You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jay Kreps <ja...@confluent.io> on 2017/01/09 19:19:00 UTC
Re: [DISCUSS] KIP-82 - Add Record Headers
Hey Michael,
Sorry for lagging, I've been out for the holiday break. What would help me
would be to get to a consolidated list of use cases we've worked out and
think add real value. I think getting down to the details really matters
for these.
The two lists I'm aware of are the one in the KIP and the one you gave in
this email thread. Both seem to be more at the sketch level. Let me review
the use cases and see if I have it right. If so let's get it in the design
proposal.
Your use cases:
1. Tracing IDs -- I get this use case and think I see how it would work.
I think because the tracing is from a third party system there is some
value to having a header versus a custom format to make this work fully
transparently.
2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
thinking below
3. End-to-end encryption -- I think what you are saying is that if you
want "routing' fields that are accessible in transit without decrypting you
need a scheme to leave this decrypted.I think the idea is to avoid
decrypting to get a header, but I'm not sure if I get how this is better
than a custom message format that doesn't encrypt the things you don't want
encrypted since you have to be aware of the scheme anyway.
4. EOS -- I'm not convinced this is a good approach for EOS.
Linked from KIP:
1. Schema-Registry-aware avro serialization -- not clear how this is
better than keeping the schema id in the value
2. Polyglot kafka topics -- marginally better than a container format,
but of course you still have to all agree on the the header and
header=>serialization mapping so not sure if it's really better than a
container format. Also not wild about polyglot topics :-)
3. End-to-End routing trail including mirroring --- I'd put this under
tracing ids
4. Cross-service process tracing -- ditto
5. Signing -- I like this use case but it seems to depend on a feature
we don't have and aren't proposing adding, right?
6. Per-record custom compression -- why do you want this? compressing
per message won't work well compared to batch compression, right?
7. Large Message Support -- I think we can agree that although this
might work it's clearly not the right way to support large messages in Kafka
Matthias
1. Control messages --- Matthias, not sure if I get the motivation in
streams for this. If we did want to do it, wouldn't it be easier to
implement and by allowing users to send control messages (in the EOS sense)
and adding a consumer config to allow either passing these on or
suppressing them?
So I if I were to summarize I'd list the use cases as follows:
- Much better with headers: tracing IDs
- Somewhat better: end-to-end encryption, polyglot topics
- Don't buy: Mirroring (though perhaps could be made to work with
changes to the proposal), EOS, large messages, compression
- Not sure: Control messages
Does this match where other people are at? From my point of view for
something like this where we are changing the core data model we should
ideally get a handful of use cases to the point where they are really well
thought out and we know they can use the feature. I think we have one solid
one with tracing IDs, and I think maybe mirroring could be another with
some work. I'm less sure about the others.
Bi-directional Mirroring
I think bidirectional mirroring could potentially become a compelling use
case if we an find a way to make it work. If I understand it, the proposal
is to add a header like "origin-cluster" with the cluster id of the Kafka
cluster. Then modify the mirroring tool to disallow mirroring a message
back to it's own origin cluster. This basic idea is definitely workable and
useful, but can we really get to a good implementation using headers?
I imagine the approach would be to add this header in the producer if some
config is set. There are two problems:
- You add a pretty big honking cluster ID to each message. For small
messages this maybe larger than the contents itself.
- What happens if someone somewhere forgets to set that config on some
client somewhere?
If someone forgets to set the config in some client somewhere I guess you
have various options in mirror maker:
1. drop the data (pretty bad, now your data is out of sync)
2. crash (pretty bad--how do you recover)
3. insert some default (pretty bad since would also lead to data loss if
that's not right)
Basically although I think a cluster tag could be a good way to do
bi-directional mirroring, I'm not sure this naive approach I described
would get us to a usable implementation. This could be fixable. I think if
there was some scheme to check required headers on the server that could
alleviate the missing header problem. Not really sure what to do about the
message bloat problem, but maybe there could be some way to shrink the
cluster id?
Cheers,
-Jay
On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Mi...@ig.com>
wrote:
> @Jay, Ismael and a few others.
>
> Based on the discussion on kip-87, and the additional use case by Matthias.
>
> Just for clarity do we now have consensus on the earlier points (1)
> headers are useful (2) headers should be native in Kafka?
>
> Cheers
> Mike
> ________________________________________
> From: radai <ra...@gmail.com>
> Sent: Monday, December 19, 2016 5:01 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> having said that im perfectly fine with something like a "no payload" flag
> (as opposed to null payload), which would mean such a msg is not returned
> from poll() unless the user code specifically "opts in" and would serve
> only to carry headers (making it a control message)
>
> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com> wrote:
>
> > if all you want is to tag "end of stream" wouldnt a header on the last
> msg
> > in the stream be enough?
> >
> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> >> The details about headers for control messages are still to define. But
> >> yes, the idea is to have some common default behavior that clients would
> >> need to implement.
> >>
> >> The point is, that "regular headers" add meta data to regular messages.
> >> Thus, those messages will be returned to the user via .poll(). And after
> >> the message is received the user can check if meta data is present and
> >> read it.
> >>
> >> For control messages, we do not want those to pop up via .poll() as
> >> those are no regular messages. A client would need to opt-in to see
> >> those messages (either via poll() or maybe a callback). Thus, we need
> >> some special (standardized) header IDs that indicate control messages
> >> that should not be returned to the user via poll() by default.
> >>
> >>
> >> -Matthias
> >>
> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
> >> > Matthias,
> >> >
> >> > Thanks for your input. I'm +1 on control messages as they seem to be
> >> the
> >> > simplest way to implement watermarks (
> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102),
> a
> >> > feature that would add a lot of value to Kafka Streams IMHO.
> >> >
> >> > Your argument that the control-message indicator belongs in the
> >> client-only
> >> > section of the record format make sense. Just to make sure I
> >> understand,
> >> > are you suggesting that control messages would be indicated by a
> >> standard
> >> > reserved header? By standard, I mean that ALL Kafka consumers would
> >> know
> >> > to handle these messages differently (possibly just ignoring them).
> >> This
> >> > would need to be added to the specification of the consumer protocol
> so
> >> > that all Kafka clients implement it, right? I think it's a good idea
> >> but
> >> > just checking.
> >> >
> >> > Cheers,
> >> >
> >> > Roger
> >> >
> >> >
> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> matthias@confluent.io
> >> >
> >> > wrote:
> >> >
> >> >> Yes and no. I did overload the term "control message".
> >> >>
> >> >> EOS control messages are for client-broker communication and thus
> never
> >> >> exposed to any application. And I think this is a good design because
> >> >> broker needs to understand those control messages. Thus, this should
> be
> >> >> a protocol change.
> >> >>
> >> >> The type of control messages I have in mind are for client-client
> >> >> (application-application) communication and the broker is agnostic to
> >> >> them. Thus, it should not be a protocol change.
> >> >>
> >> >>
> >> >> -Matthias
> >> >>
> >> >>
> >> >>
> >> >> On 12/14/16 9:42 AM, radai wrote:
> >> >>> arent control messages getting pushed as their own top level
> protocol
> >> >>> change (and a fairly massive one) for the transactions KIP ?
> >> >>>
> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> >> matthias@confluent.io>
> >> >>> wrote:
> >> >>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> I want to add a completely new angle to this discussion. For this,
> I
> >> >>>> want to propose an extension for the headers feature that enables
> new
> >> >>>> uses cases -- and those new use cases might convince people to
> >> support
> >> >>>> headers (of course including the larger scoped proposal).
> >> >>>>
> >> >>>> Extended Proposal:
> >> >>>>
> >> >>>> Allow messages with a certain header key to be special "control
> >> >>>> messages" (w/ o w/o payload) that are not exposed to an application
> >> via
> >> >>>> .poll().
> >> >>>>
> >> >>>> Thus, a consumer client would automatically skip over those
> >> messages. If
> >> >>>> an application knows about embedded control messages, it can "sing
> >> up"
> >> >>>> to those messages by the consumer client and either get a callback
> or
> >> >>>> the consumer auto-drop for this messages gets disabled (allowing to
> >> >>>> consumer those messages via poll()).
> >> >>>>
> >> >>>> (The details need further considerations/discussion. I just want to
> >> >>>> sketch the main idea.)
> >> >>>>
> >> >>>> Usage:
> >> >>>>
> >> >>>> There is a shared topic (ie, used by multiple applications) and a
> >> >>>> producer application wants to embed a special message in the topic
> >> for a
> >> >>>> dedicated consumer application. Because only one application will
> >> >>>> understand this message, it cannot be a regular message as this
> would
> >> >>>> break all applications that do not understand this message. The
> >> producer
> >> >>>> application would set a special metadata key and no consumer
> >> application
> >> >>>> would see this control message by default because they did not
> enable
> >> >>>> their consumer client to return this message in poll() (and the
> >> client
> >> >>>> would just drop this message with special metadata key). Only the
> >> single
> >> >>>> application that should receive this message, will subscribe to
> this
> >> >>>> message on its consumer client and process it.
> >> >>>>
> >> >>>>
> >> >>>> Concrete Use Case: Kafka Streams
> >> >>>>
> >> >>>> In Kafka Streams, we would like to propagate "control messages"
> from
> >> >>>> subtopology to subtopology. There are multiple scenarios for which
> >> this
> >> >>>> would be useful. For example, currently we do not guarantee a
> >> >>>> "consistent shutdown" of an application. By this, I mean that input
> >> >>>> records might not be completely processed by the whole topology
> >> because
> >> >>>> the application shutdown happens "in between" and an intermediate
> >> result
> >> >>>> topic gets "stock" in an intermediate topic. Thus, a user would see
> >> an
> >> >>>> committed offset of the source topic of the application, but no
> >> >>>> corresponding result record in the output topic.
> >> >>>>
> >> >>>> Having "shutdown markers" would allow us, to first stop the
> upstream
> >> >>>> subtopology and write this marker into the intermediate topic and
> the
> >> >>>> downstream subtopology would only shut down itself after is sees
> the
> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
> >> >>>> "in-flight" messages got stuck in intermediate topics.
> >> >>>>
> >> >>>>
> >> >>>> A similar usage would be for KIP-95 (Incremental Batch Processing).
> >> >>>> There was a discussion about the proposed metadata topic, and we
> >> could
> >> >>>> avoid this metadata topic if we would have "control messages".
> >> >>>>
> >> >>>>
> >> >>>> Right now, we cannot insert an "application control message"
> because
> >> >>>> Kafka Streams does not own all topics it read/writes and thus might
> >> >>>> break other consumer application (as described above) if we inject
> >> >>>> random messages that are not understood by other apps.
> >> >>>>
> >> >>>>
> >> >>>> Of course, one can work around "embedded control messaged" by using
> >> an
> >> >>>> additional topic to propagate control messaged between application
> >> (as
> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
> >> there
> >> >>>> are major concerns about adding this metadata topic in the KIP and
> >> this
> >> >>>> shows that other application that need a similar pattern might
> profit
> >> >>>> from topic embedded "control messages", too.
> >> >>>>
> >> >>>>
> >> >>>> One last important consideration: those "control messages" are used
> >> for
> >> >>>> client to client communication and are not understood by the
> broker.
> >> >>>> Thus, those messages should not be enabled within the message
> format
> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
> >> headers
> >> >>>> would be a nice way to implement them. Because KIP-82 did consider
> >> key
> >> >>>> namespaces for metatdata keys, this extension should not be an own
> >> KIP
> >> >>>> but should be included in KIP-82 to reserve a namespace for
> "control
> >> >>>> message" in the first place.
> >> >>>>
> >> >>>>
> >> >>>> Sorry for the long email... Looking forward to your feedback.
> >> >>>>
> >> >>>>
> >> >>>> -Matthias
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> >> >>>>> Hi Jun
> >> >>>>>
> >> >>>>> 100) each time a transaction exits a jvm for a remote system
> >> (HTTP/JMS/
> >> >>>> Hopefully one day kafka) the APM tools stich in a unique id
> (though I
> >> >>>> believe it contains the end2end uuid embedded in this id), on
> >> receiving
> >> >> the
> >> >>>> message at the receiving JVM the apm code takes this out, and
> >> continues
> >> >> its
> >> >>>> tracing on the that new thread. Both JVM’s (and other languages the
> >> APM
> >> >>>> tool supports) send this data async back to the central controllers
> >> >> where
> >> >>>> the stiching togeather occurs. For this they need some header space
> >> for
> >> >>>> them to put this id.
> >> >>>>>
> >> >>>>> 101) Yes indeed we have a business transaction Id in the payload.
> >> >> Though
> >> >>>> this is a system level tracing, that we need to have marry up. Also
> >> as
> >> >> per
> >> >>>> note on end2end encryption we’d be unable to prove the flow if the
> >> >> payload
> >> >>>> is encrypted as we’d not have access to this at certain points of
> the
> >> >> flow
> >> >>>> through the infrastructure/platform.
> >> >>>>>
> >> >>>>>
> >> >>>>> 103) As said we use this mechanism in IG very successfully, as
> >> stated
> >> >>>> per key we guarantee the transaction producing app to handle the
> >> >>>> transaction of a key at one DC unless at point of critical failure
> >> >> where we
> >> >>>> have to flip processing to another. We care about key ordering.
> >> >>>>> I disagree on the offset comment for the partition solution unless
> >> you
> >> >>>> do full ISR, or expensive full XA transactions even with partitions
> >> you
> >> >>>> cannot fully guarantee offsets would match.
> >> >>>>>
> >> >>>>> 105) Very much so, I need to have access at the platform level to
> >> the
> >> >>>> other meta data all mentioned, without having to need to have
> access
> >> to
> >> >> the
> >> >>>> encryption keys of the payload.
> >> >>>>>
> >> >>>>> 106)
> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d need to have
> a
> >> >>>> global producerId register which would be very hard to
> >> enforce/ensure is
> >> >>>> current and correct, just to understand the message origins of its
> >> >>>> region/az/cluster for routing.
> >> >>>>> The client wrapper version, producerId can be the same, as
> obviously
> >> >> the
> >> >>>> producer could upgrade its wrapper, as such we need to know what
> >> wrapper
> >> >>>> version the message is created with.
> >> >>>>> Likewise the IP address, as stated we can have our producer move,
> >> where
> >> >>>> its IP would change.
> >> >>>>>
> >> >>>>> 107)
> >> >>>>> UUID is set on the message by interceptors before actual producer
> >> >>>> transport send. This is for platform level message dedupe
> guarantee,
> >> the
> >> >>>> business payload should be agnostic to this. Please see
> >> >>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
> >> ection.html
> >> >>>> note this is not touching business payloads.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
> >> >>>>>
> >> >>>>> Hi, Michael,
> >> >>>>>
> >> >>>>> Thanks for the reply. I find it very helpful.
> >> >>>>>
> >> >>>>> Data lineage:
> >> >>>>> 100. I'd like to understand the APM use case a bit more. It
> >> sounds
> >> >>>> like
> >> >>>>> that those APM plugins can generate a transaction id that we
> >> could
> >> >>>>> potentially put in the header of every message. How would you
> >> >>>> typically
> >> >>>>> make use of such transaction ids? Are there other metadata
> >> >>>> associated with
> >> >>>>> the transaction id and if so, how are they propagated
> >> downstream?
> >> >>>>>
> >> >>>>> 101. For the finance use case, if the concept of transaction
> is
> >> >>>> important,
> >> >>>>> wouldn't it be typically included in the message payload
> >> instead of
> >> >>>> as an
> >> >>>>> optional header field?
> >> >>>>>
> >> >>>>> 102. The data lineage that Altas and Navigator support seems
> to
> >> be
> >> >>>> at the
> >> >>>>> dataset level, not per record level? So, not sure if per
> message
> >> >>>> headers
> >> >>>>> are relevant there.
> >> >>>>>
> >> >>>>> Mirroring:
> >> >>>>> 103. The benefit of using separate partitions is that it
> >> >> potentially
> >> >>>> makes
> >> >>>>> it easy to preserve offsets during mirroring. This will make
> it
> >> >>>> easier for
> >> >>>>> consumer to switch clusters. Currently, the consumers can
> switch
> >> >>>> clusters
> >> >>>>> by using the timestampToOffset() api, but it has to deal with
> >> >>>> duplicates.
> >> >>>>> Good point on the issue with log compact and I am not sure how
> >> to
> >> >>>> address
> >> >>>>> this. However, even if we mirror into the existing partitions,
> >> the
> >> >>>> ordering
> >> >>>>> for messages generated from different clusters seems
> >> >>>> non-deterministic
> >> >>>>> anyway. So, it seems that the consumers already have to deal
> >> with
> >> >>>> that? If
> >> >>>>> a topic is compacted, does that mean which messages are
> >> preserved
> >> >> is
> >> >>>> also
> >> >>>>> non-deterministic across clusters?
> >> >>>>>
> >> >>>>> 104. Good point on partition key.
> >> >>>>>
> >> >>>>> End-to-end encryption:
> >> >>>>> 105. So, it seems end-to-end encryption is useful. Are headers
> >> >>>> useful there?
> >> >>>>>
> >> >>>>> Auditing:
> >> >>>>> 106. It seems other than the UUID, all other metadata are per
> >> >>>> producer?
> >> >>>>>
> >> >>>>> EOS:
> >> >>>>> 107. How are those UUIDs generated? I am not sure if they can
> be
> >> >>>> generated
> >> >>>>> in the producer library. An application may send messages
> >> through a
> >> >>>> load
> >> >>>>> balancer and on retry, the same message could be routed to a
> >> >>>> different
> >> >>>>> producer instance. So, it seems that the application has to
> >> >> generate
> >> >>>> the
> >> >>>>> UUIDs. In that case, shouldn't the application just put the
> >> UUID in
> >> >>>> the
> >> >>>>> payload?
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>>
> >> >>>>> Jun
> >> >>>>>
> >> >>>>>
> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> >> >>>> Michael.Pearce@ig.com>
> >> >>>>> wrote:
> >> >>>>>
> >> >>>>> > Hi Jun.
> >> >>>>> >
> >> >>>>> > Per Transaction Tracing / Data Lineage.
> >> >>>>> >
> >> >>>>> > As Stated in the KIP this has the first use case of how many
> >> APM
> >> >>>> tools now
> >> >>>>> > work.
> >> >>>>> > I would find it impossible for any one to argue this is not
> >> >>>> important or a
> >> >>>>> > niche market as it has its own gartner report for this
> space.
> >> >> Such
> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular are
> >> but a
> >> >>>> few.
> >> >>>>> >
> >> >>>>> > Likewise these APM tools can help very rapidly track down
> >> issues
> >> >>>> and
> >> >>>>> > automatically capture metrics, perform actions based on
> >> >> unexpected
> >> >>>> behavior
> >> >>>>> > to auto recover services.
> >> >>>>> >
> >> >>>>> > Before mentioning looking at aggregated stats, in these
> cases
> >> >> where
> >> >>>>> > actually on critical flows we cannot afford to have
> aggregated
> >> >>>> rolled up
> >> >>>>> > stats only.
> >> >>>>> >
> >> >>>>> > With the APM tool we use its actually able to detect a
> single
> >> >>>> transaction
> >> >>>>> > failure and capture the thread traces in the JVM where it
> >> failed
> >> >>>> and
> >> >>>>> > everything for us, to the point it sends us alerts where we
> >> have
> >> >>>> this
> >> >>>>> > giving the line number of the code that caused it, the
> >> >> transaction
> >> >>>> trace
> >> >>>>> > through all the services and endpoints (supported) upto the
> >> point
> >> >>>> of
> >> >>>>> > failure, it can also capture the data in and out (so we can
> >> >>>> replay).
> >> >>>>> > Because atm Kafka doesn’t support us being able to stich in
> >> these
> >> >>>> tracing
> >> >>>>> > transaction ids natively, we cannot get these benefits as
> >> such is
> >> >>>> limiting
> >> >>>>> > our ability support apps and monitor them to the same
> >> standards
> >> >> we
> >> >>>> come to
> >> >>>>> > expect when on a kafka flow.
> >> >>>>> >
> >> >>>>> > This actually ties in with Data Lineage, as the same tracing
> >> can
> >> >>>> be used
> >> >>>>> > to back stich this. Essentially many times due to the sums
> of
> >> >> money
> >> >>>>> > involved there are disputes, and typically as a financial
> >> >>>> institute the
> >> >>>>> > easiest and cleanest way to prove when disputes arise is to
> >> >>>> present the
> >> >>>>> > actual flow and processes involved in a transaction.
> >> >>>>> >
> >> >>>>> > Likewise as Hadoop matures its evident this case is
> >> important, as
> >> >>>> tools
> >> >>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera led)
> >> are
> >> >>>> evident
> >> >>>>> > also I believe the importance here is very much NOT just a
> >> >>>> financial issue.
> >> >>>>> >
> >> >>>>> > From a MDM point of view any company wanting to care about
> >> Data
> >> >>>> Quality
> >> >>>>> > and Data Governance - Data Lineage is a key piece in this
> >> puzzle.
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > RE Mirroring,
> >> >>>>> >
> >> >>>>> > As per the KIP in-fact this is exactly what we do re cluster
> >> id,
> >> >>>> to mirror
> >> >>>>> > a network of clusters between AZ’s / Regions. We know a
> >> >>>> transaction for a
> >> >>>>> > key will be done within a AZ/Region, as such we know the
> >> write
> >> >> to
> >> >>>> kafka
> >> >>>>> > would be ordered per key. But we need eventual view of that
> >> >> across
> >> >>>> in our
> >> >>>>> > other regions/az’s. When we have complete AZ or Region
> >> failure we
> >> >>>> know
> >> >>>>> > there will be a brief interruption whilst those transactions
> >> are
> >> >>>> moved to
> >> >>>>> > another region but we expect after it to continue.
> >> >>>>> >
> >> >>>>> > As mentioned having separate Partions to do this starts to
> get
> >> >>>>> > ugly/complicated for us:
> >> >>>>> > how would I do compaction where a key is in two partitions?
> >> >>>>> > How do we balance consumers so where multiple partitions
> with
> >> the
> >> >>>> same key
> >> >>>>> > goto the same consumer
> >> >>>>> > What do you do if cluster 1 has 5 partitions but cluster 20
> >> has
> >> >> 10
> >> >>>> because
> >> >>>>> > its larger kit in our more core DC’s, as such key to
> partition
> >> >>>> mappings for
> >> >>>>> > consumers get even more complicated.
> >> >>>>> > What do you do if we add or remove a complete region
> >> >>>>> >
> >> >>>>> > Where as simple mirror will work we just need to ensure we
> >> don’t
> >> >>>> have a
> >> >>>>> > cycle which we can do with clusterId.
> >> >>>>> >
> >> >>>>> > We even have started to look at shortest path mirror routing
> >> >> based
> >> >>>> on
> >> >>>>> > clusterId, if we also had the region and az info on the
> >> >> originating
> >> >>>>> > message, this we have not implemented but some ideas come
> from
> >> >>>> network
> >> >>>>> > routing, and also the dispatcher router in apache qpid.
> >> >>>>> >
> >> >>>>> > Also we need to have data perimeters e.g. certain data
> cannot
> >> >> leave
> >> >>>>> > certain countries borders. We want this all automated so
> that
> >> at
> >> >>>> the
> >> >>>>> > platform level without having to touch or look at the
> business
> >> >>>> data inside
> >> >>>>> > we can have headers we can put tags into so that we can
> ensure
> >> >>>> this doesn’t
> >> >>>>> > occur when we mirror. (actually links in to data lineage /
> >> >> tracing
> >> >>>> as again
> >> >>>>> > we need to tag messages at a platform level) Examples are we
> >> are
> >> >>>> not
> >> >>>>> > allowed Private customer details to leave Switzerland, yet
> we
> >> >> need
> >> >>>> those
> >> >>>>> > systems integrated.
> >> >>>>> >
> >> >>>>> > Lastly around mirroring we have a partionKey field, as the
> key
> >> >>>> used for
> >> >>>>> > portioning logic != compaction key all the time but we want
> to
> >> >>>> preserve it
> >> >>>>> > for when we mirror so that if source cluster partition count
> >> !=
> >> >>>> destination
> >> >>>>> > cluster partition count we can honour the same partitioning
> >> >> logic.
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > RE End 2 End encryption
> >> >>>>> >
> >> >>>>> > As I believe mentioned just before, the solution you mention
> >> just
> >> >>>> doesn’t
> >> >>>>> > cut the mustard these days with many regulators. An
> operations
> >> >>>> person with
> >> >>>>> > access to the box should not be able to have access to the
> >> data.
> >> >>>> Many now
> >> >>>>> > actually impose quite literally the implementation expected
> >> being
> >> >>>> end2end
> >> >>>>> > encryption for certain data (Singapore for us is one that I
> am
> >> >>>> most aware
> >> >>>>> > of). In fact we’re even now needing encrypt the data and
> store
> >> >> the
> >> >>>> keys in
> >> >>>>> > HSM modules.
> >> >>>>> >
> >> >>>>> > Likewise the performance penalty on encrypting decrypting as
> >> you
> >> >>>> produce
> >> >>>>> > over wire, then again encrypt decrypt as the data is stored
> on
> >> >> the
> >> >>>> brokers
> >> >>>>> > disks and back again, then again encrypted and decrypted
> back
> >> >> over
> >> >>>> the wire
> >> >>>>> > each time for each consumer all adds up, ignoring this
> >> doubling
> >> >>>> with mirror
> >> >>>>> > makers etc. simply encrypting the value once on write by the
> >> >>>> client and
> >> >>>>> > again decrypting on consume by the consumer is far more
> >> >>>> performant, but
> >> >>>>> > then the routing and platform meta data needs to be separate
> >> >> (thus
> >> >>>> headers)
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > RE Auditing:
> >> >>>>> >
> >> >>>>> > Our Auditing needs are:
> >> >>>>> > Producer Id,
> >> >>>>> > Origin Cluster Id that message first produced into
> >> >>>>> > Origin AZ – agreed we can derive this if we have cluster id,
> >> but
> >> >>>> it makes
> >> >>>>> > resolving this for audit reporting a lot easier.
> >> >>>>> > Origin Region – agreed we can derive this if we have cluster
> >> id,
> >> >>>> but it
> >> >>>>> > makes resolving this for audit reporting a lot easier.
> >> >>>>> > Unique Message Identification (this is not the same as
> >> >> transaction
> >> >>>>> > tracing) – note offset and partition are not the same, as
> >> when we
> >> >>>> mirror or
> >> >>>>> > have for what ever system failure duplicate send,
> >> >>>>> > Custom Client wrapper version (where organizations have to
> >> wrap
> >> >>>> the kafka
> >> >>>>> > client for added features) so we know what version of the
> >> wrapper
> >> >>>> is used
> >> >>>>> > Producer IP address (in case of clients being in our vm/open
> >> >> stack
> >> >>>> infra
> >> >>>>> > where they can move around, producer id will stay the same
> but
> >> >>>> this would
> >> >>>>> > change)
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > RE Once and only once delivery case
> >> >>>>> >
> >> >>>>> > Using the same Message UUID for auditing we can achieve this
> >> >> quite
> >> >>>> simply.
> >> >>>>> >
> >> >>>>> > As per how some other brokers do this (cough qpid, artemis)
> >> >>>> message uuid
> >> >>>>> > are used to dedupe where message is sent and produced but
> the
> >> >>>> client didn’t
> >> >>>>> > receive the ack, and there for replays the send, by having a
> >> >>>> unique message
> >> >>>>> > id per message, this can be filtered out, on consumers where
> >> >>>> message
> >> >>>>> > delivery may occur twice for what ever reasons a message
> uuid
> >> can
> >> >>>> be used
> >> >>>>> > to remove duplicates being deliverd , like wise we can do
> >> this in
> >> >>>> the
> >> >>>>> > mirrormakers so if we detect a dupe message we can avoid
> >> >>>> replicating it.
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > Cheers
> >> >>>>> > Mike
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io> wrote:
> >> >>>>> >
> >> >>>>> > Since this KIP affects message format, wire protocol,
> >> apis, I
> >> >>>> think
> >> >>>>> > it's
> >> >>>>> > worth spending a bit more time to nail down the concrete
> >> use
> >> >>>> cases. It
> >> >>>>> > would be bad if we add this feature, but when start
> >> >>>> implementing it
> >> >>>>> > for say
> >> >>>>> > mirroring, we then realize that header is not the best
> >> >>>> approach.
> >> >>>>> > Initially,
> >> >>>>> > I thought I was convinced of the use cases of headers
> and
> >> was
> >> >>>> trying to
> >> >>>>> > write down a few use cases to convince others. That's
> >> when I
> >> >>>> became
> >> >>>>> > less
> >> >>>>> > certain. For me to be convinced, I just want to see two
> >> >> strong
> >> >>>> use
> >> >>>>> > cases
> >> >>>>> > (instead of 10 maybe use cases) in the third-party
> space.
> >> The
> >> >>>> reason is
> >> >>>>> > that when we discussed the use cases within a company,
> >> often
> >> >>>> it ends
> >> >>>>> > with
> >> >>>>> > "we can't force everyone to use this standard since we
> may
> >> >>>> have to
> >> >>>>> > integrate with third-party tools".
> >> >>>>> >
> >> >>>>> > At present, I am not sure why headers are useful for
> >> things
> >> >>>> like
> >> >>>>> > schemaId
> >> >>>>> > or encryption. In order to do anything useful to the
> >> value,
> >> >>>> one needs
> >> >>>>> > to
> >> >>>>> > know the schemaId or how data is encrypted, but header
> is
> >> >>>> optional.
> >> >>>>> > But, I
> >> >>>>> > can be convinced if someone (Radai, Sean, Todd?)
> provides
> >> >> more
> >> >>>> details
> >> >>>>> > on
> >> >>>>> > the argument.
> >> >>>>> >
> >> >>>>> > I am not very sure header is the best approach for
> >> mirroring
> >> >>>> either. If
> >> >>>>> > someone has thought about this more, I'd be happy to
> hear.
> >> >>>>> >
> >> >>>>> > I can see the data lineage use case. I am just not sure
> >> how
> >> >>>> widely
> >> >>>>> > applicable this is. If someone familiar with this space
> >> can
> >> >>>> justify
> >> >>>>> > this is
> >> >>>>> > a significant use case, say in the finance industry,
> this
> >> >>>> would be a
> >> >>>>> > strong
> >> >>>>> > use case.
> >> >>>>> >
> >> >>>>> > I can see the auditing use case. I am just not sure if a
> >> >> native
> >> >>>>> > producer id
> >> >>>>> > solves that problem. If there are additional metadata
> >> that's
> >> >>>> worth
> >> >>>>> > collecting but not covered by the producer id, that
> would
> >> >> make
> >> >>>> this a
> >> >>>>> > strong use case.
> >> >>>>> >
> >> >>>>> > Thanks,
> >> >>>>> >
> >> >>>>> > Jun
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> >> >>>> radai.rosenblatt@gmail.com>
> >> >>>>> > wrote:
> >> >>>>> >
> >> >>>>> > > this KIP is about enabling headers, nothing more
> nothing
> >> >>>> less - so
> >> >>>>> > no,
> >> >>>>> > > broker-side use of headers is not in the KIP scope.
> >> >>>>> > >
> >> >>>>> > > obviously though, once you have headers potential use
> >> cases
> >> >>>> could
> >> >>>>> > include
> >> >>>>> > > broker-side header-aware interceptors (which would be
> >> the
> >> >>>> topic of
> >> >>>>> > other
> >> >>>>> > > future KIPs).
> >> >>>>> > >
> >> >>>>> > > a trivially clear use case (to me) would be using such
> >> >>>> broker-side
> >> >>>>> > > interceptors to enforce compliance with organizational
> >> >>>> policies - it
> >> >>>>> > would
> >> >>>>> > > make our SREs lives much easier if instead of
> >> retroactively
> >> >>>>> > discovering
> >> >>>>> > > "rogue" topics/users those messages would have been
> >> >> rejected
> >> >>>>> > up-front.
> >> >>>>> > >
> >> >>>>> > > the kafka broker code is lacking any such
> extensibility
> >> >>>> support
> >> >>>>> > (beyond
> >> >>>>> > > maybe authorizer) which is why these use cases were
> left
> >> >> out
> >> >>>> of the
> >> >>>>> > "case
> >> >>>>> > > for headers" doc - broker extensibility is a separate
> >> >>>> discussion.
> >> >>>>> > >
> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
> >> >>>> gwen@confluent.io>
> >> >>>>> > wrote:
> >> >>>>> > >
> >> >>>>> > > > Woah, I wasn't aware this is something we'll do. It
> >> >> wasn't
> >> >>>> in the
> >> >>>>> > KIP,
> >> >>>>> > > > right?
> >> >>>>> > > >
> >> >>>>> > > > I guess we could do it the same way ACLs currently
> >> work.
> >> >>>>> > > > I had in mind something that will allow admins to
> >> apply
> >> >>>> rules to
> >> >>>>> > the
> >> >>>>> > > > new create/delete/config topic APIs. So Todd can
> >> decide
> >> >> to
> >> >>>> reject
> >> >>>>> > > > "create topic" requests that ask for more than 40
> >> >>>> partitions, or
> >> >>>>> > > > require exactly 3 replicas, or no more than 50GB
> >> >> partition
> >> >>>> size,
> >> >>>>> > etc.
> >> >>>>> > > >
> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are planning to
> >> apply
> >> >>>> more
> >> >>>>> > rules
> >> >>>>> > > > to requests (and I think we should), we may want a
> bit
> >> >>>> more generic
> >> >>>>> > > > design around that.
> >> >>>>> > > >
> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> >> >>>> radai.rosenblatt@gmail.com>
> >> >>>>> > > wrote:
> >> >>>>> > > > > "wouldn't you be in the business of making sure
> >> >> everyone
> >> >>>> uses
> >> >>>>> > them
> >> >>>>> > > > > properly?"
> >> >>>>> > > > >
> >> >>>>> > > > > thats where a broker-side plugin would come handy
> -
> >> any
> >> >>>> incoming
> >> >>>>> > > message
> >> >>>>> > > > > that does not conform to org policy (read - does
> not
> >> >>>> have the
> >> >>>>> > proper
> >> >>>>> > > > > headers) gets thrown out (with an error returned
> to
> >> >> user)
> >> >>>>> > > > >
> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
> >> >>>> tpalino@gmail.com>
> >> >>>>> > wrote:
> >> >>>>> > > > >
> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on this one
> :)
> >> >>>>> > > > >>
> >> >>>>> > > > >> Producing counts to a topic is part of it, but
> >> that’s
> >> >>>> only
> >> >>>>> > part. So
> >> >>>>> > > you
> >> >>>>> > > > >> count you have 100 messages in topic A. When you
> >> >> mirror
> >> >>>> topic A
> >> >>>>> > to
> >> >>>>> > > > another
> >> >>>>> > > > >> cluster, you have 99 messages. Where was your
> >> problem?
> >> >>>> Or
> >> >>>>> > worse, you
> >> >>>>> > > > have
> >> >>>>> > > > >> 100 messages, but one producer duplicated
> messages
> >> and
> >> >>>> another
> >> >>>>> > one
> >> >>>>> > > lost
> >> >>>>> > > > >> messages. You need details about where the
> message
> >> >> came
> >> >>>> from in
> >> >>>>> > order
> >> >>>>> > > to
> >> >>>>> > > > >> pinpoint problems when they happen. Source
> producer
> >> >>>> info, where
> >> >>>>> > it was
> >> >>>>> > > > >> produced into your infrastructure, and when it
> was
> >> >>>> produced.
> >> >>>>> > This
> >> >>>>> > > > requires
> >> >>>>> > > > >> you to add the information to the message.
> >> >>>>> > > > >>
> >> >>>>> > > > >> And yes, you still need to maintain your clients.
> >> So
> >> >>>> maybe my
> >> >>>>> > original
> >> >>>>> > > > >> example was not the best. My thoughts on not
> >> wanting
> >> >> to
> >> >>>> be
> >> >>>>> > responsible
> >> >>>>> > > > for
> >> >>>>> > > > >> message formats stands, because that’s very much
> >> >>>> separate from
> >> >>>>> > the
> >> >>>>> > > > client.
> >> >>>>> > > > >> As you know, we have our own internal client
> >> library
> >> >>>> that can
> >> >>>>> > insert
> >> >>>>> > > the
> >> >>>>> > > > >> right headers, and right now inserts the right
> >> audit
> >> >>>>> > information into
> >> >>>>> > > > the
> >> >>>>> > > > >> message fields. If they exist, and assuming the
> >> >> message
> >> >>>> is Avro
> >> >>>>> > > encoded.
> >> >>>>> > > > >> What if someone wants to use JSON instead for a
> >> good
> >> >>>> reason?
> >> >>>>> > What if
> >> >>>>> > > > user X
> >> >>>>> > > > >> wants to encrypt messages, but user Y does not?
> >> >>>> Maintaining the
> >> >>>>> > client
> >> >>>>> > > > >> library is still much easier than maintaining the
> >> >>>> message
> >> >>>>> > formats.
> >> >>>>> > > > >>
> >> >>>>> > > > >> -Todd
> >> >>>>> > > > >>
> >> >>>>> > > > >>
> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
> >> >>>> gwen@confluent.io
> >> >>>>> > >
> >> >>>>> > > wrote:
> >> >>>>> > > > >>
> >> >>>>> > > > >> > Based on your last sentence, consider me
> >> convinced
> >> >> :)
> >> >>>>> > > > >> >
> >> >>>>> > > > >> > I get why headers are critical for Mirroring
> (you
> >> >>>> need tags to
> >> >>>>> > > prevent
> >> >>>>> > > > >> > loops and sometimes to route messages to the
> >> correct
> >> >>>>> > destination).
> >> >>>>> > > > >> > But why do you need headers to audit? We are
> >> >> auditing
> >> >>>> by
> >> >>>>> > producing
> >> >>>>> > > > >> > counts to a side topic (and I was under the
> >> >>>> impression you do
> >> >>>>> > the
> >> >>>>> > > > >> > same), so we never need to modify the message.
> >> >>>>> > > > >> >
> >> >>>>> > > > >> > Another thing - after we added headers,
> wouldn't
> >> you
> >> >>>> be in the
> >> >>>>> > > > >> > business of making sure everyone uses them
> >> properly?
> >> >>>> Making
> >> >>>>> > sure
> >> >>>>> > > > >> > everyone includes the right headers you need,
> not
> >> >>>> using the
> >> >>>>> > header
> >> >>>>> > > > >> > names you intend to use, etc. I don't think the
> >> >>>> "policing"
> >> >>>>> > business
> >> >>>>> > > > >> > will ever go away.
> >> >>>>> > > > >> >
> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <
> >> >>>>> > tpalino@gmail.com>
> >> >>>>> > > > wrote:
> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
> with
> >> the
> >> >>>>> > workaround.
> >> >>>>> > > Avro
> >> >>>>> > > > >> means
> >> >>>>> > > > >> > > that I have to be concerned with the format
> of
> >> the
> >> >>>> messages
> >> >>>>> > in
> >> >>>>> > > > order to
> >> >>>>> > > > >> > run
> >> >>>>> > > > >> > > the infrastructure (audit, mirroring, etc.).
> >> That
> >> >>>> means
> >> >>>>> > that I
> >> >>>>> > > have
> >> >>>>> > > > to
> >> >>>>> > > > >> > > handle the schemas, and I have to enforce
> rules
> >> >>>> about good
> >> >>>>> > > formats.
> >> >>>>> > > > >> This
> >> >>>>> > > > >> > is
> >> >>>>> > > > >> > > not something I want to be in the business
> of,
> >> >>>> because I
> >> >>>>> > should be
> >> >>>>> > > > able
> >> >>>>> > > > >> > to
> >> >>>>> > > > >> > > run a service infrastructure without needing
> >> to be
> >> >>>> in the
> >> >>>>> > weeds of
> >> >>>>> > > > >> > dealing
> >> >>>>> > > > >> > > with customer data formats.
> >> >>>>> > > > >> > >
> >> >>>>> > > > >> > > Trust me, a sizable portion of my support
> time
> >> is
> >> >>>> spent
> >> >>>>> > dealing
> >> >>>>> > > with
> >> >>>>> > > > >> > schema
> >> >>>>> > > > >> > > issues. I really would like to get away from
> >> that.
> >> >>>> Maybe
> >> >>>>> > I'd have
> >> >>>>> > > > more
> >> >>>>> > > > >> > time
> >> >>>>> > > > >> > > for other hobbies. Like writing. ;)
> >> >>>>> > > > >> > >
> >> >>>>> > > > >> > > -Todd
> >> >>>>> > > > >> > >
> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <
> >> >>>>> > gwen@confluent.io>
> >> >>>>> > > > wrote:
> >> >>>>> > > > >> > >
> >> >>>>> > > > >> > >> I'm pretty satisfied with the current
> >> workarounds
> >> >>>> (Avro
> >> >>>>> > container
> >> >>>>> > > > >> > >> format), so I'm not too excited about the
> >> extra
> >> >>>> work
> >> >>>>> > required to
> >> >>>>> > > do
> >> >>>>> > > > >> > >> headers in Kafka. I absolutely don't mind it
> >> if
> >> >>>> you do
> >> >>>>> > it...
> >> >>>>> > > > >> > >> I think the Apache convention for "good
> idea,
> >> but
> >> >>>> not
> >> >>>>> > willing to
> >> >>>>> > > > put
> >> >>>>> > > > >> > >> any work toward it" is +0.5? anyway, that's
> >> what
> >> >> I
> >> >>>> was
> >> >>>>> > trying to
> >> >>>>> > > > >> > >> convey :)
> >> >>>>> > > > >> > >>
> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino
> <
> >> >>>>> > tpalino@gmail.com>
> >> >>>>> > > > >> wrote:
> >> >>>>> > > > >> > >> > Well I guess my question for you, then, is
> >> what
> >> >>>> is
> >> >>>>> > holding you
> >> >>>>> > > > back
> >> >>>>> > > > >> > from
> >> >>>>> > > > >> > >> > full support for headers? What’s the bit
> >> that
> >> >>>> you’re
> >> >>>>> > missing
> >> >>>>> > > that
> >> >>>>> > > > >> has
> >> >>>>> > > > >> > you
> >> >>>>> > > > >> > >> > under a full +1?
> >> >>>>> > > > >> > >> >
> >> >>>>> > > > >> > >> > -Todd
> >> >>>>> > > > >> > >> >
> >> >>>>> > > > >> > >> >
> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
> >> Shapira <
> >> >>>>> > > gwen@confluent.io>
> >> >>>>> > > > >> > wrote:
> >> >>>>> > > > >> > >> >
> >> >>>>> > > > >> > >> >> I know why people who support headers
> >> support
> >> >>>> them, and
> >> >>>>> > I've
> >> >>>>> > > > seen
> >> >>>>> > > > >> > what
> >> >>>>> > > > >> > >> >> the discussion is like.
> >> >>>>> > > > >> > >> >>
> >> >>>>> > > > >> > >> >> This is why I'm asking people who are
> >> against
> >> >>>> headers
> >> >>>>> > > > (especially
> >> >>>>> > > > >> > >> >> committers) what will make them change
> >> their
> >> >>>> mind - so
> >> >>>>> > we can
> >> >>>>> > > > get
> >> >>>>> > > > >> > this
> >> >>>>> > > > >> > >> >> part over one way or another.
> >> >>>>> > > > >> > >> >>
> >> >>>>> > > > >> > >> >> If I sound frustrated it is not at Radai,
> >> Jun
> >> >>>> or you
> >> >>>>> > (Todd)...
> >> >>>>> > > > I am
> >> >>>>> > > > >> > >> >> just looking for something concrete we
> can
> >> do
> >> >>>> to move
> >> >>>>> > the
> >> >>>>> > > > >> discussion
> >> >>>>> > > > >> > >> >> along to the yummy design details (which
> is
> >> >> the
> >> >>>>> > argument I
> >> >>>>> > > > really
> >> >>>>> > > > >> am
> >> >>>>> > > > >> > >> >> looking forward to).
> >> >>>>> > > > >> > >> >>
> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
> >> Palino <
> >> >>>>> > > tpalino@gmail.com>
> >> >>>>> > > > >> > wrote:
> >> >>>>> > > > >> > >> >> > So, Gwen, to your question (even though
> >> I’m
> >> >>>> not a
> >> >>>>> > > > committer)...
> >> >>>>> > > > >> > >> >> >
> >> >>>>> > > > >> > >> >> > I have always been a strong supporter
> of
> >> >>>> introducing
> >> >>>>> > the
> >> >>>>> > > > concept
> >> >>>>> > > > >> > of an
> >> >>>>> > > > >> > >> >> > envelope to messages, which headers
> >> >>>> accomplishes. The
> >> >>>>> > > message
> >> >>>>> > > > key
> >> >>>>> > > > >> > is
> >> >>>>> > > > >> > >> >> > already an example of a piece of
> envelope
> >> >>>>> > information. By
> >> >>>>> > > > >> > providing a
> >> >>>>> > > > >> > >> >> means
> >> >>>>> > > > >> > >> >> > to do this within Kafka itself, and not
> >> >>>> relying on
> >> >>>>> > use-case
> >> >>>>> > > > >> > specific
> >> >>>>> > > > >> > >> >> > implementations, you make it much
> easier
> >> for
> >> >>>>> > components to
> >> >>>>> > > > >> > >> interoperate.
> >> >>>>> > > > >> > >> >> It
> >> >>>>> > > > >> > >> >> > simplifies development of all these
> >> things
> >> >>>> (message
> >> >>>>> > routing,
> >> >>>>> > > > >> > auditing,
> >> >>>>> > > > >> > >> >> > encryption, etc.) because each one does
> >> not
> >> >>>> have to
> >> >>>>> > reinvent
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> wheel.
> >> >>>>> > > > >> > >> >> >
> >> >>>>> > > > >> > >> >> > It also makes it much easier from a
> >> client
> >> >>>> point of
> >> >>>>> > view if
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> headers
> >> >>>>> > > > >> > >> >> are
> >> >>>>> > > > >> > >> >> > defined as part of the protocol and/or
> >> >>>> message format
> >> >>>>> > in
> >> >>>>> > > > general
> >> >>>>> > > > >> > >> because
> >> >>>>> > > > >> > >> >> > you can easily produce and consume
> >> messages
> >> >>>> without
> >> >>>>> > having
> >> >>>>> > > to
> >> >>>>> > > > >> take
> >> >>>>> > > > >> > >> into
> >> >>>>> > > > >> > >> >> > account specific cases. For example, I
> >> want
> >> >>>> to route
> >> >>>>> > > messages,
> >> >>>>> > > > >> but
> >> >>>>> > > > >> > >> >> client A
> >> >>>>> > > > >> > >> >> > doesn’t support the way audit
> implemented
> >> >>>> headers, and
> >> >>>>> > > client
> >> >>>>> > > > B
> >> >>>>> > > > >> > >> doesn’t
> >> >>>>> > > > >> > >> >> > support the way encryption or routing
> >> >>>> implemented
> >> >>>>> > headers,
> >> >>>>> > > so
> >> >>>>> > > > now
> >> >>>>> > > > >> > my
> >> >>>>> > > > >> > >> >> > application has to create some really
> >> >> fragile
> >> >>>> (my
> >> >>>>> > > autocorrect
> >> >>>>> > > > >> just
> >> >>>>> > > > >> > >> tried
> >> >>>>> > > > >> > >> >> to
> >> >>>>> > > > >> > >> >> > make that “tragic”, which is probably
> >> >>>> appropriate
> >> >>>>> > too) code
> >> >>>>> > > to
> >> >>>>> > > > >> > strip
> >> >>>>> > > > >> > >> >> > everything off, rather than just
> >> consuming
> >> >> the
> >> >>>>> > messages,
> >> >>>>> > > > picking
> >> >>>>> > > > >> > out
> >> >>>>> > > > >> > >> the
> >> >>>>> > > > >> > >> >> 1
> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested in, and
> >> >>>> performing its
> >> >>>>> > > function.
> >> >>>>> > > > >> > >> >> >
> >> >>>>> > > > >> > >> >> > Honestly, this discussion has been
> going
> >> on
> >> >>>> for a
> >> >>>>> > long time,
> >> >>>>> > > > and
> >> >>>>> > > > >> > it’s
> >> >>>>> > > > >> > >> >> > always “Oh, you came up with 2 use
> cases,
> >> >> and
> >> >>>> yeah,
> >> >>>>> > those
> >> >>>>> > > use
> >> >>>>> > > > >> cases
> >> >>>>> > > > >> > >> are
> >> >>>>> > > > >> > >> >> > real things that someone would want to
> >> do.
> >> >>>> Here’s an
> >> >>>>> > > alternate
> >> >>>>> > > > >> way
> >> >>>>> > > > >> > to
> >> >>>>> > > > >> > >> >> > implement them so let’s not do
> headers.”
> >> If
> >> >>>> we have a
> >> >>>>> > few
> >> >>>>> > > use
> >> >>>>> > > > >> cases
> >> >>>>> > > > >> > >> that
> >> >>>>> > > > >> > >> >> we
> >> >>>>> > > > >> > >> >> > actually came up with, you can be sure
> >> that
> >> >>>> over the
> >> >>>>> > next
> >> >>>>> > > year
> >> >>>>> > > > >> > >> there’s a
> >> >>>>> > > > >> > >> >> > dozen others that we didn’t think of
> that
> >> >>>> someone
> >> >>>>> > would like
> >> >>>>> > > > to
> >> >>>>> > > > >> > do. I
> >> >>>>> > > > >> > >> >> > really think it’s time to stop
> rehashing
> >> >> this
> >> >>>>> > discussion and
> >> >>>>> > > > >> > instead
> >> >>>>> > > > >> > >> >> focus
> >> >>>>> > > > >> > >> >> > on a workable standard that we can
> adopt.
> >> >>>>> > > > >> > >> >> >
> >> >>>>> > > > >> > >> >> > -Todd
> >> >>>>> > > > >> > >> >> >
> >> >>>>> > > > >> > >> >> >
> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
> >> Palino
> >> >> <
> >> >>>>> > > > tpalino@gmail.com>
> >> >>>>> > > > >> > >> wrote:
> >> >>>>> > > > >> > >> >> >
> >> >>>>> > > > >> > >> >> >> C. per message encryption
> >> >>>>> > > > >> > >> >> >>> One drawback of this approach is that
> >> this
> >> >>>>> > significantly
> >> >>>>> > > > reduce
> >> >>>>> > > > >> > the
> >> >>>>> > > > >> > >> >> >>> effectiveness of compression, which
> >> >> happens
> >> >>>> on a
> >> >>>>> > set of
> >> >>>>> > > > >> > serialized
> >> >>>>> > > > >> > >> >> >>> messages. An alternative is to enable
> >> SSL
> >> >>>> for wire
> >> >>>>> > > > encryption
> >> >>>>> > > > >> and
> >> >>>>> > > > >> > >> rely
> >> >>>>> > > > >> > >> >> on
> >> >>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS) for at
> >> rest
> >> >>>>> > encryption.
> >> >>>>> > > > >> > >> >> >>
> >> >>>>> > > > >> > >> >> >>
> >> >>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
> this
> >> >> does
> >> >>>> cover
> >> >>>>> > the case
> >> >>>>> > > > of
> >> >>>>> > > > >> > >> removing
> >> >>>>> > > > >> > >> >> a
> >> >>>>> > > > >> > >> >> >> drive from the system, it will not
> >> satisfy
> >> >>>> most
> >> >>>>> > compliance
> >> >>>>> > > > >> > >> requirements
> >> >>>>> > > > >> > >> >> for
> >> >>>>> > > > >> > >> >> >> encryption of data as whoever has
> >> access to
> >> >>>> the
> >> >>>>> > broker
> >> >>>>> > > itself
> >> >>>>> > > > >> > still
> >> >>>>> > > > >> > >> has
> >> >>>>> > > > >> > >> >> >> access to the unencrypted data. For
> >> >>>> end-to-end
> >> >>>>> > encryption
> >> >>>>> > > you
> >> >>>>> > > > >> > need to
> >> >>>>> > > > >> > >> >> >> encrypt at the producer, before it
> >> enters
> >> >> the
> >> >>>>> > system, and
> >> >>>>> > > > >> decrypt
> >> >>>>> > > > >> > at
> >> >>>>> > > > >> > >> the
> >> >>>>> > > > >> > >> >> >> consumer, after it exits the system.
> >> >>>>> > > > >> > >> >> >>
> >> >>>>> > > > >> > >> >> >> -Todd
> >> >>>>> > > > >> > >> >> >>
> >> >>>>> > > > >> > >> >> >>
> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai
> <
> >> >>>>> > > > >> radai.rosenblatt@gmail.com
> >> >>>>> > > > >> > >
> >> >>>>> > > > >> > >> >> wrote:
> >> >>>>> > > > >> > >> >> >>
> >> >>>>> > > > >> > >> >> >>> another big plus of headers in the
> >> >> protocol
> >> >>>> is that
> >> >>>>> > it
> >> >>>>> > > would
> >> >>>>> > > > >> > enable
> >> >>>>> > > > >> > >> >> rapid
> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside of core
> >> kafka
> >> >>>> and would
> >> >>>>> > reduce
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> number of
> >> >>>>> > > > >> > >> >> >>> future wire format changes required.
> >> >>>>> > > > >> > >> >> >>>
> >> >>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
> >> >> represents
> >> >>>> use
> >> >>>>> > cases that
> >> >>>>> > > > are
> >> >>>>> > > > >> > not
> >> >>>>> > > > >> > >> 100%
> >> >>>>> > > > >> > >> >> >>> relevant to all users, and some of
> them
> >> >>>> require
> >> >>>>> > rather
> >> >>>>> > > > invasive
> >> >>>>> > > > >> > wire
> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing a good
> recent
> >> >>>> example of
> >> >>>>> > this is
> >> >>>>> > > > >> > kip-98.
> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected to
> be
> >> a
> >> >>>> very small
> >> >>>>> > > > fraction of
> >> >>>>> > > > >> > >> total
> >> >>>>> > > > >> > >> >> >>> traffic and yet the changes are
> >> invasive.
> >> >>>>> > > > >> > >> >> >>>
> >> >>>>> > > > >> > >> >> >>> every such wire format change
> >> translates
> >> >>>> into
> >> >>>>> > painful and
> >> >>>>> > > > slow
> >> >>>>> > > > >> > >> >> adoption of
> >> >>>>> > > > >> > >> >> >>> new versions.
> >> >>>>> > > > >> > >> >> >>>
> >> >>>>> > > > >> > >> >> >>> i think a lot of functionality
> >> currently
> >> >> in
> >> >>>> KIPs
> >> >>>>> > could be
> >> >>>>> > > > "spun
> >> >>>>> > > > >> > out"
> >> >>>>> > > > >> > >> >> and
> >> >>>>> > > > >> > >> >> >>> implemented as opt-in plugins
> >> transmitting
> >> >>>> data over
> >> >>>>> > > > headers.
> >> >>>>> > > > >> > this
> >> >>>>> > > > >> > >> >> would
> >> >>>>> > > > >> > >> >> >>> keep the core wire format stable(r),
> >> core
> >> >>>> codebase
> >> >>>>> > > smaller,
> >> >>>>> > > > and
> >> >>>>> > > > >> > >> avoid
> >> >>>>> > > > >> > >> >> the
> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
> >> required
> >> >>>> to prove
> >> >>>>> > a
> >> >>>>> > > > certain
> >> >>>>> > > > >> > >> feature
> >> >>>>> > > > >> > >> >> is
> >> >>>>> > > > >> > >> >> >>> useful enough for a wide-enough
> >> audience
> >> >> to
> >> >>>> warrant
> >> >>>>> > a wire
> >> >>>>> > > > >> format
> >> >>>>> > > > >> > >> >> change
> >> >>>>> > > > >> > >> >> >>> and code complexity additions.
> >> >>>>> > > > >> > >> >> >>>
> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
> >> "mere"
> >> >>>> wire format
> >> >>>>> > > changes
> >> >>>>> > > > >> and
> >> >>>>> > > > >> > im
> >> >>>>> > > > >> > >> not
> >> >>>>> > > > >> > >> >> >>> saying it could have been completely
> >> done
> >> >>>> with
> >> >>>>> > headers,
> >> >>>>> > > but
> >> >>>>> > > > >> > >> >> exactly-once
> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
> >> >>>>> > > > >> > >> >> >>>
> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen
> >> >>>> Shapira <
> >> >>>>> > > > >> gwen@confluent.io
> >> >>>>> > > > >> > >
> >> >>>>> > > > >> > >> >> wrote:
> >> >>>>> > > > >> > >> >> >>>
> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
> >> radai <
> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> >> >>>>> > > > >> > >> >> >>> wrote:
> >> >>>>> > > > >> > >> >> >>> > > "For use cases within an
> >> organization,
> >> >>>> one could
> >> >>>>> > > always
> >> >>>>> > > > use
> >> >>>>> > > > >> > >> other
> >> >>>>> > > > >> > >> >> >>> > > approaches such as company-wise
> >> >>>> containers"
> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin has
> >> >> traditionally
> >> >>>> done
> >> >>>>> > but there
> >> >>>>> > > > are
> >> >>>>> > > > >> > now
> >> >>>>> > > > >> > >> >> cases
> >> >>>>> > > > >> > >> >> >>> > (read
> >> >>>>> > > > >> > >> >> >>> > > - topics) where this is not
> >> >> acceptable.
> >> >>>> this
> >> >>>>> > makes
> >> >>>>> > > > headers
> >> >>>>> > > > >> > >> useful
> >> >>>>> > > > >> > >> >> even
> >> >>>>> > > > >> > >> >> >>> > > within single orgs for cases
> where
> >> >>>>> > > > one-container-fits-all
> >> >>>>> > > > >> > cannot
> >> >>>>> > > > >> > >> >> >>> apply.
> >> >>>>> > > > >> > >> >> >>> > >
> >> >>>>> > > > >> > >> >> >>> > > as for the particular use cases
> >> >> listed,
> >> >>>> i dont
> >> >>>>> > want
> >> >>>>> > > > this to
> >> >>>>> > > > >> > >> devolve
> >> >>>>> > > > >> > >> >> >>> to a
> >> >>>>> > > > >> > >> >> >>> > > discussion of particular use
> cases
> >> - i
> >> >>>> think its
> >> >>>>> > > enough
> >> >>>>> > > > >> that
> >> >>>>> > > > >> > >> some
> >> >>>>> > > > >> > >> >> of
> >> >>>>> > > > >> > >> >> >>> them
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > I think a main point of contention
> is
> >> >>>> that: We
> >> >>>>> > > identified
> >> >>>>> > > > few
> >> >>>>> > > > >> > >> >> >>> > use-cases where headers are useful,
> >> do
> >> >> we
> >> >>>> want
> >> >>>>> > Kafka to
> >> >>>>> > > > be a
> >> >>>>> > > > >> > >> system
> >> >>>>> > > > >> > >> >> >>> > that supports those use-cases?
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely useful
> >> record-level
> >> >>>> lineage
> >> >>>>> > is
> >> >>>>> > > though
> >> >>>>> > > > >> > since
> >> >>>>> > > > >> > >> the
> >> >>>>> > > > >> > >> >> >>> > overhead could
> >> >>>>> > > > >> > >> >> >>> > be significant."
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports record level
> >> >>>> lineage. I
> >> >>>>> > don't
> >> >>>>> > > think
> >> >>>>> > > > it
> >> >>>>> > > > >> > was
> >> >>>>> > > > >> > >> >> >>> > developed for lols, I think it is
> >> safe
> >> >> to
> >> >>>> assume
> >> >>>>> > that
> >> >>>>> > > the
> >> >>>>> > > > NSA
> >> >>>>> > > > >> > >> needed
> >> >>>>> > > > >> > >> >> >>> > that functionality. We also know
> that
> >> >>>> certain
> >> >>>>> > financial
> >> >>>>> > > > >> > institutes
> >> >>>>> > > > >> > >> >> >>> > need to track tampering with
> records
> >> at
> >> >> a
> >> >>>> record
> >> >>>>> > level
> >> >>>>> > > and
> >> >>>>> > > > >> > there
> >> >>>>> > > > >> > >> are
> >> >>>>> > > > >> > >> >> >>> > federal regulations that absolutely
> >> >>>> require
> >> >>>>> > this. They
> >> >>>>> > > > also
> >> >>>>> > > > >> > need
> >> >>>>> > > > >> > >> to
> >> >>>>> > > > >> > >> >> >>> > prove that routing apps that
> >> "touches"
> >> >> the
> >> >>>>> > messages and
> >> >>>>> > > > >> either
> >> >>>>> > > > >> > >> reads
> >> >>>>> > > > >> > >> >> >>> > or updates headers couldn't have
> >> >> possibly
> >> >>>>> > modified the
> >> >>>>> > > > >> payload
> >> >>>>> > > > >> > >> >> itself.
> >> >>>>> > > > >> > >> >> >>> > They use record level encryption to
> >> do
> >> >>>> that -
> >> >>>>> > apps can
> >> >>>>> > > > read
> >> >>>>> > > > >> and
> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
> can't
> >> >>>> touch the
> >> >>>>> > payload.
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > We can totally say "those are
> corner
> >> >>>> cases and
> >> >>>>> > not worth
> >> >>>>> > > > >> adding
> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for", they should
> >> use a
> >> >>>> different
> >> >>>>> > > pubsub
> >> >>>>> > > > >> > message
> >> >>>>> > > > >> > >> for
> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the other 1000
> >> that
> >> >>>> cater
> >> >>>>> > > > specifically
> >> >>>>> > > > >> to
> >> >>>>> > > > >> > the
> >> >>>>> > > > >> > >> >> >>> > financial industry).
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > But this gets us into a catch 22:
> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific use-case,
> >> >>>> someone can
> >> >>>>> > always
> >> >>>>> > > say
> >> >>>>> > > > it
> >> >>>>> > > > >> > isn't
> >> >>>>> > > > >> > >> >> >>> > interesting enough for Kafka. If we
> >> >>>> discuss more
> >> >>>>> > general
> >> >>>>> > > > >> > trends,
> >> >>>>> > > > >> > >> >> >>> > others can say "well, we are not
> sure
> >> >> any
> >> >>>> of them
> >> >>>>> > really
> >> >>>>> > > > >> needs
> >> >>>>> > > > >> > >> >> headers
> >> >>>>> > > > >> > >> >> >>> > specifically. This is just hand
> >> waving
> >> >>>> and not
> >> >>>>> > > > interesting.".
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > I think discussing use-cases in
> >> >> specifics
> >> >>>> is super
> >> >>>>> > > > important
> >> >>>>> > > > >> to
> >> >>>>> > > > >> > >> >> decide
> >> >>>>> > > > >> > >> >> >>> > implementation details for headers
> >> (my
> >> >>>> use-cases
> >> >>>>> > lean
> >> >>>>> > > > toward
> >> >>>>> > > > >> > >> >> numerical
> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and object
> >> values,
> >> >>>> others
> >> >>>>> > differ),
> >> >>>>> > > > but I
> >> >>>>> > > > >> > >> think
> >> >>>>> > > > >> > >> >> we
> >> >>>>> > > > >> > >> >> >>> > need to answer the general "Are we
> >> going
> >> >>>> to have
> >> >>>>> > > headers"
> >> >>>>> > > > >> > question
> >> >>>>> > > > >> > >> >> >>> > first.
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from the other
> >> >>>> committers in the
> >> >>>>> > > > discussion:
> >> >>>>> > > > >> > >> >> >>> > What would it take to convince you
> >> that
> >> >>>> headers
> >> >>>>> > in Kafka
> >> >>>>> > > > are
> >> >>>>> > > > >> a
> >> >>>>> > > > >> > >> good
> >> >>>>> > > > >> > >> >> >>> > idea in general, so we can move
> ahead
> >> >> and
> >> >>>> try to
> >> >>>>> > agree
> >> >>>>> > > on
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> >> details?
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > I feel like we keep moving the goal
> >> >> posts
> >> >>>> and
> >> >>>>> > this is
> >> >>>>> > > > truly
> >> >>>>> > > > >> > >> >> exhausting.
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly support
> >> adding
> >> >>>> headers
> >> >>>>> > to Kafka
> >> >>>>> > > > >> > (+0.5?).
> >> >>>>> > > > >> > >> >> >>> > The community can continue to find
> >> >>>> workarounds to
> >> >>>>> > the
> >> >>>>> > > > issue
> >> >>>>> > > > >> and
> >> >>>>> > > > >> > >> there
> >> >>>>> > > > >> > >> >> >>> > are some benefits to keeping the
> >> message
> >> >>>> format
> >> >>>>> > and
> >> >>>>> > > > clients
> >> >>>>> > > > >> > >> simpler.
> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness of headers
> >> to
> >> >>>> many
> >> >>>>> > use-cases
> >> >>>>> > > and
> >> >>>>> > > > if
> >> >>>>> > > > >> we
> >> >>>>> > > > >> > >> can
> >> >>>>> > > > >> > >> >> >>> > find a good and generally useful
> way
> >> to
> >> >>>> add it to
> >> >>>>> > Kafka,
> >> >>>>> > > > it
> >> >>>>> > > > >> > will
> >> >>>>> > > > >> > >> make
> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
> worthy
> >> >>>> goal in my
> >> >>>>> > eyes.
> >> >>>>> > > > >> > >> >> >>> >
> >> >>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are use cases
> >> for
> >> >>>> polyglot
> >> >>>>> > topics.
> >> >>>>> > > > >> > >> especially if
> >> >>>>> > > > >> > >> >> >>> kafka
> >> >>>>> > > > >> > >> >> >>> > > is being used to "trunk"
> something
> >> >> else.
> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics would make it
> >> >> harder
> >> >>>> to write
> >> >>>>> > > > portable
> >> >>>>> > > > >> > >> consumer
> >> >>>>> > > > >> > >> >> >>> code.
> >> >>>>> > > > >> > >> >> >>> > > partition remapping would mess
> with
> >> >>>> locality of
> >> >>>>> > > > consumption
> >> >>>>> > > > >> > >> >> >>> guarantees.
> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
> >> >>>> lineage/metadata is
> >> >>>>> > > > >> > >> billing/chargeback.
> >> >>>>> > > > >> > >> >> for
> >> >>>>> > > > >> > >> >> >>> > that
> >> >>>>> > > > >> > >> >> >>> > > use case it is not enough to
> simply
> >> >>>> record the
> >> >>>>> > point
> >> >>>>> > > of
> >> >>>>> > > > >> > origin,
> >> >>>>> > > > >> > >> but
> >> >>>>> > > > >> > >> >> >>> every
> >> >>>>> > > > >> > >> >> >>> > > replication stop (think mirror
> >> maker)
> >> >>>> must also
> >> >>>>> > add a
> >> >>>>> > > > >> record
> >> >>>>> > > > >> > to
> >> >>>>> > > > >> > >> >> form a
> >> >>>>> > > > >> > >> >> >>> > > "transit log".
> >> >>>>> > > > >> > >> >> >>> > >
> >> >>>>> > > > >> > >> >> >>> > > as for stream processing on top
> of
> >> >>>> kafka - i
> >> >>>>> > know
> >> >>>>> > > samza
> >> >>>>> > > > >> has a
> >> >>>>> > > > >> > >> >> metadata
> >> >>>>> > > > >> > >> >> >>> > map
> >> >>>>> > > > >> > >> >> >>> > > which they carry around in
> >> addition to
> >> >>>> user
> >> >>>>> > values.
> >> >>>>> > > > headers
> >> >>>>> > > > >> > are
> >> >>>>> > > > >> > >> the
> >> >>>>> > > > >> > >> >> >>> > perfect
> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
> >> >>>>> > > > >> > >> >> >>> > >
> >> >>>>> > > > >> > >> >> >>> > >
> >> >>>>> > > > >> > >> >> >>> > >
> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM,
> >> Jun
> >> >>>> Rao <
> >> >>>>> > > > jun@confluent.io
> >> >>>>> > > > >> >
> >> >>>>> > > > >> > >> wrote:
> >> >>>>> > > > >> > >> >> >>> > >
> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> In order to answer the first two
> >> >>>> questions, it
> >> >>>>> > would
> >> >>>>> > > be
> >> >>>>> > > > >> > helpful
> >> >>>>> > > > >> > >> >> if we
> >> >>>>> > > > >> > >> >> >>> > could
> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use cases
> >> for
> >> >>>> headers
> >> >>>>> > in the
> >> >>>>> > > > space
> >> >>>>> > > > >> > for
> >> >>>>> > > > >> > >> >> >>> > third-party
> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use cases within an
> >> >>>> organization,
> >> >>>>> > one
> >> >>>>> > > > could
> >> >>>>> > > > >> > always
> >> >>>>> > > > >> > >> >> use
> >> >>>>> > > > >> > >> >> >>> > other
> >> >>>>> > > > >> > >> >> >>> > >> approaches such as company-wise
> >> >>>> containers to
> >> >>>>> > get
> >> >>>>> > > > around
> >> >>>>> > > > >> w/o
> >> >>>>> > > > >> > >> >> >>> headers. I
> >> >>>>> > > > >> > >> >> >>> > >> went through the use cases in
> the
> >> KIP
> >> >>>> and in
> >> >>>>> > Radai's
> >> >>>>> > > > wiki
> >> >>>>> > > > >> (
> >> >>>>> > > > >> > >> >> >>> > >> https://cwiki.apache.org/confl
> >> >>>>> > uence/display/KAFKA/A+
> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> >> >>>>> > > > >> > >> >> >>> > >> ).
> >> >>>>> > > > >> > >> >> >>> > >> The following are the ones that
> >> that
> >> >> I
> >> >>>>> > understand and
> >> >>>>> > > > >> could
> >> >>>>> > > > >> > be
> >> >>>>> > > > >> > >> in
> >> >>>>> > > > >> > >> >> the
> >> >>>>> > > > >> > >> >> >>> > >> third-party use case category.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
> >> >>>>> > > > >> > >> >> >>> > >> It seems that in general,
> >> >> content-type
> >> >>>> should
> >> >>>>> > be set
> >> >>>>> > > at
> >> >>>>> > > > >> the
> >> >>>>> > > > >> > >> topic
> >> >>>>> > > > >> > >> >> >>> level.
> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages with
> >> >>>> different
> >> >>>>> > content
> >> >>>>> > > > types
> >> >>>>> > > > >> > >> should be
> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
> >> >>>>> > > > >> > >> >> >>> > >> Since the value is mostly
> useless
> >> >>>> without
> >> >>>>> > schema id,
> >> >>>>> > > it
> >> >>>>> > > > >> > seems
> >> >>>>> > > > >> > >> that
> >> >>>>> > > > >> > >> >> >>> > storing
> >> >>>>> > > > >> > >> >> >>> > >> the schema id together with
> >> >> serialized
> >> >>>> bytes
> >> >>>>> > in the
> >> >>>>> > > > value
> >> >>>>> > > > >> is
> >> >>>>> > > > >> > >> >> better?
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> C. per message encryption
> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this approach is
> >> that
> >> >>>> this
> >> >>>>> > > > significantly
> >> >>>>> > > > >> > reduce
> >> >>>>> > > > >> > >> >> the
> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
> >> which
> >> >>>> happens on
> >> >>>>> > a set
> >> >>>>> > > of
> >> >>>>> > > > >> > >> serialized
> >> >>>>> > > > >> > >> >> >>> > >> messages. An alternative is to
> >> enable
> >> >>>> SSL for
> >> >>>>> > wire
> >> >>>>> > > > >> > encryption
> >> >>>>> > > > >> > >> and
> >> >>>>> > > > >> > >> >> >>> rely
> >> >>>>> > > > >> > >> >> >>> > on
> >> >>>>> > > > >> > >> >> >>> > >> the storage system (e.g. LUKS)
> >> for at
> >> >>>> rest
> >> >>>>> > > encryption.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
> across
> >> >>>> Kafka
> >> >>>>> > clusters
> >> >>>>> > > > >> > >> >> >>> > >> This is actually interesting.
> >> Today,
> >> >>>> to avoid
> >> >>>>> > > > introducing
> >> >>>>> > > > >> > >> cycles
> >> >>>>> > > > >> > >> >> when
> >> >>>>> > > > >> > >> >> >>> > doing
> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data centers,
> one
> >> >>>> would
> >> >>>>> > either have
> >> >>>>> > > to
> >> >>>>> > > > >> set
> >> >>>>> > > > >> > up
> >> >>>>> > > > >> > >> two
> >> >>>>> > > > >> > >> >> >>> Kafka
> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and an
> >> aggregate)
> >> >>>> per data
> >> >>>>> > center
> >> >>>>> > > or
> >> >>>>> > > > >> > rename
> >> >>>>> > > > >> > >> >> topics.
> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal. With headers,
> >> the
> >> >>>> producer
> >> >>>>> > could
> >> >>>>> > > tag
> >> >>>>> > > > >> each
> >> >>>>> > > > >> > >> >> message
> >> >>>>> > > > >> > >> >> >>> with
> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster ID in the
> >> >> header.
> >> >>>>> > MirrorMaker
> >> >>>>> > > > could
> >> >>>>> > > > >> > then
> >> >>>>> > > > >> > >> >> avoid
> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages to a cluster
> if
> >> >>>> they are
> >> >>>>> > tagged
> >> >>>>> > > with
> >> >>>>> > > > >> the
> >> >>>>> > > > >> > >> same
> >> >>>>> > > > >> > >> >> >>> cluster
> >> >>>>> > > > >> > >> >> >>> > >> id.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> However, an alternative approach
> >> is
> >> >> to
> >> >>>>> > introduce sth
> >> >>>>> > > > like
> >> >>>>> > > > >> > >> >> >>> hierarchical
> >> >>>>> > > > >> > >> >> >>> > >> topic and store messages from
> >> >> different
> >> >>>>> > clusters in
> >> >>>>> > > > >> > different
> >> >>>>> > > > >> > >> >> >>> partitions
> >> >>>>> > > > >> > >> >> >>> > >> under the same topic. This
> >> approach
> >> >>>> avoids
> >> >>>>> > filtering
> >> >>>>> > > > out
> >> >>>>> > > > >> > >> unneeded
> >> >>>>> > > > >> > >> >> >>> data
> >> >>>>> > > > >> > >> >> >>> > and
> >> >>>>> > > > >> > >> >> >>> > >> makes offset preserving easier
> to
> >> >>>> support. It
> >> >>>>> > may
> >> >>>>> > > make
> >> >>>>> > > > >> > >> compaction
> >> >>>>> > > > >> > >> >> >>> > trickier
> >> >>>>> > > > >> > >> >> >>> > >> though since the same key may
> >> show up
> >> >>>> in
> >> >>>>> > different
> >> >>>>> > > > >> > partitions.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> E. record-level lineage
> >> >>>>> > > > >> > >> >> >>> > >> For example, a source connector
> >> could
> >> >>>> store in
> >> >>>>> > the
> >> >>>>> > > > message
> >> >>>>> > > > >> > the
> >> >>>>> > > > >> > >> >> >>> metadata
> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
> record.
> >> >>>> Similarly,
> >> >>>>> > if a
> >> >>>>> > > > stream
> >> >>>>> > > > >> job
> >> >>>>> > > > >> > >> >> >>> transforms
> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A to topic
> B,
> >> the
> >> >>>> library
> >> >>>>> > could
> >> >>>>> > > > >> include
> >> >>>>> > > > >> > the
> >> >>>>> > > > >> > >> >> >>> source
> >> >>>>> > > > >> > >> >> >>> > >> message offset in each of the
> >> >>>> transformed
> >> >>>>> > message in
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> header.
> >> >>>>> > > > >> > >> >> Not
> >> >>>>> > > > >> > >> >> >>> > sure
> >> >>>
> >> >>>>> > > > >> > >> >> >>> > >> how widely useful record-level
> >> >> lineage
> >> >>>> is
> >> >>>>> > though
> >> >>>>> > > since
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> >> overhead
> >> >>>>> > > > >> > >> >> >>> > could
> >> >>>>> > > > >> > >> >> >>> > >> be significant.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> >> >>>>> > > > >> > >> >> >>> > >> We could put things like
> >> >>>> clientId/host/user in
> >> >>>>> > the
> >> >>>>> > > > header
> >> >>>>> > > > >> in
> >> >>>>> > > > >> > >> each
> >> >>>>> > > > >> > >> >> >>> > message
> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These metadata are
> >> >>>> really at the
> >> >>>>> > > producer
> >> >>>>> > > > >> > level
> >> >>>>> > > > >> > >> >> though.
> >> >>>>> > > > >> > >> >> >>> > So, a
> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is to only
> >> >> include a
> >> >>>>> > "producerId"
> >> >>>>> > > > per
> >> >>>>> > > > >> > >> message
> >> >>>>> > > > >> > >> >> and
> >> >>>>> > > > >> > >> >> >>> > send
> >> >>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
> mapping
> >> >>>>> > independently.
> >> >>>>> > > > KIP-98
> >> >>>>> > > > >> is
> >> >>>>> > > > >> > >> >> actually
> >> >>>>> > > > >> > >> >> >>> > >> proposing including such a
> >> producerId
> >> >>>> natively
> >> >>>>> > in the
> >> >>>>> > > > >> > message.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that I
> am
> >> >> fully
> >> >>>>> > convinced of
> >> >>>>> > > > the
> >> >>>>> > > > >> > strong
> >> >>>>> > > > >> > >> >> >>> > third-party
> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
> Perhaps
> >> we
> >> >>>> could
> >> >>>>> > discuss a
> >> >>>>> > > > bit
> >> >>>>> > > > >> > more
> >> >>>>> > > > >> > >> to
> >> >>>>> > > > >> > >> >> make
> >> >>>>> > > > >> > >> >> >>> > one
> >> >>>>> > > > >> > >> >> >>> > >> or two really convincing use
> >> cases.
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal question is
> >> >>>> whether header
> >> >>>>> > should
> >> >>>>> > > > be
> >> >>>>> > > > >> > >> exposed
> >> >>>>> > > > >> > >> >> in
> >> >>>>> > > > >> > >> >> >>> > stream
> >> >>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
> >> stream,
> >> >>>> Samza,
> >> >>>>> > and
> >> >>>>> > > Spark
> >> >>>>> > > > >> > >> streaming.
> >> >>>>> > > > >> > >> >> >>> > >> Currently, those systems just
> deal
> >> >> with
> >> >>>>> > key/value
> >> >>>>> > > > pairs.
> >> >>>>> > > > >> > >> Should we
> >> >>>>> > > > >> > >> >> >>> > expose a
> >> >>>>> > > > >> > >> >> >>> > >> third thing header there too or
> >> >>>> somehow map
> >> >>>>> > header to
> >> >>>>> > > > key
> >> >>>>> > > > >> or
> >> >>>>> > > > >> > >> >> value?
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> Jun
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM,
> >> >>>> Michael
> >> >>>>> > Pearce <
> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> >> >>>>> > > > >> > >> >> >>> > >> wrote:
> >> >>>>> > > > >> > >> >> >>> > >>
> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that after a period
> >> of a
> >> >>>> week,
> >> >>>>> > that there
> >> >>>>> > > > is
> >> >>>>> > > > >> no
> >> >>>>> > > > >> > >> >> concerns
> >> >>>>> > > > >> > >> >> >>> now
> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and now
> we
> >> >> have
> >> >>>>> > agreement that
> >> >>>>> > > > >> > headers
> >> >>>>> > > > >> > >> are
> >> >>>>> > > > >> > >> >> >>> useful
> >> >>>>> > > > >> > >> >> >>> > >> and
> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
> put
> >> to
> >> >> a
> >> >>>> KIP
> >> >>>>> > vote, this
> >> >>>>> > > > >> > wouldn’t
> >> >>>>> > > > >> > >> be
> >> >>>>> > > > >> > >> >> a
> >> >>>>> > > > >> > >> >> >>> > reason
> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >>>>> > > > >> > >> >> >>> > >> > reject.
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > @
> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose of getting
> >> this
> >> >>>> KIP
> >> >>>>> > moving past
> >> >>>>> > > > >> this,
> >> >>>>> > > > >> > we
> >> >>>>> > > > >> > >> can
> >> >>>>> > > > >> > >> >> >>> state
> >> >>>>> > > > >> > >> >> >>> > >> the
> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
> that
> >> >> can
> >> >>>> will be
> >> >>>>> > > > naturally
> >> >>>>> > > > >> > >> >> interpreted
> >> >>>>> > > > >> > >> >> >>> as
> >> >>>>> > > > >> > >> >> >>> > an
> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is later
> >> >>>> wanted you can
> >> >>>>> > > easily
> >> >>>>> > > > >> split
> >> >>>>> > > > >> > >> this
> >> >>>>> > > > >> > >> >> >>> into
> >> >>>>> > > > >> > >> >> >>> > two
> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the wire
> >> >> protocol
> >> >>>>> > > implementation
> >> >>>>> > > > >> this
> >> >>>>> > > > >> > >> makes
> >> >>>>> > > > >> > >> >> no
> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t believe. Is
> >> this
> >> >>>>> > reasonable to
> >> >>>>> > > > all?
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4 therefor
> >> happy
> >> >>>> we keep
> >> >>>>> > with 32
> >> >>>>> > > > >> bits.
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
> >> >>>>> > ignacio.solis@gmail.com on
> >> >>>>> > > > >> behalf
> >> >>>>> > > > >> > of
> >> >>>>> > > > >> > >> >> >>> Ignacio
> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
> ignacio.solis@gmail.com
> >> on
> >> >>>> behalf of
> >> >>>>> > > > >> > isolis@igso.net
> >> >>>>> > > > >> > >> >
> >> >>>>> > > > >> > >> >> >>> wrote:
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value as
> >> byte[]
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
> maybe
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
> >> system
> >> >>>> should
> >> >>>>> > take a
> >> >>>>> > > > single
> >> >>>>> > > > >> > >> int. I
> >> >>>>> > > > >> > >> >> >>> think
> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if you want
> to
> >> >>>> interpret
> >> >>>>> > this as
> >> >>>>> > > to
> >> >>>>> > > > >> 16bit
> >> >>>>> > > > >> > >> >> numbers
> >> >>>>> > > > >> > >> >> >>> in
> >> >>>>> > > > >> > >> >> >>> > the
> >> >>>>> > > > >> > >> >> >>> > >> > layer
> >> >>>>> > > > >> > >> >> >>> > >> > above go right ahead. If
> >> >>>> somebody wants
> >> >>>>> > to
> >> >>>>> > > argue
> >> >>>>> > > > >> for
> >> >>>>> > > > >> > 16
> >> >>>>> > > > >> > >> >> bits
> >> >>>>> > > > >> > >> >> >>> or
> >> >>>>> > > > >> > >> >> >>> > 64
> >> >>>>> > > > >> > >> >> >>> > >> > bits of
> >> >>>>> > > > >> > >> >> >>> > >> > header key space I would
> >> >> listen.
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the key space
> into
> >> >>>> sub_key_1 and
> >> >>>>> > > > sub_key_2
> >> >>>>> > > > >> > >> makes no
> >> >>>>> > > > >> > >> >> >>> > sense to
> >> >>>>> > > > >> > >> >> >>> > >> > me at
> >> >>>>> > > > >> > >> >> >>> > >> > this layer. Are we going
> to
> >> >>>> start
> >> >>>>> > providing
> >> >>>>> > > > APIs to
> >> >>>>> > > > >> > get
> >> >>>>> > > > >> > >> all
> >> >>>>> > > > >> > >> >> >>> the
> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
> >> >>>> sub_key_2s? If
> >> >>>>> > there is
> >> >>>>> > > > no
> >> >>>>> > > > >> > >> >> >>> distinguishing
> >> >>>>> > > > >> > >> >> >>> > >> > functions
> >> >>>>> > > > >> > >> >> >>> > >> > that are applied to each
> one
> >> >>>> then they
> >> >>>>> > should
> >> >>>>> > > be
> >> >>>>> > > > a
> >> >>>>> > > > >> > single
> >> >>>>> > > > >> > >> >> >>> value.
> >> >>>>> > > > >> > >> >> >>> > At
> >> >>>>> > > > >> > >> >> >>> > >> > this
> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're doing is
> >> >>>> equality.
> >> >>>>> > > > >> > >> >> >>> > >> > If the above layer wants
> to
> >> >>>> interpret
> >> >>>>> > this as
> >> >>>>> > > 2,
> >> >>>>> > > > 3
> >> >>>>> > > > >> or
> >> >>>>> > > > >> > >> more
> >> >>>>> > > > >> > >> >> >>> values
> >> >>>>> > > > >> > >> >> >>> > >> > that's a
> >> >>>>> > > > >> > >> >> >>> > >> > different question. I
> >> >>>> personally think
> >> >>>>> > it's
> >> >>>>> > > all
> >> >>>>> > > > one
> >> >>>>> > > > >> > >> >> keyspace
> >> >>>>> > > > >> > >> >> >>> > that is
> >> >>>>> > > > >> > >> >> >>> > >> > getting assigned using
> some
> >> >>>> structure,
> >> >>>>> > but if
> >> >>>>> > > you
> >> >>>>> > > > >> > want to
> >> >>>>> > > > >> > >> >> >>> > sub-assign
> >> >>>>> > > > >> > >> >> >>> > >> > parts
> >> >>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > The same discussion
> applies
> >> to
> >> >>>> strings.
> >> >>>>> > If
> >> >>>>> > > > somebody
> >> >>>>> > > > >> > >> argued
> >> >>>>> > > > >> > >> >> for
> >> >>>>> > > > >> > >> >> >>> > >> > strings,
> >> >>>>> > > > >> > >> >> >>> > >> > would we be arguing to
> >> divide
> >> >> the
> >> >>>>> > strings with
> >> >>>>> > > > dots
> >> >>>>> > > > >> > ('.')
> >> >>>>> > > > >> > >> >> as a
> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
> >> >>>>> > > > >> > >> >> >>> > >> > Would we want them to give
> >> us
> >> >> the
> >> >>>>> > different
> >> >>>>> > > name
> >> >>>>> > > > >> > segments
> >> >>>>> > > > >> > >> >> >>> > separately?
> >> >>>>> > > > >> > >> >> >>> > >> > Would we be performing any
> >> >>>> actions on
> >> >>>>> > this key
> >> >>>>> > > > other
> >> >>>>> > > > >> > than
> >> >>>>> > > > >> > >> >> >>> > matching?
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
> 9:30
> >> >> AM,
> >> >>>> Michael
> >> >>>>> > > Pearce <
> >> >>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any concerns
> on
> >> 1
> >> >>>> and 2
> >> >>>>> > still?
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > @all
> >> >>>>> > > > >> > >> >> >>> > >> > > To get this moving
> along a
> >> >> bit
> >> >>>> more
> >> >>>>> > I'd also
> >> >>>>> > > > like
> >> >>>>> > > > >> to
> >> >>>>> > > > >> > >> ask
> >> >>>>> > > > >> > >> >> to
> >> >>>>> > > > >> > >> >> >>> get
> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
> >> >>>>> > > > >> > >> >> >>> > >> > > the below last points:
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're all
> >> >> roughly
> >> >>>> happy
> >> >>>>> > with the
> >> >>>>> > > > >> header
> >> >>>>> > > > >> > >> value
> >> >>>>> > > > >> > >> >> >>> > being a
> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe consensus
> has
> >> >>>> been for an
> >> >>>>> > > > namespace
> >> >>>>> > > > >> > based
> >> >>>>> > > > >> > >> int
> >> >>>>> > > > >> > >> >> >>> > approach
> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for the key.
> Any
> >> >>>> objections
> >> >>>>> > if this
> >> >>>>> > > > is
> >> >>>>> > > > >> > what
> >> >>>>> > > > >> > >> we
> >> >>>>> > > > >> > >> >> go
> >> >>>>> > > > >> > >> >> >>> > with?
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
> >> assumption
> >> >> in
> >> >>>> (4) is
> >> >>>>> > > correct,
> >> >>>>> > > > >> > >> {int,int}
> >> >>>>> > > > >> > >> >> >>> keys.
> >> >>>>> > > > >> > >> >> >>> > >> > > Should both int's be
> >> int16 or
> >> >>>> int32?
> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them being
> int16(2
> >> >>>> bytes) as
> >> >>>>> > combined
> >> >>>>> > > > is
> >> >>>>> > > > >> > space
> >> >>>>> > > > >> > >> of
> >> >>>>> > > > >> > >> >> >>> > 4bytes as
> >> >>>>> > > > >> > >> >> >>> > >> > per
> >> >>>>> > > > >> > >> >> >>> > >> > > original and gives
> plenty
> >> of
> >> >>>>> > combinations for
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> >> >>> foreseeable,
> >> >>>>> > > > >> > >> >> >>> > and
> >> >>>>> > > > >> > >> >> >>> > >> > keeps
> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead small.
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any benefit in
> >> >>>> another kip
> >> >>>>> > call to
> >> >>>>> > > > >> discuss
> >> >>>>> > > > >> > >> >> these at
> >> >>>>> > > > >> > >> >> >>> > all?
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> ______________________________
> >> >>>>> > __________
> >> >>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
> >> >>>> k.burstev@yandex.com>
> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday, November
> 18,
> >> >> 2016
> >> >>>>> > 7:07:07 AM
> >> >>>>> > > > >> > >> >> >>> > >> > > To:
> dev@kafka.apache.org
> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re: [DISCUSS]
> >> KIP-82
> >> >>>> - Add
> >> >>>>> > Record
> >> >>>>> > > > Headers
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > For what it is worth
> also
> >> i
> >> >>>> agree. As
> >> >>>>> > a user:
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
> >> >>>> worthwhile
> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers should
> >> be a
> >> >>>> top level
> >> >>>>> > > option
> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
> >> "Ignacio
> >> >>>> Solis" <
> >> >>>>> > > > >> isolis@igso.net
> >> >>>>> > > > >> > >:
> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers are
> >> >>>> worthwhile
> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
> should
> >> be
> >> >> a
> >> >>>> top
> >> >>>>> > level
> >> >>>>> > > option
> >> >>>>> > > > >> > >> >> >>> > >> > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14, 2016
> at
> >> >> 9:16
> >> >>>> AM,
> >> >>>>> > Michael
> >> >>>>> > > > Pearce
> >> >>>>> > > > >> <
> >> >>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
> >> details/examples
> >> >>>> the
> >> >>>>> > original
> >> >>>>> > > > proposal
> >> >>>>> > > > >> > for
> >> >>>>> > > > >> > >> key
> >> >>>>> > > > >> > >> >> >>> > spacing
> >> >>>>> > > > >> > >> >> >>> > >> ,
> >> >>>>> > > > >> > >> >> >>> > >> > not
> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as per
> >> >>>> discussion
> >> >>>>> > namespace
> >> >>>>> > > > >> idea.
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will need to
> update
> >> >> the
> >> >>>> kip,
> >> >>>>> > when we
> >> >>>>> > > get
> >> >>>>> > > > >> > >> agreement
> >> >>>>> > > > >> > >> >> >>> this
> >> >>>>> > > > >> > >> >> >>> > is a
> >> >>>>> > > > >> > >> >> >>> > >> > better
> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach (which
> seems
> >> to
> >> >>>> be the
> >> >>>>> > case if I
> >> >>>>> > > > have
> >> >>>>> > > > >> > >> >> understood
> >> >>>>> > > > >> > >> >> >>> > the
> >> >>>>> > > > >> > >> >> >>> > >> > general
> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
> >> >>>> conversation)
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
> ints,
> >> at
> >> >>>> very
> >> >>>>> > early stage
> >> >>>>> > > > we
> >> >>>>> > > > >> did
> >> >>>>> > > > >> > >> think
> >> >>>>> > > > >> > >> >> >>> about
> >> >>>>> > > > >> > >> >> >>> > >> > this. I
> >> >>>>> > > > >> > >> >> >>> > >> > > think
> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added complexity
> >> for
> >> >>>> the
> >> >>>>> > saving isn't
> >> >>>>> > > > >> worth
> >> >>>>> > > > >> > it.
> >> >>>>> > > > >> > >> >> I'd
> >> >>>>> > > > >> > >> >> >>> > rather
> >> >>>>> > > > >> > >> >> >>> > >> go
> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
> >> >>>> overheads and
> >> >>>>> > size
> >> >>>>> > > int16
> >> >>>>> > > > >> > (2bytes)
> >> >>>>> > > > >> > >> >> keys
> >> >>>>> > > > >> > >> >> >>> as
> >> >>>>> > > > >> > >> >> >>> > it
> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
> >> >> headers,
> >> >>>> there
> >> >>>>> > is as
> >> >>>>> > > per
> >> >>>>> > > > the
> >> >>>>> > > > >> > kip
> >> >>>>> > > > >> > >> as
> >> >>>>> > > > >> > >> >> we
> >> >>>>> > > > >> > >> >> >>> > use an
> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
> >> headers
> >> >>>> are
> >> >>>>> > present or
> >> >>>>> > > > not as
> >> >>>>> > > > >> > such
> >> >>>>> > > > >> > >> >> >>> > provides a
> >> >>>>> > > > >> > >> >> >>> > >> > zero
> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead currently
> if
> >> >>>> headers are
> >> >>>>> > not
> >> >>>>> > > used.
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
> >> mentions
> >> >>>> would be
> >> >>>>> > good
> >> >>>>> > > > first
> >> >>>>> > > > >> > if we
> >> >>>>> > > > >> > >> >> can
> >> >>>>> > > > >> > >> >> >>> get
> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
> >> >>>>> > > > >> > >> >> >>> > >> > > do
> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now have general
> >> >>>> consensus that
> >> >>>>> > (1)
> >> >>>>> > > > headers
> >> >>>>> > > > >> > are
> >> >>>>> > > > >> > >> >> >>> > worthwhile
> >> >>>>> > > > >> > >> >> >>> > >> and
> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want it
> as
> >> a
> >> >>>> top level
> >> >>>>> > entity.
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
> >> >> obvious i
> >> >>>>> > believe (1)
> >> >>>>> > > > >> headers
> >> >>>>> > > > >> > are
> >> >>>>> > > > >> > >> >> >>> > worthwhile
> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as a top level
> >> >>>> entity.
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>> ______________________________
> >> >>>>> > __________
> >> >>>>> > > > >> > >> >> >>> > >> > > >> From: Roger Hoover <
> >> >>>>> > > roger.hoover@gmail.com
> >> >>>>> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
> >> November
> >> >>>> 9, 2016
> >> >>>>> > 9:10:47
> >> >>>>> > > > PM
> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
> >> dev@kafka.apache.org
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
> [DISCUSS]
> >> >>>> KIP-82 - Add
> >> >>>>> > > Record
> >> >>>>> > > > >> > Headers
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
> >> little
> >> >>>> in the
> >> >>>>> > weeds but
> >> >>>>> > > > >> thanks
> >> >>>>> > > > >> > >> for
> >> >>>>> > > > >> > >> >> the
> >> >>>>> > > > >> > >> >> >>> > >> replies
> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a prefix
> >> and
> >> >>>> {int,
> >> >>>>> > int} can
> >> >>>>> > > be
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> same.
> >> >>>>> > > > >> > >> >> It
> >> >>>>> > > > >> > >> >> >>> > doesn't
> >> >>>>> > > > >> > >> >> >>> > >> > look
> >> >>>>> > > > >> > >> >> >>> > >> > > like
> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's what the KIP
> is
> >> >>>> saying the
> >> >>>>> > "Open"
> >> >>>>> > > > >> > section.
> >> >>>>> > > > >> > >> The
> >> >>>>> > > > >> > >> >> >>> > example
> >> >>>>> > > > >> > >> >> >>> > >> > shows
> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
> >> 210002
> >> >>>> for App
> >> >>>>> > Dynamics
> >> >>>>> > > > >> > implying
> >> >>>>> > > > >> > >> >> that
> >> >>>>> > > > >> > >> >> >>> the
> >> >>>>> > > > >> > >> >> >>> > New
> >> >>>>> > > > >> > >> >> >>> > >> > Relic
> >> >>>>> > > > >> > >> >> >>> > >> > > >> organization will
> have
> >> >>>> only a
> >> >>>>> > single
> >> >>>>> > > > header id
> >> >>>>> > > > >> > to
> >> >>>>> > > > >> > >> work
> >> >>>>> > > > >> > >> >> >>> > with. Or
> >> >>>>> > > > >> > >> >> >>> > >> > is
> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The main
> >> point
> >> >>>> of a
> >> >>>>> > namespace
> >> >>>>> > > or
> >> >>>>> > > > >> > prefix
> >> >>>>> > > > >> > >> is
> >> >>>>> > > > >> > >> >> to
> >> >>>>> > > > >> > >> >> >>> > reduce
> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead of config
> >> >> mapping
> >> >>>> or
> >> >>>>> > > registration
> >> >>>>> > > > >> > >> depending
> >> >>>>> > > > >> > >> >> on
> >> >>>>> > > > >> > >> >> >>> how
> >> >>>>> > > > >> > >> >> >>> > >> > > >> namespaces/prefixes
> >> are
> >> >>>> managed.
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would love to hear
> >> more
> >> >>>> feedback
> >> >>>>> > on the
> >> >>>>> > > > >> > >> higher-level
> >> >>>>> > > > >> > >> >> >>> > questions
> >> >>>>> > > > >> > >> >> >>> > >> > > though...
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9, 2016
> at
> >> >>>> 11:38 AM,
> >> >>>>> > radai <
> >> >>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think this
> >> discussion
> >> >>>> is
> >> >>>>> > getting a
> >> >>>>> > > bit
> >> >>>>> > > > >> into
> >> >>>>> > > > >> > the
> >> >>>>> > > > >> > >> >> >>> weeds on
> >> >>>>> > > > >> > >> >> >>> > >> > technical
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > implementation
> >> details.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to step
> >> back a
> >> >>>> minute
> >> >>>>> > and try
> >> >>>>> > > > and
> >> >>>>> > > > >> > >> establish
> >> >>>>> > > > >> > >> >> >>> > where we
> >> >>>>> > > > >> > >> >> >>> > >> > are in
> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
> nacho's
> >> >> last
> >> >>>>> > paragraph)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all in
> >> >>>> agreement that
> >> >>>>> > headers
> >> >>>>> > > > are
> >> >>>>> > > > >> a
> >> >>>>> > > > >> > >> >> >>> worthwhile
> >> >>>>> > > > >> > >> >> >>> > and
> >> >>>>> > > > >> > >> >> >>> > >> > useful
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > addition to have?
> >> this
> >> >>>> was
> >> >>>>> > contested
> >> >>>>> > > > early
> >> >>>>> > > > >> on
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all in
> >> >>>> agreement on
> >> >>>>> > headers
> >> >>>>> > > as
> >> >>>>> > > > top
> >> >>>>> > > > >> > >> level
> >> >>>>> > > > >> > >> >> >>> entity
> >> >>>>> > > > >> > >> >> >>> > vs
> >> >>>>> > > > >> > >> >> >>> > >> > headers
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > squirreled-away in
> >> V?
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if there are still
> >> >>>> concerns
> >> >>>>> > around
> >> >>>>> > > these
> >> >>>>> > > > #2
> >> >>>>> > > > >> > >> points
> >> >>>>> > > > >> > >> >> >>> (#jay?
> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and now back to
> our
> >> >>>> normal
> >> >>>>> > programming
> >> >>>>> > > > ...)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints are nice.
> >> >> having
> >> >>>> said
> >> >>>>> > that, its
> >> >>>>> > > > >> adding
> >> >>>>> > > > >> > >> >> >>> complexity
> >> >>>>> > > > >> > >> >> >>> > >> (see
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>> https://github.com/addthis/
> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>> main/java/com/clearspring/
> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
> >> result)
> >> >>>> and would
> >> >>>>> > require
> >> >>>>> > > > >> anyone
> >> >>>>> > > > >> > >> >> writing
> >> >>>>> > > > >> > >> >> >>> > other
> >> >>>>> > > > >> > >> >> >>> > >> > clients
> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go? Bash?
> >> ;-) )
> >> >>>> to
> >> >>>>> > > get/implement
> >> >>>>> > > > the
> >> >>>>> > > > >> > >> same,
> >> >>>>> > > > >> > >> >> and
> >> >>>>> > > > >> > >> >> >>> for
> >> >>>>> > > > >> > >> >> >>> > >> > relatively
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little gain (int
> vs
> >> >>>> string is
> >> >>>>> > order of
> >> >>>>> > > > >> > magnitude,
> >> >>>>> > > > >> > >> >> this
> >> >>>>> > > > >> > >> >> >>> > isnt).
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int namespacing vs
> >> >> {int,
> >> >>>> int}
> >> >>>>> > > namespacing
> >> >>>>> > > > >> are
> >> >>>>> > > > >> > >> >> basically
> >> >>>>> > > > >> > >> >> >>> > the
> >> >>>>> > > > >> > >> >> >>> > >> > same
> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre just
> >> namespacing
> >> >>>> an int64
> >> >>>>> > and
> >> >>>>> > > > giving
> >> >>>>> > > > >> > people
> >> >>>>> > > > >> > >> >> while
> >> >>>>> > > > >> > >> >> >>> > 2^32
> >> >>>>> > > > >> > >> >> >>> > >> > ranges
> >> >>>>> > > > >> > >> >> >>> > >> > > at a
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time. the part i
> >> like
> >> >>>> about this
> >> >>>>> > is
> >> >>>>> > > > letting
> >> >>>>> > > > >> > >> people
> >> >>>>> > > > >> > >> >> >>> have a
> >> >>>>> > > > >> > >> >> >>> > >> large
> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers with one
> >> >>>> registration so
> >> >>>>> > they
> >> >>>>> > > > dont
> >> >>>>> > > > >> > have
> >> >>>>> > > > >> > >> to
> >> >>>>> > > > >> > >> >> come
> >> >>>>> > > > >> > >> >> >>> > back
> >> >>>>> > > > >> > >> >> >>> > >> > for
> >> >>>>> > > > >> > >> >> >>> > >> > > every
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
> plugin/header
> >> >>>> they want to
> >> >>>>> > > > "reserve".
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
> 2016
> >> at
> >> >>>> 11:01 AM,
> >> >>>>> > Roger
> >> >>>>> > > > >> Hoover
> >> >>>>> > > > >> > <
> >> >>>>> > > > >> > >> >> >>> > >> > > roger.hoover@gmail.com>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
> the
> >> >>>> debate has
> >> >>>>> > been
> >> >>>>> > > about
> >> >>>>> > > > >> > >> overhead +
> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if we
> >> have
> >> >>>>> > considered a
> >> >>>>> > > > varint
> >> >>>>> > > > >> > >> encoding
> >> >>>>> > > > >> > >> >> (
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>> https://developers.google.com/
> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
> >> >>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the header
> length
> >> >>>> field (int32
> >> >>>>> > in the
> >> >>>>> > > > >> > proposal)
> >> >>>>> > > > >> > >> >> and
> >> >>>>> > > > >> > >> >> >>> for
> >> >>>>> > > > >> > >> >> >>> > >> > header
> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
> headers,
> >> >> the
> >> >>>>> > overhead would
> >> >>>>> > > > be a
> >> >>>>> > > > >> > >> single
> >> >>>>> > > > >> > >> >> >>> byte
> >> >>>>> > > > >> > >> >> >>> > and
> >> >>>>> > > > >> > >> >> >>> > >> > for each
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128 would
> >> also
> >> >>>> need only a
> >> >>>>> > > single
> >> >>>>> > > > >> byte?
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov 9,
> >> 2016
> >> >> at
> >> >>>> 6:43 AM,
> >> >>>>> > > radai <
> >> >>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus - and
> >> very
> >> >>>> dangerous
> >> >>>>> > (youre
> >> >>>>> > > > >> > >> essentially
> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > executing
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary code
> >> off
> >> >>>> the
> >> >>>>> > internet on
> >> >>>>> > > > your
> >> >>>>> > > > >> > >> servers
> >> >>>>> > > > >> > >> >> ...
> >> >>>>> > > > >> > >> >> >>> > bad
> >> >>>>> > > > >> > >> >> >>> > >> > idea
> >> >>>>> > > > >> > >> >> >>> > >> > > without
> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox, even
> >> with)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
> being
> >> a
> >> >>>> purely
> >> >>>>> > > > administrative
> >> >>>>> > > > >> > task
> >> >>>>> > > > >> > >> - i
> >> >>>>> > > > >> > >> >> >>> > >> disagree.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
> would,
> >> >>>> really,
> >> >>>>> > because
> >> >>>>> > > > then my
> >> >>>>> > > > >> > >> earlier
> >> >>>>> > > > >> > >> >> >>> > point on
> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > complexity
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the remapping
> >> >>>> process would
> >> >>>>> > be
> >> >>>>> > > > invalid,
> >> >>>>> > > > >> > but
> >> >>>>> > > > >> > >> at
> >> >>>>> > > > >> > >> >> >>> > linkedin,
> >> >>>>> > > > >> > >> >> >>> > >> > for
> >> >>>>> > > > >> > >> >> >>> > >> > > example,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team im
> in)
> >> >> run
> >> >>>> kafka
> >> >>>>> > as a
> >> >>>>> > > > service.
> >> >>>>> > > > >> > we
> >> >>>>> > > > >> > >> dont
> >> >>>>> > > > >> > >> >> >>> > really
> >> >>>>> > > > >> > >> >> >>> > >> > know
> >> >>>>> > > > >> > >> >> >>> > >> > > what our
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
> >> >>>> applications
> >> >>>>> > that use
> >> >>>>> > > > kafka)
> >> >>>>> > > > >> > are
> >> >>>>> > > > >> > >> up
> >> >>>>> > > > >> > >> >> to
> >> >>>>> > > > >> > >> >> >>> at
> >> >>>>> > > > >> > >> >> >>> > any
> >> >>>>> > > > >> > >> >> >>> > >> > given
> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
> possible
> >> >>>> (given the
> >> >>>>> > > > existance of
> >> >>>>> > > > >> > >> headers
> >> >>>>> > > > >> > >> >> >>> and a
> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem) for
> >> some
> >> >>>>> > application to
> >> >>>>> > > > >> "equip"
> >> >>>>> > > > >> > >> their
> >> >>>>> > > > >> > >> >> >>> > >> producers
> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > consumers
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
> >> required
> >> >>>> plugin
> >> >>>>> > without us
> >> >>>>> > > > >> > knowing.
> >> >>>>> > > > >> > >> i
> >> >>>>> > > > >> > >> >> dont
> >> >>>>> > > > >> > >> >> >>> > mean
> >> >>>>> > > > >> > >> >> >>> > >> > to imply
> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
> >> want to
> >> >>>> make the
> >> >>>>> > point
> >> >>>>> > > > that
> >> >>>>> > > > >> > its
> >> >>>>> > > > >> > >> not
> >> >>>>> > > > >> > >> >> as
> >> >>>>> > > > >> > >> >> >>> > simple
> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sync across a
> >> >>>> large-enough
> >> >>>>> > > > organization.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed, Nov 9,
> >> 2016
> >> >>>> at 6:17
> >> >>>>> > AM,
> >> >>>>> > > > Magnus
> >> >>>>> > > > >> > >> Edenhill
> >> >>>>> > > > >> > >> >> <
> >> >>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
> there
> >> is
> >> >> a
> >> >>>> piece
> >> >>>>> > missing
> >> >>>>> > > in
> >> >>>>> > > > >> the
> >> >>>>> > > > >> > >> >> Strings
> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
> >> >>>>> > > > >> > >> >> >>> > >> > > where
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> pro-Stringers
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason that
> by
> >> >>>> providing
> >> >>>>> > unique
> >> >>>>> > > > string
> >> >>>>> > > > >> > >> >> >>> identifiers
> >> >>>>> > > > >> > >> >> >>> > for
> >> >>>>> > > > >> > >> >> >>> > >> > each
> >> >>>>> > > > >> > >> >> >>> > >> > > header
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > everything
> >> will
> >> >>>> just
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
> work
> >> >> for
> >> >>>> all
> >> >>>>> > parts of
> >> >>>>> > > the
> >> >>>>> > > > >> > stream
> >> >>>>> > > > >> > >> >> >>> pipeline.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
> >> strings
> >> >>>> dont mean
> >> >>>>> > > anything
> >> >>>>> > > > by
> >> >>>>> > > > >> > >> >> themselves,
> >> >>>>> > > > >> > >> >> >>> > and
> >> >>>>> > > > >> > >> >> >>> > >> > while we
> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
> >> envision
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
> >> plugin
> >> >>>> loader
> >> >>>>> > that
> >> >>>>> > > > >> downloads,
> >> >>>>> > > > >> > >> >> compiles,
> >> >>>>> > > > >> > >> >> >>> > links
> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >>>>> > > > >> > >> >> >>> > >> > > runs
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon as
> >> >> they're
> >> >>>> seen by
> >> >>>>> > a
> >> >>>>> > > > >> consumer, I
> >> >>>>> > > > >> > >> dont
> >> >>>>> > > > >> > >> >> >>> really
> >> >>>>> > > > >> > >> >> >>> > >> see
> >> >>>>> > > > >> > >> >> >>> > >> > a
> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > so dynamic
> >> (and
> >> >>>> fragile) in
> >> >>>>> > > > practice.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the real
> >> world
> >> >>>> an
> >> >>>>> > application
> >> >>>>> > > > will
> >> >>>>> > > > >> be
> >> >>>>> > > > >> > >> >> >>> configured
> >> >>>>> > > > >> > >> >> >>> > >> with
> >> >>>>> > > > >> > >> >> >>> > >> > a set
> >> >>>>> > > > >> > >> >> >>> > >> > > of
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
> add
> >> >>>> (producer)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
> >> >> (consumer)
> >> >>>> headers.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is an
> >> >>>> administrative
> >> >>>>> > task
> >> >>>>> > > > based
> >> >>>>> > > > >> on
> >> >>>>> > > > >> > >> what
> >> >>>>> > > > >> > >> >> >>> > features a
> >> >>>>> > > > >> > >> >> >>> > >> > client
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> needs/provides
> >> >> and
> >> >>>> results
> >> >>>>> > in
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some sort of
> >> >>>> configuration
> >> >>>>> > to
> >> >>>>> > > > enable
> >> >>>>> > > > >> and
> >> >>>>> > > > >> > >> >> >>> configure
> >> >>>>> > > > >> > >> >> >>> > the
> >> >>>>> > > > >> > >> >> >>> > >> > desired
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since this
> >> needs
> >> >>>> to be kept
> >> >>>>> > > > somewhat
> >> >>>>> > > > >> in
> >> >>>>> > > > >> > >> sync
> >> >>>>> > > > >> > >> >> >>> across
> >> >>>>> > > > >> > >> >> >>> > an
> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no point
> in
> >> >>>> having
> >> >>>>> > producers
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > add headers
> no
> >> >>>> consumers
> >> >>>>> > will
> >> >>>>> > > read,
> >> >>>>> > > > >> and
> >> >>>>> > > > >> > >> vice
> >> >>>>> > > > >> > >> >> >>> versa),
> >> >>>>> > > > >> > >> >> >>> > >> the
> >> >>>>> > > > >> > >> >> >>> > >> > added
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > of assigning
> >> an
> >> >> id
> >> >>>>> > namespace
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
> >> plugin
> >> >> as
> >> >>>> it is
> >> >>>>> > being
> >> >>>>> > > > >> > configured
> >> >>>>> > > > >> > >> >> should
> >> >>>>> > > > >> > >> >> >>> be
> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > 2016-11-09
> >> 13:06
> >> >>>> GMT+01:00
> >> >>>>> > > Michael
> >> >>>>> > > > >> > Pearce <
> >> >>>>> > > > >> > >> >> >>> > >> > > Michael.Pearce@ig.com>:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
> >> >>>> following/catching
> >> >>>>> > up on
> >> >>>>> > > > what
> >> >>>>> > > > >> > seems
> >> >>>>> > > > >> > >> to
> >> >>>>> > > > >> > >> >> be
> >> >>>>> > > > >> > >> >> >>> an
> >> >>>>> > > > >> > >> >> >>> > >> > active
> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
> >> sorry if
> >> >>>> it may
> >> >>>>> > seem
> >> >>>>> > > > obvious
> >> >>>>> > > > >> > but
> >> >>>>> > > > >> > >> what
> >> >>>>> > > > >> > >> >> >>> does
> >> >>>>> > > > >> > >> >> >>> > MD
> >> >>>>> > > > >> > >> >> >>> > >> > stand
> >> >>>>> > > > >> > >> >> >>> > >> > > for?
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My take on
> >> >>>> String vs Int:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
> state
> >> >>>> first I am
> >> >>>>> > pro Int
> >> >>>>> > > > (16
> >> >>>>> > > > >> or
> >> >>>>> > > > >> > >> 32).
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
> though
> >> >>>> playing
> >> >>>>> > devils
> >> >>>>> > > > advocate
> >> >>>>> > > > >> > see a
> >> >>>>> > > > >> > >> >> big
> >> >>>>> > > > >> > >> >> >>> plus
> >> >>>>> > > > >> > >> >> >>> > >> > with the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > argument
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
> keys,
> >> >>>> this is
> >> >>>>> > around
> >> >>>>> > > > >> > integrating
> >> >>>>> > > > >> > >> >> into an
> >> >>>>> > > > >> > >> >> >>> > >> > existing
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
> >> other
> >> >>>> systems use
> >> >>>>> > > String
> >> >>>>> > > > >> based
> >> >>>>> > > > >> > >> >> headers
> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
> >> >>>>> > > > >> > >> >> >>> > >> > > it
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
> easier
> >> for
> >> >>>> these to
> >> >>>>> > be
> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
> >> >>>>> > > > >> > >> >> >>> > >> > into.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How with
> Int
> >> >>>> based
> >> >>>>> > headers
> >> >>>>> > > could
> >> >>>>> > > > we
> >> >>>>> > > > >> > >> provide
> >> >>>>> > > > >> > >> >> a
> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> integration
> >> >>>> simple /
> >> >>>>> > easy with
> >> >>>>> > > > >> > transition
> >> >>>>> > > > >> > >> >> flows
> >> >>>>> > > > >> > >> >> >>> > over
> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
> luck
> >> >>>> buddy
> >> >>>>> > you're on
> >> >>>>> > > your
> >> >>>>> > > > >> own
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * simply
> >> hash
> >> >>>> the string
> >> >>>>> > into
> >> >>>>> > > int
> >> >>>>> > > > >> code
> >> >>>>> > > > >> > >> and
> >> >>>>> > > > >> > >> >> hope
> >> >>>>> > > > >> > >> >> >>> > for
> >> >>>>> > > > >> > >> >> >>> > >> no
> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
> back
> >> >>>> though?)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
> >> style
> >> >> as
> >> >>>>> > mentioned by
> >> >>>>> > > > nacho.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > ______________________________
> >> >>>>> > > > >> > __________
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
> radai
> >> <
> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
> >> >> Wednesday,
> >> >>>>> > November 9,
> >> >>>>> > > 2016
> >> >>>>> > > > >> > 8:12 AM
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
> >> >>>> dev@kafka.apache.org
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Subject:
> Re:
> >> >>>> [DISCUSS]
> >> >>>>> > KIP-82 -
> >> >>>>> > > > Add
> >> >>>>> > > > >> > >> Record
> >> >>>>> > > > >> > >> >> >>> Headers
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > thinking
> >> about
> >> >>>> it some
> >> >>>>> > more,
> >> >>>>> > > the
> >> >>>>> > > > >> best
> >> >>>>> > > > >> > >> way to
> >> >>>>> > > > >> > >> >> >>> > transmit
> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >>>>> > > > >> > >> >> >>> > >> > > header
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
> >> >>>> consumers would
> >> >>>>> > be to
> >> >>>>> > > > put it
> >> >>>>> > > > >> > in
> >> >>>>> > > > >> > >> the
> >> >>>>> > > > >> > >> >> MD
> >> >>>>> > > > >> > >> >> >>> > >> response
> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it should
> be
> >> >>>> discussed
> >> >>>>> > now.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
> Nov
> >> 9,
> >> >>>> 2016 at
> >> >>>>> > 12:09
> >> >>>>> > > AM,
> >> >>>>> > > > >> > radai <
> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> radai.rosenblatt@gmail.com
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im not
> >> >> opposed
> >> >>>> to the
> >> >>>>> > idea of
> >> >>>>> > > > >> > namespace
> >> >>>>> > > > >> > >> >> >>> mapping.
> >> >>>>> > > > >> > >> >> >>> > >> all
> >> >>>>> > > > >> > >> >> >>> > >> > im
> >> >>>>> > > > >> > >> >> >>> > >> > > saying
> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > that
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not part
> >> of
> >> >>>> the "mvp"
> >> >>>>> > and,
> >> >>>>> > > > since
> >> >>>>> > > > >> it
> >> >>>>> > > > >> > >> >> requires
> >> >>>>> > > > >> > >> >> >>> no
> >> >>>>> > > > >> > >> >> >>> > >> wire
> >> >>>>> > > > >> > >> >> >>> > >> > format
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > always
> be
> >> >>>> added later.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
> its
> >> not
> >> >>>> as
> >> >>>>> > simple as
> >> >>>>> > > just
> >> >>>>> > > > >> > >> >> configuring
> >> >>>>> > > > >> > >> >> >>> MM
> >> >>>>> > > > >> > >> >> >>> > to
> >> >>>>> > > > >> > >> >> >>> > >> do
> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > transform:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say i've
> >> >>>> implemented
> >> >>>>> > large
> >> >>>>> > > > message
> >> >>>>> > > > >> > >> >> support as
> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
> >> >>>>> > > > >> > >> >> >>> > >> > > on
> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > target
> >> >> cluster
> >> >>>> its been
> >> >>>>> > > > remapped
> >> >>>>> > > > >> to
> >> >>>>> > > > >> > >> >> {999,1}.
> >> >>>>> > > > >> > >> >> >>> the
> >> >>>>> > > > >> > >> >> >>> > >> > consumer
> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
> need
> >> to
> >> >>>> be told
> >> >>>>> > to look
> >> >>>>> > > > for
> >> >>>>> > > > >> the
> >> >>>>> > > > >> > >> large
> >> >>>>> > > > >> > >> >> >>> > message
> >> >>>>> > > > >> > >> >> >>> > >> > "part X
> >> >>>>> > > > >> > >> >> >>> > >> > > of
> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > header
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
> >> >> {999,1}.
> >> >>>> doable,
> >> >>>>> > but
> >> >>>>> > > > tricky.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On Tue,
> >> Nov
> >> >> 8,
> >> >>>> 2016 at
> >> >>>>> > 10:29
> >> >>>>> > > > PM,
> >> >>>>> > > > >> > Gwen
> >> >>>>> > > > >> > >> >> >>> Shapira <
> >> >>>>> > > > >> > >> >> >>> > >> > > >> gwen@confluent.io
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> While
> you
> >> >> can
> >> >>>> do
> >> >>>>> > whatever
> >> >>>>> > > you
> >> >>>>> > > > >> want
> >> >>>>> > > > >> > >> with a
> >> >>>>> > > > >> > >> >> >>> > >> namespace
> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >>>>> > > > >> > >> >> >>> > >> > > your
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
> I'd
> >> >>>> expect is
> >> >>>>> > for each
> >> >>>>> > > > app
> >> >>>>> > > > >> to
> >> >>>>> > > > >> > >> >> >>> namespaces
> >> >>>>> > > > >> > >> >> >>> > >> > > configurable...
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So if I
> >> >>>> accidentally
> >> >>>>> > used
> >> >>>>> > > 666
> >> >>>>> > > > for
> >> >>>>> > > > >> > my
> >> >>>>> > > > >> > >> HR
> >> >>>>> > > > >> > >> >> >>> > >> department,
> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >>>>> > > > >> > >> >> >>> > >> > > still
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
> >> >> RadaiApp,
> >> >>>> I can
> >> >>>>> > config
> >> >>>>> > > > >> > >> "namespace=42"
> >> >>>>> > > > >> > >> >> >>> for
> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
> look
> >> >>>> normal.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
> >> means
> >> >>>> you only
> >> >>>>> > need to
> >> >>>>> > > > sync
> >> >>>>> > > > >> > usage
> >> >>>>> > > > >> > >> >> >>> inside
> >> >>>>> > > > >> > >> >> >>> > your
> >> >>>>> > > > >> > >> >> >>> > >> > own
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> Still
> >> hard,
> >> >>>> but
> >> >>>>> > somewhat
> >> >>>>> > > > easier
> >> >>>>> > > > >> > than
> >> >>>>> > > > >> > >> >> syncing
> >> >>>>> > > > >> > >> >> >>> > with
> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >>>>> > > > >> > >> >> >>> > >> > > entire
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > world.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On Tue,
> >> Nov
> >> >>>> 8, 2016
> >> >>>>> > at 10:07
> >> >>>>> > > > PM,
> >> >>>>> > > > >> > >> radai <
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >>>> radai.rosenblatt@gmail.com>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > and
> we
> >> can
> >> >>>> start
> >> >>>>> > with
> >> >>>>> > > > >> {namespace,
> >> >>>>> > > > >> > >> id}
> >> >>>>> > > > >> > >> >> and
> >> >>>>> > > > >> > >> >> >>> no
> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> always
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > add
> it
> >> >>>> later on
> >> >>>>> > if/when
> >> >>>>> > > > >> > collisions
> >> >>>>> > > > >> > >> >> >>> actually
> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
> >> >>>>> > > > >> > >> >> >>> > >> > > dont
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
> >> >> problem).
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > every
> >> >>>> interested
> >> >>>>> > party (so
> >> >>>>> > > > orgs
> >> >>>>> > > > >> > or
> >> >>>>> > > > >> > >> >> >>> > individuals)
> >> >>>>> > > > >> > >> >> >>> > >> > could
> >> >>>>> > > > >> > >> >> >>> > >> > > then
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > register
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> prefix
> >> (0
> >> >> =
> >> >>>>> > reserved, 1 =
> >> >>>>> > > > >> > confluent
> >> >>>>> > > > >> > >> ...
> >> >>>>> > > > >> > >> >> >>> 666
> >> >>>>> > > > >> > >> >> >>> > = me
> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
> >> >>>>> > > > >> > >> >> >>> > >> > > and
> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > the
> >> 2nd ID
> >> >>>> - so once
> >> >>>>> > > > linkedin
> >> >>>>> > > > >> > >> >> registers,
> >> >>>>> > > > >> > >> >> >>> say
> >> >>>>> > > > >> > >> >> >>> > 3,
> >> >>>>> > > > >> > >> >> >>> > >> > then
> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
> use
> >> {3,
> >> >>>> *} with a
> >> >>>>> > > > reasonable
> >> >>>>> > > > >> > >> >> >>> expectation
> >> >>>>> > > > >> > >> >> >>> > to
> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >>>>> > > > >> > >> >> >>> > >> > > collide
> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > anything
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > else.
> >> >>>> further
> >> >>>>> > partitioning
> >> >>>>> > > > of
> >> >>>>> > > > >> > that *
> >> >>>>> > > > >> > >> >> >>> becomes
> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > problem,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >> "upstream
> >> >>>>> > registration"
> >> >>>>> > > of a
> >> >>>>> > > > >> > >> namespace
> >> >>>>> > > > >> > >> >> >>> only
> >> >>>>> > > > >> > >> >> >>> > has
> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >>>>> > > > >> > >> >> >>> > >> > > happen
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
> Tue,
> >> >> Nov
> >> >>>> 8, 2016
> >> >>>>> > at
> >> >>>>> > > 9:03
> >> >>>>> > > > PM,
> >> >>>>> > > > >> > >> James
> >> >>>>> > > > >> > >> >> >>> Cheng <
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> wushujames@gmail.com
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
> >> Nov
> >> >>>> 8, 2016,
> >> >>>>> > at 5:54
> >> >>>>> > > > PM,
> >> >>>>> > > > >> > Gwen
> >> >>>>> > > > >> > >> >> >>> Shapira <
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > gwen@confluent.io
> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> Thank
> >> >>>> you so
> >> >>>>> > much for
> >> >>>>> > > > this
> >> >>>>> > > > >> > clear
> >> >>>>> > > > >> > >> and
> >> >>>>> > > > >> > >> >> >>> fair
> >> >>>>> > > > >> > >> >> >>> > >> > summary of
> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > arguments.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> I'm
> >> in
> >> >>>> favor of
> >> >>>>> > ints.
> >> >>>>> > > > Not a
> >> >>>>> > > > >> > >> >> >>> deal-breaker,
> >> >>>>> > > > >> > >> >> >>> > but
> >> >>>>> > > > >> > >> >> >>> > >> > in
> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> Even
> >> >>>> more in
> >> >>>>> > favor of
> >> >>>>> > > > >> Magnus's
> >> >>>>> > > > >> > >> >> >>> > decentralized
> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> tweak:
> >> >>>> add a
> >> >>>>> > namespace
> >> >>>>> > > > for
> >> >>>>> > > > >> > >> headers.
> >> >>>>> > > > >> > >> >> >>> This
> >> >>>>> > > > >> > >> >> >>> > will
> >> >>>>> > > > >> > >> >> >>> > >> > allow
> >> >>>>> > > > >> > >> >> >>> > >> > > each
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> use
> >> >>>> whatever IDs
> >> >>>>> > it
> >> >>>>> > > wants
> >> >>>>> > > > >> > >> >> internally,
> >> >>>>> > > > >> > >> >> >>> and
> >> >>>>> > > > >> > >> >> >>> > >> then
> >> >>>>> > > > >> > >> >> >>> > >> > let
> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> deploying
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> the
> >> app
> >> >>>> figure
> >> >>>>> > out an
> >> >>>>> > > > >> > available
> >> >>>>> > > > >> > >> >> >>> namespace
> >> >>>>> > > > >> > >> >> >>> > ID
> >> >>>>> > > > >> > >> >> >>> > >> > for the
> >> >>>>> > > > >> > >> >> >>> > >> > > app
> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > live
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > So
> >> >>>>> > > > >> > io.confluent.schema-registry
> >> >>>>> > > > >> > >> can
> >> >>>>> > > > >> > >> >> be
> >> >>>>> > > > >> > >> >> >>> > >> > namespace
> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> deployment
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> and
> >> >> 0x57
> >> >>>> on
> >> >>>>> > yours, and
> >> >>>>> > > > the
> >> >>>>> > > > >> > poor
> >> >>>>> > > > >> > >> guys
> >> >>>>> > > > >> > >> >> >>> > >> > developing the
> >> >>>>> > > > >> > >> >> >>> > >> > > app
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> worry
> >> >>>> about that.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> Gwen,
> >> if
> >> >> I
> >> >>>>> > understand
> >> >>>>> > > your
> >> >>>>> > > > >> > example
> >> >>>>> > > > >> > >> >> >>> right, an
> >> >>>>> > > > >> > >> >> >>> > >> > > application
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> decide to
> >> >>>> use 0x01
> >> >>>>> > in one
> >> >>>>> > > > >> > >> deployment,
> >> >>>>> > > > >> > >> >> and
> >> >>>>> > > > >> > >> >> >>> > that
> >> >>>>> > > > >> > >> >> >>> > >> > means
> >> >>>>> > > > >> > >> >> >>> > >> > > that
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> message
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> is
> >> >> written
> >> >>>> into the
> >> >>>>> > > > broker, it
> >> >>>>> > > > >> > >> will be
> >> >>>>> > > > >> > >> >> >>> > saved on
> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >>>>> > > > >> > >> >> >>> > >> > > broker
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> specific
> >> >>>> namespace
> >> >>>>> > > (0x01).
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> If
> you
> >> >>>> were to
> >> >>>>> > mirror
> >> >>>>> > > that
> >> >>>>> > > > >> > message
> >> >>>>> > > > >> > >> >> into
> >> >>>>> > > > >> > >> >> >>> > another
> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> accompany
> >> >>>> the
> >> >>>>> > message,
> >> >>>>> > > > right?
> >> >>>>> > > > >> > What
> >> >>>>> > > > >> > >> if
> >> >>>>> > > > >> > >> >> the
> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> other
> >> >>>> cluster uses
> >> >>>>> > 0x57?
> >> >>>>> > > > They
> >> >>>>> > > > >> > won't
> >> >>>>> > > > >> > >> >> >>> > understand
> >> >>>>> > > > >> > >> >> >>> > >> > each
> >> >>>>> > > > >> > >> >> >>> > >> > > other?
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> I'm
> >> not
> >> >>>> sure
> >> >>>>> > that's an
> >> >>>>> > > > >> avoidable
> >> >>>>> > > > >> > >> >> >>> problem. I
> >> >>>>> > > > >> > >> >> >>> > >> > think it
> >> >>>>> > > > >> > >> >> >>> > >> > > simply
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> order
> >> to
> >> >>>> share
> >> >>>>> > data, you
> >> >>>>> > > > have
> >> >>>>> > > > >> to
> >> >>>>> > > > >> > >> also
> >> >>>>> > > > >> > >> >> >>> have a
> >> >>>>> > > > >> > >> >> >>> > >> > shared
> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>> understanding of
> >> >>>>> > what the
> >> >>>>> > > > >> > >> namespaces
> >> >>>>> > > > >> > >> >> >>> mean.
> >> >>>>> > > > >> > >> >> >>> > >> Which
> >> >>>>> > > > >> > >> >> >>> > >> > I
> >> >>>>> > > > >> > >> >> >>> > >> > > think
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> because
> >> >> the
> >> >>>>> > alternate
> >> >>>>> > > > (sharing
> >> >>>>> > > > >> > >> >> *nothing*
> >> >>>>> > > > >> > >> >> >>> at
> >> >>>>> > > > >> > >> >> >>> > >> all)
> >> >>>>> > > > >> > >> >> >>> > >> > would
> >> >>>>> > > > >> > >> >> >>> > >> > > mean
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> would
> >> be
> >> >>>> no way to
> >> >>>>> > > > understand
> >> >>>>> > > > >> > each
> >> >>>>> > > > >> > >> >> other.
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> -James
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> Gwen
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
> >> Tue,
> >> >>>> Nov 8,
> >> >>>>> > 2016 at
> >> >>>>> > > > 4:23
> >> >>>>> > > > >> > PM,
> >> >>>>> > > > >> > >> >> radai <
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >>>> radai.rosenblatt@gmail.com
> >> >>>>> > >
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> wrote:
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
> +1
> >> for
> >> >>>> sean's
> >> >>>>> > > document.
> >> >>>>> > > > it
> >> >>>>> > > > >> > >> covers
> >> >>>>> > > > >> > >> >> >>> pretty
> >> >>>>> > > > >> > >> >> >>> > >> much
> >> >>>>> > > > >> > >> >> >>> > >> > all
> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
> >> >> provides
> >> >>>>> > concrete
> >> >>>>> > > > figures
> >> >>>>> > > > >> to
> >> >>>>> > > > >> > >> argue
> >> >>>>> > > > >> > >> >> >>> about
> >> >>>>> > > > >> > >> >> >>> > :-)
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
> >> >>>> (nit-picking -
> >> >>>>> > used
> >> >>>>> > > the
> >> >>>>> > > > >> same
> >> >>>>> > > > >> > >> xkcd
> >> >>>>> > > > >> > >> >> >>> twice,
> >> >>>>> > > > >> > >> >> >>> > >> also
> >> >>>>> > > > >> > >> >> >>> > >> > trove
> >> >>>>> > > > >> > >> >> >>> > >> > > has
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> superceded
> >> >>>>> > > > >> > >> >
> >> >>>>> > > > >> >
> >> >>>>> > > > >> >
> >> >>>>> > > > >> >
> >> >>>>> > > > >> > --
> >> >>>>> > > > >> > Gwen Shapira
> >> >>>>> > > > >> > Product Manager | Confluent
> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >> >>>>> > > > >> > Follow us: Twitter | blog
> >> >>>>> > > > >> >
> >> >>>>> > > > >>
> >> >>>>> > > > >>
> >> >>>>> > > > >>
> >> >>>>> > > > >> --
> >> >>>>> > > > >> *Todd Palino*
> >> >>>>> > > > >> Staff Site Reliability Engineer
> >> >>>>> > > > >> Data Infrastructure Streaming
> >> >>>>> > > > >>
> >> >>>>> > > > >>
> >> >>>>> > > > >>
> >> >>>>> > > > >> linkedin.com/in/toddpalino
> >> >>>>> > > > >>
> >> >>>>> > > >
> >> >>>>> > > >
> >> >>>>> > > >
> >> >>>>> > > > --
> >> >>>>> > > > Gwen Shapira
> >> >>>>> > > > Product Manager | Confluent
> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >> >>>>> > > > Follow us: Twitter | blog
> >> >>>>> > > >
> >> >>>>> > >
> >> >>>>> >
> >> >>>>> >
> >> >>>>> > The information contained in this email is strictly
> >> confidential
> >> >>>> and for
> >> >>>>> > the use of the addressee only, unless otherwise indicated.
> If
> >> you
> >> >>>> are not
> >> >>>>> > the intended recipient, please do not read, copy, use or
> >> disclose
> >> >>>> to others
> >> >>>>> > this message or any attachment. Please also notify the
> sender
> >> by
> >> >>>> replying
> >> >>>>> > to this email or by telephone (+44(020 7896 0011) and then
> >> delete
> >> >>>> the email
> >> >>>>> > and any copies of it. Opinions, conclusion (etc) that do not
> >> >>>> relate to the
> >> >>>>> > official business of this company shall be understood as
> >> neither
> >> >>>> given nor
> >> >>>>> > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> >> >>>> company
> >> >>>>> > registered in England and Wales, company number 04008957)
> and
> >> IG
> >> >>>> Index
> >> >>>>> > Limited (a company registered in England and Wales, company
> >> >> number
> >> >>>>> > 01190902). Registered address at Cannon Bridge House, 25
> >> Dowgate
> >> >>>> Hill,
> >> >>>>> > London EC4R 2YA. Both IG Markets Limited (register number
> >> 195355)
> >> >>>> and IG
> >> >>>>> > Index Limited (register number 114059) are authorised and
> >> >>>> regulated by the
> >> >>>>> > Financial Conduct Authority.
> >> >>>>> >
> >> >>>>>
> >> >>>>>
> >> >>>>> The information contained in this email is strictly confidential
> and
> >> >> for
> >> >>>> the use of the addressee only, unless otherwise indicated. If you
> are
> >> >> not
> >> >>>> the intended recipient, please do not read, copy, use or disclose
> to
> >> >> others
> >> >>>> this message or any attachment. Please also notify the sender by
> >> >> replying
> >> >>>> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> >> >> email
> >> >>>> and any copies of it. Opinions, conclusion (etc) that do not relate
> >> to
> >> >> the
> >> >>>> official business of this company shall be understood as neither
> >> given
> >> >> nor
> >> >>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >> >>>> registered in England and Wales, company number 04008957) and IG
> >> Index
> >> >>>> Limited (a company registered in England and Wales, company number
> >> >>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> >> Hill,
> >> >>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> >> and IG
> >> >>>> Index Limited (register number 114059) are authorised and regulated
> >> by
> >> >> the
> >> >>>> Financial Conduct Authority.
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>
> >> >>
> >> >>
> >> >
> >>
> >>
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Late reply to Jays comment:
Control messages for Streams can be implements in different ways and
using record headers is just one possibility.
However, control messages for Streams a different than for EOS --
brokers do not need to understand them, they are at the application
level (ie, client-to-client) level.
However, there was already a new mailing thread to discuss control
messages and I will pick up this thread to drive the discussion.
The beauty about using record headers to implement control messages
would still be, that we do not need a new concept but can build one upon
another.
-Matthias
On 1/9/17 11:19 AM, Jay Kreps wrote:
> Hey Michael,
>
> Sorry for lagging, I've been out for the holiday break. What would help me
> would be to get to a consolidated list of use cases we've worked out and
> think add real value. I think getting down to the details really matters
> for these.
>
> The two lists I'm aware of are the one in the KIP and the one you gave in
> this email thread. Both seem to be more at the sketch level. Let me review
> the use cases and see if I have it right. If so let's get it in the design
> proposal.
>
> Your use cases:
>
> 1. Tracing IDs -- I get this use case and think I see how it would work.
> I think because the tracing is from a third party system there is some
> value to having a header versus a custom format to make this work fully
> transparently.
> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
> thinking below
> 3. End-to-end encryption -- I think what you are saying is that if you
> want "routing' fields that are accessible in transit without decrypting you
> need a scheme to leave this decrypted.I think the idea is to avoid
> decrypting to get a header, but I'm not sure if I get how this is better
> than a custom message format that doesn't encrypt the things you don't want
> encrypted since you have to be aware of the scheme anyway.
> 4. EOS -- I'm not convinced this is a good approach for EOS.
>
> Linked from KIP:
>
> 1. Schema-Registry-aware avro serialization -- not clear how this is
> better than keeping the schema id in the value
> 2. Polyglot kafka topics -- marginally better than a container format,
> but of course you still have to all agree on the the header and
> header=>serialization mapping so not sure if it's really better than a
> container format. Also not wild about polyglot topics :-)
> 3. End-to-End routing trail including mirroring --- I'd put this under
> tracing ids
> 4. Cross-service process tracing -- ditto
> 5. Signing -- I like this use case but it seems to depend on a feature
> we don't have and aren't proposing adding, right?
> 6. Per-record custom compression -- why do you want this? compressing
> per message won't work well compared to batch compression, right?
> 7. Large Message Support -- I think we can agree that although this
> might work it's clearly not the right way to support large messages in Kafka
>
> Matthias
>
> 1. Control messages --- Matthias, not sure if I get the motivation in
> streams for this. If we did want to do it, wouldn't it be easier to
> implement and by allowing users to send control messages (in the EOS sense)
> and adding a consumer config to allow either passing these on or
> suppressing them?
>
> So I if I were to summarize I'd list the use cases as follows:
>
> - Much better with headers: tracing IDs
> - Somewhat better: end-to-end encryption, polyglot topics
> - Don't buy: Mirroring (though perhaps could be made to work with
> changes to the proposal), EOS, large messages, compression
> - Not sure: Control messages
>
> Does this match where other people are at? From my point of view for
> something like this where we are changing the core data model we should
> ideally get a handful of use cases to the point where they are really well
> thought out and we know they can use the feature. I think we have one solid
> one with tracing IDs, and I think maybe mirroring could be another with
> some work. I'm less sure about the others.
>
> Bi-directional Mirroring
> I think bidirectional mirroring could potentially become a compelling use
> case if we an find a way to make it work. If I understand it, the proposal
> is to add a header like "origin-cluster" with the cluster id of the Kafka
> cluster. Then modify the mirroring tool to disallow mirroring a message
> back to it's own origin cluster. This basic idea is definitely workable and
> useful, but can we really get to a good implementation using headers?
> I imagine the approach would be to add this header in the producer if some
> config is set. There are two problems:
>
> - You add a pretty big honking cluster ID to each message. For small
> messages this maybe larger than the contents itself.
> - What happens if someone somewhere forgets to set that config on some
> client somewhere?
>
> If someone forgets to set the config in some client somewhere I guess you
> have various options in mirror maker:
>
> 1. drop the data (pretty bad, now your data is out of sync)
> 2. crash (pretty bad--how do you recover)
> 3. insert some default (pretty bad since would also lead to data loss if
> that's not right)
>
> Basically although I think a cluster tag could be a good way to do
> bi-directional mirroring, I'm not sure this naive approach I described
> would get us to a usable implementation. This could be fixable. I think if
> there was some scheme to check required headers on the server that could
> alleviate the missing header problem. Not really sure what to do about the
> message bloat problem, but maybe there could be some way to shrink the
> cluster id?
>
> Cheers,
>
> -Jay
>
>
> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> @Jay, Ismael and a few others.
>>
>> Based on the discussion on kip-87, and the additional use case by Matthias.
>>
>> Just for clarity do we now have consensus on the earlier points (1)
>> headers are useful (2) headers should be native in Kafka?
>>
>> Cheers
>> Mike
>> ________________________________________
>> From: radai <ra...@gmail.com>
>> Sent: Monday, December 19, 2016 5:01 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>> having said that im perfectly fine with something like a "no payload" flag
>> (as opposed to null payload), which would mean such a msg is not returned
>> from poll() unless the user code specifically "opts in" and would serve
>> only to carry headers (making it a control message)
>>
>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com> wrote:
>>
>>> if all you want is to tag "end of stream" wouldnt a header on the last
>> msg
>>> in the stream be enough?
>>>
>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> The details about headers for control messages are still to define. But
>>>> yes, the idea is to have some common default behavior that clients would
>>>> need to implement.
>>>>
>>>> The point is, that "regular headers" add meta data to regular messages.
>>>> Thus, those messages will be returned to the user via .poll(). And after
>>>> the message is received the user can check if meta data is present and
>>>> read it.
>>>>
>>>> For control messages, we do not want those to pop up via .poll() as
>>>> those are no regular messages. A client would need to opt-in to see
>>>> those messages (either via poll() or maybe a callback). Thus, we need
>>>> some special (standardized) header IDs that indicate control messages
>>>> that should not be returned to the user via poll() by default.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
>>>>> Matthias,
>>>>>
>>>>> Thanks for your input. I'm +1 on control messages as they seem to be
>>>> the
>>>>> simplest way to implement watermarks (
>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102),
>> a
>>>>> feature that would add a lot of value to Kafka Streams IMHO.
>>>>>
>>>>> Your argument that the control-message indicator belongs in the
>>>> client-only
>>>>> section of the record format make sense. Just to make sure I
>>>> understand,
>>>>> are you suggesting that control messages would be indicated by a
>>>> standard
>>>>> reserved header? By standard, I mean that ALL Kafka consumers would
>>>> know
>>>>> to handle these messages differently (possibly just ignoring them).
>>>> This
>>>>> would need to be added to the specification of the consumer protocol
>> so
>>>>> that all Kafka clients implement it, right? I think it's a good idea
>>>> but
>>>>> just checking.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Roger
>>>>>
>>>>>
>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>> matthias@confluent.io
>>>>>
>>>>> wrote:
>>>>>
>>>>>> Yes and no. I did overload the term "control message".
>>>>>>
>>>>>> EOS control messages are for client-broker communication and thus
>> never
>>>>>> exposed to any application. And I think this is a good design because
>>>>>> broker needs to understand those control messages. Thus, this should
>> be
>>>>>> a protocol change.
>>>>>>
>>>>>> The type of control messages I have in mind are for client-client
>>>>>> (application-application) communication and the broker is agnostic to
>>>>>> them. Thus, it should not be a protocol change.
>>>>>>
>>>>>>
>>>>>> -Matthias
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 12/14/16 9:42 AM, radai wrote:
>>>>>>> arent control messages getting pushed as their own top level
>> protocol
>>>>>>> change (and a fairly massive one) for the transactions KIP ?
>>>>>>>
>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>>>> matthias@confluent.io>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I want to add a completely new angle to this discussion. For this,
>> I
>>>>>>>> want to propose an extension for the headers feature that enables
>> new
>>>>>>>> uses cases -- and those new use cases might convince people to
>>>> support
>>>>>>>> headers (of course including the larger scoped proposal).
>>>>>>>>
>>>>>>>> Extended Proposal:
>>>>>>>>
>>>>>>>> Allow messages with a certain header key to be special "control
>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an application
>>>> via
>>>>>>>> .poll().
>>>>>>>>
>>>>>>>> Thus, a consumer client would automatically skip over those
>>>> messages. If
>>>>>>>> an application knows about embedded control messages, it can "sing
>>>> up"
>>>>>>>> to those messages by the consumer client and either get a callback
>> or
>>>>>>>> the consumer auto-drop for this messages gets disabled (allowing to
>>>>>>>> consumer those messages via poll()).
>>>>>>>>
>>>>>>>> (The details need further considerations/discussion. I just want to
>>>>>>>> sketch the main idea.)
>>>>>>>>
>>>>>>>> Usage:
>>>>>>>>
>>>>>>>> There is a shared topic (ie, used by multiple applications) and a
>>>>>>>> producer application wants to embed a special message in the topic
>>>> for a
>>>>>>>> dedicated consumer application. Because only one application will
>>>>>>>> understand this message, it cannot be a regular message as this
>> would
>>>>>>>> break all applications that do not understand this message. The
>>>> producer
>>>>>>>> application would set a special metadata key and no consumer
>>>> application
>>>>>>>> would see this control message by default because they did not
>> enable
>>>>>>>> their consumer client to return this message in poll() (and the
>>>> client
>>>>>>>> would just drop this message with special metadata key). Only the
>>>> single
>>>>>>>> application that should receive this message, will subscribe to
>> this
>>>>>>>> message on its consumer client and process it.
>>>>>>>>
>>>>>>>>
>>>>>>>> Concrete Use Case: Kafka Streams
>>>>>>>>
>>>>>>>> In Kafka Streams, we would like to propagate "control messages"
>> from
>>>>>>>> subtopology to subtopology. There are multiple scenarios for which
>>>> this
>>>>>>>> would be useful. For example, currently we do not guarantee a
>>>>>>>> "consistent shutdown" of an application. By this, I mean that input
>>>>>>>> records might not be completely processed by the whole topology
>>>> because
>>>>>>>> the application shutdown happens "in between" and an intermediate
>>>> result
>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user would see
>>>> an
>>>>>>>> committed offset of the source topic of the application, but no
>>>>>>>> corresponding result record in the output topic.
>>>>>>>>
>>>>>>>> Having "shutdown markers" would allow us, to first stop the
>> upstream
>>>>>>>> subtopology and write this marker into the intermediate topic and
>> the
>>>>>>>> downstream subtopology would only shut down itself after is sees
>> the
>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
>>>>>>>> "in-flight" messages got stuck in intermediate topics.
>>>>>>>>
>>>>>>>>
>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch Processing).
>>>>>>>> There was a discussion about the proposed metadata topic, and we
>>>> could
>>>>>>>> avoid this metadata topic if we would have "control messages".
>>>>>>>>
>>>>>>>>
>>>>>>>> Right now, we cannot insert an "application control message"
>> because
>>>>>>>> Kafka Streams does not own all topics it read/writes and thus might
>>>>>>>> break other consumer application (as described above) if we inject
>>>>>>>> random messages that are not understood by other apps.
>>>>>>>>
>>>>>>>>
>>>>>>>> Of course, one can work around "embedded control messaged" by using
>>>> an
>>>>>>>> additional topic to propagate control messaged between application
>>>> (as
>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
>>>> there
>>>>>>>> are major concerns about adding this metadata topic in the KIP and
>>>> this
>>>>>>>> shows that other application that need a similar pattern might
>> profit
>>>>>>>> from topic embedded "control messages", too.
>>>>>>>>
>>>>>>>>
>>>>>>>> One last important consideration: those "control messages" are used
>>>> for
>>>>>>>> client to client communication and are not understood by the
>> broker.
>>>>>>>> Thus, those messages should not be enabled within the message
>> format
>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
>>>> headers
>>>>>>>> would be a nice way to implement them. Because KIP-82 did consider
>>>> key
>>>>>>>> namespaces for metatdata keys, this extension should not be an own
>>>> KIP
>>>>>>>> but should be included in KIP-82 to reserve a namespace for
>> "control
>>>>>>>> message" in the first place.
>>>>>>>>
>>>>>>>>
>>>>>>>> Sorry for the long email... Looking forward to your feedback.
>>>>>>>>
>>>>>>>>
>>>>>>>> -Matthias
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>>>>>>>>> Hi Jun
>>>>>>>>>
>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
>>>> (HTTP/JMS/
>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
>> (though I
>>>>>>>> believe it contains the end2end uuid embedded in this id), on
>>>> receiving
>>>>>> the
>>>>>>>> message at the receiving JVM the apm code takes this out, and
>>>> continues
>>>>>> its
>>>>>>>> tracing on the that new thread. Both JVM’s (and other languages the
>>>> APM
>>>>>>>> tool supports) send this data async back to the central controllers
>>>>>> where
>>>>>>>> the stiching togeather occurs. For this they need some header space
>>>> for
>>>>>>>> them to put this id.
>>>>>>>>>
>>>>>>>>> 101) Yes indeed we have a business transaction Id in the payload.
>>>>>> Though
>>>>>>>> this is a system level tracing, that we need to have marry up. Also
>>>> as
>>>>>> per
>>>>>>>> note on end2end encryption we’d be unable to prove the flow if the
>>>>>> payload
>>>>>>>> is encrypted as we’d not have access to this at certain points of
>> the
>>>>>> flow
>>>>>>>> through the infrastructure/platform.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 103) As said we use this mechanism in IG very successfully, as
>>>> stated
>>>>>>>> per key we guarantee the transaction producing app to handle the
>>>>>>>> transaction of a key at one DC unless at point of critical failure
>>>>>> where we
>>>>>>>> have to flip processing to another. We care about key ordering.
>>>>>>>>> I disagree on the offset comment for the partition solution unless
>>>> you
>>>>>>>> do full ISR, or expensive full XA transactions even with partitions
>>>> you
>>>>>>>> cannot fully guarantee offsets would match.
>>>>>>>>>
>>>>>>>>> 105) Very much so, I need to have access at the platform level to
>>>> the
>>>>>>>> other meta data all mentioned, without having to need to have
>> access
>>>> to
>>>>>> the
>>>>>>>> encryption keys of the payload.
>>>>>>>>>
>>>>>>>>> 106)
>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we’d need to have
>> a
>>>>>>>> global producerId register which would be very hard to
>>>> enforce/ensure is
>>>>>>>> current and correct, just to understand the message origins of its
>>>>>>>> region/az/cluster for routing.
>>>>>>>>> The client wrapper version, producerId can be the same, as
>> obviously
>>>>>> the
>>>>>>>> producer could upgrade its wrapper, as such we need to know what
>>>> wrapper
>>>>>>>> version the message is created with.
>>>>>>>>> Likewise the IP address, as stated we can have our producer move,
>>>> where
>>>>>>>> its IP would change.
>>>>>>>>>
>>>>>>>>> 107)
>>>>>>>>> UUID is set on the message by interceptors before actual producer
>>>>>>>> transport send. This is for platform level message dedupe
>> guarantee,
>>>> the
>>>>>>>> business payload should be agnostic to this. Please see
>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
>>>> ection.html
>>>>>>>> note this is not touching business payloads.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
>>>>>>>>>
>>>>>>>>> Hi, Michael,
>>>>>>>>>
>>>>>>>>> Thanks for the reply. I find it very helpful.
>>>>>>>>>
>>>>>>>>> Data lineage:
>>>>>>>>> 100. I'd like to understand the APM use case a bit more. It
>>>> sounds
>>>>>>>> like
>>>>>>>>> that those APM plugins can generate a transaction id that we
>>>> could
>>>>>>>>> potentially put in the header of every message. How would you
>>>>>>>> typically
>>>>>>>>> make use of such transaction ids? Are there other metadata
>>>>>>>> associated with
>>>>>>>>> the transaction id and if so, how are they propagated
>>>> downstream?
>>>>>>>>>
>>>>>>>>> 101. For the finance use case, if the concept of transaction
>> is
>>>>>>>> important,
>>>>>>>>> wouldn't it be typically included in the message payload
>>>> instead of
>>>>>>>> as an
>>>>>>>>> optional header field?
>>>>>>>>>
>>>>>>>>> 102. The data lineage that Altas and Navigator support seems
>> to
>>>> be
>>>>>>>> at the
>>>>>>>>> dataset level, not per record level? So, not sure if per
>> message
>>>>>>>> headers
>>>>>>>>> are relevant there.
>>>>>>>>>
>>>>>>>>> Mirroring:
>>>>>>>>> 103. The benefit of using separate partitions is that it
>>>>>> potentially
>>>>>>>> makes
>>>>>>>>> it easy to preserve offsets during mirroring. This will make
>> it
>>>>>>>> easier for
>>>>>>>>> consumer to switch clusters. Currently, the consumers can
>> switch
>>>>>>>> clusters
>>>>>>>>> by using the timestampToOffset() api, but it has to deal with
>>>>>>>> duplicates.
>>>>>>>>> Good point on the issue with log compact and I am not sure how
>>>> to
>>>>>>>> address
>>>>>>>>> this. However, even if we mirror into the existing partitions,
>>>> the
>>>>>>>> ordering
>>>>>>>>> for messages generated from different clusters seems
>>>>>>>> non-deterministic
>>>>>>>>> anyway. So, it seems that the consumers already have to deal
>>>> with
>>>>>>>> that? If
>>>>>>>>> a topic is compacted, does that mean which messages are
>>>> preserved
>>>>>> is
>>>>>>>> also
>>>>>>>>> non-deterministic across clusters?
>>>>>>>>>
>>>>>>>>> 104. Good point on partition key.
>>>>>>>>>
>>>>>>>>> End-to-end encryption:
>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are headers
>>>>>>>> useful there?
>>>>>>>>>
>>>>>>>>> Auditing:
>>>>>>>>> 106. It seems other than the UUID, all other metadata are per
>>>>>>>> producer?
>>>>>>>>>
>>>>>>>>> EOS:
>>>>>>>>> 107. How are those UUIDs generated? I am not sure if they can
>> be
>>>>>>>> generated
>>>>>>>>> in the producer library. An application may send messages
>>>> through a
>>>>>>>> load
>>>>>>>>> balancer and on retry, the same message could be routed to a
>>>>>>>> different
>>>>>>>>> producer instance. So, it seems that the application has to
>>>>>> generate
>>>>>>>> the
>>>>>>>>> UUIDs. In that case, shouldn't the application just put the
>>>> UUID in
>>>>>>>> the
>>>>>>>>> payload?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>>>>>>>> Michael.Pearce@ig.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> > Hi Jun.
>>>>>>>>> >
>>>>>>>>> > Per Transaction Tracing / Data Lineage.
>>>>>>>>> >
>>>>>>>>> > As Stated in the KIP this has the first use case of how many
>>>> APM
>>>>>>>> tools now
>>>>>>>>> > work.
>>>>>>>>> > I would find it impossible for any one to argue this is not
>>>>>>>> important or a
>>>>>>>>> > niche market as it has its own gartner report for this
>> space.
>>>>>> Such
>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular are
>>>> but a
>>>>>>>> few.
>>>>>>>>> >
>>>>>>>>> > Likewise these APM tools can help very rapidly track down
>>>> issues
>>>>>>>> and
>>>>>>>>> > automatically capture metrics, perform actions based on
>>>>>> unexpected
>>>>>>>> behavior
>>>>>>>>> > to auto recover services.
>>>>>>>>> >
>>>>>>>>> > Before mentioning looking at aggregated stats, in these
>> cases
>>>>>> where
>>>>>>>>> > actually on critical flows we cannot afford to have
>> aggregated
>>>>>>>> rolled up
>>>>>>>>> > stats only.
>>>>>>>>> >
>>>>>>>>> > With the APM tool we use its actually able to detect a
>> single
>>>>>>>> transaction
>>>>>>>>> > failure and capture the thread traces in the JVM where it
>>>> failed
>>>>>>>> and
>>>>>>>>> > everything for us, to the point it sends us alerts where we
>>>> have
>>>>>>>> this
>>>>>>>>> > giving the line number of the code that caused it, the
>>>>>> transaction
>>>>>>>> trace
>>>>>>>>> > through all the services and endpoints (supported) upto the
>>>> point
>>>>>>>> of
>>>>>>>>> > failure, it can also capture the data in and out (so we can
>>>>>>>> replay).
>>>>>>>>> > Because atm Kafka doesn’t support us being able to stich in
>>>> these
>>>>>>>> tracing
>>>>>>>>> > transaction ids natively, we cannot get these benefits as
>>>> such is
>>>>>>>> limiting
>>>>>>>>> > our ability support apps and monitor them to the same
>>>> standards
>>>>>> we
>>>>>>>> come to
>>>>>>>>> > expect when on a kafka flow.
>>>>>>>>> >
>>>>>>>>> > This actually ties in with Data Lineage, as the same tracing
>>>> can
>>>>>>>> be used
>>>>>>>>> > to back stich this. Essentially many times due to the sums
>> of
>>>>>> money
>>>>>>>>> > involved there are disputes, and typically as a financial
>>>>>>>> institute the
>>>>>>>>> > easiest and cleanest way to prove when disputes arise is to
>>>>>>>> present the
>>>>>>>>> > actual flow and processes involved in a transaction.
>>>>>>>>> >
>>>>>>>>> > Likewise as Hadoop matures its evident this case is
>>>> important, as
>>>>>>>> tools
>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera led)
>>>> are
>>>>>>>> evident
>>>>>>>>> > also I believe the importance here is very much NOT just a
>>>>>>>> financial issue.
>>>>>>>>> >
>>>>>>>>> > From a MDM point of view any company wanting to care about
>>>> Data
>>>>>>>> Quality
>>>>>>>>> > and Data Governance - Data Lineage is a key piece in this
>>>> puzzle.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > RE Mirroring,
>>>>>>>>> >
>>>>>>>>> > As per the KIP in-fact this is exactly what we do re cluster
>>>> id,
>>>>>>>> to mirror
>>>>>>>>> > a network of clusters between AZ’s / Regions. We know a
>>>>>>>> transaction for a
>>>>>>>>> > key will be done within a AZ/Region, as such we know the
>>>> write
>>>>>> to
>>>>>>>> kafka
>>>>>>>>> > would be ordered per key. But we need eventual view of that
>>>>>> across
>>>>>>>> in our
>>>>>>>>> > other regions/az’s. When we have complete AZ or Region
>>>> failure we
>>>>>>>> know
>>>>>>>>> > there will be a brief interruption whilst those transactions
>>>> are
>>>>>>>> moved to
>>>>>>>>> > another region but we expect after it to continue.
>>>>>>>>> >
>>>>>>>>> > As mentioned having separate Partions to do this starts to
>> get
>>>>>>>>> > ugly/complicated for us:
>>>>>>>>> > how would I do compaction where a key is in two partitions?
>>>>>>>>> > How do we balance consumers so where multiple partitions
>> with
>>>> the
>>>>>>>> same key
>>>>>>>>> > goto the same consumer
>>>>>>>>> > What do you do if cluster 1 has 5 partitions but cluster 20
>>>> has
>>>>>> 10
>>>>>>>> because
>>>>>>>>> > its larger kit in our more core DC’s, as such key to
>> partition
>>>>>>>> mappings for
>>>>>>>>> > consumers get even more complicated.
>>>>>>>>> > What do you do if we add or remove a complete region
>>>>>>>>> >
>>>>>>>>> > Where as simple mirror will work we just need to ensure we
>>>> don’t
>>>>>>>> have a
>>>>>>>>> > cycle which we can do with clusterId.
>>>>>>>>> >
>>>>>>>>> > We even have started to look at shortest path mirror routing
>>>>>> based
>>>>>>>> on
>>>>>>>>> > clusterId, if we also had the region and az info on the
>>>>>> originating
>>>>>>>>> > message, this we have not implemented but some ideas come
>> from
>>>>>>>> network
>>>>>>>>> > routing, and also the dispatcher router in apache qpid.
>>>>>>>>> >
>>>>>>>>> > Also we need to have data perimeters e.g. certain data
>> cannot
>>>>>> leave
>>>>>>>>> > certain countries borders. We want this all automated so
>> that
>>>> at
>>>>>>>> the
>>>>>>>>> > platform level without having to touch or look at the
>> business
>>>>>>>> data inside
>>>>>>>>> > we can have headers we can put tags into so that we can
>> ensure
>>>>>>>> this doesn’t
>>>>>>>>> > occur when we mirror. (actually links in to data lineage /
>>>>>> tracing
>>>>>>>> as again
>>>>>>>>> > we need to tag messages at a platform level) Examples are we
>>>> are
>>>>>>>> not
>>>>>>>>> > allowed Private customer details to leave Switzerland, yet
>> we
>>>>>> need
>>>>>>>> those
>>>>>>>>> > systems integrated.
>>>>>>>>> >
>>>>>>>>> > Lastly around mirroring we have a partionKey field, as the
>> key
>>>>>>>> used for
>>>>>>>>> > portioning logic != compaction key all the time but we want
>> to
>>>>>>>> preserve it
>>>>>>>>> > for when we mirror so that if source cluster partition count
>>>> !=
>>>>>>>> destination
>>>>>>>>> > cluster partition count we can honour the same partitioning
>>>>>> logic.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > RE End 2 End encryption
>>>>>>>>> >
>>>>>>>>> > As I believe mentioned just before, the solution you mention
>>>> just
>>>>>>>> doesn’t
>>>>>>>>> > cut the mustard these days with many regulators. An
>> operations
>>>>>>>> person with
>>>>>>>>> > access to the box should not be able to have access to the
>>>> data.
>>>>>>>> Many now
>>>>>>>>> > actually impose quite literally the implementation expected
>>>> being
>>>>>>>> end2end
>>>>>>>>> > encryption for certain data (Singapore for us is one that I
>> am
>>>>>>>> most aware
>>>>>>>>> > of). In fact we’re even now needing encrypt the data and
>> store
>>>>>> the
>>>>>>>> keys in
>>>>>>>>> > HSM modules.
>>>>>>>>> >
>>>>>>>>> > Likewise the performance penalty on encrypting decrypting as
>>>> you
>>>>>>>> produce
>>>>>>>>> > over wire, then again encrypt decrypt as the data is stored
>> on
>>>>>> the
>>>>>>>> brokers
>>>>>>>>> > disks and back again, then again encrypted and decrypted
>> back
>>>>>> over
>>>>>>>> the wire
>>>>>>>>> > each time for each consumer all adds up, ignoring this
>>>> doubling
>>>>>>>> with mirror
>>>>>>>>> > makers etc. simply encrypting the value once on write by the
>>>>>>>> client and
>>>>>>>>> > again decrypting on consume by the consumer is far more
>>>>>>>> performant, but
>>>>>>>>> > then the routing and platform meta data needs to be separate
>>>>>> (thus
>>>>>>>> headers)
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > RE Auditing:
>>>>>>>>> >
>>>>>>>>> > Our Auditing needs are:
>>>>>>>>> > Producer Id,
>>>>>>>>> > Origin Cluster Id that message first produced into
>>>>>>>>> > Origin AZ – agreed we can derive this if we have cluster id,
>>>> but
>>>>>>>> it makes
>>>>>>>>> > resolving this for audit reporting a lot easier.
>>>>>>>>> > Origin Region – agreed we can derive this if we have cluster
>>>> id,
>>>>>>>> but it
>>>>>>>>> > makes resolving this for audit reporting a lot easier.
>>>>>>>>> > Unique Message Identification (this is not the same as
>>>>>> transaction
>>>>>>>>> > tracing) – note offset and partition are not the same, as
>>>> when we
>>>>>>>> mirror or
>>>>>>>>> > have for what ever system failure duplicate send,
>>>>>>>>> > Custom Client wrapper version (where organizations have to
>>>> wrap
>>>>>>>> the kafka
>>>>>>>>> > client for added features) so we know what version of the
>>>> wrapper
>>>>>>>> is used
>>>>>>>>> > Producer IP address (in case of clients being in our vm/open
>>>>>> stack
>>>>>>>> infra
>>>>>>>>> > where they can move around, producer id will stay the same
>> but
>>>>>>>> this would
>>>>>>>>> > change)
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > RE Once and only once delivery case
>>>>>>>>> >
>>>>>>>>> > Using the same Message UUID for auditing we can achieve this
>>>>>> quite
>>>>>>>> simply.
>>>>>>>>> >
>>>>>>>>> > As per how some other brokers do this (cough qpid, artemis)
>>>>>>>> message uuid
>>>>>>>>> > are used to dedupe where message is sent and produced but
>> the
>>>>>>>> client didn’t
>>>>>>>>> > receive the ack, and there for replays the send, by having a
>>>>>>>> unique message
>>>>>>>>> > id per message, this can be filtered out, on consumers where
>>>>>>>> message
>>>>>>>>> > delivery may occur twice for what ever reasons a message
>> uuid
>>>> can
>>>>>>>> be used
>>>>>>>>> > to remove duplicates being deliverd , like wise we can do
>>>> this in
>>>>>>>> the
>>>>>>>>> > mirrormakers so if we detect a dupe message we can avoid
>>>>>>>> replicating it.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Cheers
>>>>>>>>> > Mike
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io> wrote:
>>>>>>>>> >
>>>>>>>>> > Since this KIP affects message format, wire protocol,
>>>> apis, I
>>>>>>>> think
>>>>>>>>> > it's
>>>>>>>>> > worth spending a bit more time to nail down the concrete
>>>> use
>>>>>>>> cases. It
>>>>>>>>> > would be bad if we add this feature, but when start
>>>>>>>> implementing it
>>>>>>>>> > for say
>>>>>>>>> > mirroring, we then realize that header is not the best
>>>>>>>> approach.
>>>>>>>>> > Initially,
>>>>>>>>> > I thought I was convinced of the use cases of headers
>> and
>>>> was
>>>>>>>> trying to
>>>>>>>>> > write down a few use cases to convince others. That's
>>>> when I
>>>>>>>> became
>>>>>>>>> > less
>>>>>>>>> > certain. For me to be convinced, I just want to see two
>>>>>> strong
>>>>>>>> use
>>>>>>>>> > cases
>>>>>>>>> > (instead of 10 maybe use cases) in the third-party
>> space.
>>>> The
>>>>>>>> reason is
>>>>>>>>> > that when we discussed the use cases within a company,
>>>> often
>>>>>>>> it ends
>>>>>>>>> > with
>>>>>>>>> > "we can't force everyone to use this standard since we
>> may
>>>>>>>> have to
>>>>>>>>> > integrate with third-party tools".
>>>>>>>>> >
>>>>>>>>> > At present, I am not sure why headers are useful for
>>>> things
>>>>>>>> like
>>>>>>>>> > schemaId
>>>>>>>>> > or encryption. In order to do anything useful to the
>>>> value,
>>>>>>>> one needs
>>>>>>>>> > to
>>>>>>>>> > know the schemaId or how data is encrypted, but header
>> is
>>>>>>>> optional.
>>>>>>>>> > But, I
>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
>> provides
>>>>>> more
>>>>>>>> details
>>>>>>>>> > on
>>>>>>>>> > the argument.
>>>>>>>>> >
>>>>>>>>> > I am not very sure header is the best approach for
>>>> mirroring
>>>>>>>> either. If
>>>>>>>>> > someone has thought about this more, I'd be happy to
>> hear.
>>>>>>>>> >
>>>>>>>>> > I can see the data lineage use case. I am just not sure
>>>> how
>>>>>>>> widely
>>>>>>>>> > applicable this is. If someone familiar with this space
>>>> can
>>>>>>>> justify
>>>>>>>>> > this is
>>>>>>>>> > a significant use case, say in the finance industry,
>> this
>>>>>>>> would be a
>>>>>>>>> > strong
>>>>>>>>> > use case.
>>>>>>>>> >
>>>>>>>>> > I can see the auditing use case. I am just not sure if a
>>>>>> native
>>>>>>>>> > producer id
>>>>>>>>> > solves that problem. If there are additional metadata
>>>> that's
>>>>>>>> worth
>>>>>>>>> > collecting but not covered by the producer id, that
>> would
>>>>>> make
>>>>>>>> this a
>>>>>>>>> > strong use case.
>>>>>>>>> >
>>>>>>>>> > Thanks,
>>>>>>>>> >
>>>>>>>>> > Jun
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>> > wrote:
>>>>>>>>> >
>>>>>>>>> > > this KIP is about enabling headers, nothing more
>> nothing
>>>>>>>> less - so
>>>>>>>>> > no,
>>>>>>>>> > > broker-side use of headers is not in the KIP scope.
>>>>>>>>> > >
>>>>>>>>> > > obviously though, once you have headers potential use
>>>> cases
>>>>>>>> could
>>>>>>>>> > include
>>>>>>>>> > > broker-side header-aware interceptors (which would be
>>>> the
>>>>>>>> topic of
>>>>>>>>> > other
>>>>>>>>> > > future KIPs).
>>>>>>>>> > >
>>>>>>>>> > > a trivially clear use case (to me) would be using such
>>>>>>>> broker-side
>>>>>>>>> > > interceptors to enforce compliance with organizational
>>>>>>>> policies - it
>>>>>>>>> > would
>>>>>>>>> > > make our SREs lives much easier if instead of
>>>> retroactively
>>>>>>>>> > discovering
>>>>>>>>> > > "rogue" topics/users those messages would have been
>>>>>> rejected
>>>>>>>>> > up-front.
>>>>>>>>> > >
>>>>>>>>> > > the kafka broker code is lacking any such
>> extensibility
>>>>>>>> support
>>>>>>>>> > (beyond
>>>>>>>>> > > maybe authorizer) which is why these use cases were
>> left
>>>>>> out
>>>>>>>> of the
>>>>>>>>> > "case
>>>>>>>>> > > for headers" doc - broker extensibility is a separate
>>>>>>>> discussion.
>>>>>>>>> > >
>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
>>>>>>>> gwen@confluent.io>
>>>>>>>>> > wrote:
>>>>>>>>> > >
>>>>>>>>> > > > Woah, I wasn't aware this is something we'll do. It
>>>>>> wasn't
>>>>>>>> in the
>>>>>>>>> > KIP,
>>>>>>>>> > > > right?
>>>>>>>>> > > >
>>>>>>>>> > > > I guess we could do it the same way ACLs currently
>>>> work.
>>>>>>>>> > > > I had in mind something that will allow admins to
>>>> apply
>>>>>>>> rules to
>>>>>>>>> > the
>>>>>>>>> > > > new create/delete/config topic APIs. So Todd can
>>>> decide
>>>>>> to
>>>>>>>> reject
>>>>>>>>> > > > "create topic" requests that ask for more than 40
>>>>>>>> partitions, or
>>>>>>>>> > > > require exactly 3 replicas, or no more than 50GB
>>>>>> partition
>>>>>>>> size,
>>>>>>>>> > etc.
>>>>>>>>> > > >
>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are planning to
>>>> apply
>>>>>>>> more
>>>>>>>>> > rules
>>>>>>>>> > > > to requests (and I think we should), we may want a
>> bit
>>>>>>>> more generic
>>>>>>>>> > > > design around that.
>>>>>>>>> > > >
>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>> > > wrote:
>>>>>>>>> > > > > "wouldn't you be in the business of making sure
>>>>>> everyone
>>>>>>>> uses
>>>>>>>>> > them
>>>>>>>>> > > > > properly?"
>>>>>>>>> > > > >
>>>>>>>>> > > > > thats where a broker-side plugin would come handy
>> -
>>>> any
>>>>>>>> incoming
>>>>>>>>> > > message
>>>>>>>>> > > > > that does not conform to org policy (read - does
>> not
>>>>>>>> have the
>>>>>>>>> > proper
>>>>>>>>> > > > > headers) gets thrown out (with an error returned
>> to
>>>>>> user)
>>>>>>>>> > > > >
>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
>>>>>>>> tpalino@gmail.com>
>>>>>>>>> > wrote:
>>>>>>>>> > > > >
>>>>>>>>> > > > >> Come on, I’ve done at least 2 talks on this one
>> :)
>>>>>>>>> > > > >>
>>>>>>>>> > > > >> Producing counts to a topic is part of it, but
>>>> that’s
>>>>>>>> only
>>>>>>>>> > part. So
>>>>>>>>> > > you
>>>>>>>>> > > > >> count you have 100 messages in topic A. When you
>>>>>> mirror
>>>>>>>> topic A
>>>>>>>>> > to
>>>>>>>>> > > > another
>>>>>>>>> > > > >> cluster, you have 99 messages. Where was your
>>>> problem?
>>>>>>>> Or
>>>>>>>>> > worse, you
>>>>>>>>> > > > have
>>>>>>>>> > > > >> 100 messages, but one producer duplicated
>> messages
>>>> and
>>>>>>>> another
>>>>>>>>> > one
>>>>>>>>> > > lost
>>>>>>>>> > > > >> messages. You need details about where the
>> message
>>>>>> came
>>>>>>>> from in
>>>>>>>>> > order
>>>>>>>>> > > to
>>>>>>>>> > > > >> pinpoint problems when they happen. Source
>> producer
>>>>>>>> info, where
>>>>>>>>> > it was
>>>>>>>>> > > > >> produced into your infrastructure, and when it
>> was
>>>>>>>> produced.
>>>>>>>>> > This
>>>>>>>>> > > > requires
>>>>>>>>> > > > >> you to add the information to the message.
>>>>>>>>> > > > >>
>>>>>>>>> > > > >> And yes, you still need to maintain your clients.
>>>> So
>>>>>>>> maybe my
>>>>>>>>> > original
>>>>>>>>> > > > >> example was not the best. My thoughts on not
>>>> wanting
>>>>>> to
>>>>>>>> be
>>>>>>>>> > responsible
>>>>>>>>> > > > for
>>>>>>>>> > > > >> message formats stands, because that’s very much
>>>>>>>> separate from
>>>>>>>>> > the
>>>>>>>>> > > > client.
>>>>>>>>> > > > >> As you know, we have our own internal client
>>>> library
>>>>>>>> that can
>>>>>>>>> > insert
>>>>>>>>> > > the
>>>>>>>>> > > > >> right headers, and right now inserts the right
>>>> audit
>>>>>>>>> > information into
>>>>>>>>> > > > the
>>>>>>>>> > > > >> message fields. If they exist, and assuming the
>>>>>> message
>>>>>>>> is Avro
>>>>>>>>> > > encoded.
>>>>>>>>> > > > >> What if someone wants to use JSON instead for a
>>>> good
>>>>>>>> reason?
>>>>>>>>> > What if
>>>>>>>>> > > > user X
>>>>>>>>> > > > >> wants to encrypt messages, but user Y does not?
>>>>>>>> Maintaining the
>>>>>>>>> > client
>>>>>>>>> > > > >> library is still much easier than maintaining the
>>>>>>>> message
>>>>>>>>> > formats.
>>>>>>>>> > > > >>
>>>>>>>>> > > > >> -Todd
>>>>>>>>> > > > >>
>>>>>>>>> > > > >>
>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
>>>>>>>> gwen@confluent.io
>>>>>>>>> > >
>>>>>>>>> > > wrote:
>>>>>>>>> > > > >>
>>>>>>>>> > > > >> > Based on your last sentence, consider me
>>>> convinced
>>>>>> :)
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >> > I get why headers are critical for Mirroring
>> (you
>>>>>>>> need tags to
>>>>>>>>> > > prevent
>>>>>>>>> > > > >> > loops and sometimes to route messages to the
>>>> correct
>>>>>>>>> > destination).
>>>>>>>>> > > > >> > But why do you need headers to audit? We are
>>>>>> auditing
>>>>>>>> by
>>>>>>>>> > producing
>>>>>>>>> > > > >> > counts to a side topic (and I was under the
>>>>>>>> impression you do
>>>>>>>>> > the
>>>>>>>>> > > > >> > same), so we never need to modify the message.
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >> > Another thing - after we added headers,
>> wouldn't
>>>> you
>>>>>>>> be in the
>>>>>>>>> > > > >> > business of making sure everyone uses them
>>>> properly?
>>>>>>>> Making
>>>>>>>>> > sure
>>>>>>>>> > > > >> > everyone includes the right headers you need,
>> not
>>>>>>>> using the
>>>>>>>>> > header
>>>>>>>>> > > > >> > names you intend to use, etc. I don't think the
>>>>>>>> "policing"
>>>>>>>>> > business
>>>>>>>>> > > > >> > will ever go away.
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <
>>>>>>>>> > tpalino@gmail.com>
>>>>>>>>> > > > wrote:
>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
>> with
>>>> the
>>>>>>>>> > workaround.
>>>>>>>>> > > Avro
>>>>>>>>> > > > >> means
>>>>>>>>> > > > >> > > that I have to be concerned with the format
>> of
>>>> the
>>>>>>>> messages
>>>>>>>>> > in
>>>>>>>>> > > > order to
>>>>>>>>> > > > >> > run
>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring, etc.).
>>>> That
>>>>>>>> means
>>>>>>>>> > that I
>>>>>>>>> > > have
>>>>>>>>> > > > to
>>>>>>>>> > > > >> > > handle the schemas, and I have to enforce
>> rules
>>>>>>>> about good
>>>>>>>>> > > formats.
>>>>>>>>> > > > >> This
>>>>>>>>> > > > >> > is
>>>>>>>>> > > > >> > > not something I want to be in the business
>> of,
>>>>>>>> because I
>>>>>>>>> > should be
>>>>>>>>> > > > able
>>>>>>>>> > > > >> > to
>>>>>>>>> > > > >> > > run a service infrastructure without needing
>>>> to be
>>>>>>>> in the
>>>>>>>>> > weeds of
>>>>>>>>> > > > >> > dealing
>>>>>>>>> > > > >> > > with customer data formats.
>>>>>>>>> > > > >> > >
>>>>>>>>> > > > >> > > Trust me, a sizable portion of my support
>> time
>>>> is
>>>>>>>> spent
>>>>>>>>> > dealing
>>>>>>>>> > > with
>>>>>>>>> > > > >> > schema
>>>>>>>>> > > > >> > > issues. I really would like to get away from
>>>> that.
>>>>>>>> Maybe
>>>>>>>>> > I'd have
>>>>>>>>> > > > more
>>>>>>>>> > > > >> > time
>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
>>>>>>>>> > > > >> > >
>>>>>>>>> > > > >> > > -Todd
>>>>>>>>> > > > >> > >
>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <
>>>>>>>>> > gwen@confluent.io>
>>>>>>>>> > > > wrote:
>>>>>>>>> > > > >> > >
>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
>>>> workarounds
>>>>>>>> (Avro
>>>>>>>>> > container
>>>>>>>>> > > > >> > >> format), so I'm not too excited about the
>>>> extra
>>>>>>>> work
>>>>>>>>> > required to
>>>>>>>>> > > do
>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't mind it
>>>> if
>>>>>>>> you do
>>>>>>>>> > it...
>>>>>>>>> > > > >> > >> I think the Apache convention for "good
>> idea,
>>>> but
>>>>>>>> not
>>>>>>>>> > willing to
>>>>>>>>> > > > put
>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway, that's
>>>> what
>>>>>> I
>>>>>>>> was
>>>>>>>>> > trying to
>>>>>>>>> > > > >> > >> convey :)
>>>>>>>>> > > > >> > >>
>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino
>> <
>>>>>>>>> > tpalino@gmail.com>
>>>>>>>>> > > > >> wrote:
>>>>>>>>> > > > >> > >> > Well I guess my question for you, then, is
>>>> what
>>>>>>>> is
>>>>>>>>> > holding you
>>>>>>>>> > > > back
>>>>>>>>> > > > >> > from
>>>>>>>>> > > > >> > >> > full support for headers? What’s the bit
>>>> that
>>>>>>>> you’re
>>>>>>>>> > missing
>>>>>>>>> > > that
>>>>>>>>> > > > >> has
>>>>>>>>> > > > >> > you
>>>>>>>>> > > > >> > >> > under a full +1?
>>>>>>>>> > > > >> > >> >
>>>>>>>>> > > > >> > >> > -Todd
>>>>>>>>> > > > >> > >> >
>>>>>>>>> > > > >> > >> >
>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
>>>> Shapira <
>>>>>>>>> > > gwen@confluent.io>
>>>>>>>>> > > > >> > wrote:
>>>>>>>>> > > > >> > >> >
>>>>>>>>> > > > >> > >> >> I know why people who support headers
>>>> support
>>>>>>>> them, and
>>>>>>>>> > I've
>>>>>>>>> > > > seen
>>>>>>>>> > > > >> > what
>>>>>>>>> > > > >> > >> >> the discussion is like.
>>>>>>>>> > > > >> > >> >>
>>>>>>>>> > > > >> > >> >> This is why I'm asking people who are
>>>> against
>>>>>>>> headers
>>>>>>>>> > > > (especially
>>>>>>>>> > > > >> > >> >> committers) what will make them change
>>>> their
>>>>>>>> mind - so
>>>>>>>>> > we can
>>>>>>>>> > > > get
>>>>>>>>> > > > >> > this
>>>>>>>>> > > > >> > >> >> part over one way or another.
>>>>>>>>> > > > >> > >> >>
>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at Radai,
>>>> Jun
>>>>>>>> or you
>>>>>>>>> > (Todd)...
>>>>>>>>> > > > I am
>>>>>>>>> > > > >> > >> >> just looking for something concrete we
>> can
>>>> do
>>>>>>>> to move
>>>>>>>>> > the
>>>>>>>>> > > > >> discussion
>>>>>>>>> > > > >> > >> >> along to the yummy design details (which
>> is
>>>>>> the
>>>>>>>>> > argument I
>>>>>>>>> > > > really
>>>>>>>>> > > > >> am
>>>>>>>>> > > > >> > >> >> looking forward to).
>>>>>>>>> > > > >> > >> >>
>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
>>>> Palino <
>>>>>>>>> > > tpalino@gmail.com>
>>>>>>>>> > > > >> > wrote:
>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even though
>>>> I’m
>>>>>>>> not a
>>>>>>>>> > > > committer)...
>>>>>>>>> > > > >> > >> >> >
>>>>>>>>> > > > >> > >> >> > I have always been a strong supporter
>> of
>>>>>>>> introducing
>>>>>>>>> > the
>>>>>>>>> > > > concept
>>>>>>>>> > > > >> > of an
>>>>>>>>> > > > >> > >> >> > envelope to messages, which headers
>>>>>>>> accomplishes. The
>>>>>>>>> > > message
>>>>>>>>> > > > key
>>>>>>>>> > > > >> > is
>>>>>>>>> > > > >> > >> >> > already an example of a piece of
>> envelope
>>>>>>>>> > information. By
>>>>>>>>> > > > >> > providing a
>>>>>>>>> > > > >> > >> >> means
>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself, and not
>>>>>>>> relying on
>>>>>>>>> > use-case
>>>>>>>>> > > > >> > specific
>>>>>>>>> > > > >> > >> >> > implementations, you make it much
>> easier
>>>> for
>>>>>>>>> > components to
>>>>>>>>> > > > >> > >> interoperate.
>>>>>>>>> > > > >> > >> >> It
>>>>>>>>> > > > >> > >> >> > simplifies development of all these
>>>> things
>>>>>>>> (message
>>>>>>>>> > routing,
>>>>>>>>> > > > >> > auditing,
>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each one does
>>>> not
>>>>>>>> have to
>>>>>>>>> > reinvent
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> wheel.
>>>>>>>>> > > > >> > >> >> >
>>>>>>>>> > > > >> > >> >> > It also makes it much easier from a
>>>> client
>>>>>>>> point of
>>>>>>>>> > view if
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> headers
>>>>>>>>> > > > >> > >> >> are
>>>>>>>>> > > > >> > >> >> > defined as part of the protocol and/or
>>>>>>>> message format
>>>>>>>>> > in
>>>>>>>>> > > > general
>>>>>>>>> > > > >> > >> because
>>>>>>>>> > > > >> > >> >> > you can easily produce and consume
>>>> messages
>>>>>>>> without
>>>>>>>>> > having
>>>>>>>>> > > to
>>>>>>>>> > > > >> take
>>>>>>>>> > > > >> > >> into
>>>>>>>>> > > > >> > >> >> > account specific cases. For example, I
>>>> want
>>>>>>>> to route
>>>>>>>>> > > messages,
>>>>>>>>> > > > >> but
>>>>>>>>> > > > >> > >> >> client A
>>>>>>>>> > > > >> > >> >> > doesn’t support the way audit
>> implemented
>>>>>>>> headers, and
>>>>>>>>> > > client
>>>>>>>>> > > > B
>>>>>>>>> > > > >> > >> doesn’t
>>>>>>>>> > > > >> > >> >> > support the way encryption or routing
>>>>>>>> implemented
>>>>>>>>> > headers,
>>>>>>>>> > > so
>>>>>>>>> > > > now
>>>>>>>>> > > > >> > my
>>>>>>>>> > > > >> > >> >> > application has to create some really
>>>>>> fragile
>>>>>>>> (my
>>>>>>>>> > > autocorrect
>>>>>>>>> > > > >> just
>>>>>>>>> > > > >> > >> tried
>>>>>>>>> > > > >> > >> >> to
>>>>>>>>> > > > >> > >> >> > make that “tragic”, which is probably
>>>>>>>> appropriate
>>>>>>>>> > too) code
>>>>>>>>> > > to
>>>>>>>>> > > > >> > strip
>>>>>>>>> > > > >> > >> >> > everything off, rather than just
>>>> consuming
>>>>>> the
>>>>>>>>> > messages,
>>>>>>>>> > > > picking
>>>>>>>>> > > > >> > out
>>>>>>>>> > > > >> > >> the
>>>>>>>>> > > > >> > >> >> 1
>>>>>>>>> > > > >> > >> >> > or 2 headers it’s interested in, and
>>>>>>>> performing its
>>>>>>>>> > > function.
>>>>>>>>> > > > >> > >> >> >
>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has been
>> going
>>>> on
>>>>>>>> for a
>>>>>>>>> > long time,
>>>>>>>>> > > > and
>>>>>>>>> > > > >> > it’s
>>>>>>>>> > > > >> > >> >> > always “Oh, you came up with 2 use
>> cases,
>>>>>> and
>>>>>>>> yeah,
>>>>>>>>> > those
>>>>>>>>> > > use
>>>>>>>>> > > > >> cases
>>>>>>>>> > > > >> > >> are
>>>>>>>>> > > > >> > >> >> > real things that someone would want to
>>>> do.
>>>>>>>> Here’s an
>>>>>>>>> > > alternate
>>>>>>>>> > > > >> way
>>>>>>>>> > > > >> > to
>>>>>>>>> > > > >> > >> >> > implement them so let’s not do
>> headers.”
>>>> If
>>>>>>>> we have a
>>>>>>>>> > few
>>>>>>>>> > > use
>>>>>>>>> > > > >> cases
>>>>>>>>> > > > >> > >> that
>>>>>>>>> > > > >> > >> >> we
>>>>>>>>> > > > >> > >> >> > actually came up with, you can be sure
>>>> that
>>>>>>>> over the
>>>>>>>>> > next
>>>>>>>>> > > year
>>>>>>>>> > > > >> > >> there’s a
>>>>>>>>> > > > >> > >> >> > dozen others that we didn’t think of
>> that
>>>>>>>> someone
>>>>>>>>> > would like
>>>>>>>>> > > > to
>>>>>>>>> > > > >> > do. I
>>>>>>>>> > > > >> > >> >> > really think it’s time to stop
>> rehashing
>>>>>> this
>>>>>>>>> > discussion and
>>>>>>>>> > > > >> > instead
>>>>>>>>> > > > >> > >> >> focus
>>>>>>>>> > > > >> > >> >> > on a workable standard that we can
>> adopt.
>>>>>>>>> > > > >> > >> >> >
>>>>>>>>> > > > >> > >> >> > -Todd
>>>>>>>>> > > > >> > >> >> >
>>>>>>>>> > > > >> > >> >> >
>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
>>>> Palino
>>>>>> <
>>>>>>>>> > > > tpalino@gmail.com>
>>>>>>>>> > > > >> > >> wrote:
>>>>>>>>> > > > >> > >> >> >
>>>>>>>>> > > > >> > >> >> >> C. per message encryption
>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach is that
>>>> this
>>>>>>>>> > significantly
>>>>>>>>> > > > reduce
>>>>>>>>> > > > >> > the
>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression, which
>>>>>> happens
>>>>>>>> on a
>>>>>>>>> > set of
>>>>>>>>> > > > >> > serialized
>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to enable
>>>> SSL
>>>>>>>> for wire
>>>>>>>>> > > > encryption
>>>>>>>>> > > > >> and
>>>>>>>>> > > > >> > >> rely
>>>>>>>>> > > > >> > >> >> on
>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS) for at
>>>> rest
>>>>>>>>> > encryption.
>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
>> this
>>>>>> does
>>>>>>>> cover
>>>>>>>>> > the case
>>>>>>>>> > > > of
>>>>>>>>> > > > >> > >> removing
>>>>>>>>> > > > >> > >> >> a
>>>>>>>>> > > > >> > >> >> >> drive from the system, it will not
>>>> satisfy
>>>>>>>> most
>>>>>>>>> > compliance
>>>>>>>>> > > > >> > >> requirements
>>>>>>>>> > > > >> > >> >> for
>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever has
>>>> access to
>>>>>>>> the
>>>>>>>>> > broker
>>>>>>>>> > > itself
>>>>>>>>> > > > >> > still
>>>>>>>>> > > > >> > >> has
>>>>>>>>> > > > >> > >> >> >> access to the unencrypted data. For
>>>>>>>> end-to-end
>>>>>>>>> > encryption
>>>>>>>>> > > you
>>>>>>>>> > > > >> > need to
>>>>>>>>> > > > >> > >> >> >> encrypt at the producer, before it
>>>> enters
>>>>>> the
>>>>>>>>> > system, and
>>>>>>>>> > > > >> decrypt
>>>>>>>>> > > > >> > at
>>>>>>>>> > > > >> > >> the
>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the system.
>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>> > > > >> > >> >> >> -Todd
>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai
>> <
>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
>>>>>>>>> > > > >> > >
>>>>>>>>> > > > >> > >> >> wrote:
>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>> > > > >> > >> >> >>> another big plus of headers in the
>>>>>> protocol
>>>>>>>> is that
>>>>>>>>> > it
>>>>>>>>> > > would
>>>>>>>>> > > > >> > enable
>>>>>>>>> > > > >> > >> >> rapid
>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of core
>>>> kafka
>>>>>>>> and would
>>>>>>>>> > reduce
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> number of
>>>>>>>>> > > > >> > >> >> >>> future wire format changes required.
>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
>>>>>> represents
>>>>>>>> use
>>>>>>>>> > cases that
>>>>>>>>> > > > are
>>>>>>>>> > > > >> > not
>>>>>>>>> > > > >> > >> 100%
>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and some of
>> them
>>>>>>>> require
>>>>>>>>> > rather
>>>>>>>>> > > > invasive
>>>>>>>>> > > > >> > wire
>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a good
>> recent
>>>>>>>> example of
>>>>>>>>> > this is
>>>>>>>>> > > > >> > kip-98.
>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected to
>> be
>>>> a
>>>>>>>> very small
>>>>>>>>> > > > fraction of
>>>>>>>>> > > > >> > >> total
>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes are
>>>> invasive.
>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>> > > > >> > >> >> >>> every such wire format change
>>>> translates
>>>>>>>> into
>>>>>>>>> > painful and
>>>>>>>>> > > > slow
>>>>>>>>> > > > >> > >> >> adoption of
>>>>>>>>> > > > >> > >> >> >>> new versions.
>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
>>>> currently
>>>>>> in
>>>>>>>> KIPs
>>>>>>>>> > could be
>>>>>>>>> > > > "spun
>>>>>>>>> > > > >> > out"
>>>>>>>>> > > > >> > >> >> and
>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
>>>> transmitting
>>>>>>>> data over
>>>>>>>>> > > > headers.
>>>>>>>>> > > > >> > this
>>>>>>>>> > > > >> > >> >> would
>>>>>>>>> > > > >> > >> >> >>> keep the core wire format stable(r),
>>>> core
>>>>>>>> codebase
>>>>>>>>> > > smaller,
>>>>>>>>> > > > and
>>>>>>>>> > > > >> > >> avoid
>>>>>>>>> > > > >> > >> >> the
>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
>>>> required
>>>>>>>> to prove
>>>>>>>>> > a
>>>>>>>>> > > > certain
>>>>>>>>> > > > >> > >> feature
>>>>>>>>> > > > >> > >> >> is
>>>>>>>>> > > > >> > >> >> >>> useful enough for a wide-enough
>>>> audience
>>>>>> to
>>>>>>>> warrant
>>>>>>>>> > a wire
>>>>>>>>> > > > >> format
>>>>>>>>> > > > >> > >> >> change
>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
>>>> "mere"
>>>>>>>> wire format
>>>>>>>>> > > changes
>>>>>>>>> > > > >> and
>>>>>>>>> > > > >> > im
>>>>>>>>> > > > >> > >> not
>>>>>>>>> > > > >> > >> >> >>> saying it could have been completely
>>>> done
>>>>>>>> with
>>>>>>>>> > headers,
>>>>>>>>> > > but
>>>>>>>>> > > > >> > >> >> exactly-once
>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen
>>>>>>>> Shapira <
>>>>>>>>> > > > >> gwen@confluent.io
>>>>>>>>> > > > >> > >
>>>>>>>>> > > > >> > >> >> wrote:
>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
>>>> radai <
>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>>>>>>>>> > > > >> > >> >> >>> wrote:
>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
>>>> organization,
>>>>>>>> one could
>>>>>>>>> > > always
>>>>>>>>> > > > use
>>>>>>>>> > > > >> > >> other
>>>>>>>>> > > > >> > >> >> >>> > > approaches such as company-wise
>>>>>>>> containers"
>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
>>>>>> traditionally
>>>>>>>> done
>>>>>>>>> > but there
>>>>>>>>> > > > are
>>>>>>>>> > > > >> > now
>>>>>>>>> > > > >> > >> >> cases
>>>>>>>>> > > > >> > >> >> >>> > (read
>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is not
>>>>>> acceptable.
>>>>>>>> this
>>>>>>>>> > makes
>>>>>>>>> > > > headers
>>>>>>>>> > > > >> > >> useful
>>>>>>>>> > > > >> > >> >> even
>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for cases
>> where
>>>>>>>>> > > > one-container-fits-all
>>>>>>>>> > > > >> > cannot
>>>>>>>>> > > > >> > >> >> >>> apply.
>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use cases
>>>>>> listed,
>>>>>>>> i dont
>>>>>>>>> > want
>>>>>>>>> > > > this to
>>>>>>>>> > > > >> > >> devolve
>>>>>>>>> > > > >> > >> >> >>> to a
>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular use
>> cases
>>>> - i
>>>>>>>> think its
>>>>>>>>> > > enough
>>>>>>>>> > > > >> that
>>>>>>>>> > > > >> > >> some
>>>>>>>>> > > > >> > >> >> of
>>>>>>>>> > > > >> > >> >> >>> them
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > I think a main point of contention
>> is
>>>>>>>> that: We
>>>>>>>>> > > identified
>>>>>>>>> > > > few
>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are useful,
>>>> do
>>>>>> we
>>>>>>>> want
>>>>>>>>> > Kafka to
>>>>>>>>> > > > be a
>>>>>>>>> > > > >> > >> system
>>>>>>>>> > > > >> > >> >> >>> > that supports those use-cases?
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
>>>> record-level
>>>>>>>> lineage
>>>>>>>>> > is
>>>>>>>>> > > though
>>>>>>>>> > > > >> > since
>>>>>>>>> > > > >> > >> the
>>>>>>>>> > > > >> > >> >> >>> > overhead could
>>>>>>>>> > > > >> > >> >> >>> > be significant."
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record level
>>>>>>>> lineage. I
>>>>>>>>> > don't
>>>>>>>>> > > think
>>>>>>>>> > > > it
>>>>>>>>> > > > >> > was
>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think it is
>>>> safe
>>>>>> to
>>>>>>>> assume
>>>>>>>>> > that
>>>>>>>>> > > the
>>>>>>>>> > > > NSA
>>>>>>>>> > > > >> > >> needed
>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also know
>> that
>>>>>>>> certain
>>>>>>>>> > financial
>>>>>>>>> > > > >> > institutes
>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
>> records
>>>> at
>>>>>> a
>>>>>>>> record
>>>>>>>>> > level
>>>>>>>>> > > and
>>>>>>>>> > > > >> > there
>>>>>>>>> > > > >> > >> are
>>>>>>>>> > > > >> > >> >> >>> > federal regulations that absolutely
>>>>>>>> require
>>>>>>>>> > this. They
>>>>>>>>> > > > also
>>>>>>>>> > > > >> > need
>>>>>>>>> > > > >> > >> to
>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
>>>> "touches"
>>>>>> the
>>>>>>>>> > messages and
>>>>>>>>> > > > >> either
>>>>>>>>> > > > >> > >> reads
>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't have
>>>>>> possibly
>>>>>>>>> > modified the
>>>>>>>>> > > > >> payload
>>>>>>>>> > > > >> > >> >> itself.
>>>>>>>>> > > > >> > >> >> >>> > They use record level encryption to
>>>> do
>>>>>>>> that -
>>>>>>>>> > apps can
>>>>>>>>> > > > read
>>>>>>>>> > > > >> and
>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
>> can't
>>>>>>>> touch the
>>>>>>>>> > payload.
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those are
>> corner
>>>>>>>> cases and
>>>>>>>>> > not worth
>>>>>>>>> > > > >> adding
>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they should
>>>> use a
>>>>>>>> different
>>>>>>>>> > > pubsub
>>>>>>>>> > > > >> > message
>>>>>>>>> > > > >> > >> for
>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the other 1000
>>>> that
>>>>>>>> cater
>>>>>>>>> > > > specifically
>>>>>>>>> > > > >> to
>>>>>>>>> > > > >> > the
>>>>>>>>> > > > >> > >> >> >>> > financial industry).
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a catch 22:
>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific use-case,
>>>>>>>> someone can
>>>>>>>>> > always
>>>>>>>>> > > say
>>>>>>>>> > > > it
>>>>>>>>> > > > >> > isn't
>>>>>>>>> > > > >> > >> >> >>> > interesting enough for Kafka. If we
>>>>>>>> discuss more
>>>>>>>>> > general
>>>>>>>>> > > > >> > trends,
>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we are not
>> sure
>>>>>> any
>>>>>>>> of them
>>>>>>>>> > really
>>>>>>>>> > > > >> needs
>>>>>>>>> > > > >> > >> >> headers
>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just hand
>>>> waving
>>>>>>>> and not
>>>>>>>>> > > > interesting.".
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > I think discussing use-cases in
>>>>>> specifics
>>>>>>>> is super
>>>>>>>>> > > > important
>>>>>>>>> > > > >> to
>>>>>>>>> > > > >> > >> >> decide
>>>>>>>>> > > > >> > >> >> >>> > implementation details for headers
>>>> (my
>>>>>>>> use-cases
>>>>>>>>> > lean
>>>>>>>>> > > > toward
>>>>>>>>> > > > >> > >> >> numerical
>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and object
>>>> values,
>>>>>>>> others
>>>>>>>>> > differ),
>>>>>>>>> > > > but I
>>>>>>>>> > > > >> > >> think
>>>>>>>>> > > > >> > >> >> we
>>>>>>>>> > > > >> > >> >> >>> > need to answer the general "Are we
>>>> going
>>>>>>>> to have
>>>>>>>>> > > headers"
>>>>>>>>> > > > >> > question
>>>>>>>>> > > > >> > >> >> >>> > first.
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the other
>>>>>>>> committers in the
>>>>>>>>> > > > discussion:
>>>>>>>>> > > > >> > >> >> >>> > What would it take to convince you
>>>> that
>>>>>>>> headers
>>>>>>>>> > in Kafka
>>>>>>>>> > > > are
>>>>>>>>> > > > >> a
>>>>>>>>> > > > >> > >> good
>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can move
>> ahead
>>>>>> and
>>>>>>>> try to
>>>>>>>>> > agree
>>>>>>>>> > > on
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> >> details?
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving the goal
>>>>>> posts
>>>>>>>> and
>>>>>>>>> > this is
>>>>>>>>> > > > truly
>>>>>>>>> > > > >> > >> >> exhausting.
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly support
>>>> adding
>>>>>>>> headers
>>>>>>>>> > to Kafka
>>>>>>>>> > > > >> > (+0.5?).
>>>>>>>>> > > > >> > >> >> >>> > The community can continue to find
>>>>>>>> workarounds to
>>>>>>>>> > the
>>>>>>>>> > > > issue
>>>>>>>>> > > > >> and
>>>>>>>>> > > > >> > >> there
>>>>>>>>> > > > >> > >> >> >>> > are some benefits to keeping the
>>>> message
>>>>>>>> format
>>>>>>>>> > and
>>>>>>>>> > > > clients
>>>>>>>>> > > > >> > >> simpler.
>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of headers
>>>> to
>>>>>>>> many
>>>>>>>>> > use-cases
>>>>>>>>> > > and
>>>>>>>>> > > > if
>>>>>>>>> > > > >> we
>>>>>>>>> > > > >> > >> can
>>>>>>>>> > > > >> > >> >> >>> > find a good and generally useful
>> way
>>>> to
>>>>>>>> add it to
>>>>>>>>> > Kafka,
>>>>>>>>> > > > it
>>>>>>>>> > > > >> > will
>>>>>>>>> > > > >> > >> make
>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
>> worthy
>>>>>>>> goal in my
>>>>>>>>> > eyes.
>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use cases
>>>> for
>>>>>>>> polyglot
>>>>>>>>> > topics.
>>>>>>>>> > > > >> > >> especially if
>>>>>>>>> > > > >> > >> >> >>> kafka
>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
>> something
>>>>>> else.
>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would make it
>>>>>> harder
>>>>>>>> to write
>>>>>>>>> > > > portable
>>>>>>>>> > > > >> > >> consumer
>>>>>>>>> > > > >> > >> >> >>> code.
>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would mess
>> with
>>>>>>>> locality of
>>>>>>>>> > > > consumption
>>>>>>>>> > > > >> > >> >> >>> guarantees.
>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
>>>>>>>> lineage/metadata is
>>>>>>>>> > > > >> > >> billing/chargeback.
>>>>>>>>> > > > >> > >> >> for
>>>>>>>>> > > > >> > >> >> >>> > that
>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough to
>> simply
>>>>>>>> record the
>>>>>>>>> > point
>>>>>>>>> > > of
>>>>>>>>> > > > >> > origin,
>>>>>>>>> > > > >> > >> but
>>>>>>>>> > > > >> > >> >> >>> every
>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think mirror
>>>> maker)
>>>>>>>> must also
>>>>>>>>> > add a
>>>>>>>>> > > > >> record
>>>>>>>>> > > > >> > to
>>>>>>>>> > > > >> > >> >> form a
>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing on top
>> of
>>>>>>>> kafka - i
>>>>>>>>> > know
>>>>>>>>> > > samza
>>>>>>>>> > > > >> has a
>>>>>>>>> > > > >> > >> >> metadata
>>>>>>>>> > > > >> > >> >> >>> > map
>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
>>>> addition to
>>>>>>>> user
>>>>>>>>> > values.
>>>>>>>>> > > > headers
>>>>>>>>> > > > >> > are
>>>>>>>>> > > > >> > >> the
>>>>>>>>> > > > >> > >> >> >>> > perfect
>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM,
>>>> Jun
>>>>>>>> Rao <
>>>>>>>>> > > > jun@confluent.io
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >> > >> wrote:
>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the first two
>>>>>>>> questions, it
>>>>>>>>> > would
>>>>>>>>> > > be
>>>>>>>>> > > > >> > helpful
>>>>>>>>> > > > >> > >> >> if we
>>>>>>>>> > > > >> > >> >> >>> > could
>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use cases
>>>> for
>>>>>>>> headers
>>>>>>>>> > in the
>>>>>>>>> > > > space
>>>>>>>>> > > > >> > for
>>>>>>>>> > > > >> > >> >> >>> > third-party
>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases within an
>>>>>>>> organization,
>>>>>>>>> > one
>>>>>>>>> > > > could
>>>>>>>>> > > > >> > always
>>>>>>>>> > > > >> > >> >> use
>>>>>>>>> > > > >> > >> >> >>> > other
>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as company-wise
>>>>>>>> containers to
>>>>>>>>> > get
>>>>>>>>> > > > around
>>>>>>>>> > > > >> w/o
>>>>>>>>> > > > >> > >> >> >>> headers. I
>>>>>>>>> > > > >> > >> >> >>> > >> went through the use cases in
>> the
>>>> KIP
>>>>>>>> and in
>>>>>>>>> > Radai's
>>>>>>>>> > > > wiki
>>>>>>>>> > > > >> (
>>>>>>>>> > > > >> > >> >> >>> > >> https://cwiki.apache.org/confl
>>>>>>>>> > uence/display/KAFKA/A+
>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>>>>>>>>> > > > >> > >> >> >>> > >> ).
>>>>>>>>> > > > >> > >> >> >>> > >> The following are the ones that
>>>> that
>>>>>> I
>>>>>>>>> > understand and
>>>>>>>>> > > > >> could
>>>>>>>>> > > > >> > be
>>>>>>>>> > > > >> > >> in
>>>>>>>>> > > > >> > >> >> the
>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case category.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
>>>>>> content-type
>>>>>>>> should
>>>>>>>>> > be set
>>>>>>>>> > > at
>>>>>>>>> > > > >> the
>>>>>>>>> > > > >> > >> topic
>>>>>>>>> > > > >> > >> >> >>> level.
>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages with
>>>>>>>> different
>>>>>>>>> > content
>>>>>>>>> > > > types
>>>>>>>>> > > > >> > >> should be
>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
>> useless
>>>>>>>> without
>>>>>>>>> > schema id,
>>>>>>>>> > > it
>>>>>>>>> > > > >> > seems
>>>>>>>>> > > > >> > >> that
>>>>>>>>> > > > >> > >> >> >>> > storing
>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together with
>>>>>> serialized
>>>>>>>> bytes
>>>>>>>>> > in the
>>>>>>>>> > > > value
>>>>>>>>> > > > >> is
>>>>>>>>> > > > >> > >> >> better?
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this approach is
>>>> that
>>>>>>>> this
>>>>>>>>> > > > significantly
>>>>>>>>> > > > >> > reduce
>>>>>>>>> > > > >> > >> >> the
>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
>>>> which
>>>>>>>> happens on
>>>>>>>>> > a set
>>>>>>>>> > > of
>>>>>>>>> > > > >> > >> serialized
>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative is to
>>>> enable
>>>>>>>> SSL for
>>>>>>>>> > wire
>>>>>>>>> > > > >> > encryption
>>>>>>>>> > > > >> > >> and
>>>>>>>>> > > > >> > >> >> >>> rely
>>>>>>>>> > > > >> > >> >> >>> > on
>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g. LUKS)
>>>> for at
>>>>>>>> rest
>>>>>>>>> > > encryption.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
>> across
>>>>>>>> Kafka
>>>>>>>>> > clusters
>>>>>>>>> > > > >> > >> >> >>> > >> This is actually interesting.
>>>> Today,
>>>>>>>> to avoid
>>>>>>>>> > > > introducing
>>>>>>>>> > > > >> > >> cycles
>>>>>>>>> > > > >> > >> >> when
>>>>>>>>> > > > >> > >> >> >>> > doing
>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data centers,
>> one
>>>>>>>> would
>>>>>>>>> > either have
>>>>>>>>> > > to
>>>>>>>>> > > > >> set
>>>>>>>>> > > > >> > up
>>>>>>>>> > > > >> > >> two
>>>>>>>>> > > > >> > >> >> >>> Kafka
>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
>>>> aggregate)
>>>>>>>> per data
>>>>>>>>> > center
>>>>>>>>> > > or
>>>>>>>>> > > > >> > rename
>>>>>>>>> > > > >> > >> >> topics.
>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With headers,
>>>> the
>>>>>>>> producer
>>>>>>>>> > could
>>>>>>>>> > > tag
>>>>>>>>> > > > >> each
>>>>>>>>> > > > >> > >> >> message
>>>>>>>>> > > > >> > >> >> >>> with
>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID in the
>>>>>> header.
>>>>>>>>> > MirrorMaker
>>>>>>>>> > > > could
>>>>>>>>> > > > >> > then
>>>>>>>>> > > > >> > >> >> avoid
>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a cluster
>> if
>>>>>>>> they are
>>>>>>>>> > tagged
>>>>>>>>> > > with
>>>>>>>>> > > > >> the
>>>>>>>>> > > > >> > >> same
>>>>>>>>> > > > >> > >> >> >>> cluster
>>>>>>>>> > > > >> > >> >> >>> > >> id.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative approach
>>>> is
>>>>>> to
>>>>>>>>> > introduce sth
>>>>>>>>> > > > like
>>>>>>>>> > > > >> > >> >> >>> hierarchical
>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages from
>>>>>> different
>>>>>>>>> > clusters in
>>>>>>>>> > > > >> > different
>>>>>>>>> > > > >> > >> >> >>> partitions
>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic. This
>>>> approach
>>>>>>>> avoids
>>>>>>>>> > filtering
>>>>>>>>> > > > out
>>>>>>>>> > > > >> > >> unneeded
>>>>>>>>> > > > >> > >> >> >>> data
>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving easier
>> to
>>>>>>>> support. It
>>>>>>>>> > may
>>>>>>>>> > > make
>>>>>>>>> > > > >> > >> compaction
>>>>>>>>> > > > >> > >> >> >>> > trickier
>>>>>>>>> > > > >> > >> >> >>> > >> though since the same key may
>>>> show up
>>>>>>>> in
>>>>>>>>> > different
>>>>>>>>> > > > >> > partitions.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source connector
>>>> could
>>>>>>>> store in
>>>>>>>>> > the
>>>>>>>>> > > > message
>>>>>>>>> > > > >> > the
>>>>>>>>> > > > >> > >> >> >>> metadata
>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
>> record.
>>>>>>>> Similarly,
>>>>>>>>> > if a
>>>>>>>>> > > > stream
>>>>>>>>> > > > >> job
>>>>>>>>> > > > >> > >> >> >>> transforms
>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to topic
>> B,
>>>> the
>>>>>>>> library
>>>>>>>>> > could
>>>>>>>>> > > > >> include
>>>>>>>>> > > > >> > the
>>>>>>>>> > > > >> > >> >> >>> source
>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each of the
>>>>>>>> transformed
>>>>>>>>> > message in
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> header.
>>>>>>>>> > > > >> > >> >> Not
>>>>>>>>> > > > >> > >> >> >>> > sure
>>>>>>>
>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful record-level
>>>>>> lineage
>>>>>>>> is
>>>>>>>>> > though
>>>>>>>>> > > since
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> >> overhead
>>>>>>>>> > > > >> > >> >> >>> > could
>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
>>>>>>>> clientId/host/user in
>>>>>>>>> > the
>>>>>>>>> > > > header
>>>>>>>>> > > > >> in
>>>>>>>>> > > > >> > >> each
>>>>>>>>> > > > >> > >> >> >>> > message
>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These metadata are
>>>>>>>> really at the
>>>>>>>>> > > producer
>>>>>>>>> > > > >> > level
>>>>>>>>> > > > >> > >> >> though.
>>>>>>>>> > > > >> > >> >> >>> > So, a
>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to only
>>>>>> include a
>>>>>>>>> > "producerId"
>>>>>>>>> > > > per
>>>>>>>>> > > > >> > >> message
>>>>>>>>> > > > >> > >> >> and
>>>>>>>>> > > > >> > >> >> >>> > send
>>>>>>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
>> mapping
>>>>>>>>> > independently.
>>>>>>>>> > > > KIP-98
>>>>>>>>> > > > >> is
>>>>>>>>> > > > >> > >> >> actually
>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such a
>>>> producerId
>>>>>>>> natively
>>>>>>>>> > in the
>>>>>>>>> > > > >> > message.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that I
>> am
>>>>>> fully
>>>>>>>>> > convinced of
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > strong
>>>>>>>>> > > > >> > >> >> >>> > third-party
>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
>> Perhaps
>>>> we
>>>>>>>> could
>>>>>>>>> > discuss a
>>>>>>>>> > > > bit
>>>>>>>>> > > > >> > more
>>>>>>>>> > > > >> > >> to
>>>>>>>>> > > > >> > >> >> make
>>>>>>>>> > > > >> > >> >> >>> > one
>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing use
>>>> cases.
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal question is
>>>>>>>> whether header
>>>>>>>>> > should
>>>>>>>>> > > > be
>>>>>>>>> > > > >> > >> exposed
>>>>>>>>> > > > >> > >> >> in
>>>>>>>>> > > > >> > >> >> >>> > stream
>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
>>>> stream,
>>>>>>>> Samza,
>>>>>>>>> > and
>>>>>>>>> > > Spark
>>>>>>>>> > > > >> > >> streaming.
>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems just
>> deal
>>>>>> with
>>>>>>>>> > key/value
>>>>>>>>> > > > pairs.
>>>>>>>>> > > > >> > >> Should we
>>>>>>>>> > > > >> > >> >> >>> > expose a
>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there too or
>>>>>>>> somehow map
>>>>>>>>> > header to
>>>>>>>>> > > > key
>>>>>>>>> > > > >> or
>>>>>>>>> > > > >> > >> >> value?
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> Jun
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM,
>>>>>>>> Michael
>>>>>>>>> > Pearce <
>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a period
>>>> of a
>>>>>>>> week,
>>>>>>>>> > that there
>>>>>>>>> > > > is
>>>>>>>>> > > > >> no
>>>>>>>>> > > > >> > >> >> concerns
>>>>>>>>> > > > >> > >> >> >>> now
>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and now
>> we
>>>>>> have
>>>>>>>>> > agreement that
>>>>>>>>> > > > >> > headers
>>>>>>>>> > > > >> > >> are
>>>>>>>>> > > > >> > >> >> >>> useful
>>>>>>>>> > > > >> > >> >> >>> > >> and
>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
>> put
>>>> to
>>>>>> a
>>>>>>>> KIP
>>>>>>>>> > vote, this
>>>>>>>>> > > > >> > wouldn’t
>>>>>>>>> > > > >> > >> be
>>>>>>>>> > > > >> > >> >> a
>>>>>>>>> > > > >> > >> >> >>> > reason
>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > @
>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of getting
>>>> this
>>>>>>>> KIP
>>>>>>>>> > moving past
>>>>>>>>> > > > >> this,
>>>>>>>>> > > > >> > we
>>>>>>>>> > > > >> > >> can
>>>>>>>>> > > > >> > >> >> >>> state
>>>>>>>>> > > > >> > >> >> >>> > >> the
>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
>> that
>>>>>> can
>>>>>>>> will be
>>>>>>>>> > > > naturally
>>>>>>>>> > > > >> > >> >> interpreted
>>>>>>>>> > > > >> > >> >> >>> as
>>>>>>>>> > > > >> > >> >> >>> > an
>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is later
>>>>>>>> wanted you can
>>>>>>>>> > > easily
>>>>>>>>> > > > >> split
>>>>>>>>> > > > >> > >> this
>>>>>>>>> > > > >> > >> >> >>> into
>>>>>>>>> > > > >> > >> >> >>> > two
>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the wire
>>>>>> protocol
>>>>>>>>> > > implementation
>>>>>>>>> > > > >> this
>>>>>>>>> > > > >> > >> makes
>>>>>>>>> > > > >> > >> >> no
>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don’t believe. Is
>>>> this
>>>>>>>>> > reasonable to
>>>>>>>>> > > > all?
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4 therefor
>>>> happy
>>>>>>>> we keep
>>>>>>>>> > with 32
>>>>>>>>> > > > >> bits.
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
>>>>>>>>> > ignacio.solis@gmail.com on
>>>>>>>>> > > > >> behalf
>>>>>>>>> > > > >> > of
>>>>>>>>> > > > >> > >> >> >>> Ignacio
>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
>> ignacio.solis@gmail.com
>>>> on
>>>>>>>> behalf of
>>>>>>>>> > > > >> > isolis@igso.net
>>>>>>>>> > > > >> > >> >
>>>>>>>>> > > > >> > >> >> >>> wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value as
>>>> byte[]
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
>> maybe
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
>>>> system
>>>>>>>> should
>>>>>>>>> > take a
>>>>>>>>> > > > single
>>>>>>>>> > > > >> > >> int. I
>>>>>>>>> > > > >> > >> >> >>> think
>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you want
>> to
>>>>>>>> interpret
>>>>>>>>> > this as
>>>>>>>>> > > to
>>>>>>>>> > > > >> 16bit
>>>>>>>>> > > > >> > >> >> numbers
>>>>>>>>> > > > >> > >> >> >>> in
>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > layer
>>>>>>>>> > > > >> > >> >> >>> > >> > above go right ahead. If
>>>>>>>> somebody wants
>>>>>>>>> > to
>>>>>>>>> > > argue
>>>>>>>>> > > > >> for
>>>>>>>>> > > > >> > 16
>>>>>>>>> > > > >> > >> >> bits
>>>>>>>>> > > > >> > >> >> >>> or
>>>>>>>>> > > > >> > >> >> >>> > 64
>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I would
>>>>>> listen.
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key space
>> into
>>>>>>>> sub_key_1 and
>>>>>>>>> > > > sub_key_2
>>>>>>>>> > > > >> > >> makes no
>>>>>>>>> > > > >> > >> >> >>> > sense to
>>>>>>>>> > > > >> > >> >> >>> > >> > me at
>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we going
>> to
>>>>>>>> start
>>>>>>>>> > providing
>>>>>>>>> > > > APIs to
>>>>>>>>> > > > >> > get
>>>>>>>>> > > > >> > >> all
>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
>>>>>>>> sub_key_2s? If
>>>>>>>>> > there is
>>>>>>>>> > > > no
>>>>>>>>> > > > >> > >> >> >>> distinguishing
>>>>>>>>> > > > >> > >> >> >>> > >> > functions
>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to each
>> one
>>>>>>>> then they
>>>>>>>>> > should
>>>>>>>>> > > be
>>>>>>>>> > > > a
>>>>>>>>> > > > >> > single
>>>>>>>>> > > > >> > >> >> >>> value.
>>>>>>>>> > > > >> > >> >> >>> > At
>>>>>>>>> > > > >> > >> >> >>> > >> > this
>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're doing is
>>>>>>>> equality.
>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer wants
>> to
>>>>>>>> interpret
>>>>>>>>> > this as
>>>>>>>>> > > 2,
>>>>>>>>> > > > 3
>>>>>>>>> > > > >> or
>>>>>>>>> > > > >> > >> more
>>>>>>>>> > > > >> > >> >> >>> values
>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
>>>>>>>>> > > > >> > >> >> >>> > >> > different question. I
>>>>>>>> personally think
>>>>>>>>> > it's
>>>>>>>>> > > all
>>>>>>>>> > > > one
>>>>>>>>> > > > >> > >> >> keyspace
>>>>>>>>> > > > >> > >> >> >>> > that is
>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned using
>> some
>>>>>>>> structure,
>>>>>>>>> > but if
>>>>>>>>> > > you
>>>>>>>>> > > > >> > want to
>>>>>>>>> > > > >> > >> >> >>> > sub-assign
>>>>>>>>> > > > >> > >> >> >>> > >> > parts
>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
>> applies
>>>> to
>>>>>>>> strings.
>>>>>>>>> > If
>>>>>>>>> > > > somebody
>>>>>>>>> > > > >> > >> argued
>>>>>>>>> > > > >> > >> >> for
>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
>>>>>>>>> > > > >> > >> >> >>> > >> > would we be arguing to
>>>> divide
>>>>>> the
>>>>>>>>> > strings with
>>>>>>>>> > > > dots
>>>>>>>>> > > > >> > ('.')
>>>>>>>>> > > > >> > >> >> as a
>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them to give
>>>> us
>>>>>> the
>>>>>>>>> > different
>>>>>>>>> > > name
>>>>>>>>> > > > >> > segments
>>>>>>>>> > > > >> > >> >> >>> > separately?
>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be performing any
>>>>>>>> actions on
>>>>>>>>> > this key
>>>>>>>>> > > > other
>>>>>>>>> > > > >> > than
>>>>>>>>> > > > >> > >> >> >>> > matching?
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
>> 9:30
>>>>>> AM,
>>>>>>>> Michael
>>>>>>>>> > > Pearce <
>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any concerns
>> on
>>>> 1
>>>>>>>> and 2
>>>>>>>>> > still?
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this moving
>> along a
>>>>>> bit
>>>>>>>> more
>>>>>>>>> > I'd also
>>>>>>>>> > > > like
>>>>>>>>> > > > >> to
>>>>>>>>> > > > >> > >> ask
>>>>>>>>> > > > >> > >> >> to
>>>>>>>>> > > > >> > >> >> >>> get
>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last points:
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're all
>>>>>> roughly
>>>>>>>> happy
>>>>>>>>> > with the
>>>>>>>>> > > > >> header
>>>>>>>>> > > > >> > >> value
>>>>>>>>> > > > >> > >> >> >>> > being a
>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe consensus
>> has
>>>>>>>> been for an
>>>>>>>>> > > > namespace
>>>>>>>>> > > > >> > based
>>>>>>>>> > > > >> > >> int
>>>>>>>>> > > > >> > >> >> >>> > approach
>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the key.
>> Any
>>>>>>>> objections
>>>>>>>>> > if this
>>>>>>>>> > > > is
>>>>>>>>> > > > >> > what
>>>>>>>>> > > > >> > >> we
>>>>>>>>> > > > >> > >> >> go
>>>>>>>>> > > > >> > >> >> >>> > with?
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
>>>> assumption
>>>>>> in
>>>>>>>> (4) is
>>>>>>>>> > > correct,
>>>>>>>>> > > > >> > >> {int,int}
>>>>>>>>> > > > >> > >> >> >>> keys.
>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both int's be
>>>> int16 or
>>>>>>>> int32?
>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them being
>> int16(2
>>>>>>>> bytes) as
>>>>>>>>> > combined
>>>>>>>>> > > > is
>>>>>>>>> > > > >> > space
>>>>>>>>> > > > >> > >> of
>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
>>>>>>>>> > > > >> > >> >> >>> > >> > per
>>>>>>>>> > > > >> > >> >> >>> > >> > > original and gives
>> plenty
>>>> of
>>>>>>>>> > combinations for
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> >> >>> foreseeable,
>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead small.
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any benefit in
>>>>>>>> another kip
>>>>>>>>> > call to
>>>>>>>>> > > > >> discuss
>>>>>>>>> > > > >> > >> >> these at
>>>>>>>>> > > > >> > >> >> >>> > all?
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>> ______________________________
>>>>>>>>> > __________
>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
>>>>>>>> k.burstev@yandex.com>
>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday, November
>> 18,
>>>>>> 2016
>>>>>>>>> > 7:07:07 AM
>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
>> dev@kafka.apache.org
>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re: [DISCUSS]
>>>> KIP-82
>>>>>>>> - Add
>>>>>>>>> > Record
>>>>>>>>> > > > Headers
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is worth
>> also
>>>> i
>>>>>>>> agree. As
>>>>>>>>> > a user:
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
>>>>>>>> worthwhile
>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers should
>>>> be a
>>>>>>>> top level
>>>>>>>>> > > option
>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
>>>> "Ignacio
>>>>>>>> Solis" <
>>>>>>>>> > > > >> isolis@igso.net
>>>>>>>>> > > > >> > >:
>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers are
>>>>>>>> worthwhile
>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
>> should
>>>> be
>>>>>> a
>>>>>>>> top
>>>>>>>>> > level
>>>>>>>>> > > option
>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14, 2016
>> at
>>>>>> 9:16
>>>>>>>> AM,
>>>>>>>>> > Michael
>>>>>>>>> > > > Pearce
>>>>>>>>> > > > >> <
>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
>>>> details/examples
>>>>>>>> the
>>>>>>>>> > original
>>>>>>>>> > > > proposal
>>>>>>>>> > > > >> > for
>>>>>>>>> > > > >> > >> key
>>>>>>>>> > > > >> > >> >> >>> > spacing
>>>>>>>>> > > > >> > >> >> >>> > >> ,
>>>>>>>>> > > > >> > >> >> >>> > >> > not
>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as per
>>>>>>>> discussion
>>>>>>>>> > namespace
>>>>>>>>> > > > >> idea.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need to
>> update
>>>>>> the
>>>>>>>> kip,
>>>>>>>>> > when we
>>>>>>>>> > > get
>>>>>>>>> > > > >> > >> agreement
>>>>>>>>> > > > >> > >> >> >>> this
>>>>>>>>> > > > >> > >> >> >>> > is a
>>>>>>>>> > > > >> > >> >> >>> > >> > better
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach (which
>> seems
>>>> to
>>>>>>>> be the
>>>>>>>>> > case if I
>>>>>>>>> > > > have
>>>>>>>>> > > > >> > >> >> understood
>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > general
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
>>>>>>>> conversation)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
>> ints,
>>>> at
>>>>>>>> very
>>>>>>>>> > early stage
>>>>>>>>> > > > we
>>>>>>>>> > > > >> did
>>>>>>>>> > > > >> > >> think
>>>>>>>>> > > > >> > >> >> >>> about
>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added complexity
>>>> for
>>>>>>>> the
>>>>>>>>> > saving isn't
>>>>>>>>> > > > >> worth
>>>>>>>>> > > > >> > it.
>>>>>>>>> > > > >> > >> >> I'd
>>>>>>>>> > > > >> > >> >> >>> > rather
>>>>>>>>> > > > >> > >> >> >>> > >> go
>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
>>>>>>>> overheads and
>>>>>>>>> > size
>>>>>>>>> > > int16
>>>>>>>>> > > > >> > (2bytes)
>>>>>>>>> > > > >> > >> >> keys
>>>>>>>>> > > > >> > >> >> >>> as
>>>>>>>>> > > > >> > >> >> >>> > it
>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
>>>>>> headers,
>>>>>>>> there
>>>>>>>>> > is as
>>>>>>>>> > > per
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > kip
>>>>>>>>> > > > >> > >> as
>>>>>>>>> > > > >> > >> >> we
>>>>>>>>> > > > >> > >> >> >>> > use an
>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
>>>> headers
>>>>>>>> are
>>>>>>>>> > present or
>>>>>>>>> > > > not as
>>>>>>>>> > > > >> > such
>>>>>>>>> > > > >> > >> >> >>> > provides a
>>>>>>>>> > > > >> > >> >> >>> > >> > zero
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead currently
>> if
>>>>>>>> headers are
>>>>>>>>> > not
>>>>>>>>> > > used.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
>>>> mentions
>>>>>>>> would be
>>>>>>>>> > good
>>>>>>>>> > > > first
>>>>>>>>> > > > >> > if we
>>>>>>>>> > > > >> > >> >> can
>>>>>>>>> > > > >> > >> >> >>> get
>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
>>>>>>>>> > > > >> > >> >> >>> > >> > > do
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have general
>>>>>>>> consensus that
>>>>>>>>> > (1)
>>>>>>>>> > > > headers
>>>>>>>>> > > > >> > are
>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>>>>>>>> > > > >> > >> >> >>> > >> and
>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want it
>> as
>>>> a
>>>>>>>> top level
>>>>>>>>> > entity.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
>>>>>> obvious i
>>>>>>>>> > believe (1)
>>>>>>>>> > > > >> headers
>>>>>>>>> > > > >> > are
>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a top level
>>>>>>>> entity.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>> ______________________________
>>>>>>>>> > __________
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger Hoover <
>>>>>>>>> > > roger.hoover@gmail.com
>>>>>>>>> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
>>>> November
>>>>>>>> 9, 2016
>>>>>>>>> > 9:10:47
>>>>>>>>> > > > PM
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
>>>> dev@kafka.apache.org
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
>> [DISCUSS]
>>>>>>>> KIP-82 - Add
>>>>>>>>> > > Record
>>>>>>>>> > > > >> > Headers
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
>>>> little
>>>>>>>> in the
>>>>>>>>> > weeds but
>>>>>>>>> > > > >> thanks
>>>>>>>>> > > > >> > >> for
>>>>>>>>> > > > >> > >> >> the
>>>>>>>>> > > > >> > >> >> >>> > >> replies
>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a prefix
>>>> and
>>>>>>>> {int,
>>>>>>>>> > int} can
>>>>>>>>> > > be
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> same.
>>>>>>>>> > > > >> > >> >> It
>>>>>>>>> > > > >> > >> >> >>> > doesn't
>>>>>>>>> > > > >> > >> >> >>> > >> > look
>>>>>>>>> > > > >> > >> >> >>> > >> > > like
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what the KIP
>> is
>>>>>>>> saying the
>>>>>>>>> > "Open"
>>>>>>>>> > > > >> > section.
>>>>>>>>> > > > >> > >> The
>>>>>>>>> > > > >> > >> >> >>> > example
>>>>>>>>> > > > >> > >> >> >>> > >> > shows
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
>>>> 210002
>>>>>>>> for App
>>>>>>>>> > Dynamics
>>>>>>>>> > > > >> > implying
>>>>>>>>> > > > >> > >> >> that
>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>> > > > >> > >> >> >>> > New
>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization will
>> have
>>>>>>>> only a
>>>>>>>>> > single
>>>>>>>>> > > > header id
>>>>>>>>> > > > >> > to
>>>>>>>>> > > > >> > >> work
>>>>>>>>> > > > >> > >> >> >>> > with. Or
>>>>>>>>> > > > >> > >> >> >>> > >> > is
>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The main
>>>> point
>>>>>>>> of a
>>>>>>>>> > namespace
>>>>>>>>> > > or
>>>>>>>>> > > > >> > prefix
>>>>>>>>> > > > >> > >> is
>>>>>>>>> > > > >> > >> >> to
>>>>>>>>> > > > >> > >> >> >>> > reduce
>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of config
>>>>>> mapping
>>>>>>>> or
>>>>>>>>> > > registration
>>>>>>>>> > > > >> > >> depending
>>>>>>>>> > > > >> > >> >> on
>>>>>>>>> > > > >> > >> >> >>> how
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> namespaces/prefixes
>>>> are
>>>>>>>> managed.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to hear
>>>> more
>>>>>>>> feedback
>>>>>>>>> > on the
>>>>>>>>> > > > >> > >> higher-level
>>>>>>>>> > > > >> > >> >> >>> > questions
>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9, 2016
>> at
>>>>>>>> 11:38 AM,
>>>>>>>>> > radai <
>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think this
>>>> discussion
>>>>>>>> is
>>>>>>>>> > getting a
>>>>>>>>> > > bit
>>>>>>>>> > > > >> into
>>>>>>>>> > > > >> > the
>>>>>>>>> > > > >> > >> >> >>> weeds on
>>>>>>>>> > > > >> > >> >> >>> > >> > technical
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > implementation
>>>> details.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to step
>>>> back a
>>>>>>>> minute
>>>>>>>>> > and try
>>>>>>>>> > > > and
>>>>>>>>> > > > >> > >> establish
>>>>>>>>> > > > >> > >> >> >>> > where we
>>>>>>>>> > > > >> > >> >> >>> > >> > are in
>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
>> nacho's
>>>>>> last
>>>>>>>>> > paragraph)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all in
>>>>>>>> agreement that
>>>>>>>>> > headers
>>>>>>>>> > > > are
>>>>>>>>> > > > >> a
>>>>>>>>> > > > >> > >> >> >>> worthwhile
>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > useful
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to have?
>>>> this
>>>>>>>> was
>>>>>>>>> > contested
>>>>>>>>> > > > early
>>>>>>>>> > > > >> on
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all in
>>>>>>>> agreement on
>>>>>>>>> > headers
>>>>>>>>> > > as
>>>>>>>>> > > > top
>>>>>>>>> > > > >> > >> level
>>>>>>>>> > > > >> > >> >> >>> entity
>>>>>>>>> > > > >> > >> >> >>> > vs
>>>>>>>>> > > > >> > >> >> >>> > >> > headers
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > squirreled-away in
>>>> V?
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there are still
>>>>>>>> concerns
>>>>>>>>> > around
>>>>>>>>> > > these
>>>>>>>>> > > > #2
>>>>>>>>> > > > >> > >> points
>>>>>>>>> > > > >> > >> >> >>> (#jay?
>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now back to
>> our
>>>>>>>> normal
>>>>>>>>> > programming
>>>>>>>>> > > > ...)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are nice.
>>>>>> having
>>>>>>>> said
>>>>>>>>> > that, its
>>>>>>>>> > > > >> adding
>>>>>>>>> > > > >> > >> >> >>> complexity
>>>>>>>>> > > > >> > >> >> >>> > >> (see
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>> https://github.com/addthis/
>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>> main/java/com/clearspring/
>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
>>>> result)
>>>>>>>> and would
>>>>>>>>> > require
>>>>>>>>> > > > >> anyone
>>>>>>>>> > > > >> > >> >> writing
>>>>>>>>> > > > >> > >> >> >>> > other
>>>>>>>>> > > > >> > >> >> >>> > >> > clients
>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go? Bash?
>>>> ;-) )
>>>>>>>> to
>>>>>>>>> > > get/implement
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > >> same,
>>>>>>>>> > > > >> > >> >> and
>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain (int
>> vs
>>>>>>>> string is
>>>>>>>>> > order of
>>>>>>>>> > > > >> > magnitude,
>>>>>>>>> > > > >> > >> >> this
>>>>>>>>> > > > >> > >> >> >>> > isnt).
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int namespacing vs
>>>>>> {int,
>>>>>>>> int}
>>>>>>>>> > > namespacing
>>>>>>>>> > > > >> are
>>>>>>>>> > > > >> > >> >> basically
>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > same
>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
>>>> namespacing
>>>>>>>> an int64
>>>>>>>>> > and
>>>>>>>>> > > > giving
>>>>>>>>> > > > >> > people
>>>>>>>>> > > > >> > >> >> while
>>>>>>>>> > > > >> > >> >> >>> > 2^32
>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the part i
>>>> like
>>>>>>>> about this
>>>>>>>>> > is
>>>>>>>>> > > > letting
>>>>>>>>> > > > >> > >> people
>>>>>>>>> > > > >> > >> >> >>> have a
>>>>>>>>> > > > >> > >> >> >>> > >> large
>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers with one
>>>>>>>> registration so
>>>>>>>>> > they
>>>>>>>>> > > > dont
>>>>>>>>> > > > >> > have
>>>>>>>>> > > > >> > >> to
>>>>>>>>> > > > >> > >> >> come
>>>>>>>>> > > > >> > >> >> >>> > back
>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>>>>>>>> > > > >> > >> >> >>> > >> > > every
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
>> plugin/header
>>>>>>>> they want to
>>>>>>>>> > > > "reserve".
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
>> 2016
>>>> at
>>>>>>>> 11:01 AM,
>>>>>>>>> > Roger
>>>>>>>>> > > > >> Hoover
>>>>>>>>> > > > >> > <
>>>>>>>>> > > > >> > >> >> >>> > >> > > roger.hoover@gmail.com>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
>> the
>>>>>>>> debate has
>>>>>>>>> > been
>>>>>>>>> > > about
>>>>>>>>> > > > >> > >> overhead +
>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if we
>>>> have
>>>>>>>>> > considered a
>>>>>>>>> > > > varint
>>>>>>>>> > > > >> > >> encoding
>>>>>>>>> > > > >> > >> >> (
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>> https://developers.google.com/
>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the header
>> length
>>>>>>>> field (int32
>>>>>>>>> > in the
>>>>>>>>> > > > >> > proposal)
>>>>>>>>> > > > >> > >> >> and
>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>> > > > >> > >> >> >>> > >> > header
>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
>> headers,
>>>>>> the
>>>>>>>>> > overhead would
>>>>>>>>> > > > be a
>>>>>>>>> > > > >> > >> single
>>>>>>>>> > > > >> > >> >> >>> byte
>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > for each
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128 would
>>>> also
>>>>>>>> need only a
>>>>>>>>> > > single
>>>>>>>>> > > > >> byte?
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov 9,
>>>> 2016
>>>>>> at
>>>>>>>> 6:43 AM,
>>>>>>>>> > > radai <
>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus - and
>>>> very
>>>>>>>> dangerous
>>>>>>>>> > (youre
>>>>>>>>> > > > >> > >> essentially
>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary code
>>>> off
>>>>>>>> the
>>>>>>>>> > internet on
>>>>>>>>> > > > your
>>>>>>>>> > > > >> > >> servers
>>>>>>>>> > > > >> > >> >> ...
>>>>>>>>> > > > >> > >> >> >>> > bad
>>>>>>>>> > > > >> > >> >> >>> > >> > idea
>>>>>>>>> > > > >> > >> >> >>> > >> > > without
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox, even
>>>> with)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
>> being
>>>> a
>>>>>>>> purely
>>>>>>>>> > > > administrative
>>>>>>>>> > > > >> > task
>>>>>>>>> > > > >> > >> - i
>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
>> would,
>>>>>>>> really,
>>>>>>>>> > because
>>>>>>>>> > > > then my
>>>>>>>>> > > > >> > >> earlier
>>>>>>>>> > > > >> > >> >> >>> > point on
>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the remapping
>>>>>>>> process would
>>>>>>>>> > be
>>>>>>>>> > > > invalid,
>>>>>>>>> > > > >> > but
>>>>>>>>> > > > >> > >> at
>>>>>>>>> > > > >> > >> >> >>> > linkedin,
>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team im
>> in)
>>>>>> run
>>>>>>>> kafka
>>>>>>>>> > as a
>>>>>>>>> > > > service.
>>>>>>>>> > > > >> > we
>>>>>>>>> > > > >> > >> dont
>>>>>>>>> > > > >> > >> >> >>> > really
>>>>>>>>> > > > >> > >> >> >>> > >> > know
>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
>>>>>>>> applications
>>>>>>>>> > that use
>>>>>>>>> > > > kafka)
>>>>>>>>> > > > >> > are
>>>>>>>>> > > > >> > >> up
>>>>>>>>> > > > >> > >> >> to
>>>>>>>>> > > > >> > >> >> >>> at
>>>>>>>>> > > > >> > >> >> >>> > any
>>>>>>>>> > > > >> > >> >> >>> > >> > given
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
>> possible
>>>>>>>> (given the
>>>>>>>>> > > > existance of
>>>>>>>>> > > > >> > >> headers
>>>>>>>>> > > > >> > >> >> >>> and a
>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem) for
>>>> some
>>>>>>>>> > application to
>>>>>>>>> > > > >> "equip"
>>>>>>>>> > > > >> > >> their
>>>>>>>>> > > > >> > >> >> >>> > >> producers
>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
>>>> required
>>>>>>>> plugin
>>>>>>>>> > without us
>>>>>>>>> > > > >> > knowing.
>>>>>>>>> > > > >> > >> i
>>>>>>>>> > > > >> > >> >> dont
>>>>>>>>> > > > >> > >> >> >>> > mean
>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
>>>> want to
>>>>>>>> make the
>>>>>>>>> > point
>>>>>>>>> > > > that
>>>>>>>>> > > > >> > its
>>>>>>>>> > > > >> > >> not
>>>>>>>>> > > > >> > >> >> as
>>>>>>>>> > > > >> > >> >> >>> > simple
>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync across a
>>>>>>>> large-enough
>>>>>>>>> > > > organization.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed, Nov 9,
>>>> 2016
>>>>>>>> at 6:17
>>>>>>>>> > AM,
>>>>>>>>> > > > Magnus
>>>>>>>>> > > > >> > >> Edenhill
>>>>>>>>> > > > >> > >> >> <
>>>>>>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
>> there
>>>> is
>>>>>> a
>>>>>>>> piece
>>>>>>>>> > missing
>>>>>>>>> > > in
>>>>>>>>> > > > >> the
>>>>>>>>> > > > >> > >> >> Strings
>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
>>>>>>>>> > > > >> > >> >> >>> > >> > > where
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> pro-Stringers
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason that
>> by
>>>>>>>> providing
>>>>>>>>> > unique
>>>>>>>>> > > > string
>>>>>>>>> > > > >> > >> >> >>> identifiers
>>>>>>>>> > > > >> > >> >> >>> > for
>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > everything
>>>> will
>>>>>>>> just
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
>> work
>>>>>> for
>>>>>>>> all
>>>>>>>>> > parts of
>>>>>>>>> > > the
>>>>>>>>> > > > >> > stream
>>>>>>>>> > > > >> > >> >> >>> pipeline.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
>>>> strings
>>>>>>>> dont mean
>>>>>>>>> > > anything
>>>>>>>>> > > > by
>>>>>>>>> > > > >> > >> >> themselves,
>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > while we
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
>>>> envision
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
>>>> plugin
>>>>>>>> loader
>>>>>>>>> > that
>>>>>>>>> > > > >> downloads,
>>>>>>>>> > > > >> > >> >> compiles,
>>>>>>>>> > > > >> > >> >> >>> > links
>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon as
>>>>>> they're
>>>>>>>> seen by
>>>>>>>>> > a
>>>>>>>>> > > > >> consumer, I
>>>>>>>>> > > > >> > >> dont
>>>>>>>>> > > > >> > >> >> >>> really
>>>>>>>>> > > > >> > >> >> >>> > >> see
>>>>>>>>> > > > >> > >> >> >>> > >> > a
>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so dynamic
>>>> (and
>>>>>>>> fragile) in
>>>>>>>>> > > > practice.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the real
>>>> world
>>>>>>>> an
>>>>>>>>> > application
>>>>>>>>> > > > will
>>>>>>>>> > > > >> be
>>>>>>>>> > > > >> > >> >> >>> configured
>>>>>>>>> > > > >> > >> >> >>> > >> with
>>>>>>>>> > > > >> > >> >> >>> > >> > a set
>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
>> add
>>>>>>>> (producer)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
>>>>>> (consumer)
>>>>>>>> headers.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is an
>>>>>>>> administrative
>>>>>>>>> > task
>>>>>>>>> > > > based
>>>>>>>>> > > > >> on
>>>>>>>>> > > > >> > >> what
>>>>>>>>> > > > >> > >> >> >>> > features a
>>>>>>>>> > > > >> > >> >> >>> > >> > client
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> needs/provides
>>>>>> and
>>>>>>>> results
>>>>>>>>> > in
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some sort of
>>>>>>>> configuration
>>>>>>>>> > to
>>>>>>>>> > > > enable
>>>>>>>>> > > > >> and
>>>>>>>>> > > > >> > >> >> >>> configure
>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > desired
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since this
>>>> needs
>>>>>>>> to be kept
>>>>>>>>> > > > somewhat
>>>>>>>>> > > > >> in
>>>>>>>>> > > > >> > >> sync
>>>>>>>>> > > > >> > >> >> >>> across
>>>>>>>>> > > > >> > >> >> >>> > an
>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no point
>> in
>>>>>>>> having
>>>>>>>>> > producers
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add headers
>> no
>>>>>>>> consumers
>>>>>>>>> > will
>>>>>>>>> > > read,
>>>>>>>>> > > > >> and
>>>>>>>>> > > > >> > >> vice
>>>>>>>>> > > > >> > >> >> >>> versa),
>>>>>>>>> > > > >> > >> >> >>> > >> the
>>>>>>>>> > > > >> > >> >> >>> > >> > added
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of assigning
>>>> an
>>>>>> id
>>>>>>>>> > namespace
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
>>>> plugin
>>>>>> as
>>>>>>>> it is
>>>>>>>>> > being
>>>>>>>>> > > > >> > configured
>>>>>>>>> > > > >> > >> >> should
>>>>>>>>> > > > >> > >> >> >>> be
>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > 2016-11-09
>>>> 13:06
>>>>>>>> GMT+01:00
>>>>>>>>> > > Michael
>>>>>>>>> > > > >> > Pearce <
>>>>>>>>> > > > >> > >> >> >>> > >> > > Michael.Pearce@ig.com>:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
>>>>>>>> following/catching
>>>>>>>>> > up on
>>>>>>>>> > > > what
>>>>>>>>> > > > >> > seems
>>>>>>>>> > > > >> > >> to
>>>>>>>>> > > > >> > >> >> be
>>>>>>>>> > > > >> > >> >> >>> an
>>>>>>>>> > > > >> > >> >> >>> > >> > active
>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
>>>> sorry if
>>>>>>>> it may
>>>>>>>>> > seem
>>>>>>>>> > > > obvious
>>>>>>>>> > > > >> > but
>>>>>>>>> > > > >> > >> what
>>>>>>>>> > > > >> > >> >> >>> does
>>>>>>>>> > > > >> > >> >> >>> > MD
>>>>>>>>> > > > >> > >> >> >>> > >> > stand
>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My take on
>>>>>>>> String vs Int:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
>> state
>>>>>>>> first I am
>>>>>>>>> > pro Int
>>>>>>>>> > > > (16
>>>>>>>>> > > > >> or
>>>>>>>>> > > > >> > >> 32).
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
>> though
>>>>>>>> playing
>>>>>>>>> > devils
>>>>>>>>> > > > advocate
>>>>>>>>> > > > >> > see a
>>>>>>>>> > > > >> > >> >> big
>>>>>>>>> > > > >> > >> >> >>> plus
>>>>>>>>> > > > >> > >> >> >>> > >> > with the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
>> keys,
>>>>>>>> this is
>>>>>>>>> > around
>>>>>>>>> > > > >> > integrating
>>>>>>>>> > > > >> > >> >> into an
>>>>>>>>> > > > >> > >> >> >>> > >> > existing
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
>>>> other
>>>>>>>> systems use
>>>>>>>>> > > String
>>>>>>>>> > > > >> based
>>>>>>>>> > > > >> > >> >> headers
>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
>>>>>>>>> > > > >> > >> >> >>> > >> > > it
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
>> easier
>>>> for
>>>>>>>> these to
>>>>>>>>> > be
>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
>>>>>>>>> > > > >> > >> >> >>> > >> > into.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How with
>> Int
>>>>>>>> based
>>>>>>>>> > headers
>>>>>>>>> > > could
>>>>>>>>> > > > we
>>>>>>>>> > > > >> > >> provide
>>>>>>>>> > > > >> > >> >> a
>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> integration
>>>>>>>> simple /
>>>>>>>>> > easy with
>>>>>>>>> > > > >> > transition
>>>>>>>>> > > > >> > >> >> flows
>>>>>>>>> > > > >> > >> >> >>> > over
>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
>> luck
>>>>>>>> buddy
>>>>>>>>> > you're on
>>>>>>>>> > > your
>>>>>>>>> > > > >> own
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * simply
>>>> hash
>>>>>>>> the string
>>>>>>>>> > into
>>>>>>>>> > > int
>>>>>>>>> > > > >> code
>>>>>>>>> > > > >> > >> and
>>>>>>>>> > > > >> > >> >> hope
>>>>>>>>> > > > >> > >> >> >>> > for
>>>>>>>>> > > > >> > >> >> >>> > >> no
>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
>> back
>>>>>>>> though?)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
>>>> style
>>>>>> as
>>>>>>>>> > mentioned by
>>>>>>>>> > > > nacho.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > ______________________________
>>>>>>>>> > > > >> > __________
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
>> radai
>>>> <
>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
>>>>>> Wednesday,
>>>>>>>>> > November 9,
>>>>>>>>> > > 2016
>>>>>>>>> > > > >> > 8:12 AM
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
>>>>>>>> dev@kafka.apache.org
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Subject:
>> Re:
>>>>>>>> [DISCUSS]
>>>>>>>>> > KIP-82 -
>>>>>>>>> > > > Add
>>>>>>>>> > > > >> > >> Record
>>>>>>>>> > > > >> > >> >> >>> Headers
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > thinking
>>>> about
>>>>>>>> it some
>>>>>>>>> > more,
>>>>>>>>> > > the
>>>>>>>>> > > > >> best
>>>>>>>>> > > > >> > >> way to
>>>>>>>>> > > > >> > >> >> >>> > transmit
>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
>>>>>>>> consumers would
>>>>>>>>> > be to
>>>>>>>>> > > > put it
>>>>>>>>> > > > >> > in
>>>>>>>>> > > > >> > >> the
>>>>>>>>> > > > >> > >> >> MD
>>>>>>>>> > > > >> > >> >> >>> > >> response
>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it should
>> be
>>>>>>>> discussed
>>>>>>>>> > now.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
>> Nov
>>>> 9,
>>>>>>>> 2016 at
>>>>>>>>> > 12:09
>>>>>>>>> > > AM,
>>>>>>>>> > > > >> > radai <
>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>> radai.rosenblatt@gmail.com
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im not
>>>>>> opposed
>>>>>>>> to the
>>>>>>>>> > idea of
>>>>>>>>> > > > >> > namespace
>>>>>>>>> > > > >> > >> >> >>> mapping.
>>>>>>>>> > > > >> > >> >> >>> > >> all
>>>>>>>>> > > > >> > >> >> >>> > >> > im
>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not part
>>>> of
>>>>>>>> the "mvp"
>>>>>>>>> > and,
>>>>>>>>> > > > since
>>>>>>>>> > > > >> it
>>>>>>>>> > > > >> > >> >> requires
>>>>>>>>> > > > >> > >> >> >>> no
>>>>>>>>> > > > >> > >> >> >>> > >> wire
>>>>>>>>> > > > >> > >> >> >>> > >> > format
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > always
>> be
>>>>>>>> added later.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
>> its
>>>> not
>>>>>>>> as
>>>>>>>>> > simple as
>>>>>>>>> > > just
>>>>>>>>> > > > >> > >> >> configuring
>>>>>>>>> > > > >> > >> >> >>> MM
>>>>>>>>> > > > >> > >> >> >>> > to
>>>>>>>>> > > > >> > >> >> >>> > >> do
>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say i've
>>>>>>>> implemented
>>>>>>>>> > large
>>>>>>>>> > > > message
>>>>>>>>> > > > >> > >> >> support as
>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
>>>>>>>>> > > > >> > >> >> >>> > >> > > on
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > target
>>>>>> cluster
>>>>>>>> its been
>>>>>>>>> > > > remapped
>>>>>>>>> > > > >> to
>>>>>>>>> > > > >> > >> >> {999,1}.
>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
>> need
>>>> to
>>>>>>>> be told
>>>>>>>>> > to look
>>>>>>>>> > > > for
>>>>>>>>> > > > >> the
>>>>>>>>> > > > >> > >> large
>>>>>>>>> > > > >> > >> >> >>> > message
>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
>>>>>> {999,1}.
>>>>>>>> doable,
>>>>>>>>> > but
>>>>>>>>> > > > tricky.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On Tue,
>>>> Nov
>>>>>> 8,
>>>>>>>> 2016 at
>>>>>>>>> > 10:29
>>>>>>>>> > > > PM,
>>>>>>>>> > > > >> > Gwen
>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> gwen@confluent.io
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> While
>> you
>>>>>> can
>>>>>>>> do
>>>>>>>>> > whatever
>>>>>>>>> > > you
>>>>>>>>> > > > >> want
>>>>>>>>> > > > >> > >> with a
>>>>>>>>> > > > >> > >> >> >>> > >> namespace
>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > > your
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
>> I'd
>>>>>>>> expect is
>>>>>>>>> > for each
>>>>>>>>> > > > app
>>>>>>>>> > > > >> to
>>>>>>>>> > > > >> > >> >> >>> namespaces
>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So if I
>>>>>>>> accidentally
>>>>>>>>> > used
>>>>>>>>> > > 666
>>>>>>>>> > > > for
>>>>>>>>> > > > >> > my
>>>>>>>>> > > > >> > >> HR
>>>>>>>>> > > > >> > >> >> >>> > >> department,
>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>> > > > >> > >> >> >>> > >> > > still
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
>>>>>> RadaiApp,
>>>>>>>> I can
>>>>>>>>> > config
>>>>>>>>> > > > >> > >> "namespace=42"
>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
>> look
>>>>>>>> normal.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
>>>> means
>>>>>>>> you only
>>>>>>>>> > need to
>>>>>>>>> > > > sync
>>>>>>>>> > > > >> > usage
>>>>>>>>> > > > >> > >> >> >>> inside
>>>>>>>>> > > > >> > >> >> >>> > your
>>>>>>>>> > > > >> > >> >> >>> > >> > own
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> Still
>>>> hard,
>>>>>>>> but
>>>>>>>>> > somewhat
>>>>>>>>> > > > easier
>>>>>>>>> > > > >> > than
>>>>>>>>> > > > >> > >> >> syncing
>>>>>>>>> > > > >> > >> >> >>> > with
>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On Tue,
>>>> Nov
>>>>>>>> 8, 2016
>>>>>>>>> > at 10:07
>>>>>>>>> > > > PM,
>>>>>>>>> > > > >> > >> radai <
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > and
>> we
>>>> can
>>>>>>>> start
>>>>>>>>> > with
>>>>>>>>> > > > >> {namespace,
>>>>>>>>> > > > >> > >> id}
>>>>>>>>> > > > >> > >> >> and
>>>>>>>>> > > > >> > >> >> >>> no
>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> always
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > add
>> it
>>>>>>>> later on
>>>>>>>>> > if/when
>>>>>>>>> > > > >> > collisions
>>>>>>>>> > > > >> > >> >> >>> actually
>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
>>>>>> problem).
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > every
>>>>>>>> interested
>>>>>>>>> > party (so
>>>>>>>>> > > > orgs
>>>>>>>>> > > > >> > or
>>>>>>>>> > > > >> > >> >> >>> > individuals)
>>>>>>>>> > > > >> > >> >> >>> > >> > could
>>>>>>>>> > > > >> > >> >> >>> > >> > > then
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > register
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> prefix
>>>> (0
>>>>>> =
>>>>>>>>> > reserved, 1 =
>>>>>>>>> > > > >> > confluent
>>>>>>>>> > > > >> > >> ...
>>>>>>>>> > > > >> > >> >> >>> 666
>>>>>>>>> > > > >> > >> >> >>> > = me
>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
>>>>>>>>> > > > >> > >> >> >>> > >> > > and
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > the
>>>> 2nd ID
>>>>>>>> - so once
>>>>>>>>> > > > linkedin
>>>>>>>>> > > > >> > >> >> registers,
>>>>>>>>> > > > >> > >> >> >>> say
>>>>>>>>> > > > >> > >> >> >>> > 3,
>>>>>>>>> > > > >> > >> >> >>> > >> > then
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
>> use
>>>> {3,
>>>>>>>> *} with a
>>>>>>>>> > > > reasonable
>>>>>>>>> > > > >> > >> >> >>> expectation
>>>>>>>>> > > > >> > >> >> >>> > to
>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > anything
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > else.
>>>>>>>> further
>>>>>>>>> > partitioning
>>>>>>>>> > > > of
>>>>>>>>> > > > >> > that *
>>>>>>>>> > > > >> > >> >> >>> becomes
>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>> "upstream
>>>>>>>>> > registration"
>>>>>>>>> > > of a
>>>>>>>>> > > > >> > >> namespace
>>>>>>>>> > > > >> > >> >> >>> only
>>>>>>>>> > > > >> > >> >> >>> > has
>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
>> Tue,
>>>>>> Nov
>>>>>>>> 8, 2016
>>>>>>>>> > at
>>>>>>>>> > > 9:03
>>>>>>>>> > > > PM,
>>>>>>>>> > > > >> > >> James
>>>>>>>>> > > > >> > >> >> >>> Cheng <
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>> wushujames@gmail.com
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>>>> Nov
>>>>>>>> 8, 2016,
>>>>>>>>> > at 5:54
>>>>>>>>> > > > PM,
>>>>>>>>> > > > >> > Gwen
>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > gwen@confluent.io
>>>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>> Thank
>>>>>>>> you so
>>>>>>>>> > much for
>>>>>>>>> > > > this
>>>>>>>>> > > > >> > clear
>>>>>>>>> > > > >> > >> and
>>>>>>>>> > > > >> > >> >> >>> fair
>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > arguments.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> I'm
>>>> in
>>>>>>>> favor of
>>>>>>>>> > ints.
>>>>>>>>> > > > Not a
>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
>>>>>>>>> > > > >> > >> >> >>> > but
>>>>>>>>> > > > >> > >> >> >>> > >> > in
>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> Even
>>>>>>>> more in
>>>>>>>>> > favor of
>>>>>>>>> > > > >> Magnus's
>>>>>>>>> > > > >> > >> >> >>> > decentralized
>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>> tweak:
>>>>>>>> add a
>>>>>>>>> > namespace
>>>>>>>>> > > > for
>>>>>>>>> > > > >> > >> headers.
>>>>>>>>> > > > >> > >> >> >>> This
>>>>>>>>> > > > >> > >> >> >>> > will
>>>>>>>>> > > > >> > >> >> >>> > >> > allow
>>>>>>>>> > > > >> > >> >> >>> > >> > > each
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> use
>>>>>>>> whatever IDs
>>>>>>>>> > it
>>>>>>>>> > > wants
>>>>>>>>> > > > >> > >> >> internally,
>>>>>>>>> > > > >> > >> >> >>> and
>>>>>>>>> > > > >> > >> >> >>> > >> then
>>>>>>>>> > > > >> > >> >> >>> > >> > let
>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> deploying
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> the
>>>> app
>>>>>>>> figure
>>>>>>>>> > out an
>>>>>>>>> > > > >> > available
>>>>>>>>> > > > >> > >> >> >>> namespace
>>>>>>>>> > > > >> > >> >> >>> > ID
>>>>>>>>> > > > >> > >> >> >>> > >> > for the
>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > So
>>>>>>>>> > > > >> > io.confluent.schema-registry
>>>>>>>>> > > > >> > >> can
>>>>>>>>> > > > >> > >> >> be
>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>> deployment
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> and
>>>>>> 0x57
>>>>>>>> on
>>>>>>>>> > yours, and
>>>>>>>>> > > > the
>>>>>>>>> > > > >> > poor
>>>>>>>>> > > > >> > >> guys
>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>> worry
>>>>>>>> about that.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> Gwen,
>>>> if
>>>>>> I
>>>>>>>>> > understand
>>>>>>>>> > > your
>>>>>>>>> > > > >> > example
>>>>>>>>> > > > >> > >> >> >>> right, an
>>>>>>>>> > > > >> > >> >> >>> > >> > > application
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>> decide to
>>>>>>>> use 0x01
>>>>>>>>> > in one
>>>>>>>>> > > > >> > >> deployment,
>>>>>>>>> > > > >> > >> >> and
>>>>>>>>> > > > >> > >> >> >>> > that
>>>>>>>>> > > > >> > >> >> >>> > >> > means
>>>>>>>>> > > > >> > >> >> >>> > >> > > that
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> message
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> is
>>>>>> written
>>>>>>>> into the
>>>>>>>>> > > > broker, it
>>>>>>>>> > > > >> > >> will be
>>>>>>>>> > > > >> > >> >> >>> > saved on
>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>> specific
>>>>>>>> namespace
>>>>>>>>> > > (0x01).
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> If
>> you
>>>>>>>> were to
>>>>>>>>> > mirror
>>>>>>>>> > > that
>>>>>>>>> > > > >> > message
>>>>>>>>> > > > >> > >> >> into
>>>>>>>>> > > > >> > >> >> >>> > another
>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>> accompany
>>>>>>>> the
>>>>>>>>> > message,
>>>>>>>>> > > > right?
>>>>>>>>> > > > >> > What
>>>>>>>>> > > > >> > >> if
>>>>>>>>> > > > >> > >> >> the
>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> other
>>>>>>>> cluster uses
>>>>>>>>> > 0x57?
>>>>>>>>> > > > They
>>>>>>>>> > > > >> > won't
>>>>>>>>> > > > >> > >> >> >>> > understand
>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> I'm
>>>> not
>>>>>>>> sure
>>>>>>>>> > that's an
>>>>>>>>> > > > >> avoidable
>>>>>>>>> > > > >> > >> >> >>> problem. I
>>>>>>>>> > > > >> > >> >> >>> > >> > think it
>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> order
>>>> to
>>>>>>>> share
>>>>>>>>> > data, you
>>>>>>>>> > > > have
>>>>>>>>> > > > >> to
>>>>>>>>> > > > >> > >> also
>>>>>>>>> > > > >> > >> >> >>> have a
>>>>>>>>> > > > >> > >> >> >>> > >> > shared
>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>> understanding of
>>>>>>>>> > what the
>>>>>>>>> > > > >> > >> namespaces
>>>>>>>>> > > > >> > >> >> >>> mean.
>>>>>>>>> > > > >> > >> >> >>> > >> Which
>>>>>>>>> > > > >> > >> >> >>> > >> > I
>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>> because
>>>>>> the
>>>>>>>>> > alternate
>>>>>>>>> > > > (sharing
>>>>>>>>> > > > >> > >> >> *nothing*
>>>>>>>>> > > > >> > >> >> >>> at
>>>>>>>>> > > > >> > >> >> >>> > >> all)
>>>>>>>>> > > > >> > >> >> >>> > >> > would
>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> would
>>>> be
>>>>>>>> no way to
>>>>>>>>> > > > understand
>>>>>>>>> > > > >> > each
>>>>>>>>> > > > >> > >> >> other.
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> -James
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> Gwen
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>>>> Tue,
>>>>>>>> Nov 8,
>>>>>>>>> > 2016 at
>>>>>>>>> > > > 4:23
>>>>>>>>> > > > >> > PM,
>>>>>>>>> > > > >> > >> >> radai <
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>> radai.rosenblatt@gmail.com
>>>>>>>>> > >
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> wrote:
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>> +1
>>>> for
>>>>>>>> sean's
>>>>>>>>> > > document.
>>>>>>>>> > > > it
>>>>>>>>> > > > >> > >> covers
>>>>>>>>> > > > >> > >> >> >>> pretty
>>>>>>>>> > > > >> > >> >> >>> > >> much
>>>>>>>>> > > > >> > >> >> >>> > >> > all
>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>>>>> provides
>>>>>>>>> > concrete
>>>>>>>>> > > > figures
>>>>>>>>> > > > >> to
>>>>>>>>> > > > >> > >> argue
>>>>>>>>> > > > >> > >> >> >>> about
>>>>>>>>> > > > >> > >> >> >>> > :-)
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>>>>>>> (nit-picking -
>>>>>>>>> > used
>>>>>>>>> > > the
>>>>>>>>> > > > >> same
>>>>>>>>> > > > >> > >> xkcd
>>>>>>>>> > > > >> > >> >> >>> twice,
>>>>>>>>> > > > >> > >> >> >>> > >> also
>>>>>>>>> > > > >> > >> >> >>> > >> > trove
>>>>>>>>> > > > >> > >> >> >>> > >> > > has
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>> superceded
>>>>>>>>> > > > >> > >> >
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >> > --
>>>>>>>>> > > > >> > Gwen Shapira
>>>>>>>>> > > > >> > Product Manager | Confluent
>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>>>>>>>> > > > >> > Follow us: Twitter | blog
>>>>>>>>> > > > >> >
>>>>>>>>> > > > >>
>>>>>>>>> > > > >>
>>>>>>>>> > > > >>
>>>>>>>>> > > > >> --
>>>>>>>>> > > > >> *Todd Palino*
>>>>>>>>> > > > >> Staff Site Reliability Engineer
>>>>>>>>> > > > >> Data Infrastructure Streaming
>>>>>>>>> > > > >>
>>>>>>>>> > > > >>
>>>>>>>>> > > > >>
>>>>>>>>> > > > >> linkedin.com/in/toddpalino
>>>>>>>>> > > > >>
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > >
>>>>>>>>> > > > --
>>>>>>>>> > > > Gwen Shapira
>>>>>>>>> > > > Product Manager | Confluent
>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>>>>>>>> > > > Follow us: Twitter | blog
>>>>>>>>> > > >
>>>>>>>>> > >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > The information contained in this email is strictly
>>>> confidential
>>>>>>>> and for
>>>>>>>>> > the use of the addressee only, unless otherwise indicated.
>> If
>>>> you
>>>>>>>> are not
>>>>>>>>> > the intended recipient, please do not read, copy, use or
>>>> disclose
>>>>>>>> to others
>>>>>>>>> > this message or any attachment. Please also notify the
>> sender
>>>> by
>>>>>>>> replying
>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and then
>>>> delete
>>>>>>>> the email
>>>>>>>>> > and any copies of it. Opinions, conclusion (etc) that do not
>>>>>>>> relate to the
>>>>>>>>> > official business of this company shall be understood as
>>>> neither
>>>>>>>> given nor
>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets Limited
>> (a
>>>>>>>> company
>>>>>>>>> > registered in England and Wales, company number 04008957)
>> and
>>>> IG
>>>>>>>> Index
>>>>>>>>> > Limited (a company registered in England and Wales, company
>>>>>> number
>>>>>>>>> > 01190902). Registered address at Cannon Bridge House, 25
>>>> Dowgate
>>>>>>>> Hill,
>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register number
>>>> 195355)
>>>>>>>> and IG
>>>>>>>>> > Index Limited (register number 114059) are authorised and
>>>>>>>> regulated by the
>>>>>>>>> > Financial Conduct Authority.
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The information contained in this email is strictly confidential
>> and
>>>>>> for
>>>>>>>> the use of the addressee only, unless otherwise indicated. If you
>> are
>>>>>> not
>>>>>>>> the intended recipient, please do not read, copy, use or disclose
>> to
>>>>>> others
>>>>>>>> this message or any attachment. Please also notify the sender by
>>>>>> replying
>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
>> the
>>>>>> email
>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not relate
>>>> to
>>>>>> the
>>>>>>>> official business of this company shall be understood as neither
>>>> given
>>>>>> nor
>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>>>>>>>> registered in England and Wales, company number 04008957) and IG
>>>> Index
>>>>>>>> Limited (a company registered in England and Wales, company number
>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>>>> Hill,
>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
>>>> and IG
>>>>>>>> Index Limited (register number 114059) are authorised and regulated
>>>> by
>>>>>> the
>>>>>>>> Financial Conduct Authority.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi,
I did indeed discuss this offline with Jason. I was toying with going Map.Entry also, and wasn’t fully sold on exposing Header interface outwardly.
e.g. for end users its a lot easier to make a hashmap and pass that in etc, aka its more standard.
Sounds like I wasn’t alone.
Though I also don’t really like having the idea of interfaces extend as Jason mentions <? extend Map.Entry<String, byte[]> and then we make header interface extend Map.Entry<String, byte[]>, this I don’t think achieves much, as then we end up with something like below which is getting plain ugly – I don’t think id want to my name to it ;) .
public interface Header extends Map.Entry<String, byte[]> {
}
public interface Headers extends Iterable<? extends Map.Entry>
{
Headers append(Iterable<? extends Map.Entry> values);
Collection<? extends Map.Entry<String, byte[]> filter(String key);
}
My opinion is we have to choose that we have a custom interface or not for Headers. I don’t like the idea of mixing it, its making things messy.
A) Do we have an extension of classical nice Map interface. (e.g. like Properties does)
B) Or do we go with 100% custom interface (e.g. like JMS does)
For me it seems we are heading on option (B), but I think we need to all be on board. (I actually prefer A btw, but I care more for having headers feature than pushing for A)
I am strongly for though that with going on route (B), making the custom API useable and friendly as JMS / Map interface are, and not forcing users to write lines of boiler code to copy and recreate new records.
I do think we should support for initial release if going with B:
A way to parse over all headers, currently this is handled by Iterable<Header>
A method to add header(s), currently this looks like its append.
A method to retrieve header(s) by key, currently this looks like filter
I think we have these agreed on right?
As noted by Radai we have missed on method we need to ensure we support for usability to avoid having uses make boiler plate code to fully copy and recreate.
A method to remove a header (we need to agree/discuss)
I propose either:
1) remove(Header header)
would remove a single header where the key and value inside are equal
2) remove(String key)
would remove all headers with matching key
My preference is for 1) remove(Header header)
Cheers
Mike
On 23/02/2017, 19:13, "Jason Gustafson" <ja...@confluent.io> wrote:
If we go the mutable route (not totally sold on this yet), letting Header
implement Map.Entry<String, byte[]> may be reasonable. I discussed this
with Michael offline. You can then use the following constructor for
ProducerRecord.
ProducerRecord(K key, V value, Iterable<? extends Map.Entry<String,
byte[]>> headers);
That allows you to pass a Headers instance directly without modification or
to construct them from a map and use map.entries(). I think it would be
wise to use classes like Headers and Header regardless in case we need to
add anything in the future.
-Jason
On Thu, Feb 23, 2017 at 10:48 AM, Sean McCauliff <se...@gmail.com>
wrote:
> Instead of making a Header type visible in the producer or consumer
> record, the API could use Iterator<Map.Entry<String, byte[]>>. This
> would make the API more compatible with the multimap implementations
> I'm aware of (apache commons and google guava) should one want to use
> an existing implementation of a multimap data structure at a later
> date.
>
> Sean
>
> On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com>
> wrote:
> > I think I'd like some opinions from some of the active chaps at LinkedIn
> so it's not just a two way decision (or any others for that matter).
> >
> > Any opinions on the below or no real concerns / indifferent by anyone
> with this change?
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Tuesday, February 21, 2017 6:59:38 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > Jun's suggestion gives us the best of both worlds. Because the headers
> are
> > the last field in the record and you have the total size of the record,
> > then you can easily get the size of headers in bytes. So they can be
> copied
> > without parsing and we can do the lazy initialization without an extra
> cost.
> >
> > The message set notion has changed in KIP-98 so that the message set can
> > have its own fields. See here for more detail:
> > https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF
> 0wSw9ra8/edit#heading=h.pau6n6buvoee.
> > What we're proposing to change here is the schema for the message (aka
> > record) itself. We'll update that document depending on the outcome of
> the
> > discussion here.
> >
> > -Jason
> >
> >
> > On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> >> This works whilst no other feature needs similar. I prefer to keep it
> >> contained/encapsulated it then allows the bytes of the headers to also
> be
> >> copied transferred as one without any parsing what so ever, but if you
> >> really feel strongly on the below ill go with it, just want to flag this
> >> concern.
> >>
> >> Btw, the below notation seems to be different to the notation used in
> >> previous kips like kip-32 which I took the notation from, where the
> length
> >> of the following bytes for byte arrays is clearly defined. So im 100% on
> >> what you propose could you define it in this original styled notation?
> >>
> >>
> >> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
> >>
> >> Hey Jun, that's a good thought. Maybe then the full schema could
> look
> >> like
> >> this:
> >>
> >> Record =>
> >> Length => uintVar
> >> Attributes => int8
> >> TimestampDelta => intVar
> >> OffsetDelta => uintVar
> >> Key => bytes
> >> Value => bytes
> >> Headers => [HeaderKey HeaderValue]
> >> HeaderKey => string
> >> HeaderValue => bytes
> >>
> >> (where the bytes and string type use a variable length size field)
> >>
> >> Does that work for you, Michael?
> >>
> >> -Jason
> >>
> >> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
> >>
> >> > Regarding the overhead of skipping the headers in the message.
> >> Another way
> >> > could be to add the headers at the end of the message after the
> >> value part.
> >> > This way, there is no need to skip the headers unless you really
> >> need it.
> >> > This will allow us to represent the headers explicitly as an
> array of
> >> > key/value pairs without the skipping overhead when headers are not
> >> needed.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <
> jason@confluent.io
> >> >
> >> > wrote:
> >> >
> >> > > Michael,
> >> > >
> >> > > Here's a link to the branch with the message format changes:
> >> > > https://github.com/confluentinc/kafka/tree/
> >> exactly-once-message-format.
> >> > > The
> >> > > class to modify to add headers is EosLogRecord (name will
> >> definitely
> >> > change
> >> > > prior to merging in trunk). This branch uses attributes to
> >> indicate null
> >> > > key and value. I really don't have a strong preference for or
> >> against
> >> > this
> >> > > as long as we're consistent. I do feel it's kind of a pity to
> need
> >> the
> >> > > signed version of the varint just to be able to indicate null.
> For
> >> > headers,
> >> > > it's an easier call because we don't actually need to represent
> >> null
> >> > > headers (using size zero will work fine).
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
> >> Michael.Pearce@ig.com>
> >> > > wrote:
> >> > >
> >> > > > We can make it, implement Iterable<Header>, no concerns on
> this.
> >> > > >
> >> > > > Ive noted this on the KIP.
> >> > > >
> >> > > >
> >> > > > On 20/02/2017, 01:24, "Michael Pearce" <Michael.Pearce@ig.com
> >
> >> wrote:
> >> > > >
> >> > > > Btw Jason, can you share the branch with the VarInts
> >> helper/util
> >> > > class
> >> > > > that KIP-98 will be introducing and also will have the current
> >> modified
> >> > > > (key and value lengths) message, so we can look at, thanks.
> >> > > >
> >> > > > On 20/02/2017, 01:11, "Michael Pearce" <
> >> Michael.Pearce@ig.com>
> >> > > wrote:
> >> > > >
> >> > > > RE “ProducerRecord and ConsumerRecord are immutable
> >> > currently”. I
> >> > > > beg to differ on this, if I set a byte[] as my value of my
> >> record, I
> >> > can
> >> > > > later call value() and then alter the bytes in the array. As
> >> such it is
> >> > > > very much mutable.
> >> > > >
> >> > > > Headers also will anyhow by nature be mutable on
> >> produce, as
> >> > you
> >> > > > add to them.
> >> > > >
> >> > > >
> >> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
> >> jason@confluent.io>
> >> > > > wrote:
> >> > > >
> >> > > > Re; having a dedicated Headers object. I think it
> >> makes
> >> > > sense.
> >> > > > Maybe it
> >> > > > could implement Iterable<Header>?
> >> > > >
> >> > > > One thing I'm not too sure about is mutability.
> Both
> >> the
> >> > > > ProducerRecord and
> >> > > > ConsumerRecord are immutable currently. It would
> be
> >> nice to
> >> > > > keep it that
> >> > > > way. Perhaps Headers could itself be immutable and
> >> passed
> >> > > > through the
> >> > > > constructors?
> >> > > >
> >> > > > Just going through how this is, I don’t believe
> >> this would
> >> > > > provide much
> >> > > > > saving. If issue trying to resolve is around
> >> > object/garbage
> >> > > > creation of
> >> > > > > Map.Entry objects.
> >> > > >
> >> > > >
> >> > > > It's definitely better because we only allocate
> the
> >> space
> >> > we
> >> > > > will use, but
> >> > > > I assume most of the actual memory cost is in the
> >> header
> >> > data
> >> > > > itself which
> >> > > > we can't do much about. Maybe we can start with
> the
> >> simple
> >> > > > array
> >> > > > representation and optimize from there after we
> find
> >> it's
> >> > > > worthwhile to do
> >> > > > so? If we use the Headers object, we can change
> this
> >> later
> >> > > > without breaking
> >> > > > the API.
> >> > > >
> >> > > > -Jason
> >> > > >
> >> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> >> > > > radai.rosenblatt@gmail.com> wrote:
> >> > > >
> >> > > > > "(By the way, doesn't it feel a bit odd that we
> >> seem to
> >> > be
> >> > > > designing a
> >> > > > > feature which is optimized for people not
> using?)"
> >> > > > >
> >> > > > > very. this (i claim :-) ) is the result of very
> >> intense
> >> > > > opposition to the
> >> > > > > usefulness of the feature early on, and not a
> real
> >> design
> >> > > > goal
> >> > > > >
> >> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce
> <
> >> > > > Michael.Pearce@ig.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Whilst if having an array of a Header[] would
> >> mean
> >> > > getting
> >> > > > this through,
> >> > > > > > im happy changing this, as already done.
> >> > > > > >
> >> > > > > > Just going through how this is, I don’t
> believe
> >> this
> >> > > > would provide much
> >> > > > > > saving. If issue trying to resolve is around
> >> > > > object/garbage creation of
> >> > > > > > Map.Entry objects.
> >> > > > > >
> >> > > > > > All you’re doing here is replacing the equiv
> of a
> >> > > > HashMap.Node
> >> > > > > (Map.Entry)
> >> > > > > > which simply holds reference to key and value
> >> objects
> >> > > with
> >> > > > a custom
> >> > > > > variant.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
> >> > > > Michael.Pearce@ig.com> wrote:
> >> > > > > >
> >> > > > > > On point 1 & 2 Ive updated KIP to show
> >> varints (and
> >> > > > removed the bit
> >> > > > > > flag). (on the assumption KIP 98 is getting
> the
> >> > agreement
> >> > > > the protocol is
> >> > > > > > moving from int32 to varInts as standard)
> >> > > > > >
> >> > > > > > On point 3 ive updated to use an array of
> >> Header
> >> > > > class, instead of a
> >> > > > > > MultiMap in the Headers class object
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
> >> > > > Michael.Pearce@ig.com>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > On points 1 and 2 I agree.
> >> > > > > >
> >> > > > > > This also affects kip-98, I should
> >> expect this
> >> > > > resolved before
> >> > > > > > that vote also passes. If it is accepted there
> >> (I’m
> >> > > > assuming this is
> >> > > > > > getting discussed on that KIP? As you’re
> driving
> >> the
> >> > move
> >> > > > to VarInts), I
> >> > > > > am
> >> > > > > > happy to make this KIP will simply follow
> suit to
> >> > > whatever
> >> > > > is agreed in
> >> > > > > > KIP-98.
> >> > > > > >
> >> > > > > > On 3) Agreed this is a simpler form,
> as
> >> long as
> >> > > no
> >> > > > one is
> >> > > > > > expecting hashmap lookup performance ( O(1)
> ), I
> >> still
> >> > > > would prefer an
> >> > > > > > encapsulated class, so if we find that holding
> >> it as an
> >> > > > array in future
> >> > > > > is
> >> > > > > > causing some perf issues, the internals are
> not
> >> exposed
> >> > > to
> >> > > > end users,
> >> > > > > > allowing the internal structure to move to a
> map.
> >> > > > > >
> >> > > > > > Can we compromise on?
> >> > > > > >
> >> > > > > > class ConsumerRecord<K, V> {
> >> > > > > > K key;
> >> > > > > > V value;
> >> > > > > > Headers headers;
> >> > > > > > }
> >> > > > > >
> >> > > > > > class Headers {
> >> > > > > > Header[] headers;
> >> > > > > >
> >> > > > > > add(String key, byte[] value)
> >> > > > > > Collection<byte[]> get(String
> key)
> >> > > > > > }
> >> > > > > >
> >> > > > > > class Header {
> >> > > > > > String key;
> >> > > > > > byte[] value;
> >> > > > > > }
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 19/02/2017, 18:54, "Jason
> Gustafson" <
> >> > > > jason@confluent.io>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > > headers dont "leak" into
> >> application
> >> > code.
> >> > > > they are useful
> >> > > > > > to application
> >> > > > > > > code as well.
> >> > > > > >
> >> > > > > >
> >> > > > > > This is exactly what I have been
> >> trying to
> >> > > get
> >> > > > at. The use
> >> > > > > > cases documented
> >> > > > > > here are middleware:
> >> > > > > > https://cwiki.apache.org/
> >> > > > confluence/display/KAFKA/A+
> >> > > > > > Case+for+Kafka+Headers.
> >> > > > > > If headers are intended for the
> >> application
> >> > > > space as well,
> >> > > > > the
> >> > > > > > document
> >> > > > > > should be updated accordingly so
> >> that it is
> >> > > an
> >> > > > explicit
> >> > > > > design
> >> > > > > > goal and not
> >> > > > > > an unfortunate design byproduct.
> >> There may
> >> > be
> >> > > > some
> >> > > > > > disagreement on whether
> >> > > > > > it _should_ be a design goal, but
> it
> >> may
> >> > not
> >> > > > matter much
> >> > > > > since
> >> > > > > > the current
> >> > > > > > interceptor API is probably
> >> insufficient
> >> > for
> >> > > > middleware
> >> > > > > > applications (I
> >> > > > > > haven't thought of a way to do
> this
> >> that
> >> > > isn't
> >> > > > cumbersome).
> >> > > > > >
> >> > > > > > That aside, the three unresolved
> >> points for
> >> > > me
> >> > > > are the
> >> > > > > > following:
> >> > > > > >
> >> > > > > > 1. The use of varints. As a
> one-off
> >> for use
> >> > > > only with record
> >> > > > > > headers, the
> >> > > > > > case was weak, but if we are using
> >> them
> >> > > > throughout the
> >> > > > > message
> >> > > > > > format, then
> >> > > > > > we should do so here as well. The
> >> > additional
> >> > > > complexity is
> >> > > > > > minimal and
> >> > > > > > early performance testing fully
> >> justifies
> >> > > > their use.
> >> > > > > >
> >> > > > > > 2. If varints are used, the case
> for
> >> using
> >> > > > attributes to
> >> > > > > > indicate null
> >> > > > > > headers also becomes weak. We only
> >> add one
> >> > > > additional byte in
> >> > > > > > each message
> >> > > > > > if there are no headers. Whatever
> >> the case,
> >> > > > let us be
> >> > > > > > consistent with how
> >> > > > > > we treat null keys and values.
> >> > > > > >
> >> > > > > > 3. We have apparently agreed that
> the
> >> > broker
> >> > > > will validate
> >> > > > > > headers (please
> >> > > > > > add this to the KIP). That being
> the
> >> case,
> >> > I
> >> > > > would prefer to
> >> > > > > > use Kafka's
> >> > > > > > conventional format for arrays.
> The
> >> > downside
> >> > > > is that it takes
> >> > > > > > more work to
> >> > > > > > skip over the headers on the
> >> consumer,
> >> > though
> >> > > > it's unclear if
> >> > > > > > the cost
> >> > > > > > would matter in practice. Some
> >> concern was
> >> > > > previously
> >> > > > > > expressed about the
> >> > > > > > allocation of maps. An alternative
> >> would be
> >> > > to
> >> > > > use arrays,
> >> > > > > i.e.
> >> > > > > >
> >> > > > > > class ConsumerRecord<K, V> {
> >> > > > > > K key;
> >> > > > > > V value;
> >> > > > > > Header[] headers;
> >> > > > > > }
> >> > > > > >
> >> > > > > > class Header {
> >> > > > > > String key;
> >> > > > > > byte[] value;
> >> > > > > > }
> >> > > > > >
> >> > > > > > This would work nicely with the
> >> > conventional
> >> > > > array format and
> >> > > > > > my guess is
> >> > > > > > it would obviate the need do any
> lazy
> >> > > > initialization. If we
> >> > > > > > use the map as
> >> > > > > > is currently documented, then it
> is
> >> > possible
> >> > > > with either
> >> > > > > > representation to
> >> > > > > > slice the headers and initialize
> them
> >> > lazily.
> >> > > > Either way, it
> >> > > > > > might be a
> >> > > > > > good idea to use a separate
> object to
> >> > > > represent the headers
> >> > > > > in
> >> > > > > > case we need
> >> > > > > > to modify it in the future in some
> >> way.
> >> > > > > >
> >> > > > > > (By the way, doesn't it feel a bit
> >> odd that
> >> > > we
> >> > > > seem to be
> >> > > > > > designing a
> >> > > > > > feature which is optimized for
> >> people not
> >> > > > using?)
> >> > > > > >
> >> > > > > >
> >> > > > > > If we can resolve these points,
> then
> >> at
> >> > least
> >> > > > you will get my
> >> > > > > > vote.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Jason
> >> > > > > >
> >> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
> >> radai <
> >> > > > > > radai.rosenblatt@gmail.com> wrote:
> >> > > > > >
> >> > > > > > > headers dont "leak" into
> >> application
> >> > code.
> >> > > > they are useful
> >> > > > > > to application
> >> > > > > > > code as well.
> >> > > > > > > IIUC samze currently has headers
> >> "in-V"
> >> > and
> >> > > > would just
> >> > > > > > switch over to kafka
> >> > > > > > > headers if they exist.
> >> > > > > > > im sure plenty of other users of
> >> kafka
> >> > > would
> >> > > > have a use for
> >> > > > > > headers.
> >> > > > > > > im pretty sure use cases exist
> >> around
> >> > > > shuffling data
> >> > > > > > into/out-of kafka
> >> > > > > > > (kafka connect or equivalent)
> where
> >> > > metadata
> >> > > > from one end
> >> > > > > > could copied over
> >> > > > > > > to the other (S3, for example
> uses
> >> http
> >> > > > headers for
> >> > > > > > user-accessible
> >> > > > > > > metadata). it will be kafka
> client
> >> code
> >> > > > getting/setting
> >> > > > > > those headers. not
> >> > > > > > > an interceptor.
> >> > > > > > >
> >> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
> >> Michael
> >> > > > Pearce <
> >> > > > > > Michael.Pearce@ig.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > For APM single event tracing,
> >> need
> >> > access
> >> > > > to the header
> >> > > > > at
> >> > > > > > the point of
> >> > > > > > > > processing on the processing
> >> thread.
> >> > > > > > > >
> >> > > > > > > > As such interceptors will not
> >> work/be
> >> > > > suitable for these,
> >> > > > > > due to the fact
> >> > > > > > > > they act on the
> ConsumerRecords
> >> as a
> >> > > > batch, before the
> >> > > > > > handling thread
> >> > > > > > > can
> >> > > > > > > > split out and process per
> >> message which
> >> > > is
> >> > > > the point
> >> > > > > these
> >> > > > > > tools will
> >> > > > > > > need
> >> > > > > > > > to continue to transaction
> >> tracing.
> >> > > > > > > >
> >> > > > > > > > Like wise tools and infra
> pieces
> >> will
> >> > > need
> >> > > > access to the
> >> > > > > > message outside
> >> > > > > > > > the interceptor realm.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On 17/02/2017, 21:26, "Jason
> >> > Gustafson" <
> >> > > > > > jason@confluent.io> wrote:
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > That’s exactly what
> we’re
> >> doing
> >> > the
> >> > > > headers are a
> >> > > > > > slice of bytes,
> >> > > > > > > > which
> >> > > > > > > > > then gets parsed later
> if
> >> needed,
> >> > > or
> >> > > > can be parsed
> >> > > > > > right away, the
> >> > > > > > > > headers
> >> > > > > > > > > is part of the protocol,
> >> so can
> >> > > > still be validated
> >> > > > > > if wanted.
> >> > > > > > > > > If you had a header
> count
> >> then
> >> > you
> >> > > > would have to go
> >> > > > > > through each
> >> > > > > > > > header
> >> > > > > > > > > key and value length
> value
> >> to
> >> > work
> >> > > > out how much to
> >> > > > > > skip to get to
> >> > > > > > > > say the
> >> > > > > > > > > value or any future
> >> component in
> >> > > the
> >> > > > message after
> >> > > > > > the headers.
> >> > > > > > > > Having it
> >> > > > > > > > > as a byte[] with length
> >> value
> >> > makes
> >> > > > this a lot
> >> > > > > > easier to skip.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > So the broker will parse
> the
> >> > headers
> >> > > > and validate
> >> > > > > > them. Good. The
> >> > > > > > > only
> >> > > > > > > > reason remaining that I
> can
> >> see to
> >> > > > leave the headers
> >> > > > > > as a byte array
> >> > > > > > > > is to
> >> > > > > > > > make it easier for the
> >> client to
> >> > skip
> >> > > > past them. Are
> >> > > > > > we sure this is
> >> > > > > > > > not
> >> > > > > > > > premature optimization?
> Are
> >> there
> >> > any
> >> > > > performance
> >> > > > > > results which show
> >> > > > > > > > that
> >> > > > > > > > this is worthwhile?
> >> > > > > > > >
> >> > > > > > > > What’s the issue with
> >> exposing a
> >> > > > method getHeaders on
> >> > > > > > the
> >> > > > > > > > producer/consumer
> >> > > > > > > > > record? It doesn’t break
> >> > anything.
> >> > > > We don’t need
> >> > > > > any
> >> > > > > > special
> >> > > > > > > version.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > See my previous
> explanation.
> >> What I
> >> > > am
> >> > > > trying to
> >> > > > > > resist is the
> >> > > > > > > headers
> >> > > > > > > > becoming a general
> >> > application-level
> >> > > > facility. The
> >> > > > > > primary use case
> >> > > > > > > as
> >> > > > > > > > far
> >> > > > > > > > as I can tell is
> middleware,
> >> which
> >> > is
> >> > > > the use case
> >> > > > > > that the
> >> > > > > > > > interceptors
> >> > > > > > > > are providing.
> >> > > > > > > >
> >> > > > > > > > Current batch consumer
> model
> >> and
> >> > > > consumer
> >> > > > > interceptors
> >> > > > > > don’t work
> >> > > > > > > where
> >> > > > > > > > > headers need to be acted
> >> on at
> >> > per
> >> > > > message level at
> >> > > > > > time of
> >> > > > > > > > processing,
> >> > > > > > > > > very case is APM (the
> core
> >> one),
> >> > > > where the header
> >> > > > > > value is used to
> >> > > > > > > > continue
> >> > > > > > > > > tracing.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I still don't understand
> the
> >> point
> >> > > > about batching.
> >> > > > > The
> >> > > > > > consumer
> >> > > > > > > > records are
> >> > > > > > > > exposed as a batch in the
> >> consumer
> >> > > > interceptor, but
> >> > > > > > you can still
> >> > > > > > > > iterate
> >> > > > > > > > them individually. It is
> no
> >> > different
> >> > > > for the
> >> > > > > consumer
> >> > > > > > API itself.
> >> > > > > > > >
> >> > > > > > > > -Jason
> >> > > > > > > >
> >> > > > > > > > On Fri, Feb 17, 2017 at
> >> 12:48 PM,
> >> > > > Michael Pearce <
> >> > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Re:
> >> > > > > > > > >
> >> > > > > > > > > “ The point about
> >> creation of
> >> > > > maps seems
> >> > > > > > orthogonal. We can
> >> > > > > > > still
> >> > > > > > > > > represent
> >> > > > > > > > > the headers as a
> slice
> >> of
> >> > bytes
> >> > > > until the time
> >> > > > > > it is accessed.”
> >> > > > > > > > >
> >> > > > > > > > > That’s exactly what
> we’re
> >> doing
> >> > the
> >> > > > headers are a
> >> > > > > > slice of bytes,
> >> > > > > > > > which
> >> > > > > > > > > then gets parsed later
> if
> >> needed,
> >> > > or
> >> > > > can be parsed
> >> > > > > > right away, the
> >> > > > > > > > headers
> >> > > > > > > > > is part of the protocol,
> >> so can
> >> > > > still be validated
> >> > > > > > if wanted.
> >> > > > > > > > >
> >> > > > > > > > > If you had a header
> count
> >> then
> >> > you
> >> > > > would have to go
> >> > > > > > through each
> >> > > > > > > > header
> >> > > > > > > > > key and value length
> value
> >> to
> >> > work
> >> > > > out how much to
> >> > > > > > skip to get to
> >> > > > > > > > say the
> >> > > > > > > > > value or any future
> >> component in
> >> > > the
> >> > > > message after
> >> > > > > > the headers.
> >> > > > > > > > Having it
> >> > > > > > > > > as a byte[] with length
> >> value
> >> > makes
> >> > > > this a lot
> >> > > > > > easier to skip.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On 17/02/2017, 20:37,
> >> "Michael
> >> > > > Pearce" <
> >> > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > What’s the issue
> with
> >> > exposing
> >> > > a
> >> > > > method
> >> > > > > > getHeaders on the
> >> > > > > > > > > producer/consumer
> record?
> >> It
> >> > > doesn’t
> >> > > > break
> >> > > > > anything.
> >> > > > > > We don’t need
> >> > > > > > > > any
> >> > > > > > > > > special version.
> >> > > > > > > > >
> >> > > > > > > > > Current batch
> consumer
> >> model
> >> > > and
> >> > > > consumer
> >> > > > > > interceptors don’t
> >> > > > > > > work
> >> > > > > > > > > where headers need to be
> >> acted on
> >> > > at
> >> > > > per message
> >> > > > > > level at time of
> >> > > > > > > > > processing, very case is
> >> APM (the
> >> > > > core one), where
> >> > > > > > the header value
> >> > > > > > > > is used
> >> > > > > > > > > to continue tracing.
> >> JMS/HTTP etc
> >> > > > all expose these,
> >> > > > > > without issues.
> >> > > > > > > > I would
> >> > > > > > > > > NOT want to lock this
> down
> >> to
> >> > only
> >> > > > be usable
> >> > > > > > accessible via
> >> > > > > > > > interceptors,
> >> > > > > > > > > as we’d fail on one of
> the
> >> main
> >> > > > goals.
> >> > > > > > > > >
> >> > > > > > > > > Regards
> >> > > > > > > > > Mike
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On 17/02/2017,
> 20:21,
> >> "Jason
> >> > > > Gustafson" <
> >> > > > > > jason@confluent.io>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > The point about
> >> creation
> >> > of
> >> > > > maps seems
> >> > > > > > orthogonal. We can
> >> > > > > > > > still
> >> > > > > > > > > represent
> >> > > > > > > > > the headers as a
> >> slice of
> >> > > > bytes until the
> >> > > > > > time it is
> >> > > > > > > > accessed.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > Yes exactly we
> >> have
> >> > > access
> >> > > > to the records
> >> > > > > > thus why the
> >> > > > > > > > header
> >> > > > > > > > > should be
> >> > > > > > > > > > accessible via
> >> it and
> >> > not
> >> > > > hidden for only
> >> > > > > > interceptors to
> >> > > > > > > > access.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > As explained
> >> above, the
> >> > > > point is to make
> >> > > > > the
> >> > > > > > intended usage
> >> > > > > > > > clear.
> >> > > > > > > > > Applications
> should
> >> > > continue
> >> > > > to rely on the
> >> > > > > > key/value
> >> > > > > > > fields
> >> > > > > > > > to
> >> > > > > > > > > serialize
> >> > > > > > > > > their own
> headers,
> >> and it
> >> > > > would be more
> >> > > > > > ideal if we can
> >> > > > > > > avoid
> >> > > > > > > > > leaking
> >> > > > > > > > > third-party
> >> headers into
> >> > > > applications. This
> >> > > > > > is difficult to
> >> > > > > > > > do
> >> > > > > > > > > with the
> >> > > > > > > > > current
> >> interceptors
> >> > > because
> >> > > > they share the
> >> > > > > > record objects
> >> > > > > > > > with
> >> > > > > > > > > the common
> >> > > > > > > > > API. What I had
> in
> >> mind
> >> > is
> >> > > > something like
> >> > > > > an
> >> > > > > > extension of
> >> > > > > > > the
> >> > > > > > > > > current
> >> > > > > > > > > interceptors
> which
> >> > exposed
> >> > > a
> >> > > > different
> >> > > > > > object (e.g.
> >> > > > > > > > > `RecordAndHeaders`).
> >> > > > > > > > > The challenge is
> >> for
> >> > > MM-like
> >> > > > use cases. Let
> >> > > > > > me see if I can
> >> > > > > > > > come
> >> > > > > > > > > up with a
> >> > > > > > > > > concrete
> proposal
> >> for
> >> > that
> >> > > > problem.
> >> > > > > > > > >
> >> > > > > > > > > -Jason
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Feb 17,
> >> 2017 at
> >> > > > 11:55 AM, Michael
> >> > > > > > Pearce <
> >> > > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > I am happy to
> >> move the
> >> > > > definition of the
> >> > > > > > header into the
> >> > > > > > > > message
> >> > > > > > > > > body, but
> >> > > > > > > > > > would cause us
> >> not to
> >> > > lazy
> >> > > > > > initialise/parse the headers,
> >> > > > > > > as
> >> > > > > > > > > obviously, we
> >> > > > > > > > > > would have to
> >> traverse
> >> > > > these reading the
> >> > > > > > message.
> >> > > > > > > > > >
> >> > > > > > > > > > This was
> >> actually one
> >> > of
> >> > > > Jay’s requests:
> >> > > > > > > > > >
> >> > > > > > > > > > “ 2. I
> think
> >> we
> >> > should
> >> > > > think about
> >> > > > > > creating the Map
> >> > > > > > > > lazily to
> >> > > > > > > > > avoid
> >> > > > > > > > > > parsing
> out
> >> all the
> >> > > > headers into
> >> > > > > > little objects.
> >> > > > > > > > HashMaps
> >> > > > > > > > > themselves
> >> > > > > > > > > > are
> >> > > > > > > > > > kind of
> >> expensive
> >> > and
> >> > > > the consumer is
> >> > > > > > very perf
> >> > > > > > > > sensitive so
> >> > > > > > > > > and making
> >> > > > > > > > > >
> gazillions of
> >> > > hashmaps
> >> > > > that may or
> >> > > > > may
> >> > > > > > not get used
> >> > > > > > > is
> >> > > > > > > > > probably a bad
> >> > > > > > > > > > idea.”
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On 17/02/2017,
> >> 19:44,
> >> > > > "Michael Pearce" <
> >> > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > Yes
> exactly
> >> we have
> >> > > > access to the
> >> > > > > > records thus why
> >> > > > > > > the
> >> > > > > > > > > header should
> >> > > > > > > > > > be accessible
> >> via it
> >> > and
> >> > > > not hidden for
> >> > > > > > only interceptors
> >> > > > > > > > to
> >> > > > > > > > > access.
> >> > > > > > > > > >
> >> > > > > > > > > > Sent using
> >> OWA for
> >> > > > iPhone
> >> > > > > > > > > >
> >> > > > ______________________________
> >> > > > > > __________
> >> > > > > > > > > > From:
> Magnus
> >> > > Edenhill <
> >> > > > > > magnus@edenhill.se>
> >> > > > > > > > > > Sent:
> Friday,
> >> > > February
> >> > > > 17, 2017
> >> > > > > > 7:34:49 PM
> >> > > > > > > > > > To:
> >> > > > dev@kafka.apache.org
> >> > > > > > > > > > Subject:
> Re:
> >> > > [DISCUSS]
> >> > > > KIP-82 - Add
> >> > > > > > Record Headers
> >> > > > > > > > > >
> >> > > > > > > > > > Big +1 on
> >> VarInts.
> >> > > > > > > > > > CPUs are
> >> fast,
> >> > memory
> >> > > > is slow.
> >> > > > > > > > > >
> >> > > > > > > > > > I agree
> with
> >> Jason
> >> > > > that we'll want to
> >> > > > > > continue
> >> > > > > > > > verifying
> >> > > > > > > > > messages,
> >> > > > > > > > > > including
> >> their
> >> > > > headers, so while I
> >> > > > > > appreciate the
> >> > > > > > > > idea of
> >> > > > > > > > > the opaque
> >> > > > > > > > > > header
> blob
> >> it
> >> > won't
> >> > > > be useful in
> >> > > > > > practice.
> >> > > > > > > > > >
> >> > > > > > > > > > /Magnus
> >> > > > > > > > > >
> >> > > > > > > > > > 2017-02-17
> >> 10:41
> >> > > > GMT-08:00 Jason
> >> > > > > > Gustafson <
> >> > > > > > > > > jason@confluent.io>:
> >> > > > > > > > > >
> >> > > > > > > > > > > Sorry,
> my
> >> > mistake.
> >> > > > The consumer
> >> > > > > > interceptor is per
> >> > > > > > > > batch,
> >> > > > > > > > > though I'm
> >> > > > > > > > > > not
> >> > > > > > > > > > > sure
> >> that's an
> >> > > > actual limitation
> >> > > > > > since you still
> >> > > > > > > have
> >> > > > > > > > > access to the
> >> > > > > > > > > > >
> individual
> >> > records.
> >> > > > > > > > > > >
> >> > > > > > > > > > > -Jason
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Fri,
> >> Feb 17,
> >> > > 2017
> >> > > > at 10:39 AM,
> >> > > > > > Jason Gustafson <
> >> > > > > > > > > >
> >> jason@confluent.io>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Re
> >> headers as
> >> > > byte
> >> > > > array and
> >> > > > > > future use by
> >> > > > > > > broker.
> >> > > > > > > > This
> >> > > > > > > > > doesn't
> >> > > > > > > > > > take away
> >> > > > > > > > > > > >> from
> >> that at
> >> > > all.
> >> > > > Nor makes it
> >> > > > > > difficult at all
> >> > > > > > > > in my
> >> > > > > > > > > opinion.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Yeah,
> I
> >> didn't
> >> > > say
> >> > > > it was
> >> > > > > > difficult, only
> >> > > > > > > awkward.
> >> > > > > > > > You
> >> > > > > > > > > wouldn't
> >> > > > > > > > > > write the
> >> > > > > > > > > > > > schema
> >> that way
> >> > > if
> >> > > > you were
> >> > > > > > planning to use it on
> >> > > > > > > > the
> >> > > > > > > > > brokers from
> >> > > > > > > > > > the
> >> > > > > > > > > > > >
> >> beginning. Note
> >> > > > also that one of
> >> > > > > > the benefits of
> >> > > > > > > > letting
> >> > > > > > > > > the broker
> >> > > > > > > > > > > >
> >> understand
> >> > > headers
> >> > > > is that it can
> >> > > > > > validate that
> >> > > > > > > > they are
> >> > > > > > > > > properly
> >> > > > > > > > > > > >
> >> formatted. If
> >> > > cost
> >> > > > is the only
> >> > > > > > concern, we should
> >> > > > > > > > > confirm its
> >> > > > > > > > > > impact
> >> > > > > > > > > > > >
> through
> >> > > > performance testing.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > One of
> >> the key
> >> > > use
> >> > > > cases requires
> >> > > > > > access on
> >> > > > > > > > consume at
> >> > > > > > > > > per
> >> > > > > > > > > > event/message
> >> > > > > > > > > > > >> level
> >> at the
> >> > > > point that message
> >> > > > > > is being
> >> > > > > > > > processed, as
> >> > > > > > > > > such the
> >> > > > > > > > > > batch
> >> > > > > > > > > > > >>
> >> interceptors
> >> > and
> >> > > > batch consume
> >> > > > > > api isn't
> >> > > > > > > > suitable. It
> >> > > > > > > > > needs to be
> >> > > > > > > > > > at the
> >> > > > > > > > > > > >>
> record
> >> level.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I'm
> not
> >> sure I
> >> > > > understand the
> >> > > > > > point about
> >> > > > > > > batching.
> >> > > > > > > > > Interceptors
> >> > > > > > > > > > are
> >> > > > > > > > > > > >
> applied
> >> > > > per-message, right?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > My
> >> intent on
> >> > > > interceptors is to
> >> > > > > > keep the usage of
> >> > > > > > > > headers
> >> > > > > > > > > > well-defined
> so
> >> > > > > > > > > > > > that
> >> they don't
> >> > > > start leaking
> >> > > > > > unnecessarily into
> >> > > > > > > > > applications. My
> >> > > > > > > > > > guess
> >> > > > > > > > > > > is
> >> > > > > > > > > > > > that
> it's
> >> > > probably
> >> > > > inevitable,
> >> > > > > but
> >> > > > > > isolating it
> >> > > > > > > in
> >> > > > > > > > the
> >> > > > > > > > > > interceptors
> >> would
> >> > > > > > > > > > > > at
> least
> >> give
> >> > > > people a second
> >> > > > > > thought before
> >> > > > > > > > deciding to
> >> > > > > > > > > use it.
> >> > > > > > > > > > The main
> >> > > > > > > > > > > >
> >> challenge in my
> >> > > > mind is figuring
> >> > > > > > out how an MM
> >> > > > > > > use
> >> > > > > > > > case
> >> > > > > > > > > would
> >> > > > > > > > > > work. It
> >> > > > > > > > > > > > would
> be
> >> more
> >> > > > cumbersome to
> >> > > > > > replicate headers
> >> > > > > > > > through an
> >> > > > > > > > > > interceptor,
> >> > > > > > > > > > > > though
> >> arguably
> >> > > MM
> >> > > > should be
> >> > > > > > working at a lower
> >> > > > > > > > level
> >> > > > > > > > > anyway.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > -Jason
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On
> Fri,
> >> Feb 17,
> >> > > > 2017 at 10:16 AM,
> >> > > > > > Michael Pearce
> >> > > > > > > <
> >> > > > > > > > > >
> >> Michael.Pearce@ig.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >> Re
> >> headers
> >> > > > available on the
> >> > > > > > record va
> >> > > > > > > > interceptors only
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> One
> of
> >> the key
> >> > > > use cases
> >> > > > > requires
> >> > > > > > access on
> >> > > > > > > > consume at
> >> > > > > > > > > per
> >> > > > > > > > > > event/message
> >> > > > > > > > > > > >> level
> >> at the
> >> > > > point that message
> >> > > > > > is being
> >> > > > > > > > processed, as
> >> > > > > > > > > such the
> >> > > > > > > > > > batch
> >> > > > > > > > > > > >>
> >> interceptors
> >> > and
> >> > > > batch consume
> >> > > > > > api isn't
> >> > > > > > > > suitable. It
> >> > > > > > > > > needs to be
> >> > > > > > > > > > at the
> >> > > > > > > > > > > >>
> record
> >> level.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> This
> >> anyhow is
> >> > > > similar to
> >> > > > > > jms/http/amqp where
> >> > > > > > > > headers
> >> > > > > > > > > are
> >> > > > > > > > > > available to
> >> > > > > > > > > > > >>
> >> consuming
> >> > > > applications.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> Re
> >> headers as
> >> > > > byte array and
> >> > > > > > future use by
> >> > > > > > > > broker. This
> >> > > > > > > > > doesn't
> >> > > > > > > > > > take
> >> > > > > > > > > > > away
> >> > > > > > > > > > > >> from
> >> that at
> >> > > all.
> >> > > > Nor makes it
> >> > > > > > difficult at all
> >> > > > > > > > in my
> >> > > > > > > > > opinion.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> Sent
> >> using OWA
> >> > > > for iPhone
> >> > > > > > > > > > > >>
> >> > > > ______________________________
> >> > > > > > __________
> >> > > > > > > > > > > >> From:
> >> Jason
> >> > > > Gustafson <
> >> > > > > > jason@confluent.io>
> >> > > > > > > > > > > >> Sent:
> >> Friday,
> >> > > > February 17, 2017
> >> > > > > > 5:55:42 PM
> >> > > > > > > > > > > >> To:
> >> > > > dev@kafka.apache.org
> >> > > > > > > > > > > >>
> >> Subject: Re:
> >> > > > [DISCUSS] KIP-82 -
> >> > > > > > Add Record
> >> > > > > > > Headers
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> Would
> >> you be
> >> > > > proposing in
> >> > > > > > KIP-98 to convert
> >> > > > > > > the
> >> > > > > > > > other
> >> > > > > > > > > message
> >> > > > > > > > > > int’s
> >> > > > > > > > > > > (key
> >> > > > > > > > > > > >> >
> >> length,
> >> > value
> >> > > > length) also to
> >> > > > > > varint to keep
> >> > > > > > > it
> >> > > > > > > > > uniform.
> >> > > > > > > > > > > >> >
> Also I
> >> > assume
> >> > > > there will be a
> >> > > > > > static or helper
> >> > > > > > > > method
> >> > > > > > > > > made to
> >> > > > > > > > > > >
> write/read
> >> > > > > > > > > > > >> >
> these
> >> in the
> >> > > > client and
> >> > > > > server.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> Yes,
> >> that is
> >> > > what
> >> > > > we are
> >> > > > > > proposing, so using
> >> > > > > > > > varints
> >> > > > > > > > > for headers
> >> > > > > > > > > > would
> >> > > > > > > > > > > be
> >> > > > > > > > > > > >>
> >> consistent
> >> > with
> >> > > > the rest of the
> >> > > > > > message. We have
> >> > > > > > > > used
> >> > > > > > > > > static
> >> > > > > > > > > > helper
> >> > > > > > > > > > > >>
> methods
> >> > > > > > > > > > > >> in
> our
> >> > prototype
> >> > > > implementation.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> The
> >> cost of
> >> > > > parsing, we want to
> >> > > > > > parse/interpret
> >> > > > > > > > the
> >> > > > > > > > > headers
> >> > > > > > > > > > lazily (this
> >> > > > > > > > > > > >> is
> >> > > > > > > > > > > >> > a
> key
> >> point
> >> > > > brought up earlier
> >> > > > > > in discussions)
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> I'm a
> >> bit
> >> > > > skeptical of this. Has
> >> > > > > > anyone done the
> >> > > > > > > > > performance
> >> > > > > > > > > > testing? I
> >> > > > > > > > > > > >> can
> >> > > > > > > > > > > >>
> probably
> >> > > > implement it and test
> >> > > > > it
> >> > > > > > if no one else
> >> > > > > > > > has. I
> >> > > > > > > > > was also
> >> > > > > > > > > > under
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >>
> >> impression
> >> > that
> >> > > > there may be use
> >> > > > > > cases down the
> >> > > > > > > > road
> >> > > > > > > > > where the
> >> > > > > > > > > > broker
> >> > > > > > > > > > > >> would
> >> > > > > > > > > > > >> need
> to
> >> > > interpret
> >> > > > headers. That
> >> > > > > > wouldn't be off
> >> > > > > > > > the
> >> > > > > > > > > table in the
> >> > > > > > > > > > future
> >> > > > > > > > > > > if
> >> > > > > > > > > > > >> it's
> >> > represented
> >> > > > as bytes, but
> >> > > > > it
> >> > > > > > would be quite
> >> > > > > > > > a bit
> >> > > > > > > > > more
> >> > > > > > > > > > awkward,
> >> > > > > > > > > > > >>
> right?
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> By
> the
> >> way,
> >> > one
> >> > > > question I have
> >> > > > > > been wondering
> >> > > > > > > > about. My
> >> > > > > > > > > > understanding
> >> > > > > > > > > > > is
> >> > > > > > > > > > > >> that
> >> headers
> >> > are
> >> > > > primarily for
> >> > > > > > use cases where a
> >> > > > > > > > > third-party
> >> > > > > > > > > > components
> >> > > > > > > > > > > >>
> wants to
> >> > enrich
> >> > > > messages without
> >> > > > > > needing to
> >> > > > > > > > understand
> >> > > > > > > > > or modify
> >> > > > > > > > > > the
> >> > > > > > > > > > > >>
> schema
> >> > > > > > > > > > > >> of
> the
> >> message
> >> > > > key and value.
> >> > > > > For
> >> > > > > > the
> >> > > > > > > applications
> >> > > > > > > > > which directly
> >> > > > > > > > > > > produce
> >> > > > > > > > > > > >> and
> >> consume
> >> > the
> >> > > > messages and
> >> > > > > > control the
> >> > > > > > > key/value
> >> > > > > > > > > schema
> >> > > > > > > > > > directly, it
> >> > > > > > > > > > > >> seems
> >> we would
> >> > > > rather have them
> >> > > > > > implement
> >> > > > > > > headers
> >> > > > > > > > > directly in
> >> > > > > > > > > > their own
> >> > > > > > > > > > > >>
> schema.
> >> > > Supposing
> >> > > > for the sake
> >> > > > > of
> >> > > > > > argument that
> >> > > > > > > > it was
> >> > > > > > > > > possible,
> >> > > > > > > > > > my
> >> > > > > > > > > > > >>
> >> question is
> >> > > > whether it be
> >> > > > > > sufficient to expose
> >> > > > > > > the
> >> > > > > > > > > headers in the
> >> > > > > > > > > > > >>
> >> interceptor
> >> > API
> >> > > > and not in the
> >> > > > > > common API?
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> -Jason
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> On
> Fri,
> >> Feb
> >> > 17,
> >> > > > 2017 at 3:26 AM,
> >> > > > > > Michael Pearce
> >> > > > > > > <
> >> > > > > > > > > >
> >> Michael.Pearce@ig.com>
> >> > > > > > > > > > > >>
> wrote:
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> > On
> >> the point
> >> > > of
> >> > > > varInts
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> Would
> >> you be
> >> > > > proposing in
> >> > > > > > KIP-98 to convert
> >> > > > > > > the
> >> > > > > > > > other
> >> > > > > > > > > message
> >> > > > > > > > > > int’s
> >> > > > > > > > > > > (key
> >> > > > > > > > > > > >> >
> >> length,
> >> > value
> >> > > > length) also to
> >> > > > > > varint to keep
> >> > > > > > > it
> >> > > > > > > > > uniform.
> >> > > > > > > > > > > >> >
> Also I
> >> > assume
> >> > > > there will be a
> >> > > > > > static or helper
> >> > > > > > > > method
> >> > > > > > > > > made to
> >> > > > > > > > > > >
> write/read
> >> > > > > > > > > > > >> >
> these
> >> in the
> >> > > > client and
> >> > > > > server.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> Cheers
> >> > > > > > > > > > > >> >
> Mike
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> > On
> >> > 17/02/2017,
> >> > > > 11:22, "Michael
> >> > > > > > Pearce" <
> >> > > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> On the
> >> > > > point re: headers
> >> > > > > in
> >> > > > > > the message
> >> > > > > > > > protocol
> >> > > > > > > > > being a
> >> > > > > > > > > > byte
> >> > > > > > > > > > > array
> >> > > > > > > > > > > >> > and
> >> not a
> >> > > count
> >> > > > of elements
> >> > > > > > followed by the
> >> > > > > > > > elements.
> >> > > > > > > > > Again
> >> > > > > > > > > > this was
> >> > > > > > > > > > > >> >
> >> > > > discussed/argued previously.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> It was
> >> > > > agreed on for a few
> >> > > > > > reasons some of
> >> > > > > > > > which
> >> > > > > > > > > you have
> >> > > > > > > > > > >
> obviously
> >> > > > > > > > > > > >> >
> >> picked up
> >> > on:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> Broker
> >> > is
> >> > > > able to pass it
> >> > > > > > through opaquely
> >> > > > > > > > > > > >> >
> >> The cost
> >> > > of
> >> > > > parsing, we
> >> > > > > > want to
> >> > > > > > > > parse/interpret
> >> > > > > > > > > the headers
> >> > > > > > > > > > lazily
> >> > > > > > > > > > > >> >
> (this
> >> is a
> >> > key
> >> > > > point brought
> >> > > > > up
> >> > > > > > earlier in
> >> > > > > > > > > discussions)
> >> > > > > > > > > > > >> >
> >> Headers
> >> > > can
> >> > > > be copied from
> >> > > > > > consumer record
> >> > > > > > > > to
> >> > > > > > > > > producer
> >> > > > > > > > > > record (aka
> >> > > > > > > > > > > >> >
> mirror
> >> > makers
> >> > > > etc) without
> >> > > > > > parsing if no
> >> > > > > > > > changes are
> >> > > > > > > > > being made
> >> > > > > > > > > > or
> >> > > > > > > > > > > being
> >> > > > > > > > > > > >> >
> >> looked at.
> >> > > > > > > > > > > >> >
> >> Keeps
> >> > the
> >> > > > broker agnostic
> >> > > > > > to the format
> >> > > > > > > > > > > >> >
> >> You need
> >> > > an
> >> > > > int32 either
> >> > > > > > for the byte size
> >> > > > > > > > of the
> >> > > > > > > > > headers,
> >> > > > > > > > > > or for
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> count
> >> of
> >> > > > elements, so
> >> > > > > overheads
> >> > > > > > are the same,
> >> > > > > > > > but
> >> > > > > > > > > with going
> >> > > > > > > > > > with an
> >> > > > > > > > > > > >>
> opaque
> >> > > > > > > > > > > >> >
> byte
> >> array
> >> > has
> >> > > > the above
> >> > > > > > advantages.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> Cheers
> >> > > > > > > > > > > >> >
> >> Mike
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> On
> >> > > > 17/02/2017, 02:50,
> >> > > > > > "Jason Gustafson" <
> >> > > > > > > > > jason@confluent.io
> >> > > > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > Sorry,
> >> > > > should have
> >> > > > > > noted that the
> >> > > > > > > > performance
> >> > > > > > > > > testing
> >> > > > > > > > > > was done
> >> > > > > > > > > > > >> >
> using
> >> the
> >> > > > > > > > > > > >> >
> >> > > > producer performance
> >> > > > > > tool shipped with
> >> > > > > > > > Kafka.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > -Jason
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> On
> >> > > Thu,
> >> > > > Feb 16, 2017
> >> > > > > at
> >> > > > > > 6:44 PM, Jason
> >> > > > > > > > > Gustafson <
> >> > > > > > > > > > > >> >
> >> > > > jason@confluent.io> wrote:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > Hey
> >> > > > Nacho,
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > I've
> >> > > > compared
> >> > > > > > performance of our
> >> > > > > > > > KIP-98
> >> > > > > > > > > > implementation
> >> with
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> >
> >> without
> >> > > > > > > > > > > >> >
> >> >
> >> > > > varints. For
> >> > > > > messages
> >> > > > > > around 128
> >> > > > > > > > bytes, we
> >> > > > > > > > > see an
> >> > > > > > > > > > increase
> >> > > > > > > > > > > in
> >> > > > > > > > > > > >> >
> >> throughput
> >> > of
> >> > > > > > > > > > > >> >
> >> >
> >> > > about
> >> > > > 30% using the
> >> > > > > > default
> >> > > > > > > > configuration
> >> > > > > > > > > settings.
> >> > > > > > > > > > At 256
> >> > > > > > > > > > > >> >
> >> bytes, the
> >> > > > > > > > > > > >> >
> >> >
> >> > > > increase is around
> >> > > > > > 16%. Obviously
> >> > > > > > > the
> >> > > > > > > > > performance
> >> > > > > > > > > > converge
> >> > > > > > > > > > > as
> >> > > > > > > > > > > >> >
> >> messages get
> >> > > > > > > > > > > >> >
> >> >
> >> > > > larger, but it seems
> >> > > > > > well worth the
> >> > > > > > > > cost.
> >> > > > > > > > > Note that
> >> > > > > > > > > > we are
> >> > > > > > > > > > > >> also
> >> > > > > > > > > > > >> >
> >> seeing a
> >> > > > > > > > > > > >> >
> >> >
> >> > > > substantial
> >> > > > > > performance increase
> >> > > > > > > > against
> >> > > > > > > > > trunk
> >> > > > > > > > > > primarily
> >> > > > > > > > > > > >>
> because
> >> > > > > > > > > > > >> > of
> the
> >> > > > > > > > > > > >> >
> >> >
> >> > much
> >> > > > more efficient
> >> > > > > > packing that
> >> > > > > > > > varints
> >> > > > > > > > > provide us.
> >> > > > > > > > > > > Anything
> >> > > > > > > > > > > >> >
> >> adding to
> >> > > > > > > > > > > >> >
> >> >
> >> > > > message overhead,
> >> > > > > > such as record
> >> > > > > > > > headers,
> >> > > > > > > > > would only
> >> > > > > > > > > > > increase
> >> > > > > > > > > > > >> > the
> >> relative
> >> > > > > > > > > > > >> >
> >> >
> >> > > > difference. (Of
> >> > > > > > course take these
> >> > > > > > > > numbers
> >> > > > > > > > > with a
> >> > > > > > > > > > grain of
> >> > > > > > > > > > > salt
> >> > > > > > > > > > > >> >
> since
> >> I have
> >> > > > > > > > > > > >> >
> >> >
> >> > only
> >> > > > used the
> >> > > > > default
> >> > > > > > settings with
> >> > > > > > > > both
> >> > > > > > > > > the producer
> >> > > > > > > > > > and
> >> > > > > > > > > > > >>
> broker
> >> > > > > > > > > > > >> > on
> my
> >> > > > > > > > > > > >> >
> >> >
> >> > > local
> >> > > > machine. We
> >> > > > > > intend to provide
> >> > > > > > > > more
> >> > > > > > > > > extensive
> >> > > > > > > > > > >
> performance
> >> > > > > > > > > > > >> >
> >> details as
> >> > > > > > > > > > > >> >
> >> >
> >> > part
> >> > > > of the work for
> >> > > > > > KIP-98.)
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > The
> >> > > > implementation
> >> > > > > we
> >> > > > > > are using is
> >> > > > > > > > from
> >> > > > > > > > > protobuf (
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > https://developers.google.com/
> >> > > > > > > > > > >
> >> > > > protocol-buffers/docs/encoding
> >> > > > > > > > > > > >> ),
> >> > > > > > > > > > > >> >
> which
> >> is
> >> > > > > > > > > > > >> >
> >> >
> >> > also
> >> > > > used in HBase.
> >> > > > > > It is trivial to
> >> > > > > > > > > implement and as
> >> > > > > > > > > > far
> >> > > > > > > > > > > as I
> >> > > > > > > > > > > >> >
> know
> >> doesn't
> >> > > > > > > > > > > >> >
> >> >
> >> > > > suffer from the
> >> > > > > > aliasing problem you
> >> > > > > > > > are
> >> > > > > > > > > describing. I
> >> > > > > > > > > > > checked
> >> > > > > > > > > > > >> >
> with
> >> Magnus
> >> > > > > > > > > > > >> >
> >> >
> >> > (the
> >> > > > author of
> >> > > > > > librdkafka) and he
> >> > > > > > > > agreed
> >> > > > > > > > > that the
> >> > > > > > > > > > savings
> >> > > > > > > > > > > >>
> seemed
> >> > > > > > > > > > > >> >
> worth
> >> the
> >> > > > > > > > > > > >> >
> >> >
> >> > cost
> >> > > > of
> >> > > > > > implementation.
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > > -Jason
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> > On
> >> > > > Thu, Feb 16, 2017
> >> > > > > > at 4:32 PM,
> >> > > > > > > > Ignacio
> >> > > > > > > > > Solis <
> >> > > > > > > > > > > >>
> >> > isolis@igso.net
> >> > > >
> >> > > > > > > > > > > >> >
> wrote:
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > -VarInts
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > I'm
> >> > > > one of the
> >> > > > > > people (if not the
> >> > > > > > > > most)
> >> > > > > > > > > opposed to
> >> > > > > > > > > > VarInts.
> >> > > > > > > > > > > >> >
> >> VarInts
> >> > > > > > > > > > > >> >
> >> >>
> >> > > have
> >> > > > a place, but
> >> > > > > > this is not it.
> >> > > > > > > > (We
> >> > > > > > > > > had a large
> >> > > > > > > > > > > >>
> >> discussion
> >> > > > > > > > > > > >> >
> about
> >> > > > > > > > > > > >> >
> >> >>
> >> > > them
> >> > > > at the
> >> > > > > > beginning of KIP-82
> >> > > > > > > time)
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > If
> >> > > > anybody has real
> >> > > > > > life
> >> > > > > > > performance
> >> > > > > > > > > numbers of
> >> > > > > > > > > > VarInts
> >> > > > > > > > > > > >> >
> >> improving
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > things or
> >> > > > > > significantly reducing
> >> > > > > > > > resources
> >> > > > > > > > > I would
> >> > > > > > > > > > like to
> >> > > > > > > > > > > >> know
> >> > > > > > > > > > > >> >
> what
> >> > > > > > > > > > > >> >
> >> >>
> >> > > that
> >> > > > case may be.
> >> > > > > > Yes, you can save
> >> > > > > > > > some
> >> > > > > > > > > bytes here
> >> > > > > > > > > > and
> >> > > > > > > > > > > >>
> there,
> >> > > > > > > > > > > >> > but
> >> > > > > > > > > > > >> >
> >> >>
> >> > > this
> >> > > > is probably
> >> > > > > > insignificant to
> >> > > > > > > the
> >> > > > > > > > > overall system
> >> > > > > > > > > > > behavior
> >> > > > > > > > > > > >> > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > storage
> >> > > > > > requirements. -- I say
> >> > > > > > > this
> >> > > > > > > > with
> >> > > > > > > > > respect to
> >> > > > > > > > > > using
> >> > > > > > > > > > > >> >
> >> VarInts in
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > protocol
> >> > > > > itself,
> >> > > > > > not as part of
> >> > > > > > > > the
> >> > > > > > > > > data.
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > VarInts require you
> >> > > > > > to parse the
> >> > > > > > > Int
> >> > > > > > > > > before using it
> >> > > > > > > > > > and
> >> > > > > > > > > > > >> >
> >> depending on
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > encoding they
> >> > > > > > can suffer from
> >> > > > > > > > aliasing
> >> > > > > > > > > (multiple
> >> > > > > > > > > > > >> >
> >> > > representations
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > the same
> >> > > > > value).
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > Why
> >> > > > add complexity?
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > Nacho
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > On
> >> > > > Thu, Feb 16,
> >> > > > > 2017
> >> > > > > > at 10:29 AM,
> >> > > > > > > > Colin
> >> > > > > > > > > McCabe <
> >> > > > > > > > > > > >> >
> >> > > > cmccabe@apache.org>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > wrote:
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > +1
> >> > > > for varints
> >> > > > > > here-- it would
> >> > > > > > > save
> >> > > > > > > > > quite a bit of
> >> > > > > > > > > > space.
> >> > > > > > > > > > > >> >
> They
> >> are
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > pretty quick to
> >> > > > > > implement as
> >> > > > > > > well.
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >>
> >> > > I
> >> > > > think it makes
> >> > > > > > sense for values
> >> > > > > > > > to be
> >> > > > > > > > > byte
> >> > > > > > > > > > arrays.
> >> > > > > > > > > > > Users
> >> > > > > > > > > > > >> >
> might
> >> want
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > to
> >> > > > attach
> >> > > > > > arbitrary payloads;
> >> > > > > > > they
> >> > > > > > > > > shouldn't be
> >> > > > > > > > > > forced to
> >> > > > > > > > > > > >> >
> >> serialize
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > everything to
> >> > > > > Java
> >> > > > > > strings.
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > best,
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > Colin
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > On
> >> > > > Thu, Feb 16,
> >> > > > > > 2017, at 09:52,
> >> > > > > > > > Jason
> >> > > > > > > > > Gustafson
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > Hey Michael,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > Hmm, I guess the
> >> > > > > > point of
> >> > > > > > > > representing
> >> > > > > > > > > it as
> >> > > > > > > > > > bytes is to
> >> > > > > > > > > > > >> >
> allow
> >> the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > broker
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > to pass it
> >> > > > > > through opaquely? Is
> >> > > > > > > > the
> >> > > > > > > > > cost of
> >> > > > > > > > > > parsing
> >> > > > > > > > > > > them a
> >> > > > > > > > > > > >> >
> >> concern, or
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > are
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > we simply trying
> >> > > > > > to ensure that
> >> > > > > > > > the
> >> > > > > > > > > broker stays
> >> > > > > > > > > > > agnostic
> >> > > > > > > > > > > >> to
> >> > > > > > > > > > > >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > format?
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > On varints, I
> >> > > > > > think adding
> >> > > > > > > > support for
> >> > > > > > > > > them makes
> >> > > > > > > > > > less
> >> > > > > > > > > > > >> sense
> >> > > > > > > > > > > >> >
> for an
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > isolated use
> >> > > > > > case, but as part
> >> > > > > > > of
> >> > > > > > > > a
> >> > > > > > > > > more holistic
> >> > > > > > > > > > change
> >> > > > > > > > > > > >> >
> (such
> >> as
> >> > what
> >> > > > > > > > > > > >> >
> >> >>
> >> > we
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > have proposed in
> >> > > > > > KIP-98), I
> >> > > > > > > think
> >> > > > > > > > they
> >> > > > > > > > > are
> >> > > > > > > > > > justifiable.
> >> > > > > > > > > > > If
> >> > > > > > > > > > > >> > we
> >> add them,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > then the need to
> >> > > > > > use attributes
> >> > > > > > > > becomes
> >> > > > > > > > > quite a
> >> > > > > > > > > > bit
> >> > > > > > > > > > > >>
> weaker,
> >> > > > > > > > > > > >> >
> >> right? The
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > other thing I
> >> > > > > > find slightly odd
> >> > > > > > > > is the
> >> > > > > > > > > fact that
> >> > > > > > > > > > null
> >> > > > > > > > > > > >> >
> >> headers has
> >> > no
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > actual
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > semantic meaning
> >> > > > > > for the message
> >> > > > > > > > > (unlike null
> >> > > > > > > > > > keys and
> >> > > > > > > > > > > >> >
> >> values). It
> >> > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > just
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > a
> >> > > > space
> >> > > > > > optimization. It seems a
> >> > > > > > > > bit
> >> > > > > > > > > better to
> >> > > > > > > > > > always
> >> > > > > > > > > > > use
> >> > > > > > > > > > > >> >
> size
> >> 0 to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > indicate having
> >> > > > > > no headers.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > Overall, the
> >> > > > > main
> >> > > > > > point is
> >> > > > > > > > ensuring
> >> > > > > > > > > that the
> >> > > > > > > > > > message
> >> > > > > > > > > > > >>
> schema
> >> > > > > > > > > > > >> >
> >> remains
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > consistent,
> >> > > > > > either within the
> >> > > > > > > > larger
> >> > > > > > > > > protocol, or
> >> > > > > > > > > > at a
> >> > > > > > > > > > > >> >
> >> minimum
> >> > within
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > message itself.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > -Jason
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > On Thu, Feb 16,
> >> > > > > > 2017 at 6:39 AM,
> >> > > > > > > > > Michael Pearce <
> >> > > > > > > > > > > >> >
> >> > > > Michael.Pearce@ig.com
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Hi Jason,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On point 1) in
> >> > > > > > the message
> >> > > > > > > > protocol
> >> > > > > > > > > the headers
> >> > > > > > > > > > are
> >> > > > > > > > > > > >>
> simply
> >> > > > > > > > > > > >> > a
> byte
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > array,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > as like the
> >> > > > > key
> >> > > > > > or value, this
> >> > > > > > > > is to
> >> > > > > > > > > clearly
> >> > > > > > > > > > demarcate
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> >> header in
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > core message.
> >> > > > > > Then the header
> >> > > > > > > > byte
> >> > > > > > > > > array in the
> >> > > > > > > > > > core
> >> > > > > > > > > > > >> >
> >> message is
> >> > an
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > array of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > key, value
> >> > > > > > pairs. This is what
> >> > > > > > > > it is
> >> > > > > > > > > denoting.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Then this
> >> > > > > would
> >> > > > > > be I guess in
> >> > > > > > > > the
> >> > > > > > > > > given
> >> > > > > > > > > > notation:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Headers =>
> >> > > > > > [KeyLength, Key,
> >> > > > > > > > > ValueLength, Value]
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > KeyLength
> >> > > > > > => int32
> >> > > > > > > > > <-----------------NEW
> >> > > > > > > > > > size of
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> byte[] of
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > serialised key
> >> > > > > > value
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Key =>
> >> > > > > bytes
> >> > > > > > > > > <----------------------
> NEW
> >> > > > > > > > > > >
> serialised
> >> > > > > > > > > > > >> >
> string
> >> > (UTF8)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > bytes of the
> >> > > > > > header key
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > ValueLength
> >> > > > > > => int32
> >> > > > > > > > > <-------------- NEW
> >> > > > > > > > > > size of
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> byte[] of
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > serialised
> >> > > > > > header value
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Value =>
> >> > > > > > bytes
> >> > > > > > > > > <--------------------
> NEW
> >> > > > > > > > > > >
> serialised
> >> > > > > > > > > > > >> >
> form
> >> of the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > header
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > value
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > The key length
> >> > > > > > and value
> >> > > > > > > length
> >> > > > > > > > is
> >> > > > > > > > > matching the
> >> > > > > > > > > > way
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> protocol is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > defined in the
> >> > > > > > core message
> >> > > > > > > > currently.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On point 2)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Var sized
> >> > > > > ints,
> >> > > > > > this was
> >> > > > > > > > discussed
> >> > > > > > > > > much earlier
> >> > > > > > > > > > on, in
> >> > > > > > > > > > > >> >
> fact
> >> I had
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > suggested it
> >> > > > > > myself (with
> >> > > > > > > Hadoop
> >> > > > > > > > > references),
> >> > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> complexity
> >> > of
> >> > > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > compared to
> >> > > > > > having a simpler
> >> > > > > > > > protocol
> >> > > > > > > > > was
> >> > > > > > > > > > argued and
> >> > > > > > > > > > > >> >
> >> agreed it
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > wasn’t worth
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the complexity
> >> > > > > > as all other
> >> > > > > > > > clients
> >> > > > > > > > > in other
> >> > > > > > > > > > languages
> >> > > > > > > > > > > >> >
> would
> >> need
> >> > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > ensure
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > theyre using
> >> > > > > > the right var
> >> > > > > > > size
> >> > > > > > > > > algorithm, as
> >> > > > > > > > > > there
> >> > > > > > > > > > > is a
> >> > > > > > > > > > > >> >
> few.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On point 3)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > We did the
> >> > > > > > attributes,
> >> > > > > > > optional
> >> > > > > > > > > approach as
> >> > > > > > > > > > originally
> >> > > > > > > > > > > >> >
> there
> >> was
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > marked
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > concern that
> >> > > > > > headers would
> >> > > > > > > > cause a
> >> > > > > > > > > message size
> >> > > > > > > > > > > overhead
> >> > > > > > > > > > > >> > for
> >> others,
> >> > > > > > > > > > > >> >
> >> >>
> >> > who
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > don’t want
> >> > > > > > them. As such this
> >> > > > > > > > is the
> >> > > > > > > > > clean
> >> > > > > > > > > > solution to
> >> > > > > > > > > > > >> >
> >> achieve
> >> > that.
> >> > > > > > > > > > > >> >
> >> >>
> >> > If
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > that no longer
> >> > > > > > holds, and we
> >> > > > > > > > don’t
> >> > > > > > > > > care that we
> >> > > > > > > > > > add
> >> > > > > > > > > > > >>
> 4bytes
> >> > > > > > > > > > > >> >
> >> overhead,
> >> > > > > > > > > > > >> >
> >> >>
> >> > > then
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > im happy to
> >> > > > > > remove.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > I’m personally
> >> > > > > > in favour of
> >> > > > > > > > keeping
> >> > > > > > > > > the message
> >> > > > > > > > > > as
> >> > > > > > > > > > > small
> >> > > > > > > > > > > >> > as
> >> possible
> >> > > > > > > > > > > >> >
> >> >>
> >> > so
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > people don’t
> >> > > > > > get shocks in
> >> > > > > > > perf
> >> > > > > > > > and
> >> > > > > > > > > throughputs
> >> > > > > > > > > > dues
> >> > > > > > > > > > > to
> >> > > > > > > > > > > >> >
> >> message
> >> > size,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > unless they
> >> > > > > > actively use the
> >> > > > > > > > feature,
> >> > > > > > > > > as such I
> >> > > > > > > > > > do
> >> > > > > > > > > > > >>
> prefer
> >> > > > > > > > > > > >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > attribute bit
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > wise feature
> >> > > > > > flag approach
> >> > > > > > > > myself.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On 16/02/2017,
> >> > > > > > 05:40, "Jason
> >> > > > > > > > > Gustafson" <
> >> > > > > > > > > > > >> >
> >> > > > jason@confluent.io> wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > We have
> >> > > > > > proposed a few
> >> > > > > > > > > significant changes
> >> > > > > > > > > > to the
> >> > > > > > > > > > > >> >
> >> message
> >> > format
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > KIP-98
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > which now
> >> > > > > > seems likely to
> >> > > > > > > > pass
> >> > > > > > > > > (perhaps
> >> > > > > > > > > > with some
> >> > > > > > > > > > > >> >
> >> iterations
> >> > on
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > implementation details).
> >> > > > > > > It
> >> > > > > > > > would
> >> > > > > > > > > be good
> >> > > > > > > > > > to try
> >> > > > > > > > > > > and
> >> > > > > > > > > > > >> >
> >> coordinate
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > changes
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > in both of
> >> > > > > > the proposals
> >> > > > > > > to
> >> > > > > > > > make
> >> > > > > > > > > sure they
> >> > > > > > > > > > are
> >> > > > > > > > > > > >> >
> >> consistent
> >> > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > compatible.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > I think
> >> > > > > > using the
> >> > > > > > > > attributes to
> >> > > > > > > > > indicate
> >> > > > > > > > > > null
> >> > > > > > > > > > > >>
> headers
> >> > > > > > > > > > > >> > is
> a
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > reasonable
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > approach.
> >> > > > > > We have proposed
> >> > > > > > > > to do
> >> > > > > > > > > the same
> >> > > > > > > > > > thing
> >> > > > > > > > > > > for
> >> > > > > > > > > > > >> > the
> >> message
> >> > > > > > > > > > > >> >
> >> >>
> >> > key
> >> > > > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > value.
> >> > > > > That
> >> > > > > > said, I
> >> > > > > > > > sympathize
> >> > > > > > > > > with Jay's
> >> > > > > > > > > > >
> argument.
> >> > > > > > > > > > > >> >
> Having
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > multiple
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > ways to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > specify a
> >> > > > > > null value
> >> > > > > > > > increases
> >> > > > > > > > > the overall
> >> > > > > > > > > > > >>
> >> complexity
> >> > > > > > > > > > > >> > of
> the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > protocol.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > You
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > can see
> >> > > > > > this just from the
> >> > > > > > > > fact
> >> > > > > > > > > that you
> >> > > > > > > > > > need the
> >> > > > > > > > > > > >> >
> extra
> >> > verbiage
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > protocol
> >> > > > > > specification in
> >> > > > > > > > this
> >> > > > > > > > > KIP and in
> >> > > > > > > > > > KIP-98
> >> > > > > > > > > > > to
> >> > > > > > > > > > > >> >
> >> describe the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > dependence
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > between
> >> > > > > the
> >> > > > > > fields and the
> >> > > > > > > > > attributes. It
> >> > > > > > > > > > seems
> >> > > > > > > > > > > >> like
> a
> >> > > > > > > > > > > >> >
> >> slippery
> >> > > > > > > > > > > >> >
> >> >>
> >> > > slope
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > if
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > you start
> >> > > > > > allowing
> >> > > > > > > different
> >> > > > > > > > > request types
> >> > > > > > > > > > to
> >> > > > > > > > > > > >> >
> >> implement
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > specification differently.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > You can
> >> > > > > > also argue that
> >> > > > > > > the
> >> > > > > > > > > messages
> >> > > > > > > > > > already are
> >> > > > > > > > > > > and
> >> > > > > > > > > > > >> > are
> >> likely
> >> > > > > > > > > > > >> >
> >> >>
> >> > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > remain a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > special
> >> > > > > > case. For example,
> >> > > > > > > > there
> >> > > > > > > > > is
> >> > > > > > > > > > currently no
> >> > > > > > > > > > > >> >
> >> generality
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > how
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > compressed
> >> > > > > > message sets
> >> > > > > > > are
> >> > > > > > > > > represented
> >> > > > > > > > > > that would
> >> > > > > > > > > > > >> be
> >> > > > > > > > > > > >> >
> >> applicable
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > other
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > request
> >> > > > > > types. Some might
> >> > > > > > > > see this
> >> > > > > > > > > > divergence as
> >> > > > > > > > > > > an
> >> > > > > > > > > > > >> >
> >> unfortunate
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > deficiency
> >> > > > > > which should be
> >> > > > > > > > fixed;
> >> > > > > > > > > others
> >> > > > > > > > > > might see
> >> > > > > > > > > > > >> it
> >> > > > > > > > > > > >> > as
> >> sort of
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > inevitability of needing
> >> > > > > > > to
> >> > > > > > > > > optimize where
> >> > > > > > > > > > it
> >> > > > > > > > > > > counts
> >> > > > > > > > > > > >> >
> most.
> >> I'm
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > probably
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > somewhere
> >> > > > > > in between, but
> >> > > > > > > I
> >> > > > > > > > think
> >> > > > > > > > > we
> >> > > > > > > > > > probably all
> >> > > > > > > > > > > >> >
> share
> >> the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > intuition
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > > protocol should be
> >> > > > > > > kept
> >> > > > > > > > as
> >> > > > > > > > > consistent as
> >> > > > > > > > > > > >>
> >> possible.
> >> > > > > > > > > > > >> >
> With
> >> that
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > mind,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > here are a
> >> > > > > > few comments:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > 1. One
> >> > > > > > thing I found a
> >> > > > > > > > little odd
> >> > > > > > > > > when
> >> > > > > > > > > > reading the
> >> > > > > > > > > > > >> >
> >> current
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > proposal is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > headers
> >> > > > > > are both
> >> > > > > > > > represented
> >> > > > > > > > > as an
> >> > > > > > > > > > array of
> >> > > > > > > > > > > >> bytes
> >> > > > > > > > > > > >> > and
> >> as an
> >> > > > > > > > > > > >> >
> >> >>
> >> > > array
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > key/value
> >> > > > > > pairs. I'd
> >> > > > > > > > probably
> >> > > > > > > > > suggest
> >> > > > > > > > > > something
> >> > > > > > > > > > > like
> >> > > > > > > > > > > >> >
> this:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Headers =>
> >> > > > > > [HeaderKey
> >> > > > > > > > HeaderValue]
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > HeaderKey
> >> > > > > > => String
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > HeaderValue => Bytes
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > An array
> >> > > > > in
> >> > > > > > the Kafka
> >> > > > > > > > protocol is
> >> > > > > > > > > > represented
> as a
> >> > > > > > > > > > > >> >
> 4-byte
> >> > integer
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > indicating
> >> > > > > > the number of
> >> > > > > > > > elements
> >> > > > > > > > > in the
> >> > > > > > > > > > array
> >> > > > > > > > > > > >> >
> >> followed by
> >> > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > serialization of the
> >> > > > > > > > elements.
> >> > > > > > > > > Unless I'm
> >> > > > > > > > > > > >> >
> >> > > > misunderstanding, what
> >> > > > > > > > > > > >> >
> >> >>
> >> > you
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > have
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > instead is
> >> > > > > > the total size
> >> > > > > > > > of the
> >> > > > > > > > > headers in
> >> > > > > > > > > > bytes
> >> > > > > > > > > > > >> >
> >> followed by
> >> > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > elements.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > I'm not
> >> > > > > > sure I see any
> >> > > > > > > > reason for
> >> > > > > > > > > this
> >> > > > > > > > > > > >>
> >> inconsistency.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > 2. In
> >> > > > > > KIP-98, we've
> >> > > > > > > > introduced
> >> > > > > > > > > >
> variable-length
> >> > > > > > > > > > > >>
> integer
> >> > > > > > > > > > > >> >
> >> fields.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Effectively,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > we've
> >> > > > > > enriched (or
> >> > > > > > > > "complicated"
> >> > > > > > > > > as Jay
> >> > > > > > > > > > might say
> >> > > > > > > > > > > ;)
> >> > > > > > > > > > > >> > the
> >> protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > specification to include
> >> > > > > > > the
> >> > > > > > > > > following
> >> > > > > > > > > > types:
> >> > > > > > > > > > > >>
> VarInt,
> >> > > > > > > > > > > >> >
> >> VarLong,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > UnsignedVarInt and
> >> > > > > > > > > UnsignedVarLong.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Along with
> >> > > > > > these
> >> > > > > > > > primitives, we
> >> > > > > > > > > could
> >> > > > > > > > > > introduce
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> following
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > types:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > VarSizeArray =>
> >> > > > > > > > NumberOfItems
> >> > > > > > > > > Item1 Item2
> >> > > > > > > > > > .. ItemN
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > NumberOfItems =>
> >> > > > > > > > UnsignedVarInt
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > VarSizeNullableArray =>
> >> > > > > > > > > NumberOfItemsOrNull
> >> > > > > > > > > > Item1
> >> > > > > > > > > > > >> >
> Item2
> >> ..
> >> > ItemN
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > NumberOfItemsOrNull =>
> >> > > > > > > > VarInt
> >> > > > > > > > > (-1 means
> >> > > > > > > > > > null)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > And
> >> > > > > > similarly for the
> >> > > > > > > > `String`
> >> > > > > > > > > and `Bytes`
> >> > > > > > > > > > types.
> >> > > > > > > > > > > >> >
> These
> >> types
> >> > > > > > > > > > > >> >
> >> >>
> >> > can
> >> > > > save
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > considerable amount of
> >> > > > > > > > space in
> >> > > > > > > > > this
> >> > > > > > > > > > proposal
> >> > > > > > > > > > > >>
> because
> >> > > > > > > > > > > >> >
> they
> >> can
> >> > > > > > > > > > > >> >
> >> >>
> >> > be
> >> > > > used
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > both the
> >> > > > > > number of headers
> >> > > > > > > > > included in the
> >> > > > > > > > > > message
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > lengths of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > header
> >> > > > > keys
> >> > > > > > and values. We
> >> > > > > > > > could
> >> > > > > > > > > do this
> >> > > > > > > > > > instead:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Headers =>
> >> > > > > > > > VarSizeArray[HeaderKey
> >> > > > > > > > > > HeaderValue]
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > HeaderKey
> >> > > > > > =>
> >> > > > > > > VarSizeString
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > HeaderValue =>
> >> > > > > > > > VarSizeBytes
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Combining
> >> > > > > > the savings from
> >> > > > > > > > the
> >> > > > > > > > > use of
> >> > > > > > > > > > variable
> >> > > > > > > > > > > >>
> length
> >> > > > > > > > > > > >> >
> >> fields, the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > benefit
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > of using
> >> > > > > > the attributes to
> >> > > > > > > > > represent null
> >> > > > > > > > > > seems
> >> > > > > > > > > > > >>
> pretty
> >> > > > > > > > > > > >> >
> small.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > 3.
> >> > > > > > Whichever way we go
> >> > > > > > > > (whether
> >> > > > > > > > > we use the
> >> > > > > > > > > > > >>
> >> attributes
> >> > > > > > > > > > > >> > or
> >> not), we
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > should at
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > least be
> >> > > > > > consistent
> >> > > > > > > between
> >> > > > > > > > this
> >> > > > > > > > > KIP and
> >> > > > > > > > > > KIP-98.
> >> > > > > > > > > > > It
> >> > > > > > > > > > > >> >
> would
> >> be
> >> > very
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > strange
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > to have
> >> > > > > two
> >> > > > > > ways to
> >> > > > > > > > represent
> >> > > > > > > > > null values
> >> > > > > > > > > > in the
> >> > > > > > > > > > > >> same
> >> > > > > > > > > > > >> >
> >> schema.
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > Either
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > way is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > OK with
> >> > > > > me.
> >> > > > > > I think some
> >> > > > > > > > > message-level
> >> > > > > > > > > > >
> >> optimizations
> >> > > > > > > > > > > >> > are
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > justifiable,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > but
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > savings
> >> > > > > > here seem
> >> > > > > > > > minimal (a
> >> > > > > > > > > few bytes
> >> > > > > > > > > > per
> >> > > > > > > > > > > >> >
> >> message), so
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > maybe it's
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > not
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > worth the
> >> > > > > > cost of letting
> >> > > > > > > > the
> >> > > > > > > > > message
> >> > > > > > > > > > diverge even
> >> > > > > > > > > > > >> >
> >> further from
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > rest of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > > protocol.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > -Jason
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > On Wed,
> >> > > > > Feb
> >> > > > > > 15, 2017 at
> >> > > > > > > > 8:52 AM,
> >> > > > > > > > > radai <
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > radai.rosenblatt@gmail.com>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > I've
> >> > > > > > trimmed the inline
> >> > > > > > > > > contents as this
> >> > > > > > > > > > mail is
> >> > > > > > > > > > > >> >
> >> getting too
> >> > > > > > > > > > > >> >
> >> >>
> >> > big
> >> > > > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > apache
> >> > > > > > mailing list
> >> > > > > > > > software to
> >> > > > > > > > > deliver
> >> > > > > > > > > > :-(
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 1. the
> >> > > > > > important thing
> >> > > > > > > for
> >> > > > > > > > > >
> interoperability
> >> is
> >> > > > > > > > > > > for
> >> > > > > > > > > > > >> >
> >> different
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > "interested
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > parties"
> >> > > > > > (plugins, infra
> >> > > > > > > > > layers/wrappers,
> >> > > > > > > > > > > >>
> >> user-code)
> >> > > > > > > > > > > >> > to
> be
> >> > > > > > > > > > > >> >
> >> >>
> >> > > able
> >> > > > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > stick
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > pieces
> >> > > > > of
> >> > > > > > metadata onto
> >> > > > > > > > msgs
> >> > > > > > > > > without
> >> > > > > > > > > > getting in
> >> > > > > > > > > > > >> each
> >> > > > > > > > > > > >> >
> >> other's
> >> > > > > > > > > > > >> >
> >> >>
> >> > > way.
> >> > > > a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > common
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > key
> >> > > > > > scheme (Strings, as
> >> > > > > > > > of the
> >> > > > > > > > > time of
> >> > > > > > > > > > this
> >> > > > > > > > > > > >> >
> >> writing?) is
> >> > > all
> >> > > > > > > > > > > >> >
> >> >>
> >> > > thats
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > required
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > for
> >> > > > > that.
> >> > > > > > it is assumed
> >> > > > > > > > that
> >> > > > > > > > > the other end
> >> > > > > > > > > > > >> >
> >> interested
> >> > in
> >> > > > any
> >> > > > > > > > > > > >> >
> >> >>
> >> > > such
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > piece of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > metadata
> >> > > > > > knows the
> >> > > > > > > > encoding,
> >> > > > > > > > > and byte[]
> >> > > > > > > > > > provides
> >> > > > > > > > > > > >> for
> >> > > > > > > > > > > >> > the
> >> most
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > flexibility.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > i
> >> > > > > believe
> >> > > > > > this is the
> >> > > > > > > same
> >> > > > > > > > > logic behind
> >> > > > > > > > > > core
> >> > > > > > > > > > > kafka
> >> > > > > > > > > > > >> >
> being
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > byte[]/byte[]
> >> > > > > -
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > Strings
> >> > > > > > are more
> >> > > > > > > "usable"
> >> > > > > > > > but
> >> > > > > > > > > bytes are
> >> > > > > > > > > > flexible
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> > so
> >> were
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > chosen.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > Also -
> >> > > > > > core kafka doesnt
> >> > > > > > > > even
> >> > > > > > > > > do that
> >> > > > > > > > > > good of a
> >> > > > > > > > > > > >> job
> >> > > > > > > > > > > >> > on
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > usability of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > payload
> >> > > > > > (example - i
> >> > > > > > > have
> >> > > > > > > > to
> >> > > > > > > > > specify the
> >> > > > > > > > > > nop
> >> > > > > > > > > > > >>
> byte[]
> >> > > > > > > > > > > >> >
> >> "decoders"
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > explicitly
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > in
> >> > > > > conf),
> >> > > > > > and again
> >> > > > > > > > sacrificies
> >> > > > > > > > > usability
> >> > > > > > > > > > for
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> sake
> >> of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > performance
> >> > > > > (no
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > convenient single-record
> >> > > > > > > > > processing as
> >> > > > > > > > > > poll is a
> >> > > > > > > > > > > >> >
> >> batch, lots
> >> > of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > obscure
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > little
> >> > > > > > config details
> >> > > > > > > > exposing
> >> > > > > > > > > internals
> >> > > > > > > > > > of the
> >> > > > > > > > > > > >> >
> >> batching
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > mechanism,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > etc)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > this is
> >> > > > > > also why i
> >> > > > > > > really
> >> > > > > > > > > dislike the
> >> > > > > > > > > > idea of a
> >> > > > > > > > > > > >> >
> "type
> >> > system"
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > header
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > values,
> >> > > > > > it further
> >> > > > > > > > degrades the
> >> > > > > > > > > > usability,
> adds
> >> > > > > > > > > > > >> >
> >> complexity
> >> > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > > will
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > eventually get in
> >> > > > > > > > people's way,
> >> > > > > > > > > also, it
> >> > > > > > > > > > would
> >> > > > > > > > > > > be
> >> > > > > > > > > > > >> > the
> >> 2nd/3rd
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > home-group
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > serialization mechanism
> >> > > > > > > > in core
> >> > > > > > > > > kafka
> >> > > > > > > > > > (counting
> >> > > > > > > > > > > 2
> >> > > > > > > > > > > >> >
> >> iterations
> >> > > > > > > > > > > >> >
> >> >>
> >> > of
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > "type
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > definition DSL")
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 2. this
> >> > > > > > is an
> >> > > > > > > > implementation
> >> > > > > > > > > detail, and
> >> > > > > > > > > > not
> >> > > > > > > > > > > even
> >> > > > > > > > > > > >> a
> >> > > > > > > > > > > >> >
> very
> >> "user
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > facing" one?
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > to the
> >> > > > > > best of my
> >> > > > > > > > understanding
> >> > > > > > > > > the vote
> >> > > > > > > > > > process
> >> > > > > > > > > > > >> is
> >> > > > > > > > > > > >> > on
> >> proposed
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > API/behaviour. also -
> >> > > > > > > > since
> >> > > > > > > > > we're willing
> >> > > > > > > > > > to go
> >> > > > > > > > > > > >> with
> >> > > > > > > > > > > >> >
> >> strings
> >> > > > > > > > > > > >> >
> >> >>
> >> > > just
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > serialize
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > a
> >> > > > > 0-sized
> >> > > > > > header blob
> >> > > > > > > and
> >> > > > > > > > IIUC
> >> > > > > > > > > you dont
> >> > > > > > > > > > need any
> >> > > > > > > > > > > >> >
> >> optionals
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > anymore.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 3. yes,
> >> > > > > > we can :-)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > On Tue,
> >> > > > > > Feb 14, 2017 at
> >> > > > > > > > 11:56
> >> > > > > > > > > PM, Michael
> >> > > > > > > > > > > Pearce <
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Hi
> >> > > > > Jay,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 1)
> >> > > > > > There was some
> >> > > > > > > > initial
> >> > > > > > > > > debate on the
> >> > > > > > > > > > value
> >> > > > > > > > > > > >> >
> part,
> >> as
> >> > youll
> >> > > > > > > > > > > >> >
> >> >>
> >> > > note
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > String,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > String
> >> > > > > > headers were
> >> > > > > > > > > discounted early
> >> > > > > > > > > > on. The
> >> > > > > > > > > > > >> >
> >> reason for
> >> > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > flexibility
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > and
> >> > > > > > keeping in line
> >> > > > > > > > with the
> >> > > > > > > > > > flexibility of
> >> > > > > > > > > > > key,
> >> > > > > > > > > > > >> >
> value
> >> of the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > message
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > object
> >> > > > > > itself. I don’t
> >> > > > > > > > think
> >> > > > > > > > > it takes
> >> > > > > > > > > > away
> >> > > > > > > > > > > from
> >> > > > > > > > > > > >> an
> >> > > > > > > > > > > >> >
> >> ecosystem
> >> > > > > > > > > > > >> >
> >> >>
> >> > as
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > each
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > plugin
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > will
> >> > > > > > care for their
> >> > > > > > > own
> >> > > > > > > > key,
> >> > > > > > > > > this way
> >> > > > > > > > > > ints,
> >> > > > > > > > > > > >> >
> >> booleans ,
> >> > > > exotic
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > custom
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > binary
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > can
> >> > > > > all
> >> > > > > > be catered
> >> > > > > > > for=.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a. If
> >> > > > > > you really
> >> > > > > > > wanted
> >> > > > > > > > to
> >> > > > > > > > > push for a
> >> > > > > > > > > > typed
> >> > > > > > > > > > > >> value
> >> > > > > > > > > > > >> >
> >> interface,
> >> > > > > > > > > > > >> >
> >> >> I
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > wouldn’t
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > want
> >> > > > > > just String
> >> > > > > > > values
> >> > > > > > > > > supported, but
> >> > > > > > > > > > the the
> >> > > > > > > > > > > >> >
> >> primatives
> >> > > > > > > > > > > >> >
> >> >>
> >> > > plus
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > string and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > also
> >> > > > > > still keeping the
> >> > > > > > > > > ability to have a
> >> > > > > > > > > > > binary
> >> > > > > > > > > > > >> > for
> >> custom
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > binaries that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > some
> >> > > > > > organisations may
> >> > > > > > > > have.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > i. I
> >> > > > > > have written this
> >> > > > > > > > slight
> >> > > > > > > > > > alternative
> >> > > > > > > > > > > here,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > https://cwiki.apache.org/
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > confluence/display/KAFKA/KIP-
> >> > > > > > > > > > > >> >
> >> > > > 82+-+Add+Record+Headers+-+
> >> > > > > Typed
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > ii.
> >> > > > > > Essentially the
> >> > > > > > > > value
> >> > > > > > > > > bytes, has a
> >> > > > > > > > > > leading
> >> > > > > > > > > > > >> >
> byte
> >> > overhead.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 1.
> >> > > > > > This tells you
> >> > > > > > > what
> >> > > > > > > > type
> >> > > > > > > > > the value
> >> > > > > > > > > > is,
> >> > > > > > > > > > > >>
> before
> >> > > > > > > > > > > >> >
> >> reading
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > rest
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > of the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > bytes,
> >> > > > > > allowing
> >> > > > > > > > > >
> >> > > > serialisation/deserialization
> >> > > > > > > > > > > to
> >> > > > > > > > > > > >> > and
> >> from the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > primitives,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > string
> >> > > > > > and byte[].
> >> > > > > > > This
> >> > > > > > > > is
> >> > > > > > > > > akin to some
> >> > > > > > > > > > other
> >> > > > > > > > > > > >> >
> >> messaging
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > systems.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 2) We
> >> > > > > > are making it
> >> > > > > > > > optional,
> >> > > > > > > > > so that
> >> > > > > > > > > > for
> >> > > > > > > > > > > those
> >> > > > > > > > > > > >> > not
> >> wanting
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > headers have
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 0
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > bytes
> >> > > > > > overhead (think
> >> > > > > > > > of it
> >> > > > > > > > > as a feature
> >> > > > > > > > > > > flag),
> >> > > > > > > > > > > >> I
> >> > > > > > > > > > > >> >
> don’t
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > think this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > complex, especially if
> >> > > > > > > > > comparing to
> >> > > > > > > > > > changes
> >> > > > > > > > > > > >> >
> >> proposed in
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > other kips
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > like
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > kip-98.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a. If
> >> > > > > > you really
> >> > > > > > > really
> >> > > > > > > > don’t
> >> > > > > > > > > like
> >> > > > > > > > > > this, we
> >> > > > > > > > > > > can
> >> > > > > > > > > > > >> >
> drop
> >> it, but
> >> > > > > > > > > > > >> >
> >> >>
> >> > it
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > would
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > mean
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > buying
> >> > > > > > into 4 bytes
> >> > > > > > > > extra
> >> > > > > > > > > overhead for
> >> > > > > > > > > > users
> >> > > > > > > > > > > who
> >> > > > > > > > > > > >> > do
> >> not want
> >> > > > > > > > > > > >> >
> >> >>
> >> > to
> >> > > > use
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > headers.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 3) In
> >> > > > > > the summary yes,
> >> > > > > > > > it is
> >> > > > > > > > > at a higher
> >> > > > > > > > > > > level,
> >> > > > > > > > > > > >> >
> but I
> >> think
> >> > > > > > > > > > > >> >
> >> >>
> >> > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > is well
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > documented in the
> >> > > > > > > > proposed
> >> > > > > > > > > changes
> >> > > > > > > > > > section.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a.
> >> > > > > > Added getHeaders
> >> > > > > > > > method to
> >> > > > > > > > > > >
> >> Producer/Consumer
> >> > > > > > > > > > > >> >
> >> record (that
> >> > > > > > > > > > > >> >
> >> >>
> >> > is
> >> > > > it)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > b.
> >> > > > > > We’ve also detailed
> >> > > > > > > > the
> >> > > > > > > > > new Headers
> >> > > > > > > > > > class
> >> > > > > > > > > > > >> that
> >> > > > > > > > > > > >> >
> this
> >> method
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > returns
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > encapsulates the
> >> > > > > > > headers
> >> > > > > > > > > protocol and
> >> > > > > > > > > > logic.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Best,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Mike
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > ==Original questions
> >> > > > > > > > from the
> >> > > > > > > > > vote
> >> > > > > > > > > > thread from
> >> > > > > > > > > > > >> >
> Jay.==
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Couple
> >> > > > > > of things I
> >> > > > > > > > think we
> >> > > > > > > > > still need
> >> > > > > > > > > > to work
> >> > > > > > > > > > > >> out:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 1.
> >> > > > > I
> >> > > > > > think we agree
> >> > > > > > > > about
> >> > > > > > > > > the key,
> >> > > > > > > > > > but I
> >> > > > > > > > > > > >> think
> >> > > > > > > > > > > >> > we
> >> haven't
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > talked about
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > the
> >> > > > > > value yet. I
> >> > > > > > > > think if
> >> > > > > > > > > our goal
> >> > > > > > > > > > is an
> >> > > > > > > > > > > open
> >> > > > > > > > > > > >> >
> >> ecosystem
> >> > > > > > > > > > > >> >
> >> >>
> >> > of
> >> > > > these
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > header
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > spread across many
> >> > > > > > > > plugins
> >> > > > > > > > > from many
> >> > > > > > > > > > > systems
> >> > > > > > > > > > > >> we
> >> > > > > > > > > > > >> >
> should
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > consider
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > making
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a
> >> > > > > > string as well so
> >> > > > > > > > it can
> >> > > > > > > > > be
> >> > > > > > > > > > printed, set
> >> > > > > > > > > > > >> via
> >> > > > > > > > > > > >> > a
> UI,
> >> set
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > config,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > etc.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > Basically
> >> > > > > > > encouraging
> >> > > > > > > > > pluggable
> >> > > > > > > > > > >
> >> serialization
> >> > > > > > > > > > > >> >
> >> formats
> >> > > > > > > > > > > >> >
> >> >>
> >> > > here
> >> > > > will
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > lead
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > bit
> >> > > > > > of a tower of
> >> > > > > > > > babel.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 2.
> >> > > > > > This proposal
> >> > > > > > > > still
> >> > > > > > > > > includes a
> >> > > > > > > > > > pretty
> >> > > > > > > > > > > big
> >> > > > > > > > > > > >> >
> >> change to
> >> > our
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > serialization
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > and
> >> > > > > > protocol
> >> > > > > > > > definition
> >> > > > > > > > > layer.
> >> > > > > > > > > > Essentially
> >> > > > > > > > > > > >> it is
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > introducing an
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > optional
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > type, where the
> >> > > > > > > > format is
> >> > > > > > > > > data
> >> > > > > > > > > > dependent. I
> >> > > > > > > > > > > >> >
> think
> >> this
> >> > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > actually a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > big
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > change though it
> >> > > > > > > > doesn't
> >> > > > > > > > > seem like
> >> > > > > > > > > > it. It
> >> > > > > > > > > > > >> means
> >> > > > > > > > > > > >> > you
> >> can no
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > longer
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > specify
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > this
> >> > > > > > type with our
> >> > > > > > > > type
> >> > > > > > > > > definition
> >> > > > > > > > > > DSL, and
> >> > > > > > > > > > > >> >
> >> likewise it
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > requires
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > custom
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > handling in client
> >> > > > > > > > libs.
> >> > > > > > > > > This isn't
> >> > > > > > > > > > a huge
> >> > > > > > > > > > > >> >
> >> thing, since
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Record
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > definition is
> >> > > > > > > custom
> >> > > > > > > > > anyway, but I
> >> > > > > > > > > > think
> >> > > > > > > > > > > this
> >> > > > > > > > > > > >> >
> kind
> >> of
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > inconsistency is
> >> > > > > > > very
> >> > > > > > > > > non-desirable
> >> > > > > > > > > > and
> >> > > > > > > > > > > ties
> >> > > > > > > > > > > >> >
> you to
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > hand-coding
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > things.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > I
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > think the type
> >> > > > > > > should
> >> > > > > > > > > instead by [Key
> >> > > > > > > > > > > Value]
> >> > > > > > > > > > > >> in
> >> > > > > > > > > > > >> > our
> >> BNF,
> >> > > > > > > > > > > >> >
> >> >>
> >> > > where
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > key and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > value are both
> >> > > > > > > short
> >> > > > > > > > > strings as used
> >> > > > > > > > > > > >>
> >> elsewhere.
> >> > > > > > > > > > > >> >
> This
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > brings it
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > in line
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > with
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > the
> >> > > > > > rest of the
> >> > > > > > > > protocol.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 3.
> >> > > > > > Could we get
> >> > > > > > > more
> >> > > > > > > > > specific about
> >> > > > > > > > > > the
> >> > > > > > > > > > > exact
> >> > > > > > > > > > > >> >
> Java
> >> API
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > change to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > ProducerRecord,
> >> > > > > > > > > ConsumerRecord,
> >> > > > > > > > > > Record,
> >> > > > > > > > > > > etc?
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > -Jay
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > The
> >> > > > > information
> >> > > > > > contained in
> >> > > > > > > > this
> >> > > > > > > > > email is
> >> > > > > > > > > > strictly
> >> > > > > > > > > > > >> >
> >> confidential
> >> > > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the use of the
> >> > > > > > addressee only,
> >> > > > > > > > unless
> >> > > > > > > > > otherwise
> >> > > > > > > > > > > >>
> >> indicated.
> >> > > > > > > > > > > >> > If
> you
> >> > > > > > > > > > > >> >
> >> >>
> >> > are
> >> > > > not
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the intended
> >> > > > > > recipient, please
> >> > > > > > > > do not
> >> > > > > > > > > read,
> >> > > > > > > > > > copy, use
> >> > > > > > > > > > > or
> >> > > > > > > > > > > >> >
> >> disclose to
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > others
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > this message
> >> > > > > or
> >> > > > > > any
> >> > > > > > > attachment.
> >> > > > > > > > > Please also
> >> > > > > > > > > > notify the
> >> > > > > > > > > > > >> >
> >> sender by
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > replying
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > to this email
> >> > > > > > or by telephone
> >> > > > > > > > > (+44(020 7896
> >> > > > > > > > > > 0011) and
> >> > > > > > > > > > > >> then
> >> > > > > > > > > > > >> >
> delete
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > email
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > and any copies
> >> > > > > > of it.
> >> > > > > > > Opinions,
> >> > > > > > > > > conclusion
> >> > > > > > > > > > (etc) that
> >> > > > > > > > > > > do
> >> > > > > > > > > > > >> > not
> >> relate
> >> > > > > > > > > > > >> >
> >> >>
> >> > to
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > official
> >> > > > > > business of this
> >> > > > > > > > company
> >> > > > > > > > > shall be
> >> > > > > > > > > > understood
> >> > > > > > > > > > > as
> >> > > > > > > > > > > >> >
> >> neither
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > given nor
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > endorsed by
> >> > > > > it.
> >> > > > > > IG is a
> >> > > > > > > trading
> >> > > > > > > > name
> >> > > > > > > > > of IG
> >> > > > > > > > > > Markets
> >> > > > > > > > > > > >>
> Limited
> >> > > > > > > > > > > >> > (a
> >> company
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > registered in
> >> > > > > > England and
> >> > > > > > > Wales,
> >> > > > > > > > > company number
> >> > > > > > > > > > > >>
> >> 04008957)
> >> > > > > > > > > > > >> >
> and IG
> >> > > > > > > > > > > >> >
> >> >>
> >> > > Index
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Limited (a
> >> > > > > > company registered
> >> > > > > > > in
> >> > > > > > > > > England and
> >> > > > > > > > > > Wales,
> >> > > > > > > > > > > >> >
> >> company
> >> > number
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > 01190902).
> >> > > > > > Registered address
> >> > > > > > > at
> >> > > > > > > > > Cannon Bridge
> >> > > > > > > > > > House,
> >> > > > > > > > > > > 25
> >> > > > > > > > > > > >> >
> >> Dowgate
> >> > > > > > > > > > > >> >
> >> >>
> >> > > Hill,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > London EC4R
> >> > > > > > 2YA. Both IG
> >> > > > > > > Markets
> >> > > > > > > > > Limited
> >> > > > > > > > > > (register
> >> > > > > > > > > > > >>
> number
> >> > > > > > > > > > > >> >
> >> 195355)
> >> > > > > > > > > > > >> >
> >> >>
> >> > and
> >> > > > IG
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Index Limited
> >> > > > > > (register number
> >> > > > > > > > > 114059) are
> >> > > > > > > > > > authorised
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> >
> >> regulated
> >> > > > > > > > > > > >> >
> >> >>
> >> > by
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Financial
> >> > > > > > Conduct Authority.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > --
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > Nacho - Ignacio
> >> > > > > > Solis -
> >> > > > > > > > isolis@igso.net
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> > The
> >> > > information
> >> > > > contained in
> >> > > > > > this email is
> >> > > > > > > > strictly
> >> > > > > > > > > > confidential
> and
> >> > > > > > > > > > > for
> >> > > > > > > > > > > >> > the
> >> use of
> >> > the
> >> > > > addressee only,
> >> > > > > > unless
> >> > > > > > > otherwise
> >> > > > > > > > > indicated. If
> >> > > > > > > > > > you are
> >> > > > > > > > > > > >> not
> >> > > > > > > > > > > >> > the
> >> intended
> >> > > > recipient, please
> >> > > > > > do not read,
> >> > > > > > > > copy, use
> >> > > > > > > > > or
> >> > > > > > > > > > disclose to
> >> > > > > > > > > > > >>
> others
> >> > > > > > > > > > > >> >
> this
> >> message
> >> > > or
> >> > > > any
> >> > > > > attachment.
> >> > > > > > Please also
> >> > > > > > > > notify
> >> > > > > > > > > the sender by
> >> > > > > > > > > > > >>
> replying
> >> > > > > > > > > > > >> > to
> >> this
> >> > email
> >> > > > or by telephone
> >> > > > > > (+44(020 7896
> >> > > > > > > > 0011) and
> >> > > > > > > > > then
> >> > > > > > > > > > delete the
> >> > > > > > > > > > > >> email
> >> > > > > > > > > > > >> > and
> >> any
> >> > copies
> >> > > > of it.
> >> > > > > Opinions,
> >> > > > > > conclusion
> >> > > > > > > > (etc) that
> >> > > > > > > > > do not
> >> > > > > > > > > > relate to
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> >> official
> >> > > > business of this
> >> > > > > > company shall be
> >> > > > > > > > understood
> >> > > > > > > > > as
> >> > > > > > > > > > neither given
> >> > > > > > > > > > > >> nor
> >> > > > > > > > > > > >> >
> >> endorsed by
> >> > > it.
> >> > > > IG is a
> >> > > > > trading
> >> > > > > > name of IG
> >> > > > > > > > Markets
> >> > > > > > > > > Limited (a
> >> > > > > > > > > > company
> >> > > > > > > > > > > >> >
> >> registered
> >> > in
> >> > > > England and
> >> > > > > > Wales, company
> >> > > > > > > number
> >> > > > > > > > > 04008957) and
> >> > > > > > > > > > IG Index
> >> > > > > > > > > > > >> >
> >> Limited (a
> >> > > > company registered
> >> > > > > > in England and
> >> > > > > > > > Wales,
> >> > > > > > > > > company
> >> > > > > > > > > > number
> >> > > > > > > > > > > >> >
> >> 01190902).
> >> > > > Registered address
> >> > > > > > at Cannon Bridge
> >> > > > > > > > House,
> >> > > > > > > > > 25
> >> > > > > > > > > > Dowgate Hill,
> >> > > > > > > > > > > >> >
> >> London EC4R
> >> > > > 2YA. Both IG
> >> > > > > > Markets Limited
> >> > > > > > > > (register
> >> > > > > > > > > number
> >> > > > > > > > > > 195355) and
> >> > > > > > > > > > > IG
> >> > > > > > > > > > > >> >
> Index
> >> > Limited
> >> > > > (register number
> >> > > > > > 114059) are
> >> > > > > > > > authorised
> >> > > > > > > > > and
> >> > > > > > > > > > regulated by
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> >> Financial
> >> > > > Conduct Authority.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> The
> >> > information
> >> > > > contained in
> >> > > > > this
> >> > > > > > email is
> >> > > > > > > > strictly
> >> > > > > > > > > confidential
> >> > > > > > > > > > and for
> >> > > > > > > > > > > >> the
> use
> >> of the
> >> > > > addressee only,
> >> > > > > > unless otherwise
> >> > > > > > > > > indicated. If you
> >> > > > > > > > > > are
> >> > > > > > > > > > > not
> >> > > > > > > > > > > >> the
> >> intended
> >> > > > recipient, please
> >> > > > > do
> >> > > > > > not read,
> >> > > > > > > copy,
> >> > > > > > > > use
> >> > > > > > > > > or disclose
> >> > > > > > > > > > to
> >> > > > > > > > > > > others
> >> > > > > > > > > > > >> this
> >> message
> >> > or
> >> > > > any attachment.
> >> > > > > > Please also
> >> > > > > > > > notify the
> >> > > > > > > > > sender by
> >> > > > > > > > > > > replying
> >> > > > > > > > > > > >> to
> this
> >> email
> >> > or
> >> > > > by telephone
> >> > > > > > (+44(020 7896
> >> > > > > > > 0011)
> >> > > > > > > > and
> >> > > > > > > > > then delete
> >> > > > > > > > > > the
> >> > > > > > > > > > > email
> >> > > > > > > > > > > >> and
> any
> >> copies
> >> > > of
> >> > > > it. Opinions,
> >> > > > > > conclusion (etc)
> >> > > > > > > > that
> >> > > > > > > > > do not
> >> > > > > > > > > > relate to
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >>
> official
> >> > > business
> >> > > > of this
> >> > > > > company
> >> > > > > > shall be
> >> > > > > > > > understood
> >> > > > > > > > > as neither
> >> > > > > > > > > > given
> >> > > > > > > > > > > nor
> >> > > > > > > > > > > >>
> >> endorsed by
> >> > it.
> >> > > > IG is a trading
> >> > > > > > name of IG
> >> > > > > > > Markets
> >> > > > > > > > > Limited (a
> >> > > > > > > > > > company
> >> > > > > > > > > > > >>
> >> registered in
> >> > > > England and Wales,
> >> > > > > > company number
> >> > > > > > > > > 04008957) and IG
> >> > > > > > > > > > Index
> >> > > > > > > > > > > >>
> Limited
> >> (a
> >> > > > company registered in
> >> > > > > > England and
> >> > > > > > > > Wales,
> >> > > > > > > > > company number
> >> > > > > > > > > > > >>
> >> 01190902).
> >> > > > Registered address at
> >> > > > > > Cannon Bridge
> >> > > > > > > > House,
> >> > > > > > > > > 25 Dowgate
> >> > > > > > > > > > Hill,
> >> > > > > > > > > > > >>
> London
> >> EC4R
> >> > 2YA.
> >> > > > Both IG Markets
> >> > > > > > Limited
> >> > > > > > > (register
> >> > > > > > > > > number 195355)
> >> > > > > > > > > > and IG
> >> > > > > > > > > > > >> Index
> >> Limited
> >> > > > (register number
> >> > > > > > 114059) are
> >> > > > > > > > authorised
> >> > > > > > > > > and
> >> > > > > > > > > > regulated by
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >>
> >> Financial
> >> > > Conduct
> >> > > > Authority.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > The
> >> information
> >> > > > contained in this
> >> > > > > > email is strictly
> >> > > > > > > > > confidential and
> >> > > > > > > > > > for the use of
> >> the
> >> > > > addressee only, unless
> >> > > > > > otherwise
> >> > > > > > > > indicated.
> >> > > > > > > > > If you are
> >> > > > > > > > > > not the
> intended
> >> > > > recipient, please do not
> >> > > > > > read, copy, use
> >> > > > > > > > or
> >> > > > > > > > > disclose to
> >> > > > > > > > > > others this
> >> message or
> >> > > any
> >> > > > attachment.
> >> > > > > > Please also notify
> >> > > > > > > > the
> >> > > > > > > > > sender by
> >> > > > > > > > > > replying to
> this
> >> email
> >> > or
> >> > > > by telephone
> >> > > > > > (+44(020 7896
> >> > > > > > > 0011)
> >> > > > > > > > and
> >> > > > > > > > > then delete
> >> > > > > > > > > > the email and
> any
> >> > copies
> >> > > > of it. Opinions,
> >> > > > > > conclusion
> >> > > > > > > (etc)
> >> > > > > > > > that
> >> > > > > > > > > do not
> >> > > > > > > > > > relate to the
> >> official
> >> > > > business of this
> >> > > > > > company shall be
> >> > > > > > > > > understood as
> >> > > > > > > > > > neither given
> nor
> >> > > endorsed
> >> > > > by it. IG is a
> >> > > > > > trading name of
> >> > > > > > > > IG
> >> > > > > > > > > Markets
> >> > > > > > > > > > Limited (a
> >> company
> >> > > > registered in England
> >> > > > > > and Wales,
> >> > > > > > > company
> >> > > > > > > > > number
> >> > > > > > > > > > 04008957) and
> IG
> >> Index
> >> > > > Limited (a company
> >> > > > > > registered in
> >> > > > > > > > England
> >> > > > > > > > > and Wales,
> >> > > > > > > > > > company number
> >> > 01190902).
> >> > > > Registered
> >> > > > > > address at Cannon
> >> > > > > > > > Bridge
> >> > > > > > > > > House, 25
> >> > > > > > > > > > Dowgate Hill,
> >> London
> >> > EC4R
> >> > > > 2YA. Both IG
> >> > > > > > Markets Limited
> >> > > > > > > > (register
> >> > > > > > > > > number
> >> > > > > > > > > > 195355) and IG
> >> Index
> >> > > > Limited (register
> >> > > > > > number 114059) are
> >> > > > > > > > > authorised and
> >> > > > > > > > > > regulated by
> the
> >> > > Financial
> >> > > > Conduct
> >> > > > > > Authority.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > The information
> >> contained in
> >> > > > this email is
> >> > > > > > strictly
> >> > > > > > > confidential
> >> > > > > > > > and
> >> > > > > > > > > for the use of the
> >> addressee
> >> > only,
> >> > > > unless otherwise
> >> > > > > > indicated. If
> >> > > > > > > > you are
> >> > > > > > > > > not the intended
> recipient,
> >> > please
> >> > > > do not read,
> >> > > > > > copy, use or
> >> > > > > > > > disclose to
> >> > > > > > > > > others this message or
> any
> >> > > > attachment. Please also
> >> > > > > > notify the
> >> > > > > > > sender
> >> > > > > > > > by
> >> > > > > > > > > replying to this email
> or
> >> by
> >> > > > telephone (+44(020
> >> > > > > 7896
> >> > > > > > 0011) and then
> >> > > > > > > > delete
> >> > > > > > > > > the email and any copies
> >> of it.
> >> > > > Opinions,
> >> > > > > conclusion
> >> > > > > > (etc) that do
> >> > > > > > > > not
> >> > > > > > > > > relate to the official
> >> business
> >> > of
> >> > > > this company
> >> > > > > > shall be understood
> >> > > > > > > > as
> >> > > > > > > > > neither given nor
> endorsed
> >> by it.
> >> > > IG
> >> > > > is a trading
> >> > > > > > name of IG
> >> > > > > > > Markets
> >> > > > > > > > > Limited (a company
> >> registered in
> >> > > > England and Wales,
> >> > > > > > company number
> >> > > > > > > > > 04008957) and IG Index
> >> Limited (a
> >> > > > company
> >> > > > > registered
> >> > > > > > in England and
> >> > > > > > > > Wales,
> >> > > > > > > > > company number
> 01190902).
> >> > > Registered
> >> > > > address at
> >> > > > > > Cannon Bridge
> >> > > > > > > House,
> >> > > > > > > > 25
> >> > > > > > > > > Dowgate Hill, London
> EC4R
> >> 2YA.
> >> > Both
> >> > > > IG Markets
> >> > > > > > Limited (register
> >> > > > > > > > number
> >> > > > > > > > > 195355) and IG Index
> >> Limited
> >> > > > (register number
> >> > > > > > 114059) are
> >> > > > > > > authorised
> >> > > > > > > > and
> >> > > > > > > > > regulated by the
> Financial
> >> > Conduct
> >> > > > Authority.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > The information contained in
> >> this email
> >> > > is
> >> > > > strictly
> >> > > > > > confidential and for
> >> > > > > > > > the use of the addressee only,
> >> unless
> >> > > > otherwise
> >> > > > > indicated.
> >> > > > > > If you are not
> >> > > > > > > > the intended recipient, please
> >> do not
> >> > > > read, copy, use or
> >> > > > > > disclose to
> >> > > > > > > others
> >> > > > > > > > this message or any
> attachment.
> >> Please
> >> > > > also notify the
> >> > > > > > sender by replying
> >> > > > > > > > to this email or by telephone
> >> (+44(020
> >> > > > 7896 0011) and
> >> > > > > then
> >> > > > > > delete the
> >> > > > > > > email
> >> > > > > > > > and any copies of it.
> Opinions,
> >> > > conclusion
> >> > > > (etc) that do
> >> > > > > > not relate to
> >> > > > > > > the
> >> > > > > > > > official business of this
> >> company shall
> >> > > be
> >> > > > understood as
> >> > > > > > neither given
> >> > > > > > > nor
> >> > > > > > > > endorsed by it. IG is a
> trading
> >> name of
> >> > > IG
> >> > > > Markets
> >> > > > > Limited
> >> > > > > > (a company
> >> > > > > > > > registered in England and
> Wales,
> >> > company
> >> > > > number 04008957)
> >> > > > > > and IG Index
> >> > > > > > > > Limited (a company registered
> in
> >> > England
> >> > > > and Wales,
> >> > > > > > company number
> >> > > > > > > > 01190902). Registered address
> at
> >> Cannon
> >> > > > Bridge House, 25
> >> > > > > > Dowgate Hill,
> >> > > > > > > > London EC4R 2YA. Both IG
> Markets
> >> > Limited
> >> > > > (register number
> >> > > > > > 195355) and IG
> >> > > > > > > > Index Limited (register number
> >> 114059)
> >> > > are
> >> > > > authorised and
> >> > > > > > regulated by
> >> > > > > > > the
> >> > > > > > > > Financial Conduct Authority.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > The information contained in this email is
> >> strictly
> >> > > > confidential and
> >> > > > > > for the use of the addressee only, unless
> >> otherwise
> >> > > > indicated. If you are
> >> > > > > > not the intended recipient, please do not
> read,
> >> copy,
> >> > use
> >> > > > or disclose to
> >> > > > > > others this message or any attachment. Please
> >> also
> >> > notify
> >> > > > the sender by
> >> > > > > > replying to this email or by telephone
> (+44(020
> >> 7896
> >> > > 0011)
> >> > > > and then
> >> > > > > delete
> >> > > > > > the email and any copies of it. Opinions,
> >> conclusion
> >> > > (etc)
> >> > > > that do not
> >> > > > > > relate to the official business of this
> company
> >> shall
> >> > be
> >> > > > understood as
> >> > > > > > neither given nor endorsed by it. IG is a
> >> trading name
> >> > of
> >> > > > IG Markets
> >> > > > > > Limited (a company registered in England and
> >> Wales,
> >> > > > company number
> >> > > > > > 04008957) and IG Index Limited (a company
> >> registered in
> >> > > > England and
> >> > > > > Wales,
> >> > > > > > company number 01190902). Registered address
> at
> >> Cannon
> >> > > > Bridge House, 25
> >> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> >> Limited
> >> > > > (register number
> >> > > > > > 195355) and IG Index Limited (register number
> >> 114059)
> >> > are
> >> > > > authorised and
> >> > > > > > regulated by the Financial Conduct Authority.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > The information contained in this email is strictly
> confidential
> >> and
> >> > for
> >> > > > the use of the addressee only, unless otherwise indicated. If
> >> you are
> >> > not
> >> > > > the intended recipient, please do not read, copy, use or
> >> disclose to
> >> > > others
> >> > > > this message or any attachment. Please also notify the sender
> by
> >> > replying
> >> > > > to this email or by telephone (+44(020 7896 0011) and then
> >> delete the
> >> > > email
> >> > > > and any copies of it. Opinions, conclusion (etc) that do not
> >> relate to
> >> > > the
> >> > > > official business of this company shall be understood as
> neither
> >> given
> >> > > nor
> >> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> >> company
> >> > > > registered in England and Wales, company number 04008957) and
> IG
> >> Index
> >> > > > Limited (a company registered in England and Wales, company
> >> number
> >> > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >> Hill,
> >> > > > London EC4R 2YA. Both IG Markets Limited (register number
> >> 195355) and
> >> > IG
> >> > > > Index Limited (register number 114059) are authorised and
> >> regulated by
> >> > > the
> >> > > > Financial Conduct Authority.
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> The information contained in this email is strictly confidential and for
> >> the use of the addressee only, unless otherwise indicated. If you are
> not
> >> the intended recipient, please do not read, copy, use or disclose to
> others
> >> this message or any attachment. Please also notify the sender by
> replying
> >> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >> official business of this company shall be understood as neither given
> nor
> >> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> registered in England and Wales, company number 04008957) and IG Index
> >> Limited (a company registered in England and Wales, company number
> >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> >> Index Limited (register number 114059) are authorised and regulated by
> the
> >> Financial Conduct Authority.
> >>
> > The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
If we go the mutable route (not totally sold on this yet), letting Header
implement Map.Entry<String, byte[]> may be reasonable. I discussed this
with Michael offline. You can then use the following constructor for
ProducerRecord.
ProducerRecord(K key, V value, Iterable<? extends Map.Entry<String,
byte[]>> headers);
That allows you to pass a Headers instance directly without modification or
to construct them from a map and use map.entries(). I think it would be
wise to use classes like Headers and Header regardless in case we need to
add anything in the future.
-Jason
On Thu, Feb 23, 2017 at 10:48 AM, Sean McCauliff <se...@gmail.com>
wrote:
> Instead of making a Header type visible in the producer or consumer
> record, the API could use Iterator<Map.Entry<String, byte[]>>. This
> would make the API more compatible with the multimap implementations
> I'm aware of (apache commons and google guava) should one want to use
> an existing implementation of a multimap data structure at a later
> date.
>
> Sean
>
> On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com>
> wrote:
> > I think I'd like some opinions from some of the active chaps at LinkedIn
> so it's not just a two way decision (or any others for that matter).
> >
> > Any opinions on the below or no real concerns / indifferent by anyone
> with this change?
> >
> > Cheers
> > Mike
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Tuesday, February 21, 2017 6:59:38 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > Jun's suggestion gives us the best of both worlds. Because the headers
> are
> > the last field in the record and you have the total size of the record,
> > then you can easily get the size of headers in bytes. So they can be
> copied
> > without parsing and we can do the lazy initialization without an extra
> cost.
> >
> > The message set notion has changed in KIP-98 so that the message set can
> > have its own fields. See here for more detail:
> > https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF
> 0wSw9ra8/edit#heading=h.pau6n6buvoee.
> > What we're proposing to change here is the schema for the message (aka
> > record) itself. We'll update that document depending on the outcome of
> the
> > discussion here.
> >
> > -Jason
> >
> >
> > On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> >> This works whilst no other feature needs similar. I prefer to keep it
> >> contained/encapsulated it then allows the bytes of the headers to also
> be
> >> copied transferred as one without any parsing what so ever, but if you
> >> really feel strongly on the below ill go with it, just want to flag this
> >> concern.
> >>
> >> Btw, the below notation seems to be different to the notation used in
> >> previous kips like kip-32 which I took the notation from, where the
> length
> >> of the following bytes for byte arrays is clearly defined. So im 100% on
> >> what you propose could you define it in this original styled notation?
> >>
> >>
> >> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
> >>
> >> Hey Jun, that's a good thought. Maybe then the full schema could
> look
> >> like
> >> this:
> >>
> >> Record =>
> >> Length => uintVar
> >> Attributes => int8
> >> TimestampDelta => intVar
> >> OffsetDelta => uintVar
> >> Key => bytes
> >> Value => bytes
> >> Headers => [HeaderKey HeaderValue]
> >> HeaderKey => string
> >> HeaderValue => bytes
> >>
> >> (where the bytes and string type use a variable length size field)
> >>
> >> Does that work for you, Michael?
> >>
> >> -Jason
> >>
> >> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
> >>
> >> > Regarding the overhead of skipping the headers in the message.
> >> Another way
> >> > could be to add the headers at the end of the message after the
> >> value part.
> >> > This way, there is no need to skip the headers unless you really
> >> need it.
> >> > This will allow us to represent the headers explicitly as an
> array of
> >> > key/value pairs without the skipping overhead when headers are not
> >> needed.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <
> jason@confluent.io
> >> >
> >> > wrote:
> >> >
> >> > > Michael,
> >> > >
> >> > > Here's a link to the branch with the message format changes:
> >> > > https://github.com/confluentinc/kafka/tree/
> >> exactly-once-message-format.
> >> > > The
> >> > > class to modify to add headers is EosLogRecord (name will
> >> definitely
> >> > change
> >> > > prior to merging in trunk). This branch uses attributes to
> >> indicate null
> >> > > key and value. I really don't have a strong preference for or
> >> against
> >> > this
> >> > > as long as we're consistent. I do feel it's kind of a pity to
> need
> >> the
> >> > > signed version of the varint just to be able to indicate null.
> For
> >> > headers,
> >> > > it's an easier call because we don't actually need to represent
> >> null
> >> > > headers (using size zero will work fine).
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
> >> Michael.Pearce@ig.com>
> >> > > wrote:
> >> > >
> >> > > > We can make it, implement Iterable<Header>, no concerns on
> this.
> >> > > >
> >> > > > Ive noted this on the KIP.
> >> > > >
> >> > > >
> >> > > > On 20/02/2017, 01:24, "Michael Pearce" <Michael.Pearce@ig.com
> >
> >> wrote:
> >> > > >
> >> > > > Btw Jason, can you share the branch with the VarInts
> >> helper/util
> >> > > class
> >> > > > that KIP-98 will be introducing and also will have the current
> >> modified
> >> > > > (key and value lengths) message, so we can look at, thanks.
> >> > > >
> >> > > > On 20/02/2017, 01:11, "Michael Pearce" <
> >> Michael.Pearce@ig.com>
> >> > > wrote:
> >> > > >
> >> > > > RE “ProducerRecord and ConsumerRecord are immutable
> >> > currently”. I
> >> > > > beg to differ on this, if I set a byte[] as my value of my
> >> record, I
> >> > can
> >> > > > later call value() and then alter the bytes in the array. As
> >> such it is
> >> > > > very much mutable.
> >> > > >
> >> > > > Headers also will anyhow by nature be mutable on
> >> produce, as
> >> > you
> >> > > > add to them.
> >> > > >
> >> > > >
> >> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
> >> jason@confluent.io>
> >> > > > wrote:
> >> > > >
> >> > > > Re; having a dedicated Headers object. I think it
> >> makes
> >> > > sense.
> >> > > > Maybe it
> >> > > > could implement Iterable<Header>?
> >> > > >
> >> > > > One thing I'm not too sure about is mutability.
> Both
> >> the
> >> > > > ProducerRecord and
> >> > > > ConsumerRecord are immutable currently. It would
> be
> >> nice to
> >> > > > keep it that
> >> > > > way. Perhaps Headers could itself be immutable and
> >> passed
> >> > > > through the
> >> > > > constructors?
> >> > > >
> >> > > > Just going through how this is, I don’t believe
> >> this would
> >> > > > provide much
> >> > > > > saving. If issue trying to resolve is around
> >> > object/garbage
> >> > > > creation of
> >> > > > > Map.Entry objects.
> >> > > >
> >> > > >
> >> > > > It's definitely better because we only allocate
> the
> >> space
> >> > we
> >> > > > will use, but
> >> > > > I assume most of the actual memory cost is in the
> >> header
> >> > data
> >> > > > itself which
> >> > > > we can't do much about. Maybe we can start with
> the
> >> simple
> >> > > > array
> >> > > > representation and optimize from there after we
> find
> >> it's
> >> > > > worthwhile to do
> >> > > > so? If we use the Headers object, we can change
> this
> >> later
> >> > > > without breaking
> >> > > > the API.
> >> > > >
> >> > > > -Jason
> >> > > >
> >> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> >> > > > radai.rosenblatt@gmail.com> wrote:
> >> > > >
> >> > > > > "(By the way, doesn't it feel a bit odd that we
> >> seem to
> >> > be
> >> > > > designing a
> >> > > > > feature which is optimized for people not
> using?)"
> >> > > > >
> >> > > > > very. this (i claim :-) ) is the result of very
> >> intense
> >> > > > opposition to the
> >> > > > > usefulness of the feature early on, and not a
> real
> >> design
> >> > > > goal
> >> > > > >
> >> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce
> <
> >> > > > Michael.Pearce@ig.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Whilst if having an array of a Header[] would
> >> mean
> >> > > getting
> >> > > > this through,
> >> > > > > > im happy changing this, as already done.
> >> > > > > >
> >> > > > > > Just going through how this is, I don’t
> believe
> >> this
> >> > > > would provide much
> >> > > > > > saving. If issue trying to resolve is around
> >> > > > object/garbage creation of
> >> > > > > > Map.Entry objects.
> >> > > > > >
> >> > > > > > All you’re doing here is replacing the equiv
> of a
> >> > > > HashMap.Node
> >> > > > > (Map.Entry)
> >> > > > > > which simply holds reference to key and value
> >> objects
> >> > > with
> >> > > > a custom
> >> > > > > variant.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
> >> > > > Michael.Pearce@ig.com> wrote:
> >> > > > > >
> >> > > > > > On point 1 & 2 Ive updated KIP to show
> >> varints (and
> >> > > > removed the bit
> >> > > > > > flag). (on the assumption KIP 98 is getting
> the
> >> > agreement
> >> > > > the protocol is
> >> > > > > > moving from int32 to varInts as standard)
> >> > > > > >
> >> > > > > > On point 3 ive updated to use an array of
> >> Header
> >> > > > class, instead of a
> >> > > > > > MultiMap in the Headers class object
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
> >> > > > Michael.Pearce@ig.com>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > On points 1 and 2 I agree.
> >> > > > > >
> >> > > > > > This also affects kip-98, I should
> >> expect this
> >> > > > resolved before
> >> > > > > > that vote also passes. If it is accepted there
> >> (I’m
> >> > > > assuming this is
> >> > > > > > getting discussed on that KIP? As you’re
> driving
> >> the
> >> > move
> >> > > > to VarInts), I
> >> > > > > am
> >> > > > > > happy to make this KIP will simply follow
> suit to
> >> > > whatever
> >> > > > is agreed in
> >> > > > > > KIP-98.
> >> > > > > >
> >> > > > > > On 3) Agreed this is a simpler form,
> as
> >> long as
> >> > > no
> >> > > > one is
> >> > > > > > expecting hashmap lookup performance ( O(1)
> ), I
> >> still
> >> > > > would prefer an
> >> > > > > > encapsulated class, so if we find that holding
> >> it as an
> >> > > > array in future
> >> > > > > is
> >> > > > > > causing some perf issues, the internals are
> not
> >> exposed
> >> > > to
> >> > > > end users,
> >> > > > > > allowing the internal structure to move to a
> map.
> >> > > > > >
> >> > > > > > Can we compromise on?
> >> > > > > >
> >> > > > > > class ConsumerRecord<K, V> {
> >> > > > > > K key;
> >> > > > > > V value;
> >> > > > > > Headers headers;
> >> > > > > > }
> >> > > > > >
> >> > > > > > class Headers {
> >> > > > > > Header[] headers;
> >> > > > > >
> >> > > > > > add(String key, byte[] value)
> >> > > > > > Collection<byte[]> get(String
> key)
> >> > > > > > }
> >> > > > > >
> >> > > > > > class Header {
> >> > > > > > String key;
> >> > > > > > byte[] value;
> >> > > > > > }
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > On 19/02/2017, 18:54, "Jason
> Gustafson" <
> >> > > > jason@confluent.io>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > >
> >> > > > > > > headers dont "leak" into
> >> application
> >> > code.
> >> > > > they are useful
> >> > > > > > to application
> >> > > > > > > code as well.
> >> > > > > >
> >> > > > > >
> >> > > > > > This is exactly what I have been
> >> trying to
> >> > > get
> >> > > > at. The use
> >> > > > > > cases documented
> >> > > > > > here are middleware:
> >> > > > > > https://cwiki.apache.org/
> >> > > > confluence/display/KAFKA/A+
> >> > > > > > Case+for+Kafka+Headers.
> >> > > > > > If headers are intended for the
> >> application
> >> > > > space as well,
> >> > > > > the
> >> > > > > > document
> >> > > > > > should be updated accordingly so
> >> that it is
> >> > > an
> >> > > > explicit
> >> > > > > design
> >> > > > > > goal and not
> >> > > > > > an unfortunate design byproduct.
> >> There may
> >> > be
> >> > > > some
> >> > > > > > disagreement on whether
> >> > > > > > it _should_ be a design goal, but
> it
> >> may
> >> > not
> >> > > > matter much
> >> > > > > since
> >> > > > > > the current
> >> > > > > > interceptor API is probably
> >> insufficient
> >> > for
> >> > > > middleware
> >> > > > > > applications (I
> >> > > > > > haven't thought of a way to do
> this
> >> that
> >> > > isn't
> >> > > > cumbersome).
> >> > > > > >
> >> > > > > > That aside, the three unresolved
> >> points for
> >> > > me
> >> > > > are the
> >> > > > > > following:
> >> > > > > >
> >> > > > > > 1. The use of varints. As a
> one-off
> >> for use
> >> > > > only with record
> >> > > > > > headers, the
> >> > > > > > case was weak, but if we are using
> >> them
> >> > > > throughout the
> >> > > > > message
> >> > > > > > format, then
> >> > > > > > we should do so here as well. The
> >> > additional
> >> > > > complexity is
> >> > > > > > minimal and
> >> > > > > > early performance testing fully
> >> justifies
> >> > > > their use.
> >> > > > > >
> >> > > > > > 2. If varints are used, the case
> for
> >> using
> >> > > > attributes to
> >> > > > > > indicate null
> >> > > > > > headers also becomes weak. We only
> >> add one
> >> > > > additional byte in
> >> > > > > > each message
> >> > > > > > if there are no headers. Whatever
> >> the case,
> >> > > > let us be
> >> > > > > > consistent with how
> >> > > > > > we treat null keys and values.
> >> > > > > >
> >> > > > > > 3. We have apparently agreed that
> the
> >> > broker
> >> > > > will validate
> >> > > > > > headers (please
> >> > > > > > add this to the KIP). That being
> the
> >> case,
> >> > I
> >> > > > would prefer to
> >> > > > > > use Kafka's
> >> > > > > > conventional format for arrays.
> The
> >> > downside
> >> > > > is that it takes
> >> > > > > > more work to
> >> > > > > > skip over the headers on the
> >> consumer,
> >> > though
> >> > > > it's unclear if
> >> > > > > > the cost
> >> > > > > > would matter in practice. Some
> >> concern was
> >> > > > previously
> >> > > > > > expressed about the
> >> > > > > > allocation of maps. An alternative
> >> would be
> >> > > to
> >> > > > use arrays,
> >> > > > > i.e.
> >> > > > > >
> >> > > > > > class ConsumerRecord<K, V> {
> >> > > > > > K key;
> >> > > > > > V value;
> >> > > > > > Header[] headers;
> >> > > > > > }
> >> > > > > >
> >> > > > > > class Header {
> >> > > > > > String key;
> >> > > > > > byte[] value;
> >> > > > > > }
> >> > > > > >
> >> > > > > > This would work nicely with the
> >> > conventional
> >> > > > array format and
> >> > > > > > my guess is
> >> > > > > > it would obviate the need do any
> lazy
> >> > > > initialization. If we
> >> > > > > > use the map as
> >> > > > > > is currently documented, then it
> is
> >> > possible
> >> > > > with either
> >> > > > > > representation to
> >> > > > > > slice the headers and initialize
> them
> >> > lazily.
> >> > > > Either way, it
> >> > > > > > might be a
> >> > > > > > good idea to use a separate
> object to
> >> > > > represent the headers
> >> > > > > in
> >> > > > > > case we need
> >> > > > > > to modify it in the future in some
> >> way.
> >> > > > > >
> >> > > > > > (By the way, doesn't it feel a bit
> >> odd that
> >> > > we
> >> > > > seem to be
> >> > > > > > designing a
> >> > > > > > feature which is optimized for
> >> people not
> >> > > > using?)
> >> > > > > >
> >> > > > > >
> >> > > > > > If we can resolve these points,
> then
> >> at
> >> > least
> >> > > > you will get my
> >> > > > > > vote.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Jason
> >> > > > > >
> >> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
> >> radai <
> >> > > > > > radai.rosenblatt@gmail.com> wrote:
> >> > > > > >
> >> > > > > > > headers dont "leak" into
> >> application
> >> > code.
> >> > > > they are useful
> >> > > > > > to application
> >> > > > > > > code as well.
> >> > > > > > > IIUC samze currently has headers
> >> "in-V"
> >> > and
> >> > > > would just
> >> > > > > > switch over to kafka
> >> > > > > > > headers if they exist.
> >> > > > > > > im sure plenty of other users of
> >> kafka
> >> > > would
> >> > > > have a use for
> >> > > > > > headers.
> >> > > > > > > im pretty sure use cases exist
> >> around
> >> > > > shuffling data
> >> > > > > > into/out-of kafka
> >> > > > > > > (kafka connect or equivalent)
> where
> >> > > metadata
> >> > > > from one end
> >> > > > > > could copied over
> >> > > > > > > to the other (S3, for example
> uses
> >> http
> >> > > > headers for
> >> > > > > > user-accessible
> >> > > > > > > metadata). it will be kafka
> client
> >> code
> >> > > > getting/setting
> >> > > > > > those headers. not
> >> > > > > > > an interceptor.
> >> > > > > > >
> >> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
> >> Michael
> >> > > > Pearce <
> >> > > > > > Michael.Pearce@ig.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > For APM single event tracing,
> >> need
> >> > access
> >> > > > to the header
> >> > > > > at
> >> > > > > > the point of
> >> > > > > > > > processing on the processing
> >> thread.
> >> > > > > > > >
> >> > > > > > > > As such interceptors will not
> >> work/be
> >> > > > suitable for these,
> >> > > > > > due to the fact
> >> > > > > > > > they act on the
> ConsumerRecords
> >> as a
> >> > > > batch, before the
> >> > > > > > handling thread
> >> > > > > > > can
> >> > > > > > > > split out and process per
> >> message which
> >> > > is
> >> > > > the point
> >> > > > > these
> >> > > > > > tools will
> >> > > > > > > need
> >> > > > > > > > to continue to transaction
> >> tracing.
> >> > > > > > > >
> >> > > > > > > > Like wise tools and infra
> pieces
> >> will
> >> > > need
> >> > > > access to the
> >> > > > > > message outside
> >> > > > > > > > the interceptor realm.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On 17/02/2017, 21:26, "Jason
> >> > Gustafson" <
> >> > > > > > jason@confluent.io> wrote:
> >> > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > That’s exactly what
> we’re
> >> doing
> >> > the
> >> > > > headers are a
> >> > > > > > slice of bytes,
> >> > > > > > > > which
> >> > > > > > > > > then gets parsed later
> if
> >> needed,
> >> > > or
> >> > > > can be parsed
> >> > > > > > right away, the
> >> > > > > > > > headers
> >> > > > > > > > > is part of the protocol,
> >> so can
> >> > > > still be validated
> >> > > > > > if wanted.
> >> > > > > > > > > If you had a header
> count
> >> then
> >> > you
> >> > > > would have to go
> >> > > > > > through each
> >> > > > > > > > header
> >> > > > > > > > > key and value length
> value
> >> to
> >> > work
> >> > > > out how much to
> >> > > > > > skip to get to
> >> > > > > > > > say the
> >> > > > > > > > > value or any future
> >> component in
> >> > > the
> >> > > > message after
> >> > > > > > the headers.
> >> > > > > > > > Having it
> >> > > > > > > > > as a byte[] with length
> >> value
> >> > makes
> >> > > > this a lot
> >> > > > > > easier to skip.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > So the broker will parse
> the
> >> > headers
> >> > > > and validate
> >> > > > > > them. Good. The
> >> > > > > > > only
> >> > > > > > > > reason remaining that I
> can
> >> see to
> >> > > > leave the headers
> >> > > > > > as a byte array
> >> > > > > > > > is to
> >> > > > > > > > make it easier for the
> >> client to
> >> > skip
> >> > > > past them. Are
> >> > > > > > we sure this is
> >> > > > > > > > not
> >> > > > > > > > premature optimization?
> Are
> >> there
> >> > any
> >> > > > performance
> >> > > > > > results which show
> >> > > > > > > > that
> >> > > > > > > > this is worthwhile?
> >> > > > > > > >
> >> > > > > > > > What’s the issue with
> >> exposing a
> >> > > > method getHeaders on
> >> > > > > > the
> >> > > > > > > > producer/consumer
> >> > > > > > > > > record? It doesn’t break
> >> > anything.
> >> > > > We don’t need
> >> > > > > any
> >> > > > > > special
> >> > > > > > > version.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > See my previous
> explanation.
> >> What I
> >> > > am
> >> > > > trying to
> >> > > > > > resist is the
> >> > > > > > > headers
> >> > > > > > > > becoming a general
> >> > application-level
> >> > > > facility. The
> >> > > > > > primary use case
> >> > > > > > > as
> >> > > > > > > > far
> >> > > > > > > > as I can tell is
> middleware,
> >> which
> >> > is
> >> > > > the use case
> >> > > > > > that the
> >> > > > > > > > interceptors
> >> > > > > > > > are providing.
> >> > > > > > > >
> >> > > > > > > > Current batch consumer
> model
> >> and
> >> > > > consumer
> >> > > > > interceptors
> >> > > > > > don’t work
> >> > > > > > > where
> >> > > > > > > > > headers need to be acted
> >> on at
> >> > per
> >> > > > message level at
> >> > > > > > time of
> >> > > > > > > > processing,
> >> > > > > > > > > very case is APM (the
> core
> >> one),
> >> > > > where the header
> >> > > > > > value is used to
> >> > > > > > > > continue
> >> > > > > > > > > tracing.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > I still don't understand
> the
> >> point
> >> > > > about batching.
> >> > > > > The
> >> > > > > > consumer
> >> > > > > > > > records are
> >> > > > > > > > exposed as a batch in the
> >> consumer
> >> > > > interceptor, but
> >> > > > > > you can still
> >> > > > > > > > iterate
> >> > > > > > > > them individually. It is
> no
> >> > different
> >> > > > for the
> >> > > > > consumer
> >> > > > > > API itself.
> >> > > > > > > >
> >> > > > > > > > -Jason
> >> > > > > > > >
> >> > > > > > > > On Fri, Feb 17, 2017 at
> >> 12:48 PM,
> >> > > > Michael Pearce <
> >> > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Re:
> >> > > > > > > > >
> >> > > > > > > > > “ The point about
> >> creation of
> >> > > > maps seems
> >> > > > > > orthogonal. We can
> >> > > > > > > still
> >> > > > > > > > > represent
> >> > > > > > > > > the headers as a
> slice
> >> of
> >> > bytes
> >> > > > until the time
> >> > > > > > it is accessed.”
> >> > > > > > > > >
> >> > > > > > > > > That’s exactly what
> we’re
> >> doing
> >> > the
> >> > > > headers are a
> >> > > > > > slice of bytes,
> >> > > > > > > > which
> >> > > > > > > > > then gets parsed later
> if
> >> needed,
> >> > > or
> >> > > > can be parsed
> >> > > > > > right away, the
> >> > > > > > > > headers
> >> > > > > > > > > is part of the protocol,
> >> so can
> >> > > > still be validated
> >> > > > > > if wanted.
> >> > > > > > > > >
> >> > > > > > > > > If you had a header
> count
> >> then
> >> > you
> >> > > > would have to go
> >> > > > > > through each
> >> > > > > > > > header
> >> > > > > > > > > key and value length
> value
> >> to
> >> > work
> >> > > > out how much to
> >> > > > > > skip to get to
> >> > > > > > > > say the
> >> > > > > > > > > value or any future
> >> component in
> >> > > the
> >> > > > message after
> >> > > > > > the headers.
> >> > > > > > > > Having it
> >> > > > > > > > > as a byte[] with length
> >> value
> >> > makes
> >> > > > this a lot
> >> > > > > > easier to skip.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On 17/02/2017, 20:37,
> >> "Michael
> >> > > > Pearce" <
> >> > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > What’s the issue
> with
> >> > exposing
> >> > > a
> >> > > > method
> >> > > > > > getHeaders on the
> >> > > > > > > > > producer/consumer
> record?
> >> It
> >> > > doesn’t
> >> > > > break
> >> > > > > anything.
> >> > > > > > We don’t need
> >> > > > > > > > any
> >> > > > > > > > > special version.
> >> > > > > > > > >
> >> > > > > > > > > Current batch
> consumer
> >> model
> >> > > and
> >> > > > consumer
> >> > > > > > interceptors don’t
> >> > > > > > > work
> >> > > > > > > > > where headers need to be
> >> acted on
> >> > > at
> >> > > > per message
> >> > > > > > level at time of
> >> > > > > > > > > processing, very case is
> >> APM (the
> >> > > > core one), where
> >> > > > > > the header value
> >> > > > > > > > is used
> >> > > > > > > > > to continue tracing.
> >> JMS/HTTP etc
> >> > > > all expose these,
> >> > > > > > without issues.
> >> > > > > > > > I would
> >> > > > > > > > > NOT want to lock this
> down
> >> to
> >> > only
> >> > > > be usable
> >> > > > > > accessible via
> >> > > > > > > > interceptors,
> >> > > > > > > > > as we’d fail on one of
> the
> >> main
> >> > > > goals.
> >> > > > > > > > >
> >> > > > > > > > > Regards
> >> > > > > > > > > Mike
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On 17/02/2017,
> 20:21,
> >> "Jason
> >> > > > Gustafson" <
> >> > > > > > jason@confluent.io>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > The point about
> >> creation
> >> > of
> >> > > > maps seems
> >> > > > > > orthogonal. We can
> >> > > > > > > > still
> >> > > > > > > > > represent
> >> > > > > > > > > the headers as a
> >> slice of
> >> > > > bytes until the
> >> > > > > > time it is
> >> > > > > > > > accessed.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > > Yes exactly we
> >> have
> >> > > access
> >> > > > to the records
> >> > > > > > thus why the
> >> > > > > > > > header
> >> > > > > > > > > should be
> >> > > > > > > > > > accessible via
> >> it and
> >> > not
> >> > > > hidden for only
> >> > > > > > interceptors to
> >> > > > > > > > access.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > As explained
> >> above, the
> >> > > > point is to make
> >> > > > > the
> >> > > > > > intended usage
> >> > > > > > > > clear.
> >> > > > > > > > > Applications
> should
> >> > > continue
> >> > > > to rely on the
> >> > > > > > key/value
> >> > > > > > > fields
> >> > > > > > > > to
> >> > > > > > > > > serialize
> >> > > > > > > > > their own
> headers,
> >> and it
> >> > > > would be more
> >> > > > > > ideal if we can
> >> > > > > > > avoid
> >> > > > > > > > > leaking
> >> > > > > > > > > third-party
> >> headers into
> >> > > > applications. This
> >> > > > > > is difficult to
> >> > > > > > > > do
> >> > > > > > > > > with the
> >> > > > > > > > > current
> >> interceptors
> >> > > because
> >> > > > they share the
> >> > > > > > record objects
> >> > > > > > > > with
> >> > > > > > > > > the common
> >> > > > > > > > > API. What I had
> in
> >> mind
> >> > is
> >> > > > something like
> >> > > > > an
> >> > > > > > extension of
> >> > > > > > > the
> >> > > > > > > > > current
> >> > > > > > > > > interceptors
> which
> >> > exposed
> >> > > a
> >> > > > different
> >> > > > > > object (e.g.
> >> > > > > > > > > `RecordAndHeaders`).
> >> > > > > > > > > The challenge is
> >> for
> >> > > MM-like
> >> > > > use cases. Let
> >> > > > > > me see if I can
> >> > > > > > > > come
> >> > > > > > > > > up with a
> >> > > > > > > > > concrete
> proposal
> >> for
> >> > that
> >> > > > problem.
> >> > > > > > > > >
> >> > > > > > > > > -Jason
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Feb 17,
> >> 2017 at
> >> > > > 11:55 AM, Michael
> >> > > > > > Pearce <
> >> > > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > I am happy to
> >> move the
> >> > > > definition of the
> >> > > > > > header into the
> >> > > > > > > > message
> >> > > > > > > > > body, but
> >> > > > > > > > > > would cause us
> >> not to
> >> > > lazy
> >> > > > > > initialise/parse the headers,
> >> > > > > > > as
> >> > > > > > > > > obviously, we
> >> > > > > > > > > > would have to
> >> traverse
> >> > > > these reading the
> >> > > > > > message.
> >> > > > > > > > > >
> >> > > > > > > > > > This was
> >> actually one
> >> > of
> >> > > > Jay’s requests:
> >> > > > > > > > > >
> >> > > > > > > > > > “ 2. I
> think
> >> we
> >> > should
> >> > > > think about
> >> > > > > > creating the Map
> >> > > > > > > > lazily to
> >> > > > > > > > > avoid
> >> > > > > > > > > > parsing
> out
> >> all the
> >> > > > headers into
> >> > > > > > little objects.
> >> > > > > > > > HashMaps
> >> > > > > > > > > themselves
> >> > > > > > > > > > are
> >> > > > > > > > > > kind of
> >> expensive
> >> > and
> >> > > > the consumer is
> >> > > > > > very perf
> >> > > > > > > > sensitive so
> >> > > > > > > > > and making
> >> > > > > > > > > >
> gazillions of
> >> > > hashmaps
> >> > > > that may or
> >> > > > > may
> >> > > > > > not get used
> >> > > > > > > is
> >> > > > > > > > > probably a bad
> >> > > > > > > > > > idea.”
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On 17/02/2017,
> >> 19:44,
> >> > > > "Michael Pearce" <
> >> > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > Yes
> exactly
> >> we have
> >> > > > access to the
> >> > > > > > records thus why
> >> > > > > > > the
> >> > > > > > > > > header should
> >> > > > > > > > > > be accessible
> >> via it
> >> > and
> >> > > > not hidden for
> >> > > > > > only interceptors
> >> > > > > > > > to
> >> > > > > > > > > access.
> >> > > > > > > > > >
> >> > > > > > > > > > Sent using
> >> OWA for
> >> > > > iPhone
> >> > > > > > > > > >
> >> > > > ______________________________
> >> > > > > > __________
> >> > > > > > > > > > From:
> Magnus
> >> > > Edenhill <
> >> > > > > > magnus@edenhill.se>
> >> > > > > > > > > > Sent:
> Friday,
> >> > > February
> >> > > > 17, 2017
> >> > > > > > 7:34:49 PM
> >> > > > > > > > > > To:
> >> > > > dev@kafka.apache.org
> >> > > > > > > > > > Subject:
> Re:
> >> > > [DISCUSS]
> >> > > > KIP-82 - Add
> >> > > > > > Record Headers
> >> > > > > > > > > >
> >> > > > > > > > > > Big +1 on
> >> VarInts.
> >> > > > > > > > > > CPUs are
> >> fast,
> >> > memory
> >> > > > is slow.
> >> > > > > > > > > >
> >> > > > > > > > > > I agree
> with
> >> Jason
> >> > > > that we'll want to
> >> > > > > > continue
> >> > > > > > > > verifying
> >> > > > > > > > > messages,
> >> > > > > > > > > > including
> >> their
> >> > > > headers, so while I
> >> > > > > > appreciate the
> >> > > > > > > > idea of
> >> > > > > > > > > the opaque
> >> > > > > > > > > > header
> blob
> >> it
> >> > won't
> >> > > > be useful in
> >> > > > > > practice.
> >> > > > > > > > > >
> >> > > > > > > > > > /Magnus
> >> > > > > > > > > >
> >> > > > > > > > > > 2017-02-17
> >> 10:41
> >> > > > GMT-08:00 Jason
> >> > > > > > Gustafson <
> >> > > > > > > > > jason@confluent.io>:
> >> > > > > > > > > >
> >> > > > > > > > > > > Sorry,
> my
> >> > mistake.
> >> > > > The consumer
> >> > > > > > interceptor is per
> >> > > > > > > > batch,
> >> > > > > > > > > though I'm
> >> > > > > > > > > > not
> >> > > > > > > > > > > sure
> >> that's an
> >> > > > actual limitation
> >> > > > > > since you still
> >> > > > > > > have
> >> > > > > > > > > access to the
> >> > > > > > > > > > >
> individual
> >> > records.
> >> > > > > > > > > > >
> >> > > > > > > > > > > -Jason
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Fri,
> >> Feb 17,
> >> > > 2017
> >> > > > at 10:39 AM,
> >> > > > > > Jason Gustafson <
> >> > > > > > > > > >
> >> jason@confluent.io>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Re
> >> headers as
> >> > > byte
> >> > > > array and
> >> > > > > > future use by
> >> > > > > > > broker.
> >> > > > > > > > This
> >> > > > > > > > > doesn't
> >> > > > > > > > > > take away
> >> > > > > > > > > > > >> from
> >> that at
> >> > > all.
> >> > > > Nor makes it
> >> > > > > > difficult at all
> >> > > > > > > > in my
> >> > > > > > > > > opinion.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Yeah,
> I
> >> didn't
> >> > > say
> >> > > > it was
> >> > > > > > difficult, only
> >> > > > > > > awkward.
> >> > > > > > > > You
> >> > > > > > > > > wouldn't
> >> > > > > > > > > > write the
> >> > > > > > > > > > > > schema
> >> that way
> >> > > if
> >> > > > you were
> >> > > > > > planning to use it on
> >> > > > > > > > the
> >> > > > > > > > > brokers from
> >> > > > > > > > > > the
> >> > > > > > > > > > > >
> >> beginning. Note
> >> > > > also that one of
> >> > > > > > the benefits of
> >> > > > > > > > letting
> >> > > > > > > > > the broker
> >> > > > > > > > > > > >
> >> understand
> >> > > headers
> >> > > > is that it can
> >> > > > > > validate that
> >> > > > > > > > they are
> >> > > > > > > > > properly
> >> > > > > > > > > > > >
> >> formatted. If
> >> > > cost
> >> > > > is the only
> >> > > > > > concern, we should
> >> > > > > > > > > confirm its
> >> > > > > > > > > > impact
> >> > > > > > > > > > > >
> through
> >> > > > performance testing.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > One of
> >> the key
> >> > > use
> >> > > > cases requires
> >> > > > > > access on
> >> > > > > > > > consume at
> >> > > > > > > > > per
> >> > > > > > > > > > event/message
> >> > > > > > > > > > > >> level
> >> at the
> >> > > > point that message
> >> > > > > > is being
> >> > > > > > > > processed, as
> >> > > > > > > > > such the
> >> > > > > > > > > > batch
> >> > > > > > > > > > > >>
> >> interceptors
> >> > and
> >> > > > batch consume
> >> > > > > > api isn't
> >> > > > > > > > suitable. It
> >> > > > > > > > > needs to be
> >> > > > > > > > > > at the
> >> > > > > > > > > > > >>
> record
> >> level.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > I'm
> not
> >> sure I
> >> > > > understand the
> >> > > > > > point about
> >> > > > > > > batching.
> >> > > > > > > > > Interceptors
> >> > > > > > > > > > are
> >> > > > > > > > > > > >
> applied
> >> > > > per-message, right?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > My
> >> intent on
> >> > > > interceptors is to
> >> > > > > > keep the usage of
> >> > > > > > > > headers
> >> > > > > > > > > > well-defined
> so
> >> > > > > > > > > > > > that
> >> they don't
> >> > > > start leaking
> >> > > > > > unnecessarily into
> >> > > > > > > > > applications. My
> >> > > > > > > > > > guess
> >> > > > > > > > > > > is
> >> > > > > > > > > > > > that
> it's
> >> > > probably
> >> > > > inevitable,
> >> > > > > but
> >> > > > > > isolating it
> >> > > > > > > in
> >> > > > > > > > the
> >> > > > > > > > > > interceptors
> >> would
> >> > > > > > > > > > > > at
> least
> >> give
> >> > > > people a second
> >> > > > > > thought before
> >> > > > > > > > deciding to
> >> > > > > > > > > use it.
> >> > > > > > > > > > The main
> >> > > > > > > > > > > >
> >> challenge in my
> >> > > > mind is figuring
> >> > > > > > out how an MM
> >> > > > > > > use
> >> > > > > > > > case
> >> > > > > > > > > would
> >> > > > > > > > > > work. It
> >> > > > > > > > > > > > would
> be
> >> more
> >> > > > cumbersome to
> >> > > > > > replicate headers
> >> > > > > > > > through an
> >> > > > > > > > > > interceptor,
> >> > > > > > > > > > > > though
> >> arguably
> >> > > MM
> >> > > > should be
> >> > > > > > working at a lower
> >> > > > > > > > level
> >> > > > > > > > > anyway.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > -Jason
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On
> Fri,
> >> Feb 17,
> >> > > > 2017 at 10:16 AM,
> >> > > > > > Michael Pearce
> >> > > > > > > <
> >> > > > > > > > > >
> >> Michael.Pearce@ig.com>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >> Re
> >> headers
> >> > > > available on the
> >> > > > > > record va
> >> > > > > > > > interceptors only
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> One
> of
> >> the key
> >> > > > use cases
> >> > > > > requires
> >> > > > > > access on
> >> > > > > > > > consume at
> >> > > > > > > > > per
> >> > > > > > > > > > event/message
> >> > > > > > > > > > > >> level
> >> at the
> >> > > > point that message
> >> > > > > > is being
> >> > > > > > > > processed, as
> >> > > > > > > > > such the
> >> > > > > > > > > > batch
> >> > > > > > > > > > > >>
> >> interceptors
> >> > and
> >> > > > batch consume
> >> > > > > > api isn't
> >> > > > > > > > suitable. It
> >> > > > > > > > > needs to be
> >> > > > > > > > > > at the
> >> > > > > > > > > > > >>
> record
> >> level.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> This
> >> anyhow is
> >> > > > similar to
> >> > > > > > jms/http/amqp where
> >> > > > > > > > headers
> >> > > > > > > > > are
> >> > > > > > > > > > available to
> >> > > > > > > > > > > >>
> >> consuming
> >> > > > applications.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> Re
> >> headers as
> >> > > > byte array and
> >> > > > > > future use by
> >> > > > > > > > broker. This
> >> > > > > > > > > doesn't
> >> > > > > > > > > > take
> >> > > > > > > > > > > away
> >> > > > > > > > > > > >> from
> >> that at
> >> > > all.
> >> > > > Nor makes it
> >> > > > > > difficult at all
> >> > > > > > > > in my
> >> > > > > > > > > opinion.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> Sent
> >> using OWA
> >> > > > for iPhone
> >> > > > > > > > > > > >>
> >> > > > ______________________________
> >> > > > > > __________
> >> > > > > > > > > > > >> From:
> >> Jason
> >> > > > Gustafson <
> >> > > > > > jason@confluent.io>
> >> > > > > > > > > > > >> Sent:
> >> Friday,
> >> > > > February 17, 2017
> >> > > > > > 5:55:42 PM
> >> > > > > > > > > > > >> To:
> >> > > > dev@kafka.apache.org
> >> > > > > > > > > > > >>
> >> Subject: Re:
> >> > > > [DISCUSS] KIP-82 -
> >> > > > > > Add Record
> >> > > > > > > Headers
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> Would
> >> you be
> >> > > > proposing in
> >> > > > > > KIP-98 to convert
> >> > > > > > > the
> >> > > > > > > > other
> >> > > > > > > > > message
> >> > > > > > > > > > int’s
> >> > > > > > > > > > > (key
> >> > > > > > > > > > > >> >
> >> length,
> >> > value
> >> > > > length) also to
> >> > > > > > varint to keep
> >> > > > > > > it
> >> > > > > > > > > uniform.
> >> > > > > > > > > > > >> >
> Also I
> >> > assume
> >> > > > there will be a
> >> > > > > > static or helper
> >> > > > > > > > method
> >> > > > > > > > > made to
> >> > > > > > > > > > >
> write/read
> >> > > > > > > > > > > >> >
> these
> >> in the
> >> > > > client and
> >> > > > > server.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> Yes,
> >> that is
> >> > > what
> >> > > > we are
> >> > > > > > proposing, so using
> >> > > > > > > > varints
> >> > > > > > > > > for headers
> >> > > > > > > > > > would
> >> > > > > > > > > > > be
> >> > > > > > > > > > > >>
> >> consistent
> >> > with
> >> > > > the rest of the
> >> > > > > > message. We have
> >> > > > > > > > used
> >> > > > > > > > > static
> >> > > > > > > > > > helper
> >> > > > > > > > > > > >>
> methods
> >> > > > > > > > > > > >> in
> our
> >> > prototype
> >> > > > implementation.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> The
> >> cost of
> >> > > > parsing, we want to
> >> > > > > > parse/interpret
> >> > > > > > > > the
> >> > > > > > > > > headers
> >> > > > > > > > > > lazily (this
> >> > > > > > > > > > > >> is
> >> > > > > > > > > > > >> > a
> key
> >> point
> >> > > > brought up earlier
> >> > > > > > in discussions)
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> I'm a
> >> bit
> >> > > > skeptical of this. Has
> >> > > > > > anyone done the
> >> > > > > > > > > performance
> >> > > > > > > > > > testing? I
> >> > > > > > > > > > > >> can
> >> > > > > > > > > > > >>
> probably
> >> > > > implement it and test
> >> > > > > it
> >> > > > > > if no one else
> >> > > > > > > > has. I
> >> > > > > > > > > was also
> >> > > > > > > > > > under
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >>
> >> impression
> >> > that
> >> > > > there may be use
> >> > > > > > cases down the
> >> > > > > > > > road
> >> > > > > > > > > where the
> >> > > > > > > > > > broker
> >> > > > > > > > > > > >> would
> >> > > > > > > > > > > >> need
> to
> >> > > interpret
> >> > > > headers. That
> >> > > > > > wouldn't be off
> >> > > > > > > > the
> >> > > > > > > > > table in the
> >> > > > > > > > > > future
> >> > > > > > > > > > > if
> >> > > > > > > > > > > >> it's
> >> > represented
> >> > > > as bytes, but
> >> > > > > it
> >> > > > > > would be quite
> >> > > > > > > > a bit
> >> > > > > > > > > more
> >> > > > > > > > > > awkward,
> >> > > > > > > > > > > >>
> right?
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> By
> the
> >> way,
> >> > one
> >> > > > question I have
> >> > > > > > been wondering
> >> > > > > > > > about. My
> >> > > > > > > > > > understanding
> >> > > > > > > > > > > is
> >> > > > > > > > > > > >> that
> >> headers
> >> > are
> >> > > > primarily for
> >> > > > > > use cases where a
> >> > > > > > > > > third-party
> >> > > > > > > > > > components
> >> > > > > > > > > > > >>
> wants to
> >> > enrich
> >> > > > messages without
> >> > > > > > needing to
> >> > > > > > > > understand
> >> > > > > > > > > or modify
> >> > > > > > > > > > the
> >> > > > > > > > > > > >>
> schema
> >> > > > > > > > > > > >> of
> the
> >> message
> >> > > > key and value.
> >> > > > > For
> >> > > > > > the
> >> > > > > > > applications
> >> > > > > > > > > which directly
> >> > > > > > > > > > > produce
> >> > > > > > > > > > > >> and
> >> consume
> >> > the
> >> > > > messages and
> >> > > > > > control the
> >> > > > > > > key/value
> >> > > > > > > > > schema
> >> > > > > > > > > > directly, it
> >> > > > > > > > > > > >> seems
> >> we would
> >> > > > rather have them
> >> > > > > > implement
> >> > > > > > > headers
> >> > > > > > > > > directly in
> >> > > > > > > > > > their own
> >> > > > > > > > > > > >>
> schema.
> >> > > Supposing
> >> > > > for the sake
> >> > > > > of
> >> > > > > > argument that
> >> > > > > > > > it was
> >> > > > > > > > > possible,
> >> > > > > > > > > > my
> >> > > > > > > > > > > >>
> >> question is
> >> > > > whether it be
> >> > > > > > sufficient to expose
> >> > > > > > > the
> >> > > > > > > > > headers in the
> >> > > > > > > > > > > >>
> >> interceptor
> >> > API
> >> > > > and not in the
> >> > > > > > common API?
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >>
> -Jason
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> On
> Fri,
> >> Feb
> >> > 17,
> >> > > > 2017 at 3:26 AM,
> >> > > > > > Michael Pearce
> >> > > > > > > <
> >> > > > > > > > > >
> >> Michael.Pearce@ig.com>
> >> > > > > > > > > > > >>
> wrote:
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >> > On
> >> the point
> >> > > of
> >> > > > varInts
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> Would
> >> you be
> >> > > > proposing in
> >> > > > > > KIP-98 to convert
> >> > > > > > > the
> >> > > > > > > > other
> >> > > > > > > > > message
> >> > > > > > > > > > int’s
> >> > > > > > > > > > > (key
> >> > > > > > > > > > > >> >
> >> length,
> >> > value
> >> > > > length) also to
> >> > > > > > varint to keep
> >> > > > > > > it
> >> > > > > > > > > uniform.
> >> > > > > > > > > > > >> >
> Also I
> >> > assume
> >> > > > there will be a
> >> > > > > > static or helper
> >> > > > > > > > method
> >> > > > > > > > > made to
> >> > > > > > > > > > >
> write/read
> >> > > > > > > > > > > >> >
> these
> >> in the
> >> > > > client and
> >> > > > > server.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> Cheers
> >> > > > > > > > > > > >> >
> Mike
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> > On
> >> > 17/02/2017,
> >> > > > 11:22, "Michael
> >> > > > > > Pearce" <
> >> > > > > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> On the
> >> > > > point re: headers
> >> > > > > in
> >> > > > > > the message
> >> > > > > > > > protocol
> >> > > > > > > > > being a
> >> > > > > > > > > > byte
> >> > > > > > > > > > > array
> >> > > > > > > > > > > >> > and
> >> not a
> >> > > count
> >> > > > of elements
> >> > > > > > followed by the
> >> > > > > > > > elements.
> >> > > > > > > > > Again
> >> > > > > > > > > > this was
> >> > > > > > > > > > > >> >
> >> > > > discussed/argued previously.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> It was
> >> > > > agreed on for a few
> >> > > > > > reasons some of
> >> > > > > > > > which
> >> > > > > > > > > you have
> >> > > > > > > > > > >
> obviously
> >> > > > > > > > > > > >> >
> >> picked up
> >> > on:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> Broker
> >> > is
> >> > > > able to pass it
> >> > > > > > through opaquely
> >> > > > > > > > > > > >> >
> >> The cost
> >> > > of
> >> > > > parsing, we
> >> > > > > > want to
> >> > > > > > > > parse/interpret
> >> > > > > > > > > the headers
> >> > > > > > > > > > lazily
> >> > > > > > > > > > > >> >
> (this
> >> is a
> >> > key
> >> > > > point brought
> >> > > > > up
> >> > > > > > earlier in
> >> > > > > > > > > discussions)
> >> > > > > > > > > > > >> >
> >> Headers
> >> > > can
> >> > > > be copied from
> >> > > > > > consumer record
> >> > > > > > > > to
> >> > > > > > > > > producer
> >> > > > > > > > > > record (aka
> >> > > > > > > > > > > >> >
> mirror
> >> > makers
> >> > > > etc) without
> >> > > > > > parsing if no
> >> > > > > > > > changes are
> >> > > > > > > > > being made
> >> > > > > > > > > > or
> >> > > > > > > > > > > being
> >> > > > > > > > > > > >> >
> >> looked at.
> >> > > > > > > > > > > >> >
> >> Keeps
> >> > the
> >> > > > broker agnostic
> >> > > > > > to the format
> >> > > > > > > > > > > >> >
> >> You need
> >> > > an
> >> > > > int32 either
> >> > > > > > for the byte size
> >> > > > > > > > of the
> >> > > > > > > > > headers,
> >> > > > > > > > > > or for
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> count
> >> of
> >> > > > elements, so
> >> > > > > overheads
> >> > > > > > are the same,
> >> > > > > > > > but
> >> > > > > > > > > with going
> >> > > > > > > > > > with an
> >> > > > > > > > > > > >>
> opaque
> >> > > > > > > > > > > >> >
> byte
> >> array
> >> > has
> >> > > > the above
> >> > > > > > advantages.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> Cheers
> >> > > > > > > > > > > >> >
> >> Mike
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> On
> >> > > > 17/02/2017, 02:50,
> >> > > > > > "Jason Gustafson" <
> >> > > > > > > > > jason@confluent.io
> >> > > > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > Sorry,
> >> > > > should have
> >> > > > > > noted that the
> >> > > > > > > > performance
> >> > > > > > > > > testing
> >> > > > > > > > > > was done
> >> > > > > > > > > > > >> >
> using
> >> the
> >> > > > > > > > > > > >> >
> >> > > > producer performance
> >> > > > > > tool shipped with
> >> > > > > > > > Kafka.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > -Jason
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> On
> >> > > Thu,
> >> > > > Feb 16, 2017
> >> > > > > at
> >> > > > > > 6:44 PM, Jason
> >> > > > > > > > > Gustafson <
> >> > > > > > > > > > > >> >
> >> > > > jason@confluent.io> wrote:
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > Hey
> >> > > > Nacho,
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > I've
> >> > > > compared
> >> > > > > > performance of our
> >> > > > > > > > KIP-98
> >> > > > > > > > > > implementation
> >> with
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> >
> >> without
> >> > > > > > > > > > > >> >
> >> >
> >> > > > varints. For
> >> > > > > messages
> >> > > > > > around 128
> >> > > > > > > > bytes, we
> >> > > > > > > > > see an
> >> > > > > > > > > > increase
> >> > > > > > > > > > > in
> >> > > > > > > > > > > >> >
> >> throughput
> >> > of
> >> > > > > > > > > > > >> >
> >> >
> >> > > about
> >> > > > 30% using the
> >> > > > > > default
> >> > > > > > > > configuration
> >> > > > > > > > > settings.
> >> > > > > > > > > > At 256
> >> > > > > > > > > > > >> >
> >> bytes, the
> >> > > > > > > > > > > >> >
> >> >
> >> > > > increase is around
> >> > > > > > 16%. Obviously
> >> > > > > > > the
> >> > > > > > > > > performance
> >> > > > > > > > > > converge
> >> > > > > > > > > > > as
> >> > > > > > > > > > > >> >
> >> messages get
> >> > > > > > > > > > > >> >
> >> >
> >> > > > larger, but it seems
> >> > > > > > well worth the
> >> > > > > > > > cost.
> >> > > > > > > > > Note that
> >> > > > > > > > > > we are
> >> > > > > > > > > > > >> also
> >> > > > > > > > > > > >> >
> >> seeing a
> >> > > > > > > > > > > >> >
> >> >
> >> > > > substantial
> >> > > > > > performance increase
> >> > > > > > > > against
> >> > > > > > > > > trunk
> >> > > > > > > > > > primarily
> >> > > > > > > > > > > >>
> because
> >> > > > > > > > > > > >> > of
> the
> >> > > > > > > > > > > >> >
> >> >
> >> > much
> >> > > > more efficient
> >> > > > > > packing that
> >> > > > > > > > varints
> >> > > > > > > > > provide us.
> >> > > > > > > > > > > Anything
> >> > > > > > > > > > > >> >
> >> adding to
> >> > > > > > > > > > > >> >
> >> >
> >> > > > message overhead,
> >> > > > > > such as record
> >> > > > > > > > headers,
> >> > > > > > > > > would only
> >> > > > > > > > > > > increase
> >> > > > > > > > > > > >> > the
> >> relative
> >> > > > > > > > > > > >> >
> >> >
> >> > > > difference. (Of
> >> > > > > > course take these
> >> > > > > > > > numbers
> >> > > > > > > > > with a
> >> > > > > > > > > > grain of
> >> > > > > > > > > > > salt
> >> > > > > > > > > > > >> >
> since
> >> I have
> >> > > > > > > > > > > >> >
> >> >
> >> > only
> >> > > > used the
> >> > > > > default
> >> > > > > > settings with
> >> > > > > > > > both
> >> > > > > > > > > the producer
> >> > > > > > > > > > and
> >> > > > > > > > > > > >>
> broker
> >> > > > > > > > > > > >> > on
> my
> >> > > > > > > > > > > >> >
> >> >
> >> > > local
> >> > > > machine. We
> >> > > > > > intend to provide
> >> > > > > > > > more
> >> > > > > > > > > extensive
> >> > > > > > > > > > >
> performance
> >> > > > > > > > > > > >> >
> >> details as
> >> > > > > > > > > > > >> >
> >> >
> >> > part
> >> > > > of the work for
> >> > > > > > KIP-98.)
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > The
> >> > > > implementation
> >> > > > > we
> >> > > > > > are using is
> >> > > > > > > > from
> >> > > > > > > > > protobuf (
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > https://developers.google.com/
> >> > > > > > > > > > >
> >> > > > protocol-buffers/docs/encoding
> >> > > > > > > > > > > >> ),
> >> > > > > > > > > > > >> >
> which
> >> is
> >> > > > > > > > > > > >> >
> >> >
> >> > also
> >> > > > used in HBase.
> >> > > > > > It is trivial to
> >> > > > > > > > > implement and as
> >> > > > > > > > > > far
> >> > > > > > > > > > > as I
> >> > > > > > > > > > > >> >
> know
> >> doesn't
> >> > > > > > > > > > > >> >
> >> >
> >> > > > suffer from the
> >> > > > > > aliasing problem you
> >> > > > > > > > are
> >> > > > > > > > > describing. I
> >> > > > > > > > > > > checked
> >> > > > > > > > > > > >> >
> with
> >> Magnus
> >> > > > > > > > > > > >> >
> >> >
> >> > (the
> >> > > > author of
> >> > > > > > librdkafka) and he
> >> > > > > > > > agreed
> >> > > > > > > > > that the
> >> > > > > > > > > > savings
> >> > > > > > > > > > > >>
> seemed
> >> > > > > > > > > > > >> >
> worth
> >> the
> >> > > > > > > > > > > >> >
> >> >
> >> > cost
> >> > > > of
> >> > > > > > implementation.
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > > -Jason
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> > On
> >> > > > Thu, Feb 16, 2017
> >> > > > > > at 4:32 PM,
> >> > > > > > > > Ignacio
> >> > > > > > > > > Solis <
> >> > > > > > > > > > > >>
> >> > isolis@igso.net
> >> > > >
> >> > > > > > > > > > > >> >
> wrote:
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > -VarInts
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > I'm
> >> > > > one of the
> >> > > > > > people (if not the
> >> > > > > > > > most)
> >> > > > > > > > > opposed to
> >> > > > > > > > > > VarInts.
> >> > > > > > > > > > > >> >
> >> VarInts
> >> > > > > > > > > > > >> >
> >> >>
> >> > > have
> >> > > > a place, but
> >> > > > > > this is not it.
> >> > > > > > > > (We
> >> > > > > > > > > had a large
> >> > > > > > > > > > > >>
> >> discussion
> >> > > > > > > > > > > >> >
> about
> >> > > > > > > > > > > >> >
> >> >>
> >> > > them
> >> > > > at the
> >> > > > > > beginning of KIP-82
> >> > > > > > > time)
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > If
> >> > > > anybody has real
> >> > > > > > life
> >> > > > > > > performance
> >> > > > > > > > > numbers of
> >> > > > > > > > > > VarInts
> >> > > > > > > > > > > >> >
> >> improving
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > things or
> >> > > > > > significantly reducing
> >> > > > > > > > resources
> >> > > > > > > > > I would
> >> > > > > > > > > > like to
> >> > > > > > > > > > > >> know
> >> > > > > > > > > > > >> >
> what
> >> > > > > > > > > > > >> >
> >> >>
> >> > > that
> >> > > > case may be.
> >> > > > > > Yes, you can save
> >> > > > > > > > some
> >> > > > > > > > > bytes here
> >> > > > > > > > > > and
> >> > > > > > > > > > > >>
> there,
> >> > > > > > > > > > > >> > but
> >> > > > > > > > > > > >> >
> >> >>
> >> > > this
> >> > > > is probably
> >> > > > > > insignificant to
> >> > > > > > > the
> >> > > > > > > > > overall system
> >> > > > > > > > > > > behavior
> >> > > > > > > > > > > >> > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > storage
> >> > > > > > requirements. -- I say
> >> > > > > > > this
> >> > > > > > > > with
> >> > > > > > > > > respect to
> >> > > > > > > > > > using
> >> > > > > > > > > > > >> >
> >> VarInts in
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > protocol
> >> > > > > itself,
> >> > > > > > not as part of
> >> > > > > > > > the
> >> > > > > > > > > data.
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > VarInts require you
> >> > > > > > to parse the
> >> > > > > > > Int
> >> > > > > > > > > before using it
> >> > > > > > > > > > and
> >> > > > > > > > > > > >> >
> >> depending on
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > encoding they
> >> > > > > > can suffer from
> >> > > > > > > > aliasing
> >> > > > > > > > > (multiple
> >> > > > > > > > > > > >> >
> >> > > representations
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > the same
> >> > > > > value).
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > Why
> >> > > > add complexity?
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > Nacho
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > On
> >> > > > Thu, Feb 16,
> >> > > > > 2017
> >> > > > > > at 10:29 AM,
> >> > > > > > > > Colin
> >> > > > > > > > > McCabe <
> >> > > > > > > > > > > >> >
> >> > > > cmccabe@apache.org>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > wrote:
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > +1
> >> > > > for varints
> >> > > > > > here-- it would
> >> > > > > > > save
> >> > > > > > > > > quite a bit of
> >> > > > > > > > > > space.
> >> > > > > > > > > > > >> >
> They
> >> are
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > pretty quick to
> >> > > > > > implement as
> >> > > > > > > well.
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >>
> >> > > I
> >> > > > think it makes
> >> > > > > > sense for values
> >> > > > > > > > to be
> >> > > > > > > > > byte
> >> > > > > > > > > > arrays.
> >> > > > > > > > > > > Users
> >> > > > > > > > > > > >> >
> might
> >> want
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > to
> >> > > > attach
> >> > > > > > arbitrary payloads;
> >> > > > > > > they
> >> > > > > > > > > shouldn't be
> >> > > > > > > > > > forced to
> >> > > > > > > > > > > >> >
> >> serialize
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > everything to
> >> > > > > Java
> >> > > > > > strings.
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > best,
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > Colin
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > On
> >> > > > Thu, Feb 16,
> >> > > > > > 2017, at 09:52,
> >> > > > > > > > Jason
> >> > > > > > > > > Gustafson
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > Hey Michael,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > Hmm, I guess the
> >> > > > > > point of
> >> > > > > > > > representing
> >> > > > > > > > > it as
> >> > > > > > > > > > bytes is to
> >> > > > > > > > > > > >> >
> allow
> >> the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > broker
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > to pass it
> >> > > > > > through opaquely? Is
> >> > > > > > > > the
> >> > > > > > > > > cost of
> >> > > > > > > > > > parsing
> >> > > > > > > > > > > them a
> >> > > > > > > > > > > >> >
> >> concern, or
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > are
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > we simply trying
> >> > > > > > to ensure that
> >> > > > > > > > the
> >> > > > > > > > > broker stays
> >> > > > > > > > > > > agnostic
> >> > > > > > > > > > > >> to
> >> > > > > > > > > > > >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > format?
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > On varints, I
> >> > > > > > think adding
> >> > > > > > > > support for
> >> > > > > > > > > them makes
> >> > > > > > > > > > less
> >> > > > > > > > > > > >> sense
> >> > > > > > > > > > > >> >
> for an
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > isolated use
> >> > > > > > case, but as part
> >> > > > > > > of
> >> > > > > > > > a
> >> > > > > > > > > more holistic
> >> > > > > > > > > > change
> >> > > > > > > > > > > >> >
> (such
> >> as
> >> > what
> >> > > > > > > > > > > >> >
> >> >>
> >> > we
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > have proposed in
> >> > > > > > KIP-98), I
> >> > > > > > > think
> >> > > > > > > > they
> >> > > > > > > > > are
> >> > > > > > > > > > justifiable.
> >> > > > > > > > > > > If
> >> > > > > > > > > > > >> > we
> >> add them,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > then the need to
> >> > > > > > use attributes
> >> > > > > > > > becomes
> >> > > > > > > > > quite a
> >> > > > > > > > > > bit
> >> > > > > > > > > > > >>
> weaker,
> >> > > > > > > > > > > >> >
> >> right? The
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > other thing I
> >> > > > > > find slightly odd
> >> > > > > > > > is the
> >> > > > > > > > > fact that
> >> > > > > > > > > > null
> >> > > > > > > > > > > >> >
> >> headers has
> >> > no
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > actual
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > semantic meaning
> >> > > > > > for the message
> >> > > > > > > > > (unlike null
> >> > > > > > > > > > keys and
> >> > > > > > > > > > > >> >
> >> values). It
> >> > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > just
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > a
> >> > > > space
> >> > > > > > optimization. It seems a
> >> > > > > > > > bit
> >> > > > > > > > > better to
> >> > > > > > > > > > always
> >> > > > > > > > > > > use
> >> > > > > > > > > > > >> >
> size
> >> 0 to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > indicate having
> >> > > > > > no headers.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > Overall, the
> >> > > > > main
> >> > > > > > point is
> >> > > > > > > > ensuring
> >> > > > > > > > > that the
> >> > > > > > > > > > message
> >> > > > > > > > > > > >>
> schema
> >> > > > > > > > > > > >> >
> >> remains
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > consistent,
> >> > > > > > either within the
> >> > > > > > > > larger
> >> > > > > > > > > protocol, or
> >> > > > > > > > > > at a
> >> > > > > > > > > > > >> >
> >> minimum
> >> > within
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > message itself.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > -Jason
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > On Thu, Feb 16,
> >> > > > > > 2017 at 6:39 AM,
> >> > > > > > > > > Michael Pearce <
> >> > > > > > > > > > > >> >
> >> > > > Michael.Pearce@ig.com
> >> > > > > > > > > > > >> >
> >> >> >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Hi Jason,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On point 1) in
> >> > > > > > the message
> >> > > > > > > > protocol
> >> > > > > > > > > the headers
> >> > > > > > > > > > are
> >> > > > > > > > > > > >>
> simply
> >> > > > > > > > > > > >> > a
> byte
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > array,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > as like the
> >> > > > > key
> >> > > > > > or value, this
> >> > > > > > > > is to
> >> > > > > > > > > clearly
> >> > > > > > > > > > demarcate
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> >> header in
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > core message.
> >> > > > > > Then the header
> >> > > > > > > > byte
> >> > > > > > > > > array in the
> >> > > > > > > > > > core
> >> > > > > > > > > > > >> >
> >> message is
> >> > an
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > array of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > key, value
> >> > > > > > pairs. This is what
> >> > > > > > > > it is
> >> > > > > > > > > denoting.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Then this
> >> > > > > would
> >> > > > > > be I guess in
> >> > > > > > > > the
> >> > > > > > > > > given
> >> > > > > > > > > > notation:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Headers =>
> >> > > > > > [KeyLength, Key,
> >> > > > > > > > > ValueLength, Value]
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > KeyLength
> >> > > > > > => int32
> >> > > > > > > > > <-----------------NEW
> >> > > > > > > > > > size of
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> byte[] of
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > serialised key
> >> > > > > > value
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Key =>
> >> > > > > bytes
> >> > > > > > > > > <----------------------
> NEW
> >> > > > > > > > > > >
> serialised
> >> > > > > > > > > > > >> >
> string
> >> > (UTF8)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > bytes of the
> >> > > > > > header key
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > ValueLength
> >> > > > > > => int32
> >> > > > > > > > > <-------------- NEW
> >> > > > > > > > > > size of
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> byte[] of
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > serialised
> >> > > > > > header value
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Value =>
> >> > > > > > bytes
> >> > > > > > > > > <--------------------
> NEW
> >> > > > > > > > > > >
> serialised
> >> > > > > > > > > > > >> >
> form
> >> of the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > header
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > value
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > The key length
> >> > > > > > and value
> >> > > > > > > length
> >> > > > > > > > is
> >> > > > > > > > > matching the
> >> > > > > > > > > > way
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> protocol is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > defined in the
> >> > > > > > core message
> >> > > > > > > > currently.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On point 2)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Var sized
> >> > > > > ints,
> >> > > > > > this was
> >> > > > > > > > discussed
> >> > > > > > > > > much earlier
> >> > > > > > > > > > on, in
> >> > > > > > > > > > > >> >
> fact
> >> I had
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > suggested it
> >> > > > > > myself (with
> >> > > > > > > Hadoop
> >> > > > > > > > > references),
> >> > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> complexity
> >> > of
> >> > > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > compared to
> >> > > > > > having a simpler
> >> > > > > > > > protocol
> >> > > > > > > > > was
> >> > > > > > > > > > argued and
> >> > > > > > > > > > > >> >
> >> agreed it
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > wasn’t worth
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the complexity
> >> > > > > > as all other
> >> > > > > > > > clients
> >> > > > > > > > > in other
> >> > > > > > > > > > languages
> >> > > > > > > > > > > >> >
> would
> >> need
> >> > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > ensure
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > theyre using
> >> > > > > > the right var
> >> > > > > > > size
> >> > > > > > > > > algorithm, as
> >> > > > > > > > > > there
> >> > > > > > > > > > > is a
> >> > > > > > > > > > > >> >
> few.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On point 3)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > We did the
> >> > > > > > attributes,
> >> > > > > > > optional
> >> > > > > > > > > approach as
> >> > > > > > > > > > originally
> >> > > > > > > > > > > >> >
> there
> >> was
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > marked
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > concern that
> >> > > > > > headers would
> >> > > > > > > > cause a
> >> > > > > > > > > message size
> >> > > > > > > > > > > overhead
> >> > > > > > > > > > > >> > for
> >> others,
> >> > > > > > > > > > > >> >
> >> >>
> >> > who
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > don’t want
> >> > > > > > them. As such this
> >> > > > > > > > is the
> >> > > > > > > > > clean
> >> > > > > > > > > > solution to
> >> > > > > > > > > > > >> >
> >> achieve
> >> > that.
> >> > > > > > > > > > > >> >
> >> >>
> >> > If
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > that no longer
> >> > > > > > holds, and we
> >> > > > > > > > don’t
> >> > > > > > > > > care that we
> >> > > > > > > > > > add
> >> > > > > > > > > > > >>
> 4bytes
> >> > > > > > > > > > > >> >
> >> overhead,
> >> > > > > > > > > > > >> >
> >> >>
> >> > > then
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > im happy to
> >> > > > > > remove.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > I’m personally
> >> > > > > > in favour of
> >> > > > > > > > keeping
> >> > > > > > > > > the message
> >> > > > > > > > > > as
> >> > > > > > > > > > > small
> >> > > > > > > > > > > >> > as
> >> possible
> >> > > > > > > > > > > >> >
> >> >>
> >> > so
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > people don’t
> >> > > > > > get shocks in
> >> > > > > > > perf
> >> > > > > > > > and
> >> > > > > > > > > throughputs
> >> > > > > > > > > > dues
> >> > > > > > > > > > > to
> >> > > > > > > > > > > >> >
> >> message
> >> > size,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > unless they
> >> > > > > > actively use the
> >> > > > > > > > feature,
> >> > > > > > > > > as such I
> >> > > > > > > > > > do
> >> > > > > > > > > > > >>
> prefer
> >> > > > > > > > > > > >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > attribute bit
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > wise feature
> >> > > > > > flag approach
> >> > > > > > > > myself.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > On 16/02/2017,
> >> > > > > > 05:40, "Jason
> >> > > > > > > > > Gustafson" <
> >> > > > > > > > > > > >> >
> >> > > > jason@confluent.io> wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > We have
> >> > > > > > proposed a few
> >> > > > > > > > > significant changes
> >> > > > > > > > > > to the
> >> > > > > > > > > > > >> >
> >> message
> >> > format
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > KIP-98
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > which now
> >> > > > > > seems likely to
> >> > > > > > > > pass
> >> > > > > > > > > (perhaps
> >> > > > > > > > > > with some
> >> > > > > > > > > > > >> >
> >> iterations
> >> > on
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > implementation details).
> >> > > > > > > It
> >> > > > > > > > would
> >> > > > > > > > > be good
> >> > > > > > > > > > to try
> >> > > > > > > > > > > and
> >> > > > > > > > > > > >> >
> >> coordinate
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > changes
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > in both of
> >> > > > > > the proposals
> >> > > > > > > to
> >> > > > > > > > make
> >> > > > > > > > > sure they
> >> > > > > > > > > > are
> >> > > > > > > > > > > >> >
> >> consistent
> >> > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > compatible.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > I think
> >> > > > > > using the
> >> > > > > > > > attributes to
> >> > > > > > > > > indicate
> >> > > > > > > > > > null
> >> > > > > > > > > > > >>
> headers
> >> > > > > > > > > > > >> > is
> a
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > reasonable
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > approach.
> >> > > > > > We have proposed
> >> > > > > > > > to do
> >> > > > > > > > > the same
> >> > > > > > > > > > thing
> >> > > > > > > > > > > for
> >> > > > > > > > > > > >> > the
> >> message
> >> > > > > > > > > > > >> >
> >> >>
> >> > key
> >> > > > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > value.
> >> > > > > That
> >> > > > > > said, I
> >> > > > > > > > sympathize
> >> > > > > > > > > with Jay's
> >> > > > > > > > > > >
> argument.
> >> > > > > > > > > > > >> >
> Having
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > multiple
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > ways to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > specify a
> >> > > > > > null value
> >> > > > > > > > increases
> >> > > > > > > > > the overall
> >> > > > > > > > > > > >>
> >> complexity
> >> > > > > > > > > > > >> > of
> the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > protocol.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > You
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > can see
> >> > > > > > this just from the
> >> > > > > > > > fact
> >> > > > > > > > > that you
> >> > > > > > > > > > need the
> >> > > > > > > > > > > >> >
> extra
> >> > verbiage
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > protocol
> >> > > > > > specification in
> >> > > > > > > > this
> >> > > > > > > > > KIP and in
> >> > > > > > > > > > KIP-98
> >> > > > > > > > > > > to
> >> > > > > > > > > > > >> >
> >> describe the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > dependence
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > between
> >> > > > > the
> >> > > > > > fields and the
> >> > > > > > > > > attributes. It
> >> > > > > > > > > > seems
> >> > > > > > > > > > > >> like
> a
> >> > > > > > > > > > > >> >
> >> slippery
> >> > > > > > > > > > > >> >
> >> >>
> >> > > slope
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > if
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > you start
> >> > > > > > allowing
> >> > > > > > > different
> >> > > > > > > > > request types
> >> > > > > > > > > > to
> >> > > > > > > > > > > >> >
> >> implement
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > specification differently.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > You can
> >> > > > > > also argue that
> >> > > > > > > the
> >> > > > > > > > > messages
> >> > > > > > > > > > already are
> >> > > > > > > > > > > and
> >> > > > > > > > > > > >> > are
> >> likely
> >> > > > > > > > > > > >> >
> >> >>
> >> > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > remain a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > special
> >> > > > > > case. For example,
> >> > > > > > > > there
> >> > > > > > > > > is
> >> > > > > > > > > > currently no
> >> > > > > > > > > > > >> >
> >> generality
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > how
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > compressed
> >> > > > > > message sets
> >> > > > > > > are
> >> > > > > > > > > represented
> >> > > > > > > > > > that would
> >> > > > > > > > > > > >> be
> >> > > > > > > > > > > >> >
> >> applicable
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > other
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > request
> >> > > > > > types. Some might
> >> > > > > > > > see this
> >> > > > > > > > > > divergence as
> >> > > > > > > > > > > an
> >> > > > > > > > > > > >> >
> >> unfortunate
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > deficiency
> >> > > > > > which should be
> >> > > > > > > > fixed;
> >> > > > > > > > > others
> >> > > > > > > > > > might see
> >> > > > > > > > > > > >> it
> >> > > > > > > > > > > >> > as
> >> sort of
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > inevitability of needing
> >> > > > > > > to
> >> > > > > > > > > optimize where
> >> > > > > > > > > > it
> >> > > > > > > > > > > counts
> >> > > > > > > > > > > >> >
> most.
> >> I'm
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > probably
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > somewhere
> >> > > > > > in between, but
> >> > > > > > > I
> >> > > > > > > > think
> >> > > > > > > > > we
> >> > > > > > > > > > probably all
> >> > > > > > > > > > > >> >
> share
> >> the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > intuition
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > > protocol should be
> >> > > > > > > kept
> >> > > > > > > > as
> >> > > > > > > > > consistent as
> >> > > > > > > > > > > >>
> >> possible.
> >> > > > > > > > > > > >> >
> With
> >> that
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > mind,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > here are a
> >> > > > > > few comments:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > 1. One
> >> > > > > > thing I found a
> >> > > > > > > > little odd
> >> > > > > > > > > when
> >> > > > > > > > > > reading the
> >> > > > > > > > > > > >> >
> >> current
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > proposal is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > headers
> >> > > > > > are both
> >> > > > > > > > represented
> >> > > > > > > > > as an
> >> > > > > > > > > > array of
> >> > > > > > > > > > > >> bytes
> >> > > > > > > > > > > >> > and
> >> as an
> >> > > > > > > > > > > >> >
> >> >>
> >> > > array
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > key/value
> >> > > > > > pairs. I'd
> >> > > > > > > > probably
> >> > > > > > > > > suggest
> >> > > > > > > > > > something
> >> > > > > > > > > > > like
> >> > > > > > > > > > > >> >
> this:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Headers =>
> >> > > > > > [HeaderKey
> >> > > > > > > > HeaderValue]
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > HeaderKey
> >> > > > > > => String
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > HeaderValue => Bytes
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > An array
> >> > > > > in
> >> > > > > > the Kafka
> >> > > > > > > > protocol is
> >> > > > > > > > > > represented
> as a
> >> > > > > > > > > > > >> >
> 4-byte
> >> > integer
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > indicating
> >> > > > > > the number of
> >> > > > > > > > elements
> >> > > > > > > > > in the
> >> > > > > > > > > > array
> >> > > > > > > > > > > >> >
> >> followed by
> >> > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > serialization of the
> >> > > > > > > > elements.
> >> > > > > > > > > Unless I'm
> >> > > > > > > > > > > >> >
> >> > > > misunderstanding, what
> >> > > > > > > > > > > >> >
> >> >>
> >> > you
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > have
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > instead is
> >> > > > > > the total size
> >> > > > > > > > of the
> >> > > > > > > > > headers in
> >> > > > > > > > > > bytes
> >> > > > > > > > > > > >> >
> >> followed by
> >> > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > elements.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > I'm not
> >> > > > > > sure I see any
> >> > > > > > > > reason for
> >> > > > > > > > > this
> >> > > > > > > > > > > >>
> >> inconsistency.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > 2. In
> >> > > > > > KIP-98, we've
> >> > > > > > > > introduced
> >> > > > > > > > > >
> variable-length
> >> > > > > > > > > > > >>
> integer
> >> > > > > > > > > > > >> >
> >> fields.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Effectively,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > we've
> >> > > > > > enriched (or
> >> > > > > > > > "complicated"
> >> > > > > > > > > as Jay
> >> > > > > > > > > > might say
> >> > > > > > > > > > > ;)
> >> > > > > > > > > > > >> > the
> >> protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > specification to include
> >> > > > > > > the
> >> > > > > > > > > following
> >> > > > > > > > > > types:
> >> > > > > > > > > > > >>
> VarInt,
> >> > > > > > > > > > > >> >
> >> VarLong,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > UnsignedVarInt and
> >> > > > > > > > > UnsignedVarLong.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Along with
> >> > > > > > these
> >> > > > > > > > primitives, we
> >> > > > > > > > > could
> >> > > > > > > > > > introduce
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> >> following
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > types:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > VarSizeArray =>
> >> > > > > > > > NumberOfItems
> >> > > > > > > > > Item1 Item2
> >> > > > > > > > > > .. ItemN
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > NumberOfItems =>
> >> > > > > > > > UnsignedVarInt
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > VarSizeNullableArray =>
> >> > > > > > > > > NumberOfItemsOrNull
> >> > > > > > > > > > Item1
> >> > > > > > > > > > > >> >
> Item2
> >> ..
> >> > ItemN
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > NumberOfItemsOrNull =>
> >> > > > > > > > VarInt
> >> > > > > > > > > (-1 means
> >> > > > > > > > > > null)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > And
> >> > > > > > similarly for the
> >> > > > > > > > `String`
> >> > > > > > > > > and `Bytes`
> >> > > > > > > > > > types.
> >> > > > > > > > > > > >> >
> These
> >> types
> >> > > > > > > > > > > >> >
> >> >>
> >> > can
> >> > > > save
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > considerable amount of
> >> > > > > > > > space in
> >> > > > > > > > > this
> >> > > > > > > > > > proposal
> >> > > > > > > > > > > >>
> because
> >> > > > > > > > > > > >> >
> they
> >> can
> >> > > > > > > > > > > >> >
> >> >>
> >> > be
> >> > > > used
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > both the
> >> > > > > > number of headers
> >> > > > > > > > > included in the
> >> > > > > > > > > > message
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > lengths of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > header
> >> > > > > keys
> >> > > > > > and values. We
> >> > > > > > > > could
> >> > > > > > > > > do this
> >> > > > > > > > > > instead:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Headers =>
> >> > > > > > > > VarSizeArray[HeaderKey
> >> > > > > > > > > > HeaderValue]
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > HeaderKey
> >> > > > > > =>
> >> > > > > > > VarSizeString
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > HeaderValue =>
> >> > > > > > > > VarSizeBytes
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > Combining
> >> > > > > > the savings from
> >> > > > > > > > the
> >> > > > > > > > > use of
> >> > > > > > > > > > variable
> >> > > > > > > > > > > >>
> length
> >> > > > > > > > > > > >> >
> >> fields, the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > benefit
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > of using
> >> > > > > > the attributes to
> >> > > > > > > > > represent null
> >> > > > > > > > > > seems
> >> > > > > > > > > > > >>
> pretty
> >> > > > > > > > > > > >> >
> small.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > 3.
> >> > > > > > Whichever way we go
> >> > > > > > > > (whether
> >> > > > > > > > > we use the
> >> > > > > > > > > > > >>
> >> attributes
> >> > > > > > > > > > > >> > or
> >> not), we
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > should at
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > least be
> >> > > > > > consistent
> >> > > > > > > between
> >> > > > > > > > this
> >> > > > > > > > > KIP and
> >> > > > > > > > > > KIP-98.
> >> > > > > > > > > > > It
> >> > > > > > > > > > > >> >
> would
> >> be
> >> > very
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > strange
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > to have
> >> > > > > two
> >> > > > > > ways to
> >> > > > > > > > represent
> >> > > > > > > > > null values
> >> > > > > > > > > > in the
> >> > > > > > > > > > > >> same
> >> > > > > > > > > > > >> >
> >> schema.
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > Either
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > way is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > OK with
> >> > > > > me.
> >> > > > > > I think some
> >> > > > > > > > > message-level
> >> > > > > > > > > > >
> >> optimizations
> >> > > > > > > > > > > >> > are
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > justifiable,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > but
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > savings
> >> > > > > > here seem
> >> > > > > > > > minimal (a
> >> > > > > > > > > few bytes
> >> > > > > > > > > > per
> >> > > > > > > > > > > >> >
> >> message), so
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > maybe it's
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > not
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > worth the
> >> > > > > > cost of letting
> >> > > > > > > > the
> >> > > > > > > > > message
> >> > > > > > > > > > diverge even
> >> > > > > > > > > > > >> >
> >> further from
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > rest of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > the
> >> > > > > > protocol.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > -Jason
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > On Wed,
> >> > > > > Feb
> >> > > > > > 15, 2017 at
> >> > > > > > > > 8:52 AM,
> >> > > > > > > > > radai <
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > radai.rosenblatt@gmail.com>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > I've
> >> > > > > > trimmed the inline
> >> > > > > > > > > contents as this
> >> > > > > > > > > > mail is
> >> > > > > > > > > > > >> >
> >> getting too
> >> > > > > > > > > > > >> >
> >> >>
> >> > big
> >> > > > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > apache
> >> > > > > > mailing list
> >> > > > > > > > software to
> >> > > > > > > > > deliver
> >> > > > > > > > > > :-(
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 1. the
> >> > > > > > important thing
> >> > > > > > > for
> >> > > > > > > > > >
> interoperability
> >> is
> >> > > > > > > > > > > for
> >> > > > > > > > > > > >> >
> >> different
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > "interested
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > parties"
> >> > > > > > (plugins, infra
> >> > > > > > > > > layers/wrappers,
> >> > > > > > > > > > > >>
> >> user-code)
> >> > > > > > > > > > > >> > to
> be
> >> > > > > > > > > > > >> >
> >> >>
> >> > > able
> >> > > > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > stick
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > pieces
> >> > > > > of
> >> > > > > > metadata onto
> >> > > > > > > > msgs
> >> > > > > > > > > without
> >> > > > > > > > > > getting in
> >> > > > > > > > > > > >> each
> >> > > > > > > > > > > >> >
> >> other's
> >> > > > > > > > > > > >> >
> >> >>
> >> > > way.
> >> > > > a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > common
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > key
> >> > > > > > scheme (Strings, as
> >> > > > > > > > of the
> >> > > > > > > > > time of
> >> > > > > > > > > > this
> >> > > > > > > > > > > >> >
> >> writing?) is
> >> > > all
> >> > > > > > > > > > > >> >
> >> >>
> >> > > thats
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > required
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > for
> >> > > > > that.
> >> > > > > > it is assumed
> >> > > > > > > > that
> >> > > > > > > > > the other end
> >> > > > > > > > > > > >> >
> >> interested
> >> > in
> >> > > > any
> >> > > > > > > > > > > >> >
> >> >>
> >> > > such
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > piece of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > metadata
> >> > > > > > knows the
> >> > > > > > > > encoding,
> >> > > > > > > > > and byte[]
> >> > > > > > > > > > provides
> >> > > > > > > > > > > >> for
> >> > > > > > > > > > > >> > the
> >> most
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > flexibility.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > i
> >> > > > > believe
> >> > > > > > this is the
> >> > > > > > > same
> >> > > > > > > > > logic behind
> >> > > > > > > > > > core
> >> > > > > > > > > > > kafka
> >> > > > > > > > > > > >> >
> being
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > byte[]/byte[]
> >> > > > > -
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > Strings
> >> > > > > > are more
> >> > > > > > > "usable"
> >> > > > > > > > but
> >> > > > > > > > > bytes are
> >> > > > > > > > > > flexible
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> > so
> >> were
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > chosen.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > Also -
> >> > > > > > core kafka doesnt
> >> > > > > > > > even
> >> > > > > > > > > do that
> >> > > > > > > > > > good of a
> >> > > > > > > > > > > >> job
> >> > > > > > > > > > > >> > on
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > usability of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > payload
> >> > > > > > (example - i
> >> > > > > > > have
> >> > > > > > > > to
> >> > > > > > > > > specify the
> >> > > > > > > > > > nop
> >> > > > > > > > > > > >>
> byte[]
> >> > > > > > > > > > > >> >
> >> "decoders"
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > explicitly
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > in
> >> > > > > conf),
> >> > > > > > and again
> >> > > > > > > > sacrificies
> >> > > > > > > > > usability
> >> > > > > > > > > > for
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >> >
> sake
> >> of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > performance
> >> > > > > (no
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > convenient single-record
> >> > > > > > > > > processing as
> >> > > > > > > > > > poll is a
> >> > > > > > > > > > > >> >
> >> batch, lots
> >> > of
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > obscure
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > little
> >> > > > > > config details
> >> > > > > > > > exposing
> >> > > > > > > > > internals
> >> > > > > > > > > > of the
> >> > > > > > > > > > > >> >
> >> batching
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > mechanism,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > etc)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > this is
> >> > > > > > also why i
> >> > > > > > > really
> >> > > > > > > > > dislike the
> >> > > > > > > > > > idea of a
> >> > > > > > > > > > > >> >
> "type
> >> > system"
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > header
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > values,
> >> > > > > > it further
> >> > > > > > > > degrades the
> >> > > > > > > > > > usability,
> adds
> >> > > > > > > > > > > >> >
> >> complexity
> >> > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > > will
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > eventually get in
> >> > > > > > > > people's way,
> >> > > > > > > > > also, it
> >> > > > > > > > > > would
> >> > > > > > > > > > > be
> >> > > > > > > > > > > >> > the
> >> 2nd/3rd
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > home-group
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > serialization mechanism
> >> > > > > > > > in core
> >> > > > > > > > > kafka
> >> > > > > > > > > > (counting
> >> > > > > > > > > > > 2
> >> > > > > > > > > > > >> >
> >> iterations
> >> > > > > > > > > > > >> >
> >> >>
> >> > of
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > "type
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > definition DSL")
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 2. this
> >> > > > > > is an
> >> > > > > > > > implementation
> >> > > > > > > > > detail, and
> >> > > > > > > > > > not
> >> > > > > > > > > > > even
> >> > > > > > > > > > > >> a
> >> > > > > > > > > > > >> >
> very
> >> "user
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > facing" one?
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > to the
> >> > > > > > best of my
> >> > > > > > > > understanding
> >> > > > > > > > > the vote
> >> > > > > > > > > > process
> >> > > > > > > > > > > >> is
> >> > > > > > > > > > > >> > on
> >> proposed
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > API/behaviour. also -
> >> > > > > > > > since
> >> > > > > > > > > we're willing
> >> > > > > > > > > > to go
> >> > > > > > > > > > > >> with
> >> > > > > > > > > > > >> >
> >> strings
> >> > > > > > > > > > > >> >
> >> >>
> >> > > just
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > serialize
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > a
> >> > > > > 0-sized
> >> > > > > > header blob
> >> > > > > > > and
> >> > > > > > > > IIUC
> >> > > > > > > > > you dont
> >> > > > > > > > > > need any
> >> > > > > > > > > > > >> >
> >> optionals
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > anymore.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 3. yes,
> >> > > > > > we can :-)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > On Tue,
> >> > > > > > Feb 14, 2017 at
> >> > > > > > > > 11:56
> >> > > > > > > > > PM, Michael
> >> > > > > > > > > > > Pearce <
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > Michael.Pearce@ig.com>
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > wrote:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Hi
> >> > > > > Jay,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 1)
> >> > > > > > There was some
> >> > > > > > > > initial
> >> > > > > > > > > debate on the
> >> > > > > > > > > > value
> >> > > > > > > > > > > >> >
> part,
> >> as
> >> > youll
> >> > > > > > > > > > > >> >
> >> >>
> >> > > note
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > String,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > String
> >> > > > > > headers were
> >> > > > > > > > > discounted early
> >> > > > > > > > > > on. The
> >> > > > > > > > > > > >> >
> >> reason for
> >> > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > flexibility
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > and
> >> > > > > > keeping in line
> >> > > > > > > > with the
> >> > > > > > > > > > flexibility of
> >> > > > > > > > > > > key,
> >> > > > > > > > > > > >> >
> value
> >> of the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > message
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > object
> >> > > > > > itself. I don’t
> >> > > > > > > > think
> >> > > > > > > > > it takes
> >> > > > > > > > > > away
> >> > > > > > > > > > > from
> >> > > > > > > > > > > >> an
> >> > > > > > > > > > > >> >
> >> ecosystem
> >> > > > > > > > > > > >> >
> >> >>
> >> > as
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > each
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > plugin
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > will
> >> > > > > > care for their
> >> > > > > > > own
> >> > > > > > > > key,
> >> > > > > > > > > this way
> >> > > > > > > > > > ints,
> >> > > > > > > > > > > >> >
> >> booleans ,
> >> > > > exotic
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > custom
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > binary
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > can
> >> > > > > all
> >> > > > > > be catered
> >> > > > > > > for=.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a. If
> >> > > > > > you really
> >> > > > > > > wanted
> >> > > > > > > > to
> >> > > > > > > > > push for a
> >> > > > > > > > > > typed
> >> > > > > > > > > > > >> value
> >> > > > > > > > > > > >> >
> >> interface,
> >> > > > > > > > > > > >> >
> >> >> I
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > wouldn’t
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > want
> >> > > > > > just String
> >> > > > > > > values
> >> > > > > > > > > supported, but
> >> > > > > > > > > > the the
> >> > > > > > > > > > > >> >
> >> primatives
> >> > > > > > > > > > > >> >
> >> >>
> >> > > plus
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > string and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > also
> >> > > > > > still keeping the
> >> > > > > > > > > ability to have a
> >> > > > > > > > > > > binary
> >> > > > > > > > > > > >> > for
> >> custom
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > binaries that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > some
> >> > > > > > organisations may
> >> > > > > > > > have.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > i. I
> >> > > > > > have written this
> >> > > > > > > > slight
> >> > > > > > > > > > alternative
> >> > > > > > > > > > > here,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > https://cwiki.apache.org/
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > confluence/display/KAFKA/KIP-
> >> > > > > > > > > > > >> >
> >> > > > 82+-+Add+Record+Headers+-+
> >> > > > > Typed
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > ii.
> >> > > > > > Essentially the
> >> > > > > > > > value
> >> > > > > > > > > bytes, has a
> >> > > > > > > > > > leading
> >> > > > > > > > > > > >> >
> byte
> >> > overhead.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 1.
> >> > > > > > This tells you
> >> > > > > > > what
> >> > > > > > > > type
> >> > > > > > > > > the value
> >> > > > > > > > > > is,
> >> > > > > > > > > > > >>
> before
> >> > > > > > > > > > > >> >
> >> reading
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > rest
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > of the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > bytes,
> >> > > > > > allowing
> >> > > > > > > > > >
> >> > > > serialisation/deserialization
> >> > > > > > > > > > > to
> >> > > > > > > > > > > >> > and
> >> from the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > primitives,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > string
> >> > > > > > and byte[].
> >> > > > > > > This
> >> > > > > > > > is
> >> > > > > > > > > akin to some
> >> > > > > > > > > > other
> >> > > > > > > > > > > >> >
> >> messaging
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > systems.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 2) We
> >> > > > > > are making it
> >> > > > > > > > optional,
> >> > > > > > > > > so that
> >> > > > > > > > > > for
> >> > > > > > > > > > > those
> >> > > > > > > > > > > >> > not
> >> wanting
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > headers have
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > 0
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > bytes
> >> > > > > > overhead (think
> >> > > > > > > > of it
> >> > > > > > > > > as a feature
> >> > > > > > > > > > > flag),
> >> > > > > > > > > > > >> I
> >> > > > > > > > > > > >> >
> don’t
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > think this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > complex, especially if
> >> > > > > > > > > comparing to
> >> > > > > > > > > > changes
> >> > > > > > > > > > > >> >
> >> proposed in
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > other kips
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > like
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > kip-98.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a. If
> >> > > > > > you really
> >> > > > > > > really
> >> > > > > > > > don’t
> >> > > > > > > > > like
> >> > > > > > > > > > this, we
> >> > > > > > > > > > > can
> >> > > > > > > > > > > >> >
> drop
> >> it, but
> >> > > > > > > > > > > >> >
> >> >>
> >> > it
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > would
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > mean
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > buying
> >> > > > > > into 4 bytes
> >> > > > > > > > extra
> >> > > > > > > > > overhead for
> >> > > > > > > > > > users
> >> > > > > > > > > > > who
> >> > > > > > > > > > > >> > do
> >> not want
> >> > > > > > > > > > > >> >
> >> >>
> >> > to
> >> > > > use
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > headers.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 3) In
> >> > > > > > the summary yes,
> >> > > > > > > > it is
> >> > > > > > > > > at a higher
> >> > > > > > > > > > > level,
> >> > > > > > > > > > > >> >
> but I
> >> think
> >> > > > > > > > > > > >> >
> >> >>
> >> > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > is well
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > documented in the
> >> > > > > > > > proposed
> >> > > > > > > > > changes
> >> > > > > > > > > > section.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a.
> >> > > > > > Added getHeaders
> >> > > > > > > > method to
> >> > > > > > > > > > >
> >> Producer/Consumer
> >> > > > > > > > > > > >> >
> >> record (that
> >> > > > > > > > > > > >> >
> >> >>
> >> > is
> >> > > > it)
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > b.
> >> > > > > > We’ve also detailed
> >> > > > > > > > the
> >> > > > > > > > > new Headers
> >> > > > > > > > > > class
> >> > > > > > > > > > > >> that
> >> > > > > > > > > > > >> >
> this
> >> method
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > returns
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > that
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > encapsulates the
> >> > > > > > > headers
> >> > > > > > > > > protocol and
> >> > > > > > > > > > logic.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Best,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Mike
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > ==Original questions
> >> > > > > > > > from the
> >> > > > > > > > > vote
> >> > > > > > > > > > thread from
> >> > > > > > > > > > > >> >
> Jay.==
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > Couple
> >> > > > > > of things I
> >> > > > > > > > think we
> >> > > > > > > > > still need
> >> > > > > > > > > > to work
> >> > > > > > > > > > > >> out:
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 1.
> >> > > > > I
> >> > > > > > think we agree
> >> > > > > > > > about
> >> > > > > > > > > the key,
> >> > > > > > > > > > but I
> >> > > > > > > > > > > >> think
> >> > > > > > > > > > > >> > we
> >> haven't
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > talked about
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > the
> >> > > > > > value yet. I
> >> > > > > > > > think if
> >> > > > > > > > > our goal
> >> > > > > > > > > > is an
> >> > > > > > > > > > > open
> >> > > > > > > > > > > >> >
> >> ecosystem
> >> > > > > > > > > > > >> >
> >> >>
> >> > of
> >> > > > these
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > header
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > spread across many
> >> > > > > > > > plugins
> >> > > > > > > > > from many
> >> > > > > > > > > > > systems
> >> > > > > > > > > > > >> we
> >> > > > > > > > > > > >> >
> should
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > consider
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > making
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > this
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a
> >> > > > > > string as well so
> >> > > > > > > > it can
> >> > > > > > > > > be
> >> > > > > > > > > > printed, set
> >> > > > > > > > > > > >> via
> >> > > > > > > > > > > >> > a
> UI,
> >> set
> >> > > > > > > > > > > >> >
> >> >>
> >> > in
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > config,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > etc.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > Basically
> >> > > > > > > encouraging
> >> > > > > > > > > pluggable
> >> > > > > > > > > > >
> >> serialization
> >> > > > > > > > > > > >> >
> >> formats
> >> > > > > > > > > > > >> >
> >> >>
> >> > > here
> >> > > > will
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > lead
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > bit
> >> > > > > > of a tower of
> >> > > > > > > > babel.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 2.
> >> > > > > > This proposal
> >> > > > > > > > still
> >> > > > > > > > > includes a
> >> > > > > > > > > > pretty
> >> > > > > > > > > > > big
> >> > > > > > > > > > > >> >
> >> change to
> >> > our
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > serialization
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > and
> >> > > > > > protocol
> >> > > > > > > > definition
> >> > > > > > > > > layer.
> >> > > > > > > > > > Essentially
> >> > > > > > > > > > > >> it is
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > introducing an
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > optional
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > type, where the
> >> > > > > > > > format is
> >> > > > > > > > > data
> >> > > > > > > > > > dependent. I
> >> > > > > > > > > > > >> >
> think
> >> this
> >> > is
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > actually a
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > big
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > change though it
> >> > > > > > > > doesn't
> >> > > > > > > > > seem like
> >> > > > > > > > > > it. It
> >> > > > > > > > > > > >> means
> >> > > > > > > > > > > >> > you
> >> can no
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > longer
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > specify
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > this
> >> > > > > > type with our
> >> > > > > > > > type
> >> > > > > > > > > definition
> >> > > > > > > > > > DSL, and
> >> > > > > > > > > > > >> >
> >> likewise it
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > requires
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > custom
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > handling in client
> >> > > > > > > > libs.
> >> > > > > > > > > This isn't
> >> > > > > > > > > > a huge
> >> > > > > > > > > > > >> >
> >> thing, since
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Record
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > definition is
> >> > > > > > > custom
> >> > > > > > > > > anyway, but I
> >> > > > > > > > > > think
> >> > > > > > > > > > > this
> >> > > > > > > > > > > >> >
> kind
> >> of
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > protocol
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > inconsistency is
> >> > > > > > > very
> >> > > > > > > > > non-desirable
> >> > > > > > > > > > and
> >> > > > > > > > > > > ties
> >> > > > > > > > > > > >> >
> you to
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > hand-coding
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > things.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > I
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > think the type
> >> > > > > > > should
> >> > > > > > > > > instead by [Key
> >> > > > > > > > > > > Value]
> >> > > > > > > > > > > >> in
> >> > > > > > > > > > > >> > our
> >> BNF,
> >> > > > > > > > > > > >> >
> >> >>
> >> > > where
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > key and
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > value are both
> >> > > > > > > short
> >> > > > > > > > > strings as used
> >> > > > > > > > > > > >>
> >> elsewhere.
> >> > > > > > > > > > > >> >
> This
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > brings it
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > in line
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > with
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > the
> >> > > > > > rest of the
> >> > > > > > > > protocol.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > 3.
> >> > > > > > Could we get
> >> > > > > > > more
> >> > > > > > > > > specific about
> >> > > > > > > > > > the
> >> > > > > > > > > > > exact
> >> > > > > > > > > > > >> >
> Java
> >> API
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > change to
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > ProducerRecord,
> >> > > > > > > > > ConsumerRecord,
> >> > > > > > > > > > Record,
> >> > > > > > > > > > > etc?
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > > -Jay
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > The
> >> > > > > information
> >> > > > > > contained in
> >> > > > > > > > this
> >> > > > > > > > > email is
> >> > > > > > > > > > strictly
> >> > > > > > > > > > > >> >
> >> confidential
> >> > > and
> >> > > > > > > > > > > >> >
> >> >>
> >> > for
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the use of the
> >> > > > > > addressee only,
> >> > > > > > > > unless
> >> > > > > > > > > otherwise
> >> > > > > > > > > > > >>
> >> indicated.
> >> > > > > > > > > > > >> > If
> you
> >> > > > > > > > > > > >> >
> >> >>
> >> > are
> >> > > > not
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > the intended
> >> > > > > > recipient, please
> >> > > > > > > > do not
> >> > > > > > > > > read,
> >> > > > > > > > > > copy, use
> >> > > > > > > > > > > or
> >> > > > > > > > > > > >> >
> >> disclose to
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > others
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > this message
> >> > > > > or
> >> > > > > > any
> >> > > > > > > attachment.
> >> > > > > > > > > Please also
> >> > > > > > > > > > notify the
> >> > > > > > > > > > > >> >
> >> sender by
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > replying
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > to this email
> >> > > > > > or by telephone
> >> > > > > > > > > (+44(020 7896
> >> > > > > > > > > > 0011) and
> >> > > > > > > > > > > >> then
> >> > > > > > > > > > > >> >
> delete
> >> > > > > > > > > > > >> >
> >> >>
> >> > the
> >> > > > email
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > and any copies
> >> > > > > > of it.
> >> > > > > > > Opinions,
> >> > > > > > > > > conclusion
> >> > > > > > > > > > (etc) that
> >> > > > > > > > > > > do
> >> > > > > > > > > > > >> > not
> >> relate
> >> > > > > > > > > > > >> >
> >> >>
> >> > to
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > official
> >> > > > > > business of this
> >> > > > > > > > company
> >> > > > > > > > > shall be
> >> > > > > > > > > > understood
> >> > > > > > > > > > > as
> >> > > > > > > > > > > >> >
> >> neither
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > given nor
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > endorsed by
> >> > > > > it.
> >> > > > > > IG is a
> >> > > > > > > trading
> >> > > > > > > > name
> >> > > > > > > > > of IG
> >> > > > > > > > > > Markets
> >> > > > > > > > > > > >>
> Limited
> >> > > > > > > > > > > >> > (a
> >> company
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > registered in
> >> > > > > > England and
> >> > > > > > > Wales,
> >> > > > > > > > > company number
> >> > > > > > > > > > > >>
> >> 04008957)
> >> > > > > > > > > > > >> >
> and IG
> >> > > > > > > > > > > >> >
> >> >>
> >> > > Index
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Limited (a
> >> > > > > > company registered
> >> > > > > > > in
> >> > > > > > > > > England and
> >> > > > > > > > > > Wales,
> >> > > > > > > > > > > >> >
> >> company
> >> > number
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > 01190902).
> >> > > > > > Registered address
> >> > > > > > > at
> >> > > > > > > > > Cannon Bridge
> >> > > > > > > > > > House,
> >> > > > > > > > > > > 25
> >> > > > > > > > > > > >> >
> >> Dowgate
> >> > > > > > > > > > > >> >
> >> >>
> >> > > Hill,
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > London EC4R
> >> > > > > > 2YA. Both IG
> >> > > > > > > Markets
> >> > > > > > > > > Limited
> >> > > > > > > > > > (register
> >> > > > > > > > > > > >>
> number
> >> > > > > > > > > > > >> >
> >> 195355)
> >> > > > > > > > > > > >> >
> >> >>
> >> > and
> >> > > > IG
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Index Limited
> >> > > > > > (register number
> >> > > > > > > > > 114059) are
> >> > > > > > > > > > authorised
> >> > > > > > > > > > > >> and
> >> > > > > > > > > > > >> >
> >> regulated
> >> > > > > > > > > > > >> >
> >> >>
> >> > by
> >> > > > the
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > Financial
> >> > > > > > Conduct Authority.
> >> > > > > > > > > > > >> >
> >> >>
> >> > >>
> >> > > >
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >>
> >> > --
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > Nacho - Ignacio
> >> > > > > > Solis -
> >> > > > > > > > isolis@igso.net
> >> > > > > > > > > > > >> >
> >> >>
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> > The
> >> > > information
> >> > > > contained in
> >> > > > > > this email is
> >> > > > > > > > strictly
> >> > > > > > > > > > confidential
> and
> >> > > > > > > > > > > for
> >> > > > > > > > > > > >> > the
> >> use of
> >> > the
> >> > > > addressee only,
> >> > > > > > unless
> >> > > > > > > otherwise
> >> > > > > > > > > indicated. If
> >> > > > > > > > > > you are
> >> > > > > > > > > > > >> not
> >> > > > > > > > > > > >> > the
> >> intended
> >> > > > recipient, please
> >> > > > > > do not read,
> >> > > > > > > > copy, use
> >> > > > > > > > > or
> >> > > > > > > > > > disclose to
> >> > > > > > > > > > > >>
> others
> >> > > > > > > > > > > >> >
> this
> >> message
> >> > > or
> >> > > > any
> >> > > > > attachment.
> >> > > > > > Please also
> >> > > > > > > > notify
> >> > > > > > > > > the sender by
> >> > > > > > > > > > > >>
> replying
> >> > > > > > > > > > > >> > to
> >> this
> >> > email
> >> > > > or by telephone
> >> > > > > > (+44(020 7896
> >> > > > > > > > 0011) and
> >> > > > > > > > > then
> >> > > > > > > > > > delete the
> >> > > > > > > > > > > >> email
> >> > > > > > > > > > > >> > and
> >> any
> >> > copies
> >> > > > of it.
> >> > > > > Opinions,
> >> > > > > > conclusion
> >> > > > > > > > (etc) that
> >> > > > > > > > > do not
> >> > > > > > > > > > relate to
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> >> official
> >> > > > business of this
> >> > > > > > company shall be
> >> > > > > > > > understood
> >> > > > > > > > > as
> >> > > > > > > > > > neither given
> >> > > > > > > > > > > >> nor
> >> > > > > > > > > > > >> >
> >> endorsed by
> >> > > it.
> >> > > > IG is a
> >> > > > > trading
> >> > > > > > name of IG
> >> > > > > > > > Markets
> >> > > > > > > > > Limited (a
> >> > > > > > > > > > company
> >> > > > > > > > > > > >> >
> >> registered
> >> > in
> >> > > > England and
> >> > > > > > Wales, company
> >> > > > > > > number
> >> > > > > > > > > 04008957) and
> >> > > > > > > > > > IG Index
> >> > > > > > > > > > > >> >
> >> Limited (a
> >> > > > company registered
> >> > > > > > in England and
> >> > > > > > > > Wales,
> >> > > > > > > > > company
> >> > > > > > > > > > number
> >> > > > > > > > > > > >> >
> >> 01190902).
> >> > > > Registered address
> >> > > > > > at Cannon Bridge
> >> > > > > > > > House,
> >> > > > > > > > > 25
> >> > > > > > > > > > Dowgate Hill,
> >> > > > > > > > > > > >> >
> >> London EC4R
> >> > > > 2YA. Both IG
> >> > > > > > Markets Limited
> >> > > > > > > > (register
> >> > > > > > > > > number
> >> > > > > > > > > > 195355) and
> >> > > > > > > > > > > IG
> >> > > > > > > > > > > >> >
> Index
> >> > Limited
> >> > > > (register number
> >> > > > > > 114059) are
> >> > > > > > > > authorised
> >> > > > > > > > > and
> >> > > > > > > > > > regulated by
> >> > > > > > > > > > > >> the
> >> > > > > > > > > > > >> >
> >> Financial
> >> > > > Conduct Authority.
> >> > > > > > > > > > > >> >
> >> > > > > > > > > > > >> The
> >> > information
> >> > > > contained in
> >> > > > > this
> >> > > > > > email is
> >> > > > > > > > strictly
> >> > > > > > > > > confidential
> >> > > > > > > > > > and for
> >> > > > > > > > > > > >> the
> use
> >> of the
> >> > > > addressee only,
> >> > > > > > unless otherwise
> >> > > > > > > > > indicated. If you
> >> > > > > > > > > > are
> >> > > > > > > > > > > not
> >> > > > > > > > > > > >> the
> >> intended
> >> > > > recipient, please
> >> > > > > do
> >> > > > > > not read,
> >> > > > > > > copy,
> >> > > > > > > > use
> >> > > > > > > > > or disclose
> >> > > > > > > > > > to
> >> > > > > > > > > > > others
> >> > > > > > > > > > > >> this
> >> message
> >> > or
> >> > > > any attachment.
> >> > > > > > Please also
> >> > > > > > > > notify the
> >> > > > > > > > > sender by
> >> > > > > > > > > > > replying
> >> > > > > > > > > > > >> to
> this
> >> email
> >> > or
> >> > > > by telephone
> >> > > > > > (+44(020 7896
> >> > > > > > > 0011)
> >> > > > > > > > and
> >> > > > > > > > > then delete
> >> > > > > > > > > > the
> >> > > > > > > > > > > email
> >> > > > > > > > > > > >> and
> any
> >> copies
> >> > > of
> >> > > > it. Opinions,
> >> > > > > > conclusion (etc)
> >> > > > > > > > that
> >> > > > > > > > > do not
> >> > > > > > > > > > relate to
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >>
> official
> >> > > business
> >> > > > of this
> >> > > > > company
> >> > > > > > shall be
> >> > > > > > > > understood
> >> > > > > > > > > as neither
> >> > > > > > > > > > given
> >> > > > > > > > > > > nor
> >> > > > > > > > > > > >>
> >> endorsed by
> >> > it.
> >> > > > IG is a trading
> >> > > > > > name of IG
> >> > > > > > > Markets
> >> > > > > > > > > Limited (a
> >> > > > > > > > > > company
> >> > > > > > > > > > > >>
> >> registered in
> >> > > > England and Wales,
> >> > > > > > company number
> >> > > > > > > > > 04008957) and IG
> >> > > > > > > > > > Index
> >> > > > > > > > > > > >>
> Limited
> >> (a
> >> > > > company registered in
> >> > > > > > England and
> >> > > > > > > > Wales,
> >> > > > > > > > > company number
> >> > > > > > > > > > > >>
> >> 01190902).
> >> > > > Registered address at
> >> > > > > > Cannon Bridge
> >> > > > > > > > House,
> >> > > > > > > > > 25 Dowgate
> >> > > > > > > > > > Hill,
> >> > > > > > > > > > > >>
> London
> >> EC4R
> >> > 2YA.
> >> > > > Both IG Markets
> >> > > > > > Limited
> >> > > > > > > (register
> >> > > > > > > > > number 195355)
> >> > > > > > > > > > and IG
> >> > > > > > > > > > > >> Index
> >> Limited
> >> > > > (register number
> >> > > > > > 114059) are
> >> > > > > > > > authorised
> >> > > > > > > > > and
> >> > > > > > > > > > regulated by
> >> > > > > > > > > > > the
> >> > > > > > > > > > > >>
> >> Financial
> >> > > Conduct
> >> > > > Authority.
> >> > > > > > > > > > > >>
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > The
> >> information
> >> > > > contained in this
> >> > > > > > email is strictly
> >> > > > > > > > > confidential and
> >> > > > > > > > > > for the use of
> >> the
> >> > > > addressee only, unless
> >> > > > > > otherwise
> >> > > > > > > > indicated.
> >> > > > > > > > > If you are
> >> > > > > > > > > > not the
> intended
> >> > > > recipient, please do not
> >> > > > > > read, copy, use
> >> > > > > > > > or
> >> > > > > > > > > disclose to
> >> > > > > > > > > > others this
> >> message or
> >> > > any
> >> > > > attachment.
> >> > > > > > Please also notify
> >> > > > > > > > the
> >> > > > > > > > > sender by
> >> > > > > > > > > > replying to
> this
> >> email
> >> > or
> >> > > > by telephone
> >> > > > > > (+44(020 7896
> >> > > > > > > 0011)
> >> > > > > > > > and
> >> > > > > > > > > then delete
> >> > > > > > > > > > the email and
> any
> >> > copies
> >> > > > of it. Opinions,
> >> > > > > > conclusion
> >> > > > > > > (etc)
> >> > > > > > > > that
> >> > > > > > > > > do not
> >> > > > > > > > > > relate to the
> >> official
> >> > > > business of this
> >> > > > > > company shall be
> >> > > > > > > > > understood as
> >> > > > > > > > > > neither given
> nor
> >> > > endorsed
> >> > > > by it. IG is a
> >> > > > > > trading name of
> >> > > > > > > > IG
> >> > > > > > > > > Markets
> >> > > > > > > > > > Limited (a
> >> company
> >> > > > registered in England
> >> > > > > > and Wales,
> >> > > > > > > company
> >> > > > > > > > > number
> >> > > > > > > > > > 04008957) and
> IG
> >> Index
> >> > > > Limited (a company
> >> > > > > > registered in
> >> > > > > > > > England
> >> > > > > > > > > and Wales,
> >> > > > > > > > > > company number
> >> > 01190902).
> >> > > > Registered
> >> > > > > > address at Cannon
> >> > > > > > > > Bridge
> >> > > > > > > > > House, 25
> >> > > > > > > > > > Dowgate Hill,
> >> London
> >> > EC4R
> >> > > > 2YA. Both IG
> >> > > > > > Markets Limited
> >> > > > > > > > (register
> >> > > > > > > > > number
> >> > > > > > > > > > 195355) and IG
> >> Index
> >> > > > Limited (register
> >> > > > > > number 114059) are
> >> > > > > > > > > authorised and
> >> > > > > > > > > > regulated by
> the
> >> > > Financial
> >> > > > Conduct
> >> > > > > > Authority.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > The information
> >> contained in
> >> > > > this email is
> >> > > > > > strictly
> >> > > > > > > confidential
> >> > > > > > > > and
> >> > > > > > > > > for the use of the
> >> addressee
> >> > only,
> >> > > > unless otherwise
> >> > > > > > indicated. If
> >> > > > > > > > you are
> >> > > > > > > > > not the intended
> recipient,
> >> > please
> >> > > > do not read,
> >> > > > > > copy, use or
> >> > > > > > > > disclose to
> >> > > > > > > > > others this message or
> any
> >> > > > attachment. Please also
> >> > > > > > notify the
> >> > > > > > > sender
> >> > > > > > > > by
> >> > > > > > > > > replying to this email
> or
> >> by
> >> > > > telephone (+44(020
> >> > > > > 7896
> >> > > > > > 0011) and then
> >> > > > > > > > delete
> >> > > > > > > > > the email and any copies
> >> of it.
> >> > > > Opinions,
> >> > > > > conclusion
> >> > > > > > (etc) that do
> >> > > > > > > > not
> >> > > > > > > > > relate to the official
> >> business
> >> > of
> >> > > > this company
> >> > > > > > shall be understood
> >> > > > > > > > as
> >> > > > > > > > > neither given nor
> endorsed
> >> by it.
> >> > > IG
> >> > > > is a trading
> >> > > > > > name of IG
> >> > > > > > > Markets
> >> > > > > > > > > Limited (a company
> >> registered in
> >> > > > England and Wales,
> >> > > > > > company number
> >> > > > > > > > > 04008957) and IG Index
> >> Limited (a
> >> > > > company
> >> > > > > registered
> >> > > > > > in England and
> >> > > > > > > > Wales,
> >> > > > > > > > > company number
> 01190902).
> >> > > Registered
> >> > > > address at
> >> > > > > > Cannon Bridge
> >> > > > > > > House,
> >> > > > > > > > 25
> >> > > > > > > > > Dowgate Hill, London
> EC4R
> >> 2YA.
> >> > Both
> >> > > > IG Markets
> >> > > > > > Limited (register
> >> > > > > > > > number
> >> > > > > > > > > 195355) and IG Index
> >> Limited
> >> > > > (register number
> >> > > > > > 114059) are
> >> > > > > > > authorised
> >> > > > > > > > and
> >> > > > > > > > > regulated by the
> Financial
> >> > Conduct
> >> > > > Authority.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > The information contained in
> >> this email
> >> > > is
> >> > > > strictly
> >> > > > > > confidential and for
> >> > > > > > > > the use of the addressee only,
> >> unless
> >> > > > otherwise
> >> > > > > indicated.
> >> > > > > > If you are not
> >> > > > > > > > the intended recipient, please
> >> do not
> >> > > > read, copy, use or
> >> > > > > > disclose to
> >> > > > > > > others
> >> > > > > > > > this message or any
> attachment.
> >> Please
> >> > > > also notify the
> >> > > > > > sender by replying
> >> > > > > > > > to this email or by telephone
> >> (+44(020
> >> > > > 7896 0011) and
> >> > > > > then
> >> > > > > > delete the
> >> > > > > > > email
> >> > > > > > > > and any copies of it.
> Opinions,
> >> > > conclusion
> >> > > > (etc) that do
> >> > > > > > not relate to
> >> > > > > > > the
> >> > > > > > > > official business of this
> >> company shall
> >> > > be
> >> > > > understood as
> >> > > > > > neither given
> >> > > > > > > nor
> >> > > > > > > > endorsed by it. IG is a
> trading
> >> name of
> >> > > IG
> >> > > > Markets
> >> > > > > Limited
> >> > > > > > (a company
> >> > > > > > > > registered in England and
> Wales,
> >> > company
> >> > > > number 04008957)
> >> > > > > > and IG Index
> >> > > > > > > > Limited (a company registered
> in
> >> > England
> >> > > > and Wales,
> >> > > > > > company number
> >> > > > > > > > 01190902). Registered address
> at
> >> Cannon
> >> > > > Bridge House, 25
> >> > > > > > Dowgate Hill,
> >> > > > > > > > London EC4R 2YA. Both IG
> Markets
> >> > Limited
> >> > > > (register number
> >> > > > > > 195355) and IG
> >> > > > > > > > Index Limited (register number
> >> 114059)
> >> > > are
> >> > > > authorised and
> >> > > > > > regulated by
> >> > > > > > > the
> >> > > > > > > > Financial Conduct Authority.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > The information contained in this email is
> >> strictly
> >> > > > confidential and
> >> > > > > > for the use of the addressee only, unless
> >> otherwise
> >> > > > indicated. If you are
> >> > > > > > not the intended recipient, please do not
> read,
> >> copy,
> >> > use
> >> > > > or disclose to
> >> > > > > > others this message or any attachment. Please
> >> also
> >> > notify
> >> > > > the sender by
> >> > > > > > replying to this email or by telephone
> (+44(020
> >> 7896
> >> > > 0011)
> >> > > > and then
> >> > > > > delete
> >> > > > > > the email and any copies of it. Opinions,
> >> conclusion
> >> > > (etc)
> >> > > > that do not
> >> > > > > > relate to the official business of this
> company
> >> shall
> >> > be
> >> > > > understood as
> >> > > > > > neither given nor endorsed by it. IG is a
> >> trading name
> >> > of
> >> > > > IG Markets
> >> > > > > > Limited (a company registered in England and
> >> Wales,
> >> > > > company number
> >> > > > > > 04008957) and IG Index Limited (a company
> >> registered in
> >> > > > England and
> >> > > > > Wales,
> >> > > > > > company number 01190902). Registered address
> at
> >> Cannon
> >> > > > Bridge House, 25
> >> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> >> Limited
> >> > > > (register number
> >> > > > > > 195355) and IG Index Limited (register number
> >> 114059)
> >> > are
> >> > > > authorised and
> >> > > > > > regulated by the Financial Conduct Authority.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > The information contained in this email is strictly
> confidential
> >> and
> >> > for
> >> > > > the use of the addressee only, unless otherwise indicated. If
> >> you are
> >> > not
> >> > > > the intended recipient, please do not read, copy, use or
> >> disclose to
> >> > > others
> >> > > > this message or any attachment. Please also notify the sender
> by
> >> > replying
> >> > > > to this email or by telephone (+44(020 7896 0011) and then
> >> delete the
> >> > > email
> >> > > > and any copies of it. Opinions, conclusion (etc) that do not
> >> relate to
> >> > > the
> >> > > > official business of this company shall be understood as
> neither
> >> given
> >> > > nor
> >> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> >> company
> >> > > > registered in England and Wales, company number 04008957) and
> IG
> >> Index
> >> > > > Limited (a company registered in England and Wales, company
> >> number
> >> > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >> Hill,
> >> > > > London EC4R 2YA. Both IG Markets Limited (register number
> >> 195355) and
> >> > IG
> >> > > > Index Limited (register number 114059) are authorised and
> >> regulated by
> >> > > the
> >> > > > Financial Conduct Authority.
> >> > > >
> >> > >
> >> >
> >>
> >>
> >> The information contained in this email is strictly confidential and for
> >> the use of the addressee only, unless otherwise indicated. If you are
> not
> >> the intended recipient, please do not read, copy, use or disclose to
> others
> >> this message or any attachment. Please also notify the sender by
> replying
> >> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >> official business of this company shall be understood as neither given
> nor
> >> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> registered in England and Wales, company number 04008957) and IG Index
> >> Limited (a company registered in England and Wales, company number
> >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> >> Index Limited (register number 114059) are authorised and regulated by
> the
> >> Financial Conduct Authority.
> >>
> > The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Sean McCauliff <se...@gmail.com>.
Instead of making a Header type visible in the producer or consumer
record, the API could use Iterator<Map.Entry<String, byte[]>>. This
would make the API more compatible with the multimap implementations
I'm aware of (apache commons and google guava) should one want to use
an existing implementation of a multimap data structure at a later
date.
Sean
On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com> wrote:
> I think I'd like some opinions from some of the active chaps at LinkedIn so it's not just a two way decision (or any others for that matter).
>
> Any opinions on the below or no real concerns / indifferent by anyone with this change?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Tuesday, February 21, 2017 6:59:38 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Jun's suggestion gives us the best of both worlds. Because the headers are
> the last field in the record and you have the total size of the record,
> then you can easily get the size of headers in bytes. So they can be copied
> without parsing and we can do the lazy initialization without an extra cost.
>
> The message set notion has changed in KIP-98 so that the message set can
> have its own fields. See here for more detail:
> https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
> What we're proposing to change here is the schema for the message (aka
> record) itself. We'll update that document depending on the outcome of the
> discussion here.
>
> -Jason
>
>
> On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> This works whilst no other feature needs similar. I prefer to keep it
>> contained/encapsulated it then allows the bytes of the headers to also be
>> copied transferred as one without any parsing what so ever, but if you
>> really feel strongly on the below ill go with it, just want to flag this
>> concern.
>>
>> Btw, the below notation seems to be different to the notation used in
>> previous kips like kip-32 which I took the notation from, where the length
>> of the following bytes for byte arrays is clearly defined. So im 100% on
>> what you propose could you define it in this original styled notation?
>>
>>
>> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>>
>> Hey Jun, that's a good thought. Maybe then the full schema could look
>> like
>> this:
>>
>> Record =>
>> Length => uintVar
>> Attributes => int8
>> TimestampDelta => intVar
>> OffsetDelta => uintVar
>> Key => bytes
>> Value => bytes
>> Headers => [HeaderKey HeaderValue]
>> HeaderKey => string
>> HeaderValue => bytes
>>
>> (where the bytes and string type use a variable length size field)
>>
>> Does that work for you, Michael?
>>
>> -Jason
>>
>> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > Regarding the overhead of skipping the headers in the message.
>> Another way
>> > could be to add the headers at the end of the message after the
>> value part.
>> > This way, there is no need to skip the headers unless you really
>> need it.
>> > This will allow us to represent the headers explicitly as an array of
>> > key/value pairs without the skipping overhead when headers are not
>> needed.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
>> >
>> > wrote:
>> >
>> > > Michael,
>> > >
>> > > Here's a link to the branch with the message format changes:
>> > > https://github.com/confluentinc/kafka/tree/
>> exactly-once-message-format.
>> > > The
>> > > class to modify to add headers is EosLogRecord (name will
>> definitely
>> > change
>> > > prior to merging in trunk). This branch uses attributes to
>> indicate null
>> > > key and value. I really don't have a strong preference for or
>> against
>> > this
>> > > as long as we're consistent. I do feel it's kind of a pity to need
>> the
>> > > signed version of the varint just to be able to indicate null. For
>> > headers,
>> > > it's an easier call because we don't actually need to represent
>> null
>> > > headers (using size zero will work fine).
>> > >
>> > > -Jason
>> > >
>> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > We can make it, implement Iterable<Header>, no concerns on this.
>> > > >
>> > > > Ive noted this on the KIP.
>> > > >
>> > > >
>> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > > Btw Jason, can you share the branch with the VarInts
>> helper/util
>> > > class
>> > > > that KIP-98 will be introducing and also will have the current
>> modified
>> > > > (key and value lengths) message, so we can look at, thanks.
>> > > >
>> > > > On 20/02/2017, 01:11, "Michael Pearce" <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > > >
>> > > > RE “ProducerRecord and ConsumerRecord are immutable
>> > currently”. I
>> > > > beg to differ on this, if I set a byte[] as my value of my
>> record, I
>> > can
>> > > > later call value() and then alter the bytes in the array. As
>> such it is
>> > > > very much mutable.
>> > > >
>> > > > Headers also will anyhow by nature be mutable on
>> produce, as
>> > you
>> > > > add to them.
>> > > >
>> > > >
>> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
>> jason@confluent.io>
>> > > > wrote:
>> > > >
>> > > > Re; having a dedicated Headers object. I think it
>> makes
>> > > sense.
>> > > > Maybe it
>> > > > could implement Iterable<Header>?
>> > > >
>> > > > One thing I'm not too sure about is mutability. Both
>> the
>> > > > ProducerRecord and
>> > > > ConsumerRecord are immutable currently. It would be
>> nice to
>> > > > keep it that
>> > > > way. Perhaps Headers could itself be immutable and
>> passed
>> > > > through the
>> > > > constructors?
>> > > >
>> > > > Just going through how this is, I don’t believe
>> this would
>> > > > provide much
>> > > > > saving. If issue trying to resolve is around
>> > object/garbage
>> > > > creation of
>> > > > > Map.Entry objects.
>> > > >
>> > > >
>> > > > It's definitely better because we only allocate the
>> space
>> > we
>> > > > will use, but
>> > > > I assume most of the actual memory cost is in the
>> header
>> > data
>> > > > itself which
>> > > > we can't do much about. Maybe we can start with the
>> simple
>> > > > array
>> > > > representation and optimize from there after we find
>> it's
>> > > > worthwhile to do
>> > > > so? If we use the Headers object, we can change this
>> later
>> > > > without breaking
>> > > > the API.
>> > > >
>> > > > -Jason
>> > > >
>> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
>> > > > radai.rosenblatt@gmail.com> wrote:
>> > > >
>> > > > > "(By the way, doesn't it feel a bit odd that we
>> seem to
>> > be
>> > > > designing a
>> > > > > feature which is optimized for people not using?)"
>> > > > >
>> > > > > very. this (i claim :-) ) is the result of very
>> intense
>> > > > opposition to the
>> > > > > usefulness of the feature early on, and not a real
>> design
>> > > > goal
>> > > > >
>> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Whilst if having an array of a Header[] would
>> mean
>> > > getting
>> > > > this through,
>> > > > > > im happy changing this, as already done.
>> > > > > >
>> > > > > > Just going through how this is, I don’t believe
>> this
>> > > > would provide much
>> > > > > > saving. If issue trying to resolve is around
>> > > > object/garbage creation of
>> > > > > > Map.Entry objects.
>> > > > > >
>> > > > > > All you’re doing here is replacing the equiv of a
>> > > > HashMap.Node
>> > > > > (Map.Entry)
>> > > > > > which simply holds reference to key and value
>> objects
>> > > with
>> > > > a custom
>> > > > > variant.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com> wrote:
>> > > > > >
>> > > > > > On point 1 & 2 Ive updated KIP to show
>> varints (and
>> > > > removed the bit
>> > > > > > flag). (on the assumption KIP 98 is getting the
>> > agreement
>> > > > the protocol is
>> > > > > > moving from int32 to varInts as standard)
>> > > > > >
>> > > > > > On point 3 ive updated to use an array of
>> Header
>> > > > class, instead of a
>> > > > > > MultiMap in the Headers class object
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > On points 1 and 2 I agree.
>> > > > > >
>> > > > > > This also affects kip-98, I should
>> expect this
>> > > > resolved before
>> > > > > > that vote also passes. If it is accepted there
>> (I’m
>> > > > assuming this is
>> > > > > > getting discussed on that KIP? As you’re driving
>> the
>> > move
>> > > > to VarInts), I
>> > > > > am
>> > > > > > happy to make this KIP will simply follow suit to
>> > > whatever
>> > > > is agreed in
>> > > > > > KIP-98.
>> > > > > >
>> > > > > > On 3) Agreed this is a simpler form, as
>> long as
>> > > no
>> > > > one is
>> > > > > > expecting hashmap lookup performance ( O(1) ), I
>> still
>> > > > would prefer an
>> > > > > > encapsulated class, so if we find that holding
>> it as an
>> > > > array in future
>> > > > > is
>> > > > > > causing some perf issues, the internals are not
>> exposed
>> > > to
>> > > > end users,
>> > > > > > allowing the internal structure to move to a map.
>> > > > > >
>> > > > > > Can we compromise on?
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Headers headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Headers {
>> > > > > > Header[] headers;
>> > > > > >
>> > > > > > add(String key, byte[] value)
>> > > > > > Collection<byte[]> get(String key)
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
>> > > > jason@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > >
>> > > > > >
>> > > > > > This is exactly what I have been
>> trying to
>> > > get
>> > > > at. The use
>> > > > > > cases documented
>> > > > > > here are middleware:
>> > > > > > https://cwiki.apache.org/
>> > > > confluence/display/KAFKA/A+
>> > > > > > Case+for+Kafka+Headers.
>> > > > > > If headers are intended for the
>> application
>> > > > space as well,
>> > > > > the
>> > > > > > document
>> > > > > > should be updated accordingly so
>> that it is
>> > > an
>> > > > explicit
>> > > > > design
>> > > > > > goal and not
>> > > > > > an unfortunate design byproduct.
>> There may
>> > be
>> > > > some
>> > > > > > disagreement on whether
>> > > > > > it _should_ be a design goal, but it
>> may
>> > not
>> > > > matter much
>> > > > > since
>> > > > > > the current
>> > > > > > interceptor API is probably
>> insufficient
>> > for
>> > > > middleware
>> > > > > > applications (I
>> > > > > > haven't thought of a way to do this
>> that
>> > > isn't
>> > > > cumbersome).
>> > > > > >
>> > > > > > That aside, the three unresolved
>> points for
>> > > me
>> > > > are the
>> > > > > > following:
>> > > > > >
>> > > > > > 1. The use of varints. As a one-off
>> for use
>> > > > only with record
>> > > > > > headers, the
>> > > > > > case was weak, but if we are using
>> them
>> > > > throughout the
>> > > > > message
>> > > > > > format, then
>> > > > > > we should do so here as well. The
>> > additional
>> > > > complexity is
>> > > > > > minimal and
>> > > > > > early performance testing fully
>> justifies
>> > > > their use.
>> > > > > >
>> > > > > > 2. If varints are used, the case for
>> using
>> > > > attributes to
>> > > > > > indicate null
>> > > > > > headers also becomes weak. We only
>> add one
>> > > > additional byte in
>> > > > > > each message
>> > > > > > if there are no headers. Whatever
>> the case,
>> > > > let us be
>> > > > > > consistent with how
>> > > > > > we treat null keys and values.
>> > > > > >
>> > > > > > 3. We have apparently agreed that the
>> > broker
>> > > > will validate
>> > > > > > headers (please
>> > > > > > add this to the KIP). That being the
>> case,
>> > I
>> > > > would prefer to
>> > > > > > use Kafka's
>> > > > > > conventional format for arrays. The
>> > downside
>> > > > is that it takes
>> > > > > > more work to
>> > > > > > skip over the headers on the
>> consumer,
>> > though
>> > > > it's unclear if
>> > > > > > the cost
>> > > > > > would matter in practice. Some
>> concern was
>> > > > previously
>> > > > > > expressed about the
>> > > > > > allocation of maps. An alternative
>> would be
>> > > to
>> > > > use arrays,
>> > > > > i.e.
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Header[] headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > > This would work nicely with the
>> > conventional
>> > > > array format and
>> > > > > > my guess is
>> > > > > > it would obviate the need do any lazy
>> > > > initialization. If we
>> > > > > > use the map as
>> > > > > > is currently documented, then it is
>> > possible
>> > > > with either
>> > > > > > representation to
>> > > > > > slice the headers and initialize them
>> > lazily.
>> > > > Either way, it
>> > > > > > might be a
>> > > > > > good idea to use a separate object to
>> > > > represent the headers
>> > > > > in
>> > > > > > case we need
>> > > > > > to modify it in the future in some
>> way.
>> > > > > >
>> > > > > > (By the way, doesn't it feel a bit
>> odd that
>> > > we
>> > > > seem to be
>> > > > > > designing a
>> > > > > > feature which is optimized for
>> people not
>> > > > using?)
>> > > > > >
>> > > > > >
>> > > > > > If we can resolve these points, then
>> at
>> > least
>> > > > you will get my
>> > > > > > vote.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jason
>> > > > > >
>> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
>> radai <
>> > > > > > radai.rosenblatt@gmail.com> wrote:
>> > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > > > IIUC samze currently has headers
>> "in-V"
>> > and
>> > > > would just
>> > > > > > switch over to kafka
>> > > > > > > headers if they exist.
>> > > > > > > im sure plenty of other users of
>> kafka
>> > > would
>> > > > have a use for
>> > > > > > headers.
>> > > > > > > im pretty sure use cases exist
>> around
>> > > > shuffling data
>> > > > > > into/out-of kafka
>> > > > > > > (kafka connect or equivalent) where
>> > > metadata
>> > > > from one end
>> > > > > > could copied over
>> > > > > > > to the other (S3, for example uses
>> http
>> > > > headers for
>> > > > > > user-accessible
>> > > > > > > metadata). it will be kafka client
>> code
>> > > > getting/setting
>> > > > > > those headers. not
>> > > > > > > an interceptor.
>> > > > > > >
>> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
>> Michael
>> > > > Pearce <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > For APM single event tracing,
>> need
>> > access
>> > > > to the header
>> > > > > at
>> > > > > > the point of
>> > > > > > > > processing on the processing
>> thread.
>> > > > > > > >
>> > > > > > > > As such interceptors will not
>> work/be
>> > > > suitable for these,
>> > > > > > due to the fact
>> > > > > > > > they act on the ConsumerRecords
>> as a
>> > > > batch, before the
>> > > > > > handling thread
>> > > > > > > can
>> > > > > > > > split out and process per
>> message which
>> > > is
>> > > > the point
>> > > > > these
>> > > > > > tools will
>> > > > > > > need
>> > > > > > > > to continue to transaction
>> tracing.
>> > > > > > > >
>> > > > > > > > Like wise tools and infra pieces
>> will
>> > > need
>> > > > access to the
>> > > > > > message outside
>> > > > > > > > the interceptor realm.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 17/02/2017, 21:26, "Jason
>> > Gustafson" <
>> > > > > > jason@confluent.io> wrote:
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > So the broker will parse the
>> > headers
>> > > > and validate
>> > > > > > them. Good. The
>> > > > > > > only
>> > > > > > > > reason remaining that I can
>> see to
>> > > > leave the headers
>> > > > > > as a byte array
>> > > > > > > > is to
>> > > > > > > > make it easier for the
>> client to
>> > skip
>> > > > past them. Are
>> > > > > > we sure this is
>> > > > > > > > not
>> > > > > > > > premature optimization? Are
>> there
>> > any
>> > > > performance
>> > > > > > results which show
>> > > > > > > > that
>> > > > > > > > this is worthwhile?
>> > > > > > > >
>> > > > > > > > What’s the issue with
>> exposing a
>> > > > method getHeaders on
>> > > > > > the
>> > > > > > > > producer/consumer
>> > > > > > > > > record? It doesn’t break
>> > anything.
>> > > > We don’t need
>> > > > > any
>> > > > > > special
>> > > > > > > version.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > See my previous explanation.
>> What I
>> > > am
>> > > > trying to
>> > > > > > resist is the
>> > > > > > > headers
>> > > > > > > > becoming a general
>> > application-level
>> > > > facility. The
>> > > > > > primary use case
>> > > > > > > as
>> > > > > > > > far
>> > > > > > > > as I can tell is middleware,
>> which
>> > is
>> > > > the use case
>> > > > > > that the
>> > > > > > > > interceptors
>> > > > > > > > are providing.
>> > > > > > > >
>> > > > > > > > Current batch consumer model
>> and
>> > > > consumer
>> > > > > interceptors
>> > > > > > don’t work
>> > > > > > > where
>> > > > > > > > > headers need to be acted
>> on at
>> > per
>> > > > message level at
>> > > > > > time of
>> > > > > > > > processing,
>> > > > > > > > > very case is APM (the core
>> one),
>> > > > where the header
>> > > > > > value is used to
>> > > > > > > > continue
>> > > > > > > > > tracing.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I still don't understand the
>> point
>> > > > about batching.
>> > > > > The
>> > > > > > consumer
>> > > > > > > > records are
>> > > > > > > > exposed as a batch in the
>> consumer
>> > > > interceptor, but
>> > > > > > you can still
>> > > > > > > > iterate
>> > > > > > > > them individually. It is no
>> > different
>> > > > for the
>> > > > > consumer
>> > > > > > API itself.
>> > > > > > > >
>> > > > > > > > -Jason
>> > > > > > > >
>> > > > > > > > On Fri, Feb 17, 2017 at
>> 12:48 PM,
>> > > > Michael Pearce <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Re:
>> > > > > > > > >
>> > > > > > > > > “ The point about
>> creation of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a slice
>> of
>> > bytes
>> > > > until the time
>> > > > > > it is accessed.”
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > >
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:37,
>> "Michael
>> > > > Pearce" <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > What’s the issue with
>> > exposing
>> > > a
>> > > > method
>> > > > > > getHeaders on the
>> > > > > > > > > producer/consumer record?
>> It
>> > > doesn’t
>> > > > break
>> > > > > anything.
>> > > > > > We don’t need
>> > > > > > > > any
>> > > > > > > > > special version.
>> > > > > > > > >
>> > > > > > > > > Current batch consumer
>> model
>> > > and
>> > > > consumer
>> > > > > > interceptors don’t
>> > > > > > > work
>> > > > > > > > > where headers need to be
>> acted on
>> > > at
>> > > > per message
>> > > > > > level at time of
>> > > > > > > > > processing, very case is
>> APM (the
>> > > > core one), where
>> > > > > > the header value
>> > > > > > > > is used
>> > > > > > > > > to continue tracing.
>> JMS/HTTP etc
>> > > > all expose these,
>> > > > > > without issues.
>> > > > > > > > I would
>> > > > > > > > > NOT want to lock this down
>> to
>> > only
>> > > > be usable
>> > > > > > accessible via
>> > > > > > > > interceptors,
>> > > > > > > > > as we’d fail on one of the
>> main
>> > > > goals.
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Mike
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:21,
>> "Jason
>> > > > Gustafson" <
>> > > > > > jason@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > The point about
>> creation
>> > of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a
>> slice of
>> > > > bytes until the
>> > > > > > time it is
>> > > > > > > > accessed.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > Yes exactly we
>> have
>> > > access
>> > > > to the records
>> > > > > > thus why the
>> > > > > > > > header
>> > > > > > > > > should be
>> > > > > > > > > > accessible via
>> it and
>> > not
>> > > > hidden for only
>> > > > > > interceptors to
>> > > > > > > > access.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > As explained
>> above, the
>> > > > point is to make
>> > > > > the
>> > > > > > intended usage
>> > > > > > > > clear.
>> > > > > > > > > Applications should
>> > > continue
>> > > > to rely on the
>> > > > > > key/value
>> > > > > > > fields
>> > > > > > > > to
>> > > > > > > > > serialize
>> > > > > > > > > their own headers,
>> and it
>> > > > would be more
>> > > > > > ideal if we can
>> > > > > > > avoid
>> > > > > > > > > leaking
>> > > > > > > > > third-party
>> headers into
>> > > > applications. This
>> > > > > > is difficult to
>> > > > > > > > do
>> > > > > > > > > with the
>> > > > > > > > > current
>> interceptors
>> > > because
>> > > > they share the
>> > > > > > record objects
>> > > > > > > > with
>> > > > > > > > > the common
>> > > > > > > > > API. What I had in
>> mind
>> > is
>> > > > something like
>> > > > > an
>> > > > > > extension of
>> > > > > > > the
>> > > > > > > > > current
>> > > > > > > > > interceptors which
>> > exposed
>> > > a
>> > > > different
>> > > > > > object (e.g.
>> > > > > > > > > `RecordAndHeaders`).
>> > > > > > > > > The challenge is
>> for
>> > > MM-like
>> > > > use cases. Let
>> > > > > > me see if I can
>> > > > > > > > come
>> > > > > > > > > up with a
>> > > > > > > > > concrete proposal
>> for
>> > that
>> > > > problem.
>> > > > > > > > >
>> > > > > > > > > -Jason
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 17,
>> 2017 at
>> > > > 11:55 AM, Michael
>> > > > > > Pearce <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I am happy to
>> move the
>> > > > definition of the
>> > > > > > header into the
>> > > > > > > > message
>> > > > > > > > > body, but
>> > > > > > > > > > would cause us
>> not to
>> > > lazy
>> > > > > > initialise/parse the headers,
>> > > > > > > as
>> > > > > > > > > obviously, we
>> > > > > > > > > > would have to
>> traverse
>> > > > these reading the
>> > > > > > message.
>> > > > > > > > > >
>> > > > > > > > > > This was
>> actually one
>> > of
>> > > > Jay’s requests:
>> > > > > > > > > >
>> > > > > > > > > > “ 2. I think
>> we
>> > should
>> > > > think about
>> > > > > > creating the Map
>> > > > > > > > lazily to
>> > > > > > > > > avoid
>> > > > > > > > > > parsing out
>> all the
>> > > > headers into
>> > > > > > little objects.
>> > > > > > > > HashMaps
>> > > > > > > > > themselves
>> > > > > > > > > > are
>> > > > > > > > > > kind of
>> expensive
>> > and
>> > > > the consumer is
>> > > > > > very perf
>> > > > > > > > sensitive so
>> > > > > > > > > and making
>> > > > > > > > > > gazillions of
>> > > hashmaps
>> > > > that may or
>> > > > > may
>> > > > > > not get used
>> > > > > > > is
>> > > > > > > > > probably a bad
>> > > > > > > > > > idea.”
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 17/02/2017,
>> 19:44,
>> > > > "Michael Pearce" <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Yes exactly
>> we have
>> > > > access to the
>> > > > > > records thus why
>> > > > > > > the
>> > > > > > > > > header should
>> > > > > > > > > > be accessible
>> via it
>> > and
>> > > > not hidden for
>> > > > > > only interceptors
>> > > > > > > > to
>> > > > > > > > > access.
>> > > > > > > > > >
>> > > > > > > > > > Sent using
>> OWA for
>> > > > iPhone
>> > > > > > > > > >
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > From: Magnus
>> > > Edenhill <
>> > > > > > magnus@edenhill.se>
>> > > > > > > > > > Sent: Friday,
>> > > February
>> > > > 17, 2017
>> > > > > > 7:34:49 PM
>> > > > > > > > > > To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > Subject: Re:
>> > > [DISCUSS]
>> > > > KIP-82 - Add
>> > > > > > Record Headers
>> > > > > > > > > >
>> > > > > > > > > > Big +1 on
>> VarInts.
>> > > > > > > > > > CPUs are
>> fast,
>> > memory
>> > > > is slow.
>> > > > > > > > > >
>> > > > > > > > > > I agree with
>> Jason
>> > > > that we'll want to
>> > > > > > continue
>> > > > > > > > verifying
>> > > > > > > > > messages,
>> > > > > > > > > > including
>> their
>> > > > headers, so while I
>> > > > > > appreciate the
>> > > > > > > > idea of
>> > > > > > > > > the opaque
>> > > > > > > > > > header blob
>> it
>> > won't
>> > > > be useful in
>> > > > > > practice.
>> > > > > > > > > >
>> > > > > > > > > > /Magnus
>> > > > > > > > > >
>> > > > > > > > > > 2017-02-17
>> 10:41
>> > > > GMT-08:00 Jason
>> > > > > > Gustafson <
>> > > > > > > > > jason@confluent.io>:
>> > > > > > > > > >
>> > > > > > > > > > > Sorry, my
>> > mistake.
>> > > > The consumer
>> > > > > > interceptor is per
>> > > > > > > > batch,
>> > > > > > > > > though I'm
>> > > > > > > > > > not
>> > > > > > > > > > > sure
>> that's an
>> > > > actual limitation
>> > > > > > since you still
>> > > > > > > have
>> > > > > > > > > access to the
>> > > > > > > > > > > individual
>> > records.
>> > > > > > > > > > >
>> > > > > > > > > > > -Jason
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri,
>> Feb 17,
>> > > 2017
>> > > > at 10:39 AM,
>> > > > > > Jason Gustafson <
>> > > > > > > > > >
>> jason@confluent.io>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Re
>> headers as
>> > > byte
>> > > > array and
>> > > > > > future use by
>> > > > > > > broker.
>> > > > > > > > This
>> > > > > > > > > doesn't
>> > > > > > > > > > take away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yeah, I
>> didn't
>> > > say
>> > > > it was
>> > > > > > difficult, only
>> > > > > > > awkward.
>> > > > > > > > You
>> > > > > > > > > wouldn't
>> > > > > > > > > > write the
>> > > > > > > > > > > > schema
>> that way
>> > > if
>> > > > you were
>> > > > > > planning to use it on
>> > > > > > > > the
>> > > > > > > > > brokers from
>> > > > > > > > > > the
>> > > > > > > > > > > >
>> beginning. Note
>> > > > also that one of
>> > > > > > the benefits of
>> > > > > > > > letting
>> > > > > > > > > the broker
>> > > > > > > > > > > >
>> understand
>> > > headers
>> > > > is that it can
>> > > > > > validate that
>> > > > > > > > they are
>> > > > > > > > > properly
>> > > > > > > > > > > >
>> formatted. If
>> > > cost
>> > > > is the only
>> > > > > > concern, we should
>> > > > > > > > > confirm its
>> > > > > > > > > > impact
>> > > > > > > > > > > > through
>> > > > performance testing.
>> > > > > > > > > > > >
>> > > > > > > > > > > > One of
>> the key
>> > > use
>> > > > cases requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'm not
>> sure I
>> > > > understand the
>> > > > > > point about
>> > > > > > > batching.
>> > > > > > > > > Interceptors
>> > > > > > > > > > are
>> > > > > > > > > > > > applied
>> > > > per-message, right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > My
>> intent on
>> > > > interceptors is to
>> > > > > > keep the usage of
>> > > > > > > > headers
>> > > > > > > > > > well-defined so
>> > > > > > > > > > > > that
>> they don't
>> > > > start leaking
>> > > > > > unnecessarily into
>> > > > > > > > > applications. My
>> > > > > > > > > > guess
>> > > > > > > > > > > is
>> > > > > > > > > > > > that it's
>> > > probably
>> > > > inevitable,
>> > > > > but
>> > > > > > isolating it
>> > > > > > > in
>> > > > > > > > the
>> > > > > > > > > > interceptors
>> would
>> > > > > > > > > > > > at least
>> give
>> > > > people a second
>> > > > > > thought before
>> > > > > > > > deciding to
>> > > > > > > > > use it.
>> > > > > > > > > > The main
>> > > > > > > > > > > >
>> challenge in my
>> > > > mind is figuring
>> > > > > > out how an MM
>> > > > > > > use
>> > > > > > > > case
>> > > > > > > > > would
>> > > > > > > > > > work. It
>> > > > > > > > > > > > would be
>> more
>> > > > cumbersome to
>> > > > > > replicate headers
>> > > > > > > > through an
>> > > > > > > > > > interceptor,
>> > > > > > > > > > > > though
>> arguably
>> > > MM
>> > > > should be
>> > > > > > working at a lower
>> > > > > > > > level
>> > > > > > > > > anyway.
>> > > > > > > > > > > >
>> > > > > > > > > > > > -Jason
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri,
>> Feb 17,
>> > > > 2017 at 10:16 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Re
>> headers
>> > > > available on the
>> > > > > > record va
>> > > > > > > > interceptors only
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> One of
>> the key
>> > > > use cases
>> > > > > requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> This
>> anyhow is
>> > > > similar to
>> > > > > > jms/http/amqp where
>> > > > > > > > headers
>> > > > > > > > > are
>> > > > > > > > > > available to
>> > > > > > > > > > > >>
>> consuming
>> > > > applications.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Re
>> headers as
>> > > > byte array and
>> > > > > > future use by
>> > > > > > > > broker. This
>> > > > > > > > > doesn't
>> > > > > > > > > > take
>> > > > > > > > > > > away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Sent
>> using OWA
>> > > > for iPhone
>> > > > > > > > > > > >>
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > > >> From:
>> Jason
>> > > > Gustafson <
>> > > > > > jason@confluent.io>
>> > > > > > > > > > > >> Sent:
>> Friday,
>> > > > February 17, 2017
>> > > > > > 5:55:42 PM
>> > > > > > > > > > > >> To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > > >>
>> Subject: Re:
>> > > > [DISCUSS] KIP-82 -
>> > > > > > Add Record
>> > > > > > > Headers
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Yes,
>> that is
>> > > what
>> > > > we are
>> > > > > > proposing, so using
>> > > > > > > > varints
>> > > > > > > > > for headers
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >>
>> consistent
>> > with
>> > > > the rest of the
>> > > > > > message. We have
>> > > > > > > > used
>> > > > > > > > > static
>> > > > > > > > > > helper
>> > > > > > > > > > > >> methods
>> > > > > > > > > > > >> in our
>> > prototype
>> > > > implementation.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> The
>> cost of
>> > > > parsing, we want to
>> > > > > > parse/interpret
>> > > > > > > > the
>> > > > > > > > > headers
>> > > > > > > > > > lazily (this
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > a key
>> point
>> > > > brought up earlier
>> > > > > > in discussions)
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I'm a
>> bit
>> > > > skeptical of this. Has
>> > > > > > anyone done the
>> > > > > > > > > performance
>> > > > > > > > > > testing? I
>> > > > > > > > > > > >> can
>> > > > > > > > > > > >> probably
>> > > > implement it and test
>> > > > > it
>> > > > > > if no one else
>> > > > > > > > has. I
>> > > > > > > > > was also
>> > > > > > > > > > under
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> impression
>> > that
>> > > > there may be use
>> > > > > > cases down the
>> > > > > > > > road
>> > > > > > > > > where the
>> > > > > > > > > > broker
>> > > > > > > > > > > >> would
>> > > > > > > > > > > >> need to
>> > > interpret
>> > > > headers. That
>> > > > > > wouldn't be off
>> > > > > > > > the
>> > > > > > > > > table in the
>> > > > > > > > > > future
>> > > > > > > > > > > if
>> > > > > > > > > > > >> it's
>> > represented
>> > > > as bytes, but
>> > > > > it
>> > > > > > would be quite
>> > > > > > > > a bit
>> > > > > > > > > more
>> > > > > > > > > > awkward,
>> > > > > > > > > > > >> right?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> By the
>> way,
>> > one
>> > > > question I have
>> > > > > > been wondering
>> > > > > > > > about. My
>> > > > > > > > > > understanding
>> > > > > > > > > > > is
>> > > > > > > > > > > >> that
>> headers
>> > are
>> > > > primarily for
>> > > > > > use cases where a
>> > > > > > > > > third-party
>> > > > > > > > > > components
>> > > > > > > > > > > >> wants to
>> > enrich
>> > > > messages without
>> > > > > > needing to
>> > > > > > > > understand
>> > > > > > > > > or modify
>> > > > > > > > > > the
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> of the
>> message
>> > > > key and value.
>> > > > > For
>> > > > > > the
>> > > > > > > applications
>> > > > > > > > > which directly
>> > > > > > > > > > > produce
>> > > > > > > > > > > >> and
>> consume
>> > the
>> > > > messages and
>> > > > > > control the
>> > > > > > > key/value
>> > > > > > > > > schema
>> > > > > > > > > > directly, it
>> > > > > > > > > > > >> seems
>> we would
>> > > > rather have them
>> > > > > > implement
>> > > > > > > headers
>> > > > > > > > > directly in
>> > > > > > > > > > their own
>> > > > > > > > > > > >> schema.
>> > > Supposing
>> > > > for the sake
>> > > > > of
>> > > > > > argument that
>> > > > > > > > it was
>> > > > > > > > > possible,
>> > > > > > > > > > my
>> > > > > > > > > > > >>
>> question is
>> > > > whether it be
>> > > > > > sufficient to expose
>> > > > > > > the
>> > > > > > > > > headers in the
>> > > > > > > > > > > >>
>> interceptor
>> > API
>> > > > and not in the
>> > > > > > common API?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> -Jason
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Fri,
>> Feb
>> > 17,
>> > > > 2017 at 3:26 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > On
>> the point
>> > > of
>> > > > varInts
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Cheers
>> > > > > > > > > > > >> > Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > 17/02/2017,
>> > > > 11:22, "Michael
>> > > > > > Pearce" <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On the
>> > > > point re: headers
>> > > > > in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > being a
>> > > > > > > > > > byte
>> > > > > > > > > > > array
>> > > > > > > > > > > >> > and
>> not a
>> > > count
>> > > > of elements
>> > > > > > followed by the
>> > > > > > > > elements.
>> > > > > > > > > Again
>> > > > > > > > > > this was
>> > > > > > > > > > > >> >
>> > > > discussed/argued previously.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> It was
>> > > > agreed on for a few
>> > > > > > reasons some of
>> > > > > > > > which
>> > > > > > > > > you have
>> > > > > > > > > > > obviously
>> > > > > > > > > > > >> >
>> picked up
>> > on:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Broker
>> > is
>> > > > able to pass it
>> > > > > > through opaquely
>> > > > > > > > > > > >> >
>> The cost
>> > > of
>> > > > parsing, we
>> > > > > > want to
>> > > > > > > > parse/interpret
>> > > > > > > > > the headers
>> > > > > > > > > > lazily
>> > > > > > > > > > > >> > (this
>> is a
>> > key
>> > > > point brought
>> > > > > up
>> > > > > > earlier in
>> > > > > > > > > discussions)
>> > > > > > > > > > > >> >
>> Headers
>> > > can
>> > > > be copied from
>> > > > > > consumer record
>> > > > > > > > to
>> > > > > > > > > producer
>> > > > > > > > > > record (aka
>> > > > > > > > > > > >> > mirror
>> > makers
>> > > > etc) without
>> > > > > > parsing if no
>> > > > > > > > changes are
>> > > > > > > > > being made
>> > > > > > > > > > or
>> > > > > > > > > > > being
>> > > > > > > > > > > >> >
>> looked at.
>> > > > > > > > > > > >> >
>> Keeps
>> > the
>> > > > broker agnostic
>> > > > > > to the format
>> > > > > > > > > > > >> >
>> You need
>> > > an
>> > > > int32 either
>> > > > > > for the byte size
>> > > > > > > > of the
>> > > > > > > > > headers,
>> > > > > > > > > > or for
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > count
>> of
>> > > > elements, so
>> > > > > overheads
>> > > > > > are the same,
>> > > > > > > > but
>> > > > > > > > > with going
>> > > > > > > > > > with an
>> > > > > > > > > > > >> opaque
>> > > > > > > > > > > >> > byte
>> array
>> > has
>> > > > the above
>> > > > > > advantages.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Cheers
>> > > > > > > > > > > >> >
>> Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > > > 17/02/2017, 02:50,
>> > > > > > "Jason Gustafson" <
>> > > > > > > > > jason@confluent.io
>> > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > Sorry,
>> > > > should have
>> > > > > > noted that the
>> > > > > > > > performance
>> > > > > > > > > testing
>> > > > > > > > > > was done
>> > > > > > > > > > > >> > using
>> the
>> > > > > > > > > > > >> >
>> > > > producer performance
>> > > > > > tool shipped with
>> > > > > > > > Kafka.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > -Jason
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On
>> > > Thu,
>> > > > Feb 16, 2017
>> > > > > at
>> > > > > > 6:44 PM, Jason
>> > > > > > > > > Gustafson <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> >
>> > Hey
>> > > > Nacho,
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > I've
>> > > > compared
>> > > > > > performance of our
>> > > > > > > > KIP-98
>> > > > > > > > > > implementation
>> with
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> without
>> > > > > > > > > > > >> >
>> >
>> > > > varints. For
>> > > > > messages
>> > > > > > around 128
>> > > > > > > > bytes, we
>> > > > > > > > > see an
>> > > > > > > > > > increase
>> > > > > > > > > > > in
>> > > > > > > > > > > >> >
>> throughput
>> > of
>> > > > > > > > > > > >> >
>> >
>> > > about
>> > > > 30% using the
>> > > > > > default
>> > > > > > > > configuration
>> > > > > > > > > settings.
>> > > > > > > > > > At 256
>> > > > > > > > > > > >> >
>> bytes, the
>> > > > > > > > > > > >> >
>> >
>> > > > increase is around
>> > > > > > 16%. Obviously
>> > > > > > > the
>> > > > > > > > > performance
>> > > > > > > > > > converge
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> messages get
>> > > > > > > > > > > >> >
>> >
>> > > > larger, but it seems
>> > > > > > well worth the
>> > > > > > > > cost.
>> > > > > > > > > Note that
>> > > > > > > > > > we are
>> > > > > > > > > > > >> also
>> > > > > > > > > > > >> >
>> seeing a
>> > > > > > > > > > > >> >
>> >
>> > > > substantial
>> > > > > > performance increase
>> > > > > > > > against
>> > > > > > > > > trunk
>> > > > > > > > > > primarily
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >
>> > much
>> > > > more efficient
>> > > > > > packing that
>> > > > > > > > varints
>> > > > > > > > > provide us.
>> > > > > > > > > > > Anything
>> > > > > > > > > > > >> >
>> adding to
>> > > > > > > > > > > >> >
>> >
>> > > > message overhead,
>> > > > > > such as record
>> > > > > > > > headers,
>> > > > > > > > > would only
>> > > > > > > > > > > increase
>> > > > > > > > > > > >> > the
>> relative
>> > > > > > > > > > > >> >
>> >
>> > > > difference. (Of
>> > > > > > course take these
>> > > > > > > > numbers
>> > > > > > > > > with a
>> > > > > > > > > > grain of
>> > > > > > > > > > > salt
>> > > > > > > > > > > >> > since
>> I have
>> > > > > > > > > > > >> >
>> >
>> > only
>> > > > used the
>> > > > > default
>> > > > > > settings with
>> > > > > > > > both
>> > > > > > > > > the producer
>> > > > > > > > > > and
>> > > > > > > > > > > >> broker
>> > > > > > > > > > > >> > on my
>> > > > > > > > > > > >> >
>> >
>> > > local
>> > > > machine. We
>> > > > > > intend to provide
>> > > > > > > > more
>> > > > > > > > > extensive
>> > > > > > > > > > > performance
>> > > > > > > > > > > >> >
>> details as
>> > > > > > > > > > > >> >
>> >
>> > part
>> > > > of the work for
>> > > > > > KIP-98.)
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > The
>> > > > implementation
>> > > > > we
>> > > > > > are using is
>> > > > > > > > from
>> > > > > > > > > protobuf (
>> > > > > > > > > > > >> >
>> >
>> > > > > > https://developers.google.com/
>> > > > > > > > > > >
>> > > > protocol-buffers/docs/encoding
>> > > > > > > > > > > >> ),
>> > > > > > > > > > > >> > which
>> is
>> > > > > > > > > > > >> >
>> >
>> > also
>> > > > used in HBase.
>> > > > > > It is trivial to
>> > > > > > > > > implement and as
>> > > > > > > > > > far
>> > > > > > > > > > > as I
>> > > > > > > > > > > >> > know
>> doesn't
>> > > > > > > > > > > >> >
>> >
>> > > > suffer from the
>> > > > > > aliasing problem you
>> > > > > > > > are
>> > > > > > > > > describing. I
>> > > > > > > > > > > checked
>> > > > > > > > > > > >> > with
>> Magnus
>> > > > > > > > > > > >> >
>> >
>> > (the
>> > > > author of
>> > > > > > librdkafka) and he
>> > > > > > > > agreed
>> > > > > > > > > that the
>> > > > > > > > > > savings
>> > > > > > > > > > > >> seemed
>> > > > > > > > > > > >> > worth
>> the
>> > > > > > > > > > > >> >
>> >
>> > cost
>> > > > of
>> > > > > > implementation.
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > -Jason
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > On
>> > > > Thu, Feb 16, 2017
>> > > > > > at 4:32 PM,
>> > > > > > > > Ignacio
>> > > > > > > > > Solis <
>> > > > > > > > > > > >>
>> > isolis@igso.net
>> > > >
>> > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >>
>> > > > -VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > I'm
>> > > > one of the
>> > > > > > people (if not the
>> > > > > > > > most)
>> > > > > > > > > opposed to
>> > > > > > > > > > VarInts.
>> > > > > > > > > > > >> >
>> VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > have
>> > > > a place, but
>> > > > > > this is not it.
>> > > > > > > > (We
>> > > > > > > > > had a large
>> > > > > > > > > > > >>
>> discussion
>> > > > > > > > > > > >> > about
>> > > > > > > > > > > >> >
>> >>
>> > > them
>> > > > at the
>> > > > > > beginning of KIP-82
>> > > > > > > time)
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > anybody has real
>> > > > > > life
>> > > > > > > performance
>> > > > > > > > > numbers of
>> > > > > > > > > > VarInts
>> > > > > > > > > > > >> >
>> improving
>> > > > > > > > > > > >> >
>> >>
>> > > > things or
>> > > > > > significantly reducing
>> > > > > > > > resources
>> > > > > > > > > I would
>> > > > > > > > > > like to
>> > > > > > > > > > > >> know
>> > > > > > > > > > > >> > what
>> > > > > > > > > > > >> >
>> >>
>> > > that
>> > > > case may be.
>> > > > > > Yes, you can save
>> > > > > > > > some
>> > > > > > > > > bytes here
>> > > > > > > > > > and
>> > > > > > > > > > > >> there,
>> > > > > > > > > > > >> > but
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > is probably
>> > > > > > insignificant to
>> > > > > > > the
>> > > > > > > > > overall system
>> > > > > > > > > > > behavior
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> >
>> >>
>> > > > storage
>> > > > > > requirements. -- I say
>> > > > > > > this
>> > > > > > > > with
>> > > > > > > > > respect to
>> > > > > > > > > > using
>> > > > > > > > > > > >> >
>> VarInts in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > protocol
>> > > > > itself,
>> > > > > > not as part of
>> > > > > > > > the
>> > > > > > > > > data.
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > VarInts require you
>> > > > > > to parse the
>> > > > > > > Int
>> > > > > > > > > before using it
>> > > > > > > > > > and
>> > > > > > > > > > > >> >
>> depending on
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > encoding they
>> > > > > > can suffer from
>> > > > > > > > aliasing
>> > > > > > > > > (multiple
>> > > > > > > > > > > >> >
>> > > representations
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > the same
>> > > > > value).
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > Why
>> > > > add complexity?
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > Nacho
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > On
>> > > > Thu, Feb 16,
>> > > > > 2017
>> > > > > > at 10:29 AM,
>> > > > > > > > Colin
>> > > > > > > > > McCabe <
>> > > > > > > > > > > >> >
>> > > > cmccabe@apache.org>
>> > > > > > > > > > > >> >
>> >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >> >
>> > > +1
>> > > > for varints
>> > > > > > here-- it would
>> > > > > > > save
>> > > > > > > > > quite a bit of
>> > > > > > > > > > space.
>> > > > > > > > > > > >> > They
>> are
>> > > > > > > > > > > >> >
>> >> >
>> > > > pretty quick to
>> > > > > > implement as
>> > > > > > > well.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > > I
>> > > > think it makes
>> > > > > > sense for values
>> > > > > > > > to be
>> > > > > > > > > byte
>> > > > > > > > > > arrays.
>> > > > > > > > > > > Users
>> > > > > > > > > > > >> > might
>> want
>> > > > > > > > > > > >> >
>> >> >
>> > > to
>> > > > attach
>> > > > > > arbitrary payloads;
>> > > > > > > they
>> > > > > > > > > shouldn't be
>> > > > > > > > > > forced to
>> > > > > > > > > > > >> >
>> serialize
>> > > > > > > > > > > >> >
>> >> >
>> > > > everything to
>> > > > > Java
>> > > > > > strings.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > best,
>> > > > > > > > > > > >> >
>> >> >
>> > > > Colin
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > On
>> > > > Thu, Feb 16,
>> > > > > > 2017, at 09:52,
>> > > > > > > > Jason
>> > > > > > > > > Gustafson
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hey Michael,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hmm, I guess the
>> > > > > > point of
>> > > > > > > > representing
>> > > > > > > > > it as
>> > > > > > > > > > bytes is to
>> > > > > > > > > > > >> > allow
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > broker
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > to pass it
>> > > > > > through opaquely? Is
>> > > > > > > > the
>> > > > > > > > > cost of
>> > > > > > > > > > parsing
>> > > > > > > > > > > them a
>> > > > > > > > > > > >> >
>> concern, or
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > are
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > we simply trying
>> > > > > > to ensure that
>> > > > > > > > the
>> > > > > > > > > broker stays
>> > > > > > > > > > > agnostic
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > format?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On varints, I
>> > > > > > think adding
>> > > > > > > > support for
>> > > > > > > > > them makes
>> > > > > > > > > > less
>> > > > > > > > > > > >> sense
>> > > > > > > > > > > >> > for an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > isolated use
>> > > > > > case, but as part
>> > > > > > > of
>> > > > > > > > a
>> > > > > > > > > more holistic
>> > > > > > > > > > change
>> > > > > > > > > > > >> > (such
>> as
>> > what
>> > > > > > > > > > > >> >
>> >>
>> > we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > have proposed in
>> > > > > > KIP-98), I
>> > > > > > > think
>> > > > > > > > they
>> > > > > > > > > are
>> > > > > > > > > > justifiable.
>> > > > > > > > > > > If
>> > > > > > > > > > > >> > we
>> add them,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > then the need to
>> > > > > > use attributes
>> > > > > > > > becomes
>> > > > > > > > > quite a
>> > > > > > > > > > bit
>> > > > > > > > > > > >> weaker,
>> > > > > > > > > > > >> >
>> right? The
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > other thing I
>> > > > > > find slightly odd
>> > > > > > > > is the
>> > > > > > > > > fact that
>> > > > > > > > > > null
>> > > > > > > > > > > >> >
>> headers has
>> > no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > actual
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > semantic meaning
>> > > > > > for the message
>> > > > > > > > > (unlike null
>> > > > > > > > > > keys and
>> > > > > > > > > > > >> >
>> values). It
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > a
>> > > > space
>> > > > > > optimization. It seems a
>> > > > > > > > bit
>> > > > > > > > > better to
>> > > > > > > > > > always
>> > > > > > > > > > > use
>> > > > > > > > > > > >> > size
>> 0 to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > indicate having
>> > > > > > no headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Overall, the
>> > > > > main
>> > > > > > point is
>> > > > > > > > ensuring
>> > > > > > > > > that the
>> > > > > > > > > > message
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> >
>> remains
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > consistent,
>> > > > > > either within the
>> > > > > > > > larger
>> > > > > > > > > protocol, or
>> > > > > > > > > > at a
>> > > > > > > > > > > >> >
>> minimum
>> > within
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > message itself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On Thu, Feb 16,
>> > > > > > 2017 at 6:39 AM,
>> > > > > > > > > Michael Pearce <
>> > > > > > > > > > > >> >
>> > > > Michael.Pearce@ig.com
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Hi Jason,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 1) in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > the headers
>> > > > > > > > > > are
>> > > > > > > > > > > >> simply
>> > > > > > > > > > > >> > a byte
>> > > > > > > > > > > >> >
>> >>
>> > > > array,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > as like the
>> > > > > key
>> > > > > > or value, this
>> > > > > > > > is to
>> > > > > > > > > clearly
>> > > > > > > > > > demarcate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> header in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > core message.
>> > > > > > Then the header
>> > > > > > > > byte
>> > > > > > > > > array in the
>> > > > > > > > > > core
>> > > > > > > > > > > >> >
>> message is
>> > an
>> > > > > > > > > > > >> >
>> >>
>> > > > array of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key, value
>> > > > > > pairs. This is what
>> > > > > > > > it is
>> > > > > > > > > denoting.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Then this
>> > > > > would
>> > > > > > be I guess in
>> > > > > > > > the
>> > > > > > > > > given
>> > > > > > > > > > notation:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Headers =>
>> > > > > > [KeyLength, Key,
>> > > > > > > > > ValueLength, Value]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > KeyLength
>> > > > > > => int32
>> > > > > > > > > <-----------------NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised key
>> > > > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Key =>
>> > > > > bytes
>> > > > > > > > > <---------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > string
>> > (UTF8)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > bytes of the
>> > > > > > header key
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > ValueLength
>> > > > > > => int32
>> > > > > > > > > <-------------- NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised
>> > > > > > header value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Value =>
>> > > > > > bytes
>> > > > > > > > > <-------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > form
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The key length
>> > > > > > and value
>> > > > > > > length
>> > > > > > > > is
>> > > > > > > > > matching the
>> > > > > > > > > > way
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> protocol is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > defined in the
>> > > > > > core message
>> > > > > > > > currently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 2)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Var sized
>> > > > > ints,
>> > > > > > this was
>> > > > > > > > discussed
>> > > > > > > > > much earlier
>> > > > > > > > > > on, in
>> > > > > > > > > > > >> > fact
>> I had
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > suggested it
>> > > > > > myself (with
>> > > > > > > Hadoop
>> > > > > > > > > references),
>> > > > > > > > > > the
>> > > > > > > > > > > >> >
>> complexity
>> > of
>> > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compared to
>> > > > > > having a simpler
>> > > > > > > > protocol
>> > > > > > > > > was
>> > > > > > > > > > argued and
>> > > > > > > > > > > >> >
>> agreed it
>> > > > > > > > > > > >> >
>> >>
>> > > > wasn’t worth
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the complexity
>> > > > > > as all other
>> > > > > > > > clients
>> > > > > > > > > in other
>> > > > > > > > > > languages
>> > > > > > > > > > > >> > would
>> need
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > > > ensure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > theyre using
>> > > > > > the right var
>> > > > > > > size
>> > > > > > > > > algorithm, as
>> > > > > > > > > > there
>> > > > > > > > > > > is a
>> > > > > > > > > > > >> > few.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 3)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > We did the
>> > > > > > attributes,
>> > > > > > > optional
>> > > > > > > > > approach as
>> > > > > > > > > > originally
>> > > > > > > > > > > >> > there
>> was
>> > > > > > > > > > > >> >
>> >>
>> > > > marked
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > concern that
>> > > > > > headers would
>> > > > > > > > cause a
>> > > > > > > > > message size
>> > > > > > > > > > > overhead
>> > > > > > > > > > > >> > for
>> others,
>> > > > > > > > > > > >> >
>> >>
>> > who
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > don’t want
>> > > > > > them. As such this
>> > > > > > > > is the
>> > > > > > > > > clean
>> > > > > > > > > > solution to
>> > > > > > > > > > > >> >
>> achieve
>> > that.
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that no longer
>> > > > > > holds, and we
>> > > > > > > > don’t
>> > > > > > > > > care that we
>> > > > > > > > > > add
>> > > > > > > > > > > >> 4bytes
>> > > > > > > > > > > >> >
>> overhead,
>> > > > > > > > > > > >> >
>> >>
>> > > then
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > im happy to
>> > > > > > remove.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > I’m personally
>> > > > > > in favour of
>> > > > > > > > keeping
>> > > > > > > > > the message
>> > > > > > > > > > as
>> > > > > > > > > > > small
>> > > > > > > > > > > >> > as
>> possible
>> > > > > > > > > > > >> >
>> >>
>> > so
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > people don’t
>> > > > > > get shocks in
>> > > > > > > perf
>> > > > > > > > and
>> > > > > > > > > throughputs
>> > > > > > > > > > dues
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> message
>> > size,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > unless they
>> > > > > > actively use the
>> > > > > > > > feature,
>> > > > > > > > > as such I
>> > > > > > > > > > do
>> > > > > > > > > > > >> prefer
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > attribute bit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wise feature
>> > > > > > flag approach
>> > > > > > > > myself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On 16/02/2017,
>> > > > > > 05:40, "Jason
>> > > > > > > > > Gustafson" <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > We have
>> > > > > > proposed a few
>> > > > > > > > > significant changes
>> > > > > > > > > > to the
>> > > > > > > > > > > >> >
>> message
>> > format
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > KIP-98
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > which now
>> > > > > > seems likely to
>> > > > > > > > pass
>> > > > > > > > > (perhaps
>> > > > > > > > > > with some
>> > > > > > > > > > > >> >
>> iterations
>> > on
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > implementation details).
>> > > > > > > It
>> > > > > > > > would
>> > > > > > > > > be good
>> > > > > > > > > > to try
>> > > > > > > > > > > and
>> > > > > > > > > > > >> >
>> coordinate
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > changes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > in both of
>> > > > > > the proposals
>> > > > > > > to
>> > > > > > > > make
>> > > > > > > > > sure they
>> > > > > > > > > > are
>> > > > > > > > > > > >> >
>> consistent
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compatible.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I think
>> > > > > > using the
>> > > > > > > > attributes to
>> > > > > > > > > indicate
>> > > > > > > > > > null
>> > > > > > > > > > > >> headers
>> > > > > > > > > > > >> > is a
>> > > > > > > > > > > >> >
>> >>
>> > > > reasonable
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > approach.
>> > > > > > We have proposed
>> > > > > > > > to do
>> > > > > > > > > the same
>> > > > > > > > > > thing
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> message
>> > > > > > > > > > > >> >
>> >>
>> > key
>> > > > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > value.
>> > > > > That
>> > > > > > said, I
>> > > > > > > > sympathize
>> > > > > > > > > with Jay's
>> > > > > > > > > > > argument.
>> > > > > > > > > > > >> > Having
>> > > > > > > > > > > >> >
>> >>
>> > > > multiple
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > ways to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > specify a
>> > > > > > null value
>> > > > > > > > increases
>> > > > > > > > > the overall
>> > > > > > > > > > > >>
>> complexity
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > You
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > can see
>> > > > > > this just from the
>> > > > > > > > fact
>> > > > > > > > > that you
>> > > > > > > > > > need the
>> > > > > > > > > > > >> > extra
>> > verbiage
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > protocol
>> > > > > > specification in
>> > > > > > > > this
>> > > > > > > > > KIP and in
>> > > > > > > > > > KIP-98
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> describe the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > dependence
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > between
>> > > > > the
>> > > > > > fields and the
>> > > > > > > > > attributes. It
>> > > > > > > > > > seems
>> > > > > > > > > > > >> like a
>> > > > > > > > > > > >> >
>> slippery
>> > > > > > > > > > > >> >
>> >>
>> > > slope
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > if
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > you start
>> > > > > > allowing
>> > > > > > > different
>> > > > > > > > > request types
>> > > > > > > > > > to
>> > > > > > > > > > > >> >
>> implement
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification differently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > You can
>> > > > > > also argue that
>> > > > > > > the
>> > > > > > > > > messages
>> > > > > > > > > > already are
>> > > > > > > > > > > and
>> > > > > > > > > > > >> > are
>> likely
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > remain a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > special
>> > > > > > case. For example,
>> > > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > currently no
>> > > > > > > > > > > >> >
>> generality
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > how
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > compressed
>> > > > > > message sets
>> > > > > > > are
>> > > > > > > > > represented
>> > > > > > > > > > that would
>> > > > > > > > > > > >> be
>> > > > > > > > > > > >> >
>> applicable
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > other
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > request
>> > > > > > types. Some might
>> > > > > > > > see this
>> > > > > > > > > > divergence as
>> > > > > > > > > > > an
>> > > > > > > > > > > >> >
>> unfortunate
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > deficiency
>> > > > > > which should be
>> > > > > > > > fixed;
>> > > > > > > > > others
>> > > > > > > > > > might see
>> > > > > > > > > > > >> it
>> > > > > > > > > > > >> > as
>> sort of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > inevitability of needing
>> > > > > > > to
>> > > > > > > > > optimize where
>> > > > > > > > > > it
>> > > > > > > > > > > counts
>> > > > > > > > > > > >> > most.
>> I'm
>> > > > > > > > > > > >> >
>> >>
>> > > > probably
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > somewhere
>> > > > > > in between, but
>> > > > > > > I
>> > > > > > > > think
>> > > > > > > > > we
>> > > > > > > > > > probably all
>> > > > > > > > > > > >> > share
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > intuition
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol should be
>> > > > > > > kept
>> > > > > > > > as
>> > > > > > > > > consistent as
>> > > > > > > > > > > >>
>> possible.
>> > > > > > > > > > > >> > With
>> that
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > mind,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > here are a
>> > > > > > few comments:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 1. One
>> > > > > > thing I found a
>> > > > > > > > little odd
>> > > > > > > > > when
>> > > > > > > > > > reading the
>> > > > > > > > > > > >> >
>> current
>> > > > > > > > > > > >> >
>> >>
>> > > > proposal is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > headers
>> > > > > > are both
>> > > > > > > > represented
>> > > > > > > > > as an
>> > > > > > > > > > array of
>> > > > > > > > > > > >> bytes
>> > > > > > > > > > > >> > and
>> as an
>> > > > > > > > > > > >> >
>> >>
>> > > array
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > key/value
>> > > > > > pairs. I'd
>> > > > > > > > probably
>> > > > > > > > > suggest
>> > > > > > > > > > something
>> > > > > > > > > > > like
>> > > > > > > > > > > >> > this:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > [HeaderKey
>> > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > HeaderKey
>> > > > > > => String
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue => Bytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > An array
>> > > > > in
>> > > > > > the Kafka
>> > > > > > > > protocol is
>> > > > > > > > > > represented as a
>> > > > > > > > > > > >> > 4-byte
>> > integer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > indicating
>> > > > > > the number of
>> > > > > > > > elements
>> > > > > > > > > in the
>> > > > > > > > > > array
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > serialization of the
>> > > > > > > > elements.
>> > > > > > > > > Unless I'm
>> > > > > > > > > > > >> >
>> > > > misunderstanding, what
>> > > > > > > > > > > >> >
>> >>
>> > you
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > instead is
>> > > > > > the total size
>> > > > > > > > of the
>> > > > > > > > > headers in
>> > > > > > > > > > bytes
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > elements.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I'm not
>> > > > > > sure I see any
>> > > > > > > > reason for
>> > > > > > > > > this
>> > > > > > > > > > > >>
>> inconsistency.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 2. In
>> > > > > > KIP-98, we've
>> > > > > > > > introduced
>> > > > > > > > > > variable-length
>> > > > > > > > > > > >> integer
>> > > > > > > > > > > >> >
>> fields.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Effectively,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > we've
>> > > > > > enriched (or
>> > > > > > > > "complicated"
>> > > > > > > > > as Jay
>> > > > > > > > > > might say
>> > > > > > > > > > > ;)
>> > > > > > > > > > > >> > the
>> protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification to include
>> > > > > > > the
>> > > > > > > > > following
>> > > > > > > > > > types:
>> > > > > > > > > > > >> VarInt,
>> > > > > > > > > > > >> >
>> VarLong,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > UnsignedVarInt and
>> > > > > > > > > UnsignedVarLong.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Along with
>> > > > > > these
>> > > > > > > > primitives, we
>> > > > > > > > > could
>> > > > > > > > > > introduce
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> following
>> > > > > > > > > > > >> >
>> >>
>> > > > types:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeArray =>
>> > > > > > > > NumberOfItems
>> > > > > > > > > Item1 Item2
>> > > > > > > > > > .. ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItems =>
>> > > > > > > > UnsignedVarInt
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeNullableArray =>
>> > > > > > > > > NumberOfItemsOrNull
>> > > > > > > > > > Item1
>> > > > > > > > > > > >> > Item2
>> ..
>> > ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItemsOrNull =>
>> > > > > > > > VarInt
>> > > > > > > > > (-1 means
>> > > > > > > > > > null)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > And
>> > > > > > similarly for the
>> > > > > > > > `String`
>> > > > > > > > > and `Bytes`
>> > > > > > > > > > types.
>> > > > > > > > > > > >> > These
>> types
>> > > > > > > > > > > >> >
>> >>
>> > can
>> > > > save
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > considerable amount of
>> > > > > > > > space in
>> > > > > > > > > this
>> > > > > > > > > > proposal
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > they
>> can
>> > > > > > > > > > > >> >
>> >>
>> > be
>> > > > used
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > both the
>> > > > > > number of headers
>> > > > > > > > > included in the
>> > > > > > > > > > message
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > lengths of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > header
>> > > > > keys
>> > > > > > and values. We
>> > > > > > > > could
>> > > > > > > > > do this
>> > > > > > > > > > instead:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > > > VarSizeArray[HeaderKey
>> > > > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > HeaderKey
>> > > > > > =>
>> > > > > > > VarSizeString
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue =>
>> > > > > > > > VarSizeBytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Combining
>> > > > > > the savings from
>> > > > > > > > the
>> > > > > > > > > use of
>> > > > > > > > > > variable
>> > > > > > > > > > > >> length
>> > > > > > > > > > > >> >
>> fields, the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > benefit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > of using
>> > > > > > the attributes to
>> > > > > > > > > represent null
>> > > > > > > > > > seems
>> > > > > > > > > > > >> pretty
>> > > > > > > > > > > >> > small.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 3.
>> > > > > > Whichever way we go
>> > > > > > > > (whether
>> > > > > > > > > we use the
>> > > > > > > > > > > >>
>> attributes
>> > > > > > > > > > > >> > or
>> not), we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > should at
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > least be
>> > > > > > consistent
>> > > > > > > between
>> > > > > > > > this
>> > > > > > > > > KIP and
>> > > > > > > > > > KIP-98.
>> > > > > > > > > > > It
>> > > > > > > > > > > >> > would
>> be
>> > very
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > strange
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > to have
>> > > > > two
>> > > > > > ways to
>> > > > > > > > represent
>> > > > > > > > > null values
>> > > > > > > > > > in the
>> > > > > > > > > > > >> same
>> > > > > > > > > > > >> >
>> schema.
>> > > > > > > > > > > >> >
>> >>
>> > > > Either
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > way is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > OK with
>> > > > > me.
>> > > > > > I think some
>> > > > > > > > > message-level
>> > > > > > > > > > >
>> optimizations
>> > > > > > > > > > > >> > are
>> > > > > > > > > > > >> >
>> >>
>> > > > justifiable,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > but
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > savings
>> > > > > > here seem
>> > > > > > > > minimal (a
>> > > > > > > > > few bytes
>> > > > > > > > > > per
>> > > > > > > > > > > >> >
>> message), so
>> > > > > > > > > > > >> >
>> >>
>> > > > maybe it's
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > worth the
>> > > > > > cost of letting
>> > > > > > > > the
>> > > > > > > > > message
>> > > > > > > > > > diverge even
>> > > > > > > > > > > >> >
>> further from
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > rest of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > On Wed,
>> > > > > Feb
>> > > > > > 15, 2017 at
>> > > > > > > > 8:52 AM,
>> > > > > > > > > radai <
>> > > > > > > > > > > >> >
>> >>
>> > > > > > radai.rosenblatt@gmail.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > I've
>> > > > > > trimmed the inline
>> > > > > > > > > contents as this
>> > > > > > > > > > mail is
>> > > > > > > > > > > >> >
>> getting too
>> > > > > > > > > > > >> >
>> >>
>> > big
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > apache
>> > > > > > mailing list
>> > > > > > > > software to
>> > > > > > > > > deliver
>> > > > > > > > > > :-(
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 1. the
>> > > > > > important thing
>> > > > > > > for
>> > > > > > > > > > interoperability
>> is
>> > > > > > > > > > > for
>> > > > > > > > > > > >> >
>> different
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "interested
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > parties"
>> > > > > > (plugins, infra
>> > > > > > > > > layers/wrappers,
>> > > > > > > > > > > >>
>> user-code)
>> > > > > > > > > > > >> > to be
>> > > > > > > > > > > >> >
>> >>
>> > > able
>> > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > stick
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > pieces
>> > > > > of
>> > > > > > metadata onto
>> > > > > > > > msgs
>> > > > > > > > > without
>> > > > > > > > > > getting in
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> >
>> other's
>> > > > > > > > > > > >> >
>> >>
>> > > way.
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > common
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > key
>> > > > > > scheme (Strings, as
>> > > > > > > > of the
>> > > > > > > > > time of
>> > > > > > > > > > this
>> > > > > > > > > > > >> >
>> writing?) is
>> > > all
>> > > > > > > > > > > >> >
>> >>
>> > > thats
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > required
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > for
>> > > > > that.
>> > > > > > it is assumed
>> > > > > > > > that
>> > > > > > > > > the other end
>> > > > > > > > > > > >> >
>> interested
>> > in
>> > > > any
>> > > > > > > > > > > >> >
>> >>
>> > > such
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > piece of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > metadata
>> > > > > > knows the
>> > > > > > > > encoding,
>> > > > > > > > > and byte[]
>> > > > > > > > > > provides
>> > > > > > > > > > > >> for
>> > > > > > > > > > > >> > the
>> most
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > flexibility.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > i
>> > > > > believe
>> > > > > > this is the
>> > > > > > > same
>> > > > > > > > > logic behind
>> > > > > > > > > > core
>> > > > > > > > > > > kafka
>> > > > > > > > > > > >> > being
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > byte[]/byte[]
>> > > > > -
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Strings
>> > > > > > are more
>> > > > > > > "usable"
>> > > > > > > > but
>> > > > > > > > > bytes are
>> > > > > > > > > > flexible
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > so
>> were
>> > > > > > > > > > > >> >
>> >>
>> > > > chosen.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Also -
>> > > > > > core kafka doesnt
>> > > > > > > > even
>> > > > > > > > > do that
>> > > > > > > > > > good of a
>> > > > > > > > > > > >> job
>> > > > > > > > > > > >> > on
>> > > > > > > > > > > >> >
>> >>
>> > > > usability of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > payload
>> > > > > > (example - i
>> > > > > > > have
>> > > > > > > > to
>> > > > > > > > > specify the
>> > > > > > > > > > nop
>> > > > > > > > > > > >> byte[]
>> > > > > > > > > > > >> >
>> "decoders"
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > explicitly
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > in
>> > > > > conf),
>> > > > > > and again
>> > > > > > > > sacrificies
>> > > > > > > > > usability
>> > > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > sake
>> of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > performance
>> > > > > (no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > convenient single-record
>> > > > > > > > > processing as
>> > > > > > > > > > poll is a
>> > > > > > > > > > > >> >
>> batch, lots
>> > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > obscure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > little
>> > > > > > config details
>> > > > > > > > exposing
>> > > > > > > > > internals
>> > > > > > > > > > of the
>> > > > > > > > > > > >> >
>> batching
>> > > > > > > > > > > >> >
>> >>
>> > > > mechanism,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > etc)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > this is
>> > > > > > also why i
>> > > > > > > really
>> > > > > > > > > dislike the
>> > > > > > > > > > idea of a
>> > > > > > > > > > > >> > "type
>> > system"
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > values,
>> > > > > > it further
>> > > > > > > > degrades the
>> > > > > > > > > > usability, adds
>> > > > > > > > > > > >> >
>> complexity
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > eventually get in
>> > > > > > > > people's way,
>> > > > > > > > > also, it
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >> > the
>> 2nd/3rd
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > home-group
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization mechanism
>> > > > > > > > in core
>> > > > > > > > > kafka
>> > > > > > > > > > (counting
>> > > > > > > > > > > 2
>> > > > > > > > > > > >> >
>> iterations
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "type
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > definition DSL")
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 2. this
>> > > > > > is an
>> > > > > > > > implementation
>> > > > > > > > > detail, and
>> > > > > > > > > > not
>> > > > > > > > > > > even
>> > > > > > > > > > > >> a
>> > > > > > > > > > > >> > very
>> "user
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > facing" one?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to the
>> > > > > > best of my
>> > > > > > > > understanding
>> > > > > > > > > the vote
>> > > > > > > > > > process
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > on
>> proposed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > API/behaviour. also -
>> > > > > > > > since
>> > > > > > > > > we're willing
>> > > > > > > > > > to go
>> > > > > > > > > > > >> with
>> > > > > > > > > > > >> >
>> strings
>> > > > > > > > > > > >> >
>> >>
>> > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialize
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > a
>> > > > > 0-sized
>> > > > > > header blob
>> > > > > > > and
>> > > > > > > > IIUC
>> > > > > > > > > you dont
>> > > > > > > > > > need any
>> > > > > > > > > > > >> >
>> optionals
>> > > > > > > > > > > >> >
>> >>
>> > > > anymore.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 3. yes,
>> > > > > > we can :-)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > On Tue,
>> > > > > > Feb 14, 2017 at
>> > > > > > > > 11:56
>> > > > > > > > > PM, Michael
>> > > > > > > > > > > Pearce <
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Hi
>> > > > > Jay,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1)
>> > > > > > There was some
>> > > > > > > > initial
>> > > > > > > > > debate on the
>> > > > > > > > > > value
>> > > > > > > > > > > >> > part,
>> as
>> > youll
>> > > > > > > > > > > >> >
>> >>
>> > > note
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > String,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > String
>> > > > > > headers were
>> > > > > > > > > discounted early
>> > > > > > > > > > on. The
>> > > > > > > > > > > >> >
>> reason for
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > flexibility
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > keeping in line
>> > > > > > > > with the
>> > > > > > > > > > flexibility of
>> > > > > > > > > > > key,
>> > > > > > > > > > > >> > value
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > message
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > object
>> > > > > > itself. I don’t
>> > > > > > > > think
>> > > > > > > > > it takes
>> > > > > > > > > > away
>> > > > > > > > > > > from
>> > > > > > > > > > > >> an
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > as
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > each
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > plugin
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > will
>> > > > > > care for their
>> > > > > > > own
>> > > > > > > > key,
>> > > > > > > > > this way
>> > > > > > > > > > ints,
>> > > > > > > > > > > >> >
>> booleans ,
>> > > > exotic
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > binary
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > can
>> > > > > all
>> > > > > > be catered
>> > > > > > > for=.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > wanted
>> > > > > > > > to
>> > > > > > > > > push for a
>> > > > > > > > > > typed
>> > > > > > > > > > > >> value
>> > > > > > > > > > > >> >
>> interface,
>> > > > > > > > > > > >> >
>> >> I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wouldn’t
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > want
>> > > > > > just String
>> > > > > > > values
>> > > > > > > > > supported, but
>> > > > > > > > > > the the
>> > > > > > > > > > > >> >
>> primatives
>> > > > > > > > > > > >> >
>> >>
>> > > plus
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > string and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > also
>> > > > > > still keeping the
>> > > > > > > > > ability to have a
>> > > > > > > > > > > binary
>> > > > > > > > > > > >> > for
>> custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > binaries that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > some
>> > > > > > organisations may
>> > > > > > > > have.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > i. I
>> > > > > > have written this
>> > > > > > > > slight
>> > > > > > > > > > alternative
>> > > > > > > > > > > here,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > https://cwiki.apache.org/
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > > >> >
>> > > > 82+-+Add+Record+Headers+-+
>> > > > > Typed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > ii.
>> > > > > > Essentially the
>> > > > > > > > value
>> > > > > > > > > bytes, has a
>> > > > > > > > > > leading
>> > > > > > > > > > > >> > byte
>> > overhead.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > > This tells you
>> > > > > > > what
>> > > > > > > > type
>> > > > > > > > > the value
>> > > > > > > > > > is,
>> > > > > > > > > > > >> before
>> > > > > > > > > > > >> >
>> reading
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > rest
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes,
>> > > > > > allowing
>> > > > > > > > > >
>> > > > serialisation/deserialization
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > and
>> from the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > primitives,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > string
>> > > > > > and byte[].
>> > > > > > > This
>> > > > > > > > is
>> > > > > > > > > akin to some
>> > > > > > > > > > other
>> > > > > > > > > > > >> >
>> messaging
>> > > > > > > > > > > >> >
>> >>
>> > > > systems.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2) We
>> > > > > > are making it
>> > > > > > > > optional,
>> > > > > > > > > so that
>> > > > > > > > > > for
>> > > > > > > > > > > those
>> > > > > > > > > > > >> > not
>> wanting
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > headers have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 0
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes
>> > > > > > overhead (think
>> > > > > > > > of it
>> > > > > > > > > as a feature
>> > > > > > > > > > > flag),
>> > > > > > > > > > > >> I
>> > > > > > > > > > > >> > don’t
>> > > > > > > > > > > >> >
>> >>
>> > > > think this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > complex, especially if
>> > > > > > > > > comparing to
>> > > > > > > > > > changes
>> > > > > > > > > > > >> >
>> proposed in
>> > > > > > > > > > > >> >
>> >>
>> > > > other kips
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > like
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > kip-98.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > really
>> > > > > > > > don’t
>> > > > > > > > > like
>> > > > > > > > > > this, we
>> > > > > > > > > > > can
>> > > > > > > > > > > >> > drop
>> it, but
>> > > > > > > > > > > >> >
>> >>
>> > it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > would
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > mean
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > buying
>> > > > > > into 4 bytes
>> > > > > > > > extra
>> > > > > > > > > overhead for
>> > > > > > > > > > users
>> > > > > > > > > > > who
>> > > > > > > > > > > >> > do
>> not want
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > use
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3) In
>> > > > > > the summary yes,
>> > > > > > > > it is
>> > > > > > > > > at a higher
>> > > > > > > > > > > level,
>> > > > > > > > > > > >> > but I
>> think
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is well
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > documented in the
>> > > > > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > section.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a.
>> > > > > > Added getHeaders
>> > > > > > > > method to
>> > > > > > > > > > >
>> Producer/Consumer
>> > > > > > > > > > > >> >
>> record (that
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > it)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > b.
>> > > > > > We’ve also detailed
>> > > > > > > > the
>> > > > > > > > > new Headers
>> > > > > > > > > > class
>> > > > > > > > > > > >> that
>> > > > > > > > > > > >> > this
>> method
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > returns
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > encapsulates the
>> > > > > > > headers
>> > > > > > > > > protocol and
>> > > > > > > > > > logic.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Best,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Mike
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ==Original questions
>> > > > > > > > from the
>> > > > > > > > > vote
>> > > > > > > > > > thread from
>> > > > > > > > > > > >> > Jay.==
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Couple
>> > > > > > of things I
>> > > > > > > > think we
>> > > > > > > > > still need
>> > > > > > > > > > to work
>> > > > > > > > > > > >> out:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > I
>> > > > > > think we agree
>> > > > > > > > about
>> > > > > > > > > the key,
>> > > > > > > > > > but I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > we
>> haven't
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > talked about
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > value yet. I
>> > > > > > > > think if
>> > > > > > > > > our goal
>> > > > > > > > > > is an
>> > > > > > > > > > > open
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > these
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > spread across many
>> > > > > > > > plugins
>> > > > > > > > > from many
>> > > > > > > > > > > systems
>> > > > > > > > > > > >> we
>> > > > > > > > > > > >> > should
>> > > > > > > > > > > >> >
>> >>
>> > > > consider
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > making
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > string as well so
>> > > > > > > > it can
>> > > > > > > > > be
>> > > > > > > > > > printed, set
>> > > > > > > > > > > >> via
>> > > > > > > > > > > >> > a UI,
>> set
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > config,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > etc.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > Basically
>> > > > > > > encouraging
>> > > > > > > > > pluggable
>> > > > > > > > > > >
>> serialization
>> > > > > > > > > > > >> >
>> formats
>> > > > > > > > > > > >> >
>> >>
>> > > here
>> > > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > lead
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bit
>> > > > > > of a tower of
>> > > > > > > > babel.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2.
>> > > > > > This proposal
>> > > > > > > > still
>> > > > > > > > > includes a
>> > > > > > > > > > pretty
>> > > > > > > > > > > big
>> > > > > > > > > > > >> >
>> change to
>> > our
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > protocol
>> > > > > > > > definition
>> > > > > > > > > layer.
>> > > > > > > > > > Essentially
>> > > > > > > > > > > >> it is
>> > > > > > > > > > > >> >
>> >>
>> > > > introducing an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > optional
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > type, where the
>> > > > > > > > format is
>> > > > > > > > > data
>> > > > > > > > > > dependent. I
>> > > > > > > > > > > >> > think
>> this
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > actually a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > big
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > change though it
>> > > > > > > > doesn't
>> > > > > > > > > seem like
>> > > > > > > > > > it. It
>> > > > > > > > > > > >> means
>> > > > > > > > > > > >> > you
>> can no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > longer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > specify
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > this
>> > > > > > type with our
>> > > > > > > > type
>> > > > > > > > > definition
>> > > > > > > > > > DSL, and
>> > > > > > > > > > > >> >
>> likewise it
>> > > > > > > > > > > >> >
>> >>
>> > > > requires
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > handling in client
>> > > > > > > > libs.
>> > > > > > > > > This isn't
>> > > > > > > > > > a huge
>> > > > > > > > > > > >> >
>> thing, since
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Record
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > definition is
>> > > > > > > custom
>> > > > > > > > > anyway, but I
>> > > > > > > > > > think
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > kind
>> of
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > inconsistency is
>> > > > > > > very
>> > > > > > > > > non-desirable
>> > > > > > > > > > and
>> > > > > > > > > > > ties
>> > > > > > > > > > > >> > you to
>> > > > > > > > > > > >> >
>> >>
>> > > > hand-coding
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > things.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > think the type
>> > > > > > > should
>> > > > > > > > > instead by [Key
>> > > > > > > > > > > Value]
>> > > > > > > > > > > >> in
>> > > > > > > > > > > >> > our
>> BNF,
>> > > > > > > > > > > >> >
>> >>
>> > > where
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > value are both
>> > > > > > > short
>> > > > > > > > > strings as used
>> > > > > > > > > > > >>
>> elsewhere.
>> > > > > > > > > > > >> > This
>> > > > > > > > > > > >> >
>> >>
>> > > > brings it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > in line
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > with
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > rest of the
>> > > > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3.
>> > > > > > Could we get
>> > > > > > > more
>> > > > > > > > > specific about
>> > > > > > > > > > the
>> > > > > > > > > > > exact
>> > > > > > > > > > > >> > Java
>> API
>> > > > > > > > > > > >> >
>> >>
>> > > > change to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ProducerRecord,
>> > > > > > > > > ConsumerRecord,
>> > > > > > > > > > Record,
>> > > > > > > > > > > etc?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > -Jay
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The
>> > > > > information
>> > > > > > contained in
>> > > > > > > > this
>> > > > > > > > > email is
>> > > > > > > > > > strictly
>> > > > > > > > > > > >> >
>> confidential
>> > > and
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the use of the
>> > > > > > addressee only,
>> > > > > > > > unless
>> > > > > > > > > otherwise
>> > > > > > > > > > > >>
>> indicated.
>> > > > > > > > > > > >> > If you
>> > > > > > > > > > > >> >
>> >>
>> > are
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the intended
>> > > > > > recipient, please
>> > > > > > > > do not
>> > > > > > > > > read,
>> > > > > > > > > > copy, use
>> > > > > > > > > > > or
>> > > > > > > > > > > >> >
>> disclose to
>> > > > > > > > > > > >> >
>> >>
>> > > > others
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > this message
>> > > > > or
>> > > > > > any
>> > > > > > > attachment.
>> > > > > > > > > Please also
>> > > > > > > > > > notify the
>> > > > > > > > > > > >> >
>> sender by
>> > > > > > > > > > > >> >
>> >>
>> > > > replying
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > to this email
>> > > > > > or by telephone
>> > > > > > > > > (+44(020 7896
>> > > > > > > > > > 0011) and
>> > > > > > > > > > > >> then
>> > > > > > > > > > > >> > delete
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > email
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > and any copies
>> > > > > > of it.
>> > > > > > > Opinions,
>> > > > > > > > > conclusion
>> > > > > > > > > > (etc) that
>> > > > > > > > > > > do
>> > > > > > > > > > > >> > not
>> relate
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > official
>> > > > > > business of this
>> > > > > > > > company
>> > > > > > > > > shall be
>> > > > > > > > > > understood
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> neither
>> > > > > > > > > > > >> >
>> >>
>> > > > given nor
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > endorsed by
>> > > > > it.
>> > > > > > IG is a
>> > > > > > > trading
>> > > > > > > > name
>> > > > > > > > > of IG
>> > > > > > > > > > Markets
>> > > > > > > > > > > >> Limited
>> > > > > > > > > > > >> > (a
>> company
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > registered in
>> > > > > > England and
>> > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 04008957)
>> > > > > > > > > > > >> > and IG
>> > > > > > > > > > > >> >
>> >>
>> > > Index
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Limited (a
>> > > > > > company registered
>> > > > > > > in
>> > > > > > > > > England and
>> > > > > > > > > > Wales,
>> > > > > > > > > > > >> >
>> company
>> > number
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > 01190902).
>> > > > > > Registered address
>> > > > > > > at
>> > > > > > > > > Cannon Bridge
>> > > > > > > > > > House,
>> > > > > > > > > > > 25
>> > > > > > > > > > > >> >
>> Dowgate
>> > > > > > > > > > > >> >
>> >>
>> > > Hill,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > London EC4R
>> > > > > > 2YA. Both IG
>> > > > > > > Markets
>> > > > > > > > > Limited
>> > > > > > > > > > (register
>> > > > > > > > > > > >> number
>> > > > > > > > > > > >> >
>> 195355)
>> > > > > > > > > > > >> >
>> >>
>> > and
>> > > > IG
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Index Limited
>> > > > > > (register number
>> > > > > > > > > 114059) are
>> > > > > > > > > > authorised
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> regulated
>> > > > > > > > > > > >> >
>> >>
>> > by
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Financial
>> > > > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > --
>> > > > > > > > > > > >> >
>> >>
>> > > > Nacho - Ignacio
>> > > > > > Solis -
>> > > > > > > > isolis@igso.net
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > The
>> > > information
>> > > > contained in
>> > > > > > this email is
>> > > > > > > > strictly
>> > > > > > > > > > confidential and
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> use of
>> > the
>> > > > addressee only,
>> > > > > > unless
>> > > > > > > otherwise
>> > > > > > > > > indicated. If
>> > > > > > > > > > you are
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > the
>> intended
>> > > > recipient, please
>> > > > > > do not read,
>> > > > > > > > copy, use
>> > > > > > > > > or
>> > > > > > > > > > disclose to
>> > > > > > > > > > > >> others
>> > > > > > > > > > > >> > this
>> message
>> > > or
>> > > > any
>> > > > > attachment.
>> > > > > > Please also
>> > > > > > > > notify
>> > > > > > > > > the sender by
>> > > > > > > > > > > >> replying
>> > > > > > > > > > > >> > to
>> this
>> > email
>> > > > or by telephone
>> > > > > > (+44(020 7896
>> > > > > > > > 0011) and
>> > > > > > > > > then
>> > > > > > > > > > delete the
>> > > > > > > > > > > >> email
>> > > > > > > > > > > >> > and
>> any
>> > copies
>> > > > of it.
>> > > > > Opinions,
>> > > > > > conclusion
>> > > > > > > > (etc) that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > understood
>> > > > > > > > > as
>> > > > > > > > > > neither given
>> > > > > > > > > > > >> nor
>> > > > > > > > > > > >> >
>> endorsed by
>> > > it.
>> > > > IG is a
>> > > > > trading
>> > > > > > name of IG
>> > > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >> >
>> registered
>> > in
>> > > > England and
>> > > > > > Wales, company
>> > > > > > > number
>> > > > > > > > > 04008957) and
>> > > > > > > > > > IG Index
>> > > > > > > > > > > >> >
>> Limited (a
>> > > > company registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > >> >
>> 01190902).
>> > > > Registered address
>> > > > > > at Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25
>> > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > >> >
>> London EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and
>> > > > > > > > > > > IG
>> > > > > > > > > > > >> > Index
>> > Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> Financial
>> > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> The
>> > information
>> > > > contained in
>> > > > > this
>> > > > > > email is
>> > > > > > > > strictly
>> > > > > > > > > confidential
>> > > > > > > > > > and for
>> > > > > > > > > > > >> the use
>> of the
>> > > > addressee only,
>> > > > > > unless otherwise
>> > > > > > > > > indicated. If you
>> > > > > > > > > > are
>> > > > > > > > > > > not
>> > > > > > > > > > > >> the
>> intended
>> > > > recipient, please
>> > > > > do
>> > > > > > not read,
>> > > > > > > copy,
>> > > > > > > > use
>> > > > > > > > > or disclose
>> > > > > > > > > > to
>> > > > > > > > > > > others
>> > > > > > > > > > > >> this
>> message
>> > or
>> > > > any attachment.
>> > > > > > Please also
>> > > > > > > > notify the
>> > > > > > > > > sender by
>> > > > > > > > > > > replying
>> > > > > > > > > > > >> to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the
>> > > > > > > > > > > email
>> > > > > > > > > > > >> and any
>> copies
>> > > of
>> > > > it. Opinions,
>> > > > > > conclusion (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > the
>> > > > > > > > > > > >> official
>> > > business
>> > > > of this
>> > > > > company
>> > > > > > shall be
>> > > > > > > > understood
>> > > > > > > > > as neither
>> > > > > > > > > > given
>> > > > > > > > > > > nor
>> > > > > > > > > > > >>
>> endorsed by
>> > it.
>> > > > IG is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >>
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG
>> > > > > > > > > > Index
>> > > > > > > > > > > >> Limited
>> (a
>> > > > company registered in
>> > > > > > England and
>> > > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 01190902).
>> > > > Registered address at
>> > > > > > Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25 Dowgate
>> > > > > > > > > > Hill,
>> > > > > > > > > > > >> London
>> EC4R
>> > 2YA.
>> > > > Both IG Markets
>> > > > > > Limited
>> > > > > > > (register
>> > > > > > > > > number 195355)
>> > > > > > > > > > and IG
>> > > > > > > > > > > >> Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> Financial
>> > > Conduct
>> > > > Authority.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > The
>> information
>> > > > contained in this
>> > > > > > email is strictly
>> > > > > > > > > confidential and
>> > > > > > > > > > for the use of
>> the
>> > > > addressee only, unless
>> > > > > > otherwise
>> > > > > > > > indicated.
>> > > > > > > > > If you are
>> > > > > > > > > > not the intended
>> > > > recipient, please do not
>> > > > > > read, copy, use
>> > > > > > > > or
>> > > > > > > > > disclose to
>> > > > > > > > > > others this
>> message or
>> > > any
>> > > > attachment.
>> > > > > > Please also notify
>> > > > > > > > the
>> > > > > > > > > sender by
>> > > > > > > > > > replying to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the email and any
>> > copies
>> > > > of it. Opinions,
>> > > > > > conclusion
>> > > > > > > (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to the
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > > understood as
>> > > > > > > > > > neither given nor
>> > > endorsed
>> > > > by it. IG is a
>> > > > > > trading name of
>> > > > > > > > IG
>> > > > > > > > > Markets
>> > > > > > > > > > Limited (a
>> company
>> > > > registered in England
>> > > > > > and Wales,
>> > > > > > > company
>> > > > > > > > > number
>> > > > > > > > > > 04008957) and IG
>> Index
>> > > > Limited (a company
>> > > > > > registered in
>> > > > > > > > England
>> > > > > > > > > and Wales,
>> > > > > > > > > > company number
>> > 01190902).
>> > > > Registered
>> > > > > > address at Cannon
>> > > > > > > > Bridge
>> > > > > > > > > House, 25
>> > > > > > > > > > Dowgate Hill,
>> London
>> > EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and IG
>> Index
>> > > > Limited (register
>> > > > > > number 114059) are
>> > > > > > > > > authorised and
>> > > > > > > > > > regulated by the
>> > > Financial
>> > > > Conduct
>> > > > > > Authority.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The information
>> contained in
>> > > > this email is
>> > > > > > strictly
>> > > > > > > confidential
>> > > > > > > > and
>> > > > > > > > > for the use of the
>> addressee
>> > only,
>> > > > unless otherwise
>> > > > > > indicated. If
>> > > > > > > > you are
>> > > > > > > > > not the intended recipient,
>> > please
>> > > > do not read,
>> > > > > > copy, use or
>> > > > > > > > disclose to
>> > > > > > > > > others this message or any
>> > > > attachment. Please also
>> > > > > > notify the
>> > > > > > > sender
>> > > > > > > > by
>> > > > > > > > > replying to this email or
>> by
>> > > > telephone (+44(020
>> > > > > 7896
>> > > > > > 0011) and then
>> > > > > > > > delete
>> > > > > > > > > the email and any copies
>> of it.
>> > > > Opinions,
>> > > > > conclusion
>> > > > > > (etc) that do
>> > > > > > > > not
>> > > > > > > > > relate to the official
>> business
>> > of
>> > > > this company
>> > > > > > shall be understood
>> > > > > > > > as
>> > > > > > > > > neither given nor endorsed
>> by it.
>> > > IG
>> > > > is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a company
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG Index
>> Limited (a
>> > > > company
>> > > > > registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company number 01190902).
>> > > Registered
>> > > > address at
>> > > > > > Cannon Bridge
>> > > > > > > House,
>> > > > > > > > 25
>> > > > > > > > > Dowgate Hill, London EC4R
>> 2YA.
>> > Both
>> > > > IG Markets
>> > > > > > Limited (register
>> > > > > > > > number
>> > > > > > > > > 195355) and IG Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > authorised
>> > > > > > > > and
>> > > > > > > > > regulated by the Financial
>> > Conduct
>> > > > Authority.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > The information contained in
>> this email
>> > > is
>> > > > strictly
>> > > > > > confidential and for
>> > > > > > > > the use of the addressee only,
>> unless
>> > > > otherwise
>> > > > > indicated.
>> > > > > > If you are not
>> > > > > > > > the intended recipient, please
>> do not
>> > > > read, copy, use or
>> > > > > > disclose to
>> > > > > > > others
>> > > > > > > > this message or any attachment.
>> Please
>> > > > also notify the
>> > > > > > sender by replying
>> > > > > > > > to this email or by telephone
>> (+44(020
>> > > > 7896 0011) and
>> > > > > then
>> > > > > > delete the
>> > > > > > > email
>> > > > > > > > and any copies of it. Opinions,
>> > > conclusion
>> > > > (etc) that do
>> > > > > > not relate to
>> > > > > > > the
>> > > > > > > > official business of this
>> company shall
>> > > be
>> > > > understood as
>> > > > > > neither given
>> > > > > > > nor
>> > > > > > > > endorsed by it. IG is a trading
>> name of
>> > > IG
>> > > > Markets
>> > > > > Limited
>> > > > > > (a company
>> > > > > > > > registered in England and Wales,
>> > company
>> > > > number 04008957)
>> > > > > > and IG Index
>> > > > > > > > Limited (a company registered in
>> > England
>> > > > and Wales,
>> > > > > > company number
>> > > > > > > > 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill,
>> > > > > > > > London EC4R 2YA. Both IG Markets
>> > Limited
>> > > > (register number
>> > > > > > 195355) and IG
>> > > > > > > > Index Limited (register number
>> 114059)
>> > > are
>> > > > authorised and
>> > > > > > regulated by
>> > > > > > > the
>> > > > > > > > Financial Conduct Authority.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > The information contained in this email is
>> strictly
>> > > > confidential and
>> > > > > > for the use of the addressee only, unless
>> otherwise
>> > > > indicated. If you are
>> > > > > > not the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or disclose to
>> > > > > > others this message or any attachment. Please
>> also
>> > notify
>> > > > the sender by
>> > > > > > replying to this email or by telephone (+44(020
>> 7896
>> > > 0011)
>> > > > and then
>> > > > > delete
>> > > > > > the email and any copies of it. Opinions,
>> conclusion
>> > > (etc)
>> > > > that do not
>> > > > > > relate to the official business of this company
>> shall
>> > be
>> > > > understood as
>> > > > > > neither given nor endorsed by it. IG is a
>> trading name
>> > of
>> > > > IG Markets
>> > > > > > Limited (a company registered in England and
>> Wales,
>> > > > company number
>> > > > > > 04008957) and IG Index Limited (a company
>> registered in
>> > > > England and
>> > > > > Wales,
>> > > > > > company number 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register number
>> > > > > > 195355) and IG Index Limited (register number
>> 114059)
>> > are
>> > > > authorised and
>> > > > > > regulated by the Financial Conduct Authority.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential
>> and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If
>> you are
>> > not
>> > > > the intended recipient, please do not read, copy, use or
>> disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then
>> delete the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company
>> number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number
>> 195355) and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and
>> regulated by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > >
>> >
>>
>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Yeah, additional feedback is welcome. Note that KIP-98 has unofficially
passed (we have 3 binding votes, though we are awaiting the outcome of this
discussion to call it), so I don't think there will be significant changes
to what we have there. If you think the suggestion sounds reasonable, maybe
update the KIP and restart the vote? That will give others an extra push to
comment. If that sounds good, I will go ahead and update KIP-98 accordingly.
By the way, if the APIs in the KIP are adapted to use the Headers object, I
think it's fine to leave out the details about how we will represent the
headers (whether as a byte slice, an array, or whatever). We can leave that
for implementation. If the only interface we expose is Iterable<Header>,
then we retain a lot of flexibility.
-Jason
On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com>
wrote:
> I think I'd like some opinions from some of the active chaps at LinkedIn
> so it's not just a two way decision (or any others for that matter).
>
> Any opinions on the below or no real concerns / indifferent by anyone with
> this change?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Tuesday, February 21, 2017 6:59:38 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Jun's suggestion gives us the best of both worlds. Because the headers are
> the last field in the record and you have the total size of the record,
> then you can easily get the size of headers in bytes. So they can be copied
> without parsing and we can do the lazy initialization without an extra
> cost.
>
> The message set notion has changed in KIP-98 so that the message set can
> have its own fields. See here for more detail:
> https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF
> 0wSw9ra8/edit#heading=h.pau6n6buvoee.
> What we're proposing to change here is the schema for the message (aka
> record) itself. We'll update that document depending on the outcome of the
> discussion here.
>
> -Jason
>
>
> On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > This works whilst no other feature needs similar. I prefer to keep it
> > contained/encapsulated it then allows the bytes of the headers to also be
> > copied transferred as one without any parsing what so ever, but if you
> > really feel strongly on the below ill go with it, just want to flag this
> > concern.
> >
> > Btw, the below notation seems to be different to the notation used in
> > previous kips like kip-32 which I took the notation from, where the
> length
> > of the following bytes for byte arrays is clearly defined. So im 100% on
> > what you propose could you define it in this original styled notation?
> >
> >
> > On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > Hey Jun, that's a good thought. Maybe then the full schema could look
> > like
> > this:
> >
> > Record =>
> > Length => uintVar
> > Attributes => int8
> > TimestampDelta => intVar
> > OffsetDelta => uintVar
> > Key => bytes
> > Value => bytes
> > Headers => [HeaderKey HeaderValue]
> > HeaderKey => string
> > HeaderValue => bytes
> >
> > (where the bytes and string type use a variable length size field)
> >
> > Does that work for you, Michael?
> >
> > -Jason
> >
> > On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
> >
> > > Regarding the overhead of skipping the headers in the message.
> > Another way
> > > could be to add the headers at the end of the message after the
> > value part.
> > > This way, there is no need to skip the headers unless you really
> > need it.
> > > This will allow us to represent the headers explicitly as an array
> of
> > > key/value pairs without the skipping overhead when headers are not
> > needed.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <
> jason@confluent.io
> > >
> > > wrote:
> > >
> > > > Michael,
> > > >
> > > > Here's a link to the branch with the message format changes:
> > > > https://github.com/confluentinc/kafka/tree/
> > exactly-once-message-format.
> > > > The
> > > > class to modify to add headers is EosLogRecord (name will
> > definitely
> > > change
> > > > prior to merging in trunk). This branch uses attributes to
> > indicate null
> > > > key and value. I really don't have a strong preference for or
> > against
> > > this
> > > > as long as we're consistent. I do feel it's kind of a pity to
> need
> > the
> > > > signed version of the varint just to be able to indicate null.
> For
> > > headers,
> > > > it's an easier call because we don't actually need to represent
> > null
> > > > headers (using size zero will work fine).
> > > >
> > > > -Jason
> > > >
> > > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > We can make it, implement Iterable<Header>, no concerns on
> this.
> > > > >
> > > > > Ive noted this on the KIP.
> > > > >
> > > > >
> > > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > > > >
> > > > > Btw Jason, can you share the branch with the VarInts
> > helper/util
> > > > class
> > > > > that KIP-98 will be introducing and also will have the current
> > modified
> > > > > (key and value lengths) message, so we can look at, thanks.
> > > > >
> > > > > On 20/02/2017, 01:11, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > RE “ProducerRecord and ConsumerRecord are immutable
> > > currently”. I
> > > > > beg to differ on this, if I set a byte[] as my value of my
> > record, I
> > > can
> > > > > later call value() and then alter the bytes in the array. As
> > such it is
> > > > > very much mutable.
> > > > >
> > > > > Headers also will anyhow by nature be mutable on
> > produce, as
> > > you
> > > > > add to them.
> > > > >
> > > > >
> > > > > On 20/02/2017, 00:22, "Jason Gustafson" <
> > jason@confluent.io>
> > > > > wrote:
> > > > >
> > > > > Re; having a dedicated Headers object. I think it
> > makes
> > > > sense.
> > > > > Maybe it
> > > > > could implement Iterable<Header>?
> > > > >
> > > > > One thing I'm not too sure about is mutability.
> Both
> > the
> > > > > ProducerRecord and
> > > > > ConsumerRecord are immutable currently. It would be
> > nice to
> > > > > keep it that
> > > > > way. Perhaps Headers could itself be immutable and
> > passed
> > > > > through the
> > > > > constructors?
> > > > >
> > > > > Just going through how this is, I don’t believe
> > this would
> > > > > provide much
> > > > > > saving. If issue trying to resolve is around
> > > object/garbage
> > > > > creation of
> > > > > > Map.Entry objects.
> > > > >
> > > > >
> > > > > It's definitely better because we only allocate the
> > space
> > > we
> > > > > will use, but
> > > > > I assume most of the actual memory cost is in the
> > header
> > > data
> > > > > itself which
> > > > > we can't do much about. Maybe we can start with the
> > simple
> > > > > array
> > > > > representation and optimize from there after we
> find
> > it's
> > > > > worthwhile to do
> > > > > so? If we use the Headers object, we can change
> this
> > later
> > > > > without breaking
> > > > > the API.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> > > > > radai.rosenblatt@gmail.com> wrote:
> > > > >
> > > > > > "(By the way, doesn't it feel a bit odd that we
> > seem to
> > > be
> > > > > designing a
> > > > > > feature which is optimized for people not
> using?)"
> > > > > >
> > > > > > very. this (i claim :-) ) is the result of very
> > intense
> > > > > opposition to the
> > > > > > usefulness of the feature early on, and not a
> real
> > design
> > > > > goal
> > > > > >
> > > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Whilst if having an array of a Header[] would
> > mean
> > > > getting
> > > > > this through,
> > > > > > > im happy changing this, as already done.
> > > > > > >
> > > > > > > Just going through how this is, I don’t
> believe
> > this
> > > > > would provide much
> > > > > > > saving. If issue trying to resolve is around
> > > > > object/garbage creation of
> > > > > > > Map.Entry objects.
> > > > > > >
> > > > > > > All you’re doing here is replacing the equiv
> of a
> > > > > HashMap.Node
> > > > > > (Map.Entry)
> > > > > > > which simply holds reference to key and value
> > objects
> > > > with
> > > > > a custom
> > > > > > variant.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
> > > > > Michael.Pearce@ig.com> wrote:
> > > > > > >
> > > > > > > On point 1 & 2 Ive updated KIP to show
> > varints (and
> > > > > removed the bit
> > > > > > > flag). (on the assumption KIP 98 is getting the
> > > agreement
> > > > > the protocol is
> > > > > > > moving from int32 to varInts as standard)
> > > > > > >
> > > > > > > On point 3 ive updated to use an array of
> > Header
> > > > > class, instead of a
> > > > > > > MultiMap in the Headers class object
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > On points 1 and 2 I agree.
> > > > > > >
> > > > > > > This also affects kip-98, I should
> > expect this
> > > > > resolved before
> > > > > > > that vote also passes. If it is accepted there
> > (I’m
> > > > > assuming this is
> > > > > > > getting discussed on that KIP? As you’re
> driving
> > the
> > > move
> > > > > to VarInts), I
> > > > > > am
> > > > > > > happy to make this KIP will simply follow suit
> to
> > > > whatever
> > > > > is agreed in
> > > > > > > KIP-98.
> > > > > > >
> > > > > > > On 3) Agreed this is a simpler form, as
> > long as
> > > > no
> > > > > one is
> > > > > > > expecting hashmap lookup performance ( O(1) ),
> I
> > still
> > > > > would prefer an
> > > > > > > encapsulated class, so if we find that holding
> > it as an
> > > > > array in future
> > > > > > is
> > > > > > > causing some perf issues, the internals are not
> > exposed
> > > > to
> > > > > end users,
> > > > > > > allowing the internal structure to move to a
> map.
> > > > > > >
> > > > > > > Can we compromise on?
> > > > > > >
> > > > > > > class ConsumerRecord<K, V> {
> > > > > > > K key;
> > > > > > > V value;
> > > > > > > Headers headers;
> > > > > > > }
> > > > > > >
> > > > > > > class Headers {
> > > > > > > Header[] headers;
> > > > > > >
> > > > > > > add(String key, byte[] value)
> > > > > > > Collection<byte[]> get(String
> key)
> > > > > > > }
> > > > > > >
> > > > > > > class Header {
> > > > > > > String key;
> > > > > > > byte[] value;
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 19/02/2017, 18:54, "Jason
> Gustafson" <
> > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > headers dont "leak" into
> > application
> > > code.
> > > > > they are useful
> > > > > > > to application
> > > > > > > > code as well.
> > > > > > >
> > > > > > >
> > > > > > > This is exactly what I have been
> > trying to
> > > > get
> > > > > at. The use
> > > > > > > cases documented
> > > > > > > here are middleware:
> > > > > > > https://cwiki.apache.org/
> > > > > confluence/display/KAFKA/A+
> > > > > > > Case+for+Kafka+Headers.
> > > > > > > If headers are intended for the
> > application
> > > > > space as well,
> > > > > > the
> > > > > > > document
> > > > > > > should be updated accordingly so
> > that it is
> > > > an
> > > > > explicit
> > > > > > design
> > > > > > > goal and not
> > > > > > > an unfortunate design byproduct.
> > There may
> > > be
> > > > > some
> > > > > > > disagreement on whether
> > > > > > > it _should_ be a design goal, but
> it
> > may
> > > not
> > > > > matter much
> > > > > > since
> > > > > > > the current
> > > > > > > interceptor API is probably
> > insufficient
> > > for
> > > > > middleware
> > > > > > > applications (I
> > > > > > > haven't thought of a way to do this
> > that
> > > > isn't
> > > > > cumbersome).
> > > > > > >
> > > > > > > That aside, the three unresolved
> > points for
> > > > me
> > > > > are the
> > > > > > > following:
> > > > > > >
> > > > > > > 1. The use of varints. As a one-off
> > for use
> > > > > only with record
> > > > > > > headers, the
> > > > > > > case was weak, but if we are using
> > them
> > > > > throughout the
> > > > > > message
> > > > > > > format, then
> > > > > > > we should do so here as well. The
> > > additional
> > > > > complexity is
> > > > > > > minimal and
> > > > > > > early performance testing fully
> > justifies
> > > > > their use.
> > > > > > >
> > > > > > > 2. If varints are used, the case
> for
> > using
> > > > > attributes to
> > > > > > > indicate null
> > > > > > > headers also becomes weak. We only
> > add one
> > > > > additional byte in
> > > > > > > each message
> > > > > > > if there are no headers. Whatever
> > the case,
> > > > > let us be
> > > > > > > consistent with how
> > > > > > > we treat null keys and values.
> > > > > > >
> > > > > > > 3. We have apparently agreed that
> the
> > > broker
> > > > > will validate
> > > > > > > headers (please
> > > > > > > add this to the KIP). That being
> the
> > case,
> > > I
> > > > > would prefer to
> > > > > > > use Kafka's
> > > > > > > conventional format for arrays. The
> > > downside
> > > > > is that it takes
> > > > > > > more work to
> > > > > > > skip over the headers on the
> > consumer,
> > > though
> > > > > it's unclear if
> > > > > > > the cost
> > > > > > > would matter in practice. Some
> > concern was
> > > > > previously
> > > > > > > expressed about the
> > > > > > > allocation of maps. An alternative
> > would be
> > > > to
> > > > > use arrays,
> > > > > > i.e.
> > > > > > >
> > > > > > > class ConsumerRecord<K, V> {
> > > > > > > K key;
> > > > > > > V value;
> > > > > > > Header[] headers;
> > > > > > > }
> > > > > > >
> > > > > > > class Header {
> > > > > > > String key;
> > > > > > > byte[] value;
> > > > > > > }
> > > > > > >
> > > > > > > This would work nicely with the
> > > conventional
> > > > > array format and
> > > > > > > my guess is
> > > > > > > it would obviate the need do any
> lazy
> > > > > initialization. If we
> > > > > > > use the map as
> > > > > > > is currently documented, then it is
> > > possible
> > > > > with either
> > > > > > > representation to
> > > > > > > slice the headers and initialize
> them
> > > lazily.
> > > > > Either way, it
> > > > > > > might be a
> > > > > > > good idea to use a separate object
> to
> > > > > represent the headers
> > > > > > in
> > > > > > > case we need
> > > > > > > to modify it in the future in some
> > way.
> > > > > > >
> > > > > > > (By the way, doesn't it feel a bit
> > odd that
> > > > we
> > > > > seem to be
> > > > > > > designing a
> > > > > > > feature which is optimized for
> > people not
> > > > > using?)
> > > > > > >
> > > > > > >
> > > > > > > If we can resolve these points,
> then
> > at
> > > least
> > > > > you will get my
> > > > > > > vote.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jason
> > > > > > >
> > > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
> > radai <
> > > > > > > radai.rosenblatt@gmail.com> wrote:
> > > > > > >
> > > > > > > > headers dont "leak" into
> > application
> > > code.
> > > > > they are useful
> > > > > > > to application
> > > > > > > > code as well.
> > > > > > > > IIUC samze currently has headers
> > "in-V"
> > > and
> > > > > would just
> > > > > > > switch over to kafka
> > > > > > > > headers if they exist.
> > > > > > > > im sure plenty of other users of
> > kafka
> > > > would
> > > > > have a use for
> > > > > > > headers.
> > > > > > > > im pretty sure use cases exist
> > around
> > > > > shuffling data
> > > > > > > into/out-of kafka
> > > > > > > > (kafka connect or equivalent)
> where
> > > > metadata
> > > > > from one end
> > > > > > > could copied over
> > > > > > > > to the other (S3, for example
> uses
> > http
> > > > > headers for
> > > > > > > user-accessible
> > > > > > > > metadata). it will be kafka
> client
> > code
> > > > > getting/setting
> > > > > > > those headers. not
> > > > > > > > an interceptor.
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
> > Michael
> > > > > Pearce <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > For APM single event tracing,
> > need
> > > access
> > > > > to the header
> > > > > > at
> > > > > > > the point of
> > > > > > > > > processing on the processing
> > thread.
> > > > > > > > >
> > > > > > > > > As such interceptors will not
> > work/be
> > > > > suitable for these,
> > > > > > > due to the fact
> > > > > > > > > they act on the ConsumerRecords
> > as a
> > > > > batch, before the
> > > > > > > handling thread
> > > > > > > > can
> > > > > > > > > split out and process per
> > message which
> > > > is
> > > > > the point
> > > > > > these
> > > > > > > tools will
> > > > > > > > need
> > > > > > > > > to continue to transaction
> > tracing.
> > > > > > > > >
> > > > > > > > > Like wise tools and infra
> pieces
> > will
> > > > need
> > > > > access to the
> > > > > > > message outside
> > > > > > > > > the interceptor realm.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 17/02/2017, 21:26, "Jason
> > > Gustafson" <
> > > > > > > jason@confluent.io> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > That’s exactly what we’re
> > doing
> > > the
> > > > > headers are a
> > > > > > > slice of bytes,
> > > > > > > > > which
> > > > > > > > > > then gets parsed later if
> > needed,
> > > > or
> > > > > can be parsed
> > > > > > > right away, the
> > > > > > > > > headers
> > > > > > > > > > is part of the protocol,
> > so can
> > > > > still be validated
> > > > > > > if wanted.
> > > > > > > > > > If you had a header count
> > then
> > > you
> > > > > would have to go
> > > > > > > through each
> > > > > > > > > header
> > > > > > > > > > key and value length
> value
> > to
> > > work
> > > > > out how much to
> > > > > > > skip to get to
> > > > > > > > > say the
> > > > > > > > > > value or any future
> > component in
> > > > the
> > > > > message after
> > > > > > > the headers.
> > > > > > > > > Having it
> > > > > > > > > > as a byte[] with length
> > value
> > > makes
> > > > > this a lot
> > > > > > > easier to skip.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > So the broker will parse
> the
> > > headers
> > > > > and validate
> > > > > > > them. Good. The
> > > > > > > > only
> > > > > > > > > reason remaining that I can
> > see to
> > > > > leave the headers
> > > > > > > as a byte array
> > > > > > > > > is to
> > > > > > > > > make it easier for the
> > client to
> > > skip
> > > > > past them. Are
> > > > > > > we sure this is
> > > > > > > > > not
> > > > > > > > > premature optimization? Are
> > there
> > > any
> > > > > performance
> > > > > > > results which show
> > > > > > > > > that
> > > > > > > > > this is worthwhile?
> > > > > > > > >
> > > > > > > > > What’s the issue with
> > exposing a
> > > > > method getHeaders on
> > > > > > > the
> > > > > > > > > producer/consumer
> > > > > > > > > > record? It doesn’t break
> > > anything.
> > > > > We don’t need
> > > > > > any
> > > > > > > special
> > > > > > > > version.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > See my previous
> explanation.
> > What I
> > > > am
> > > > > trying to
> > > > > > > resist is the
> > > > > > > > headers
> > > > > > > > > becoming a general
> > > application-level
> > > > > facility. The
> > > > > > > primary use case
> > > > > > > > as
> > > > > > > > > far
> > > > > > > > > as I can tell is
> middleware,
> > which
> > > is
> > > > > the use case
> > > > > > > that the
> > > > > > > > > interceptors
> > > > > > > > > are providing.
> > > > > > > > >
> > > > > > > > > Current batch consumer
> model
> > and
> > > > > consumer
> > > > > > interceptors
> > > > > > > don’t work
> > > > > > > > where
> > > > > > > > > > headers need to be acted
> > on at
> > > per
> > > > > message level at
> > > > > > > time of
> > > > > > > > > processing,
> > > > > > > > > > very case is APM (the
> core
> > one),
> > > > > where the header
> > > > > > > value is used to
> > > > > > > > > continue
> > > > > > > > > > tracing.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I still don't understand
> the
> > point
> > > > > about batching.
> > > > > > The
> > > > > > > consumer
> > > > > > > > > records are
> > > > > > > > > exposed as a batch in the
> > consumer
> > > > > interceptor, but
> > > > > > > you can still
> > > > > > > > > iterate
> > > > > > > > > them individually. It is no
> > > different
> > > > > for the
> > > > > > consumer
> > > > > > > API itself.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > > On Fri, Feb 17, 2017 at
> > 12:48 PM,
> > > > > Michael Pearce <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Re:
> > > > > > > > > >
> > > > > > > > > > “ The point about
> > creation of
> > > > > maps seems
> > > > > > > orthogonal. We can
> > > > > > > > still
> > > > > > > > > > represent
> > > > > > > > > > the headers as a
> slice
> > of
> > > bytes
> > > > > until the time
> > > > > > > it is accessed.”
> > > > > > > > > >
> > > > > > > > > > That’s exactly what we’re
> > doing
> > > the
> > > > > headers are a
> > > > > > > slice of bytes,
> > > > > > > > > which
> > > > > > > > > > then gets parsed later if
> > needed,
> > > > or
> > > > > can be parsed
> > > > > > > right away, the
> > > > > > > > > headers
> > > > > > > > > > is part of the protocol,
> > so can
> > > > > still be validated
> > > > > > > if wanted.
> > > > > > > > > >
> > > > > > > > > > If you had a header count
> > then
> > > you
> > > > > would have to go
> > > > > > > through each
> > > > > > > > > header
> > > > > > > > > > key and value length
> value
> > to
> > > work
> > > > > out how much to
> > > > > > > skip to get to
> > > > > > > > > say the
> > > > > > > > > > value or any future
> > component in
> > > > the
> > > > > message after
> > > > > > > the headers.
> > > > > > > > > Having it
> > > > > > > > > > as a byte[] with length
> > value
> > > makes
> > > > > this a lot
> > > > > > > easier to skip.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 17/02/2017, 20:37,
> > "Michael
> > > > > Pearce" <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > What’s the issue with
> > > exposing
> > > > a
> > > > > method
> > > > > > > getHeaders on the
> > > > > > > > > > producer/consumer record?
> > It
> > > > doesn’t
> > > > > break
> > > > > > anything.
> > > > > > > We don’t need
> > > > > > > > > any
> > > > > > > > > > special version.
> > > > > > > > > >
> > > > > > > > > > Current batch
> consumer
> > model
> > > > and
> > > > > consumer
> > > > > > > interceptors don’t
> > > > > > > > work
> > > > > > > > > > where headers need to be
> > acted on
> > > > at
> > > > > per message
> > > > > > > level at time of
> > > > > > > > > > processing, very case is
> > APM (the
> > > > > core one), where
> > > > > > > the header value
> > > > > > > > > is used
> > > > > > > > > > to continue tracing.
> > JMS/HTTP etc
> > > > > all expose these,
> > > > > > > without issues.
> > > > > > > > > I would
> > > > > > > > > > NOT want to lock this
> down
> > to
> > > only
> > > > > be usable
> > > > > > > accessible via
> > > > > > > > > interceptors,
> > > > > > > > > > as we’d fail on one of
> the
> > main
> > > > > goals.
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Mike
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 17/02/2017, 20:21,
> > "Jason
> > > > > Gustafson" <
> > > > > > > jason@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > The point about
> > creation
> > > of
> > > > > maps seems
> > > > > > > orthogonal. We can
> > > > > > > > > still
> > > > > > > > > > represent
> > > > > > > > > > the headers as a
> > slice of
> > > > > bytes until the
> > > > > > > time it is
> > > > > > > > > accessed.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > Yes exactly we
> > have
> > > > access
> > > > > to the records
> > > > > > > thus why the
> > > > > > > > > header
> > > > > > > > > > should be
> > > > > > > > > > > accessible via
> > it and
> > > not
> > > > > hidden for only
> > > > > > > interceptors to
> > > > > > > > > access.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > As explained
> > above, the
> > > > > point is to make
> > > > > > the
> > > > > > > intended usage
> > > > > > > > > clear.
> > > > > > > > > > Applications
> should
> > > > continue
> > > > > to rely on the
> > > > > > > key/value
> > > > > > > > fields
> > > > > > > > > to
> > > > > > > > > > serialize
> > > > > > > > > > their own
> headers,
> > and it
> > > > > would be more
> > > > > > > ideal if we can
> > > > > > > > avoid
> > > > > > > > > > leaking
> > > > > > > > > > third-party
> > headers into
> > > > > applications. This
> > > > > > > is difficult to
> > > > > > > > > do
> > > > > > > > > > with the
> > > > > > > > > > current
> > interceptors
> > > > because
> > > > > they share the
> > > > > > > record objects
> > > > > > > > > with
> > > > > > > > > > the common
> > > > > > > > > > API. What I had
> in
> > mind
> > > is
> > > > > something like
> > > > > > an
> > > > > > > extension of
> > > > > > > > the
> > > > > > > > > > current
> > > > > > > > > > interceptors
> which
> > > exposed
> > > > a
> > > > > different
> > > > > > > object (e.g.
> > > > > > > > > > `RecordAndHeaders`).
> > > > > > > > > > The challenge is
> > for
> > > > MM-like
> > > > > use cases. Let
> > > > > > > me see if I can
> > > > > > > > > come
> > > > > > > > > > up with a
> > > > > > > > > > concrete proposal
> > for
> > > that
> > > > > problem.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 17,
> > 2017 at
> > > > > 11:55 AM, Michael
> > > > > > > Pearce <
> > > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I am happy to
> > move the
> > > > > definition of the
> > > > > > > header into the
> > > > > > > > > message
> > > > > > > > > > body, but
> > > > > > > > > > > would cause us
> > not to
> > > > lazy
> > > > > > > initialise/parse the headers,
> > > > > > > > as
> > > > > > > > > > obviously, we
> > > > > > > > > > > would have to
> > traverse
> > > > > these reading the
> > > > > > > message.
> > > > > > > > > > >
> > > > > > > > > > > This was
> > actually one
> > > of
> > > > > Jay’s requests:
> > > > > > > > > > >
> > > > > > > > > > > “ 2. I think
> > we
> > > should
> > > > > think about
> > > > > > > creating the Map
> > > > > > > > > lazily to
> > > > > > > > > > avoid
> > > > > > > > > > > parsing out
> > all the
> > > > > headers into
> > > > > > > little objects.
> > > > > > > > > HashMaps
> > > > > > > > > > themselves
> > > > > > > > > > > are
> > > > > > > > > > > kind of
> > expensive
> > > and
> > > > > the consumer is
> > > > > > > very perf
> > > > > > > > > sensitive so
> > > > > > > > > > and making
> > > > > > > > > > > gazillions
> of
> > > > hashmaps
> > > > > that may or
> > > > > > may
> > > > > > > not get used
> > > > > > > > is
> > > > > > > > > > probably a bad
> > > > > > > > > > > idea.”
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 17/02/2017,
> > 19:44,
> > > > > "Michael Pearce" <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Yes exactly
> > we have
> > > > > access to the
> > > > > > > records thus why
> > > > > > > > the
> > > > > > > > > > header should
> > > > > > > > > > > be accessible
> > via it
> > > and
> > > > > not hidden for
> > > > > > > only interceptors
> > > > > > > > > to
> > > > > > > > > > access.
> > > > > > > > > > >
> > > > > > > > > > > Sent using
> > OWA for
> > > > > iPhone
> > > > > > > > > > >
> > > > > ______________________________
> > > > > > > __________
> > > > > > > > > > > From:
> Magnus
> > > > Edenhill <
> > > > > > > magnus@edenhill.se>
> > > > > > > > > > > Sent:
> Friday,
> > > > February
> > > > > 17, 2017
> > > > > > > 7:34:49 PM
> > > > > > > > > > > To:
> > > > > dev@kafka.apache.org
> > > > > > > > > > > Subject:
> Re:
> > > > [DISCUSS]
> > > > > KIP-82 - Add
> > > > > > > Record Headers
> > > > > > > > > > >
> > > > > > > > > > > Big +1 on
> > VarInts.
> > > > > > > > > > > CPUs are
> > fast,
> > > memory
> > > > > is slow.
> > > > > > > > > > >
> > > > > > > > > > > I agree
> with
> > Jason
> > > > > that we'll want to
> > > > > > > continue
> > > > > > > > > verifying
> > > > > > > > > > messages,
> > > > > > > > > > > including
> > their
> > > > > headers, so while I
> > > > > > > appreciate the
> > > > > > > > > idea of
> > > > > > > > > > the opaque
> > > > > > > > > > > header blob
> > it
> > > won't
> > > > > be useful in
> > > > > > > practice.
> > > > > > > > > > >
> > > > > > > > > > > /Magnus
> > > > > > > > > > >
> > > > > > > > > > > 2017-02-17
> > 10:41
> > > > > GMT-08:00 Jason
> > > > > > > Gustafson <
> > > > > > > > > > jason@confluent.io>:
> > > > > > > > > > >
> > > > > > > > > > > > Sorry, my
> > > mistake.
> > > > > The consumer
> > > > > > > interceptor is per
> > > > > > > > > batch,
> > > > > > > > > > though I'm
> > > > > > > > > > > not
> > > > > > > > > > > > sure
> > that's an
> > > > > actual limitation
> > > > > > > since you still
> > > > > > > > have
> > > > > > > > > > access to the
> > > > > > > > > > > >
> individual
> > > records.
> > > > > > > > > > > >
> > > > > > > > > > > > -Jason
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri,
> > Feb 17,
> > > > 2017
> > > > > at 10:39 AM,
> > > > > > > Jason Gustafson <
> > > > > > > > > > >
> > jason@confluent.io>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Re
> > headers as
> > > > byte
> > > > > array and
> > > > > > > future use by
> > > > > > > > broker.
> > > > > > > > > This
> > > > > > > > > > doesn't
> > > > > > > > > > > take away
> > > > > > > > > > > > >> from
> > that at
> > > > all.
> > > > > Nor makes it
> > > > > > > difficult at all
> > > > > > > > > in my
> > > > > > > > > > opinion.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Yeah, I
> > didn't
> > > > say
> > > > > it was
> > > > > > > difficult, only
> > > > > > > > awkward.
> > > > > > > > > You
> > > > > > > > > > wouldn't
> > > > > > > > > > > write the
> > > > > > > > > > > > > schema
> > that way
> > > > if
> > > > > you were
> > > > > > > planning to use it on
> > > > > > > > > the
> > > > > > > > > > brokers from
> > > > > > > > > > > the
> > > > > > > > > > > > >
> > beginning. Note
> > > > > also that one of
> > > > > > > the benefits of
> > > > > > > > > letting
> > > > > > > > > > the broker
> > > > > > > > > > > > >
> > understand
> > > > headers
> > > > > is that it can
> > > > > > > validate that
> > > > > > > > > they are
> > > > > > > > > > properly
> > > > > > > > > > > > >
> > formatted. If
> > > > cost
> > > > > is the only
> > > > > > > concern, we should
> > > > > > > > > > confirm its
> > > > > > > > > > > impact
> > > > > > > > > > > > > through
> > > > > performance testing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > One of
> > the key
> > > > use
> > > > > cases requires
> > > > > > > access on
> > > > > > > > > consume at
> > > > > > > > > > per
> > > > > > > > > > > event/message
> > > > > > > > > > > > >> level
> > at the
> > > > > point that message
> > > > > > > is being
> > > > > > > > > processed, as
> > > > > > > > > > such the
> > > > > > > > > > > batch
> > > > > > > > > > > > >>
> > interceptors
> > > and
> > > > > batch consume
> > > > > > > api isn't
> > > > > > > > > suitable. It
> > > > > > > > > > needs to be
> > > > > > > > > > > at the
> > > > > > > > > > > > >> record
> > level.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm not
> > sure I
> > > > > understand the
> > > > > > > point about
> > > > > > > > batching.
> > > > > > > > > > Interceptors
> > > > > > > > > > > are
> > > > > > > > > > > > > applied
> > > > > per-message, right?
> > > > > > > > > > > > >
> > > > > > > > > > > > > My
> > intent on
> > > > > interceptors is to
> > > > > > > keep the usage of
> > > > > > > > > headers
> > > > > > > > > > > well-defined so
> > > > > > > > > > > > > that
> > they don't
> > > > > start leaking
> > > > > > > unnecessarily into
> > > > > > > > > > applications. My
> > > > > > > > > > > guess
> > > > > > > > > > > > is
> > > > > > > > > > > > > that
> it's
> > > > probably
> > > > > inevitable,
> > > > > > but
> > > > > > > isolating it
> > > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > interceptors
> > would
> > > > > > > > > > > > > at
> least
> > give
> > > > > people a second
> > > > > > > thought before
> > > > > > > > > deciding to
> > > > > > > > > > use it.
> > > > > > > > > > > The main
> > > > > > > > > > > > >
> > challenge in my
> > > > > mind is figuring
> > > > > > > out how an MM
> > > > > > > > use
> > > > > > > > > case
> > > > > > > > > > would
> > > > > > > > > > > work. It
> > > > > > > > > > > > > would
> be
> > more
> > > > > cumbersome to
> > > > > > > replicate headers
> > > > > > > > > through an
> > > > > > > > > > > interceptor,
> > > > > > > > > > > > > though
> > arguably
> > > > MM
> > > > > should be
> > > > > > > working at a lower
> > > > > > > > > level
> > > > > > > > > > anyway.
> > > > > > > > > > > > >
> > > > > > > > > > > > > -Jason
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri,
> > Feb 17,
> > > > > 2017 at 10:16 AM,
> > > > > > > Michael Pearce
> > > > > > > > <
> > > > > > > > > > >
> > Michael.Pearce@ig.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Re
> > headers
> > > > > available on the
> > > > > > > record va
> > > > > > > > > interceptors only
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> One of
> > the key
> > > > > use cases
> > > > > > requires
> > > > > > > access on
> > > > > > > > > consume at
> > > > > > > > > > per
> > > > > > > > > > > event/message
> > > > > > > > > > > > >> level
> > at the
> > > > > point that message
> > > > > > > is being
> > > > > > > > > processed, as
> > > > > > > > > > such the
> > > > > > > > > > > batch
> > > > > > > > > > > > >>
> > interceptors
> > > and
> > > > > batch consume
> > > > > > > api isn't
> > > > > > > > > suitable. It
> > > > > > > > > > needs to be
> > > > > > > > > > > at the
> > > > > > > > > > > > >> record
> > level.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> This
> > anyhow is
> > > > > similar to
> > > > > > > jms/http/amqp where
> > > > > > > > > headers
> > > > > > > > > > are
> > > > > > > > > > > available to
> > > > > > > > > > > > >>
> > consuming
> > > > > applications.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Re
> > headers as
> > > > > byte array and
> > > > > > > future use by
> > > > > > > > > broker. This
> > > > > > > > > > doesn't
> > > > > > > > > > > take
> > > > > > > > > > > > away
> > > > > > > > > > > > >> from
> > that at
> > > > all.
> > > > > Nor makes it
> > > > > > > difficult at all
> > > > > > > > > in my
> > > > > > > > > > opinion.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Sent
> > using OWA
> > > > > for iPhone
> > > > > > > > > > > > >>
> > > > > ______________________________
> > > > > > > __________
> > > > > > > > > > > > >> From:
> > Jason
> > > > > Gustafson <
> > > > > > > jason@confluent.io>
> > > > > > > > > > > > >> Sent:
> > Friday,
> > > > > February 17, 2017
> > > > > > > 5:55:42 PM
> > > > > > > > > > > > >> To:
> > > > > dev@kafka.apache.org
> > > > > > > > > > > > >>
> > Subject: Re:
> > > > > [DISCUSS] KIP-82 -
> > > > > > > Add Record
> > > > > > > > Headers
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> Would
> > you be
> > > > > proposing in
> > > > > > > KIP-98 to convert
> > > > > > > > the
> > > > > > > > > other
> > > > > > > > > > message
> > > > > > > > > > > int’s
> > > > > > > > > > > > (key
> > > > > > > > > > > > >> >
> > length,
> > > value
> > > > > length) also to
> > > > > > > varint to keep
> > > > > > > > it
> > > > > > > > > > uniform.
> > > > > > > > > > > > >> >
> Also I
> > > assume
> > > > > there will be a
> > > > > > > static or helper
> > > > > > > > > method
> > > > > > > > > > made to
> > > > > > > > > > > >
> write/read
> > > > > > > > > > > > >> >
> these
> > in the
> > > > > client and
> > > > > > server.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Yes,
> > that is
> > > > what
> > > > > we are
> > > > > > > proposing, so using
> > > > > > > > > varints
> > > > > > > > > > for headers
> > > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > >>
> > consistent
> > > with
> > > > > the rest of the
> > > > > > > message. We have
> > > > > > > > > used
> > > > > > > > > > static
> > > > > > > > > > > helper
> > > > > > > > > > > > >>
> methods
> > > > > > > > > > > > >> in our
> > > prototype
> > > > > implementation.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> The
> > cost of
> > > > > parsing, we want to
> > > > > > > parse/interpret
> > > > > > > > > the
> > > > > > > > > > headers
> > > > > > > > > > > lazily (this
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >> > a
> key
> > point
> > > > > brought up earlier
> > > > > > > in discussions)
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I'm a
> > bit
> > > > > skeptical of this. Has
> > > > > > > anyone done the
> > > > > > > > > > performance
> > > > > > > > > > > testing? I
> > > > > > > > > > > > >> can
> > > > > > > > > > > > >>
> probably
> > > > > implement it and test
> > > > > > it
> > > > > > > if no one else
> > > > > > > > > has. I
> > > > > > > > > > was also
> > > > > > > > > > > under
> > > > > > > > > > > > the
> > > > > > > > > > > > >>
> > impression
> > > that
> > > > > there may be use
> > > > > > > cases down the
> > > > > > > > > road
> > > > > > > > > > where the
> > > > > > > > > > > broker
> > > > > > > > > > > > >> would
> > > > > > > > > > > > >> need
> to
> > > > interpret
> > > > > headers. That
> > > > > > > wouldn't be off
> > > > > > > > > the
> > > > > > > > > > table in the
> > > > > > > > > > > future
> > > > > > > > > > > > if
> > > > > > > > > > > > >> it's
> > > represented
> > > > > as bytes, but
> > > > > > it
> > > > > > > would be quite
> > > > > > > > > a bit
> > > > > > > > > > more
> > > > > > > > > > > awkward,
> > > > > > > > > > > > >> right?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> By the
> > way,
> > > one
> > > > > question I have
> > > > > > > been wondering
> > > > > > > > > about. My
> > > > > > > > > > > understanding
> > > > > > > > > > > > is
> > > > > > > > > > > > >> that
> > headers
> > > are
> > > > > primarily for
> > > > > > > use cases where a
> > > > > > > > > > third-party
> > > > > > > > > > > components
> > > > > > > > > > > > >> wants
> to
> > > enrich
> > > > > messages without
> > > > > > > needing to
> > > > > > > > > understand
> > > > > > > > > > or modify
> > > > > > > > > > > the
> > > > > > > > > > > > >> schema
> > > > > > > > > > > > >> of the
> > message
> > > > > key and value.
> > > > > > For
> > > > > > > the
> > > > > > > > applications
> > > > > > > > > > which directly
> > > > > > > > > > > > produce
> > > > > > > > > > > > >> and
> > consume
> > > the
> > > > > messages and
> > > > > > > control the
> > > > > > > > key/value
> > > > > > > > > > schema
> > > > > > > > > > > directly, it
> > > > > > > > > > > > >> seems
> > we would
> > > > > rather have them
> > > > > > > implement
> > > > > > > > headers
> > > > > > > > > > directly in
> > > > > > > > > > > their own
> > > > > > > > > > > > >>
> schema.
> > > > Supposing
> > > > > for the sake
> > > > > > of
> > > > > > > argument that
> > > > > > > > > it was
> > > > > > > > > > possible,
> > > > > > > > > > > my
> > > > > > > > > > > > >>
> > question is
> > > > > whether it be
> > > > > > > sufficient to expose
> > > > > > > > the
> > > > > > > > > > headers in the
> > > > > > > > > > > > >>
> > interceptor
> > > API
> > > > > and not in the
> > > > > > > common API?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -Jason
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On
> Fri,
> > Feb
> > > 17,
> > > > > 2017 at 3:26 AM,
> > > > > > > Michael Pearce
> > > > > > > > <
> > > > > > > > > > >
> > Michael.Pearce@ig.com>
> > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > On
> > the point
> > > > of
> > > > > varInts
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> Would
> > you be
> > > > > proposing in
> > > > > > > KIP-98 to convert
> > > > > > > > the
> > > > > > > > > other
> > > > > > > > > > message
> > > > > > > > > > > int’s
> > > > > > > > > > > > (key
> > > > > > > > > > > > >> >
> > length,
> > > value
> > > > > length) also to
> > > > > > > varint to keep
> > > > > > > > it
> > > > > > > > > > uniform.
> > > > > > > > > > > > >> >
> Also I
> > > assume
> > > > > there will be a
> > > > > > > static or helper
> > > > > > > > > method
> > > > > > > > > > made to
> > > > > > > > > > > >
> write/read
> > > > > > > > > > > > >> >
> these
> > in the
> > > > > client and
> > > > > > server.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> Cheers
> > > > > > > > > > > > >> > Mike
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On
> > > 17/02/2017,
> > > > > 11:22, "Michael
> > > > > > > Pearce" <
> > > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > On the
> > > > > point re: headers
> > > > > > in
> > > > > > > the message
> > > > > > > > > protocol
> > > > > > > > > > being a
> > > > > > > > > > > byte
> > > > > > > > > > > > array
> > > > > > > > > > > > >> > and
> > not a
> > > > count
> > > > > of elements
> > > > > > > followed by the
> > > > > > > > > elements.
> > > > > > > > > > Again
> > > > > > > > > > > this was
> > > > > > > > > > > > >> >
> > > > > discussed/argued previously.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > It was
> > > > > agreed on for a few
> > > > > > > reasons some of
> > > > > > > > > which
> > > > > > > > > > you have
> > > > > > > > > > > > obviously
> > > > > > > > > > > > >> >
> > picked up
> > > on:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > Broker
> > > is
> > > > > able to pass it
> > > > > > > through opaquely
> > > > > > > > > > > > >> >
> > The cost
> > > > of
> > > > > parsing, we
> > > > > > > want to
> > > > > > > > > parse/interpret
> > > > > > > > > > the headers
> > > > > > > > > > > lazily
> > > > > > > > > > > > >> >
> (this
> > is a
> > > key
> > > > > point brought
> > > > > > up
> > > > > > > earlier in
> > > > > > > > > > discussions)
> > > > > > > > > > > > >> >
> > Headers
> > > > can
> > > > > be copied from
> > > > > > > consumer record
> > > > > > > > > to
> > > > > > > > > > producer
> > > > > > > > > > > record (aka
> > > > > > > > > > > > >> >
> mirror
> > > makers
> > > > > etc) without
> > > > > > > parsing if no
> > > > > > > > > changes are
> > > > > > > > > > being made
> > > > > > > > > > > or
> > > > > > > > > > > > being
> > > > > > > > > > > > >> >
> > looked at.
> > > > > > > > > > > > >> >
> > Keeps
> > > the
> > > > > broker agnostic
> > > > > > > to the format
> > > > > > > > > > > > >> >
> > You need
> > > > an
> > > > > int32 either
> > > > > > > for the byte size
> > > > > > > > > of the
> > > > > > > > > > headers,
> > > > > > > > > > > or for
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >
> count
> > of
> > > > > elements, so
> > > > > > overheads
> > > > > > > are the same,
> > > > > > > > > but
> > > > > > > > > > with going
> > > > > > > > > > > with an
> > > > > > > > > > > > >> opaque
> > > > > > > > > > > > >> > byte
> > array
> > > has
> > > > > the above
> > > > > > > advantages.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > Cheers
> > > > > > > > > > > > >> >
> > Mike
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> On
> > > > > 17/02/2017, 02:50,
> > > > > > > "Jason Gustafson" <
> > > > > > > > > > jason@confluent.io
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > Sorry,
> > > > > should have
> > > > > > > noted that the
> > > > > > > > > performance
> > > > > > > > > > testing
> > > > > > > > > > > was done
> > > > > > > > > > > > >> >
> using
> > the
> > > > > > > > > > > > >> >
> > > > > producer performance
> > > > > > > tool shipped with
> > > > > > > > > Kafka.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > -Jason
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > On
> > > > Thu,
> > > > > Feb 16, 2017
> > > > > > at
> > > > > > > 6:44 PM, Jason
> > > > > > > > > > Gustafson <
> > > > > > > > > > > > >> >
> > > > > jason@confluent.io> wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > >
> > > Hey
> > > > > Nacho,
> > > > > > > > > > > > >> >
> > >
> > > > > > > > > > > > >> >
> > >
> > > I've
> > > > > compared
> > > > > > > performance of our
> > > > > > > > > KIP-98
> > > > > > > > > > > implementation
> > with
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> >
> > without
> > > > > > > > > > > > >> >
> > >
> > > > > varints. For
> > > > > > messages
> > > > > > > around 128
> > > > > > > > > bytes, we
> > > > > > > > > > see an
> > > > > > > > > > > increase
> > > > > > > > > > > > in
> > > > > > > > > > > > >> >
> > throughput
> > > of
> > > > > > > > > > > > >> >
> > >
> > > > about
> > > > > 30% using the
> > > > > > > default
> > > > > > > > > configuration
> > > > > > > > > > settings.
> > > > > > > > > > > At 256
> > > > > > > > > > > > >> >
> > bytes, the
> > > > > > > > > > > > >> >
> > >
> > > > > increase is around
> > > > > > > 16%. Obviously
> > > > > > > > the
> > > > > > > > > > performance
> > > > > > > > > > > converge
> > > > > > > > > > > > as
> > > > > > > > > > > > >> >
> > messages get
> > > > > > > > > > > > >> >
> > >
> > > > > larger, but it seems
> > > > > > > well worth the
> > > > > > > > > cost.
> > > > > > > > > > Note that
> > > > > > > > > > > we are
> > > > > > > > > > > > >> also
> > > > > > > > > > > > >> >
> > seeing a
> > > > > > > > > > > > >> >
> > >
> > > > > substantial
> > > > > > > performance increase
> > > > > > > > > against
> > > > > > > > > > trunk
> > > > > > > > > > > primarily
> > > > > > > > > > > > >>
> because
> > > > > > > > > > > > >> > of
> the
> > > > > > > > > > > > >> >
> > >
> > > much
> > > > > more efficient
> > > > > > > packing that
> > > > > > > > > varints
> > > > > > > > > > provide us.
> > > > > > > > > > > > Anything
> > > > > > > > > > > > >> >
> > adding to
> > > > > > > > > > > > >> >
> > >
> > > > > message overhead,
> > > > > > > such as record
> > > > > > > > > headers,
> > > > > > > > > > would only
> > > > > > > > > > > > increase
> > > > > > > > > > > > >> > the
> > relative
> > > > > > > > > > > > >> >
> > >
> > > > > difference. (Of
> > > > > > > course take these
> > > > > > > > > numbers
> > > > > > > > > > with a
> > > > > > > > > > > grain of
> > > > > > > > > > > > salt
> > > > > > > > > > > > >> >
> since
> > I have
> > > > > > > > > > > > >> >
> > >
> > > only
> > > > > used the
> > > > > > default
> > > > > > > settings with
> > > > > > > > > both
> > > > > > > > > > the producer
> > > > > > > > > > > and
> > > > > > > > > > > > >> broker
> > > > > > > > > > > > >> > on
> my
> > > > > > > > > > > > >> >
> > >
> > > > local
> > > > > machine. We
> > > > > > > intend to provide
> > > > > > > > > more
> > > > > > > > > > extensive
> > > > > > > > > > > >
> performance
> > > > > > > > > > > > >> >
> > details as
> > > > > > > > > > > > >> >
> > >
> > > part
> > > > > of the work for
> > > > > > > KIP-98.)
> > > > > > > > > > > > >> >
> > >
> > > > > > > > > > > > >> >
> > >
> > > The
> > > > > implementation
> > > > > > we
> > > > > > > are using is
> > > > > > > > > from
> > > > > > > > > > protobuf (
> > > > > > > > > > > > >> >
> > >
> > > > > > > https://developers.google.com/
> > > > > > > > > > > >
> > > > > protocol-buffers/docs/encoding
> > > > > > > > > > > > >> ),
> > > > > > > > > > > > >> >
> which
> > is
> > > > > > > > > > > > >> >
> > >
> > > also
> > > > > used in HBase.
> > > > > > > It is trivial to
> > > > > > > > > > implement and as
> > > > > > > > > > > far
> > > > > > > > > > > > as I
> > > > > > > > > > > > >> > know
> > doesn't
> > > > > > > > > > > > >> >
> > >
> > > > > suffer from the
> > > > > > > aliasing problem you
> > > > > > > > > are
> > > > > > > > > > describing. I
> > > > > > > > > > > > checked
> > > > > > > > > > > > >> > with
> > Magnus
> > > > > > > > > > > > >> >
> > >
> > > (the
> > > > > author of
> > > > > > > librdkafka) and he
> > > > > > > > > agreed
> > > > > > > > > > that the
> > > > > > > > > > > savings
> > > > > > > > > > > > >> seemed
> > > > > > > > > > > > >> >
> worth
> > the
> > > > > > > > > > > > >> >
> > >
> > > cost
> > > > > of
> > > > > > > implementation.
> > > > > > > > > > > > >> >
> > >
> > > > > > > > > > > > >> >
> > >
> > > > -Jason
> > > > > > > > > > > > >> >
> > >
> > > > > > > > > > > > >> >
> > > On
> > > > > Thu, Feb 16, 2017
> > > > > > > at 4:32 PM,
> > > > > > > > > Ignacio
> > > > > > > > > > Solis <
> > > > > > > > > > > > >>
> > > isolis@igso.net
> > > > >
> > > > > > > > > > > > >> >
> wrote:
> > > > > > > > > > > > >> >
> > >
> > > > > > > > > > > > >> >
> > >>
> > > > > -VarInts
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > I'm
> > > > > one of the
> > > > > > > people (if not the
> > > > > > > > > most)
> > > > > > > > > > opposed to
> > > > > > > > > > > VarInts.
> > > > > > > > > > > > >> >
> > VarInts
> > > > > > > > > > > > >> >
> > >>
> > > > have
> > > > > a place, but
> > > > > > > this is not it.
> > > > > > > > > (We
> > > > > > > > > > had a large
> > > > > > > > > > > > >>
> > discussion
> > > > > > > > > > > > >> >
> about
> > > > > > > > > > > > >> >
> > >>
> > > > them
> > > > > at the
> > > > > > > beginning of KIP-82
> > > > > > > > time)
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > If
> > > > > anybody has real
> > > > > > > life
> > > > > > > > performance
> > > > > > > > > > numbers of
> > > > > > > > > > > VarInts
> > > > > > > > > > > > >> >
> > improving
> > > > > > > > > > > > >> >
> > >>
> > > > > things or
> > > > > > > significantly reducing
> > > > > > > > > resources
> > > > > > > > > > I would
> > > > > > > > > > > like to
> > > > > > > > > > > > >> know
> > > > > > > > > > > > >> > what
> > > > > > > > > > > > >> >
> > >>
> > > > that
> > > > > case may be.
> > > > > > > Yes, you can save
> > > > > > > > > some
> > > > > > > > > > bytes here
> > > > > > > > > > > and
> > > > > > > > > > > > >> there,
> > > > > > > > > > > > >> > but
> > > > > > > > > > > > >> >
> > >>
> > > > this
> > > > > is probably
> > > > > > > insignificant to
> > > > > > > > the
> > > > > > > > > > overall system
> > > > > > > > > > > > behavior
> > > > > > > > > > > > >> > and
> > > > > > > > > > > > >> >
> > >>
> > > > > storage
> > > > > > > requirements. -- I say
> > > > > > > > this
> > > > > > > > > with
> > > > > > > > > > respect to
> > > > > > > > > > > using
> > > > > > > > > > > > >> >
> > VarInts in
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > protocol
> > > > > > itself,
> > > > > > > not as part of
> > > > > > > > > the
> > > > > > > > > > data.
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > > > VarInts require you
> > > > > > > to parse the
> > > > > > > > Int
> > > > > > > > > > before using it
> > > > > > > > > > > and
> > > > > > > > > > > > >> >
> > depending on
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > encoding they
> > > > > > > can suffer from
> > > > > > > > > aliasing
> > > > > > > > > > (multiple
> > > > > > > > > > > > >> >
> > > > representations
> > > > > > > > > > > > >> >
> > >>
> > > for
> > > > > the same
> > > > > > value).
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > Why
> > > > > add complexity?
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > > Nacho
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > On
> > > > > Thu, Feb 16,
> > > > > > 2017
> > > > > > > at 10:29 AM,
> > > > > > > > > Colin
> > > > > > > > > > McCabe <
> > > > > > > > > > > > >> >
> > > > > cmccabe@apache.org>
> > > > > > > > > > > > >> >
> > >>
> > > > > wrote:
> > > > > > > > > > > > >> >
> > >> >
> > > > +1
> > > > > for varints
> > > > > > > here-- it would
> > > > > > > > save
> > > > > > > > > > quite a bit of
> > > > > > > > > > > space.
> > > > > > > > > > > > >> > They
> > are
> > > > > > > > > > > > >> >
> > >> >
> > > > > pretty quick to
> > > > > > > implement as
> > > > > > > > well.
> > > > > > > > > > > > >> >
> > >> >
> > > > > > > > > > > > >> >
> > >>
> > > > I
> > > > > think it makes
> > > > > > > sense for values
> > > > > > > > > to be
> > > > > > > > > > byte
> > > > > > > > > > > arrays.
> > > > > > > > > > > > Users
> > > > > > > > > > > > >> >
> might
> > want
> > > > > > > > > > > > >> >
> > >> >
> > > > to
> > > > > attach
> > > > > > > arbitrary payloads;
> > > > > > > > they
> > > > > > > > > > shouldn't be
> > > > > > > > > > > forced to
> > > > > > > > > > > > >> >
> > serialize
> > > > > > > > > > > > >> >
> > >> >
> > > > > everything to
> > > > > > Java
> > > > > > > strings.
> > > > > > > > > > > > >> >
> > >> >
> > > > > > > > > > > > >> >
> > >> >
> > > > > best,
> > > > > > > > > > > > >> >
> > >> >
> > > > > Colin
> > > > > > > > > > > > >> >
> > >> >
> > > > > > > > > > > > >> >
> > >> >
> > > > > > > > > > > > >> >
> > >> >
> > > > On
> > > > > Thu, Feb 16,
> > > > > > > 2017, at 09:52,
> > > > > > > > > Jason
> > > > > > > > > > Gustafson
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > Hey Michael,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > Hmm, I guess the
> > > > > > > point of
> > > > > > > > > representing
> > > > > > > > > > it as
> > > > > > > > > > > bytes is to
> > > > > > > > > > > > >> >
> allow
> > the
> > > > > > > > > > > > >> >
> > >>
> > > > > broker
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > to pass it
> > > > > > > through opaquely? Is
> > > > > > > > > the
> > > > > > > > > > cost of
> > > > > > > > > > > parsing
> > > > > > > > > > > > them a
> > > > > > > > > > > > >> >
> > concern, or
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > are
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > we simply trying
> > > > > > > to ensure that
> > > > > > > > > the
> > > > > > > > > > broker stays
> > > > > > > > > > > > agnostic
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > the
> > > > > > > > > > > > >> >
> > >>
> > > > > format?
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > On varints, I
> > > > > > > think adding
> > > > > > > > > support for
> > > > > > > > > > them makes
> > > > > > > > > > > less
> > > > > > > > > > > > >> sense
> > > > > > > > > > > > >> > for
> an
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > isolated use
> > > > > > > case, but as part
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > > more holistic
> > > > > > > > > > > change
> > > > > > > > > > > > >> >
> (such
> > as
> > > what
> > > > > > > > > > > > >> >
> > >>
> > > we
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > have proposed in
> > > > > > > KIP-98), I
> > > > > > > > think
> > > > > > > > > they
> > > > > > > > > > are
> > > > > > > > > > > justifiable.
> > > > > > > > > > > > If
> > > > > > > > > > > > >> > we
> > add them,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > then the need to
> > > > > > > use attributes
> > > > > > > > > becomes
> > > > > > > > > > quite a
> > > > > > > > > > > bit
> > > > > > > > > > > > >>
> weaker,
> > > > > > > > > > > > >> >
> > right? The
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > other thing I
> > > > > > > find slightly odd
> > > > > > > > > is the
> > > > > > > > > > fact that
> > > > > > > > > > > null
> > > > > > > > > > > > >> >
> > headers has
> > > no
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > actual
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > semantic meaning
> > > > > > > for the message
> > > > > > > > > > (unlike null
> > > > > > > > > > > keys and
> > > > > > > > > > > > >> >
> > values). It
> > > is
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > just
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > a
> > > > > space
> > > > > > > optimization. It seems a
> > > > > > > > > bit
> > > > > > > > > > better to
> > > > > > > > > > > always
> > > > > > > > > > > > use
> > > > > > > > > > > > >> > size
> > 0 to
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > indicate having
> > > > > > > no headers.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > Overall, the
> > > > > > main
> > > > > > > point is
> > > > > > > > > ensuring
> > > > > > > > > > that the
> > > > > > > > > > > message
> > > > > > > > > > > > >> schema
> > > > > > > > > > > > >> >
> > remains
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > consistent,
> > > > > > > either within the
> > > > > > > > > larger
> > > > > > > > > > protocol, or
> > > > > > > > > > > at a
> > > > > > > > > > > > >> >
> > minimum
> > > within
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > message itself.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > -Jason
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > On Thu, Feb 16,
> > > > > > > 2017 at 6:39 AM,
> > > > > > > > > > Michael Pearce <
> > > > > > > > > > > > >> >
> > > > > Michael.Pearce@ig.com
> > > > > > > > > > > > >> >
> > >> >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > wrote:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Hi Jason,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > On point 1) in
> > > > > > > the message
> > > > > > > > > protocol
> > > > > > > > > > the headers
> > > > > > > > > > > are
> > > > > > > > > > > > >> simply
> > > > > > > > > > > > >> > a
> byte
> > > > > > > > > > > > >> >
> > >>
> > > > > array,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > as like the
> > > > > > key
> > > > > > > or value, this
> > > > > > > > > is to
> > > > > > > > > > clearly
> > > > > > > > > > > demarcate
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >
> > header in
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > core message.
> > > > > > > Then the header
> > > > > > > > > byte
> > > > > > > > > > array in the
> > > > > > > > > > > core
> > > > > > > > > > > > >> >
> > message is
> > > an
> > > > > > > > > > > > >> >
> > >>
> > > > > array of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > key, value
> > > > > > > pairs. This is what
> > > > > > > > > it is
> > > > > > > > > > denoting.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Then this
> > > > > > would
> > > > > > > be I guess in
> > > > > > > > > the
> > > > > > > > > > given
> > > > > > > > > > > notation:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Headers =>
> > > > > > > [KeyLength, Key,
> > > > > > > > > > ValueLength, Value]
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > KeyLength
> > > > > > > => int32
> > > > > > > > > > <-----------------NEW
> > > > > > > > > > > size of
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >
> > byte[] of
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > serialised key
> > > > > > > value
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > Key =>
> > > > > > bytes
> > > > > > > > > > <----------------------
> NEW
> > > > > > > > > > > >
> serialised
> > > > > > > > > > > > >> >
> string
> > > (UTF8)
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > bytes of the
> > > > > > > header key
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > ValueLength
> > > > > > > => int32
> > > > > > > > > > <-------------- NEW
> > > > > > > > > > > size of
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >
> > byte[] of
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > serialised
> > > > > > > header value
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > Value =>
> > > > > > > bytes
> > > > > > > > > > <-------------------- NEW
> > > > > > > > > > > >
> serialised
> > > > > > > > > > > > >> > form
> > of the
> > > > > > > > > > > > >> >
> > >>
> > > > > header
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > value
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > The key length
> > > > > > > and value
> > > > > > > > length
> > > > > > > > > is
> > > > > > > > > > matching the
> > > > > > > > > > > way
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >
> > protocol is
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > defined in the
> > > > > > > core message
> > > > > > > > > currently.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > On point 2)
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Var sized
> > > > > > ints,
> > > > > > > this was
> > > > > > > > > discussed
> > > > > > > > > > much earlier
> > > > > > > > > > > on, in
> > > > > > > > > > > > >> > fact
> > I had
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > suggested it
> > > > > > > myself (with
> > > > > > > > Hadoop
> > > > > > > > > > references),
> > > > > > > > > > > the
> > > > > > > > > > > > >> >
> > complexity
> > > of
> > > > > this
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > compared to
> > > > > > > having a simpler
> > > > > > > > > protocol
> > > > > > > > > > was
> > > > > > > > > > > argued and
> > > > > > > > > > > > >> >
> > agreed it
> > > > > > > > > > > > >> >
> > >>
> > > > > wasn’t worth
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > the complexity
> > > > > > > as all other
> > > > > > > > > clients
> > > > > > > > > > in other
> > > > > > > > > > > languages
> > > > > > > > > > > > >> >
> would
> > need
> > > to
> > > > > > > > > > > > >> >
> > >>
> > > > > ensure
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > theyre using
> > > > > > > the right var
> > > > > > > > size
> > > > > > > > > > algorithm, as
> > > > > > > > > > > there
> > > > > > > > > > > > is a
> > > > > > > > > > > > >> > few.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > On point 3)
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > We did the
> > > > > > > attributes,
> > > > > > > > optional
> > > > > > > > > > approach as
> > > > > > > > > > > originally
> > > > > > > > > > > > >> >
> there
> > was
> > > > > > > > > > > > >> >
> > >>
> > > > > marked
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > concern that
> > > > > > > headers would
> > > > > > > > > cause a
> > > > > > > > > > message size
> > > > > > > > > > > > overhead
> > > > > > > > > > > > >> > for
> > others,
> > > > > > > > > > > > >> >
> > >>
> > > who
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > don’t want
> > > > > > > them. As such this
> > > > > > > > > is the
> > > > > > > > > > clean
> > > > > > > > > > > solution to
> > > > > > > > > > > > >> >
> > achieve
> > > that.
> > > > > > > > > > > > >> >
> > >>
> > > If
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > that no longer
> > > > > > > holds, and we
> > > > > > > > > don’t
> > > > > > > > > > care that we
> > > > > > > > > > > add
> > > > > > > > > > > > >> 4bytes
> > > > > > > > > > > > >> >
> > overhead,
> > > > > > > > > > > > >> >
> > >>
> > > > then
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > im happy to
> > > > > > > remove.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > I’m personally
> > > > > > > in favour of
> > > > > > > > > keeping
> > > > > > > > > > the message
> > > > > > > > > > > as
> > > > > > > > > > > > small
> > > > > > > > > > > > >> > as
> > possible
> > > > > > > > > > > > >> >
> > >>
> > > so
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > people don’t
> > > > > > > get shocks in
> > > > > > > > perf
> > > > > > > > > and
> > > > > > > > > > throughputs
> > > > > > > > > > > dues
> > > > > > > > > > > > to
> > > > > > > > > > > > >> >
> > message
> > > size,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > unless they
> > > > > > > actively use the
> > > > > > > > > feature,
> > > > > > > > > > as such I
> > > > > > > > > > > do
> > > > > > > > > > > > >> prefer
> > > > > > > > > > > > >> > the
> > > > > > > > > > > > >> >
> > >>
> > > > > attribute bit
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > wise feature
> > > > > > > flag approach
> > > > > > > > > myself.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > On 16/02/2017,
> > > > > > > 05:40, "Jason
> > > > > > > > > > Gustafson" <
> > > > > > > > > > > > >> >
> > > > > jason@confluent.io> wrote:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > We have
> > > > > > > proposed a few
> > > > > > > > > > significant changes
> > > > > > > > > > > to the
> > > > > > > > > > > > >> >
> > message
> > > format
> > > > > > > > > > > > >> >
> > >>
> > > in
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > KIP-98
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > which now
> > > > > > > seems likely to
> > > > > > > > > pass
> > > > > > > > > > (perhaps
> > > > > > > > > > > with some
> > > > > > > > > > > > >> >
> > iterations
> > > on
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > implementation details).
> > > > > > > > It
> > > > > > > > > would
> > > > > > > > > > be good
> > > > > > > > > > > to try
> > > > > > > > > > > > and
> > > > > > > > > > > > >> >
> > coordinate
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > changes
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > in both of
> > > > > > > the proposals
> > > > > > > > to
> > > > > > > > > make
> > > > > > > > > > sure they
> > > > > > > > > > > are
> > > > > > > > > > > > >> >
> > consistent
> > > and
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > compatible.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > I think
> > > > > > > using the
> > > > > > > > > attributes to
> > > > > > > > > > indicate
> > > > > > > > > > > null
> > > > > > > > > > > > >>
> headers
> > > > > > > > > > > > >> > is a
> > > > > > > > > > > > >> >
> > >>
> > > > > reasonable
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > approach.
> > > > > > > We have proposed
> > > > > > > > > to do
> > > > > > > > > > the same
> > > > > > > > > > > thing
> > > > > > > > > > > > for
> > > > > > > > > > > > >> > the
> > message
> > > > > > > > > > > > >> >
> > >>
> > > key
> > > > > and
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > value.
> > > > > > That
> > > > > > > said, I
> > > > > > > > > sympathize
> > > > > > > > > > with Jay's
> > > > > > > > > > > > argument.
> > > > > > > > > > > > >> >
> Having
> > > > > > > > > > > > >> >
> > >>
> > > > > multiple
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > ways to
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > specify a
> > > > > > > null value
> > > > > > > > > increases
> > > > > > > > > > the overall
> > > > > > > > > > > > >>
> > complexity
> > > > > > > > > > > > >> > of
> the
> > > > > > > > > > > > >> >
> > >>
> > > > > protocol.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > You
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > can see
> > > > > > > this just from the
> > > > > > > > > fact
> > > > > > > > > > that you
> > > > > > > > > > > need the
> > > > > > > > > > > > >> >
> extra
> > > verbiage
> > > > > > > > > > > > >> >
> > >>
> > > in
> > > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > protocol
> > > > > > > specification in
> > > > > > > > > this
> > > > > > > > > > KIP and in
> > > > > > > > > > > KIP-98
> > > > > > > > > > > > to
> > > > > > > > > > > > >> >
> > describe the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > dependence
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > between
> > > > > > the
> > > > > > > fields and the
> > > > > > > > > > attributes. It
> > > > > > > > > > > seems
> > > > > > > > > > > > >> like a
> > > > > > > > > > > > >> >
> > slippery
> > > > > > > > > > > > >> >
> > >>
> > > > slope
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > if
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > you start
> > > > > > > allowing
> > > > > > > > different
> > > > > > > > > > request types
> > > > > > > > > > > to
> > > > > > > > > > > > >> >
> > implement
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > > > protocol
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > specification differently.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > You can
> > > > > > > also argue that
> > > > > > > > the
> > > > > > > > > > messages
> > > > > > > > > > > already are
> > > > > > > > > > > > and
> > > > > > > > > > > > >> > are
> > likely
> > > > > > > > > > > > >> >
> > >>
> > > to
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > remain a
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > special
> > > > > > > case. For example,
> > > > > > > > > there
> > > > > > > > > > is
> > > > > > > > > > > currently no
> > > > > > > > > > > > >> >
> > generality
> > > in
> > > > > > > > > > > > >> >
> > >>
> > > how
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > compressed
> > > > > > > message sets
> > > > > > > > are
> > > > > > > > > > represented
> > > > > > > > > > > that would
> > > > > > > > > > > > >> be
> > > > > > > > > > > > >> >
> > applicable
> > > > > > > > > > > > >> >
> > >>
> > > for
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > other
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > request
> > > > > > > types. Some might
> > > > > > > > > see this
> > > > > > > > > > > divergence as
> > > > > > > > > > > > an
> > > > > > > > > > > > >> >
> > unfortunate
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > protocol
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > deficiency
> > > > > > > which should be
> > > > > > > > > fixed;
> > > > > > > > > > others
> > > > > > > > > > > might see
> > > > > > > > > > > > >> it
> > > > > > > > > > > > >> > as
> > sort of
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > inevitability of needing
> > > > > > > > to
> > > > > > > > > > optimize where
> > > > > > > > > > > it
> > > > > > > > > > > > counts
> > > > > > > > > > > > >> >
> most.
> > I'm
> > > > > > > > > > > > >> >
> > >>
> > > > > probably
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > somewhere
> > > > > > > in between, but
> > > > > > > > I
> > > > > > > > > think
> > > > > > > > > > we
> > > > > > > > > > > probably all
> > > > > > > > > > > > >> >
> share
> > the
> > > > > > > > > > > > >> >
> > >>
> > > > > intuition
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > that
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > the
> > > > > > > protocol should be
> > > > > > > > kept
> > > > > > > > > as
> > > > > > > > > > consistent as
> > > > > > > > > > > > >>
> > possible.
> > > > > > > > > > > > >> > With
> > that
> > > > > > > > > > > > >> >
> > >>
> > > in
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > mind,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > here are a
> > > > > > > few comments:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > 1. One
> > > > > > > thing I found a
> > > > > > > > > little odd
> > > > > > > > > > when
> > > > > > > > > > > reading the
> > > > > > > > > > > > >> >
> > current
> > > > > > > > > > > > >> >
> > >>
> > > > > proposal is
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > that
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > the
> > > > > > headers
> > > > > > > are both
> > > > > > > > > represented
> > > > > > > > > > as an
> > > > > > > > > > > array of
> > > > > > > > > > > > >> bytes
> > > > > > > > > > > > >> > and
> > as an
> > > > > > > > > > > > >> >
> > >>
> > > > array
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > key/value
> > > > > > > pairs. I'd
> > > > > > > > > probably
> > > > > > > > > > suggest
> > > > > > > > > > > something
> > > > > > > > > > > > like
> > > > > > > > > > > > >> >
> this:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > Headers =>
> > > > > > > [HeaderKey
> > > > > > > > > HeaderValue]
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > HeaderKey
> > > > > > > => String
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > HeaderValue => Bytes
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > An array
> > > > > > in
> > > > > > > the Kafka
> > > > > > > > > protocol is
> > > > > > > > > > > represented as
> a
> > > > > > > > > > > > >> >
> 4-byte
> > > integer
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > indicating
> > > > > > > the number of
> > > > > > > > > elements
> > > > > > > > > > in the
> > > > > > > > > > > array
> > > > > > > > > > > > >> >
> > followed by
> > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > serialization of the
> > > > > > > > > elements.
> > > > > > > > > > Unless I'm
> > > > > > > > > > > > >> >
> > > > > misunderstanding, what
> > > > > > > > > > > > >> >
> > >>
> > > you
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > have
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > instead is
> > > > > > > the total size
> > > > > > > > > of the
> > > > > > > > > > headers in
> > > > > > > > > > > bytes
> > > > > > > > > > > > >> >
> > followed by
> > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > elements.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > I'm not
> > > > > > > sure I see any
> > > > > > > > > reason for
> > > > > > > > > > this
> > > > > > > > > > > > >>
> > inconsistency.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > 2. In
> > > > > > > KIP-98, we've
> > > > > > > > > introduced
> > > > > > > > > > > variable-length
> > > > > > > > > > > > >>
> integer
> > > > > > > > > > > > >> >
> > fields.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Effectively,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > we've
> > > > > > > enriched (or
> > > > > > > > > "complicated"
> > > > > > > > > > as Jay
> > > > > > > > > > > might say
> > > > > > > > > > > > ;)
> > > > > > > > > > > > >> > the
> > protocol
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > specification to include
> > > > > > > > the
> > > > > > > > > > following
> > > > > > > > > > > types:
> > > > > > > > > > > > >>
> VarInt,
> > > > > > > > > > > > >> >
> > VarLong,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > UnsignedVarInt and
> > > > > > > > > > UnsignedVarLong.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > Along with
> > > > > > > these
> > > > > > > > > primitives, we
> > > > > > > > > > could
> > > > > > > > > > > introduce
> > > > > > > > > > > > the
> > > > > > > > > > > > >> >
> > following
> > > > > > > > > > > > >> >
> > >>
> > > > > types:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > VarSizeArray =>
> > > > > > > > > NumberOfItems
> > > > > > > > > > Item1 Item2
> > > > > > > > > > > .. ItemN
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > NumberOfItems =>
> > > > > > > > > UnsignedVarInt
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > VarSizeNullableArray =>
> > > > > > > > > > NumberOfItemsOrNull
> > > > > > > > > > > Item1
> > > > > > > > > > > > >> >
> Item2
> > ..
> > > ItemN
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > NumberOfItemsOrNull =>
> > > > > > > > > VarInt
> > > > > > > > > > (-1 means
> > > > > > > > > > > null)
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > And
> > > > > > > similarly for the
> > > > > > > > > `String`
> > > > > > > > > > and `Bytes`
> > > > > > > > > > > types.
> > > > > > > > > > > > >> >
> These
> > types
> > > > > > > > > > > > >> >
> > >>
> > > can
> > > > > save
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > a
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > considerable amount of
> > > > > > > > > space in
> > > > > > > > > > this
> > > > > > > > > > > proposal
> > > > > > > > > > > > >>
> because
> > > > > > > > > > > > >> > they
> > can
> > > > > > > > > > > > >> >
> > >>
> > > be
> > > > > used
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > for
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > both the
> > > > > > > number of headers
> > > > > > > > > > included in the
> > > > > > > > > > > message
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> > the
> > > > > > > > > > > > >> >
> > >>
> > > > > lengths of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > header
> > > > > > keys
> > > > > > > and values. We
> > > > > > > > > could
> > > > > > > > > > do this
> > > > > > > > > > > instead:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > Headers =>
> > > > > > > > > VarSizeArray[HeaderKey
> > > > > > > > > > > HeaderValue]
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > HeaderKey
> > > > > > > =>
> > > > > > > > VarSizeString
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > HeaderValue =>
> > > > > > > > > VarSizeBytes
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > Combining
> > > > > > > the savings from
> > > > > > > > > the
> > > > > > > > > > use of
> > > > > > > > > > > variable
> > > > > > > > > > > > >> length
> > > > > > > > > > > > >> >
> > fields, the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > benefit
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > of using
> > > > > > > the attributes to
> > > > > > > > > > represent null
> > > > > > > > > > > seems
> > > > > > > > > > > > >> pretty
> > > > > > > > > > > > >> >
> small.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > 3.
> > > > > > > Whichever way we go
> > > > > > > > > (whether
> > > > > > > > > > we use the
> > > > > > > > > > > > >>
> > attributes
> > > > > > > > > > > > >> > or
> > not), we
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > should at
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > least be
> > > > > > > consistent
> > > > > > > > between
> > > > > > > > > this
> > > > > > > > > > KIP and
> > > > > > > > > > > KIP-98.
> > > > > > > > > > > > It
> > > > > > > > > > > > >> >
> would
> > be
> > > very
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > strange
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > to have
> > > > > > two
> > > > > > > ways to
> > > > > > > > > represent
> > > > > > > > > > null values
> > > > > > > > > > > in the
> > > > > > > > > > > > >> same
> > > > > > > > > > > > >> >
> > schema.
> > > > > > > > > > > > >> >
> > >>
> > > > > Either
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > way is
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > OK with
> > > > > > me.
> > > > > > > I think some
> > > > > > > > > > message-level
> > > > > > > > > > > >
> > optimizations
> > > > > > > > > > > > >> > are
> > > > > > > > > > > > >> >
> > >>
> > > > > justifiable,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > but
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > the
> > > > > > savings
> > > > > > > here seem
> > > > > > > > > minimal (a
> > > > > > > > > > few bytes
> > > > > > > > > > > per
> > > > > > > > > > > > >> >
> > message), so
> > > > > > > > > > > > >> >
> > >>
> > > > > maybe it's
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > not
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > worth the
> > > > > > > cost of letting
> > > > > > > > > the
> > > > > > > > > > message
> > > > > > > > > > > diverge even
> > > > > > > > > > > > >> >
> > further from
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > rest of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > the
> > > > > > > protocol.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > -Jason
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > On Wed,
> > > > > > Feb
> > > > > > > 15, 2017 at
> > > > > > > > > 8:52 AM,
> > > > > > > > > > radai <
> > > > > > > > > > > > >> >
> > >>
> > > > > > > radai.rosenblatt@gmail.com>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > wrote:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > I've
> > > > > > > trimmed the inline
> > > > > > > > > > contents as this
> > > > > > > > > > > mail is
> > > > > > > > > > > > >> >
> > getting too
> > > > > > > > > > > > >> >
> > >>
> > > big
> > > > > for
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > apache
> > > > > > > mailing list
> > > > > > > > > software to
> > > > > > > > > > deliver
> > > > > > > > > > > :-(
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > 1. the
> > > > > > > important thing
> > > > > > > > for
> > > > > > > > > > >
> interoperability
> > is
> > > > > > > > > > > > for
> > > > > > > > > > > > >> >
> > different
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > "interested
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > parties"
> > > > > > > (plugins, infra
> > > > > > > > > > layers/wrappers,
> > > > > > > > > > > > >>
> > user-code)
> > > > > > > > > > > > >> > to
> be
> > > > > > > > > > > > >> >
> > >>
> > > > able
> > > > > to
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > stick
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > pieces
> > > > > > of
> > > > > > > metadata onto
> > > > > > > > > msgs
> > > > > > > > > > without
> > > > > > > > > > > getting in
> > > > > > > > > > > > >> each
> > > > > > > > > > > > >> >
> > other's
> > > > > > > > > > > > >> >
> > >>
> > > > way.
> > > > > a
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > common
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > key
> > > > > > > scheme (Strings, as
> > > > > > > > > of the
> > > > > > > > > > time of
> > > > > > > > > > > this
> > > > > > > > > > > > >> >
> > writing?) is
> > > > all
> > > > > > > > > > > > >> >
> > >>
> > > > thats
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > required
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > for
> > > > > > that.
> > > > > > > it is assumed
> > > > > > > > > that
> > > > > > > > > > the other end
> > > > > > > > > > > > >> >
> > interested
> > > in
> > > > > any
> > > > > > > > > > > > >> >
> > >>
> > > > such
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > piece of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > metadata
> > > > > > > knows the
> > > > > > > > > encoding,
> > > > > > > > > > and byte[]
> > > > > > > > > > > provides
> > > > > > > > > > > > >> for
> > > > > > > > > > > > >> > the
> > most
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > flexibility.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > i
> > > > > > believe
> > > > > > > this is the
> > > > > > > > same
> > > > > > > > > > logic behind
> > > > > > > > > > > core
> > > > > > > > > > > > kafka
> > > > > > > > > > > > >> >
> being
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > byte[]/byte[]
> > > > > > -
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > Strings
> > > > > > > are more
> > > > > > > > "usable"
> > > > > > > > > but
> > > > > > > > > > bytes are
> > > > > > > > > > > flexible
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> > so
> > were
> > > > > > > > > > > > >> >
> > >>
> > > > > chosen.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > Also -
> > > > > > > core kafka doesnt
> > > > > > > > > even
> > > > > > > > > > do that
> > > > > > > > > > > good of a
> > > > > > > > > > > > >> job
> > > > > > > > > > > > >> > on
> > > > > > > > > > > > >> >
> > >>
> > > > > usability of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > payload
> > > > > > > (example - i
> > > > > > > > have
> > > > > > > > > to
> > > > > > > > > > specify the
> > > > > > > > > > > nop
> > > > > > > > > > > > >> byte[]
> > > > > > > > > > > > >> >
> > "decoders"
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > explicitly
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > in
> > > > > > conf),
> > > > > > > and again
> > > > > > > > > sacrificies
> > > > > > > > > > usability
> > > > > > > > > > > for
> > > > > > > > > > > > the
> > > > > > > > > > > > >> > sake
> > of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > performance
> > > > > > (no
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > convenient single-record
> > > > > > > > > > processing as
> > > > > > > > > > > poll is a
> > > > > > > > > > > > >> >
> > batch, lots
> > > of
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > obscure
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > little
> > > > > > > config details
> > > > > > > > > exposing
> > > > > > > > > > internals
> > > > > > > > > > > of the
> > > > > > > > > > > > >> >
> > batching
> > > > > > > > > > > > >> >
> > >>
> > > > > mechanism,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > etc)
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > this is
> > > > > > > also why i
> > > > > > > > really
> > > > > > > > > > dislike the
> > > > > > > > > > > idea of a
> > > > > > > > > > > > >> >
> "type
> > > system"
> > > > > > > > > > > > >> >
> > >>
> > > for
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > header
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > values,
> > > > > > > it further
> > > > > > > > > degrades the
> > > > > > > > > > > usability, adds
> > > > > > > > > > > > >> >
> > complexity
> > > and
> > > > > > > > > > > > >> >
> > >>
> > > > will
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > eventually get in
> > > > > > > > > people's way,
> > > > > > > > > > also, it
> > > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > >> > the
> > 2nd/3rd
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > home-group
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > serialization mechanism
> > > > > > > > > in core
> > > > > > > > > > kafka
> > > > > > > > > > > (counting
> > > > > > > > > > > > 2
> > > > > > > > > > > > >> >
> > iterations
> > > > > > > > > > > > >> >
> > >>
> > > of
> > > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > "type
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > definition DSL")
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > 2. this
> > > > > > > is an
> > > > > > > > > implementation
> > > > > > > > > > detail, and
> > > > > > > > > > > not
> > > > > > > > > > > > even
> > > > > > > > > > > > >> a
> > > > > > > > > > > > >> > very
> > "user
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > facing" one?
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > to the
> > > > > > > best of my
> > > > > > > > > understanding
> > > > > > > > > > the vote
> > > > > > > > > > > process
> > > > > > > > > > > > >> is
> > > > > > > > > > > > >> > on
> > proposed
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > API/behaviour. also -
> > > > > > > > > since
> > > > > > > > > > we're willing
> > > > > > > > > > > to go
> > > > > > > > > > > > >> with
> > > > > > > > > > > > >> >
> > strings
> > > > > > > > > > > > >> >
> > >>
> > > > just
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > serialize
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > a
> > > > > > 0-sized
> > > > > > > header blob
> > > > > > > > and
> > > > > > > > > IIUC
> > > > > > > > > > you dont
> > > > > > > > > > > need any
> > > > > > > > > > > > >> >
> > optionals
> > > > > > > > > > > > >> >
> > >>
> > > > > anymore.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > 3. yes,
> > > > > > > we can :-)
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > On Tue,
> > > > > > > Feb 14, 2017 at
> > > > > > > > > 11:56
> > > > > > > > > > PM, Michael
> > > > > > > > > > > > Pearce <
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > wrote:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > Hi
> > > > > > Jay,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > 1)
> > > > > > > There was some
> > > > > > > > > initial
> > > > > > > > > > debate on the
> > > > > > > > > > > value
> > > > > > > > > > > > >> >
> part,
> > as
> > > youll
> > > > > > > > > > > > >> >
> > >>
> > > > note
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > String,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > String
> > > > > > > headers were
> > > > > > > > > > discounted early
> > > > > > > > > > > on. The
> > > > > > > > > > > > >> >
> > reason for
> > > > this
> > > > > > > > > > > > >> >
> > >>
> > > is
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > flexibility
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > and
> > > > > > > keeping in line
> > > > > > > > > with the
> > > > > > > > > > > flexibility of
> > > > > > > > > > > > key,
> > > > > > > > > > > > >> >
> value
> > of the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > message
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > object
> > > > > > > itself. I don’t
> > > > > > > > > think
> > > > > > > > > > it takes
> > > > > > > > > > > away
> > > > > > > > > > > > from
> > > > > > > > > > > > >> an
> > > > > > > > > > > > >> >
> > ecosystem
> > > > > > > > > > > > >> >
> > >>
> > > as
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > each
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > plugin
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > will
> > > > > > > care for their
> > > > > > > > own
> > > > > > > > > key,
> > > > > > > > > > this way
> > > > > > > > > > > ints,
> > > > > > > > > > > > >> >
> > booleans ,
> > > > > exotic
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > custom
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > binary
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > can
> > > > > > all
> > > > > > > be catered
> > > > > > > > for=.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > a. If
> > > > > > > you really
> > > > > > > > wanted
> > > > > > > > > to
> > > > > > > > > > push for a
> > > > > > > > > > > typed
> > > > > > > > > > > > >> value
> > > > > > > > > > > > >> >
> > interface,
> > > > > > > > > > > > >> >
> > >> I
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > wouldn’t
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > want
> > > > > > > just String
> > > > > > > > values
> > > > > > > > > > supported, but
> > > > > > > > > > > the the
> > > > > > > > > > > > >> >
> > primatives
> > > > > > > > > > > > >> >
> > >>
> > > > plus
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > string and
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > also
> > > > > > > still keeping the
> > > > > > > > > > ability to have a
> > > > > > > > > > > > binary
> > > > > > > > > > > > >> > for
> > custom
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > binaries that
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > some
> > > > > > > organisations may
> > > > > > > > > have.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > i. I
> > > > > > > have written this
> > > > > > > > > slight
> > > > > > > > > > > alternative
> > > > > > > > > > > > here,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > https://cwiki.apache.org/
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > > > >> >
> > > > > 82+-+Add+Record+Headers+-+
> > > > > > Typed
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > ii.
> > > > > > > Essentially the
> > > > > > > > > value
> > > > > > > > > > bytes, has a
> > > > > > > > > > > leading
> > > > > > > > > > > > >> > byte
> > > overhead.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > 1.
> > > > > > > This tells you
> > > > > > > > what
> > > > > > > > > type
> > > > > > > > > > the value
> > > > > > > > > > > is,
> > > > > > > > > > > > >> before
> > > > > > > > > > > > >> >
> > reading
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > rest
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > of the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > bytes,
> > > > > > > allowing
> > > > > > > > > > >
> > > > > serialisation/deserialization
> > > > > > > > > > > > to
> > > > > > > > > > > > >> > and
> > from the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > primitives,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > string
> > > > > > > and byte[].
> > > > > > > > This
> > > > > > > > > is
> > > > > > > > > > akin to some
> > > > > > > > > > > other
> > > > > > > > > > > > >> >
> > messaging
> > > > > > > > > > > > >> >
> > >>
> > > > > systems.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > 2) We
> > > > > > > are making it
> > > > > > > > > optional,
> > > > > > > > > > so that
> > > > > > > > > > > for
> > > > > > > > > > > > those
> > > > > > > > > > > > >> > not
> > wanting
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > headers have
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > 0
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > bytes
> > > > > > > overhead (think
> > > > > > > > > of it
> > > > > > > > > > as a feature
> > > > > > > > > > > > flag),
> > > > > > > > > > > > >> I
> > > > > > > > > > > > >> >
> don’t
> > > > > > > > > > > > >> >
> > >>
> > > > > think this
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > is
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > complex, especially if
> > > > > > > > > > comparing to
> > > > > > > > > > > changes
> > > > > > > > > > > > >> >
> > proposed in
> > > > > > > > > > > > >> >
> > >>
> > > > > other kips
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > like
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > kip-98.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > a. If
> > > > > > > you really
> > > > > > > > really
> > > > > > > > > don’t
> > > > > > > > > > like
> > > > > > > > > > > this, we
> > > > > > > > > > > > can
> > > > > > > > > > > > >> > drop
> > it, but
> > > > > > > > > > > > >> >
> > >>
> > > it
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > would
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > mean
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > buying
> > > > > > > into 4 bytes
> > > > > > > > > extra
> > > > > > > > > > overhead for
> > > > > > > > > > > users
> > > > > > > > > > > > who
> > > > > > > > > > > > >> > do
> > not want
> > > > > > > > > > > > >> >
> > >>
> > > to
> > > > > use
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > headers.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > 3) In
> > > > > > > the summary yes,
> > > > > > > > > it is
> > > > > > > > > > at a higher
> > > > > > > > > > > > level,
> > > > > > > > > > > > >> > but
> I
> > think
> > > > > > > > > > > > >> >
> > >>
> > > > this
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > is well
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > documented in the
> > > > > > > > > proposed
> > > > > > > > > > changes
> > > > > > > > > > > section.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > a.
> > > > > > > Added getHeaders
> > > > > > > > > method to
> > > > > > > > > > > >
> > Producer/Consumer
> > > > > > > > > > > > >> >
> > record (that
> > > > > > > > > > > > >> >
> > >>
> > > is
> > > > > it)
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > b.
> > > > > > > We’ve also detailed
> > > > > > > > > the
> > > > > > > > > > new Headers
> > > > > > > > > > > class
> > > > > > > > > > > > >> that
> > > > > > > > > > > > >> > this
> > method
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > returns
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > that
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > encapsulates the
> > > > > > > > headers
> > > > > > > > > > protocol and
> > > > > > > > > > > logic.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > Best,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > Mike
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > ==Original questions
> > > > > > > > > from the
> > > > > > > > > > vote
> > > > > > > > > > > thread from
> > > > > > > > > > > > >> >
> Jay.==
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > Couple
> > > > > > > of things I
> > > > > > > > > think we
> > > > > > > > > > still need
> > > > > > > > > > > to work
> > > > > > > > > > > > >> out:
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > 1.
> > > > > > I
> > > > > > > think we agree
> > > > > > > > > about
> > > > > > > > > > the key,
> > > > > > > > > > > but I
> > > > > > > > > > > > >> think
> > > > > > > > > > > > >> > we
> > haven't
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > talked about
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > the
> > > > > > > value yet. I
> > > > > > > > > think if
> > > > > > > > > > our goal
> > > > > > > > > > > is an
> > > > > > > > > > > > open
> > > > > > > > > > > > >> >
> > ecosystem
> > > > > > > > > > > > >> >
> > >>
> > > of
> > > > > these
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > header
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > spread across many
> > > > > > > > > plugins
> > > > > > > > > > from many
> > > > > > > > > > > > systems
> > > > > > > > > > > > >> we
> > > > > > > > > > > > >> >
> should
> > > > > > > > > > > > >> >
> > >>
> > > > > consider
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > making
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > this
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > a
> > > > > > > string as well so
> > > > > > > > > it can
> > > > > > > > > > be
> > > > > > > > > > > printed, set
> > > > > > > > > > > > >> via
> > > > > > > > > > > > >> > a
> UI,
> > set
> > > > > > > > > > > > >> >
> > >>
> > > in
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > config,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > etc.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > Basically
> > > > > > > > encouraging
> > > > > > > > > > pluggable
> > > > > > > > > > > >
> > serialization
> > > > > > > > > > > > >> >
> > formats
> > > > > > > > > > > > >> >
> > >>
> > > > here
> > > > > will
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > lead
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > to
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > a
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > bit
> > > > > > > of a tower of
> > > > > > > > > babel.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > 2.
> > > > > > > This proposal
> > > > > > > > > still
> > > > > > > > > > includes a
> > > > > > > > > > > pretty
> > > > > > > > > > > > big
> > > > > > > > > > > > >> >
> > change to
> > > our
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > serialization
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > and
> > > > > > > protocol
> > > > > > > > > definition
> > > > > > > > > > layer.
> > > > > > > > > > > Essentially
> > > > > > > > > > > > >> it is
> > > > > > > > > > > > >> >
> > >>
> > > > > introducing an
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > optional
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > type, where the
> > > > > > > > > format is
> > > > > > > > > > data
> > > > > > > > > > > dependent. I
> > > > > > > > > > > > >> >
> think
> > this
> > > is
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > actually a
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > big
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > change though it
> > > > > > > > > doesn't
> > > > > > > > > > seem like
> > > > > > > > > > > it. It
> > > > > > > > > > > > >> means
> > > > > > > > > > > > >> > you
> > can no
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > longer
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > specify
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > this
> > > > > > > type with our
> > > > > > > > > type
> > > > > > > > > > definition
> > > > > > > > > > > DSL, and
> > > > > > > > > > > > >> >
> > likewise it
> > > > > > > > > > > > >> >
> > >>
> > > > > requires
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > custom
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > handling in client
> > > > > > > > > libs.
> > > > > > > > > > This isn't
> > > > > > > > > > > a huge
> > > > > > > > > > > > >> >
> > thing, since
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Record
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > definition is
> > > > > > > > custom
> > > > > > > > > > anyway, but I
> > > > > > > > > > > think
> > > > > > > > > > > > this
> > > > > > > > > > > > >> > kind
> > of
> > > > > > > > > > > > >> >
> > >>
> > > > > protocol
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > inconsistency is
> > > > > > > > very
> > > > > > > > > > non-desirable
> > > > > > > > > > > and
> > > > > > > > > > > > ties
> > > > > > > > > > > > >> > you
> to
> > > > > > > > > > > > >> >
> > >>
> > > > > hand-coding
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > things.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > I
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > think the type
> > > > > > > > should
> > > > > > > > > > instead by [Key
> > > > > > > > > > > > Value]
> > > > > > > > > > > > >> in
> > > > > > > > > > > > >> > our
> > BNF,
> > > > > > > > > > > > >> >
> > >>
> > > > where
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > key and
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > value are both
> > > > > > > > short
> > > > > > > > > > strings as used
> > > > > > > > > > > > >>
> > elsewhere.
> > > > > > > > > > > > >> > This
> > > > > > > > > > > > >> >
> > >>
> > > > > brings it
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > in line
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > with
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > the
> > > > > > > rest of the
> > > > > > > > > protocol.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > 3.
> > > > > > > Could we get
> > > > > > > > more
> > > > > > > > > > specific about
> > > > > > > > > > > the
> > > > > > > > > > > > exact
> > > > > > > > > > > > >> > Java
> > API
> > > > > > > > > > > > >> >
> > >>
> > > > > change to
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > ProducerRecord,
> > > > > > > > > > ConsumerRecord,
> > > > > > > > > > > Record,
> > > > > > > > > > > > etc?
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > > -Jay
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > The
> > > > > > information
> > > > > > > contained in
> > > > > > > > > this
> > > > > > > > > > email is
> > > > > > > > > > > strictly
> > > > > > > > > > > > >> >
> > confidential
> > > > and
> > > > > > > > > > > > >> >
> > >>
> > > for
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > the use of the
> > > > > > > addressee only,
> > > > > > > > > unless
> > > > > > > > > > otherwise
> > > > > > > > > > > > >>
> > indicated.
> > > > > > > > > > > > >> > If
> you
> > > > > > > > > > > > >> >
> > >>
> > > are
> > > > > not
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > the intended
> > > > > > > recipient, please
> > > > > > > > > do not
> > > > > > > > > > read,
> > > > > > > > > > > copy, use
> > > > > > > > > > > > or
> > > > > > > > > > > > >> >
> > disclose to
> > > > > > > > > > > > >> >
> > >>
> > > > > others
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > this message
> > > > > > or
> > > > > > > any
> > > > > > > > attachment.
> > > > > > > > > > Please also
> > > > > > > > > > > notify the
> > > > > > > > > > > > >> >
> > sender by
> > > > > > > > > > > > >> >
> > >>
> > > > > replying
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > to this email
> > > > > > > or by telephone
> > > > > > > > > > (+44(020 7896
> > > > > > > > > > > 0011) and
> > > > > > > > > > > > >> then
> > > > > > > > > > > > >> >
> delete
> > > > > > > > > > > > >> >
> > >>
> > > the
> > > > > email
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > and any copies
> > > > > > > of it.
> > > > > > > > Opinions,
> > > > > > > > > > conclusion
> > > > > > > > > > > (etc) that
> > > > > > > > > > > > do
> > > > > > > > > > > > >> > not
> > relate
> > > > > > > > > > > > >> >
> > >>
> > > to
> > > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > official
> > > > > > > business of this
> > > > > > > > > company
> > > > > > > > > > shall be
> > > > > > > > > > > understood
> > > > > > > > > > > > as
> > > > > > > > > > > > >> >
> > neither
> > > > > > > > > > > > >> >
> > >>
> > > > > given nor
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > endorsed by
> > > > > > it.
> > > > > > > IG is a
> > > > > > > > trading
> > > > > > > > > name
> > > > > > > > > > of IG
> > > > > > > > > > > Markets
> > > > > > > > > > > > >>
> Limited
> > > > > > > > > > > > >> > (a
> > company
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > registered in
> > > > > > > England and
> > > > > > > > Wales,
> > > > > > > > > > company number
> > > > > > > > > > > > >>
> > 04008957)
> > > > > > > > > > > > >> > and
> IG
> > > > > > > > > > > > >> >
> > >>
> > > > Index
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Limited (a
> > > > > > > company registered
> > > > > > > > in
> > > > > > > > > > England and
> > > > > > > > > > > Wales,
> > > > > > > > > > > > >> >
> > company
> > > number
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > 01190902).
> > > > > > > Registered address
> > > > > > > > at
> > > > > > > > > > Cannon Bridge
> > > > > > > > > > > House,
> > > > > > > > > > > > 25
> > > > > > > > > > > > >> >
> > Dowgate
> > > > > > > > > > > > >> >
> > >>
> > > > Hill,
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > London EC4R
> > > > > > > 2YA. Both IG
> > > > > > > > Markets
> > > > > > > > > > Limited
> > > > > > > > > > > (register
> > > > > > > > > > > > >> number
> > > > > > > > > > > > >> >
> > 195355)
> > > > > > > > > > > > >> >
> > >>
> > > and
> > > > > IG
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Index Limited
> > > > > > > (register number
> > > > > > > > > > 114059) are
> > > > > > > > > > > authorised
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> >
> > regulated
> > > > > > > > > > > > >> >
> > >>
> > > by
> > > > > the
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > Financial
> > > > > > > Conduct Authority.
> > > > > > > > > > > > >> >
> > >>
> > > >>
> > > > >
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >>
> > > --
> > > > > > > > > > > > >> >
> > >>
> > > > > Nacho - Ignacio
> > > > > > > Solis -
> > > > > > > > > isolis@igso.net
> > > > > > > > > > > > >> >
> > >>
> > > > > > > > > > > > >> >
> > >
> > > > > > > > > > > > >> >
> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > The
> > > > information
> > > > > contained in
> > > > > > > this email is
> > > > > > > > > strictly
> > > > > > > > > > > confidential
> and
> > > > > > > > > > > > for
> > > > > > > > > > > > >> > the
> > use of
> > > the
> > > > > addressee only,
> > > > > > > unless
> > > > > > > > otherwise
> > > > > > > > > > indicated. If
> > > > > > > > > > > you are
> > > > > > > > > > > > >> not
> > > > > > > > > > > > >> > the
> > intended
> > > > > recipient, please
> > > > > > > do not read,
> > > > > > > > > copy, use
> > > > > > > > > > or
> > > > > > > > > > > disclose to
> > > > > > > > > > > > >> others
> > > > > > > > > > > > >> > this
> > message
> > > > or
> > > > > any
> > > > > > attachment.
> > > > > > > Please also
> > > > > > > > > notify
> > > > > > > > > > the sender by
> > > > > > > > > > > > >>
> replying
> > > > > > > > > > > > >> > to
> > this
> > > email
> > > > > or by telephone
> > > > > > > (+44(020 7896
> > > > > > > > > 0011) and
> > > > > > > > > > then
> > > > > > > > > > > delete the
> > > > > > > > > > > > >> email
> > > > > > > > > > > > >> > and
> > any
> > > copies
> > > > > of it.
> > > > > > Opinions,
> > > > > > > conclusion
> > > > > > > > > (etc) that
> > > > > > > > > > do not
> > > > > > > > > > > relate to
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >
> > official
> > > > > business of this
> > > > > > > company shall be
> > > > > > > > > understood
> > > > > > > > > > as
> > > > > > > > > > > neither given
> > > > > > > > > > > > >> nor
> > > > > > > > > > > > >> >
> > endorsed by
> > > > it.
> > > > > IG is a
> > > > > > trading
> > > > > > > name of IG
> > > > > > > > > Markets
> > > > > > > > > > Limited (a
> > > > > > > > > > > company
> > > > > > > > > > > > >> >
> > registered
> > > in
> > > > > England and
> > > > > > > Wales, company
> > > > > > > > number
> > > > > > > > > > 04008957) and
> > > > > > > > > > > IG Index
> > > > > > > > > > > > >> >
> > Limited (a
> > > > > company registered
> > > > > > > in England and
> > > > > > > > > Wales,
> > > > > > > > > > company
> > > > > > > > > > > number
> > > > > > > > > > > > >> >
> > 01190902).
> > > > > Registered address
> > > > > > > at Cannon Bridge
> > > > > > > > > House,
> > > > > > > > > > 25
> > > > > > > > > > > Dowgate Hill,
> > > > > > > > > > > > >> >
> > London EC4R
> > > > > 2YA. Both IG
> > > > > > > Markets Limited
> > > > > > > > > (register
> > > > > > > > > > number
> > > > > > > > > > > 195355) and
> > > > > > > > > > > > IG
> > > > > > > > > > > > >> >
> Index
> > > Limited
> > > > > (register number
> > > > > > > 114059) are
> > > > > > > > > authorised
> > > > > > > > > > and
> > > > > > > > > > > regulated by
> > > > > > > > > > > > >> the
> > > > > > > > > > > > >> >
> > Financial
> > > > > Conduct Authority.
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> The
> > > information
> > > > > contained in
> > > > > > this
> > > > > > > email is
> > > > > > > > > strictly
> > > > > > > > > > confidential
> > > > > > > > > > > and for
> > > > > > > > > > > > >> the
> use
> > of the
> > > > > addressee only,
> > > > > > > unless otherwise
> > > > > > > > > > indicated. If you
> > > > > > > > > > > are
> > > > > > > > > > > > not
> > > > > > > > > > > > >> the
> > intended
> > > > > recipient, please
> > > > > > do
> > > > > > > not read,
> > > > > > > > copy,
> > > > > > > > > use
> > > > > > > > > > or disclose
> > > > > > > > > > > to
> > > > > > > > > > > > others
> > > > > > > > > > > > >> this
> > message
> > > or
> > > > > any attachment.
> > > > > > > Please also
> > > > > > > > > notify the
> > > > > > > > > > sender by
> > > > > > > > > > > > replying
> > > > > > > > > > > > >> to
> this
> > email
> > > or
> > > > > by telephone
> > > > > > > (+44(020 7896
> > > > > > > > 0011)
> > > > > > > > > and
> > > > > > > > > > then delete
> > > > > > > > > > > the
> > > > > > > > > > > > email
> > > > > > > > > > > > >> and
> any
> > copies
> > > > of
> > > > > it. Opinions,
> > > > > > > conclusion (etc)
> > > > > > > > > that
> > > > > > > > > > do not
> > > > > > > > > > > relate to
> > > > > > > > > > > > the
> > > > > > > > > > > > >>
> official
> > > > business
> > > > > of this
> > > > > > company
> > > > > > > shall be
> > > > > > > > > understood
> > > > > > > > > > as neither
> > > > > > > > > > > given
> > > > > > > > > > > > nor
> > > > > > > > > > > > >>
> > endorsed by
> > > it.
> > > > > IG is a trading
> > > > > > > name of IG
> > > > > > > > Markets
> > > > > > > > > > Limited (a
> > > > > > > > > > > company
> > > > > > > > > > > > >>
> > registered in
> > > > > England and Wales,
> > > > > > > company number
> > > > > > > > > > 04008957) and IG
> > > > > > > > > > > Index
> > > > > > > > > > > > >>
> Limited
> > (a
> > > > > company registered in
> > > > > > > England and
> > > > > > > > > Wales,
> > > > > > > > > > company number
> > > > > > > > > > > > >>
> > 01190902).
> > > > > Registered address at
> > > > > > > Cannon Bridge
> > > > > > > > > House,
> > > > > > > > > > 25 Dowgate
> > > > > > > > > > > Hill,
> > > > > > > > > > > > >> London
> > EC4R
> > > 2YA.
> > > > > Both IG Markets
> > > > > > > Limited
> > > > > > > > (register
> > > > > > > > > > number 195355)
> > > > > > > > > > > and IG
> > > > > > > > > > > > >> Index
> > Limited
> > > > > (register number
> > > > > > > 114059) are
> > > > > > > > > authorised
> > > > > > > > > > and
> > > > > > > > > > > regulated by
> > > > > > > > > > > > the
> > > > > > > > > > > > >>
> > Financial
> > > > Conduct
> > > > > Authority.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > The
> > information
> > > > > contained in this
> > > > > > > email is strictly
> > > > > > > > > > confidential and
> > > > > > > > > > > for the use of
> > the
> > > > > addressee only, unless
> > > > > > > otherwise
> > > > > > > > > indicated.
> > > > > > > > > > If you are
> > > > > > > > > > > not the
> intended
> > > > > recipient, please do not
> > > > > > > read, copy, use
> > > > > > > > > or
> > > > > > > > > > disclose to
> > > > > > > > > > > others this
> > message or
> > > > any
> > > > > attachment.
> > > > > > > Please also notify
> > > > > > > > > the
> > > > > > > > > > sender by
> > > > > > > > > > > replying to
> this
> > email
> > > or
> > > > > by telephone
> > > > > > > (+44(020 7896
> > > > > > > > 0011)
> > > > > > > > > and
> > > > > > > > > > then delete
> > > > > > > > > > > the email and
> any
> > > copies
> > > > > of it. Opinions,
> > > > > > > conclusion
> > > > > > > > (etc)
> > > > > > > > > that
> > > > > > > > > > do not
> > > > > > > > > > > relate to the
> > official
> > > > > business of this
> > > > > > > company shall be
> > > > > > > > > > understood as
> > > > > > > > > > > neither given
> nor
> > > > endorsed
> > > > > by it. IG is a
> > > > > > > trading name of
> > > > > > > > > IG
> > > > > > > > > > Markets
> > > > > > > > > > > Limited (a
> > company
> > > > > registered in England
> > > > > > > and Wales,
> > > > > > > > company
> > > > > > > > > > number
> > > > > > > > > > > 04008957) and
> IG
> > Index
> > > > > Limited (a company
> > > > > > > registered in
> > > > > > > > > England
> > > > > > > > > > and Wales,
> > > > > > > > > > > company number
> > > 01190902).
> > > > > Registered
> > > > > > > address at Cannon
> > > > > > > > > Bridge
> > > > > > > > > > House, 25
> > > > > > > > > > > Dowgate Hill,
> > London
> > > EC4R
> > > > > 2YA. Both IG
> > > > > > > Markets Limited
> > > > > > > > > (register
> > > > > > > > > > number
> > > > > > > > > > > 195355) and IG
> > Index
> > > > > Limited (register
> > > > > > > number 114059) are
> > > > > > > > > > authorised and
> > > > > > > > > > > regulated by
> the
> > > > Financial
> > > > > Conduct
> > > > > > > Authority.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The information
> > contained in
> > > > > this email is
> > > > > > > strictly
> > > > > > > > confidential
> > > > > > > > > and
> > > > > > > > > > for the use of the
> > addressee
> > > only,
> > > > > unless otherwise
> > > > > > > indicated. If
> > > > > > > > > you are
> > > > > > > > > > not the intended
> recipient,
> > > please
> > > > > do not read,
> > > > > > > copy, use or
> > > > > > > > > disclose to
> > > > > > > > > > others this message or
> any
> > > > > attachment. Please also
> > > > > > > notify the
> > > > > > > > sender
> > > > > > > > > by
> > > > > > > > > > replying to this email or
> > by
> > > > > telephone (+44(020
> > > > > > 7896
> > > > > > > 0011) and then
> > > > > > > > > delete
> > > > > > > > > > the email and any copies
> > of it.
> > > > > Opinions,
> > > > > > conclusion
> > > > > > > (etc) that do
> > > > > > > > > not
> > > > > > > > > > relate to the official
> > business
> > > of
> > > > > this company
> > > > > > > shall be understood
> > > > > > > > > as
> > > > > > > > > > neither given nor
> endorsed
> > by it.
> > > > IG
> > > > > is a trading
> > > > > > > name of IG
> > > > > > > > Markets
> > > > > > > > > > Limited (a company
> > registered in
> > > > > England and Wales,
> > > > > > > company number
> > > > > > > > > > 04008957) and IG Index
> > Limited (a
> > > > > company
> > > > > > registered
> > > > > > > in England and
> > > > > > > > > Wales,
> > > > > > > > > > company number 01190902).
> > > > Registered
> > > > > address at
> > > > > > > Cannon Bridge
> > > > > > > > House,
> > > > > > > > > 25
> > > > > > > > > > Dowgate Hill, London EC4R
> > 2YA.
> > > Both
> > > > > IG Markets
> > > > > > > Limited (register
> > > > > > > > > number
> > > > > > > > > > 195355) and IG Index
> > Limited
> > > > > (register number
> > > > > > > 114059) are
> > > > > > > > authorised
> > > > > > > > > and
> > > > > > > > > > regulated by the
> Financial
> > > Conduct
> > > > > Authority.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information contained in
> > this email
> > > > is
> > > > > strictly
> > > > > > > confidential and for
> > > > > > > > > the use of the addressee only,
> > unless
> > > > > otherwise
> > > > > > indicated.
> > > > > > > If you are not
> > > > > > > > > the intended recipient, please
> > do not
> > > > > read, copy, use or
> > > > > > > disclose to
> > > > > > > > others
> > > > > > > > > this message or any attachment.
> > Please
> > > > > also notify the
> > > > > > > sender by replying
> > > > > > > > > to this email or by telephone
> > (+44(020
> > > > > 7896 0011) and
> > > > > > then
> > > > > > > delete the
> > > > > > > > email
> > > > > > > > > and any copies of it. Opinions,
> > > > conclusion
> > > > > (etc) that do
> > > > > > > not relate to
> > > > > > > > the
> > > > > > > > > official business of this
> > company shall
> > > > be
> > > > > understood as
> > > > > > > neither given
> > > > > > > > nor
> > > > > > > > > endorsed by it. IG is a trading
> > name of
> > > > IG
> > > > > Markets
> > > > > > Limited
> > > > > > > (a company
> > > > > > > > > registered in England and
> Wales,
> > > company
> > > > > number 04008957)
> > > > > > > and IG Index
> > > > > > > > > Limited (a company registered
> in
> > > England
> > > > > and Wales,
> > > > > > > company number
> > > > > > > > > 01190902). Registered address
> at
> > Cannon
> > > > > Bridge House, 25
> > > > > > > Dowgate Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> Markets
> > > Limited
> > > > > (register number
> > > > > > > 195355) and IG
> > > > > > > > > Index Limited (register number
> > 114059)
> > > > are
> > > > > authorised and
> > > > > > > regulated by
> > > > > > > > the
> > > > > > > > > Financial Conduct Authority.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > strictly
> > > > > confidential and
> > > > > > > for the use of the addressee only, unless
> > otherwise
> > > > > indicated. If you are
> > > > > > > not the intended recipient, please do not read,
> > copy,
> > > use
> > > > > or disclose to
> > > > > > > others this message or any attachment. Please
> > also
> > > notify
> > > > > the sender by
> > > > > > > replying to this email or by telephone (+44(020
> > 7896
> > > > 0011)
> > > > > and then
> > > > > > delete
> > > > > > > the email and any copies of it. Opinions,
> > conclusion
> > > > (etc)
> > > > > that do not
> > > > > > > relate to the official business of this company
> > shall
> > > be
> > > > > understood as
> > > > > > > neither given nor endorsed by it. IG is a
> > trading name
> > > of
> > > > > IG Markets
> > > > > > > Limited (a company registered in England and
> > Wales,
> > > > > company number
> > > > > > > 04008957) and IG Index Limited (a company
> > registered in
> > > > > England and
> > > > > > Wales,
> > > > > > > company number 01190902). Registered address at
> > Cannon
> > > > > Bridge House, 25
> > > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > Limited
> > > > > (register number
> > > > > > > 195355) and IG Index Limited (register number
> > 114059)
> > > are
> > > > > authorised and
> > > > > > > regulated by the Financial Conduct Authority.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> confidential
> > and
> > > for
> > > > > the use of the addressee only, unless otherwise indicated. If
> > you are
> > > not
> > > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify the sender
> by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to
> > > > the
> > > > > official business of this company shall be understood as
> neither
> > given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > > > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > > > Limited (a company registered in England and Wales, company
> > number
> > > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and
> > > IG
> > > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
So currently KIP has:
append
filter (get by key, but not inferring perf as noted expectation on O(1) perf with a get method)
I agree that supporting some form of
remove
replace
is standard in this kind of API and for the benefit of making a usable API, we should explore adding these.
If we’re avoid the put, get styled interface as there seems contention with this naming, we happy with naming these as I mentioned?
On 23/02/2017, 12:16, "radai" <ra...@gmail.com> wrote:
append-only would mean that if (for whatever reason) i want to replace a
header or strip it out i'd need to copy the whole record.
On Wed, Feb 22, 2017 at 5:09 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Im happy to compromise to keep it mutable but move to an append style api.
> (as in guava interables concat)
>
> class Headers {
> Headers append(Iterable<Header> headers);
> }
>
>
> I don’t think we’d want prepend, this would give the idea of guaranteed
> ordering, when in actual fact we don’t provide that guarantee (.e.g one
> client can put headerA, then headerB, but another could put headerB then
> headerA, this shouldn’t cause issues), Also what if we changed to a hashmap
> for the internal implementation, its just a bucket of entries no ordering.
> I think we just need to provide an api to add/append headers.
>
> This ok? If so ill update KIP to record this.
>
> Cheers
> Mike
>
> On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> The point about usability is fair. It's also reasonable to expect that
> common use cases such as appending headers should be done efficiently.
>
> Perhaps we could compromise with something like this?
>
> class Headers {
> Headers append(Iterable<Header> headers);
> Headers prepend(Iterable<Header> headers);
> }
>
> That retains ease of use while still giving ourselves some flexibility
> in
> the implementation.
>
> -Jason
>
>
> On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > I wasn’t referring to the headers needing to be copied, im meaning
> the
> > fact we’d be forcing a new producer record to be created, with all
> the
> > contents copied.
> >
> > i.e what will happen is utility method will be created or end up
> being
> > used, which does this, and returns the new ProducerRecord instance.
> >
> > ProducerRecord addHeader(ProducerRecord record, Header header){
> > Return New ProducerRecord(record.key, record.value,
> record.timestamp…..,
> > record.headers.concat(header))
> > }
> >
> > To me this seems ugly, but will be inevitable if we don’t make adding
> > headers to existing records a simple clean method call.
> >
> >
> >
> > On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Lazy init can achieve/avoid that.
> >
> > Re the concat, why don’t we implement that inside the Headers
> rather
> > than causing everyone to implement this as adding headers in
> interceptors
> > will be a dominant use case. We want a user friendly API. Having as
> a user
> > having to code this instead of having the headers handle this for me
> seems
> > redundant.
> >
> > On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >
> > I thought the argument was against creating the extra objects
> > unnecessarily
> > (i.e. if they were not accessed). And note that making the
> Headers
> > immutable doesn't necessarily mean that they need to be
> copied:
> > you can do
> > a trick like Guava's Iterables.concat to add additional
> headers
> > without
> > changing the underlying collections.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > If the argument for not having a map holding the key, value
> > pairs is due
> > > to garbage creation of HashMap entry's, forcing the
> creation of
> > a whole new
> > > producer record to simply add a head, surely is creating
> a-lot
> > more?
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > The current producer interceptor API is this:
> > >
> > > ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
> > >
> > > So adding a header means creating a new ProducerRecord
> with a
> > new header
> > > added to the current headers and returning it. Would that
> not
> > work?
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > So how would you have this work if not mutable where
> > interceptors would
> > > > add headers?
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > I think the point on the mutability of Headers is worth
> > discussing a
> > > little
> > > > more. As far as I can tell, once the ProducerRecord (or
> > ConsumerRecord)
> > > is
> > > > constructed, there should be no need to further change
> the
> > headers. Is
> > > that
> > > > correct? If so, then why not enforce that that is the
> case
> > through the
> > > API?
> > > > One problem with mutability it that it constrains the
> > implementation of
> > > > Headers. For example, if we were backing with a byte
> slice,
> > would we
> > > recopy
> > > > the bytes if a header is added or would we maintain a
> satellite
> > > collection
> > > > of added records. Seems not great either way. If we
> really
> > think
> > > mutability
> > > > is needed, perhaps we could add a method to headers to
> convert
> > it to a
> > > > mutable type (e.g. a Headers.toMap)?
> > > >
> > > > I'm also with Ismael about exposing Headers.get(). I
> thought
> > it might
> > > make
> > > > sense to have a method like this instead:
> > > >
> > > > Iterable<Header> findMatching(Pattern pattern);
> > > >
> > > > This makes the (potential) need to scan the headers
> clear in
> > the API. I'd
> > > > also be fine exposing no getter at all. In general, Ï
> think
> > it's good to
> > > > start with a minimalistic API and work from there since
> it's
> > always
> > > easier
> > > > to add methods than remove them.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > >
> > > > > Hi Ismael
> > > > >
> > > > > On point 1,
> > > > >
> > > > > Sure makes sense will update shortly.
> > > > >
> > > > > On point 2,
> > > > >
> > > > > Setter/getter typical to properties/headers api’s
> > traditionally are map
> > > > > styled interfaces and what I believe is most expected
> styled
> > thus the
> > > > Key,
> > > > > Value setter.
> > > > > Also it would mean rather than an interface, we would
> be
> > making our
> > > > > internal header impl object we have for the array,
> exposed.
> > E.g. if we
> > > > had
> > > > > a Map really this would be Map.Entry interface, this
> is the
> > same
> > > reasons
> > > > on
> > > > > the map interface I cannot actually make the
> underlying Node
> > object
> > > > that’s
> > > > > the implementation for HashMap, so that internals can
> be
> > changed.
> > > > >
> > > > >
> > > > > On point 3,
> > > > >
> > > > > I think it people do expect it to be performant, thus
> why
> > originally
> > > > > concern I raised with using an array for to me is an
> early
> > memory
> > > > > optimisation. I think the user experience of
> > properties/headers is on a
> > > > > get/set model. This is why its important we have
> > encapsulated logic
> > > that
> > > > > then allows us to change this to a map, if this
> becomes and
> > issue, and
> > > > the
> > > > > memory overhead of hashmap is less so.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of
> > Ismael Juma" <
> > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk>
> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Great to see the progress that has been achieved
> on this
> > one. :) A
> > > > few
> > > > > comments regarding the APIs (I'm still reviewing
> the
> > message format
> > > > > changes):
> > > > >
> > > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> > `ConsumerRecord`
> > > should
> > > > be
> > > > > named `headers` (we avoid the `get` prefix in
> Kafka)
> > > > >
> > > > > 2. The `Headers` class is mutable (there's an `add`
> > method). Does
> > > it
> > > > > need
> > > > > to be? If so, it would be good to explain why.
> Related
> > to that, we
> > > > > should
> > > > > also explain the thinking around thread-safety. If
> we
> > keep the
> > > `add`
> > > > > method, it may make sense for it to take a `Header`
> > (that way we
> > > can
> > > > > add
> > > > > things to `Header` without changing the interface).
> > > > >
> > > > > 3. Do we need the `Headers.get()` method? People
> usually
> > assume
> > > that
> > > > > `get`
> > > > > would be efficient, but depending on the
> implementation
> > (the
> > > current
> > > > > proposal states that an array would be used), it
> may not
> > be. If we
> > > > > expect
> > > > > the number of headers to be small, it doesn't
> matter
> > though.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Have converted the interface/api bullets into
> > interface code
> > > > > snippets.
> > > > > >
> > > > > > Agreed implementation won’t take too long. We
> have
> > early versions
> > > > > already.
> > > > > > Maybe a week before you think about merging I
> would
> > assume it
> > > would
> > > > > be more
> > > > > > stabilised? I was thinking then we could fork
> from
> > your confluent
> > > > > branch,
> > > > > > making and then holding KIP-82 changes in a patch
> > file, that we
> > > can
> > > > > then
> > > > > > re-fork from apache once KIP98 final is merged,
> and
> > apply patch
> > > > with
> > > > > last
> > > > > > minute changes.
> > > > > >
> > > > > > Cheers
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > Hey Michael,
> > > > > >
> > > > > > Awesome. I have a minor request. The APIs are
> > currently
> > > > > documented as a
> > > > > > wiki list. Would you mind adding a code
> snippet
> > instead?
> > > It's a
> > > > > bit
> > > > > > easier
> > > > > > to process.
> > > > > >
> > > > > > How will be best to manage this, as we will
> > obviously build
> > > off
> > > > > your
> > > > > > KIP’s
> > > > > > > protocol changes, to avoid a merge hell,
> should
> > we branch
> > > > from
> > > > > your
> > > > > > branch
> > > > > > > in the confluent repo or is it worth
> having a
> > KIP-98
> > > special
> > > > > branch
> > > > > > in the
> > > > > > > apache git, that we can branch/fork from?
> > > > > >
> > > > > >
> > > > > > I was thinking about this also. Ideally we'd
> like
> > to get the
> > > > > changes
> > > > > > in as
> > > > > > close together as possible since we only
> want one
> > magic bump
> > > > and
> > > > > some
> > > > > > users
> > > > > > deploy trunk. The level of effort to change
> the
> > format for
> > > > > headers
> > > > > > seems
> > > > > > not too high. Do you agree? My guess is that
> the
> > KIP-98
> > > message
> > > > > format
> > > > > > patch will take 2-3 weeks to review before we
> > merge to trunk,
> > > > so
> > > > > you
> > > > > > could
> > > > > > hold off implementing until that patch has
> somewhat
> > > stabilized.
> > > > > That
> > > > > > would
> > > > > > save some potential rebase pain.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > confidential
> > > > and
> > > > > for
> > > > > > the use of the addressee only, unless otherwise
> > indicated. If you
> > > > > are not
> > > > > > the intended recipient, please do not read,
> copy, use
> > or disclose
> > > > to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify the
> > sender by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896
> 0011) and
> > then delete
> > > > the
> > > > > > email and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > > relate
> > > > > > to the official business of this company shall be
> > understood as
> > > > > neither
> > > > > > given nor endorsed by it. IG is a trading name
> of IG
> > Markets
> > > > Limited
> > > > > (a
> > > > > > company registered in England and Wales, company
> > number 04008957)
> > > > > and IG
> > > > > > Index Limited (a company registered in England
> and
> > Wales, company
> > > > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25 Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are
> > > not
> > > > > the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify the
> > sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > > the
> > > > > official business of this company shall be understood
> as
> > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > > registered in England and Wales, company number
> 04008957)
> > and IG Index
> > > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > > 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and
> > > IG
> > > > > Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential
> > and for
> > > the use of the addressee only, unless otherwise indicated.
> If
> > you are not
> > > the intended recipient, please do not read, copy, use or
> > disclose to others
> > > this message or any attachment. Please also notify the
> sender by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the email
> > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > company
> > > registered in England and Wales, company number 04008957)
> and IG
> > Index
> > > Limited (a company registered in England and Wales, company
> > number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> > regulated by the
> > > Financial Conduct Authority.
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
append-only would mean that if (for whatever reason) i want to replace a
header or strip it out i'd need to copy the whole record.
On Wed, Feb 22, 2017 at 5:09 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Im happy to compromise to keep it mutable but move to an append style api.
> (as in guava interables concat)
>
> class Headers {
> Headers append(Iterable<Header> headers);
> }
>
>
> I don’t think we’d want prepend, this would give the idea of guaranteed
> ordering, when in actual fact we don’t provide that guarantee (.e.g one
> client can put headerA, then headerB, but another could put headerB then
> headerA, this shouldn’t cause issues), Also what if we changed to a hashmap
> for the internal implementation, its just a bucket of entries no ordering.
> I think we just need to provide an api to add/append headers.
>
> This ok? If so ill update KIP to record this.
>
> Cheers
> Mike
>
> On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> The point about usability is fair. It's also reasonable to expect that
> common use cases such as appending headers should be done efficiently.
>
> Perhaps we could compromise with something like this?
>
> class Headers {
> Headers append(Iterable<Header> headers);
> Headers prepend(Iterable<Header> headers);
> }
>
> That retains ease of use while still giving ourselves some flexibility
> in
> the implementation.
>
> -Jason
>
>
> On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > I wasn’t referring to the headers needing to be copied, im meaning
> the
> > fact we’d be forcing a new producer record to be created, with all
> the
> > contents copied.
> >
> > i.e what will happen is utility method will be created or end up
> being
> > used, which does this, and returns the new ProducerRecord instance.
> >
> > ProducerRecord addHeader(ProducerRecord record, Header header){
> > Return New ProducerRecord(record.key, record.value,
> record.timestamp…..,
> > record.headers.concat(header))
> > }
> >
> > To me this seems ugly, but will be inevitable if we don’t make adding
> > headers to existing records a simple clean method call.
> >
> >
> >
> > On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Lazy init can achieve/avoid that.
> >
> > Re the concat, why don’t we implement that inside the Headers
> rather
> > than causing everyone to implement this as adding headers in
> interceptors
> > will be a dominant use case. We want a user friendly API. Having as
> a user
> > having to code this instead of having the headers handle this for me
> seems
> > redundant.
> >
> > On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >
> > I thought the argument was against creating the extra objects
> > unnecessarily
> > (i.e. if they were not accessed). And note that making the
> Headers
> > immutable doesn't necessarily mean that they need to be
> copied:
> > you can do
> > a trick like Guava's Iterables.concat to add additional
> headers
> > without
> > changing the underlying collections.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > If the argument for not having a map holding the key, value
> > pairs is due
> > > to garbage creation of HashMap entry's, forcing the
> creation of
> > a whole new
> > > producer record to simply add a head, surely is creating
> a-lot
> > more?
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > The current producer interceptor API is this:
> > >
> > > ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
> > >
> > > So adding a header means creating a new ProducerRecord
> with a
> > new header
> > > added to the current headers and returning it. Would that
> not
> > work?
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > So how would you have this work if not mutable where
> > interceptors would
> > > > add headers?
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > I think the point on the mutability of Headers is worth
> > discussing a
> > > little
> > > > more. As far as I can tell, once the ProducerRecord (or
> > ConsumerRecord)
> > > is
> > > > constructed, there should be no need to further change
> the
> > headers. Is
> > > that
> > > > correct? If so, then why not enforce that that is the
> case
> > through the
> > > API?
> > > > One problem with mutability it that it constrains the
> > implementation of
> > > > Headers. For example, if we were backing with a byte
> slice,
> > would we
> > > recopy
> > > > the bytes if a header is added or would we maintain a
> satellite
> > > collection
> > > > of added records. Seems not great either way. If we
> really
> > think
> > > mutability
> > > > is needed, perhaps we could add a method to headers to
> convert
> > it to a
> > > > mutable type (e.g. a Headers.toMap)?
> > > >
> > > > I'm also with Ismael about exposing Headers.get(). I
> thought
> > it might
> > > make
> > > > sense to have a method like this instead:
> > > >
> > > > Iterable<Header> findMatching(Pattern pattern);
> > > >
> > > > This makes the (potential) need to scan the headers
> clear in
> > the API. I'd
> > > > also be fine exposing no getter at all. In general, Ï
> think
> > it's good to
> > > > start with a minimalistic API and work from there since
> it's
> > always
> > > easier
> > > > to add methods than remove them.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > >
> > > > > Hi Ismael
> > > > >
> > > > > On point 1,
> > > > >
> > > > > Sure makes sense will update shortly.
> > > > >
> > > > > On point 2,
> > > > >
> > > > > Setter/getter typical to properties/headers api’s
> > traditionally are map
> > > > > styled interfaces and what I believe is most expected
> styled
> > thus the
> > > > Key,
> > > > > Value setter.
> > > > > Also it would mean rather than an interface, we would
> be
> > making our
> > > > > internal header impl object we have for the array,
> exposed.
> > E.g. if we
> > > > had
> > > > > a Map really this would be Map.Entry interface, this
> is the
> > same
> > > reasons
> > > > on
> > > > > the map interface I cannot actually make the
> underlying Node
> > object
> > > > that’s
> > > > > the implementation for HashMap, so that internals can
> be
> > changed.
> > > > >
> > > > >
> > > > > On point 3,
> > > > >
> > > > > I think it people do expect it to be performant, thus
> why
> > originally
> > > > > concern I raised with using an array for to me is an
> early
> > memory
> > > > > optimisation. I think the user experience of
> > properties/headers is on a
> > > > > get/set model. This is why its important we have
> > encapsulated logic
> > > that
> > > > > then allows us to change this to a map, if this
> becomes and
> > issue, and
> > > > the
> > > > > memory overhead of hashmap is less so.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of
> > Ismael Juma" <
> > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk>
> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Great to see the progress that has been achieved
> on this
> > one. :) A
> > > > few
> > > > > comments regarding the APIs (I'm still reviewing
> the
> > message format
> > > > > changes):
> > > > >
> > > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> > `ConsumerRecord`
> > > should
> > > > be
> > > > > named `headers` (we avoid the `get` prefix in
> Kafka)
> > > > >
> > > > > 2. The `Headers` class is mutable (there's an `add`
> > method). Does
> > > it
> > > > > need
> > > > > to be? If so, it would be good to explain why.
> Related
> > to that, we
> > > > > should
> > > > > also explain the thinking around thread-safety. If
> we
> > keep the
> > > `add`
> > > > > method, it may make sense for it to take a `Header`
> > (that way we
> > > can
> > > > > add
> > > > > things to `Header` without changing the interface).
> > > > >
> > > > > 3. Do we need the `Headers.get()` method? People
> usually
> > assume
> > > that
> > > > > `get`
> > > > > would be efficient, but depending on the
> implementation
> > (the
> > > current
> > > > > proposal states that an array would be used), it
> may not
> > be. If we
> > > > > expect
> > > > > the number of headers to be small, it doesn't
> matter
> > though.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Have converted the interface/api bullets into
> > interface code
> > > > > snippets.
> > > > > >
> > > > > > Agreed implementation won’t take too long. We
> have
> > early versions
> > > > > already.
> > > > > > Maybe a week before you think about merging I
> would
> > assume it
> > > would
> > > > > be more
> > > > > > stabilised? I was thinking then we could fork
> from
> > your confluent
> > > > > branch,
> > > > > > making and then holding KIP-82 changes in a patch
> > file, that we
> > > can
> > > > > then
> > > > > > re-fork from apache once KIP98 final is merged,
> and
> > apply patch
> > > > with
> > > > > last
> > > > > > minute changes.
> > > > > >
> > > > > > Cheers
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > Hey Michael,
> > > > > >
> > > > > > Awesome. I have a minor request. The APIs are
> > currently
> > > > > documented as a
> > > > > > wiki list. Would you mind adding a code
> snippet
> > instead?
> > > It's a
> > > > > bit
> > > > > > easier
> > > > > > to process.
> > > > > >
> > > > > > How will be best to manage this, as we will
> > obviously build
> > > off
> > > > > your
> > > > > > KIP’s
> > > > > > > protocol changes, to avoid a merge hell,
> should
> > we branch
> > > > from
> > > > > your
> > > > > > branch
> > > > > > > in the confluent repo or is it worth
> having a
> > KIP-98
> > > special
> > > > > branch
> > > > > > in the
> > > > > > > apache git, that we can branch/fork from?
> > > > > >
> > > > > >
> > > > > > I was thinking about this also. Ideally we'd
> like
> > to get the
> > > > > changes
> > > > > > in as
> > > > > > close together as possible since we only
> want one
> > magic bump
> > > > and
> > > > > some
> > > > > > users
> > > > > > deploy trunk. The level of effort to change
> the
> > format for
> > > > > headers
> > > > > > seems
> > > > > > not too high. Do you agree? My guess is that
> the
> > KIP-98
> > > message
> > > > > format
> > > > > > patch will take 2-3 weeks to review before we
> > merge to trunk,
> > > > so
> > > > > you
> > > > > > could
> > > > > > hold off implementing until that patch has
> somewhat
> > > stabilized.
> > > > > That
> > > > > > would
> > > > > > save some potential rebase pain.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > confidential
> > > > and
> > > > > for
> > > > > > the use of the addressee only, unless otherwise
> > indicated. If you
> > > > > are not
> > > > > > the intended recipient, please do not read,
> copy, use
> > or disclose
> > > > to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify the
> > sender by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896
> 0011) and
> > then delete
> > > > the
> > > > > > email and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > > relate
> > > > > > to the official business of this company shall be
> > understood as
> > > > > neither
> > > > > > given nor endorsed by it. IG is a trading name
> of IG
> > Markets
> > > > Limited
> > > > > (a
> > > > > > company registered in England and Wales, company
> > number 04008957)
> > > > > and IG
> > > > > > Index Limited (a company registered in England
> and
> > Wales, company
> > > > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25 Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are
> > > not
> > > > > the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify the
> > sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > > the
> > > > > official business of this company shall be understood
> as
> > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > > registered in England and Wales, company number
> 04008957)
> > and IG Index
> > > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > > 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and
> > > IG
> > > > > Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential
> > and for
> > > the use of the addressee only, unless otherwise indicated.
> If
> > you are not
> > > the intended recipient, please do not read, copy, use or
> > disclose to others
> > > this message or any attachment. Please also notify the
> sender by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the email
> > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > company
> > > registered in England and Wales, company number 04008957)
> and IG
> > Index
> > > Limited (a company registered in England and Wales, company
> > number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> > regulated by the
> > > Financial Conduct Authority.
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Becket Qin <be...@gmail.com>.
Hi Ismael,
Yes, it makes sense to do benchmark. My concern was based on the
observation in KAFKA-3994 where we saw GC problem when creating new lists
in the purgatory.
Thanks,
Jiangjie (Becket) Qin
On Fri, Mar 10, 2017 at 8:54 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Becket,
>
> Sorry for the delay and thanks for your comments. Comments inline.
>
> On Wed, Mar 1, 2017 at 8:59 PM, Becket Qin <be...@gmail.com> wrote:
>
> > > The difference is that the user chooses the value type. They are free
> to
> > > choose a mutable or immutable type. A generic interceptor cannot mutate
> > the
> > > value because it doesn't know the type (and it could be immutable). One
> > > could write an interceptor that checked the type of the value at
> runtime
> > > and did things based on that. But again, the user who creates the
> record
> > is
> > > in control.
> > >
> > But there is no generic interceptor, right? The interceptor always takes
> > specific K, V type.
> >
>
> No, generic interceptors exist, are very useful and we use them at
> Confluent.
>
> If we want to let the users control the mutability, users can always call
> > headers.close() before calling producer.send() and that will force the
> > interceptor to create new ProducerRecord object.
> >
>
> If we expect users to call `close()`, then this is an important point to
> document because interceptors would have to catch an exception or check if
> the record has already been closed. I'm not sure if we should allow or
> encourage that.
>
>
> > If we do not allow the users to add headers on existing ProducerRecord
> > objects, each interceptor who wants to add headers will have to create a
> > new ProducerRecord. So we will have to create NUM_INTERCEPTOR times of
> > ProducerRecord, if a producer is sending 100K messages per second, it
> would
> > be a lot of new objects, right?
> >
>
> Note we are allocating one Header instance per header. If you are concerned
> about the number of ProducerRecord instances, then it seems like you should
> also be concerned about the number of Header instances, which is likely to
> be a multiple of the former. In addition, if we use a Map to hold the
> headers, we'll also need to allocate the map entry instances. This is not
> to say that we should be unnecessarily wasteful, but it's worth
> understanding what we're trying to achieve to make sure that we are not
> solving the wrong bottleneck.
>
> It may make sense to do some benchmarking during the implementation phase
> to verify that the solution achieves whatever performance goal we're aiming
> for.
>
> Thanks,
> Ismael
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Becket,
Sorry for the delay and thanks for your comments. Comments inline.
On Wed, Mar 1, 2017 at 8:59 PM, Becket Qin <be...@gmail.com> wrote:
> > The difference is that the user chooses the value type. They are free to
> > choose a mutable or immutable type. A generic interceptor cannot mutate
> the
> > value because it doesn't know the type (and it could be immutable). One
> > could write an interceptor that checked the type of the value at runtime
> > and did things based on that. But again, the user who creates the record
> is
> > in control.
> >
> But there is no generic interceptor, right? The interceptor always takes
> specific K, V type.
>
No, generic interceptors exist, are very useful and we use them at
Confluent.
If we want to let the users control the mutability, users can always call
> headers.close() before calling producer.send() and that will force the
> interceptor to create new ProducerRecord object.
>
If we expect users to call `close()`, then this is an important point to
document because interceptors would have to catch an exception or check if
the record has already been closed. I'm not sure if we should allow or
encourage that.
> If we do not allow the users to add headers on existing ProducerRecord
> objects, each interceptor who wants to add headers will have to create a
> new ProducerRecord. So we will have to create NUM_INTERCEPTOR times of
> ProducerRecord, if a producer is sending 100K messages per second, it would
> be a lot of new objects, right?
>
Note we are allocating one Header instance per header. If you are concerned
about the number of ProducerRecord instances, then it seems like you should
also be concerned about the number of Header instances, which is likely to
be a multiple of the former. In addition, if we use a Map to hold the
headers, we'll also need to allocate the map entry instances. This is not
to say that we should be unnecessarily wasteful, but it's worth
understanding what we're trying to achieve to make sure that we are not
solving the wrong bottleneck.
It may make sense to do some benchmarking during the implementation phase
to verify that the solution achieves whatever performance goal we're aiming
for.
Thanks,
Ismael
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
just to clarify - ListIterator is a nice API, and doesnt constrain the
implementation a lot more than Iterator (especially if we implement
previous() very inefficiently :-) ), but changing
Iterable<Header> headers(String key)
into
ListIterator<Header> headers(String key)
would lose us the ability to easily write what i think is the most common
case - a for each loop:
for (Header stop : headers("lineage")) {
//examine stop
}
On Tue, Mar 7, 2017 at 12:31 PM, radai <ra...@gmail.com> wrote:
> ing, as you call it, would probably be implemente
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
where do you see insert-in-the-middle/replace being commonly used?
lineage tracing, as you call it, would probably be implemented by way of:
1. every "stop" along the way appending itself (at the end)
2. some replication technologies, instead of just doing #1, may clear out
everything when they replicate (starting from a clean slate)
On Mon, Mar 6, 2017 at 11:00 AM, Colin McCabe <cm...@apache.org> wrote:
> As others have mentioned, it seems clear that we want to preserve the
> ordering of message headers, so that we can implement things like
> lineage tracing. (For example, each stage could add a "lineage:"
> header.) I also think that we want the ability to add and remove
> headers as needed. It would be really unfortunate if the only way to
> remove a message header or add a header at a certain position was the
> duplicate the whole message and re-create everything.
>
> So how about implementing ListIterator?
> https://docs.oracle.com/javase/7/docs/api/java/util/ListIterator.html
> It supports adding and removing things at arbitrary positions. For
> people who want to use it as a simple Iterator, it is one (and you can
> use all the fancy syntax such as Java's foreach with it). For people
> who want to add and remove things at arbitrary locations, they can. And
> it doesn't expose the implementation, so that can be changed later. We
> can materialize things in memory lazily if we want to, and so forth. I
> think using the standard interface is better than rolling our own
> nonstandard collection or iterator interfaces.
>
> regards,
> Colin
>
>
> On Wed, Mar 1, 2017, at 12:59, Becket Qin wrote:
> > Hi Ismael,
> >
> > Thanks for the reply. Please see the comments inline.
> >
> > On Wed, Mar 1, 2017 at 6:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Becket,
> > >
> > > Thanks for sharing your thoughts. More inline.
> > >
> > > On Wed, Mar 1, 2017 at 2:54 AM, Becket Qin <be...@gmail.com>
> wrote:
> > >
> > > > As you can imagine if the ProducerRecord has a value as a List and
> the
> > > > Interceptor.onSend() can actually add an element to the List. If the
> > > > producer.send() is called on the same ProducerRecord again, the value
> > > list
> > > > would have one more element than the previously sent ProducerRecord
> even
> > > > though the ProducerRecord itself is not mutable, right? Same thing
> can
> > > > apply to any modifiable class type.
> > > >
> > >
> > > The difference is that the user chooses the value type. They are free
> to
> > > choose a mutable or immutable type. A generic interceptor cannot
> mutate the
> > > value because it doesn't know the type (and it could be immutable). One
> > > could write an interceptor that checked the type of the value at
> runtime
> > > and did things based on that. But again, the user who creates the
> record is
> > > in control.
> > >
> > But there is no generic interceptor, right? The interceptor always takes
> > specific K, V type.
> >
> >
> > > From this standpoint allowing headers to be mutable doesn't really
> weaken
> > > > the mutability we already have. Admittedly a mutable header is kind
> of
> > > > guiding user towards to change the headers in the existing object
> instead
> > > > of creating a new one.
> > >
> > >
> > > Yes, with headers, we are providing a model for the user (the user
> doesn't
> > > get to choose it like with keys and values) and for the interceptors.
> So, I
> > > think it's not the same.
> >
> >
> > >
> > > > But I think reusing an object while it is possible
> > > > to be modified by user code is a risk that users themselves are
> willing
> > > to
> > > > take. And we do not really protect against that.
> > >
> > >
> > > If users want to take that risk, it's fine. But we give them the
> option to
> > > protect themselves. With mutable headers, there is no option.
> >
> > If we want to let the users control the mutability, users can always call
> > headers.close() before calling producer.send() and that will force the
> > interceptor to create new ProducerRecord object.
> >
> > Because the headers are mostly useful for interceptors, unless the users
> > do
> > not want the interceptors to change their records, it seems reasonable to
> > say that by default modification of headers are allowed for the
> > interceptors.
> >
> > >
> > >
> > > > But there still seems
> > > > value to allow the users to not pay the overhead of creating tons of
> > > > objects if they do not reuse an object to send it twice, which is
> > > probably
> > > > a more common case.
> > > >
> > >
> > > I think the assumption that there would be tons of objects created is
> > > incorrect (I suggested a solution that would only involve one
> additional
> > > reference in the `Header` instance). The usability of the immutable API
> > > seemed to be more of an issue.
> > >
> > If we do not allow the users to add headers on existing ProducerRecord
> > objects, each interceptor who wants to add headers will have to create a
> > new ProducerRecord. So we will have to create NUM_INTERCEPTOR times of
> > ProducerRecord, if a producer is sending 100K messages per second, it
> > would
> > be a lot of new objects, right?
> >
> > >
> > > In any case, if we do add the `close()` method, then we need to add a
> note
> > > to the compatibility section stating that once a producer record is
> sent,
> > > it cannot be sent again as this would cause interceptors that add
> headers
> > > to fail.
> > >
> > Agreed, clear documentation is important.
> >
> > >
> > > Ismael
> > >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Colin McCabe <cm...@apache.org>.
As others have mentioned, it seems clear that we want to preserve the
ordering of message headers, so that we can implement things like
lineage tracing. (For example, each stage could add a "lineage:"
header.) I also think that we want the ability to add and remove
headers as needed. It would be really unfortunate if the only way to
remove a message header or add a header at a certain position was the
duplicate the whole message and re-create everything.
So how about implementing ListIterator?
https://docs.oracle.com/javase/7/docs/api/java/util/ListIterator.html
It supports adding and removing things at arbitrary positions. For
people who want to use it as a simple Iterator, it is one (and you can
use all the fancy syntax such as Java's foreach with it). For people
who want to add and remove things at arbitrary locations, they can. And
it doesn't expose the implementation, so that can be changed later. We
can materialize things in memory lazily if we want to, and so forth. I
think using the standard interface is better than rolling our own
nonstandard collection or iterator interfaces.
regards,
Colin
On Wed, Mar 1, 2017, at 12:59, Becket Qin wrote:
> Hi Ismael,
>
> Thanks for the reply. Please see the comments inline.
>
> On Wed, Mar 1, 2017 at 6:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Becket,
> >
> > Thanks for sharing your thoughts. More inline.
> >
> > On Wed, Mar 1, 2017 at 2:54 AM, Becket Qin <be...@gmail.com> wrote:
> >
> > > As you can imagine if the ProducerRecord has a value as a List and the
> > > Interceptor.onSend() can actually add an element to the List. If the
> > > producer.send() is called on the same ProducerRecord again, the value
> > list
> > > would have one more element than the previously sent ProducerRecord even
> > > though the ProducerRecord itself is not mutable, right? Same thing can
> > > apply to any modifiable class type.
> > >
> >
> > The difference is that the user chooses the value type. They are free to
> > choose a mutable or immutable type. A generic interceptor cannot mutate the
> > value because it doesn't know the type (and it could be immutable). One
> > could write an interceptor that checked the type of the value at runtime
> > and did things based on that. But again, the user who creates the record is
> > in control.
> >
> But there is no generic interceptor, right? The interceptor always takes
> specific K, V type.
>
>
> > From this standpoint allowing headers to be mutable doesn't really weaken
> > > the mutability we already have. Admittedly a mutable header is kind of
> > > guiding user towards to change the headers in the existing object instead
> > > of creating a new one.
> >
> >
> > Yes, with headers, we are providing a model for the user (the user doesn't
> > get to choose it like with keys and values) and for the interceptors. So, I
> > think it's not the same.
>
>
> >
> > > But I think reusing an object while it is possible
> > > to be modified by user code is a risk that users themselves are willing
> > to
> > > take. And we do not really protect against that.
> >
> >
> > If users want to take that risk, it's fine. But we give them the option to
> > protect themselves. With mutable headers, there is no option.
>
> If we want to let the users control the mutability, users can always call
> headers.close() before calling producer.send() and that will force the
> interceptor to create new ProducerRecord object.
>
> Because the headers are mostly useful for interceptors, unless the users
> do
> not want the interceptors to change their records, it seems reasonable to
> say that by default modification of headers are allowed for the
> interceptors.
>
> >
> >
> > > But there still seems
> > > value to allow the users to not pay the overhead of creating tons of
> > > objects if they do not reuse an object to send it twice, which is
> > probably
> > > a more common case.
> > >
> >
> > I think the assumption that there would be tons of objects created is
> > incorrect (I suggested a solution that would only involve one additional
> > reference in the `Header` instance). The usability of the immutable API
> > seemed to be more of an issue.
> >
> If we do not allow the users to add headers on existing ProducerRecord
> objects, each interceptor who wants to add headers will have to create a
> new ProducerRecord. So we will have to create NUM_INTERCEPTOR times of
> ProducerRecord, if a producer is sending 100K messages per second, it
> would
> be a lot of new objects, right?
>
> >
> > In any case, if we do add the `close()` method, then we need to add a note
> > to the compatibility section stating that once a producer record is sent,
> > it cannot be sent again as this would cause interceptors that add headers
> > to fail.
> >
> Agreed, clear documentation is important.
>
> >
> > Ismael
> >
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Becket Qin <be...@gmail.com>.
Hi Ismael,
Thanks for the reply. Please see the comments inline.
On Wed, Mar 1, 2017 at 6:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Becket,
>
> Thanks for sharing your thoughts. More inline.
>
> On Wed, Mar 1, 2017 at 2:54 AM, Becket Qin <be...@gmail.com> wrote:
>
> > As you can imagine if the ProducerRecord has a value as a List and the
> > Interceptor.onSend() can actually add an element to the List. If the
> > producer.send() is called on the same ProducerRecord again, the value
> list
> > would have one more element than the previously sent ProducerRecord even
> > though the ProducerRecord itself is not mutable, right? Same thing can
> > apply to any modifiable class type.
> >
>
> The difference is that the user chooses the value type. They are free to
> choose a mutable or immutable type. A generic interceptor cannot mutate the
> value because it doesn't know the type (and it could be immutable). One
> could write an interceptor that checked the type of the value at runtime
> and did things based on that. But again, the user who creates the record is
> in control.
>
But there is no generic interceptor, right? The interceptor always takes
specific K, V type.
> From this standpoint allowing headers to be mutable doesn't really weaken
> > the mutability we already have. Admittedly a mutable header is kind of
> > guiding user towards to change the headers in the existing object instead
> > of creating a new one.
>
>
> Yes, with headers, we are providing a model for the user (the user doesn't
> get to choose it like with keys and values) and for the interceptors. So, I
> think it's not the same.
>
> > But I think reusing an object while it is possible
> > to be modified by user code is a risk that users themselves are willing
> to
> > take. And we do not really protect against that.
>
>
> If users want to take that risk, it's fine. But we give them the option to
> protect themselves. With mutable headers, there is no option.
If we want to let the users control the mutability, users can always call
headers.close() before calling producer.send() and that will force the
interceptor to create new ProducerRecord object.
Because the headers are mostly useful for interceptors, unless the users do
not want the interceptors to change their records, it seems reasonable to
say that by default modification of headers are allowed for the
interceptors.
>
>
> > But there still seems
> > value to allow the users to not pay the overhead of creating tons of
> > objects if they do not reuse an object to send it twice, which is
> probably
> > a more common case.
> >
>
> I think the assumption that there would be tons of objects created is
> incorrect (I suggested a solution that would only involve one additional
> reference in the `Header` instance). The usability of the immutable API
> seemed to be more of an issue.
>
If we do not allow the users to add headers on existing ProducerRecord
objects, each interceptor who wants to add headers will have to create a
new ProducerRecord. So we will have to create NUM_INTERCEPTOR times of
ProducerRecord, if a producer is sending 100K messages per second, it would
be a lot of new objects, right?
>
> In any case, if we do add the `close()` method, then we need to add a note
> to the compatibility section stating that once a producer record is sent,
> it cannot be sent again as this would cause interceptors that add headers
> to fail.
>
Agreed, clear documentation is important.
>
> Ismael
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
@michael:
i used void because im used to java beans. thinking about it, i dont see
much use for returning false from adding a header: if the headers are in
read-only you should probably thrown an IllegalStateException because lets
face it, 99% of users dont check return values.
returning "this" is probably more useful because it would allow chaining:
Headers.add().add().remove()
On Wed, Mar 1, 2017 at 6:47 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Becket,
>
> Thanks for sharing your thoughts. More inline.
>
> On Wed, Mar 1, 2017 at 2:54 AM, Becket Qin <be...@gmail.com> wrote:
>
> > As you can imagine if the ProducerRecord has a value as a List and the
> > Interceptor.onSend() can actually add an element to the List. If the
> > producer.send() is called on the same ProducerRecord again, the value
> list
> > would have one more element than the previously sent ProducerRecord even
> > though the ProducerRecord itself is not mutable, right? Same thing can
> > apply to any modifiable class type.
> >
>
> The difference is that the user chooses the value type. They are free to
> choose a mutable or immutable type. A generic interceptor cannot mutate the
> value because it doesn't know the type (and it could be immutable). One
> could write an interceptor that checked the type of the value at runtime
> and did things based on that. But again, the user who creates the record is
> in control.
>
> From this standpoint allowing headers to be mutable doesn't really weaken
> > the mutability we already have. Admittedly a mutable header is kind of
> > guiding user towards to change the headers in the existing object instead
> > of creating a new one.
>
>
> Yes, with headers, we are providing a model for the user (the user doesn't
> get to choose it like with keys and values) and for the interceptors. So, I
> think it's not the same.
>
>
> > But I think reusing an object while it is possible
> > to be modified by user code is a risk that users themselves are willing
> to
> > take. And we do not really protect against that.
>
>
> If users want to take that risk, it's fine. But we give them the option to
> protect themselves. With mutable headers, there is no option.
>
>
> > But there still seems
> > value to allow the users to not pay the overhead of creating tons of
> > objects if they do not reuse an object to send it twice, which is
> probably
> > a more common case.
> >
>
> I think the assumption that there would be tons of objects created is
> incorrect (I suggested a solution that would only involve one additional
> reference in the `Header` instance). The usability of the immutable API
> seemed to be more of an issue.
>
> In any case, if we do add the `close()` method, then we need to add a note
> to the compatibility section stating that once a producer record is sent,
> it cannot be sent again as this would cause interceptors that add headers
> to fail.
>
> Ismael
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Becket,
Thanks for sharing your thoughts. More inline.
On Wed, Mar 1, 2017 at 2:54 AM, Becket Qin <be...@gmail.com> wrote:
> As you can imagine if the ProducerRecord has a value as a List and the
> Interceptor.onSend() can actually add an element to the List. If the
> producer.send() is called on the same ProducerRecord again, the value list
> would have one more element than the previously sent ProducerRecord even
> though the ProducerRecord itself is not mutable, right? Same thing can
> apply to any modifiable class type.
>
The difference is that the user chooses the value type. They are free to
choose a mutable or immutable type. A generic interceptor cannot mutate the
value because it doesn't know the type (and it could be immutable). One
could write an interceptor that checked the type of the value at runtime
and did things based on that. But again, the user who creates the record is
in control.
From this standpoint allowing headers to be mutable doesn't really weaken
> the mutability we already have. Admittedly a mutable header is kind of
> guiding user towards to change the headers in the existing object instead
> of creating a new one.
Yes, with headers, we are providing a model for the user (the user doesn't
get to choose it like with keys and values) and for the interceptors. So, I
think it's not the same.
> But I think reusing an object while it is possible
> to be modified by user code is a risk that users themselves are willing to
> take. And we do not really protect against that.
If users want to take that risk, it's fine. But we give them the option to
protect themselves. With mutable headers, there is no option.
> But there still seems
> value to allow the users to not pay the overhead of creating tons of
> objects if they do not reuse an object to send it twice, which is probably
> a more common case.
>
I think the assumption that there would be tons of objects created is
incorrect (I suggested a solution that would only involve one additional
reference in the `Header` instance). The usability of the immutable API
seemed to be more of an issue.
In any case, if we do add the `close()` method, then we need to add a note
to the compatibility section stating that once a producer record is sent,
it cannot be sent again as this would cause interceptors that add headers
to fail.
Ismael
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Sent to early:
Hi Radai:
RE:
Header header(String key) - returns JUST ONE (the very last) value given a
key
Iterable<Header> headers(String key) - returns ALL under a key
Iterable<Header> headers() - returns all, period. maybe allow null as key
to prev method instead?
void add(Header header) - appends a header (key inside).
void remove(String key) - removes ALL HEADERS under a key.
I don't think this one is needed:
"Iterable<Header> headers() - returns all, period. maybe allow null as key to prev method instead"
The class Headers implements Iterable<Header> anyhow.
On another note of the response for the add, remove. I note you put void. Any particular reason?
for add, i would normally either expect a boolean response if it succeeded or not (classical case) or the instance object returned like the "with" pattern. This avoids in the case of when its in read only, needing to throw exception, simply return false (in classical case).
Like wise on remove normally i would expect a boolean or previous object returned, so again operation didn't succeed either i get a false, or a null object.
any objections if i make the response's for these to booleans to denote if the operation succeeded?
Cheers
Mike
________________________________________
From: Michael Pearce
Sent: Wednesday, March 1, 2017 5:55 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Hi Radai:
RE:
Header header(String key) - returns JUST ONE (the very last) value given a
key
Iterable<Header> headers(String key) - returns ALL under a key
void add(Header header) - appends a header (key inside).
void remove(String key) - removes ALL HEADERS under a key.
I don't think this one is needed:
________________________________________
From: Becket Qin <be...@gmail.com>
Sent: Wednesday, March 1, 2017 2:54 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Hi Ismael,
Yes, there is a difference between Batch and Headers. I was just trying to
see if that would work. Good point about sending the same ProducerRecord
twice, but in fact in that case any reuse of objects would cause problem.
As you can imagine if the ProducerRecord has a value as a List and the
Interceptor.onSend() can actually add an element to the List. If the
producer.send() is called on the same ProducerRecord again, the value list
would have one more element than the previously sent ProducerRecord even
though the ProducerRecord itself is not mutable, right? Same thing can
apply to any modifiable class type.
From this standpoint allowing headers to be mutable doesn't really weaken
the mutability we already have. Admittedly a mutable header is kind of
guiding user towards to change the headers in the existing object instead
of creating a new one. But I think reusing an object while it is possible
to be modified by user code is a risk that users themselves are willing to
take. And we do not really protect against that. But there still seems
value to allow the users to not pay the overhead of creating tons of
objects if they do not reuse an object to send it twice, which is probably
a more common case.
Thanks,
Jiangjie (Becket) Qin
On Tue, Feb 28, 2017 at 12:43 PM, radai <ra...@gmail.com> wrote:
> I will settle for any API really, but just wanted to point out that as it
> stands right now the API targets the most "advanced" (hence obscure and
> rare) use cases, at the expense of the simple and common ones. i'd suggest
> (as the minimal set):
>
> Header header(String key) - returns JUST ONE (the very last) value given a
> key
> Iterable<Header> headers(String key) - returns ALL under a key
> Iterable<Header> headers() - returns all, period. maybe allow null as key
> to prev method instead?
> void add(Header header) - appends a header (key inside).
> void remove(String key) - removes ALL HEADERS under a key.
>
> this way naive get/set semantics map to header(key)/add(Header) cleanly and
> simply while preserving the ability to handle more advanced use cases.
> we can always add more convenience methods (like those dealing with lists -
> addAll etc) but i think the 5 (potentially 4) above are sufficient for
> basically everything.
>
> On Tue, Feb 28, 2017 at 4:08 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Becket,
> >
> > Comments inline.
> >
> > On Sat, Feb 25, 2017 at 10:33 PM, Becket Qin <be...@gmail.com>
> wrote:
> > >
> > > 1. Regarding the mutability.
> > >
> > > I think it would be a big convenience to have headers mutable during
> > > certain stage in the message life cycle for the use cases you
> mentioned.
> > I
> > > agree there is a material benefit especially given that we may have to
> > > modify the headers for each message.
> > >
> > > That said, I also think it is fair to say that in the producer, in
> order
> > to
> > > guarantee the correctness of the entire logic, it is necessary that at
> > some
> > > point we need to make producer record immutable. For example we
> probably
> > > don't want to see that users accidentally updated the headers when the
> > > producer is doing the serialization or compression.
> > >
> > > Given that, would it be possible to make Headers to be able to switch
> > from
> > > mutable to immutable? We have done this for the Batch in the producer.
> > For
> > > example, initially the headers are mutable, but after it has gone
> through
> > > all the interceptors, we can call Headers.close() to make it immutable
> > > afterwards.
> > >
> >
> > The difference is that the batch is an internal class that is not exposed
> > to users. Can you please explain what happens if a user tries to send the
> > same ProducerRecord twice? Would an interceptor fail when trying to
> mutate
> > the header that is now closed? Or did you have something else in mind?
> >
> > Thanks,
> > Ismael
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Radai:
RE:
Header header(String key) - returns JUST ONE (the very last) value given a
key
Iterable<Header> headers(String key) - returns ALL under a key
void add(Header header) - appends a header (key inside).
void remove(String key) - removes ALL HEADERS under a key.
I don't think this one is needed:
________________________________________
From: Becket Qin <be...@gmail.com>
Sent: Wednesday, March 1, 2017 2:54 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Hi Ismael,
Yes, there is a difference between Batch and Headers. I was just trying to
see if that would work. Good point about sending the same ProducerRecord
twice, but in fact in that case any reuse of objects would cause problem.
As you can imagine if the ProducerRecord has a value as a List and the
Interceptor.onSend() can actually add an element to the List. If the
producer.send() is called on the same ProducerRecord again, the value list
would have one more element than the previously sent ProducerRecord even
though the ProducerRecord itself is not mutable, right? Same thing can
apply to any modifiable class type.
From this standpoint allowing headers to be mutable doesn't really weaken
the mutability we already have. Admittedly a mutable header is kind of
guiding user towards to change the headers in the existing object instead
of creating a new one. But I think reusing an object while it is possible
to be modified by user code is a risk that users themselves are willing to
take. And we do not really protect against that. But there still seems
value to allow the users to not pay the overhead of creating tons of
objects if they do not reuse an object to send it twice, which is probably
a more common case.
Thanks,
Jiangjie (Becket) Qin
On Tue, Feb 28, 2017 at 12:43 PM, radai <ra...@gmail.com> wrote:
> I will settle for any API really, but just wanted to point out that as it
> stands right now the API targets the most "advanced" (hence obscure and
> rare) use cases, at the expense of the simple and common ones. i'd suggest
> (as the minimal set):
>
> Header header(String key) - returns JUST ONE (the very last) value given a
> key
> Iterable<Header> headers(String key) - returns ALL under a key
> Iterable<Header> headers() - returns all, period. maybe allow null as key
> to prev method instead?
> void add(Header header) - appends a header (key inside).
> void remove(String key) - removes ALL HEADERS under a key.
>
> this way naive get/set semantics map to header(key)/add(Header) cleanly and
> simply while preserving the ability to handle more advanced use cases.
> we can always add more convenience methods (like those dealing with lists -
> addAll etc) but i think the 5 (potentially 4) above are sufficient for
> basically everything.
>
> On Tue, Feb 28, 2017 at 4:08 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Becket,
> >
> > Comments inline.
> >
> > On Sat, Feb 25, 2017 at 10:33 PM, Becket Qin <be...@gmail.com>
> wrote:
> > >
> > > 1. Regarding the mutability.
> > >
> > > I think it would be a big convenience to have headers mutable during
> > > certain stage in the message life cycle for the use cases you
> mentioned.
> > I
> > > agree there is a material benefit especially given that we may have to
> > > modify the headers for each message.
> > >
> > > That said, I also think it is fair to say that in the producer, in
> order
> > to
> > > guarantee the correctness of the entire logic, it is necessary that at
> > some
> > > point we need to make producer record immutable. For example we
> probably
> > > don't want to see that users accidentally updated the headers when the
> > > producer is doing the serialization or compression.
> > >
> > > Given that, would it be possible to make Headers to be able to switch
> > from
> > > mutable to immutable? We have done this for the Batch in the producer.
> > For
> > > example, initially the headers are mutable, but after it has gone
> through
> > > all the interceptors, we can call Headers.close() to make it immutable
> > > afterwards.
> > >
> >
> > The difference is that the batch is an internal class that is not exposed
> > to users. Can you please explain what happens if a user tries to send the
> > same ProducerRecord twice? Would an interceptor fail when trying to
> mutate
> > the header that is now closed? Or did you have something else in mind?
> >
> > Thanks,
> > Ismael
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Becket Qin <be...@gmail.com>.
Hi Ismael,
Yes, there is a difference between Batch and Headers. I was just trying to
see if that would work. Good point about sending the same ProducerRecord
twice, but in fact in that case any reuse of objects would cause problem.
As you can imagine if the ProducerRecord has a value as a List and the
Interceptor.onSend() can actually add an element to the List. If the
producer.send() is called on the same ProducerRecord again, the value list
would have one more element than the previously sent ProducerRecord even
though the ProducerRecord itself is not mutable, right? Same thing can
apply to any modifiable class type.
From this standpoint allowing headers to be mutable doesn't really weaken
the mutability we already have. Admittedly a mutable header is kind of
guiding user towards to change the headers in the existing object instead
of creating a new one. But I think reusing an object while it is possible
to be modified by user code is a risk that users themselves are willing to
take. And we do not really protect against that. But there still seems
value to allow the users to not pay the overhead of creating tons of
objects if they do not reuse an object to send it twice, which is probably
a more common case.
Thanks,
Jiangjie (Becket) Qin
On Tue, Feb 28, 2017 at 12:43 PM, radai <ra...@gmail.com> wrote:
> I will settle for any API really, but just wanted to point out that as it
> stands right now the API targets the most "advanced" (hence obscure and
> rare) use cases, at the expense of the simple and common ones. i'd suggest
> (as the minimal set):
>
> Header header(String key) - returns JUST ONE (the very last) value given a
> key
> Iterable<Header> headers(String key) - returns ALL under a key
> Iterable<Header> headers() - returns all, period. maybe allow null as key
> to prev method instead?
> void add(Header header) - appends a header (key inside).
> void remove(String key) - removes ALL HEADERS under a key.
>
> this way naive get/set semantics map to header(key)/add(Header) cleanly and
> simply while preserving the ability to handle more advanced use cases.
> we can always add more convenience methods (like those dealing with lists -
> addAll etc) but i think the 5 (potentially 4) above are sufficient for
> basically everything.
>
> On Tue, Feb 28, 2017 at 4:08 AM, Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Becket,
> >
> > Comments inline.
> >
> > On Sat, Feb 25, 2017 at 10:33 PM, Becket Qin <be...@gmail.com>
> wrote:
> > >
> > > 1. Regarding the mutability.
> > >
> > > I think it would be a big convenience to have headers mutable during
> > > certain stage in the message life cycle for the use cases you
> mentioned.
> > I
> > > agree there is a material benefit especially given that we may have to
> > > modify the headers for each message.
> > >
> > > That said, I also think it is fair to say that in the producer, in
> order
> > to
> > > guarantee the correctness of the entire logic, it is necessary that at
> > some
> > > point we need to make producer record immutable. For example we
> probably
> > > don't want to see that users accidentally updated the headers when the
> > > producer is doing the serialization or compression.
> > >
> > > Given that, would it be possible to make Headers to be able to switch
> > from
> > > mutable to immutable? We have done this for the Batch in the producer.
> > For
> > > example, initially the headers are mutable, but after it has gone
> through
> > > all the interceptors, we can call Headers.close() to make it immutable
> > > afterwards.
> > >
> >
> > The difference is that the batch is an internal class that is not exposed
> > to users. Can you please explain what happens if a user tries to send the
> > same ProducerRecord twice? Would an interceptor fail when trying to
> mutate
> > the header that is now closed? Or did you have something else in mind?
> >
> > Thanks,
> > Ismael
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
I will settle for any API really, but just wanted to point out that as it
stands right now the API targets the most "advanced" (hence obscure and
rare) use cases, at the expense of the simple and common ones. i'd suggest
(as the minimal set):
Header header(String key) - returns JUST ONE (the very last) value given a
key
Iterable<Header> headers(String key) - returns ALL under a key
Iterable<Header> headers() - returns all, period. maybe allow null as key
to prev method instead?
void add(Header header) - appends a header (key inside).
void remove(String key) - removes ALL HEADERS under a key.
this way naive get/set semantics map to header(key)/add(Header) cleanly and
simply while preserving the ability to handle more advanced use cases.
we can always add more convenience methods (like those dealing with lists -
addAll etc) but i think the 5 (potentially 4) above are sufficient for
basically everything.
On Tue, Feb 28, 2017 at 4:08 AM, Ismael Juma <is...@juma.me.uk> wrote:
> Hi Becket,
>
> Comments inline.
>
> On Sat, Feb 25, 2017 at 10:33 PM, Becket Qin <be...@gmail.com> wrote:
> >
> > 1. Regarding the mutability.
> >
> > I think it would be a big convenience to have headers mutable during
> > certain stage in the message life cycle for the use cases you mentioned.
> I
> > agree there is a material benefit especially given that we may have to
> > modify the headers for each message.
> >
> > That said, I also think it is fair to say that in the producer, in order
> to
> > guarantee the correctness of the entire logic, it is necessary that at
> some
> > point we need to make producer record immutable. For example we probably
> > don't want to see that users accidentally updated the headers when the
> > producer is doing the serialization or compression.
> >
> > Given that, would it be possible to make Headers to be able to switch
> from
> > mutable to immutable? We have done this for the Batch in the producer.
> For
> > example, initially the headers are mutable, but after it has gone through
> > all the interceptors, we can call Headers.close() to make it immutable
> > afterwards.
> >
>
> The difference is that the batch is an internal class that is not exposed
> to users. Can you please explain what happens if a user tries to send the
> same ProducerRecord twice? Would an interceptor fail when trying to mutate
> the header that is now closed? Or did you have something else in mind?
>
> Thanks,
> Ismael
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Becket,
Comments inline.
On Sat, Feb 25, 2017 at 10:33 PM, Becket Qin <be...@gmail.com> wrote:
>
> 1. Regarding the mutability.
>
> I think it would be a big convenience to have headers mutable during
> certain stage in the message life cycle for the use cases you mentioned. I
> agree there is a material benefit especially given that we may have to
> modify the headers for each message.
>
> That said, I also think it is fair to say that in the producer, in order to
> guarantee the correctness of the entire logic, it is necessary that at some
> point we need to make producer record immutable. For example we probably
> don't want to see that users accidentally updated the headers when the
> producer is doing the serialization or compression.
>
> Given that, would it be possible to make Headers to be able to switch from
> mutable to immutable? We have done this for the Batch in the producer. For
> example, initially the headers are mutable, but after it has gone through
> all the interceptors, we can call Headers.close() to make it immutable
> afterwards.
>
The difference is that the batch is an internal class that is not exposed
to users. Can you please explain what happens if a user tries to send the
same ProducerRecord twice? Would an interceptor fail when trying to mutate
the header that is now closed? Or did you have something else in mind?
Thanks,
Ismael
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
So on mutable, and headers just at message level seems we're all agreed then.
On Radai's comments.
1) agreed - kip updated.
2) Now i totally get the nasty code having this would create as noted by your example. And obviously we want an API which means boiler plate code and most common interaction is supported.
not sure what we should do here, so via protocol and as per previous discussion, it was compromised to support multiple values for a key. I believe this was Gwen's request, and compromised accepted.
Now in Guava multimap interface it has "Collection<V> get(K key)", there is no single "V get(K key)" styled method.
org.apache.collections varient is the same.
Two ideas i had to address this would be:
A) add two additional methods something
/**
* Returns the first header for a key if present, else returns null.
*/
Header first(K key)
/**
* Replaces an existing header where the key and value for the old equal an existing header.
*/
replace(Header old, Header new)
B) Change the previous agreement to support multiple values for a key (aka multimap), and if someone wants a collection for their value they should code this in their value byte[].
My personal opinion is I think option B is a lot cleaner and will make for a cleaner interface. But it will mean negating on a previous discussion agreement. Would everyone be happy with that? If not Radai is option A ok with you? Or any other ideas?
Cheers
Mike
________________________________________
From: Jason Gustafson <ja...@confluent.io>
Sent: Tuesday, February 28, 2017 1:38 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
If I understand correctly, the suggestion is to let headers be mutable on
the producer side basically until after they've passed through the
interceptors. That sounds like a reasonable compromise to me.
@Becket
3. It might be useful to have headers at MessageSet level as well so we can
> avoid decompression in some cases. But given this KIP is already
> complicated, I would rather leave this out of the scope and address that
> later when needed, e.g. after having batch level interceptors.
Yeah, I had the same thought. I was considering factoring the map of header
names to generated integer ids into the message set and only using the
integer ids in the individual messages. It's a bit complex though, so I
agree it's probably best left out. I guess for now if users have a lot of
headers, they should just enable compression.
-Jason
On Mon, Feb 27, 2017 at 1:16 PM, radai <ra...@gmail.com> wrote:
> a few comments on the KIP as it is now:
>
> 1. instead of add(Header) + add (Iterable<Header>) i suggest we use add +
> addAll. this is more in line with how java collections work and may
> therefor be more intuitive
>
> 2. common user code dealing with headers will want get("someKey") /
> set("someKey"), or equivalent. code using multiple headers under the same
> key will be rare, and code iterating over all headers would be even rarer
> (probably only for kafka-connect equivalent use cases, really). as the API
> looks right now, the most common and trivial cases will be gnarly:
> get("someKey") -->
> record.headers().headers("someKey").iterator().next().value(). and this is
> before i start talking about how nulls/emptys are handled.
> replace("someKey") -->
> record.headers().remove(record.headers().headers("someKey"));
> record.headers().append(new Header("someKey", value));
>
> this is why i think we should start with get()/set() which are single-value
> map semantics (so set overwrites), then add getAll() (multi-map), append()
> etc on top. make the common case pretty.
>
> On Sun, Feb 26, 2017 at 2:01 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Becket,
> >
> > On 1)
> >
> > Yes truly we wanted mutable headers also. Alas we couldn't think of a
> > solution would address Jason's point around, once a record is sent it
> > shouldn't be possible to mutate it, for cases where you send twice the
> same
> > record.
> >
> > Thank you so much for your solution i think this will work very nicely :)
> >
> > Agreed we only need to do mutable to immutable conversion
> >
> > I think you solution with a ".close()" taken from else where in the kafka
> > protocol where mutability is existent is a great solution, and happy
> middle
> > ground.
> >
> > @Jason you agree, this resolves your concerns if we had mutable headers?
> >
> >
> > On 2)
> > Agreed, this was only added as i couldn't think of a solution to that
> > would address Jason's concern, but really didn't want to force end users
> to
> > constantly write ugly boiler plate code. If we agree on you solution for
> 1,
> > very happy to remove these.
> >
> > On 3)
> > I also would like to keep the scope of this KIP limited to Message
> Headers
> > for now, else we run the risk of not getting even these delivered for
> next
> > release and we're almost now there on getting this KIP to the state
> > everyone is happy. As you note address that later if theres the need.
> >
> >
> > Ill leave it 24hrs and update the kip if no strong objections based on
> > your solution for 1 & 2.
> >
> > Cheers
> > Mike
> >
> > __________ ______________________________
> > From: Becket Qin <be...@gmail.com>
> > Sent: Saturday, February 25, 2017 10:33 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > Hey Michael,
> >
> > Thanks for the KIP. It looks good overall and it looks we only have few
> > things to agree on.
> >
> > 1. Regarding the mutability.
> >
> > I think it would be a big convenience to have headers mutable during
> > certain stage in the message life cycle for the use cases you mentioned.
> I
> > agree there is a material benefit especially given that we may have to
> > modify the headers for each message.
> >
> > That said, I also think it is fair to say that in the producer, in order
> to
> > guarantee the correctness of the entire logic, it is necessary that at
> some
> > point we need to make producer record immutable. For example we probably
> > don't want to see that users accidentally updated the headers when the
> > producer is doing the serialization or compression.
> >
> > Given that, would it be possible to make Headers to be able to switch
> from
> > mutable to immutable? We have done this for the Batch in the producer.
> For
> > example, initially the headers are mutable, but after it has gone through
> > all the interceptors, we can call Headers.close() to make it immutable
> > afterwards.
> >
> > On the consumer side, we can probably always leave the the ConsumerRecord
> > mutable because after we give the messages to the users, Kafka consumer
> > itself does not care about whether the headers are modified or not
> anymore.
> >
> > So far I think we only need to do the mutable to immutable conversion. If
> > there are use case require immutable to mutable conversion, we may need
> > something more than a closable.
> >
> > 2. If we agree on what mentioned above, I think it probably makes sense
> to
> > put the addHeaders()/removeHeaders() methods into Headers class and just
> > leave the headers() method in ProducerRecord and ConsumerRecord.
> >
> > 3. It might be useful to have headers at MessageSet level as well so we
> can
> > avoid decompression in some cases. But given this KIP is already
> > complicated, I would rather leave this out of the scope and address that
> > later when needed, e.g. after having batch level interceptors.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Fri, Feb 24, 2017 at 3:56 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > KIP updated in response to the below comments:
> > >
> > > > 1. Is the intent of `Headers.filter` to include or exclude the
> > > headers
> > > > matching the key? Can you add a javadoc to clarify?
> > > > 2. The KIP mentions that we will introduce V4 of FetchRequest
> > > and V4 of
> > > > ProduceRequest. Can you change this to say that the changes
> > will
> > > > piggyback
> > > > onto V3 of ProduceRequest and V4 of FetchRequest which were
> > > introduced
> > > > in
> > > > KIP-98?
> > >
> > >
> > >
> > >
> > > On 24/02/2017, 23:20, "Michael Pearce" <Mi...@ig.com> wrote:
> > >
> > > We’re trying to make an eco-system for people to be able to use
> > > headers, I think we want to ensure some least common features are
> > supported
> > > and not limited.
> > >
> > >
> > > Some examples we have already.
> > >
> > > On consume interceptors a security interceptor may need to take the
> > > current header, decrypt the data and replace the token with the next
> > token
> > > for the next processing, in case of a single decryption token being one
> > > time use only.
> > >
> > > On produce it could be the interceptors add some values in the
> clear
> > > from the systems that supply them, but later a security header
> > interceptor
> > > needs to encrypt some headers, as such needs to replace the current
> value
> > > with new one.
> > >
> > > I note Radai already requested this in the thread, I assume he has
> > > some use case also. S
> > >
> > > Simple add / remove is a least common feature.
> > >
> > > Rgds,
> > > Mike
> > >
> > >
> > > On 24/02/2017, 23:00, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> > >
> > > Hey Michael,
> > >
> > > I'm not strongly opposed to them; I just don't see a lot of
> > > benefit. One
> > > thing it would be good to understand is why a consumer
> > interceptor
> > > would
> > > need to add headers and why a producer interceptor would need
> to
> > > remove
> > > them. Maybe we only need the common cases?
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Sorry I thought this was the agreed compromise to provide an
> > api
> > > that
> > > > avoid boiler plate in return for immutabilty.
> > > >
> > > > If not then mutability will be needed as a goal is to have a
> > > single clean
> > > > method call to append/remove a header.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > > On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io>
> > > wrote:
> > > >
> > > > Hey Michael,
> > > >
> > > > I didn't actually comment on the new methods for
> > > ProducerRecord and
> > > > ConsumerRecord. If they only save some boilerplate, I'd
> > just
> > > as well
> > > > not
> > > > have them.
> > > >
> > > > Also a couple minor comments:
> > > >
> > > > 1. Is the intent of `Headers.filter` to include or
> exclude
> > > the headers
> > > > matching the key? Can you add a javadoc to clarify?
> > > > 2. The KIP mentions that we will introduce V4 of
> > > FetchRequest and V4 of
> > > > ProduceRequest. Can you change this to say that the
> changes
> > > will
> > > > piggyback
> > > > onto V3 of ProduceRequest and V4 of FetchRequest which
> were
> > > introduced
> > > > in
> > > > KIP-98?
> > > >
> > > > The rest of the KIP looks good to me.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > I’ve added the methods on the ProducerRecord that will
> > > return a new
> > > > > instance of ProducerRecord with modified headers.
> > > > >
> > > > > On 24/02/2017, 19:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > Pattern.compile is expensive, and even if cached
> > > String.equals is
> > > > > faster than matched. also if we end up with an internal
> > > map in
> > > > future for
> > > > > performance it will be easier to be by key.
> > > > >
> > > > > As all that's needed is to get header by key.
> > > > >
> > > > > With like the other arguements of let's implement
> > > simple and
> > > > then we
> > > > > can always add pattern later as well if it's found it's
> > > needed. (As
> > > > noted
> > > > > it's easier to add methods than to take away)
> > > > >
> > > > > Great I'll update kip with extra methods on
> > > producerecord and a
> > > > note
> > > > > that new objects are returned by method calls.
> > > > >
> > > > >
> > > > >
> > > > > Sent using OWA for iPhone
> > > > > ________________________________________
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > Sent: Friday, February 24, 2017 6:51:45 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > > >
> > > > > The APIs in the current KIP look good to me. Just a
> > > couple
> > > > questions:
> > > > > why
> > > > > does filter not return Headers? Also would it be
> > > useful if the
> > > > key is a
> > > > > regex?
> > > > >
> > > > > On the point of immutability.. One option might be
> to
> > > use a
> > > > mutable
> > > > > object
> > > > > only when passing the headers through the
> interceptor
> > > chain. I
> > > > think as
> > > > > long as we resort to mutability only when clear
> > > performance
> > > > results
> > > > > show
> > > > > that it is worthwhile, I am satisfied. As Ismael
> > > noted, for
> > > > common
> > > > > scenarios it is possible to get reasonable
> > performance
> > > with
> > > > immutable
> > > > > objects.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > On 1, How can you guarantee two separate
> > > implemented clients
> > > > would
> > > > > add
> > > > > > the headers in the same order we are not
> specifying
> > > an order
> > > > at the
> > > > > > protocol level (nor should we) with regards to
> > keyA
> > > being
> > > > ordered
> > > > > before
> > > > > > keyB? We shouldn’t be expecting keyA to be always
> > > set before
> > > > keyB.
> > > > > >
> > > > > > On 2, I believe we have changed the naming based
> on
> > > feedback
> > > > from
> > > > > Jason
> > > > > > already, e.g. we don’t have “get” method that
> > > inferred O(1)
> > > > > performance,
> > > > > > like wise nor “put” but we have an “append”
> > > > > >
> > > > > > On 3, in the KafkaProducer, I think we have
> > > mutability
> > > > already, the
> > > > > value
> > > > > > for time is changed if it is null, at the point
> of
> > > send:
> > > > > > “
> > > > > > long timestamp = record.timestamp()
> ==
> > > null ?
> > > > > > time.milliseconds() : record.timestamp();
> > > > > > “
> > > > > >
> > > > > > As such the timestamp is already mutable, so
> what’s
> > > the
> > > > difference
> > > > > here,
> > > > > > we already have some mixed semantics. On
> timestamp.
> > > > > > e.g. currently if I send to records with
> timestamp
> > > not set,
> > > > the wire
> > > > > > binary sent the value for the timestamp would be
> > > different, as
> > > > such
> > > > > we have
> > > > > > mutation for the same record.
> > > > > >
> > > > > > On 4, I think we should not expect not 1 or 2
> > > headers, but
> > > > infact
> > > > > 10’s of
> > > > > > headers. This is the concern on immutable
> headers,
> > > whilst the
> > > > append
> > > > > > self-reference works nicely, what if someone
> needs
> > > to remove a
> > > > > header?
> > > > > >
> > > > > > Trying to get this moving:
> > > > > >
> > > > > > If we really wanted Immutable Headers and
> > > essentially you guys
> > > > wont
> > > > > give
> > > > > > +1 for it without.
> > > > > >
> > > > > > Whats the feeling for adding methods to
> > > ProducerRecord that
> > > > does the
> > > > > > boiler plate code or creating a new
> ProducerRecord
> > > with the
> > > > altered
> > > > > new
> > > > > > headers (appended or removed) inside. E.g.
> > > > > >
> > > > > > ProducerRecord {
> > > > > >
> > > > > >
> > > > > > ProducerRecord append(Iterable<Headers>
> > > headersToAppend){
> > > > > > return new ProducerRecord(key, value,
> > > headers.append(
> > > > > headersToAppend),
> > > > > > ….)
> > > > > > }
> > > > > >
> > > > > > ProducerRecord remove(Iterable<Headers>
> > > headersToAppend){
> > > > > > return new ProducerRecord(key, value,
> > > headers.remove(
> > > > > headersToAppend),
> > > > > > ….)
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > Were the headers methods actually returns new
> > > objects, and the
> > > > > producer
> > > > > > records methods create a new producer record with
> > > all the
> > > > current
> > > > > values,
> > > > > > but with the new modified headers.
> > > > > >
> > > > > > Then interceptors / code return this new object?
> > > > > >
> > > > > >
> > > > > > Cheers
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 24/02/2017, 16:02, "ismaelj@gmail.com on
> behalf
> > > of Ismael
> > > > Juma" <
> > > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk
> >
> > > wrote:
> > > > > >
> > > > > > Hi Michael,
> > > > > >
> > > > > > Did you mean that you were happy to
> compromise
> > > to keep it
> > > > > mutable or
> > > > > > immutable? You wrote the former, but it
> sounded
> > > from the
> > > > > sentence that
> > > > > > it
> > > > > > could have been a typo. So, my thoughts on
> this
> > > is that
> > > > there
> > > > > are a few
> > > > > > things to take into account:
> > > > > >
> > > > > > 1. Semantics
> > > > > > 2. Simplicity of use (the common operations
> > > should be easy
> > > > to do)
> > > > > > 3. If it's easy to reason about and safe
> > > (immutability
> > > > helps
> > > > > with this)
> > > > > > 4. Efficiency (both memory and CPU usage)
> > > > > >
> > > > > > Regarding 1, I think it would be good to be
> > very
> > > clear
> > > > about the
> > > > > > guarantees
> > > > > > that we are providing. It seems that we are
> > > saying that
> > > > keys are
> > > > > > unordered,
> > > > > > but what about the case where there are
> > multiple
> > > values
> > > > for the
> > > > > same
> > > > > > key?
> > > > > > It seems that for some use cases (e.g.
> > lineage),
> > > it may be
> > > > > useful to
> > > > > > add
> > > > > > values to the same key while preserving the
> > > order.
> > > > > >
> > > > > > Regarding 2, I agree that it's useful to have
> > > methods in
> > > > > `Headers` for
> > > > > > the
> > > > > > very common use cases although we have to be
> > > careful with
> > > > the
> > > > > naming to
> > > > > > avoid giving the wrong impression. Also, when
> > it
> > > comes to
> > > > > `Map.Entry`,
> > > > > > I
> > > > > > think I'd prefer a `toMap` method that simply
> > > gives the
> > > > user a
> > > > > Map if
> > > > > > that's what they want (it makes it clear that
> > > there's a
> > > > > conversion
> > > > > > happening).
> > > > > >
> > > > > > Regarding 3, the concern I have if we make
> the
> > > headers
> > > > mutable
> > > > > is that
> > > > > > it
> > > > > > seems to introduce some inconsistencies and
> > > potential edge
> > > > > cases. At
> > > > > > the
> > > > > > moment, it's safe to keep a ProducerRecord
> and
> > > resend it,
> > > > for
> > > > > example.
> > > > > > If
> > > > > > the record is mutated by an interceptor, then
> > > this can
> > > > lead to
> > > > > weird
> > > > > > behaviour. Also, it seems inconsistent that
> one
> > > has to
> > > > create a
> > > > > new
> > > > > > ProducerRecord to modify the record
> timestamp,
> > > but that
> > > > one has
> > > > > to
> > > > > > mutate
> > > > > > the record to add headers. It seems like one
> > > should either
> > > > > embrace
> > > > > > immutability or mutability, but mixing both
> is
> > > not ideal.
> > > > > >
> > > > > > Regarding 4, for the cases where there are a
> > > small number
> > > > of
> > > > > headers
> > > > > > and/or
> > > > > > 1 or 2 interceptors, it doesn't seem
> difficult
> > > to come up
> > > > with
> > > > > > reasonable
> > > > > > implementations for mutable and immutable
> cases
> > > that do a
> > > > good
> > > > > enough
> > > > > > job.
> > > > > > However, if the number of headers and
> > > interceptors is
> > > > large,
> > > > > then care
> > > > > > is
> > > > > > needed for the immutable case to avoid
> > > unnecessary
> > > > copying. A
> > > > > simple
> > > > > > option
> > > > > > that adds little memory overhead if we have
> > > Header
> > > > instances is
> > > > > to
> > > > > > simply
> > > > > > add a self-reference to the previous Header
> in
> > > the linked
> > > > list.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Feb 23, 2017 at 1:09 AM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Im happy to compromise to keep it mutable
> but
> > > move to an
> > > > append
> > > > > > style api.
> > > > > > > (as in guava interables concat)
> > > > > > >
> > > > > > > class Headers {
> > > > > > > Headers append(Iterable<Header>
> > > headers);
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > I don’t think we’d want prepend, this would
> > > give the
> > > > idea of
> > > > > > guaranteed
> > > > > > > ordering, when in actual fact we don’t
> > provide
> > > that
> > > > guarantee
> > > > > (.e.g
> > > > > > one
> > > > > > > client can put headerA, then headerB, but
> > > another could
> > > > put
> > > > > headerB
> > > > > > then
> > > > > > > headerA, this shouldn’t cause issues), Also
> > > what if we
> > > > changed
> > > > > to a
> > > > > > hashmap
> > > > > > > for the internal implementation, its just a
> > > bucket of
> > > > entries
> > > > > no
> > > > > > ordering.
> > > > > > > I think we just need to provide an api to
> > > add/append
> > > > headers.
> > > > > > >
> > > > > > > This ok? If so ill update KIP to record
> this.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Mike
> > > > > > >
> > > > > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > The point about usability is fair. It's
> > > also
> > > > reasonable to
> > > > > > expect that
> > > > > > > common use cases such as appending
> > headers
> > > should be
> > > > done
> > > > > > efficiently.
> > > > > > >
> > > > > > > Perhaps we could compromise with
> > something
> > > like this?
> > > > > > >
> > > > > > > class Headers {
> > > > > > > Headers append(Iterable<Header>
> > headers);
> > > > > > > Headers prepend(Iterable<Header>
> > headers);
> > > > > > > }
> > > > > > >
> > > > > > > That retains ease of use while still
> > giving
> > > > ourselves some
> > > > > > flexibility
> > > > > > > in
> > > > > > > the implementation.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 3:03 PM,
> Michael
> > > Pearce <
> > > > > > Michael.Pearce@ig.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I wasn’t referring to the headers
> > > needing to be
> > > > copied,
> > > > > im
> > > > > > meaning
> > > > > > > the
> > > > > > > > fact we’d be forcing a new producer
> > > record to be
> > > > > created, with
> > > > > > all
> > > > > > > the
> > > > > > > > contents copied.
> > > > > > > >
> > > > > > > > i.e what will happen is utility
> method
> > > will be
> > > > created
> > > > > or end
> > > > > > up
> > > > > > > being
> > > > > > > > used, which does this, and returns
> the
> > > new
> > > > ProducerRecord
> > > > > > instance.
> > > > > > > >
> > > > > > > > ProducerRecord
> > addHeader(ProducerRecord
> > > record,
> > > > Header
> > > > > > header){
> > > > > > > > Return New ProducerRecord(record.key,
> > > record.value,
> > > > > > > record.timestamp…..,
> > > > > > > > record.headers.concat(header))
> > > > > > > > }
> > > > > > > >
> > > > > > > > To me this seems ugly, but will be
> > > inevitable if we
> > > > > don’t make
> > > > > > adding
> > > > > > > > headers to existing records a simple
> > > clean method
> > > > call.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 22/02/2017, 22:57, "Michael
> Pearce"
> > <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Lazy init can achieve/avoid that.
> > > > > > > >
> > > > > > > > Re the concat, why don’t we
> > > implement that
> > > > inside the
> > > > > > Headers
> > > > > > > rather
> > > > > > > > than causing everyone to implement
> this
> > > as adding
> > > > > headers in
> > > > > > > interceptors
> > > > > > > > will be a dominant use case. We want
> a
> > > user
> > > > friendly API.
> > > > > > Having as
> > > > > > > a user
> > > > > > > > having to code this instead of having
> > > the headers
> > > > handle
> > > > > this
> > > > > > for me
> > > > > > > seems
> > > > > > > > redundant.
> > > > > > > >
> > > > > > > > On 22/02/2017, 22:34, "Jason
> > > Gustafson" <
> > > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I thought the argument was
> > > against
> > > > creating the
> > > > > extra
> > > > > > objects
> > > > > > > > unnecessarily
> > > > > > > > (i.e. if they were not
> > > accessed). And note
> > > > that
> > > > > making
> > > > > > the
> > > > > > > Headers
> > > > > > > > immutable doesn't necessarily
> > > mean that
> > > > they
> > > > > need to be
> > > > > > > copied:
> > > > > > > > you can do
> > > > > > > > a trick like Guava's
> > > Iterables.concat to
> > > > add
> > > > > additional
> > > > > > > headers
> > > > > > > > without
> > > > > > > > changing the underlying
> > > collections.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Wed, Feb 22, 2017 at 2:22
> > PM,
> > > Michael
> > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > If the argument for not
> > having
> > > a map
> > > > holding
> > > > > the
> > > > > > key, value
> > > > > > > > pairs is due
> > > > > > > > > to garbage creation of
> > HashMap
> > > entry's,
> > > > > forcing the
> > > > > > > creation of
> > > > > > > > a whole new
> > > > > > > > > producer record to simply
> add
> > > a head,
> > > > surely is
> > > > > > creating
> > > > > > > a-lot
> > > > > > > > more?
> > > > > > > > >
> > ______________________________
> > > __________
> > > > > > > > > From: Jason Gustafson <
> > > > jason@confluent.io>
> > > > > > > > > Sent: Wednesday, February
> 22,
> > > 2017 10:09
> > > > PM
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Subject: Re: [DISCUSS]
> KIP-82
> > > - Add
> > > > Record
> > > > > Headers
> > > > > > > > >
> > > > > > > > > The current producer
> > > interceptor API is
> > > > this:
> > > > > > > > >
> > > > > > > > > ProducerRecord<K, V>
> > > > onSend(ProducerRecord<K,
> > > > > V>
> > > > > > record);
> > > > > > > > >
> > > > > > > > > So adding a header means
> > > creating a new
> > > > > > ProducerRecord
> > > > > > > with a
> > > > > > > > new header
> > > > > > > > > added to the current
> headers
> > > and
> > > > returning it.
> > > > > Would
> > > > > > that
> > > > > > > not
> > > > > > > > work?
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > > On Wed, Feb 22, 2017 at
> 1:45
> > > PM, Michael
> > > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > So how would you have
> this
> > > work if not
> > > > > mutable
> > > > > > where
> > > > > > > > interceptors would
> > > > > > > > > > add headers?
> > > > > > > > > >
> > > > > > > > > > Sent using OWA for iPhone
> > > > > > > > > >
> > > ______________________________
> > > > __________
> > > > > > > > > > From: Jason Gustafson <
> > > > jason@confluent.io>
> > > > > > > > > > Sent: Wednesday, February
> > > 22, 2017
> > > > 8:42:27 PM
> > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > Subject: Re: [DISCUSS]
> > > KIP-82 - Add
> > > > Record
> > > > > Headers
> > > > > > > > > >
> > > > > > > > > > I think the point on the
> > > mutability of
> > > > > Headers is
> > > > > > worth
> > > > > > > > discussing a
> > > > > > > > > little
> > > > > > > > > > more. As far as I can
> tell,
> > > once the
> > > > > > ProducerRecord (or
> > > > > > > > ConsumerRecord)
> > > > > > > > > is
> > > > > > > > > > constructed, there should
> > be
> > > no need to
> > > > > further
> > > > > > change
> > > > > > > the
> > > > > > > > headers. Is
> > > > > > > > > that
> > > > > > > > > > correct? If so, then why
> > not
> > > enforce
> > > > that
> > > > > that is
> > > > > > the
> > > > > > > case
> > > > > > > > through the
> > > > > > > > > API?
> > > > > > > > > > One problem with
> mutability
> > > it that it
> > > > > constrains
> > > > > > the
> > > > > > > > implementation of
> > > > > > > > > > Headers. For example, if
> we
> > > were
> > > > backing
> > > > > with a
> > > > > > byte
> > > > > > > slice,
> > > > > > > > would we
> > > > > > > > > recopy
> > > > > > > > > > the bytes if a header is
> > > added or
> > > > would we
> > > > > > maintain a
> > > > > > > satellite
> > > > > > > > > collection
> > > > > > > > > > of added records. Seems
> not
> > > great
> > > > either
> > > > > way. If we
> > > > > > > really
> > > > > > > > think
> > > > > > > > > mutability
> > > > > > > > > > is needed, perhaps we
> could
> > > add a
> > > > method to
> > > > > > headers to
> > > > > > > convert
> > > > > > > > it to a
> > > > > > > > > > mutable type (e.g. a
> > > Headers.toMap)?
> > > > > > > > > >
> > > > > > > > > > I'm also with Ismael
> about
> > > exposing
> > > > > Headers.get().
> > > > > > I
> > > > > > > thought
> > > > > > > > it might
> > > > > > > > > make
> > > > > > > > > > sense to have a method
> like
> > > this
> > > > instead:
> > > > > > > > > >
> > > > > > > > > > Iterable<Header>
> > > findMatching(Pattern
> > > > > pattern);
> > > > > > > > > >
> > > > > > > > > > This makes the
> (potential)
> > > need to
> > > > scan the
> > > > > headers
> > > > > > > clear in
> > > > > > > > the API. I'd
> > > > > > > > > > also be fine exposing no
> > > getter at
> > > > all. In
> > > > > > general, Ï
> > > > > > > think
> > > > > > > > it's good to
> > > > > > > > > > start with a minimalistic
> > > API and work
> > > > from
> > > > > there
> > > > > > since
> > > > > > > it's
> > > > > > > > always
> > > > > > > > > easier
> > > > > > > > > > to add methods than
> remove
> > > them.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > > On Wed, Feb 22, 2017 at
> > 9:16
> > > AM,
> > > > Michael
> > > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Ismael
> > > > > > > > > > >
> > > > > > > > > > > On point 1,
> > > > > > > > > > >
> > > > > > > > > > > Sure makes sense will
> > > update shortly.
> > > > > > > > > > >
> > > > > > > > > > > On point 2,
> > > > > > > > > > >
> > > > > > > > > > > Setter/getter typical
> to
> > > > > properties/headers api’s
> > > > > > > > traditionally are map
> > > > > > > > > > > styled interfaces and
> > what
> > > I believe
> > > > is
> > > > > most
> > > > > > expected
> > > > > > > styled
> > > > > > > > thus the
> > > > > > > > > > Key,
> > > > > > > > > > > Value setter.
> > > > > > > > > > > Also it would mean
> rather
> > > than an
> > > > > interface, we
> > > > > > would
> > > > > > > be
> > > > > > > > making our
> > > > > > > > > > > internal header impl
> > > object we have
> > > > for the
> > > > > > array,
> > > > > > > exposed.
> > > > > > > > E.g. if we
> > > > > > > > > > had
> > > > > > > > > > > a Map really this would
> > be
> > > Map.Entry
> > > > > interface,
> > > > > > this
> > > > > > > is the
> > > > > > > > same
> > > > > > > > > reasons
> > > > > > > > > > on
> > > > > > > > > > > the map interface I
> > cannot
> > > actually
> > > > make
> > > > > the
> > > > > > > underlying Node
> > > > > > > > object
> > > > > > > > > > that’s
> > > > > > > > > > > the implementation for
> > > HashMap, so
> > > > that
> > > > > > internals can
> > > > > > > be
> > > > > > > > changed.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On point 3,
> > > > > > > > > > >
> > > > > > > > > > > I think it people do
> > > expect it to be
> > > > > performant,
> > > > > > thus
> > > > > > > why
> > > > > > > > originally
> > > > > > > > > > > concern I raised with
> > > using an array
> > > > for
> > > > > to me
> > > > > > is an
> > > > > > > early
> > > > > > > > memory
> > > > > > > > > > > optimisation. I think
> the
> > > user
> > > > experience
> > > > > of
> > > > > > > > properties/headers is on a
> > > > > > > > > > > get/set model. This is
> > why
> > > its
> > > > important
> > > > > we have
> > > > > > > > encapsulated logic
> > > > > > > > > that
> > > > > > > > > > > then allows us to
> change
> > > this to a
> > > > map, if
> > > > > this
> > > > > > > becomes and
> > > > > > > > issue, and
> > > > > > > > > > the
> > > > > > > > > > > memory overhead of
> > hashmap
> > > is less
> > > > so.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 22/02/2017, 15:56, "
> > > > ismaelj@gmail.com
> > > > > on
> > > > > > behalf of
> > > > > > > > Ismael Juma" <
> > > > > > > > > > > ismaelj@gmail.com on
> > > behalf of
> > > > > ismael@juma.me.uk
> > > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Great to see the
> > > progress that
> > > > has been
> > > > > > achieved
> > > > > > > on this
> > > > > > > > one. :) A
> > > > > > > > > > few
> > > > > > > > > > > comments regarding
> > the
> > > APIs (I'm
> > > > still
> > > > > > reviewing
> > > > > > > the
> > > > > > > > message format
> > > > > > > > > > > changes):
> > > > > > > > > > >
> > > > > > > > > > > 1. Nit:
> `getHeaders`
> > in
> > > > > `ProducerRecord` and
> > > > > > > > `ConsumerRecord`
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > named `headers` (we
> > > avoid the
> > > > `get`
> > > > > prefix in
> > > > > > > Kafka)
> > > > > > > > > > >
> > > > > > > > > > > 2. The `Headers`
> > class
> > > is mutable
> > > > > (there's
> > > > > > an `add`
> > > > > > > > method). Does
> > > > > > > > > it
> > > > > > > > > > > need
> > > > > > > > > > > to be? If so, it
> > would
> > > be good to
> > > > > explain
> > > > > > why.
> > > > > > > Related
> > > > > > > > to that, we
> > > > > > > > > > > should
> > > > > > > > > > > also explain the
> > > thinking around
> > > > > > thread-safety. If
> > > > > > > we
> > > > > > > > keep the
> > > > > > > > > `add`
> > > > > > > > > > > method, it may make
> > > sense for it
> > > > to
> > > > > take a
> > > > > > `Header`
> > > > > > > > (that way we
> > > > > > > > > can
> > > > > > > > > > > add
> > > > > > > > > > > things to `Header`
> > > without
> > > > changing the
> > > > > > interface).
> > > > > > > > > > >
> > > > > > > > > > > 3. Do we need the
> > > `Headers.get()`
> > > > > method?
> > > > > > People
> > > > > > > usually
> > > > > > > > assume
> > > > > > > > > that
> > > > > > > > > > > `get`
> > > > > > > > > > > would be efficient,
> > but
> > > > depending on
> > > > > the
> > > > > > > implementation
> > > > > > > > (the
> > > > > > > > > current
> > > > > > > > > > > proposal states
> that
> > > an array
> > > > would be
> > > > > > used), it
> > > > > > > may not
> > > > > > > > be. If we
> > > > > > > > > > > expect
> > > > > > > > > > > the number of
> headers
> > > to be
> > > > small, it
> > > > > doesn't
> > > > > > > matter
> > > > > > > > though.
> > > > > > > > > > >
> > > > > > > > > > > Ismael
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 21,
> 2017
> > > at 6:38 PM,
> > > > > Michael
> > > > > > Pearce <
> > > > > > > > > > Michael.Pearce@ig.com
> > > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jason,
> > > > > > > > > > > >
> > > > > > > > > > > > Have converted
> the
> > > > interface/api
> > > > > bullets
> > > > > > into
> > > > > > > > interface code
> > > > > > > > > > > snippets.
> > > > > > > > > > > >
> > > > > > > > > > > > Agreed
> > > implementation won’t
> > > > take too
> > > > > long.
> > > > > > We
> > > > > > > have
> > > > > > > > early versions
> > > > > > > > > > > already.
> > > > > > > > > > > > Maybe a week
> before
> > > you think
> > > > about
> > > > > > merging I
> > > > > > > would
> > > > > > > > assume it
> > > > > > > > > would
> > > > > > > > > > > be more
> > > > > > > > > > > > stabilised? I was
> > > thinking
> > > > then we
> > > > > could
> > > > > > fork
> > > > > > > from
> > > > > > > > your confluent
> > > > > > > > > > > branch,
> > > > > > > > > > > > making and then
> > > holding KIP-82
> > > > > changes in
> > > > > > a patch
> > > > > > > > file, that we
> > > > > > > > > can
> > > > > > > > > > > then
> > > > > > > > > > > > re-fork from
> apache
> > > once KIP98
> > > > final
> > > > > is
> > > > > > merged,
> > > > > > > and
> > > > > > > > apply patch
> > > > > > > > > > with
> > > > > > > > > > > last
> > > > > > > > > > > > minute changes.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers
> > > > > > > > > > > > Mike
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On 22/02/2017,
> > > 00:56, "Jason
> > > > > Gustafson" <
> > > > > > > > jason@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hey Michael,
> > > > > > > > > > > >
> > > > > > > > > > > > Awesome. I
> have
> > > a minor
> > > > request.
> > > > > The
> > > > > > APIs are
> > > > > > > > currently
> > > > > > > > > > > documented as a
> > > > > > > > > > > > wiki list.
> > Would
> > > you mind
> > > > adding
> > > > > a code
> > > > > > > snippet
> > > > > > > > instead?
> > > > > > > > > It's a
> > > > > > > > > > > bit
> > > > > > > > > > > > easier
> > > > > > > > > > > > to process.
> > > > > > > > > > > >
> > > > > > > > > > > > How will be
> > best
> > > to manage
> > > > this,
> > > > > as we
> > > > > > will
> > > > > > > > obviously build
> > > > > > > > > off
> > > > > > > > > > > your
> > > > > > > > > > > > KIP’s
> > > > > > > > > > > > > protocol
> > > changes, to
> > > > avoid a
> > > > > merge
> > > > > > hell,
> > > > > > > should
> > > > > > > > we branch
> > > > > > > > > > from
> > > > > > > > > > > your
> > > > > > > > > > > > branch
> > > > > > > > > > > > > in the
> > > confluent repo or
> > > > is it
> > > > > worth
> > > > > > > having a
> > > > > > > > KIP-98
> > > > > > > > > special
> > > > > > > > > > > branch
> > > > > > > > > > > > in the
> > > > > > > > > > > > > apache git,
> > > that we can
> > > > > branch/fork
> > > > > > from?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I was
> thinking
> > > about this
> > > > also.
> > > > > > Ideally we'd
> > > > > > > like
> > > > > > > > to get the
> > > > > > > > > > > changes
> > > > > > > > > > > > in as
> > > > > > > > > > > > close
> together
> > > as possible
> > > > since
> > > > > we
> > > > > > only
> > > > > > > want one
> > > > > > > > magic bump
> > > > > > > > > > and
> > > > > > > > > > > some
> > > > > > > > > > > > users
> > > > > > > > > > > > deploy trunk.
> > > The level of
> > > > > effort to
> > > > > > change
> > > > > > > the
> > > > > > > > format for
> > > > > > > > > > > headers
> > > > > > > > > > > > seems
> > > > > > > > > > > > not too high.
> > Do
> > > you
> > > > agree? My
> > > > > guess
> > > > > > is that
> > > > > > > the
> > > > > > > > KIP-98
> > > > > > > > > message
> > > > > > > > > > > format
> > > > > > > > > > > > patch will
> take
> > > 2-3 weeks
> > > > to
> > > > > review
> > > > > > before we
> > > > > > > > merge to trunk,
> > > > > > > > > > so
> > > > > > > > > > > you
> > > > > > > > > > > > could
> > > > > > > > > > > > hold off
> > > implementing
> > > > until that
> > > > > patch
> > > > > > has
> > > > > > > somewhat
> > > > > > > > > stabilized.
> > > > > > > > > > > That
> > > > > > > > > > > > would
> > > > > > > > > > > > save some
> > > potential rebase
> > > > pain.
> > > > > > > > > > > >
> > > > > > > > > > > > -Jason
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The information
> > > contained in
> > > > this
> > > > > email is
> > > > > > > strictly
> > > > > > > > confidential
> > > > > > > > > > and
> > > > > > > > > > > for
> > > > > > > > > > > > the use of the
> > > addressee only,
> > > > unless
> > > > > > otherwise
> > > > > > > > indicated. If you
> > > > > > > > > > > are not
> > > > > > > > > > > > the intended
> > > recipient, please
> > > > do
> > > > > not read,
> > > > > > > copy, use
> > > > > > > > or disclose
> > > > > > > > > > to
> > > > > > > > > > > others
> > > > > > > > > > > > this message or
> any
> > > attachment.
> > > > > Please also
> > > > > > > notify the
> > > > > > > > sender by
> > > > > > > > > > > replying
> > > > > > > > > > > > to this email or
> by
> > > telephone
> > > > > (+44(020
> > > > > > 7896 0011)
> > > > > > > and
> > > > > > > > then delete
> > > > > > > > > > the
> > > > > > > > > > > > email and any
> > copies
> > > of it.
> > > > Opinions,
> > > > > > conclusion
> > > > > > > (etc)
> > > > > > > > that do
> > > > > > > > > not
> > > > > > > > > > > relate
> > > > > > > > > > > > to the official
> > > business of
> > > > this
> > > > > company
> > > > > > shall be
> > > > > > > > understood as
> > > > > > > > > > > neither
> > > > > > > > > > > > given nor
> endorsed
> > > by it. IG
> > > > is a
> > > > > trading
> > > > > > name
> > > > > > > of IG
> > > > > > > > Markets
> > > > > > > > > > Limited
> > > > > > > > > > > (a
> > > > > > > > > > > > company
> registered
> > > in England
> > > > and
> > > > > Wales,
> > > > > > company
> > > > > > > > number 04008957)
> > > > > > > > > > > and IG
> > > > > > > > > > > > Index Limited (a
> > > company
> > > > registered
> > > > > in
> > > > > > England
> > > > > > > and
> > > > > > > > Wales, company
> > > > > > > > > > > number
> > > > > > > > > > > > 01190902).
> > > Registered address
> > > > at
> > > > > Cannon
> > > > > > Bridge
> > > > > > > House,
> > > > > > > > 25 Dowgate
> > > > > > > > > > > Hill,
> > > > > > > > > > > > London EC4R 2YA.
> > > Both IG
> > > > Markets
> > > > > Limited
> > > > > > > (register
> > > > > > > > number 195355)
> > > > > > > > > > > and IG
> > > > > > > > > > > > Index Limited
> > > (register number
> > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated
> > > > > > > > > > > by the
> > > > > > > > > > > > Financial Conduct
> > > Authority.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The information
> contained
> > > in this
> > > > email is
> > > > > > strictly
> > > > > > > > confidential and
> > > > > > > > > for
> > > > > > > > > > > the use of the
> addressee
> > > only, unless
> > > > > otherwise
> > > > > > > indicated.
> > > > > > > > If you are
> > > > > > > > > not
> > > > > > > > > > > the intended recipient,
> > > please do
> > > > not read,
> > > > > > copy, use
> > > > > > > or
> > > > > > > > disclose to
> > > > > > > > > > others
> > > > > > > > > > > this message or any
> > > attachment.
> > > > Please also
> > > > > > notify the
> > > > > > > > sender by
> > > > > > > > > replying
> > > > > > > > > > > to this email or by
> > > telephone
> > > > (+44(020 7896
> > > > > > 0011) and
> > > > > > > then
> > > > > > > > delete the
> > > > > > > > > > email
> > > > > > > > > > > and any copies of it.
> > > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that
> > > > > > > do not
> > > > > > > > relate to
> > > > > > > > > > the
> > > > > > > > > > > official business of
> this
> > > company
> > > > shall be
> > > > > > understood
> > > > > > > as
> > > > > > > > neither given
> > > > > > > > > > nor
> > > > > > > > > > > endorsed by it. IG is a
> > > trading name
> > > > of IG
> > > > > > Markets
> > > > > > > Limited
> > > > > > > > (a company
> > > > > > > > > > > registered in England
> and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > 04008957)
> > > > > > > > and IG Index
> > > > > > > > > > > Limited (a company
> > > registered in
> > > > England
> > > > > and
> > > > > > Wales,
> > > > > > > company
> > > > > > > > number
> > > > > > > > > > > 01190902). Registered
> > > address at
> > > > Cannon
> > > > > Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > > Dowgate Hill,
> > > > > > > > > > > London EC4R 2YA. Both
> IG
> > > Markets
> > > > Limited
> > > > > > (register
> > > > > > > number
> > > > > > > > 195355) and
> > > > > > > > > IG
> > > > > > > > > > > Index Limited (register
> > > number
> > > > 114059) are
> > > > > > authorised
> > > > > > > and
> > > > > > > > regulated by
> > > > > > > > > > the
> > > > > > > > > > > Financial Conduct
> > > Authority.
> > > > > > > > > > >
> > > > > > > > > > The information contained
> > in
> > > this
> > > > email is
> > > > > strictly
> > > > > > > > confidential and for
> > > > > > > > > > the use of the addressee
> > > only, unless
> > > > > otherwise
> > > > > > > indicated. If
> > > > > > > > you are not
> > > > > > > > > > the intended recipient,
> > > please do not
> > > > read,
> > > > > copy,
> > > > > > use or
> > > > > > > > disclose to
> > > > > > > > > others
> > > > > > > > > > this message or any
> > > attachment. Please
> > > > also
> > > > > notify
> > > > > > the
> > > > > > > sender
> > > > > > > > by replying
> > > > > > > > > > to this email or by
> > > telephone (+44(020
> > > > 7896
> > > > > 0011)
> > > > > > and
> > > > > > > then
> > > > > > > > delete the
> > > > > > > > > email
> > > > > > > > > > and any copies of it.
> > > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that do
> > > > > > > not
> > > > > > > > relate to
> > > > > > > > > the
> > > > > > > > > > official business of this
> > > company
> > > > shall be
> > > > > > understood as
> > > > > > > > neither given
> > > > > > > > > nor
> > > > > > > > > > endorsed by it. IG is a
> > > trading name
> > > > of IG
> > > > > Markets
> > > > > > > Limited (a
> > > > > > > > company
> > > > > > > > > > registered in England and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > 04008957) and
> > > > > > > > IG Index
> > > > > > > > > > Limited (a company
> > > registered in
> > > > England and
> > > > > Wales,
> > > > > > > company
> > > > > > > > number
> > > > > > > > > > 01190902). Registered
> > > address at Cannon
> > > > > Bridge
> > > > > > House, 25
> > > > > > > > Dowgate Hill,
> > > > > > > > > > London EC4R 2YA. Both IG
> > > Markets
> > > > Limited
> > > > > (register
> > > > > > number
> > > > > > > > 195355) and IG
> > > > > > > > > > Index Limited (register
> > > number 114059)
> > > > are
> > > > > > authorised and
> > > > > > > > regulated by
> > > > > > > > > the
> > > > > > > > > > Financial Conduct
> > Authority.
> > > > > > > > > >
> > > > > > > > > The information contained
> in
> > > this email
> > > > is
> > > > > strictly
> > > > > > > confidential
> > > > > > > > and for
> > > > > > > > > the use of the addressee
> > only,
> > > unless
> > > > otherwise
> > > > > > indicated.
> > > > > > > If
> > > > > > > > you are not
> > > > > > > > > the intended recipient,
> > please
> > > do not
> > > > read,
> > > > > copy,
> > > > > > use or
> > > > > > > > disclose to others
> > > > > > > > > this message or any
> > > attachment. Please
> > > > also
> > > > > notify
> > > > > > the
> > > > > > > sender by
> > > > > > > > replying
> > > > > > > > > to this email or by
> telephone
> > > (+44(020
> > > > 7896
> > > > > 0011)
> > > > > > and then
> > > > > > > > delete the email
> > > > > > > > > and any copies of it.
> > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that do
> > > > > > > not
> > > > > > > > relate to the
> > > > > > > > > official business of this
> > > company shall
> > > > be
> > > > > > understood as
> > > > > > > neither
> > > > > > > > given nor
> > > > > > > > > endorsed by it. IG is a
> > > trading name of
> > > > IG
> > > > > Markets
> > > > > > Limited
> > > > > > > (a
> > > > > > > > company
> > > > > > > > > registered in England and
> > > Wales, company
> > > > number
> > > > > > 04008957)
> > > > > > > and IG
> > > > > > > > Index
> > > > > > > > > Limited (a company
> registered
> > > in England
> > > > and
> > > > > Wales,
> > > > > > company
> > > > > > > > number
> > > > > > > > > 01190902). Registered
> address
> > > at Cannon
> > > > Bridge
> > > > > > House, 25
> > > > > > > Dowgate
> > > > > > > > Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> > > Markets Limited
> > > > > (register
> > > > > > number
> > > > > > > > 195355) and IG
> > > > > > > > > Index Limited (register
> > number
> > > 114059)
> > > > are
> > > > > > authorised and
> > > > > > > > regulated by the
> > > > > > > > > Financial Conduct
> Authority.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this
> email
> > > is strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > > the use of the addressee only, unless
> > > otherwise
> > > > > indicated. If
> > > > > > you
> > > > > > > are not
> > > > > > > > the intended recipient, please do not
> > > read, copy,
> > > > use or
> > > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please
> > > also notify
> > > > the
> > > > > sender
> > > > > > by
> > > > > > > replying
> > > > > > > > to this email or by telephone
> (+44(020
> > > 7896 0011)
> > > > and
> > > > > then
> > > > > > delete
> > > > > > > the email
> > > > > > > > and any copies of it. Opinions,
> > > conclusion (etc)
> > > > that do
> > > > > not
> > > > > > relate
> > > > > > > to the
> > > > > > > > official business of this company
> shall
> > > be
> > > > understood as
> > > > > > neither
> > > > > > > given nor
> > > > > > > > endorsed by it. IG is a trading name
> of
> > > IG Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > > registered in England and Wales,
> > company
> > > number
> > > > > 04008957) and
> > > > > > IG
> > > > > > > Index
> > > > > > > > Limited (a company registered in
> > England
> > > and Wales,
> > > > > company
> > > > > > number
> > > > > > > > 01190902). Registered address at
> Cannon
> > > Bridge
> > > > House, 25
> > > > > > Dowgate
> > > > > > > Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> > Limited
> > > (register
> > > > number
> > > > > > 195355)
> > > > > > > and IG
> > > > > > > > Index Limited (register number
> 114059)
> > > are
> > > > authorised and
> > > > > > regulated
> > > > > > > by the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > > strictly
> > > > > confidential and
> > > > > > for
> > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If
> > > > > you
> > > > > > are not
> > > > > > > the intended recipient, please do not read,
> > > copy, use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please also
> > > notify the
> > > > sender
> > > > > by
> > > > > > replying
> > > > > > > to this email or by telephone (+44(020 7896
> > > 0011) and
> > > > then
> > > > > delete
> > > > > > the email
> > > > > > > and any copies of it. Opinions, conclusion
> > > (etc) that do
> > > > not
> > > > > relate
> > > > > > to the
> > > > > > > official business of this company shall be
> > > understood as
> > > > > neither
> > > > > > given nor
> > > > > > > endorsed by it. IG is a trading name of IG
> > > Markets
> > > > Limited (a
> > > > > company
> > > > > > > registered in England and Wales, company
> > number
> > > > 04008957) and
> > > > > IG
> > > > > > Index
> > > > > > > Limited (a company registered in England
> and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > > Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited
> > > (register number
> > > > > 195355)
> > > > > > and IG
> > > > > > > Index Limited (register number 114059) are
> > > authorised and
> > > > > regulated
> > > > > > by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > > > confidential and
> > > > > for
> > > > > > the use of the addressee only, unless otherwise
> > > indicated. If
> > > > you
> > > > > are not
> > > > > > the intended recipient, please do not read, copy,
> > > use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify
> > > the sender
> > > > by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011)
> > > and then
> > > > delete
> > > > > the email
> > > > > > and any copies of it. Opinions, conclusion (etc)
> > > that do not
> > > > relate
> > > > > to the
> > > > > > official business of this company shall be
> > > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > > registered in England and Wales, company number
> > > 04008957) and
> > > > IG
> > > > > Index
> > > > > > Limited (a company registered in England and
> Wales,
> > > company
> > > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> > > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > > number
> > > > 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are
> > > authorised and
> > > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is strictly
> > > confidential
> > > > and
> > > > > for the use of the addressee only, unless otherwise
> > > indicated. If
> > > > you are
> > > > > not the intended recipient, please do not read, copy,
> use
> > > or
> > > > disclose to
> > > > > others this message or any attachment. Please also
> notify
> > > the sender
> > > > by
> > > > > replying to this email or by telephone (+44(020 7896
> > 0011)
> > > and then
> > > > delete
> > > > > the email and any copies of it. Opinions, conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to the official business of this company shall
> be
> > > understood
> > > > as
> > > > > neither given nor endorsed by it. IG is a trading name
> of
> > > IG Markets
> > > > > Limited (a company registered in England and Wales,
> > > company number
> > > > > 04008957) and IG Index Limited (a company registered in
> > > England and
> > > > Wales,
> > > > > company number 01190902). Registered address at Cannon
> > > Bridge House,
> > > > 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > > (register
> > > > number
> > > > > 195355) and IG Index Limited (register number 114059)
> are
> > > authorised
> > > > and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential
> > > and for
> > > > the use of the addressee only, unless otherwise indicated. If
> > > you are not
> > > > the intended recipient, please do not read, copy, use or
> > > disclose to others
> > > > this message or any attachment. Please also notify the sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then
> > > delete the email
> > > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to the
> > > > official business of this company shall be understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > > > registered in England and Wales, company number 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales, company
> > > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > > regulated by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly confidential
> and
> > > for the use of the addressee only, unless otherwise indicated. If you
> are
> > > not the intended recipient, please do not read, copy, use or disclose
> to
> > > others this message or any attachment. Please also notify the sender by
> > > replying to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to the official business of this company shall be understood as
> > > neither given nor endorsed by it. IG is a trading name of IG Markets
> > > Limited (a company registered in England and Wales, company number
> > > 04008957) and IG Index Limited (a company registered in England and
> > Wales,
> > > company number 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and IG Index Limited (register number 114059) are authorised
> and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
If I understand correctly, the suggestion is to let headers be mutable on
the producer side basically until after they've passed through the
interceptors. That sounds like a reasonable compromise to me.
@Becket
3. It might be useful to have headers at MessageSet level as well so we can
> avoid decompression in some cases. But given this KIP is already
> complicated, I would rather leave this out of the scope and address that
> later when needed, e.g. after having batch level interceptors.
Yeah, I had the same thought. I was considering factoring the map of header
names to generated integer ids into the message set and only using the
integer ids in the individual messages. It's a bit complex though, so I
agree it's probably best left out. I guess for now if users have a lot of
headers, they should just enable compression.
-Jason
On Mon, Feb 27, 2017 at 1:16 PM, radai <ra...@gmail.com> wrote:
> a few comments on the KIP as it is now:
>
> 1. instead of add(Header) + add (Iterable<Header>) i suggest we use add +
> addAll. this is more in line with how java collections work and may
> therefor be more intuitive
>
> 2. common user code dealing with headers will want get("someKey") /
> set("someKey"), or equivalent. code using multiple headers under the same
> key will be rare, and code iterating over all headers would be even rarer
> (probably only for kafka-connect equivalent use cases, really). as the API
> looks right now, the most common and trivial cases will be gnarly:
> get("someKey") -->
> record.headers().headers("someKey").iterator().next().value(). and this is
> before i start talking about how nulls/emptys are handled.
> replace("someKey") -->
> record.headers().remove(record.headers().headers("someKey"));
> record.headers().append(new Header("someKey", value));
>
> this is why i think we should start with get()/set() which are single-value
> map semantics (so set overwrites), then add getAll() (multi-map), append()
> etc on top. make the common case pretty.
>
> On Sun, Feb 26, 2017 at 2:01 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Becket,
> >
> > On 1)
> >
> > Yes truly we wanted mutable headers also. Alas we couldn't think of a
> > solution would address Jason's point around, once a record is sent it
> > shouldn't be possible to mutate it, for cases where you send twice the
> same
> > record.
> >
> > Thank you so much for your solution i think this will work very nicely :)
> >
> > Agreed we only need to do mutable to immutable conversion
> >
> > I think you solution with a ".close()" taken from else where in the kafka
> > protocol where mutability is existent is a great solution, and happy
> middle
> > ground.
> >
> > @Jason you agree, this resolves your concerns if we had mutable headers?
> >
> >
> > On 2)
> > Agreed, this was only added as i couldn't think of a solution to that
> > would address Jason's concern, but really didn't want to force end users
> to
> > constantly write ugly boiler plate code. If we agree on you solution for
> 1,
> > very happy to remove these.
> >
> > On 3)
> > I also would like to keep the scope of this KIP limited to Message
> Headers
> > for now, else we run the risk of not getting even these delivered for
> next
> > release and we're almost now there on getting this KIP to the state
> > everyone is happy. As you note address that later if theres the need.
> >
> >
> > Ill leave it 24hrs and update the kip if no strong objections based on
> > your solution for 1 & 2.
> >
> > Cheers
> > Mike
> >
> > __________ ______________________________
> > From: Becket Qin <be...@gmail.com>
> > Sent: Saturday, February 25, 2017 10:33 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > Hey Michael,
> >
> > Thanks for the KIP. It looks good overall and it looks we only have few
> > things to agree on.
> >
> > 1. Regarding the mutability.
> >
> > I think it would be a big convenience to have headers mutable during
> > certain stage in the message life cycle for the use cases you mentioned.
> I
> > agree there is a material benefit especially given that we may have to
> > modify the headers for each message.
> >
> > That said, I also think it is fair to say that in the producer, in order
> to
> > guarantee the correctness of the entire logic, it is necessary that at
> some
> > point we need to make producer record immutable. For example we probably
> > don't want to see that users accidentally updated the headers when the
> > producer is doing the serialization or compression.
> >
> > Given that, would it be possible to make Headers to be able to switch
> from
> > mutable to immutable? We have done this for the Batch in the producer.
> For
> > example, initially the headers are mutable, but after it has gone through
> > all the interceptors, we can call Headers.close() to make it immutable
> > afterwards.
> >
> > On the consumer side, we can probably always leave the the ConsumerRecord
> > mutable because after we give the messages to the users, Kafka consumer
> > itself does not care about whether the headers are modified or not
> anymore.
> >
> > So far I think we only need to do the mutable to immutable conversion. If
> > there are use case require immutable to mutable conversion, we may need
> > something more than a closable.
> >
> > 2. If we agree on what mentioned above, I think it probably makes sense
> to
> > put the addHeaders()/removeHeaders() methods into Headers class and just
> > leave the headers() method in ProducerRecord and ConsumerRecord.
> >
> > 3. It might be useful to have headers at MessageSet level as well so we
> can
> > avoid decompression in some cases. But given this KIP is already
> > complicated, I would rather leave this out of the scope and address that
> > later when needed, e.g. after having batch level interceptors.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Fri, Feb 24, 2017 at 3:56 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > KIP updated in response to the below comments:
> > >
> > > > 1. Is the intent of `Headers.filter` to include or exclude the
> > > headers
> > > > matching the key? Can you add a javadoc to clarify?
> > > > 2. The KIP mentions that we will introduce V4 of FetchRequest
> > > and V4 of
> > > > ProduceRequest. Can you change this to say that the changes
> > will
> > > > piggyback
> > > > onto V3 of ProduceRequest and V4 of FetchRequest which were
> > > introduced
> > > > in
> > > > KIP-98?
> > >
> > >
> > >
> > >
> > > On 24/02/2017, 23:20, "Michael Pearce" <Mi...@ig.com> wrote:
> > >
> > > We’re trying to make an eco-system for people to be able to use
> > > headers, I think we want to ensure some least common features are
> > supported
> > > and not limited.
> > >
> > >
> > > Some examples we have already.
> > >
> > > On consume interceptors a security interceptor may need to take the
> > > current header, decrypt the data and replace the token with the next
> > token
> > > for the next processing, in case of a single decryption token being one
> > > time use only.
> > >
> > > On produce it could be the interceptors add some values in the
> clear
> > > from the systems that supply them, but later a security header
> > interceptor
> > > needs to encrypt some headers, as such needs to replace the current
> value
> > > with new one.
> > >
> > > I note Radai already requested this in the thread, I assume he has
> > > some use case also. S
> > >
> > > Simple add / remove is a least common feature.
> > >
> > > Rgds,
> > > Mike
> > >
> > >
> > > On 24/02/2017, 23:00, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> > >
> > > Hey Michael,
> > >
> > > I'm not strongly opposed to them; I just don't see a lot of
> > > benefit. One
> > > thing it would be good to understand is why a consumer
> > interceptor
> > > would
> > > need to add headers and why a producer interceptor would need
> to
> > > remove
> > > them. Maybe we only need the common cases?
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Sorry I thought this was the agreed compromise to provide an
> > api
> > > that
> > > > avoid boiler plate in return for immutabilty.
> > > >
> > > > If not then mutability will be needed as a goal is to have a
> > > single clean
> > > > method call to append/remove a header.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > > On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io>
> > > wrote:
> > > >
> > > > Hey Michael,
> > > >
> > > > I didn't actually comment on the new methods for
> > > ProducerRecord and
> > > > ConsumerRecord. If they only save some boilerplate, I'd
> > just
> > > as well
> > > > not
> > > > have them.
> > > >
> > > > Also a couple minor comments:
> > > >
> > > > 1. Is the intent of `Headers.filter` to include or
> exclude
> > > the headers
> > > > matching the key? Can you add a javadoc to clarify?
> > > > 2. The KIP mentions that we will introduce V4 of
> > > FetchRequest and V4 of
> > > > ProduceRequest. Can you change this to say that the
> changes
> > > will
> > > > piggyback
> > > > onto V3 of ProduceRequest and V4 of FetchRequest which
> were
> > > introduced
> > > > in
> > > > KIP-98?
> > > >
> > > > The rest of the KIP looks good to me.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > I’ve added the methods on the ProducerRecord that will
> > > return a new
> > > > > instance of ProducerRecord with modified headers.
> > > > >
> > > > > On 24/02/2017, 19:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > Pattern.compile is expensive, and even if cached
> > > String.equals is
> > > > > faster than matched. also if we end up with an internal
> > > map in
> > > > future for
> > > > > performance it will be easier to be by key.
> > > > >
> > > > > As all that's needed is to get header by key.
> > > > >
> > > > > With like the other arguements of let's implement
> > > simple and
> > > > then we
> > > > > can always add pattern later as well if it's found it's
> > > needed. (As
> > > > noted
> > > > > it's easier to add methods than to take away)
> > > > >
> > > > > Great I'll update kip with extra methods on
> > > producerecord and a
> > > > note
> > > > > that new objects are returned by method calls.
> > > > >
> > > > >
> > > > >
> > > > > Sent using OWA for iPhone
> > > > > ________________________________________
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > Sent: Friday, February 24, 2017 6:51:45 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > > >
> > > > > The APIs in the current KIP look good to me. Just a
> > > couple
> > > > questions:
> > > > > why
> > > > > does filter not return Headers? Also would it be
> > > useful if the
> > > > key is a
> > > > > regex?
> > > > >
> > > > > On the point of immutability.. One option might be
> to
> > > use a
> > > > mutable
> > > > > object
> > > > > only when passing the headers through the
> interceptor
> > > chain. I
> > > > think as
> > > > > long as we resort to mutability only when clear
> > > performance
> > > > results
> > > > > show
> > > > > that it is worthwhile, I am satisfied. As Ismael
> > > noted, for
> > > > common
> > > > > scenarios it is possible to get reasonable
> > performance
> > > with
> > > > immutable
> > > > > objects.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > On 1, How can you guarantee two separate
> > > implemented clients
> > > > would
> > > > > add
> > > > > > the headers in the same order we are not
> specifying
> > > an order
> > > > at the
> > > > > > protocol level (nor should we) with regards to
> > keyA
> > > being
> > > > ordered
> > > > > before
> > > > > > keyB? We shouldn’t be expecting keyA to be always
> > > set before
> > > > keyB.
> > > > > >
> > > > > > On 2, I believe we have changed the naming based
> on
> > > feedback
> > > > from
> > > > > Jason
> > > > > > already, e.g. we don’t have “get” method that
> > > inferred O(1)
> > > > > performance,
> > > > > > like wise nor “put” but we have an “append”
> > > > > >
> > > > > > On 3, in the KafkaProducer, I think we have
> > > mutability
> > > > already, the
> > > > > value
> > > > > > for time is changed if it is null, at the point
> of
> > > send:
> > > > > > “
> > > > > > long timestamp = record.timestamp()
> ==
> > > null ?
> > > > > > time.milliseconds() : record.timestamp();
> > > > > > “
> > > > > >
> > > > > > As such the timestamp is already mutable, so
> what’s
> > > the
> > > > difference
> > > > > here,
> > > > > > we already have some mixed semantics. On
> timestamp.
> > > > > > e.g. currently if I send to records with
> timestamp
> > > not set,
> > > > the wire
> > > > > > binary sent the value for the timestamp would be
> > > different, as
> > > > such
> > > > > we have
> > > > > > mutation for the same record.
> > > > > >
> > > > > > On 4, I think we should not expect not 1 or 2
> > > headers, but
> > > > infact
> > > > > 10’s of
> > > > > > headers. This is the concern on immutable
> headers,
> > > whilst the
> > > > append
> > > > > > self-reference works nicely, what if someone
> needs
> > > to remove a
> > > > > header?
> > > > > >
> > > > > > Trying to get this moving:
> > > > > >
> > > > > > If we really wanted Immutable Headers and
> > > essentially you guys
> > > > wont
> > > > > give
> > > > > > +1 for it without.
> > > > > >
> > > > > > Whats the feeling for adding methods to
> > > ProducerRecord that
> > > > does the
> > > > > > boiler plate code or creating a new
> ProducerRecord
> > > with the
> > > > altered
> > > > > new
> > > > > > headers (appended or removed) inside. E.g.
> > > > > >
> > > > > > ProducerRecord {
> > > > > >
> > > > > >
> > > > > > ProducerRecord append(Iterable<Headers>
> > > headersToAppend){
> > > > > > return new ProducerRecord(key, value,
> > > headers.append(
> > > > > headersToAppend),
> > > > > > ….)
> > > > > > }
> > > > > >
> > > > > > ProducerRecord remove(Iterable<Headers>
> > > headersToAppend){
> > > > > > return new ProducerRecord(key, value,
> > > headers.remove(
> > > > > headersToAppend),
> > > > > > ….)
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > Were the headers methods actually returns new
> > > objects, and the
> > > > > producer
> > > > > > records methods create a new producer record with
> > > all the
> > > > current
> > > > > values,
> > > > > > but with the new modified headers.
> > > > > >
> > > > > > Then interceptors / code return this new object?
> > > > > >
> > > > > >
> > > > > > Cheers
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 24/02/2017, 16:02, "ismaelj@gmail.com on
> behalf
> > > of Ismael
> > > > Juma" <
> > > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk
> >
> > > wrote:
> > > > > >
> > > > > > Hi Michael,
> > > > > >
> > > > > > Did you mean that you were happy to
> compromise
> > > to keep it
> > > > > mutable or
> > > > > > immutable? You wrote the former, but it
> sounded
> > > from the
> > > > > sentence that
> > > > > > it
> > > > > > could have been a typo. So, my thoughts on
> this
> > > is that
> > > > there
> > > > > are a few
> > > > > > things to take into account:
> > > > > >
> > > > > > 1. Semantics
> > > > > > 2. Simplicity of use (the common operations
> > > should be easy
> > > > to do)
> > > > > > 3. If it's easy to reason about and safe
> > > (immutability
> > > > helps
> > > > > with this)
> > > > > > 4. Efficiency (both memory and CPU usage)
> > > > > >
> > > > > > Regarding 1, I think it would be good to be
> > very
> > > clear
> > > > about the
> > > > > > guarantees
> > > > > > that we are providing. It seems that we are
> > > saying that
> > > > keys are
> > > > > > unordered,
> > > > > > but what about the case where there are
> > multiple
> > > values
> > > > for the
> > > > > same
> > > > > > key?
> > > > > > It seems that for some use cases (e.g.
> > lineage),
> > > it may be
> > > > > useful to
> > > > > > add
> > > > > > values to the same key while preserving the
> > > order.
> > > > > >
> > > > > > Regarding 2, I agree that it's useful to have
> > > methods in
> > > > > `Headers` for
> > > > > > the
> > > > > > very common use cases although we have to be
> > > careful with
> > > > the
> > > > > naming to
> > > > > > avoid giving the wrong impression. Also, when
> > it
> > > comes to
> > > > > `Map.Entry`,
> > > > > > I
> > > > > > think I'd prefer a `toMap` method that simply
> > > gives the
> > > > user a
> > > > > Map if
> > > > > > that's what they want (it makes it clear that
> > > there's a
> > > > > conversion
> > > > > > happening).
> > > > > >
> > > > > > Regarding 3, the concern I have if we make
> the
> > > headers
> > > > mutable
> > > > > is that
> > > > > > it
> > > > > > seems to introduce some inconsistencies and
> > > potential edge
> > > > > cases. At
> > > > > > the
> > > > > > moment, it's safe to keep a ProducerRecord
> and
> > > resend it,
> > > > for
> > > > > example.
> > > > > > If
> > > > > > the record is mutated by an interceptor, then
> > > this can
> > > > lead to
> > > > > weird
> > > > > > behaviour. Also, it seems inconsistent that
> one
> > > has to
> > > > create a
> > > > > new
> > > > > > ProducerRecord to modify the record
> timestamp,
> > > but that
> > > > one has
> > > > > to
> > > > > > mutate
> > > > > > the record to add headers. It seems like one
> > > should either
> > > > > embrace
> > > > > > immutability or mutability, but mixing both
> is
> > > not ideal.
> > > > > >
> > > > > > Regarding 4, for the cases where there are a
> > > small number
> > > > of
> > > > > headers
> > > > > > and/or
> > > > > > 1 or 2 interceptors, it doesn't seem
> difficult
> > > to come up
> > > > with
> > > > > > reasonable
> > > > > > implementations for mutable and immutable
> cases
> > > that do a
> > > > good
> > > > > enough
> > > > > > job.
> > > > > > However, if the number of headers and
> > > interceptors is
> > > > large,
> > > > > then care
> > > > > > is
> > > > > > needed for the immutable case to avoid
> > > unnecessary
> > > > copying. A
> > > > > simple
> > > > > > option
> > > > > > that adds little memory overhead if we have
> > > Header
> > > > instances is
> > > > > to
> > > > > > simply
> > > > > > add a self-reference to the previous Header
> in
> > > the linked
> > > > list.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Thu, Feb 23, 2017 at 1:09 AM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Im happy to compromise to keep it mutable
> but
> > > move to an
> > > > append
> > > > > > style api.
> > > > > > > (as in guava interables concat)
> > > > > > >
> > > > > > > class Headers {
> > > > > > > Headers append(Iterable<Header>
> > > headers);
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > I don’t think we’d want prepend, this would
> > > give the
> > > > idea of
> > > > > > guaranteed
> > > > > > > ordering, when in actual fact we don’t
> > provide
> > > that
> > > > guarantee
> > > > > (.e.g
> > > > > > one
> > > > > > > client can put headerA, then headerB, but
> > > another could
> > > > put
> > > > > headerB
> > > > > > then
> > > > > > > headerA, this shouldn’t cause issues), Also
> > > what if we
> > > > changed
> > > > > to a
> > > > > > hashmap
> > > > > > > for the internal implementation, its just a
> > > bucket of
> > > > entries
> > > > > no
> > > > > > ordering.
> > > > > > > I think we just need to provide an api to
> > > add/append
> > > > headers.
> > > > > > >
> > > > > > > This ok? If so ill update KIP to record
> this.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Mike
> > > > > > >
> > > > > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > The point about usability is fair. It's
> > > also
> > > > reasonable to
> > > > > > expect that
> > > > > > > common use cases such as appending
> > headers
> > > should be
> > > > done
> > > > > > efficiently.
> > > > > > >
> > > > > > > Perhaps we could compromise with
> > something
> > > like this?
> > > > > > >
> > > > > > > class Headers {
> > > > > > > Headers append(Iterable<Header>
> > headers);
> > > > > > > Headers prepend(Iterable<Header>
> > headers);
> > > > > > > }
> > > > > > >
> > > > > > > That retains ease of use while still
> > giving
> > > > ourselves some
> > > > > > flexibility
> > > > > > > in
> > > > > > > the implementation.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 3:03 PM,
> Michael
> > > Pearce <
> > > > > > Michael.Pearce@ig.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I wasn’t referring to the headers
> > > needing to be
> > > > copied,
> > > > > im
> > > > > > meaning
> > > > > > > the
> > > > > > > > fact we’d be forcing a new producer
> > > record to be
> > > > > created, with
> > > > > > all
> > > > > > > the
> > > > > > > > contents copied.
> > > > > > > >
> > > > > > > > i.e what will happen is utility
> method
> > > will be
> > > > created
> > > > > or end
> > > > > > up
> > > > > > > being
> > > > > > > > used, which does this, and returns
> the
> > > new
> > > > ProducerRecord
> > > > > > instance.
> > > > > > > >
> > > > > > > > ProducerRecord
> > addHeader(ProducerRecord
> > > record,
> > > > Header
> > > > > > header){
> > > > > > > > Return New ProducerRecord(record.key,
> > > record.value,
> > > > > > > record.timestamp…..,
> > > > > > > > record.headers.concat(header))
> > > > > > > > }
> > > > > > > >
> > > > > > > > To me this seems ugly, but will be
> > > inevitable if we
> > > > > don’t make
> > > > > > adding
> > > > > > > > headers to existing records a simple
> > > clean method
> > > > call.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 22/02/2017, 22:57, "Michael
> Pearce"
> > <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Lazy init can achieve/avoid that.
> > > > > > > >
> > > > > > > > Re the concat, why don’t we
> > > implement that
> > > > inside the
> > > > > > Headers
> > > > > > > rather
> > > > > > > > than causing everyone to implement
> this
> > > as adding
> > > > > headers in
> > > > > > > interceptors
> > > > > > > > will be a dominant use case. We want
> a
> > > user
> > > > friendly API.
> > > > > > Having as
> > > > > > > a user
> > > > > > > > having to code this instead of having
> > > the headers
> > > > handle
> > > > > this
> > > > > > for me
> > > > > > > seems
> > > > > > > > redundant.
> > > > > > > >
> > > > > > > > On 22/02/2017, 22:34, "Jason
> > > Gustafson" <
> > > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > I thought the argument was
> > > against
> > > > creating the
> > > > > extra
> > > > > > objects
> > > > > > > > unnecessarily
> > > > > > > > (i.e. if they were not
> > > accessed). And note
> > > > that
> > > > > making
> > > > > > the
> > > > > > > Headers
> > > > > > > > immutable doesn't necessarily
> > > mean that
> > > > they
> > > > > need to be
> > > > > > > copied:
> > > > > > > > you can do
> > > > > > > > a trick like Guava's
> > > Iterables.concat to
> > > > add
> > > > > additional
> > > > > > > headers
> > > > > > > > without
> > > > > > > > changing the underlying
> > > collections.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Wed, Feb 22, 2017 at 2:22
> > PM,
> > > Michael
> > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > If the argument for not
> > having
> > > a map
> > > > holding
> > > > > the
> > > > > > key, value
> > > > > > > > pairs is due
> > > > > > > > > to garbage creation of
> > HashMap
> > > entry's,
> > > > > forcing the
> > > > > > > creation of
> > > > > > > > a whole new
> > > > > > > > > producer record to simply
> add
> > > a head,
> > > > surely is
> > > > > > creating
> > > > > > > a-lot
> > > > > > > > more?
> > > > > > > > >
> > ______________________________
> > > __________
> > > > > > > > > From: Jason Gustafson <
> > > > jason@confluent.io>
> > > > > > > > > Sent: Wednesday, February
> 22,
> > > 2017 10:09
> > > > PM
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Subject: Re: [DISCUSS]
> KIP-82
> > > - Add
> > > > Record
> > > > > Headers
> > > > > > > > >
> > > > > > > > > The current producer
> > > interceptor API is
> > > > this:
> > > > > > > > >
> > > > > > > > > ProducerRecord<K, V>
> > > > onSend(ProducerRecord<K,
> > > > > V>
> > > > > > record);
> > > > > > > > >
> > > > > > > > > So adding a header means
> > > creating a new
> > > > > > ProducerRecord
> > > > > > > with a
> > > > > > > > new header
> > > > > > > > > added to the current
> headers
> > > and
> > > > returning it.
> > > > > Would
> > > > > > that
> > > > > > > not
> > > > > > > > work?
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > > On Wed, Feb 22, 2017 at
> 1:45
> > > PM, Michael
> > > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > So how would you have
> this
> > > work if not
> > > > > mutable
> > > > > > where
> > > > > > > > interceptors would
> > > > > > > > > > add headers?
> > > > > > > > > >
> > > > > > > > > > Sent using OWA for iPhone
> > > > > > > > > >
> > > ______________________________
> > > > __________
> > > > > > > > > > From: Jason Gustafson <
> > > > jason@confluent.io>
> > > > > > > > > > Sent: Wednesday, February
> > > 22, 2017
> > > > 8:42:27 PM
> > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > Subject: Re: [DISCUSS]
> > > KIP-82 - Add
> > > > Record
> > > > > Headers
> > > > > > > > > >
> > > > > > > > > > I think the point on the
> > > mutability of
> > > > > Headers is
> > > > > > worth
> > > > > > > > discussing a
> > > > > > > > > little
> > > > > > > > > > more. As far as I can
> tell,
> > > once the
> > > > > > ProducerRecord (or
> > > > > > > > ConsumerRecord)
> > > > > > > > > is
> > > > > > > > > > constructed, there should
> > be
> > > no need to
> > > > > further
> > > > > > change
> > > > > > > the
> > > > > > > > headers. Is
> > > > > > > > > that
> > > > > > > > > > correct? If so, then why
> > not
> > > enforce
> > > > that
> > > > > that is
> > > > > > the
> > > > > > > case
> > > > > > > > through the
> > > > > > > > > API?
> > > > > > > > > > One problem with
> mutability
> > > it that it
> > > > > constrains
> > > > > > the
> > > > > > > > implementation of
> > > > > > > > > > Headers. For example, if
> we
> > > were
> > > > backing
> > > > > with a
> > > > > > byte
> > > > > > > slice,
> > > > > > > > would we
> > > > > > > > > recopy
> > > > > > > > > > the bytes if a header is
> > > added or
> > > > would we
> > > > > > maintain a
> > > > > > > satellite
> > > > > > > > > collection
> > > > > > > > > > of added records. Seems
> not
> > > great
> > > > either
> > > > > way. If we
> > > > > > > really
> > > > > > > > think
> > > > > > > > > mutability
> > > > > > > > > > is needed, perhaps we
> could
> > > add a
> > > > method to
> > > > > > headers to
> > > > > > > convert
> > > > > > > > it to a
> > > > > > > > > > mutable type (e.g. a
> > > Headers.toMap)?
> > > > > > > > > >
> > > > > > > > > > I'm also with Ismael
> about
> > > exposing
> > > > > Headers.get().
> > > > > > I
> > > > > > > thought
> > > > > > > > it might
> > > > > > > > > make
> > > > > > > > > > sense to have a method
> like
> > > this
> > > > instead:
> > > > > > > > > >
> > > > > > > > > > Iterable<Header>
> > > findMatching(Pattern
> > > > > pattern);
> > > > > > > > > >
> > > > > > > > > > This makes the
> (potential)
> > > need to
> > > > scan the
> > > > > headers
> > > > > > > clear in
> > > > > > > > the API. I'd
> > > > > > > > > > also be fine exposing no
> > > getter at
> > > > all. In
> > > > > > general, Ï
> > > > > > > think
> > > > > > > > it's good to
> > > > > > > > > > start with a minimalistic
> > > API and work
> > > > from
> > > > > there
> > > > > > since
> > > > > > > it's
> > > > > > > > always
> > > > > > > > > easier
> > > > > > > > > > to add methods than
> remove
> > > them.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > > On Wed, Feb 22, 2017 at
> > 9:16
> > > AM,
> > > > Michael
> > > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi Ismael
> > > > > > > > > > >
> > > > > > > > > > > On point 1,
> > > > > > > > > > >
> > > > > > > > > > > Sure makes sense will
> > > update shortly.
> > > > > > > > > > >
> > > > > > > > > > > On point 2,
> > > > > > > > > > >
> > > > > > > > > > > Setter/getter typical
> to
> > > > > properties/headers api’s
> > > > > > > > traditionally are map
> > > > > > > > > > > styled interfaces and
> > what
> > > I believe
> > > > is
> > > > > most
> > > > > > expected
> > > > > > > styled
> > > > > > > > thus the
> > > > > > > > > > Key,
> > > > > > > > > > > Value setter.
> > > > > > > > > > > Also it would mean
> rather
> > > than an
> > > > > interface, we
> > > > > > would
> > > > > > > be
> > > > > > > > making our
> > > > > > > > > > > internal header impl
> > > object we have
> > > > for the
> > > > > > array,
> > > > > > > exposed.
> > > > > > > > E.g. if we
> > > > > > > > > > had
> > > > > > > > > > > a Map really this would
> > be
> > > Map.Entry
> > > > > interface,
> > > > > > this
> > > > > > > is the
> > > > > > > > same
> > > > > > > > > reasons
> > > > > > > > > > on
> > > > > > > > > > > the map interface I
> > cannot
> > > actually
> > > > make
> > > > > the
> > > > > > > underlying Node
> > > > > > > > object
> > > > > > > > > > that’s
> > > > > > > > > > > the implementation for
> > > HashMap, so
> > > > that
> > > > > > internals can
> > > > > > > be
> > > > > > > > changed.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On point 3,
> > > > > > > > > > >
> > > > > > > > > > > I think it people do
> > > expect it to be
> > > > > performant,
> > > > > > thus
> > > > > > > why
> > > > > > > > originally
> > > > > > > > > > > concern I raised with
> > > using an array
> > > > for
> > > > > to me
> > > > > > is an
> > > > > > > early
> > > > > > > > memory
> > > > > > > > > > > optimisation. I think
> the
> > > user
> > > > experience
> > > > > of
> > > > > > > > properties/headers is on a
> > > > > > > > > > > get/set model. This is
> > why
> > > its
> > > > important
> > > > > we have
> > > > > > > > encapsulated logic
> > > > > > > > > that
> > > > > > > > > > > then allows us to
> change
> > > this to a
> > > > map, if
> > > > > this
> > > > > > > becomes and
> > > > > > > > issue, and
> > > > > > > > > > the
> > > > > > > > > > > memory overhead of
> > hashmap
> > > is less
> > > > so.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 22/02/2017, 15:56, "
> > > > ismaelj@gmail.com
> > > > > on
> > > > > > behalf of
> > > > > > > > Ismael Juma" <
> > > > > > > > > > > ismaelj@gmail.com on
> > > behalf of
> > > > > ismael@juma.me.uk
> > > > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Great to see the
> > > progress that
> > > > has been
> > > > > > achieved
> > > > > > > on this
> > > > > > > > one. :) A
> > > > > > > > > > few
> > > > > > > > > > > comments regarding
> > the
> > > APIs (I'm
> > > > still
> > > > > > reviewing
> > > > > > > the
> > > > > > > > message format
> > > > > > > > > > > changes):
> > > > > > > > > > >
> > > > > > > > > > > 1. Nit:
> `getHeaders`
> > in
> > > > > `ProducerRecord` and
> > > > > > > > `ConsumerRecord`
> > > > > > > > > should
> > > > > > > > > > be
> > > > > > > > > > > named `headers` (we
> > > avoid the
> > > > `get`
> > > > > prefix in
> > > > > > > Kafka)
> > > > > > > > > > >
> > > > > > > > > > > 2. The `Headers`
> > class
> > > is mutable
> > > > > (there's
> > > > > > an `add`
> > > > > > > > method). Does
> > > > > > > > > it
> > > > > > > > > > > need
> > > > > > > > > > > to be? If so, it
> > would
> > > be good to
> > > > > explain
> > > > > > why.
> > > > > > > Related
> > > > > > > > to that, we
> > > > > > > > > > > should
> > > > > > > > > > > also explain the
> > > thinking around
> > > > > > thread-safety. If
> > > > > > > we
> > > > > > > > keep the
> > > > > > > > > `add`
> > > > > > > > > > > method, it may make
> > > sense for it
> > > > to
> > > > > take a
> > > > > > `Header`
> > > > > > > > (that way we
> > > > > > > > > can
> > > > > > > > > > > add
> > > > > > > > > > > things to `Header`
> > > without
> > > > changing the
> > > > > > interface).
> > > > > > > > > > >
> > > > > > > > > > > 3. Do we need the
> > > `Headers.get()`
> > > > > method?
> > > > > > People
> > > > > > > usually
> > > > > > > > assume
> > > > > > > > > that
> > > > > > > > > > > `get`
> > > > > > > > > > > would be efficient,
> > but
> > > > depending on
> > > > > the
> > > > > > > implementation
> > > > > > > > (the
> > > > > > > > > current
> > > > > > > > > > > proposal states
> that
> > > an array
> > > > would be
> > > > > > used), it
> > > > > > > may not
> > > > > > > > be. If we
> > > > > > > > > > > expect
> > > > > > > > > > > the number of
> headers
> > > to be
> > > > small, it
> > > > > doesn't
> > > > > > > matter
> > > > > > > > though.
> > > > > > > > > > >
> > > > > > > > > > > Ismael
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Feb 21,
> 2017
> > > at 6:38 PM,
> > > > > Michael
> > > > > > Pearce <
> > > > > > > > > > Michael.Pearce@ig.com
> > > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jason,
> > > > > > > > > > > >
> > > > > > > > > > > > Have converted
> the
> > > > interface/api
> > > > > bullets
> > > > > > into
> > > > > > > > interface code
> > > > > > > > > > > snippets.
> > > > > > > > > > > >
> > > > > > > > > > > > Agreed
> > > implementation won’t
> > > > take too
> > > > > long.
> > > > > > We
> > > > > > > have
> > > > > > > > early versions
> > > > > > > > > > > already.
> > > > > > > > > > > > Maybe a week
> before
> > > you think
> > > > about
> > > > > > merging I
> > > > > > > would
> > > > > > > > assume it
> > > > > > > > > would
> > > > > > > > > > > be more
> > > > > > > > > > > > stabilised? I was
> > > thinking
> > > > then we
> > > > > could
> > > > > > fork
> > > > > > > from
> > > > > > > > your confluent
> > > > > > > > > > > branch,
> > > > > > > > > > > > making and then
> > > holding KIP-82
> > > > > changes in
> > > > > > a patch
> > > > > > > > file, that we
> > > > > > > > > can
> > > > > > > > > > > then
> > > > > > > > > > > > re-fork from
> apache
> > > once KIP98
> > > > final
> > > > > is
> > > > > > merged,
> > > > > > > and
> > > > > > > > apply patch
> > > > > > > > > > with
> > > > > > > > > > > last
> > > > > > > > > > > > minute changes.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers
> > > > > > > > > > > > Mike
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On 22/02/2017,
> > > 00:56, "Jason
> > > > > Gustafson" <
> > > > > > > > jason@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hey Michael,
> > > > > > > > > > > >
> > > > > > > > > > > > Awesome. I
> have
> > > a minor
> > > > request.
> > > > > The
> > > > > > APIs are
> > > > > > > > currently
> > > > > > > > > > > documented as a
> > > > > > > > > > > > wiki list.
> > Would
> > > you mind
> > > > adding
> > > > > a code
> > > > > > > snippet
> > > > > > > > instead?
> > > > > > > > > It's a
> > > > > > > > > > > bit
> > > > > > > > > > > > easier
> > > > > > > > > > > > to process.
> > > > > > > > > > > >
> > > > > > > > > > > > How will be
> > best
> > > to manage
> > > > this,
> > > > > as we
> > > > > > will
> > > > > > > > obviously build
> > > > > > > > > off
> > > > > > > > > > > your
> > > > > > > > > > > > KIP’s
> > > > > > > > > > > > > protocol
> > > changes, to
> > > > avoid a
> > > > > merge
> > > > > > hell,
> > > > > > > should
> > > > > > > > we branch
> > > > > > > > > > from
> > > > > > > > > > > your
> > > > > > > > > > > > branch
> > > > > > > > > > > > > in the
> > > confluent repo or
> > > > is it
> > > > > worth
> > > > > > > having a
> > > > > > > > KIP-98
> > > > > > > > > special
> > > > > > > > > > > branch
> > > > > > > > > > > > in the
> > > > > > > > > > > > > apache git,
> > > that we can
> > > > > branch/fork
> > > > > > from?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I was
> thinking
> > > about this
> > > > also.
> > > > > > Ideally we'd
> > > > > > > like
> > > > > > > > to get the
> > > > > > > > > > > changes
> > > > > > > > > > > > in as
> > > > > > > > > > > > close
> together
> > > as possible
> > > > since
> > > > > we
> > > > > > only
> > > > > > > want one
> > > > > > > > magic bump
> > > > > > > > > > and
> > > > > > > > > > > some
> > > > > > > > > > > > users
> > > > > > > > > > > > deploy trunk.
> > > The level of
> > > > > effort to
> > > > > > change
> > > > > > > the
> > > > > > > > format for
> > > > > > > > > > > headers
> > > > > > > > > > > > seems
> > > > > > > > > > > > not too high.
> > Do
> > > you
> > > > agree? My
> > > > > guess
> > > > > > is that
> > > > > > > the
> > > > > > > > KIP-98
> > > > > > > > > message
> > > > > > > > > > > format
> > > > > > > > > > > > patch will
> take
> > > 2-3 weeks
> > > > to
> > > > > review
> > > > > > before we
> > > > > > > > merge to trunk,
> > > > > > > > > > so
> > > > > > > > > > > you
> > > > > > > > > > > > could
> > > > > > > > > > > > hold off
> > > implementing
> > > > until that
> > > > > patch
> > > > > > has
> > > > > > > somewhat
> > > > > > > > > stabilized.
> > > > > > > > > > > That
> > > > > > > > > > > > would
> > > > > > > > > > > > save some
> > > potential rebase
> > > > pain.
> > > > > > > > > > > >
> > > > > > > > > > > > -Jason
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The information
> > > contained in
> > > > this
> > > > > email is
> > > > > > > strictly
> > > > > > > > confidential
> > > > > > > > > > and
> > > > > > > > > > > for
> > > > > > > > > > > > the use of the
> > > addressee only,
> > > > unless
> > > > > > otherwise
> > > > > > > > indicated. If you
> > > > > > > > > > > are not
> > > > > > > > > > > > the intended
> > > recipient, please
> > > > do
> > > > > not read,
> > > > > > > copy, use
> > > > > > > > or disclose
> > > > > > > > > > to
> > > > > > > > > > > others
> > > > > > > > > > > > this message or
> any
> > > attachment.
> > > > > Please also
> > > > > > > notify the
> > > > > > > > sender by
> > > > > > > > > > > replying
> > > > > > > > > > > > to this email or
> by
> > > telephone
> > > > > (+44(020
> > > > > > 7896 0011)
> > > > > > > and
> > > > > > > > then delete
> > > > > > > > > > the
> > > > > > > > > > > > email and any
> > copies
> > > of it.
> > > > Opinions,
> > > > > > conclusion
> > > > > > > (etc)
> > > > > > > > that do
> > > > > > > > > not
> > > > > > > > > > > relate
> > > > > > > > > > > > to the official
> > > business of
> > > > this
> > > > > company
> > > > > > shall be
> > > > > > > > understood as
> > > > > > > > > > > neither
> > > > > > > > > > > > given nor
> endorsed
> > > by it. IG
> > > > is a
> > > > > trading
> > > > > > name
> > > > > > > of IG
> > > > > > > > Markets
> > > > > > > > > > Limited
> > > > > > > > > > > (a
> > > > > > > > > > > > company
> registered
> > > in England
> > > > and
> > > > > Wales,
> > > > > > company
> > > > > > > > number 04008957)
> > > > > > > > > > > and IG
> > > > > > > > > > > > Index Limited (a
> > > company
> > > > registered
> > > > > in
> > > > > > England
> > > > > > > and
> > > > > > > > Wales, company
> > > > > > > > > > > number
> > > > > > > > > > > > 01190902).
> > > Registered address
> > > > at
> > > > > Cannon
> > > > > > Bridge
> > > > > > > House,
> > > > > > > > 25 Dowgate
> > > > > > > > > > > Hill,
> > > > > > > > > > > > London EC4R 2YA.
> > > Both IG
> > > > Markets
> > > > > Limited
> > > > > > > (register
> > > > > > > > number 195355)
> > > > > > > > > > > and IG
> > > > > > > > > > > > Index Limited
> > > (register number
> > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated
> > > > > > > > > > > by the
> > > > > > > > > > > > Financial Conduct
> > > Authority.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The information
> contained
> > > in this
> > > > email is
> > > > > > strictly
> > > > > > > > confidential and
> > > > > > > > > for
> > > > > > > > > > > the use of the
> addressee
> > > only, unless
> > > > > otherwise
> > > > > > > indicated.
> > > > > > > > If you are
> > > > > > > > > not
> > > > > > > > > > > the intended recipient,
> > > please do
> > > > not read,
> > > > > > copy, use
> > > > > > > or
> > > > > > > > disclose to
> > > > > > > > > > others
> > > > > > > > > > > this message or any
> > > attachment.
> > > > Please also
> > > > > > notify the
> > > > > > > > sender by
> > > > > > > > > replying
> > > > > > > > > > > to this email or by
> > > telephone
> > > > (+44(020 7896
> > > > > > 0011) and
> > > > > > > then
> > > > > > > > delete the
> > > > > > > > > > email
> > > > > > > > > > > and any copies of it.
> > > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that
> > > > > > > do not
> > > > > > > > relate to
> > > > > > > > > > the
> > > > > > > > > > > official business of
> this
> > > company
> > > > shall be
> > > > > > understood
> > > > > > > as
> > > > > > > > neither given
> > > > > > > > > > nor
> > > > > > > > > > > endorsed by it. IG is a
> > > trading name
> > > > of IG
> > > > > > Markets
> > > > > > > Limited
> > > > > > > > (a company
> > > > > > > > > > > registered in England
> and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > 04008957)
> > > > > > > > and IG Index
> > > > > > > > > > > Limited (a company
> > > registered in
> > > > England
> > > > > and
> > > > > > Wales,
> > > > > > > company
> > > > > > > > number
> > > > > > > > > > > 01190902). Registered
> > > address at
> > > > Cannon
> > > > > Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > > Dowgate Hill,
> > > > > > > > > > > London EC4R 2YA. Both
> IG
> > > Markets
> > > > Limited
> > > > > > (register
> > > > > > > number
> > > > > > > > 195355) and
> > > > > > > > > IG
> > > > > > > > > > > Index Limited (register
> > > number
> > > > 114059) are
> > > > > > authorised
> > > > > > > and
> > > > > > > > regulated by
> > > > > > > > > > the
> > > > > > > > > > > Financial Conduct
> > > Authority.
> > > > > > > > > > >
> > > > > > > > > > The information contained
> > in
> > > this
> > > > email is
> > > > > strictly
> > > > > > > > confidential and for
> > > > > > > > > > the use of the addressee
> > > only, unless
> > > > > otherwise
> > > > > > > indicated. If
> > > > > > > > you are not
> > > > > > > > > > the intended recipient,
> > > please do not
> > > > read,
> > > > > copy,
> > > > > > use or
> > > > > > > > disclose to
> > > > > > > > > others
> > > > > > > > > > this message or any
> > > attachment. Please
> > > > also
> > > > > notify
> > > > > > the
> > > > > > > sender
> > > > > > > > by replying
> > > > > > > > > > to this email or by
> > > telephone (+44(020
> > > > 7896
> > > > > 0011)
> > > > > > and
> > > > > > > then
> > > > > > > > delete the
> > > > > > > > > email
> > > > > > > > > > and any copies of it.
> > > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that do
> > > > > > > not
> > > > > > > > relate to
> > > > > > > > > the
> > > > > > > > > > official business of this
> > > company
> > > > shall be
> > > > > > understood as
> > > > > > > > neither given
> > > > > > > > > nor
> > > > > > > > > > endorsed by it. IG is a
> > > trading name
> > > > of IG
> > > > > Markets
> > > > > > > Limited (a
> > > > > > > > company
> > > > > > > > > > registered in England and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > 04008957) and
> > > > > > > > IG Index
> > > > > > > > > > Limited (a company
> > > registered in
> > > > England and
> > > > > Wales,
> > > > > > > company
> > > > > > > > number
> > > > > > > > > > 01190902). Registered
> > > address at Cannon
> > > > > Bridge
> > > > > > House, 25
> > > > > > > > Dowgate Hill,
> > > > > > > > > > London EC4R 2YA. Both IG
> > > Markets
> > > > Limited
> > > > > (register
> > > > > > number
> > > > > > > > 195355) and IG
> > > > > > > > > > Index Limited (register
> > > number 114059)
> > > > are
> > > > > > authorised and
> > > > > > > > regulated by
> > > > > > > > > the
> > > > > > > > > > Financial Conduct
> > Authority.
> > > > > > > > > >
> > > > > > > > > The information contained
> in
> > > this email
> > > > is
> > > > > strictly
> > > > > > > confidential
> > > > > > > > and for
> > > > > > > > > the use of the addressee
> > only,
> > > unless
> > > > otherwise
> > > > > > indicated.
> > > > > > > If
> > > > > > > > you are not
> > > > > > > > > the intended recipient,
> > please
> > > do not
> > > > read,
> > > > > copy,
> > > > > > use or
> > > > > > > > disclose to others
> > > > > > > > > this message or any
> > > attachment. Please
> > > > also
> > > > > notify
> > > > > > the
> > > > > > > sender by
> > > > > > > > replying
> > > > > > > > > to this email or by
> telephone
> > > (+44(020
> > > > 7896
> > > > > 0011)
> > > > > > and then
> > > > > > > > delete the email
> > > > > > > > > and any copies of it.
> > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that do
> > > > > > > not
> > > > > > > > relate to the
> > > > > > > > > official business of this
> > > company shall
> > > > be
> > > > > > understood as
> > > > > > > neither
> > > > > > > > given nor
> > > > > > > > > endorsed by it. IG is a
> > > trading name of
> > > > IG
> > > > > Markets
> > > > > > Limited
> > > > > > > (a
> > > > > > > > company
> > > > > > > > > registered in England and
> > > Wales, company
> > > > number
> > > > > > 04008957)
> > > > > > > and IG
> > > > > > > > Index
> > > > > > > > > Limited (a company
> registered
> > > in England
> > > > and
> > > > > Wales,
> > > > > > company
> > > > > > > > number
> > > > > > > > > 01190902). Registered
> address
> > > at Cannon
> > > > Bridge
> > > > > > House, 25
> > > > > > > Dowgate
> > > > > > > > Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> > > Markets Limited
> > > > > (register
> > > > > > number
> > > > > > > > 195355) and IG
> > > > > > > > > Index Limited (register
> > number
> > > 114059)
> > > > are
> > > > > > authorised and
> > > > > > > > regulated by the
> > > > > > > > > Financial Conduct
> Authority.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this
> email
> > > is strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > > the use of the addressee only, unless
> > > otherwise
> > > > > indicated. If
> > > > > > you
> > > > > > > are not
> > > > > > > > the intended recipient, please do not
> > > read, copy,
> > > > use or
> > > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please
> > > also notify
> > > > the
> > > > > sender
> > > > > > by
> > > > > > > replying
> > > > > > > > to this email or by telephone
> (+44(020
> > > 7896 0011)
> > > > and
> > > > > then
> > > > > > delete
> > > > > > > the email
> > > > > > > > and any copies of it. Opinions,
> > > conclusion (etc)
> > > > that do
> > > > > not
> > > > > > relate
> > > > > > > to the
> > > > > > > > official business of this company
> shall
> > > be
> > > > understood as
> > > > > > neither
> > > > > > > given nor
> > > > > > > > endorsed by it. IG is a trading name
> of
> > > IG Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > > registered in England and Wales,
> > company
> > > number
> > > > > 04008957) and
> > > > > > IG
> > > > > > > Index
> > > > > > > > Limited (a company registered in
> > England
> > > and Wales,
> > > > > company
> > > > > > number
> > > > > > > > 01190902). Registered address at
> Cannon
> > > Bridge
> > > > House, 25
> > > > > > Dowgate
> > > > > > > Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> > Limited
> > > (register
> > > > number
> > > > > > 195355)
> > > > > > > and IG
> > > > > > > > Index Limited (register number
> 114059)
> > > are
> > > > authorised and
> > > > > > regulated
> > > > > > > by the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > > strictly
> > > > > confidential and
> > > > > > for
> > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If
> > > > > you
> > > > > > are not
> > > > > > > the intended recipient, please do not read,
> > > copy, use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please also
> > > notify the
> > > > sender
> > > > > by
> > > > > > replying
> > > > > > > to this email or by telephone (+44(020 7896
> > > 0011) and
> > > > then
> > > > > delete
> > > > > > the email
> > > > > > > and any copies of it. Opinions, conclusion
> > > (etc) that do
> > > > not
> > > > > relate
> > > > > > to the
> > > > > > > official business of this company shall be
> > > understood as
> > > > > neither
> > > > > > given nor
> > > > > > > endorsed by it. IG is a trading name of IG
> > > Markets
> > > > Limited (a
> > > > > company
> > > > > > > registered in England and Wales, company
> > number
> > > > 04008957) and
> > > > > IG
> > > > > > Index
> > > > > > > Limited (a company registered in England
> and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > > Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited
> > > (register number
> > > > > 195355)
> > > > > > and IG
> > > > > > > Index Limited (register number 114059) are
> > > authorised and
> > > > > regulated
> > > > > > by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > > > confidential and
> > > > > for
> > > > > > the use of the addressee only, unless otherwise
> > > indicated. If
> > > > you
> > > > > are not
> > > > > > the intended recipient, please do not read, copy,
> > > use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify
> > > the sender
> > > > by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011)
> > > and then
> > > > delete
> > > > > the email
> > > > > > and any copies of it. Opinions, conclusion (etc)
> > > that do not
> > > > relate
> > > > > to the
> > > > > > official business of this company shall be
> > > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > > registered in England and Wales, company number
> > > 04008957) and
> > > > IG
> > > > > Index
> > > > > > Limited (a company registered in England and
> Wales,
> > > company
> > > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> > > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > > number
> > > > 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are
> > > authorised and
> > > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is strictly
> > > confidential
> > > > and
> > > > > for the use of the addressee only, unless otherwise
> > > indicated. If
> > > > you are
> > > > > not the intended recipient, please do not read, copy,
> use
> > > or
> > > > disclose to
> > > > > others this message or any attachment. Please also
> notify
> > > the sender
> > > > by
> > > > > replying to this email or by telephone (+44(020 7896
> > 0011)
> > > and then
> > > > delete
> > > > > the email and any copies of it. Opinions, conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to the official business of this company shall
> be
> > > understood
> > > > as
> > > > > neither given nor endorsed by it. IG is a trading name
> of
> > > IG Markets
> > > > > Limited (a company registered in England and Wales,
> > > company number
> > > > > 04008957) and IG Index Limited (a company registered in
> > > England and
> > > > Wales,
> > > > > company number 01190902). Registered address at Cannon
> > > Bridge House,
> > > > 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > > (register
> > > > number
> > > > > 195355) and IG Index Limited (register number 114059)
> are
> > > authorised
> > > > and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential
> > > and for
> > > > the use of the addressee only, unless otherwise indicated. If
> > > you are not
> > > > the intended recipient, please do not read, copy, use or
> > > disclose to others
> > > > this message or any attachment. Please also notify the sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then
> > > delete the email
> > > > and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to the
> > > > official business of this company shall be understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > > > registered in England and Wales, company number 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales, company
> > > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > > regulated by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly confidential
> and
> > > for the use of the addressee only, unless otherwise indicated. If you
> are
> > > not the intended recipient, please do not read, copy, use or disclose
> to
> > > others this message or any attachment. Please also notify the sender by
> > > replying to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc) that do not
> > > relate to the official business of this company shall be understood as
> > > neither given nor endorsed by it. IG is a trading name of IG Markets
> > > Limited (a company registered in England and Wales, company number
> > > 04008957) and IG Index Limited (a company registered in England and
> > Wales,
> > > company number 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > > 195355) and IG Index Limited (register number 114059) are authorised
> and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
a few comments on the KIP as it is now:
1. instead of add(Header) + add (Iterable<Header>) i suggest we use add +
addAll. this is more in line with how java collections work and may
therefor be more intuitive
2. common user code dealing with headers will want get("someKey") /
set("someKey"), or equivalent. code using multiple headers under the same
key will be rare, and code iterating over all headers would be even rarer
(probably only for kafka-connect equivalent use cases, really). as the API
looks right now, the most common and trivial cases will be gnarly:
get("someKey") -->
record.headers().headers("someKey").iterator().next().value(). and this is
before i start talking about how nulls/emptys are handled.
replace("someKey") -->
record.headers().remove(record.headers().headers("someKey"));
record.headers().append(new Header("someKey", value));
this is why i think we should start with get()/set() which are single-value
map semantics (so set overwrites), then add getAll() (multi-map), append()
etc on top. make the common case pretty.
On Sun, Feb 26, 2017 at 2:01 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Becket,
>
> On 1)
>
> Yes truly we wanted mutable headers also. Alas we couldn't think of a
> solution would address Jason's point around, once a record is sent it
> shouldn't be possible to mutate it, for cases where you send twice the same
> record.
>
> Thank you so much for your solution i think this will work very nicely :)
>
> Agreed we only need to do mutable to immutable conversion
>
> I think you solution with a ".close()" taken from else where in the kafka
> protocol where mutability is existent is a great solution, and happy middle
> ground.
>
> @Jason you agree, this resolves your concerns if we had mutable headers?
>
>
> On 2)
> Agreed, this was only added as i couldn't think of a solution to that
> would address Jason's concern, but really didn't want to force end users to
> constantly write ugly boiler plate code. If we agree on you solution for 1,
> very happy to remove these.
>
> On 3)
> I also would like to keep the scope of this KIP limited to Message Headers
> for now, else we run the risk of not getting even these delivered for next
> release and we're almost now there on getting this KIP to the state
> everyone is happy. As you note address that later if theres the need.
>
>
> Ill leave it 24hrs and update the kip if no strong objections based on
> your solution for 1 & 2.
>
> Cheers
> Mike
>
> __________ ______________________________
> From: Becket Qin <be...@gmail.com>
> Sent: Saturday, February 25, 2017 10:33 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Hey Michael,
>
> Thanks for the KIP. It looks good overall and it looks we only have few
> things to agree on.
>
> 1. Regarding the mutability.
>
> I think it would be a big convenience to have headers mutable during
> certain stage in the message life cycle for the use cases you mentioned. I
> agree there is a material benefit especially given that we may have to
> modify the headers for each message.
>
> That said, I also think it is fair to say that in the producer, in order to
> guarantee the correctness of the entire logic, it is necessary that at some
> point we need to make producer record immutable. For example we probably
> don't want to see that users accidentally updated the headers when the
> producer is doing the serialization or compression.
>
> Given that, would it be possible to make Headers to be able to switch from
> mutable to immutable? We have done this for the Batch in the producer. For
> example, initially the headers are mutable, but after it has gone through
> all the interceptors, we can call Headers.close() to make it immutable
> afterwards.
>
> On the consumer side, we can probably always leave the the ConsumerRecord
> mutable because after we give the messages to the users, Kafka consumer
> itself does not care about whether the headers are modified or not anymore.
>
> So far I think we only need to do the mutable to immutable conversion. If
> there are use case require immutable to mutable conversion, we may need
> something more than a closable.
>
> 2. If we agree on what mentioned above, I think it probably makes sense to
> put the addHeaders()/removeHeaders() methods into Headers class and just
> leave the headers() method in ProducerRecord and ConsumerRecord.
>
> 3. It might be useful to have headers at MessageSet level as well so we can
> avoid decompression in some cases. But given this KIP is already
> complicated, I would rather leave this out of the scope and address that
> later when needed, e.g. after having batch level interceptors.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Fri, Feb 24, 2017 at 3:56 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > KIP updated in response to the below comments:
> >
> > > 1. Is the intent of `Headers.filter` to include or exclude the
> > headers
> > > matching the key? Can you add a javadoc to clarify?
> > > 2. The KIP mentions that we will introduce V4 of FetchRequest
> > and V4 of
> > > ProduceRequest. Can you change this to say that the changes
> will
> > > piggyback
> > > onto V3 of ProduceRequest and V4 of FetchRequest which were
> > introduced
> > > in
> > > KIP-98?
> >
> >
> >
> >
> > On 24/02/2017, 23:20, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> > We’re trying to make an eco-system for people to be able to use
> > headers, I think we want to ensure some least common features are
> supported
> > and not limited.
> >
> >
> > Some examples we have already.
> >
> > On consume interceptors a security interceptor may need to take the
> > current header, decrypt the data and replace the token with the next
> token
> > for the next processing, in case of a single decryption token being one
> > time use only.
> >
> > On produce it could be the interceptors add some values in the clear
> > from the systems that supply them, but later a security header
> interceptor
> > needs to encrypt some headers, as such needs to replace the current value
> > with new one.
> >
> > I note Radai already requested this in the thread, I assume he has
> > some use case also. S
> >
> > Simple add / remove is a least common feature.
> >
> > Rgds,
> > Mike
> >
> >
> > On 24/02/2017, 23:00, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > Hey Michael,
> >
> > I'm not strongly opposed to them; I just don't see a lot of
> > benefit. One
> > thing it would be good to understand is why a consumer
> interceptor
> > would
> > need to add headers and why a producer interceptor would need to
> > remove
> > them. Maybe we only need the common cases?
> >
> > Thanks,
> > Jason
> >
> > On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Sorry I thought this was the agreed compromise to provide an
> api
> > that
> > > avoid boiler plate in return for immutabilty.
> > >
> > > If not then mutability will be needed as a goal is to have a
> > single clean
> > > method call to append/remove a header.
> > >
> > > Cheers
> > > Mike
> > >
> > > On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > >
> > > Hey Michael,
> > >
> > > I didn't actually comment on the new methods for
> > ProducerRecord and
> > > ConsumerRecord. If they only save some boilerplate, I'd
> just
> > as well
> > > not
> > > have them.
> > >
> > > Also a couple minor comments:
> > >
> > > 1. Is the intent of `Headers.filter` to include or exclude
> > the headers
> > > matching the key? Can you add a javadoc to clarify?
> > > 2. The KIP mentions that we will introduce V4 of
> > FetchRequest and V4 of
> > > ProduceRequest. Can you change this to say that the changes
> > will
> > > piggyback
> > > onto V3 of ProduceRequest and V4 of FetchRequest which were
> > introduced
> > > in
> > > KIP-98?
> > >
> > > The rest of the KIP looks good to me.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > I’ve added the methods on the ProducerRecord that will
> > return a new
> > > > instance of ProducerRecord with modified headers.
> > > >
> > > > On 24/02/2017, 19:22, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > Pattern.compile is expensive, and even if cached
> > String.equals is
> > > > faster than matched. also if we end up with an internal
> > map in
> > > future for
> > > > performance it will be easier to be by key.
> > > >
> > > > As all that's needed is to get header by key.
> > > >
> > > > With like the other arguements of let's implement
> > simple and
> > > then we
> > > > can always add pattern later as well if it's found it's
> > needed. (As
> > > noted
> > > > it's easier to add methods than to take away)
> > > >
> > > > Great I'll update kip with extra methods on
> > producerecord and a
> > > note
> > > > that new objects are returned by method calls.
> > > >
> > > >
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Friday, February 24, 2017 6:51:45 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > The APIs in the current KIP look good to me. Just a
> > couple
> > > questions:
> > > > why
> > > > does filter not return Headers? Also would it be
> > useful if the
> > > key is a
> > > > regex?
> > > >
> > > > On the point of immutability.. One option might be to
> > use a
> > > mutable
> > > > object
> > > > only when passing the headers through the interceptor
> > chain. I
> > > think as
> > > > long as we resort to mutability only when clear
> > performance
> > > results
> > > > show
> > > > that it is worthwhile, I am satisfied. As Ismael
> > noted, for
> > > common
> > > > scenarios it is possible to get reasonable
> performance
> > with
> > > immutable
> > > > objects.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > On 1, How can you guarantee two separate
> > implemented clients
> > > would
> > > > add
> > > > > the headers in the same order we are not specifying
> > an order
> > > at the
> > > > > protocol level (nor should we) with regards to
> keyA
> > being
> > > ordered
> > > > before
> > > > > keyB? We shouldn’t be expecting keyA to be always
> > set before
> > > keyB.
> > > > >
> > > > > On 2, I believe we have changed the naming based on
> > feedback
> > > from
> > > > Jason
> > > > > already, e.g. we don’t have “get” method that
> > inferred O(1)
> > > > performance,
> > > > > like wise nor “put” but we have an “append”
> > > > >
> > > > > On 3, in the KafkaProducer, I think we have
> > mutability
> > > already, the
> > > > value
> > > > > for time is changed if it is null, at the point of
> > send:
> > > > > “
> > > > > long timestamp = record.timestamp() ==
> > null ?
> > > > > time.milliseconds() : record.timestamp();
> > > > > “
> > > > >
> > > > > As such the timestamp is already mutable, so what’s
> > the
> > > difference
> > > > here,
> > > > > we already have some mixed semantics. On timestamp.
> > > > > e.g. currently if I send to records with timestamp
> > not set,
> > > the wire
> > > > > binary sent the value for the timestamp would be
> > different, as
> > > such
> > > > we have
> > > > > mutation for the same record.
> > > > >
> > > > > On 4, I think we should not expect not 1 or 2
> > headers, but
> > > infact
> > > > 10’s of
> > > > > headers. This is the concern on immutable headers,
> > whilst the
> > > append
> > > > > self-reference works nicely, what if someone needs
> > to remove a
> > > > header?
> > > > >
> > > > > Trying to get this moving:
> > > > >
> > > > > If we really wanted Immutable Headers and
> > essentially you guys
> > > wont
> > > > give
> > > > > +1 for it without.
> > > > >
> > > > > Whats the feeling for adding methods to
> > ProducerRecord that
> > > does the
> > > > > boiler plate code or creating a new ProducerRecord
> > with the
> > > altered
> > > > new
> > > > > headers (appended or removed) inside. E.g.
> > > > >
> > > > > ProducerRecord {
> > > > >
> > > > >
> > > > > ProducerRecord append(Iterable<Headers>
> > headersToAppend){
> > > > > return new ProducerRecord(key, value,
> > headers.append(
> > > > headersToAppend),
> > > > > ….)
> > > > > }
> > > > >
> > > > > ProducerRecord remove(Iterable<Headers>
> > headersToAppend){
> > > > > return new ProducerRecord(key, value,
> > headers.remove(
> > > > headersToAppend),
> > > > > ….)
> > > > > }
> > > > >
> > > > > }
> > > > >
> > > > > Were the headers methods actually returns new
> > objects, and the
> > > > producer
> > > > > records methods create a new producer record with
> > all the
> > > current
> > > > values,
> > > > > but with the new modified headers.
> > > > >
> > > > > Then interceptors / code return this new object?
> > > > >
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf
> > of Ismael
> > > Juma" <
> > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk>
> > wrote:
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > > Did you mean that you were happy to compromise
> > to keep it
> > > > mutable or
> > > > > immutable? You wrote the former, but it sounded
> > from the
> > > > sentence that
> > > > > it
> > > > > could have been a typo. So, my thoughts on this
> > is that
> > > there
> > > > are a few
> > > > > things to take into account:
> > > > >
> > > > > 1. Semantics
> > > > > 2. Simplicity of use (the common operations
> > should be easy
> > > to do)
> > > > > 3. If it's easy to reason about and safe
> > (immutability
> > > helps
> > > > with this)
> > > > > 4. Efficiency (both memory and CPU usage)
> > > > >
> > > > > Regarding 1, I think it would be good to be
> very
> > clear
> > > about the
> > > > > guarantees
> > > > > that we are providing. It seems that we are
> > saying that
> > > keys are
> > > > > unordered,
> > > > > but what about the case where there are
> multiple
> > values
> > > for the
> > > > same
> > > > > key?
> > > > > It seems that for some use cases (e.g.
> lineage),
> > it may be
> > > > useful to
> > > > > add
> > > > > values to the same key while preserving the
> > order.
> > > > >
> > > > > Regarding 2, I agree that it's useful to have
> > methods in
> > > > `Headers` for
> > > > > the
> > > > > very common use cases although we have to be
> > careful with
> > > the
> > > > naming to
> > > > > avoid giving the wrong impression. Also, when
> it
> > comes to
> > > > `Map.Entry`,
> > > > > I
> > > > > think I'd prefer a `toMap` method that simply
> > gives the
> > > user a
> > > > Map if
> > > > > that's what they want (it makes it clear that
> > there's a
> > > > conversion
> > > > > happening).
> > > > >
> > > > > Regarding 3, the concern I have if we make the
> > headers
> > > mutable
> > > > is that
> > > > > it
> > > > > seems to introduce some inconsistencies and
> > potential edge
> > > > cases. At
> > > > > the
> > > > > moment, it's safe to keep a ProducerRecord and
> > resend it,
> > > for
> > > > example.
> > > > > If
> > > > > the record is mutated by an interceptor, then
> > this can
> > > lead to
> > > > weird
> > > > > behaviour. Also, it seems inconsistent that one
> > has to
> > > create a
> > > > new
> > > > > ProducerRecord to modify the record timestamp,
> > but that
> > > one has
> > > > to
> > > > > mutate
> > > > > the record to add headers. It seems like one
> > should either
> > > > embrace
> > > > > immutability or mutability, but mixing both is
> > not ideal.
> > > > >
> > > > > Regarding 4, for the cases where there are a
> > small number
> > > of
> > > > headers
> > > > > and/or
> > > > > 1 or 2 interceptors, it doesn't seem difficult
> > to come up
> > > with
> > > > > reasonable
> > > > > implementations for mutable and immutable cases
> > that do a
> > > good
> > > > enough
> > > > > job.
> > > > > However, if the number of headers and
> > interceptors is
> > > large,
> > > > then care
> > > > > is
> > > > > needed for the immutable case to avoid
> > unnecessary
> > > copying. A
> > > > simple
> > > > > option
> > > > > that adds little memory overhead if we have
> > Header
> > > instances is
> > > > to
> > > > > simply
> > > > > add a self-reference to the previous Header in
> > the linked
> > > list.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Feb 23, 2017 at 1:09 AM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Im happy to compromise to keep it mutable but
> > move to an
> > > append
> > > > > style api.
> > > > > > (as in guava interables concat)
> > > > > >
> > > > > > class Headers {
> > > > > > Headers append(Iterable<Header>
> > headers);
> > > > > > }
> > > > > >
> > > > > >
> > > > > > I don’t think we’d want prepend, this would
> > give the
> > > idea of
> > > > > guaranteed
> > > > > > ordering, when in actual fact we don’t
> provide
> > that
> > > guarantee
> > > > (.e.g
> > > > > one
> > > > > > client can put headerA, then headerB, but
> > another could
> > > put
> > > > headerB
> > > > > then
> > > > > > headerA, this shouldn’t cause issues), Also
> > what if we
> > > changed
> > > > to a
> > > > > hashmap
> > > > > > for the internal implementation, its just a
> > bucket of
> > > entries
> > > > no
> > > > > ordering.
> > > > > > I think we just need to provide an api to
> > add/append
> > > headers.
> > > > > >
> > > > > > This ok? If so ill update KIP to record this.
> > > > > >
> > > > > > Cheers
> > > > > > Mike
> > > > > >
> > > > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > The point about usability is fair. It's
> > also
> > > reasonable to
> > > > > expect that
> > > > > > common use cases such as appending
> headers
> > should be
> > > done
> > > > > efficiently.
> > > > > >
> > > > > > Perhaps we could compromise with
> something
> > like this?
> > > > > >
> > > > > > class Headers {
> > > > > > Headers append(Iterable<Header>
> headers);
> > > > > > Headers prepend(Iterable<Header>
> headers);
> > > > > > }
> > > > > >
> > > > > > That retains ease of use while still
> giving
> > > ourselves some
> > > > > flexibility
> > > > > > in
> > > > > > the implementation.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 3:03 PM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I wasn’t referring to the headers
> > needing to be
> > > copied,
> > > > im
> > > > > meaning
> > > > > > the
> > > > > > > fact we’d be forcing a new producer
> > record to be
> > > > created, with
> > > > > all
> > > > > > the
> > > > > > > contents copied.
> > > > > > >
> > > > > > > i.e what will happen is utility method
> > will be
> > > created
> > > > or end
> > > > > up
> > > > > > being
> > > > > > > used, which does this, and returns the
> > new
> > > ProducerRecord
> > > > > instance.
> > > > > > >
> > > > > > > ProducerRecord
> addHeader(ProducerRecord
> > record,
> > > Header
> > > > > header){
> > > > > > > Return New ProducerRecord(record.key,
> > record.value,
> > > > > > record.timestamp…..,
> > > > > > > record.headers.concat(header))
> > > > > > > }
> > > > > > >
> > > > > > > To me this seems ugly, but will be
> > inevitable if we
> > > > don’t make
> > > > > adding
> > > > > > > headers to existing records a simple
> > clean method
> > > call.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 22/02/2017, 22:57, "Michael Pearce"
> <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Lazy init can achieve/avoid that.
> > > > > > >
> > > > > > > Re the concat, why don’t we
> > implement that
> > > inside the
> > > > > Headers
> > > > > > rather
> > > > > > > than causing everyone to implement this
> > as adding
> > > > headers in
> > > > > > interceptors
> > > > > > > will be a dominant use case. We want a
> > user
> > > friendly API.
> > > > > Having as
> > > > > > a user
> > > > > > > having to code this instead of having
> > the headers
> > > handle
> > > > this
> > > > > for me
> > > > > > seems
> > > > > > > redundant.
> > > > > > >
> > > > > > > On 22/02/2017, 22:34, "Jason
> > Gustafson" <
> > > > > jason@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > I thought the argument was
> > against
> > > creating the
> > > > extra
> > > > > objects
> > > > > > > unnecessarily
> > > > > > > (i.e. if they were not
> > accessed). And note
> > > that
> > > > making
> > > > > the
> > > > > > Headers
> > > > > > > immutable doesn't necessarily
> > mean that
> > > they
> > > > need to be
> > > > > > copied:
> > > > > > > you can do
> > > > > > > a trick like Guava's
> > Iterables.concat to
> > > add
> > > > additional
> > > > > > headers
> > > > > > > without
> > > > > > > changing the underlying
> > collections.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 2:22
> PM,
> > Michael
> > > Pearce <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > If the argument for not
> having
> > a map
> > > holding
> > > > the
> > > > > key, value
> > > > > > > pairs is due
> > > > > > > > to garbage creation of
> HashMap
> > entry's,
> > > > forcing the
> > > > > > creation of
> > > > > > > a whole new
> > > > > > > > producer record to simply add
> > a head,
> > > surely is
> > > > > creating
> > > > > > a-lot
> > > > > > > more?
> > > > > > > >
> ______________________________
> > __________
> > > > > > > > From: Jason Gustafson <
> > > jason@confluent.io>
> > > > > > > > Sent: Wednesday, February 22,
> > 2017 10:09
> > > PM
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Subject: Re: [DISCUSS] KIP-82
> > - Add
> > > Record
> > > > Headers
> > > > > > > >
> > > > > > > > The current producer
> > interceptor API is
> > > this:
> > > > > > > >
> > > > > > > > ProducerRecord<K, V>
> > > onSend(ProducerRecord<K,
> > > > V>
> > > > > record);
> > > > > > > >
> > > > > > > > So adding a header means
> > creating a new
> > > > > ProducerRecord
> > > > > > with a
> > > > > > > new header
> > > > > > > > added to the current headers
> > and
> > > returning it.
> > > > Would
> > > > > that
> > > > > > not
> > > > > > > work?
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Wed, Feb 22, 2017 at 1:45
> > PM, Michael
> > > > Pearce <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > So how would you have this
> > work if not
> > > > mutable
> > > > > where
> > > > > > > interceptors would
> > > > > > > > > add headers?
> > > > > > > > >
> > > > > > > > > Sent using OWA for iPhone
> > > > > > > > >
> > ______________________________
> > > __________
> > > > > > > > > From: Jason Gustafson <
> > > jason@confluent.io>
> > > > > > > > > Sent: Wednesday, February
> > 22, 2017
> > > 8:42:27 PM
> > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > Subject: Re: [DISCUSS]
> > KIP-82 - Add
> > > Record
> > > > Headers
> > > > > > > > >
> > > > > > > > > I think the point on the
> > mutability of
> > > > Headers is
> > > > > worth
> > > > > > > discussing a
> > > > > > > > little
> > > > > > > > > more. As far as I can tell,
> > once the
> > > > > ProducerRecord (or
> > > > > > > ConsumerRecord)
> > > > > > > > is
> > > > > > > > > constructed, there should
> be
> > no need to
> > > > further
> > > > > change
> > > > > > the
> > > > > > > headers. Is
> > > > > > > > that
> > > > > > > > > correct? If so, then why
> not
> > enforce
> > > that
> > > > that is
> > > > > the
> > > > > > case
> > > > > > > through the
> > > > > > > > API?
> > > > > > > > > One problem with mutability
> > it that it
> > > > constrains
> > > > > the
> > > > > > > implementation of
> > > > > > > > > Headers. For example, if we
> > were
> > > backing
> > > > with a
> > > > > byte
> > > > > > slice,
> > > > > > > would we
> > > > > > > > recopy
> > > > > > > > > the bytes if a header is
> > added or
> > > would we
> > > > > maintain a
> > > > > > satellite
> > > > > > > > collection
> > > > > > > > > of added records. Seems not
> > great
> > > either
> > > > way. If we
> > > > > > really
> > > > > > > think
> > > > > > > > mutability
> > > > > > > > > is needed, perhaps we could
> > add a
> > > method to
> > > > > headers to
> > > > > > convert
> > > > > > > it to a
> > > > > > > > > mutable type (e.g. a
> > Headers.toMap)?
> > > > > > > > >
> > > > > > > > > I'm also with Ismael about
> > exposing
> > > > Headers.get().
> > > > > I
> > > > > > thought
> > > > > > > it might
> > > > > > > > make
> > > > > > > > > sense to have a method like
> > this
> > > instead:
> > > > > > > > >
> > > > > > > > > Iterable<Header>
> > findMatching(Pattern
> > > > pattern);
> > > > > > > > >
> > > > > > > > > This makes the (potential)
> > need to
> > > scan the
> > > > headers
> > > > > > clear in
> > > > > > > the API. I'd
> > > > > > > > > also be fine exposing no
> > getter at
> > > all. In
> > > > > general, Ï
> > > > > > think
> > > > > > > it's good to
> > > > > > > > > start with a minimalistic
> > API and work
> > > from
> > > > there
> > > > > since
> > > > > > it's
> > > > > > > always
> > > > > > > > easier
> > > > > > > > > to add methods than remove
> > them.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > > On Wed, Feb 22, 2017 at
> 9:16
> > AM,
> > > Michael
> > > > Pearce <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi Ismael
> > > > > > > > > >
> > > > > > > > > > On point 1,
> > > > > > > > > >
> > > > > > > > > > Sure makes sense will
> > update shortly.
> > > > > > > > > >
> > > > > > > > > > On point 2,
> > > > > > > > > >
> > > > > > > > > > Setter/getter typical to
> > > > properties/headers api’s
> > > > > > > traditionally are map
> > > > > > > > > > styled interfaces and
> what
> > I believe
> > > is
> > > > most
> > > > > expected
> > > > > > styled
> > > > > > > thus the
> > > > > > > > > Key,
> > > > > > > > > > Value setter.
> > > > > > > > > > Also it would mean rather
> > than an
> > > > interface, we
> > > > > would
> > > > > > be
> > > > > > > making our
> > > > > > > > > > internal header impl
> > object we have
> > > for the
> > > > > array,
> > > > > > exposed.
> > > > > > > E.g. if we
> > > > > > > > > had
> > > > > > > > > > a Map really this would
> be
> > Map.Entry
> > > > interface,
> > > > > this
> > > > > > is the
> > > > > > > same
> > > > > > > > reasons
> > > > > > > > > on
> > > > > > > > > > the map interface I
> cannot
> > actually
> > > make
> > > > the
> > > > > > underlying Node
> > > > > > > object
> > > > > > > > > that’s
> > > > > > > > > > the implementation for
> > HashMap, so
> > > that
> > > > > internals can
> > > > > > be
> > > > > > > changed.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On point 3,
> > > > > > > > > >
> > > > > > > > > > I think it people do
> > expect it to be
> > > > performant,
> > > > > thus
> > > > > > why
> > > > > > > originally
> > > > > > > > > > concern I raised with
> > using an array
> > > for
> > > > to me
> > > > > is an
> > > > > > early
> > > > > > > memory
> > > > > > > > > > optimisation. I think the
> > user
> > > experience
> > > > of
> > > > > > > properties/headers is on a
> > > > > > > > > > get/set model. This is
> why
> > its
> > > important
> > > > we have
> > > > > > > encapsulated logic
> > > > > > > > that
> > > > > > > > > > then allows us to change
> > this to a
> > > map, if
> > > > this
> > > > > > becomes and
> > > > > > > issue, and
> > > > > > > > > the
> > > > > > > > > > memory overhead of
> hashmap
> > is less
> > > so.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 22/02/2017, 15:56, "
> > > ismaelj@gmail.com
> > > > on
> > > > > behalf of
> > > > > > > Ismael Juma" <
> > > > > > > > > > ismaelj@gmail.com on
> > behalf of
> > > > ismael@juma.me.uk
> > > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Great to see the
> > progress that
> > > has been
> > > > > achieved
> > > > > > on this
> > > > > > > one. :) A
> > > > > > > > > few
> > > > > > > > > > comments regarding
> the
> > APIs (I'm
> > > still
> > > > > reviewing
> > > > > > the
> > > > > > > message format
> > > > > > > > > > changes):
> > > > > > > > > >
> > > > > > > > > > 1. Nit: `getHeaders`
> in
> > > > `ProducerRecord` and
> > > > > > > `ConsumerRecord`
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > named `headers` (we
> > avoid the
> > > `get`
> > > > prefix in
> > > > > > Kafka)
> > > > > > > > > >
> > > > > > > > > > 2. The `Headers`
> class
> > is mutable
> > > > (there's
> > > > > an `add`
> > > > > > > method). Does
> > > > > > > > it
> > > > > > > > > > need
> > > > > > > > > > to be? If so, it
> would
> > be good to
> > > > explain
> > > > > why.
> > > > > > Related
> > > > > > > to that, we
> > > > > > > > > > should
> > > > > > > > > > also explain the
> > thinking around
> > > > > thread-safety. If
> > > > > > we
> > > > > > > keep the
> > > > > > > > `add`
> > > > > > > > > > method, it may make
> > sense for it
> > > to
> > > > take a
> > > > > `Header`
> > > > > > > (that way we
> > > > > > > > can
> > > > > > > > > > add
> > > > > > > > > > things to `Header`
> > without
> > > changing the
> > > > > interface).
> > > > > > > > > >
> > > > > > > > > > 3. Do we need the
> > `Headers.get()`
> > > > method?
> > > > > People
> > > > > > usually
> > > > > > > assume
> > > > > > > > that
> > > > > > > > > > `get`
> > > > > > > > > > would be efficient,
> but
> > > depending on
> > > > the
> > > > > > implementation
> > > > > > > (the
> > > > > > > > current
> > > > > > > > > > proposal states that
> > an array
> > > would be
> > > > > used), it
> > > > > > may not
> > > > > > > be. If we
> > > > > > > > > > expect
> > > > > > > > > > the number of headers
> > to be
> > > small, it
> > > > doesn't
> > > > > > matter
> > > > > > > though.
> > > > > > > > > >
> > > > > > > > > > Ismael
> > > > > > > > > >
> > > > > > > > > > On Tue, Feb 21, 2017
> > at 6:38 PM,
> > > > Michael
> > > > > Pearce <
> > > > > > > > > Michael.Pearce@ig.com
> > > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jason,
> > > > > > > > > > >
> > > > > > > > > > > Have converted the
> > > interface/api
> > > > bullets
> > > > > into
> > > > > > > interface code
> > > > > > > > > > snippets.
> > > > > > > > > > >
> > > > > > > > > > > Agreed
> > implementation won’t
> > > take too
> > > > long.
> > > > > We
> > > > > > have
> > > > > > > early versions
> > > > > > > > > > already.
> > > > > > > > > > > Maybe a week before
> > you think
> > > about
> > > > > merging I
> > > > > > would
> > > > > > > assume it
> > > > > > > > would
> > > > > > > > > > be more
> > > > > > > > > > > stabilised? I was
> > thinking
> > > then we
> > > > could
> > > > > fork
> > > > > > from
> > > > > > > your confluent
> > > > > > > > > > branch,
> > > > > > > > > > > making and then
> > holding KIP-82
> > > > changes in
> > > > > a patch
> > > > > > > file, that we
> > > > > > > > can
> > > > > > > > > > then
> > > > > > > > > > > re-fork from apache
> > once KIP98
> > > final
> > > > is
> > > > > merged,
> > > > > > and
> > > > > > > apply patch
> > > > > > > > > with
> > > > > > > > > > last
> > > > > > > > > > > minute changes.
> > > > > > > > > > >
> > > > > > > > > > > Cheers
> > > > > > > > > > > Mike
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On 22/02/2017,
> > 00:56, "Jason
> > > > Gustafson" <
> > > > > > > jason@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hey Michael,
> > > > > > > > > > >
> > > > > > > > > > > Awesome. I have
> > a minor
> > > request.
> > > > The
> > > > > APIs are
> > > > > > > currently
> > > > > > > > > > documented as a
> > > > > > > > > > > wiki list.
> Would
> > you mind
> > > adding
> > > > a code
> > > > > > snippet
> > > > > > > instead?
> > > > > > > > It's a
> > > > > > > > > > bit
> > > > > > > > > > > easier
> > > > > > > > > > > to process.
> > > > > > > > > > >
> > > > > > > > > > > How will be
> best
> > to manage
> > > this,
> > > > as we
> > > > > will
> > > > > > > obviously build
> > > > > > > > off
> > > > > > > > > > your
> > > > > > > > > > > KIP’s
> > > > > > > > > > > > protocol
> > changes, to
> > > avoid a
> > > > merge
> > > > > hell,
> > > > > > should
> > > > > > > we branch
> > > > > > > > > from
> > > > > > > > > > your
> > > > > > > > > > > branch
> > > > > > > > > > > > in the
> > confluent repo or
> > > is it
> > > > worth
> > > > > > having a
> > > > > > > KIP-98
> > > > > > > > special
> > > > > > > > > > branch
> > > > > > > > > > > in the
> > > > > > > > > > > > apache git,
> > that we can
> > > > branch/fork
> > > > > from?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I was thinking
> > about this
> > > also.
> > > > > Ideally we'd
> > > > > > like
> > > > > > > to get the
> > > > > > > > > > changes
> > > > > > > > > > > in as
> > > > > > > > > > > close together
> > as possible
> > > since
> > > > we
> > > > > only
> > > > > > want one
> > > > > > > magic bump
> > > > > > > > > and
> > > > > > > > > > some
> > > > > > > > > > > users
> > > > > > > > > > > deploy trunk.
> > The level of
> > > > effort to
> > > > > change
> > > > > > the
> > > > > > > format for
> > > > > > > > > > headers
> > > > > > > > > > > seems
> > > > > > > > > > > not too high.
> Do
> > you
> > > agree? My
> > > > guess
> > > > > is that
> > > > > > the
> > > > > > > KIP-98
> > > > > > > > message
> > > > > > > > > > format
> > > > > > > > > > > patch will take
> > 2-3 weeks
> > > to
> > > > review
> > > > > before we
> > > > > > > merge to trunk,
> > > > > > > > > so
> > > > > > > > > > you
> > > > > > > > > > > could
> > > > > > > > > > > hold off
> > implementing
> > > until that
> > > > patch
> > > > > has
> > > > > > somewhat
> > > > > > > > stabilized.
> > > > > > > > > > That
> > > > > > > > > > > would
> > > > > > > > > > > save some
> > potential rebase
> > > pain.
> > > > > > > > > > >
> > > > > > > > > > > -Jason
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The information
> > contained in
> > > this
> > > > email is
> > > > > > strictly
> > > > > > > confidential
> > > > > > > > > and
> > > > > > > > > > for
> > > > > > > > > > > the use of the
> > addressee only,
> > > unless
> > > > > otherwise
> > > > > > > indicated. If you
> > > > > > > > > > are not
> > > > > > > > > > > the intended
> > recipient, please
> > > do
> > > > not read,
> > > > > > copy, use
> > > > > > > or disclose
> > > > > > > > > to
> > > > > > > > > > others
> > > > > > > > > > > this message or any
> > attachment.
> > > > Please also
> > > > > > notify the
> > > > > > > sender by
> > > > > > > > > > replying
> > > > > > > > > > > to this email or by
> > telephone
> > > > (+44(020
> > > > > 7896 0011)
> > > > > > and
> > > > > > > then delete
> > > > > > > > > the
> > > > > > > > > > > email and any
> copies
> > of it.
> > > Opinions,
> > > > > conclusion
> > > > > > (etc)
> > > > > > > that do
> > > > > > > > not
> > > > > > > > > > relate
> > > > > > > > > > > to the official
> > business of
> > > this
> > > > company
> > > > > shall be
> > > > > > > understood as
> > > > > > > > > > neither
> > > > > > > > > > > given nor endorsed
> > by it. IG
> > > is a
> > > > trading
> > > > > name
> > > > > > of IG
> > > > > > > Markets
> > > > > > > > > Limited
> > > > > > > > > > (a
> > > > > > > > > > > company registered
> > in England
> > > and
> > > > Wales,
> > > > > company
> > > > > > > number 04008957)
> > > > > > > > > > and IG
> > > > > > > > > > > Index Limited (a
> > company
> > > registered
> > > > in
> > > > > England
> > > > > > and
> > > > > > > Wales, company
> > > > > > > > > > number
> > > > > > > > > > > 01190902).
> > Registered address
> > > at
> > > > Cannon
> > > > > Bridge
> > > > > > House,
> > > > > > > 25 Dowgate
> > > > > > > > > > Hill,
> > > > > > > > > > > London EC4R 2YA.
> > Both IG
> > > Markets
> > > > Limited
> > > > > > (register
> > > > > > > number 195355)
> > > > > > > > > > and IG
> > > > > > > > > > > Index Limited
> > (register number
> > > > 114059) are
> > > > > > authorised
> > > > > > > and
> > > > > > > > regulated
> > > > > > > > > > by the
> > > > > > > > > > > Financial Conduct
> > Authority.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The information contained
> > in this
> > > email is
> > > > > strictly
> > > > > > > confidential and
> > > > > > > > for
> > > > > > > > > > the use of the addressee
> > only, unless
> > > > otherwise
> > > > > > indicated.
> > > > > > > If you are
> > > > > > > > not
> > > > > > > > > > the intended recipient,
> > please do
> > > not read,
> > > > > copy, use
> > > > > > or
> > > > > > > disclose to
> > > > > > > > > others
> > > > > > > > > > this message or any
> > attachment.
> > > Please also
> > > > > notify the
> > > > > > > sender by
> > > > > > > > replying
> > > > > > > > > > to this email or by
> > telephone
> > > (+44(020 7896
> > > > > 0011) and
> > > > > > then
> > > > > > > delete the
> > > > > > > > > email
> > > > > > > > > > and any copies of it.
> > Opinions,
> > > conclusion
> > > > (etc)
> > > > > that
> > > > > > do not
> > > > > > > relate to
> > > > > > > > > the
> > > > > > > > > > official business of this
> > company
> > > shall be
> > > > > understood
> > > > > > as
> > > > > > > neither given
> > > > > > > > > nor
> > > > > > > > > > endorsed by it. IG is a
> > trading name
> > > of IG
> > > > > Markets
> > > > > > Limited
> > > > > > > (a company
> > > > > > > > > > registered in England and
> > Wales,
> > > company
> > > > number
> > > > > > 04008957)
> > > > > > > and IG Index
> > > > > > > > > > Limited (a company
> > registered in
> > > England
> > > > and
> > > > > Wales,
> > > > > > company
> > > > > > > number
> > > > > > > > > > 01190902). Registered
> > address at
> > > Cannon
> > > > Bridge
> > > > > House,
> > > > > > 25
> > > > > > > Dowgate Hill,
> > > > > > > > > > London EC4R 2YA. Both IG
> > Markets
> > > Limited
> > > > > (register
> > > > > > number
> > > > > > > 195355) and
> > > > > > > > IG
> > > > > > > > > > Index Limited (register
> > number
> > > 114059) are
> > > > > authorised
> > > > > > and
> > > > > > > regulated by
> > > > > > > > > the
> > > > > > > > > > Financial Conduct
> > Authority.
> > > > > > > > > >
> > > > > > > > > The information contained
> in
> > this
> > > email is
> > > > strictly
> > > > > > > confidential and for
> > > > > > > > > the use of the addressee
> > only, unless
> > > > otherwise
> > > > > > indicated. If
> > > > > > > you are not
> > > > > > > > > the intended recipient,
> > please do not
> > > read,
> > > > copy,
> > > > > use or
> > > > > > > disclose to
> > > > > > > > others
> > > > > > > > > this message or any
> > attachment. Please
> > > also
> > > > notify
> > > > > the
> > > > > > sender
> > > > > > > by replying
> > > > > > > > > to this email or by
> > telephone (+44(020
> > > 7896
> > > > 0011)
> > > > > and
> > > > > > then
> > > > > > > delete the
> > > > > > > > email
> > > > > > > > > and any copies of it.
> > Opinions,
> > > conclusion
> > > > (etc)
> > > > > that do
> > > > > > not
> > > > > > > relate to
> > > > > > > > the
> > > > > > > > > official business of this
> > company
> > > shall be
> > > > > understood as
> > > > > > > neither given
> > > > > > > > nor
> > > > > > > > > endorsed by it. IG is a
> > trading name
> > > of IG
> > > > Markets
> > > > > > Limited (a
> > > > > > > company
> > > > > > > > > registered in England and
> > Wales,
> > > company
> > > > number
> > > > > > 04008957) and
> > > > > > > IG Index
> > > > > > > > > Limited (a company
> > registered in
> > > England and
> > > > Wales,
> > > > > > company
> > > > > > > number
> > > > > > > > > 01190902). Registered
> > address at Cannon
> > > > Bridge
> > > > > House, 25
> > > > > > > Dowgate Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> > Markets
> > > Limited
> > > > (register
> > > > > number
> > > > > > > 195355) and IG
> > > > > > > > > Index Limited (register
> > number 114059)
> > > are
> > > > > authorised and
> > > > > > > regulated by
> > > > > > > > the
> > > > > > > > > Financial Conduct
> Authority.
> > > > > > > > >
> > > > > > > > The information contained in
> > this email
> > > is
> > > > strictly
> > > > > > confidential
> > > > > > > and for
> > > > > > > > the use of the addressee
> only,
> > unless
> > > otherwise
> > > > > indicated.
> > > > > > If
> > > > > > > you are not
> > > > > > > > the intended recipient,
> please
> > do not
> > > read,
> > > > copy,
> > > > > use or
> > > > > > > disclose to others
> > > > > > > > this message or any
> > attachment. Please
> > > also
> > > > notify
> > > > > the
> > > > > > sender by
> > > > > > > replying
> > > > > > > > to this email or by telephone
> > (+44(020
> > > 7896
> > > > 0011)
> > > > > and then
> > > > > > > delete the email
> > > > > > > > and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc)
> > > > > that do
> > > > > > not
> > > > > > > relate to the
> > > > > > > > official business of this
> > company shall
> > > be
> > > > > understood as
> > > > > > neither
> > > > > > > given nor
> > > > > > > > endorsed by it. IG is a
> > trading name of
> > > IG
> > > > Markets
> > > > > Limited
> > > > > > (a
> > > > > > > company
> > > > > > > > registered in England and
> > Wales, company
> > > number
> > > > > 04008957)
> > > > > > and IG
> > > > > > > Index
> > > > > > > > Limited (a company registered
> > in England
> > > and
> > > > Wales,
> > > > > company
> > > > > > > number
> > > > > > > > 01190902). Registered address
> > at Cannon
> > > Bridge
> > > > > House, 25
> > > > > > Dowgate
> > > > > > > Hill,
> > > > > > > > London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > > 195355) and IG
> > > > > > > > Index Limited (register
> number
> > 114059)
> > > are
> > > > > authorised and
> > > > > > > regulated by the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email
> > is strictly
> > > > > confidential and
> > > > > > for
> > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If
> > > > > you
> > > > > > are not
> > > > > > > the intended recipient, please do not
> > read, copy,
> > > use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please
> > also notify
> > > the
> > > > sender
> > > > > by
> > > > > > replying
> > > > > > > to this email or by telephone (+44(020
> > 7896 0011)
> > > and
> > > > then
> > > > > delete
> > > > > > the email
> > > > > > > and any copies of it. Opinions,
> > conclusion (etc)
> > > that do
> > > > not
> > > > > relate
> > > > > > to the
> > > > > > > official business of this company shall
> > be
> > > understood as
> > > > > neither
> > > > > > given nor
> > > > > > > endorsed by it. IG is a trading name of
> > IG Markets
> > > > Limited (a
> > > > > company
> > > > > > > registered in England and Wales,
> company
> > number
> > > > 04008957) and
> > > > > IG
> > > > > > Index
> > > > > > > Limited (a company registered in
> England
> > and Wales,
> > > > company
> > > > > number
> > > > > > > 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > > Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > (register
> > > number
> > > > > 195355)
> > > > > > and IG
> > > > > > > Index Limited (register number 114059)
> > are
> > > authorised and
> > > > > regulated
> > > > > > by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> > strictly
> > > > confidential and
> > > > > for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you
> > > > > are not
> > > > > > the intended recipient, please do not read,
> > copy, use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> > notify the
> > > sender
> > > > by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896
> > 0011) and
> > > then
> > > > delete
> > > > > the email
> > > > > > and any copies of it. Opinions, conclusion
> > (etc) that do
> > > not
> > > > relate
> > > > > to the
> > > > > > official business of this company shall be
> > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of IG
> > Markets
> > > Limited (a
> > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG
> > > > > Index
> > > > > > Limited (a company registered in England and
> > Wales,
> > > company
> > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register number
> > > > 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are
> > authorised and
> > > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > > confidential and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you
> > > > are not
> > > > > the intended recipient, please do not read, copy,
> > use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify
> > the sender
> > > by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896 0011)
> > and then
> > > delete
> > > > the email
> > > > > and any copies of it. Opinions, conclusion (etc)
> > that do not
> > > relate
> > > > to the
> > > > > official business of this company shall be
> > understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG
> > > > Index
> > > > > Limited (a company registered in England and Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> > House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are
> > authorised and
> > > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential
> > > and
> > > > for the use of the addressee only, unless otherwise
> > indicated. If
> > > you are
> > > > not the intended recipient, please do not read, copy, use
> > or
> > > disclose to
> > > > others this message or any attachment. Please also notify
> > the sender
> > > by
> > > > replying to this email or by telephone (+44(020 7896
> 0011)
> > and then
> > > delete
> > > > the email and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > relate to the official business of this company shall be
> > understood
> > > as
> > > > neither given nor endorsed by it. IG is a trading name of
> > IG Markets
> > > > Limited (a company registered in England and Wales,
> > company number
> > > > 04008957) and IG Index Limited (a company registered in
> > England and
> > > Wales,
> > > > company number 01190902). Registered address at Cannon
> > Bridge House,
> > > 25
> > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and IG Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by the Financial Conduct Authority.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential
> > and for
> > > the use of the addressee only, unless otherwise indicated. If
> > you are not
> > > the intended recipient, please do not read, copy, use or
> > disclose to others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> > relate to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> > number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> > regulated by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and
> > for the use of the addressee only, unless otherwise indicated. If you are
> > not the intended recipient, please do not read, copy, use or disclose to
> > others this message or any attachment. Please also notify the sender by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do not
> > relate to the official business of this company shall be understood as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG Index Limited (register number 114059) are authorised and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Becket,
On 1)
Yes truly we wanted mutable headers also. Alas we couldn't think of a solution would address Jason's point around, once a record is sent it shouldn't be possible to mutate it, for cases where you send twice the same record.
Thank you so much for your solution i think this will work very nicely :)
Agreed we only need to do mutable to immutable conversion
I think you solution with a ".close()" taken from else where in the kafka protocol where mutability is existent is a great solution, and happy middle ground.
@Jason you agree, this resolves your concerns if we had mutable headers?
On 2)
Agreed, this was only added as i couldn't think of a solution to that would address Jason's concern, but really didn't want to force end users to constantly write ugly boiler plate code. If we agree on you solution for 1, very happy to remove these.
On 3)
I also would like to keep the scope of this KIP limited to Message Headers for now, else we run the risk of not getting even these delivered for next release and we're almost now there on getting this KIP to the state everyone is happy. As you note address that later if theres the need.
Ill leave it 24hrs and update the kip if no strong objections based on your solution for 1 & 2.
Cheers
Mike
__________ ______________________________
From: Becket Qin <be...@gmail.com>
Sent: Saturday, February 25, 2017 10:33 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Hey Michael,
Thanks for the KIP. It looks good overall and it looks we only have few
things to agree on.
1. Regarding the mutability.
I think it would be a big convenience to have headers mutable during
certain stage in the message life cycle for the use cases you mentioned. I
agree there is a material benefit especially given that we may have to
modify the headers for each message.
That said, I also think it is fair to say that in the producer, in order to
guarantee the correctness of the entire logic, it is necessary that at some
point we need to make producer record immutable. For example we probably
don't want to see that users accidentally updated the headers when the
producer is doing the serialization or compression.
Given that, would it be possible to make Headers to be able to switch from
mutable to immutable? We have done this for the Batch in the producer. For
example, initially the headers are mutable, but after it has gone through
all the interceptors, we can call Headers.close() to make it immutable
afterwards.
On the consumer side, we can probably always leave the the ConsumerRecord
mutable because after we give the messages to the users, Kafka consumer
itself does not care about whether the headers are modified or not anymore.
So far I think we only need to do the mutable to immutable conversion. If
there are use case require immutable to mutable conversion, we may need
something more than a closable.
2. If we agree on what mentioned above, I think it probably makes sense to
put the addHeaders()/removeHeaders() methods into Headers class and just
leave the headers() method in ProducerRecord and ConsumerRecord.
3. It might be useful to have headers at MessageSet level as well so we can
avoid decompression in some cases. But given this KIP is already
complicated, I would rather leave this out of the scope and address that
later when needed, e.g. after having batch level interceptors.
Thanks,
Jiangjie (Becket) Qin
On Fri, Feb 24, 2017 at 3:56 PM, Michael Pearce <Mi...@ig.com>
wrote:
> KIP updated in response to the below comments:
>
> > 1. Is the intent of `Headers.filter` to include or exclude the
> headers
> > matching the key? Can you add a javadoc to clarify?
> > 2. The KIP mentions that we will introduce V4 of FetchRequest
> and V4 of
> > ProduceRequest. Can you change this to say that the changes will
> > piggyback
> > onto V3 of ProduceRequest and V4 of FetchRequest which were
> introduced
> > in
> > KIP-98?
>
>
>
>
> On 24/02/2017, 23:20, "Michael Pearce" <Mi...@ig.com> wrote:
>
> We’re trying to make an eco-system for people to be able to use
> headers, I think we want to ensure some least common features are supported
> and not limited.
>
>
> Some examples we have already.
>
> On consume interceptors a security interceptor may need to take the
> current header, decrypt the data and replace the token with the next token
> for the next processing, in case of a single decryption token being one
> time use only.
>
> On produce it could be the interceptors add some values in the clear
> from the systems that supply them, but later a security header interceptor
> needs to encrypt some headers, as such needs to replace the current value
> with new one.
>
> I note Radai already requested this in the thread, I assume he has
> some use case also. S
>
> Simple add / remove is a least common feature.
>
> Rgds,
> Mike
>
>
> On 24/02/2017, 23:00, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Michael,
>
> I'm not strongly opposed to them; I just don't see a lot of
> benefit. One
> thing it would be good to understand is why a consumer interceptor
> would
> need to add headers and why a producer interceptor would need to
> remove
> them. Maybe we only need the common cases?
>
> Thanks,
> Jason
>
> On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > Hi Jason,
> >
> > Sorry I thought this was the agreed compromise to provide an api
> that
> > avoid boiler plate in return for immutabilty.
> >
> > If not then mutability will be needed as a goal is to have a
> single clean
> > method call to append/remove a header.
> >
> > Cheers
> > Mike
> >
> > On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >
> > Hey Michael,
> >
> > I didn't actually comment on the new methods for
> ProducerRecord and
> > ConsumerRecord. If they only save some boilerplate, I'd just
> as well
> > not
> > have them.
> >
> > Also a couple minor comments:
> >
> > 1. Is the intent of `Headers.filter` to include or exclude
> the headers
> > matching the key? Can you add a javadoc to clarify?
> > 2. The KIP mentions that we will introduce V4 of
> FetchRequest and V4 of
> > ProduceRequest. Can you change this to say that the changes
> will
> > piggyback
> > onto V3 of ProduceRequest and V4 of FetchRequest which were
> introduced
> > in
> > KIP-98?
> >
> > The rest of the KIP looks good to me.
> >
> > -Jason
> >
> > On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > I’ve added the methods on the ProducerRecord that will
> return a new
> > > instance of ProducerRecord with modified headers.
> > >
> > > On 24/02/2017, 19:22, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > wrote:
> > >
> > > Pattern.compile is expensive, and even if cached
> String.equals is
> > > faster than matched. also if we end up with an internal
> map in
> > future for
> > > performance it will be easier to be by key.
> > >
> > > As all that's needed is to get header by key.
> > >
> > > With like the other arguements of let's implement
> simple and
> > then we
> > > can always add pattern later as well if it's found it's
> needed. (As
> > noted
> > > it's easier to add methods than to take away)
> > >
> > > Great I'll update kip with extra methods on
> producerecord and a
> > note
> > > that new objects are returned by method calls.
> > >
> > >
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Friday, February 24, 2017 6:51:45 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > The APIs in the current KIP look good to me. Just a
> couple
> > questions:
> > > why
> > > does filter not return Headers? Also would it be
> useful if the
> > key is a
> > > regex?
> > >
> > > On the point of immutability.. One option might be to
> use a
> > mutable
> > > object
> > > only when passing the headers through the interceptor
> chain. I
> > think as
> > > long as we resort to mutability only when clear
> performance
> > results
> > > show
> > > that it is worthwhile, I am satisfied. As Ismael
> noted, for
> > common
> > > scenarios it is possible to get reasonable performance
> with
> > immutable
> > > objects.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > On 1, How can you guarantee two separate
> implemented clients
> > would
> > > add
> > > > the headers in the same order we are not specifying
> an order
> > at the
> > > > protocol level (nor should we) with regards to keyA
> being
> > ordered
> > > before
> > > > keyB? We shouldn’t be expecting keyA to be always
> set before
> > keyB.
> > > >
> > > > On 2, I believe we have changed the naming based on
> feedback
> > from
> > > Jason
> > > > already, e.g. we don’t have “get” method that
> inferred O(1)
> > > performance,
> > > > like wise nor “put” but we have an “append”
> > > >
> > > > On 3, in the KafkaProducer, I think we have
> mutability
> > already, the
> > > value
> > > > for time is changed if it is null, at the point of
> send:
> > > > “
> > > > long timestamp = record.timestamp() ==
> null ?
> > > > time.milliseconds() : record.timestamp();
> > > > “
> > > >
> > > > As such the timestamp is already mutable, so what’s
> the
> > difference
> > > here,
> > > > we already have some mixed semantics. On timestamp.
> > > > e.g. currently if I send to records with timestamp
> not set,
> > the wire
> > > > binary sent the value for the timestamp would be
> different, as
> > such
> > > we have
> > > > mutation for the same record.
> > > >
> > > > On 4, I think we should not expect not 1 or 2
> headers, but
> > infact
> > > 10’s of
> > > > headers. This is the concern on immutable headers,
> whilst the
> > append
> > > > self-reference works nicely, what if someone needs
> to remove a
> > > header?
> > > >
> > > > Trying to get this moving:
> > > >
> > > > If we really wanted Immutable Headers and
> essentially you guys
> > wont
> > > give
> > > > +1 for it without.
> > > >
> > > > Whats the feeling for adding methods to
> ProducerRecord that
> > does the
> > > > boiler plate code or creating a new ProducerRecord
> with the
> > altered
> > > new
> > > > headers (appended or removed) inside. E.g.
> > > >
> > > > ProducerRecord {
> > > >
> > > >
> > > > ProducerRecord append(Iterable<Headers>
> headersToAppend){
> > > > return new ProducerRecord(key, value,
> headers.append(
> > > headersToAppend),
> > > > ….)
> > > > }
> > > >
> > > > ProducerRecord remove(Iterable<Headers>
> headersToAppend){
> > > > return new ProducerRecord(key, value,
> headers.remove(
> > > headersToAppend),
> > > > ….)
> > > > }
> > > >
> > > > }
> > > >
> > > > Were the headers methods actually returns new
> objects, and the
> > > producer
> > > > records methods create a new producer record with
> all the
> > current
> > > values,
> > > > but with the new modified headers.
> > > >
> > > > Then interceptors / code return this new object?
> > > >
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf
> of Ismael
> > Juma" <
> > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk>
> wrote:
> > > >
> > > > Hi Michael,
> > > >
> > > > Did you mean that you were happy to compromise
> to keep it
> > > mutable or
> > > > immutable? You wrote the former, but it sounded
> from the
> > > sentence that
> > > > it
> > > > could have been a typo. So, my thoughts on this
> is that
> > there
> > > are a few
> > > > things to take into account:
> > > >
> > > > 1. Semantics
> > > > 2. Simplicity of use (the common operations
> should be easy
> > to do)
> > > > 3. If it's easy to reason about and safe
> (immutability
> > helps
> > > with this)
> > > > 4. Efficiency (both memory and CPU usage)
> > > >
> > > > Regarding 1, I think it would be good to be very
> clear
> > about the
> > > > guarantees
> > > > that we are providing. It seems that we are
> saying that
> > keys are
> > > > unordered,
> > > > but what about the case where there are multiple
> values
> > for the
> > > same
> > > > key?
> > > > It seems that for some use cases (e.g. lineage),
> it may be
> > > useful to
> > > > add
> > > > values to the same key while preserving the
> order.
> > > >
> > > > Regarding 2, I agree that it's useful to have
> methods in
> > > `Headers` for
> > > > the
> > > > very common use cases although we have to be
> careful with
> > the
> > > naming to
> > > > avoid giving the wrong impression. Also, when it
> comes to
> > > `Map.Entry`,
> > > > I
> > > > think I'd prefer a `toMap` method that simply
> gives the
> > user a
> > > Map if
> > > > that's what they want (it makes it clear that
> there's a
> > > conversion
> > > > happening).
> > > >
> > > > Regarding 3, the concern I have if we make the
> headers
> > mutable
> > > is that
> > > > it
> > > > seems to introduce some inconsistencies and
> potential edge
> > > cases. At
> > > > the
> > > > moment, it's safe to keep a ProducerRecord and
> resend it,
> > for
> > > example.
> > > > If
> > > > the record is mutated by an interceptor, then
> this can
> > lead to
> > > weird
> > > > behaviour. Also, it seems inconsistent that one
> has to
> > create a
> > > new
> > > > ProducerRecord to modify the record timestamp,
> but that
> > one has
> > > to
> > > > mutate
> > > > the record to add headers. It seems like one
> should either
> > > embrace
> > > > immutability or mutability, but mixing both is
> not ideal.
> > > >
> > > > Regarding 4, for the cases where there are a
> small number
> > of
> > > headers
> > > > and/or
> > > > 1 or 2 interceptors, it doesn't seem difficult
> to come up
> > with
> > > > reasonable
> > > > implementations for mutable and immutable cases
> that do a
> > good
> > > enough
> > > > job.
> > > > However, if the number of headers and
> interceptors is
> > large,
> > > then care
> > > > is
> > > > needed for the immutable case to avoid
> unnecessary
> > copying. A
> > > simple
> > > > option
> > > > that adds little memory overhead if we have
> Header
> > instances is
> > > to
> > > > simply
> > > > add a self-reference to the previous Header in
> the linked
> > list.
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Im happy to compromise to keep it mutable but
> move to an
> > append
> > > > style api.
> > > > > (as in guava interables concat)
> > > > >
> > > > > class Headers {
> > > > > Headers append(Iterable<Header>
> headers);
> > > > > }
> > > > >
> > > > >
> > > > > I don’t think we’d want prepend, this would
> give the
> > idea of
> > > > guaranteed
> > > > > ordering, when in actual fact we don’t provide
> that
> > guarantee
> > > (.e.g
> > > > one
> > > > > client can put headerA, then headerB, but
> another could
> > put
> > > headerB
> > > > then
> > > > > headerA, this shouldn’t cause issues), Also
> what if we
> > changed
> > > to a
> > > > hashmap
> > > > > for the internal implementation, its just a
> bucket of
> > entries
> > > no
> > > > ordering.
> > > > > I think we just need to provide an api to
> add/append
> > headers.
> > > > >
> > > > > This ok? If so ill update KIP to record this.
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> > jason@confluent.io>
> > > wrote:
> > > > >
> > > > > The point about usability is fair. It's
> also
> > reasonable to
> > > > expect that
> > > > > common use cases such as appending headers
> should be
> > done
> > > > efficiently.
> > > > >
> > > > > Perhaps we could compromise with something
> like this?
> > > > >
> > > > > class Headers {
> > > > > Headers append(Iterable<Header> headers);
> > > > > Headers prepend(Iterable<Header> headers);
> > > > > }
> > > > >
> > > > > That retains ease of use while still giving
> > ourselves some
> > > > flexibility
> > > > > in
> > > > > the implementation.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > > On Wed, Feb 22, 2017 at 3:03 PM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I wasn’t referring to the headers
> needing to be
> > copied,
> > > im
> > > > meaning
> > > > > the
> > > > > > fact we’d be forcing a new producer
> record to be
> > > created, with
> > > > all
> > > > > the
> > > > > > contents copied.
> > > > > >
> > > > > > i.e what will happen is utility method
> will be
> > created
> > > or end
> > > > up
> > > > > being
> > > > > > used, which does this, and returns the
> new
> > ProducerRecord
> > > > instance.
> > > > > >
> > > > > > ProducerRecord addHeader(ProducerRecord
> record,
> > Header
> > > > header){
> > > > > > Return New ProducerRecord(record.key,
> record.value,
> > > > > record.timestamp…..,
> > > > > > record.headers.concat(header))
> > > > > > }
> > > > > >
> > > > > > To me this seems ugly, but will be
> inevitable if we
> > > don’t make
> > > > adding
> > > > > > headers to existing records a simple
> clean method
> > call.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 22:57, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > Lazy init can achieve/avoid that.
> > > > > >
> > > > > > Re the concat, why don’t we
> implement that
> > inside the
> > > > Headers
> > > > > rather
> > > > > > than causing everyone to implement this
> as adding
> > > headers in
> > > > > interceptors
> > > > > > will be a dominant use case. We want a
> user
> > friendly API.
> > > > Having as
> > > > > a user
> > > > > > having to code this instead of having
> the headers
> > handle
> > > this
> > > > for me
> > > > > seems
> > > > > > redundant.
> > > > > >
> > > > > > On 22/02/2017, 22:34, "Jason
> Gustafson" <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > I thought the argument was
> against
> > creating the
> > > extra
> > > > objects
> > > > > > unnecessarily
> > > > > > (i.e. if they were not
> accessed). And note
> > that
> > > making
> > > > the
> > > > > Headers
> > > > > > immutable doesn't necessarily
> mean that
> > they
> > > need to be
> > > > > copied:
> > > > > > you can do
> > > > > > a trick like Guava's
> Iterables.concat to
> > add
> > > additional
> > > > > headers
> > > > > > without
> > > > > > changing the underlying
> collections.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 2:22 PM,
> Michael
> > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > If the argument for not having
> a map
> > holding
> > > the
> > > > key, value
> > > > > > pairs is due
> > > > > > > to garbage creation of HashMap
> entry's,
> > > forcing the
> > > > > creation of
> > > > > > a whole new
> > > > > > > producer record to simply add
> a head,
> > surely is
> > > > creating
> > > > > a-lot
> > > > > > more?
> > > > > > > ______________________________
> __________
> > > > > > > From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > Sent: Wednesday, February 22,
> 2017 10:09
> > PM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-82
> - Add
> > Record
> > > Headers
> > > > > > >
> > > > > > > The current producer
> interceptor API is
> > this:
> > > > > > >
> > > > > > > ProducerRecord<K, V>
> > onSend(ProducerRecord<K,
> > > V>
> > > > record);
> > > > > > >
> > > > > > > So adding a header means
> creating a new
> > > > ProducerRecord
> > > > > with a
> > > > > > new header
> > > > > > > added to the current headers
> and
> > returning it.
> > > Would
> > > > that
> > > > > not
> > > > > > work?
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 1:45
> PM, Michael
> > > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > So how would you have this
> work if not
> > > mutable
> > > > where
> > > > > > interceptors would
> > > > > > > > add headers?
> > > > > > > >
> > > > > > > > Sent using OWA for iPhone
> > > > > > > >
> ______________________________
> > __________
> > > > > > > > From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > > Sent: Wednesday, February
> 22, 2017
> > 8:42:27 PM
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Subject: Re: [DISCUSS]
> KIP-82 - Add
> > Record
> > > Headers
> > > > > > > >
> > > > > > > > I think the point on the
> mutability of
> > > Headers is
> > > > worth
> > > > > > discussing a
> > > > > > > little
> > > > > > > > more. As far as I can tell,
> once the
> > > > ProducerRecord (or
> > > > > > ConsumerRecord)
> > > > > > > is
> > > > > > > > constructed, there should be
> no need to
> > > further
> > > > change
> > > > > the
> > > > > > headers. Is
> > > > > > > that
> > > > > > > > correct? If so, then why not
> enforce
> > that
> > > that is
> > > > the
> > > > > case
> > > > > > through the
> > > > > > > API?
> > > > > > > > One problem with mutability
> it that it
> > > constrains
> > > > the
> > > > > > implementation of
> > > > > > > > Headers. For example, if we
> were
> > backing
> > > with a
> > > > byte
> > > > > slice,
> > > > > > would we
> > > > > > > recopy
> > > > > > > > the bytes if a header is
> added or
> > would we
> > > > maintain a
> > > > > satellite
> > > > > > > collection
> > > > > > > > of added records. Seems not
> great
> > either
> > > way. If we
> > > > > really
> > > > > > think
> > > > > > > mutability
> > > > > > > > is needed, perhaps we could
> add a
> > method to
> > > > headers to
> > > > > convert
> > > > > > it to a
> > > > > > > > mutable type (e.g. a
> Headers.toMap)?
> > > > > > > >
> > > > > > > > I'm also with Ismael about
> exposing
> > > Headers.get().
> > > > I
> > > > > thought
> > > > > > it might
> > > > > > > make
> > > > > > > > sense to have a method like
> this
> > instead:
> > > > > > > >
> > > > > > > > Iterable<Header>
> findMatching(Pattern
> > > pattern);
> > > > > > > >
> > > > > > > > This makes the (potential)
> need to
> > scan the
> > > headers
> > > > > clear in
> > > > > > the API. I'd
> > > > > > > > also be fine exposing no
> getter at
> > all. In
> > > > general, Ï
> > > > > think
> > > > > > it's good to
> > > > > > > > start with a minimalistic
> API and work
> > from
> > > there
> > > > since
> > > > > it's
> > > > > > always
> > > > > > > easier
> > > > > > > > to add methods than remove
> them.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Wed, Feb 22, 2017 at 9:16
> AM,
> > Michael
> > > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Ismael
> > > > > > > > >
> > > > > > > > > On point 1,
> > > > > > > > >
> > > > > > > > > Sure makes sense will
> update shortly.
> > > > > > > > >
> > > > > > > > > On point 2,
> > > > > > > > >
> > > > > > > > > Setter/getter typical to
> > > properties/headers api’s
> > > > > > traditionally are map
> > > > > > > > > styled interfaces and what
> I believe
> > is
> > > most
> > > > expected
> > > > > styled
> > > > > > thus the
> > > > > > > > Key,
> > > > > > > > > Value setter.
> > > > > > > > > Also it would mean rather
> than an
> > > interface, we
> > > > would
> > > > > be
> > > > > > making our
> > > > > > > > > internal header impl
> object we have
> > for the
> > > > array,
> > > > > exposed.
> > > > > > E.g. if we
> > > > > > > > had
> > > > > > > > > a Map really this would be
> Map.Entry
> > > interface,
> > > > this
> > > > > is the
> > > > > > same
> > > > > > > reasons
> > > > > > > > on
> > > > > > > > > the map interface I cannot
> actually
> > make
> > > the
> > > > > underlying Node
> > > > > > object
> > > > > > > > that’s
> > > > > > > > > the implementation for
> HashMap, so
> > that
> > > > internals can
> > > > > be
> > > > > > changed.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On point 3,
> > > > > > > > >
> > > > > > > > > I think it people do
> expect it to be
> > > performant,
> > > > thus
> > > > > why
> > > > > > originally
> > > > > > > > > concern I raised with
> using an array
> > for
> > > to me
> > > > is an
> > > > > early
> > > > > > memory
> > > > > > > > > optimisation. I think the
> user
> > experience
> > > of
> > > > > > properties/headers is on a
> > > > > > > > > get/set model. This is why
> its
> > important
> > > we have
> > > > > > encapsulated logic
> > > > > > > that
> > > > > > > > > then allows us to change
> this to a
> > map, if
> > > this
> > > > > becomes and
> > > > > > issue, and
> > > > > > > > the
> > > > > > > > > memory overhead of hashmap
> is less
> > so.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 22/02/2017, 15:56, "
> > ismaelj@gmail.com
> > > on
> > > > behalf of
> > > > > > Ismael Juma" <
> > > > > > > > > ismaelj@gmail.com on
> behalf of
> > > ismael@juma.me.uk
> > > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Great to see the
> progress that
> > has been
> > > > achieved
> > > > > on this
> > > > > > one. :) A
> > > > > > > > few
> > > > > > > > > comments regarding the
> APIs (I'm
> > still
> > > > reviewing
> > > > > the
> > > > > > message format
> > > > > > > > > changes):
> > > > > > > > >
> > > > > > > > > 1. Nit: `getHeaders` in
> > > `ProducerRecord` and
> > > > > > `ConsumerRecord`
> > > > > > > should
> > > > > > > > be
> > > > > > > > > named `headers` (we
> avoid the
> > `get`
> > > prefix in
> > > > > Kafka)
> > > > > > > > >
> > > > > > > > > 2. The `Headers` class
> is mutable
> > > (there's
> > > > an `add`
> > > > > > method). Does
> > > > > > > it
> > > > > > > > > need
> > > > > > > > > to be? If so, it would
> be good to
> > > explain
> > > > why.
> > > > > Related
> > > > > > to that, we
> > > > > > > > > should
> > > > > > > > > also explain the
> thinking around
> > > > thread-safety. If
> > > > > we
> > > > > > keep the
> > > > > > > `add`
> > > > > > > > > method, it may make
> sense for it
> > to
> > > take a
> > > > `Header`
> > > > > > (that way we
> > > > > > > can
> > > > > > > > > add
> > > > > > > > > things to `Header`
> without
> > changing the
> > > > interface).
> > > > > > > > >
> > > > > > > > > 3. Do we need the
> `Headers.get()`
> > > method?
> > > > People
> > > > > usually
> > > > > > assume
> > > > > > > that
> > > > > > > > > `get`
> > > > > > > > > would be efficient, but
> > depending on
> > > the
> > > > > implementation
> > > > > > (the
> > > > > > > current
> > > > > > > > > proposal states that
> an array
> > would be
> > > > used), it
> > > > > may not
> > > > > > be. If we
> > > > > > > > > expect
> > > > > > > > > the number of headers
> to be
> > small, it
> > > doesn't
> > > > > matter
> > > > > > though.
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Tue, Feb 21, 2017
> at 6:38 PM,
> > > Michael
> > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jason,
> > > > > > > > > >
> > > > > > > > > > Have converted the
> > interface/api
> > > bullets
> > > > into
> > > > > > interface code
> > > > > > > > > snippets.
> > > > > > > > > >
> > > > > > > > > > Agreed
> implementation won’t
> > take too
> > > long.
> > > > We
> > > > > have
> > > > > > early versions
> > > > > > > > > already.
> > > > > > > > > > Maybe a week before
> you think
> > about
> > > > merging I
> > > > > would
> > > > > > assume it
> > > > > > > would
> > > > > > > > > be more
> > > > > > > > > > stabilised? I was
> thinking
> > then we
> > > could
> > > > fork
> > > > > from
> > > > > > your confluent
> > > > > > > > > branch,
> > > > > > > > > > making and then
> holding KIP-82
> > > changes in
> > > > a patch
> > > > > > file, that we
> > > > > > > can
> > > > > > > > > then
> > > > > > > > > > re-fork from apache
> once KIP98
> > final
> > > is
> > > > merged,
> > > > > and
> > > > > > apply patch
> > > > > > > > with
> > > > > > > > > last
> > > > > > > > > > minute changes.
> > > > > > > > > >
> > > > > > > > > > Cheers
> > > > > > > > > > Mike
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 22/02/2017,
> 00:56, "Jason
> > > Gustafson" <
> > > > > > jason@confluent.io>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hey Michael,
> > > > > > > > > >
> > > > > > > > > > Awesome. I have
> a minor
> > request.
> > > The
> > > > APIs are
> > > > > > currently
> > > > > > > > > documented as a
> > > > > > > > > > wiki list. Would
> you mind
> > adding
> > > a code
> > > > > snippet
> > > > > > instead?
> > > > > > > It's a
> > > > > > > > > bit
> > > > > > > > > > easier
> > > > > > > > > > to process.
> > > > > > > > > >
> > > > > > > > > > How will be best
> to manage
> > this,
> > > as we
> > > > will
> > > > > > obviously build
> > > > > > > off
> > > > > > > > > your
> > > > > > > > > > KIP’s
> > > > > > > > > > > protocol
> changes, to
> > avoid a
> > > merge
> > > > hell,
> > > > > should
> > > > > > we branch
> > > > > > > > from
> > > > > > > > > your
> > > > > > > > > > branch
> > > > > > > > > > > in the
> confluent repo or
> > is it
> > > worth
> > > > > having a
> > > > > > KIP-98
> > > > > > > special
> > > > > > > > > branch
> > > > > > > > > > in the
> > > > > > > > > > > apache git,
> that we can
> > > branch/fork
> > > > from?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I was thinking
> about this
> > also.
> > > > Ideally we'd
> > > > > like
> > > > > > to get the
> > > > > > > > > changes
> > > > > > > > > > in as
> > > > > > > > > > close together
> as possible
> > since
> > > we
> > > > only
> > > > > want one
> > > > > > magic bump
> > > > > > > > and
> > > > > > > > > some
> > > > > > > > > > users
> > > > > > > > > > deploy trunk.
> The level of
> > > effort to
> > > > change
> > > > > the
> > > > > > format for
> > > > > > > > > headers
> > > > > > > > > > seems
> > > > > > > > > > not too high. Do
> you
> > agree? My
> > > guess
> > > > is that
> > > > > the
> > > > > > KIP-98
> > > > > > > message
> > > > > > > > > format
> > > > > > > > > > patch will take
> 2-3 weeks
> > to
> > > review
> > > > before we
> > > > > > merge to trunk,
> > > > > > > > so
> > > > > > > > > you
> > > > > > > > > > could
> > > > > > > > > > hold off
> implementing
> > until that
> > > patch
> > > > has
> > > > > somewhat
> > > > > > > stabilized.
> > > > > > > > > That
> > > > > > > > > > would
> > > > > > > > > > save some
> potential rebase
> > pain.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The information
> contained in
> > this
> > > email is
> > > > > strictly
> > > > > > confidential
> > > > > > > > and
> > > > > > > > > for
> > > > > > > > > > the use of the
> addressee only,
> > unless
> > > > otherwise
> > > > > > indicated. If you
> > > > > > > > > are not
> > > > > > > > > > the intended
> recipient, please
> > do
> > > not read,
> > > > > copy, use
> > > > > > or disclose
> > > > > > > > to
> > > > > > > > > others
> > > > > > > > > > this message or any
> attachment.
> > > Please also
> > > > > notify the
> > > > > > sender by
> > > > > > > > > replying
> > > > > > > > > > to this email or by
> telephone
> > > (+44(020
> > > > 7896 0011)
> > > > > and
> > > > > > then delete
> > > > > > > > the
> > > > > > > > > > email and any copies
> of it.
> > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that do
> > > > > > > not
> > > > > > > > > relate
> > > > > > > > > > to the official
> business of
> > this
> > > company
> > > > shall be
> > > > > > understood as
> > > > > > > > > neither
> > > > > > > > > > given nor endorsed
> by it. IG
> > is a
> > > trading
> > > > name
> > > > > of IG
> > > > > > Markets
> > > > > > > > Limited
> > > > > > > > > (a
> > > > > > > > > > company registered
> in England
> > and
> > > Wales,
> > > > company
> > > > > > number 04008957)
> > > > > > > > > and IG
> > > > > > > > > > Index Limited (a
> company
> > registered
> > > in
> > > > England
> > > > > and
> > > > > > Wales, company
> > > > > > > > > number
> > > > > > > > > > 01190902).
> Registered address
> > at
> > > Cannon
> > > > Bridge
> > > > > House,
> > > > > > 25 Dowgate
> > > > > > > > > Hill,
> > > > > > > > > > London EC4R 2YA.
> Both IG
> > Markets
> > > Limited
> > > > > (register
> > > > > > number 195355)
> > > > > > > > > and IG
> > > > > > > > > > Index Limited
> (register number
> > > 114059) are
> > > > > authorised
> > > > > > and
> > > > > > > regulated
> > > > > > > > > by the
> > > > > > > > > > Financial Conduct
> Authority.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information contained
> in this
> > email is
> > > > strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > > > the use of the addressee
> only, unless
> > > otherwise
> > > > > indicated.
> > > > > > If you are
> > > > > > > not
> > > > > > > > > the intended recipient,
> please do
> > not read,
> > > > copy, use
> > > > > or
> > > > > > disclose to
> > > > > > > > others
> > > > > > > > > this message or any
> attachment.
> > Please also
> > > > notify the
> > > > > > sender by
> > > > > > > replying
> > > > > > > > > to this email or by
> telephone
> > (+44(020 7896
> > > > 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > > email
> > > > > > > > > and any copies of it.
> Opinions,
> > conclusion
> > > (etc)
> > > > that
> > > > > do not
> > > > > > relate to
> > > > > > > > the
> > > > > > > > > official business of this
> company
> > shall be
> > > > understood
> > > > > as
> > > > > > neither given
> > > > > > > > nor
> > > > > > > > > endorsed by it. IG is a
> trading name
> > of IG
> > > > Markets
> > > > > Limited
> > > > > > (a company
> > > > > > > > > registered in England and
> Wales,
> > company
> > > number
> > > > > 04008957)
> > > > > > and IG Index
> > > > > > > > > Limited (a company
> registered in
> > England
> > > and
> > > > Wales,
> > > > > company
> > > > > > number
> > > > > > > > > 01190902). Registered
> address at
> > Cannon
> > > Bridge
> > > > House,
> > > > > 25
> > > > > > Dowgate Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> Markets
> > Limited
> > > > (register
> > > > > number
> > > > > > 195355) and
> > > > > > > IG
> > > > > > > > > Index Limited (register
> number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > > the
> > > > > > > > > Financial Conduct
> Authority.
> > > > > > > > >
> > > > > > > > The information contained in
> this
> > email is
> > > strictly
> > > > > > confidential and for
> > > > > > > > the use of the addressee
> only, unless
> > > otherwise
> > > > > indicated. If
> > > > > > you are not
> > > > > > > > the intended recipient,
> please do not
> > read,
> > > copy,
> > > > use or
> > > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any
> attachment. Please
> > also
> > > notify
> > > > the
> > > > > sender
> > > > > > by replying
> > > > > > > > to this email or by
> telephone (+44(020
> > 7896
> > > 0011)
> > > > and
> > > > > then
> > > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it.
> Opinions,
> > conclusion
> > > (etc)
> > > > that do
> > > > > not
> > > > > > relate to
> > > > > > > the
> > > > > > > > official business of this
> company
> > shall be
> > > > understood as
> > > > > > neither given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a
> trading name
> > of IG
> > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > > registered in England and
> Wales,
> > company
> > > number
> > > > > 04008957) and
> > > > > > IG Index
> > > > > > > > Limited (a company
> registered in
> > England and
> > > Wales,
> > > > > company
> > > > > > number
> > > > > > > > 01190902). Registered
> address at Cannon
> > > Bridge
> > > > House, 25
> > > > > > Dowgate Hill,
> > > > > > > > London EC4R 2YA. Both IG
> Markets
> > Limited
> > > (register
> > > > number
> > > > > > 195355) and IG
> > > > > > > > Index Limited (register
> number 114059)
> > are
> > > > authorised and
> > > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > > The information contained in
> this email
> > is
> > > strictly
> > > > > confidential
> > > > > > and for
> > > > > > > the use of the addressee only,
> unless
> > otherwise
> > > > indicated.
> > > > > If
> > > > > > you are not
> > > > > > > the intended recipient, please
> do not
> > read,
> > > copy,
> > > > use or
> > > > > > disclose to others
> > > > > > > this message or any
> attachment. Please
> > also
> > > notify
> > > > the
> > > > > sender by
> > > > > > replying
> > > > > > > to this email or by telephone
> (+44(020
> > 7896
> > > 0011)
> > > > and then
> > > > > > delete the email
> > > > > > > and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that do
> > > > > not
> > > > > > relate to the
> > > > > > > official business of this
> company shall
> > be
> > > > understood as
> > > > > neither
> > > > > > given nor
> > > > > > > endorsed by it. IG is a
> trading name of
> > IG
> > > Markets
> > > > Limited
> > > > > (a
> > > > > > company
> > > > > > > registered in England and
> Wales, company
> > number
> > > > 04008957)
> > > > > and IG
> > > > > > Index
> > > > > > > Limited (a company registered
> in England
> > and
> > > Wales,
> > > > company
> > > > > > number
> > > > > > > 01190902). Registered address
> at Cannon
> > Bridge
> > > > House, 25
> > > > > Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG
> Markets Limited
> > > (register
> > > > number
> > > > > > 195355) and IG
> > > > > > > Index Limited (register number
> 114059)
> > are
> > > > authorised and
> > > > > > regulated by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email
> is strictly
> > > > confidential and
> > > > > for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you
> > > > > are not
> > > > > > the intended recipient, please do not
> read, copy,
> > use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please
> also notify
> > the
> > > sender
> > > > by
> > > > > replying
> > > > > > to this email or by telephone (+44(020
> 7896 0011)
> > and
> > > then
> > > > delete
> > > > > the email
> > > > > > and any copies of it. Opinions,
> conclusion (etc)
> > that do
> > > not
> > > > relate
> > > > > to the
> > > > > > official business of this company shall
> be
> > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of
> IG Markets
> > > Limited (a
> > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG
> > > > > Index
> > > > > > Limited (a company registered in England
> and Wales,
> > > company
> > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > > 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059)
> are
> > authorised and
> > > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is
> strictly
> > > confidential and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you
> > > > are not
> > > > > the intended recipient, please do not read,
> copy, use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also
> notify the
> > sender
> > > by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896
> 0011) and
> > then
> > > delete
> > > > the email
> > > > > and any copies of it. Opinions, conclusion
> (etc) that do
> > not
> > > relate
> > > > to the
> > > > > official business of this company shall be
> understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG
> > > > Index
> > > > > Limited (a company registered in England and
> Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited
> (register number
> > > 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you
> > > are not
> > > > the intended recipient, please do not read, copy,
> use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify
> the sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011)
> and then
> > delete
> > > the email
> > > > and any copies of it. Opinions, conclusion (etc)
> that do not
> > relate
> > > to the
> > > > official business of this company shall be
> understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355)
> > > and IG
> > > > Index Limited (register number 114059) are
> authorised and
> > regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential
> > and
> > > for the use of the addressee only, unless otherwise
> indicated. If
> > you are
> > > not the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > others this message or any attachment. Please also notify
> the sender
> > by
> > > replying to this email or by telephone (+44(020 7896 0011)
> and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to the official business of this company shall be
> understood
> > as
> > > neither given nor endorsed by it. IG is a trading name of
> IG Markets
> > > Limited (a company registered in England and Wales,
> company number
> > > 04008957) and IG Index Limited (a company registered in
> England and
> > Wales,
> > > company number 01190902). Registered address at Cannon
> Bridge House,
> > 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > 195355) and IG Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> > >
> >
> >
> > The information contained in this email is strictly confidential
> and for
> > the use of the addressee only, unless otherwise indicated. If
> you are not
> > the intended recipient, please do not read, copy, use or
> disclose to others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then
> delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company
> number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> > Index Limited (register number 114059) are authorised and
> regulated by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Becket Qin <be...@gmail.com>.
Hey Michael,
Thanks for the KIP. It looks good overall and it looks we only have few
things to agree on.
1. Regarding the mutability.
I think it would be a big convenience to have headers mutable during
certain stage in the message life cycle for the use cases you mentioned. I
agree there is a material benefit especially given that we may have to
modify the headers for each message.
That said, I also think it is fair to say that in the producer, in order to
guarantee the correctness of the entire logic, it is necessary that at some
point we need to make producer record immutable. For example we probably
don't want to see that users accidentally updated the headers when the
producer is doing the serialization or compression.
Given that, would it be possible to make Headers to be able to switch from
mutable to immutable? We have done this for the Batch in the producer. For
example, initially the headers are mutable, but after it has gone through
all the interceptors, we can call Headers.close() to make it immutable
afterwards.
On the consumer side, we can probably always leave the the ConsumerRecord
mutable because after we give the messages to the users, Kafka consumer
itself does not care about whether the headers are modified or not anymore.
So far I think we only need to do the mutable to immutable conversion. If
there are use case require immutable to mutable conversion, we may need
something more than a closable.
2. If we agree on what mentioned above, I think it probably makes sense to
put the addHeaders()/removeHeaders() methods into Headers class and just
leave the headers() method in ProducerRecord and ConsumerRecord.
3. It might be useful to have headers at MessageSet level as well so we can
avoid decompression in some cases. But given this KIP is already
complicated, I would rather leave this out of the scope and address that
later when needed, e.g. after having batch level interceptors.
Thanks,
Jiangjie (Becket) Qin
On Fri, Feb 24, 2017 at 3:56 PM, Michael Pearce <Mi...@ig.com>
wrote:
> KIP updated in response to the below comments:
>
> > 1. Is the intent of `Headers.filter` to include or exclude the
> headers
> > matching the key? Can you add a javadoc to clarify?
> > 2. The KIP mentions that we will introduce V4 of FetchRequest
> and V4 of
> > ProduceRequest. Can you change this to say that the changes will
> > piggyback
> > onto V3 of ProduceRequest and V4 of FetchRequest which were
> introduced
> > in
> > KIP-98?
>
>
>
>
> On 24/02/2017, 23:20, "Michael Pearce" <Mi...@ig.com> wrote:
>
> We’re trying to make an eco-system for people to be able to use
> headers, I think we want to ensure some least common features are supported
> and not limited.
>
>
> Some examples we have already.
>
> On consume interceptors a security interceptor may need to take the
> current header, decrypt the data and replace the token with the next token
> for the next processing, in case of a single decryption token being one
> time use only.
>
> On produce it could be the interceptors add some values in the clear
> from the systems that supply them, but later a security header interceptor
> needs to encrypt some headers, as such needs to replace the current value
> with new one.
>
> I note Radai already requested this in the thread, I assume he has
> some use case also. S
>
> Simple add / remove is a least common feature.
>
> Rgds,
> Mike
>
>
> On 24/02/2017, 23:00, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Michael,
>
> I'm not strongly opposed to them; I just don't see a lot of
> benefit. One
> thing it would be good to understand is why a consumer interceptor
> would
> need to add headers and why a producer interceptor would need to
> remove
> them. Maybe we only need the common cases?
>
> Thanks,
> Jason
>
> On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > Hi Jason,
> >
> > Sorry I thought this was the agreed compromise to provide an api
> that
> > avoid boiler plate in return for immutabilty.
> >
> > If not then mutability will be needed as a goal is to have a
> single clean
> > method call to append/remove a header.
> >
> > Cheers
> > Mike
> >
> > On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >
> > Hey Michael,
> >
> > I didn't actually comment on the new methods for
> ProducerRecord and
> > ConsumerRecord. If they only save some boilerplate, I'd just
> as well
> > not
> > have them.
> >
> > Also a couple minor comments:
> >
> > 1. Is the intent of `Headers.filter` to include or exclude
> the headers
> > matching the key? Can you add a javadoc to clarify?
> > 2. The KIP mentions that we will introduce V4 of
> FetchRequest and V4 of
> > ProduceRequest. Can you change this to say that the changes
> will
> > piggyback
> > onto V3 of ProduceRequest and V4 of FetchRequest which were
> introduced
> > in
> > KIP-98?
> >
> > The rest of the KIP looks good to me.
> >
> > -Jason
> >
> > On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > I’ve added the methods on the ProducerRecord that will
> return a new
> > > instance of ProducerRecord with modified headers.
> > >
> > > On 24/02/2017, 19:22, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > wrote:
> > >
> > > Pattern.compile is expensive, and even if cached
> String.equals is
> > > faster than matched. also if we end up with an internal
> map in
> > future for
> > > performance it will be easier to be by key.
> > >
> > > As all that's needed is to get header by key.
> > >
> > > With like the other arguements of let's implement
> simple and
> > then we
> > > can always add pattern later as well if it's found it's
> needed. (As
> > noted
> > > it's easier to add methods than to take away)
> > >
> > > Great I'll update kip with extra methods on
> producerecord and a
> > note
> > > that new objects are returned by method calls.
> > >
> > >
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Friday, February 24, 2017 6:51:45 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > The APIs in the current KIP look good to me. Just a
> couple
> > questions:
> > > why
> > > does filter not return Headers? Also would it be
> useful if the
> > key is a
> > > regex?
> > >
> > > On the point of immutability.. One option might be to
> use a
> > mutable
> > > object
> > > only when passing the headers through the interceptor
> chain. I
> > think as
> > > long as we resort to mutability only when clear
> performance
> > results
> > > show
> > > that it is worthwhile, I am satisfied. As Ismael
> noted, for
> > common
> > > scenarios it is possible to get reasonable performance
> with
> > immutable
> > > objects.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > On 1, How can you guarantee two separate
> implemented clients
> > would
> > > add
> > > > the headers in the same order we are not specifying
> an order
> > at the
> > > > protocol level (nor should we) with regards to keyA
> being
> > ordered
> > > before
> > > > keyB? We shouldn’t be expecting keyA to be always
> set before
> > keyB.
> > > >
> > > > On 2, I believe we have changed the naming based on
> feedback
> > from
> > > Jason
> > > > already, e.g. we don’t have “get” method that
> inferred O(1)
> > > performance,
> > > > like wise nor “put” but we have an “append”
> > > >
> > > > On 3, in the KafkaProducer, I think we have
> mutability
> > already, the
> > > value
> > > > for time is changed if it is null, at the point of
> send:
> > > > “
> > > > long timestamp = record.timestamp() ==
> null ?
> > > > time.milliseconds() : record.timestamp();
> > > > “
> > > >
> > > > As such the timestamp is already mutable, so what’s
> the
> > difference
> > > here,
> > > > we already have some mixed semantics. On timestamp.
> > > > e.g. currently if I send to records with timestamp
> not set,
> > the wire
> > > > binary sent the value for the timestamp would be
> different, as
> > such
> > > we have
> > > > mutation for the same record.
> > > >
> > > > On 4, I think we should not expect not 1 or 2
> headers, but
> > infact
> > > 10’s of
> > > > headers. This is the concern on immutable headers,
> whilst the
> > append
> > > > self-reference works nicely, what if someone needs
> to remove a
> > > header?
> > > >
> > > > Trying to get this moving:
> > > >
> > > > If we really wanted Immutable Headers and
> essentially you guys
> > wont
> > > give
> > > > +1 for it without.
> > > >
> > > > Whats the feeling for adding methods to
> ProducerRecord that
> > does the
> > > > boiler plate code or creating a new ProducerRecord
> with the
> > altered
> > > new
> > > > headers (appended or removed) inside. E.g.
> > > >
> > > > ProducerRecord {
> > > >
> > > >
> > > > ProducerRecord append(Iterable<Headers>
> headersToAppend){
> > > > return new ProducerRecord(key, value,
> headers.append(
> > > headersToAppend),
> > > > ….)
> > > > }
> > > >
> > > > ProducerRecord remove(Iterable<Headers>
> headersToAppend){
> > > > return new ProducerRecord(key, value,
> headers.remove(
> > > headersToAppend),
> > > > ….)
> > > > }
> > > >
> > > > }
> > > >
> > > > Were the headers methods actually returns new
> objects, and the
> > > producer
> > > > records methods create a new producer record with
> all the
> > current
> > > values,
> > > > but with the new modified headers.
> > > >
> > > > Then interceptors / code return this new object?
> > > >
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf
> of Ismael
> > Juma" <
> > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk>
> wrote:
> > > >
> > > > Hi Michael,
> > > >
> > > > Did you mean that you were happy to compromise
> to keep it
> > > mutable or
> > > > immutable? You wrote the former, but it sounded
> from the
> > > sentence that
> > > > it
> > > > could have been a typo. So, my thoughts on this
> is that
> > there
> > > are a few
> > > > things to take into account:
> > > >
> > > > 1. Semantics
> > > > 2. Simplicity of use (the common operations
> should be easy
> > to do)
> > > > 3. If it's easy to reason about and safe
> (immutability
> > helps
> > > with this)
> > > > 4. Efficiency (both memory and CPU usage)
> > > >
> > > > Regarding 1, I think it would be good to be very
> clear
> > about the
> > > > guarantees
> > > > that we are providing. It seems that we are
> saying that
> > keys are
> > > > unordered,
> > > > but what about the case where there are multiple
> values
> > for the
> > > same
> > > > key?
> > > > It seems that for some use cases (e.g. lineage),
> it may be
> > > useful to
> > > > add
> > > > values to the same key while preserving the
> order.
> > > >
> > > > Regarding 2, I agree that it's useful to have
> methods in
> > > `Headers` for
> > > > the
> > > > very common use cases although we have to be
> careful with
> > the
> > > naming to
> > > > avoid giving the wrong impression. Also, when it
> comes to
> > > `Map.Entry`,
> > > > I
> > > > think I'd prefer a `toMap` method that simply
> gives the
> > user a
> > > Map if
> > > > that's what they want (it makes it clear that
> there's a
> > > conversion
> > > > happening).
> > > >
> > > > Regarding 3, the concern I have if we make the
> headers
> > mutable
> > > is that
> > > > it
> > > > seems to introduce some inconsistencies and
> potential edge
> > > cases. At
> > > > the
> > > > moment, it's safe to keep a ProducerRecord and
> resend it,
> > for
> > > example.
> > > > If
> > > > the record is mutated by an interceptor, then
> this can
> > lead to
> > > weird
> > > > behaviour. Also, it seems inconsistent that one
> has to
> > create a
> > > new
> > > > ProducerRecord to modify the record timestamp,
> but that
> > one has
> > > to
> > > > mutate
> > > > the record to add headers. It seems like one
> should either
> > > embrace
> > > > immutability or mutability, but mixing both is
> not ideal.
> > > >
> > > > Regarding 4, for the cases where there are a
> small number
> > of
> > > headers
> > > > and/or
> > > > 1 or 2 interceptors, it doesn't seem difficult
> to come up
> > with
> > > > reasonable
> > > > implementations for mutable and immutable cases
> that do a
> > good
> > > enough
> > > > job.
> > > > However, if the number of headers and
> interceptors is
> > large,
> > > then care
> > > > is
> > > > needed for the immutable case to avoid
> unnecessary
> > copying. A
> > > simple
> > > > option
> > > > that adds little memory overhead if we have
> Header
> > instances is
> > > to
> > > > simply
> > > > add a self-reference to the previous Header in
> the linked
> > list.
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Im happy to compromise to keep it mutable but
> move to an
> > append
> > > > style api.
> > > > > (as in guava interables concat)
> > > > >
> > > > > class Headers {
> > > > > Headers append(Iterable<Header>
> headers);
> > > > > }
> > > > >
> > > > >
> > > > > I don’t think we’d want prepend, this would
> give the
> > idea of
> > > > guaranteed
> > > > > ordering, when in actual fact we don’t provide
> that
> > guarantee
> > > (.e.g
> > > > one
> > > > > client can put headerA, then headerB, but
> another could
> > put
> > > headerB
> > > > then
> > > > > headerA, this shouldn’t cause issues), Also
> what if we
> > changed
> > > to a
> > > > hashmap
> > > > > for the internal implementation, its just a
> bucket of
> > entries
> > > no
> > > > ordering.
> > > > > I think we just need to provide an api to
> add/append
> > headers.
> > > > >
> > > > > This ok? If so ill update KIP to record this.
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> > jason@confluent.io>
> > > wrote:
> > > > >
> > > > > The point about usability is fair. It's
> also
> > reasonable to
> > > > expect that
> > > > > common use cases such as appending headers
> should be
> > done
> > > > efficiently.
> > > > >
> > > > > Perhaps we could compromise with something
> like this?
> > > > >
> > > > > class Headers {
> > > > > Headers append(Iterable<Header> headers);
> > > > > Headers prepend(Iterable<Header> headers);
> > > > > }
> > > > >
> > > > > That retains ease of use while still giving
> > ourselves some
> > > > flexibility
> > > > > in
> > > > > the implementation.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > > On Wed, Feb 22, 2017 at 3:03 PM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I wasn’t referring to the headers
> needing to be
> > copied,
> > > im
> > > > meaning
> > > > > the
> > > > > > fact we’d be forcing a new producer
> record to be
> > > created, with
> > > > all
> > > > > the
> > > > > > contents copied.
> > > > > >
> > > > > > i.e what will happen is utility method
> will be
> > created
> > > or end
> > > > up
> > > > > being
> > > > > > used, which does this, and returns the
> new
> > ProducerRecord
> > > > instance.
> > > > > >
> > > > > > ProducerRecord addHeader(ProducerRecord
> record,
> > Header
> > > > header){
> > > > > > Return New ProducerRecord(record.key,
> record.value,
> > > > > record.timestamp…..,
> > > > > > record.headers.concat(header))
> > > > > > }
> > > > > >
> > > > > > To me this seems ugly, but will be
> inevitable if we
> > > don’t make
> > > > adding
> > > > > > headers to existing records a simple
> clean method
> > call.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 22:57, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > Lazy init can achieve/avoid that.
> > > > > >
> > > > > > Re the concat, why don’t we
> implement that
> > inside the
> > > > Headers
> > > > > rather
> > > > > > than causing everyone to implement this
> as adding
> > > headers in
> > > > > interceptors
> > > > > > will be a dominant use case. We want a
> user
> > friendly API.
> > > > Having as
> > > > > a user
> > > > > > having to code this instead of having
> the headers
> > handle
> > > this
> > > > for me
> > > > > seems
> > > > > > redundant.
> > > > > >
> > > > > > On 22/02/2017, 22:34, "Jason
> Gustafson" <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > I thought the argument was
> against
> > creating the
> > > extra
> > > > objects
> > > > > > unnecessarily
> > > > > > (i.e. if they were not
> accessed). And note
> > that
> > > making
> > > > the
> > > > > Headers
> > > > > > immutable doesn't necessarily
> mean that
> > they
> > > need to be
> > > > > copied:
> > > > > > you can do
> > > > > > a trick like Guava's
> Iterables.concat to
> > add
> > > additional
> > > > > headers
> > > > > > without
> > > > > > changing the underlying
> collections.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 2:22 PM,
> Michael
> > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > If the argument for not having
> a map
> > holding
> > > the
> > > > key, value
> > > > > > pairs is due
> > > > > > > to garbage creation of HashMap
> entry's,
> > > forcing the
> > > > > creation of
> > > > > > a whole new
> > > > > > > producer record to simply add
> a head,
> > surely is
> > > > creating
> > > > > a-lot
> > > > > > more?
> > > > > > > ______________________________
> __________
> > > > > > > From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > Sent: Wednesday, February 22,
> 2017 10:09
> > PM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-82
> - Add
> > Record
> > > Headers
> > > > > > >
> > > > > > > The current producer
> interceptor API is
> > this:
> > > > > > >
> > > > > > > ProducerRecord<K, V>
> > onSend(ProducerRecord<K,
> > > V>
> > > > record);
> > > > > > >
> > > > > > > So adding a header means
> creating a new
> > > > ProducerRecord
> > > > > with a
> > > > > > new header
> > > > > > > added to the current headers
> and
> > returning it.
> > > Would
> > > > that
> > > > > not
> > > > > > work?
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 1:45
> PM, Michael
> > > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > So how would you have this
> work if not
> > > mutable
> > > > where
> > > > > > interceptors would
> > > > > > > > add headers?
> > > > > > > >
> > > > > > > > Sent using OWA for iPhone
> > > > > > > >
> ______________________________
> > __________
> > > > > > > > From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > > Sent: Wednesday, February
> 22, 2017
> > 8:42:27 PM
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Subject: Re: [DISCUSS]
> KIP-82 - Add
> > Record
> > > Headers
> > > > > > > >
> > > > > > > > I think the point on the
> mutability of
> > > Headers is
> > > > worth
> > > > > > discussing a
> > > > > > > little
> > > > > > > > more. As far as I can tell,
> once the
> > > > ProducerRecord (or
> > > > > > ConsumerRecord)
> > > > > > > is
> > > > > > > > constructed, there should be
> no need to
> > > further
> > > > change
> > > > > the
> > > > > > headers. Is
> > > > > > > that
> > > > > > > > correct? If so, then why not
> enforce
> > that
> > > that is
> > > > the
> > > > > case
> > > > > > through the
> > > > > > > API?
> > > > > > > > One problem with mutability
> it that it
> > > constrains
> > > > the
> > > > > > implementation of
> > > > > > > > Headers. For example, if we
> were
> > backing
> > > with a
> > > > byte
> > > > > slice,
> > > > > > would we
> > > > > > > recopy
> > > > > > > > the bytes if a header is
> added or
> > would we
> > > > maintain a
> > > > > satellite
> > > > > > > collection
> > > > > > > > of added records. Seems not
> great
> > either
> > > way. If we
> > > > > really
> > > > > > think
> > > > > > > mutability
> > > > > > > > is needed, perhaps we could
> add a
> > method to
> > > > headers to
> > > > > convert
> > > > > > it to a
> > > > > > > > mutable type (e.g. a
> Headers.toMap)?
> > > > > > > >
> > > > > > > > I'm also with Ismael about
> exposing
> > > Headers.get().
> > > > I
> > > > > thought
> > > > > > it might
> > > > > > > make
> > > > > > > > sense to have a method like
> this
> > instead:
> > > > > > > >
> > > > > > > > Iterable<Header>
> findMatching(Pattern
> > > pattern);
> > > > > > > >
> > > > > > > > This makes the (potential)
> need to
> > scan the
> > > headers
> > > > > clear in
> > > > > > the API. I'd
> > > > > > > > also be fine exposing no
> getter at
> > all. In
> > > > general, Ï
> > > > > think
> > > > > > it's good to
> > > > > > > > start with a minimalistic
> API and work
> > from
> > > there
> > > > since
> > > > > it's
> > > > > > always
> > > > > > > easier
> > > > > > > > to add methods than remove
> them.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Wed, Feb 22, 2017 at 9:16
> AM,
> > Michael
> > > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi Ismael
> > > > > > > > >
> > > > > > > > > On point 1,
> > > > > > > > >
> > > > > > > > > Sure makes sense will
> update shortly.
> > > > > > > > >
> > > > > > > > > On point 2,
> > > > > > > > >
> > > > > > > > > Setter/getter typical to
> > > properties/headers api’s
> > > > > > traditionally are map
> > > > > > > > > styled interfaces and what
> I believe
> > is
> > > most
> > > > expected
> > > > > styled
> > > > > > thus the
> > > > > > > > Key,
> > > > > > > > > Value setter.
> > > > > > > > > Also it would mean rather
> than an
> > > interface, we
> > > > would
> > > > > be
> > > > > > making our
> > > > > > > > > internal header impl
> object we have
> > for the
> > > > array,
> > > > > exposed.
> > > > > > E.g. if we
> > > > > > > > had
> > > > > > > > > a Map really this would be
> Map.Entry
> > > interface,
> > > > this
> > > > > is the
> > > > > > same
> > > > > > > reasons
> > > > > > > > on
> > > > > > > > > the map interface I cannot
> actually
> > make
> > > the
> > > > > underlying Node
> > > > > > object
> > > > > > > > that’s
> > > > > > > > > the implementation for
> HashMap, so
> > that
> > > > internals can
> > > > > be
> > > > > > changed.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On point 3,
> > > > > > > > >
> > > > > > > > > I think it people do
> expect it to be
> > > performant,
> > > > thus
> > > > > why
> > > > > > originally
> > > > > > > > > concern I raised with
> using an array
> > for
> > > to me
> > > > is an
> > > > > early
> > > > > > memory
> > > > > > > > > optimisation. I think the
> user
> > experience
> > > of
> > > > > > properties/headers is on a
> > > > > > > > > get/set model. This is why
> its
> > important
> > > we have
> > > > > > encapsulated logic
> > > > > > > that
> > > > > > > > > then allows us to change
> this to a
> > map, if
> > > this
> > > > > becomes and
> > > > > > issue, and
> > > > > > > > the
> > > > > > > > > memory overhead of hashmap
> is less
> > so.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 22/02/2017, 15:56, "
> > ismaelj@gmail.com
> > > on
> > > > behalf of
> > > > > > Ismael Juma" <
> > > > > > > > > ismaelj@gmail.com on
> behalf of
> > > ismael@juma.me.uk
> > > > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Great to see the
> progress that
> > has been
> > > > achieved
> > > > > on this
> > > > > > one. :) A
> > > > > > > > few
> > > > > > > > > comments regarding the
> APIs (I'm
> > still
> > > > reviewing
> > > > > the
> > > > > > message format
> > > > > > > > > changes):
> > > > > > > > >
> > > > > > > > > 1. Nit: `getHeaders` in
> > > `ProducerRecord` and
> > > > > > `ConsumerRecord`
> > > > > > > should
> > > > > > > > be
> > > > > > > > > named `headers` (we
> avoid the
> > `get`
> > > prefix in
> > > > > Kafka)
> > > > > > > > >
> > > > > > > > > 2. The `Headers` class
> is mutable
> > > (there's
> > > > an `add`
> > > > > > method). Does
> > > > > > > it
> > > > > > > > > need
> > > > > > > > > to be? If so, it would
> be good to
> > > explain
> > > > why.
> > > > > Related
> > > > > > to that, we
> > > > > > > > > should
> > > > > > > > > also explain the
> thinking around
> > > > thread-safety. If
> > > > > we
> > > > > > keep the
> > > > > > > `add`
> > > > > > > > > method, it may make
> sense for it
> > to
> > > take a
> > > > `Header`
> > > > > > (that way we
> > > > > > > can
> > > > > > > > > add
> > > > > > > > > things to `Header`
> without
> > changing the
> > > > interface).
> > > > > > > > >
> > > > > > > > > 3. Do we need the
> `Headers.get()`
> > > method?
> > > > People
> > > > > usually
> > > > > > assume
> > > > > > > that
> > > > > > > > > `get`
> > > > > > > > > would be efficient, but
> > depending on
> > > the
> > > > > implementation
> > > > > > (the
> > > > > > > current
> > > > > > > > > proposal states that
> an array
> > would be
> > > > used), it
> > > > > may not
> > > > > > be. If we
> > > > > > > > > expect
> > > > > > > > > the number of headers
> to be
> > small, it
> > > doesn't
> > > > > matter
> > > > > > though.
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Tue, Feb 21, 2017
> at 6:38 PM,
> > > Michael
> > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jason,
> > > > > > > > > >
> > > > > > > > > > Have converted the
> > interface/api
> > > bullets
> > > > into
> > > > > > interface code
> > > > > > > > > snippets.
> > > > > > > > > >
> > > > > > > > > > Agreed
> implementation won’t
> > take too
> > > long.
> > > > We
> > > > > have
> > > > > > early versions
> > > > > > > > > already.
> > > > > > > > > > Maybe a week before
> you think
> > about
> > > > merging I
> > > > > would
> > > > > > assume it
> > > > > > > would
> > > > > > > > > be more
> > > > > > > > > > stabilised? I was
> thinking
> > then we
> > > could
> > > > fork
> > > > > from
> > > > > > your confluent
> > > > > > > > > branch,
> > > > > > > > > > making and then
> holding KIP-82
> > > changes in
> > > > a patch
> > > > > > file, that we
> > > > > > > can
> > > > > > > > > then
> > > > > > > > > > re-fork from apache
> once KIP98
> > final
> > > is
> > > > merged,
> > > > > and
> > > > > > apply patch
> > > > > > > > with
> > > > > > > > > last
> > > > > > > > > > minute changes.
> > > > > > > > > >
> > > > > > > > > > Cheers
> > > > > > > > > > Mike
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 22/02/2017,
> 00:56, "Jason
> > > Gustafson" <
> > > > > > jason@confluent.io>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hey Michael,
> > > > > > > > > >
> > > > > > > > > > Awesome. I have
> a minor
> > request.
> > > The
> > > > APIs are
> > > > > > currently
> > > > > > > > > documented as a
> > > > > > > > > > wiki list. Would
> you mind
> > adding
> > > a code
> > > > > snippet
> > > > > > instead?
> > > > > > > It's a
> > > > > > > > > bit
> > > > > > > > > > easier
> > > > > > > > > > to process.
> > > > > > > > > >
> > > > > > > > > > How will be best
> to manage
> > this,
> > > as we
> > > > will
> > > > > > obviously build
> > > > > > > off
> > > > > > > > > your
> > > > > > > > > > KIP’s
> > > > > > > > > > > protocol
> changes, to
> > avoid a
> > > merge
> > > > hell,
> > > > > should
> > > > > > we branch
> > > > > > > > from
> > > > > > > > > your
> > > > > > > > > > branch
> > > > > > > > > > > in the
> confluent repo or
> > is it
> > > worth
> > > > > having a
> > > > > > KIP-98
> > > > > > > special
> > > > > > > > > branch
> > > > > > > > > > in the
> > > > > > > > > > > apache git,
> that we can
> > > branch/fork
> > > > from?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I was thinking
> about this
> > also.
> > > > Ideally we'd
> > > > > like
> > > > > > to get the
> > > > > > > > > changes
> > > > > > > > > > in as
> > > > > > > > > > close together
> as possible
> > since
> > > we
> > > > only
> > > > > want one
> > > > > > magic bump
> > > > > > > > and
> > > > > > > > > some
> > > > > > > > > > users
> > > > > > > > > > deploy trunk.
> The level of
> > > effort to
> > > > change
> > > > > the
> > > > > > format for
> > > > > > > > > headers
> > > > > > > > > > seems
> > > > > > > > > > not too high. Do
> you
> > agree? My
> > > guess
> > > > is that
> > > > > the
> > > > > > KIP-98
> > > > > > > message
> > > > > > > > > format
> > > > > > > > > > patch will take
> 2-3 weeks
> > to
> > > review
> > > > before we
> > > > > > merge to trunk,
> > > > > > > > so
> > > > > > > > > you
> > > > > > > > > > could
> > > > > > > > > > hold off
> implementing
> > until that
> > > patch
> > > > has
> > > > > somewhat
> > > > > > > stabilized.
> > > > > > > > > That
> > > > > > > > > > would
> > > > > > > > > > save some
> potential rebase
> > pain.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The information
> contained in
> > this
> > > email is
> > > > > strictly
> > > > > > confidential
> > > > > > > > and
> > > > > > > > > for
> > > > > > > > > > the use of the
> addressee only,
> > unless
> > > > otherwise
> > > > > > indicated. If you
> > > > > > > > > are not
> > > > > > > > > > the intended
> recipient, please
> > do
> > > not read,
> > > > > copy, use
> > > > > > or disclose
> > > > > > > > to
> > > > > > > > > others
> > > > > > > > > > this message or any
> attachment.
> > > Please also
> > > > > notify the
> > > > > > sender by
> > > > > > > > > replying
> > > > > > > > > > to this email or by
> telephone
> > > (+44(020
> > > > 7896 0011)
> > > > > and
> > > > > > then delete
> > > > > > > > the
> > > > > > > > > > email and any copies
> of it.
> > Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that do
> > > > > > > not
> > > > > > > > > relate
> > > > > > > > > > to the official
> business of
> > this
> > > company
> > > > shall be
> > > > > > understood as
> > > > > > > > > neither
> > > > > > > > > > given nor endorsed
> by it. IG
> > is a
> > > trading
> > > > name
> > > > > of IG
> > > > > > Markets
> > > > > > > > Limited
> > > > > > > > > (a
> > > > > > > > > > company registered
> in England
> > and
> > > Wales,
> > > > company
> > > > > > number 04008957)
> > > > > > > > > and IG
> > > > > > > > > > Index Limited (a
> company
> > registered
> > > in
> > > > England
> > > > > and
> > > > > > Wales, company
> > > > > > > > > number
> > > > > > > > > > 01190902).
> Registered address
> > at
> > > Cannon
> > > > Bridge
> > > > > House,
> > > > > > 25 Dowgate
> > > > > > > > > Hill,
> > > > > > > > > > London EC4R 2YA.
> Both IG
> > Markets
> > > Limited
> > > > > (register
> > > > > > number 195355)
> > > > > > > > > and IG
> > > > > > > > > > Index Limited
> (register number
> > > 114059) are
> > > > > authorised
> > > > > > and
> > > > > > > regulated
> > > > > > > > > by the
> > > > > > > > > > Financial Conduct
> Authority.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information contained
> in this
> > email is
> > > > strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > > > the use of the addressee
> only, unless
> > > otherwise
> > > > > indicated.
> > > > > > If you are
> > > > > > > not
> > > > > > > > > the intended recipient,
> please do
> > not read,
> > > > copy, use
> > > > > or
> > > > > > disclose to
> > > > > > > > others
> > > > > > > > > this message or any
> attachment.
> > Please also
> > > > notify the
> > > > > > sender by
> > > > > > > replying
> > > > > > > > > to this email or by
> telephone
> > (+44(020 7896
> > > > 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > > email
> > > > > > > > > and any copies of it.
> Opinions,
> > conclusion
> > > (etc)
> > > > that
> > > > > do not
> > > > > > relate to
> > > > > > > > the
> > > > > > > > > official business of this
> company
> > shall be
> > > > understood
> > > > > as
> > > > > > neither given
> > > > > > > > nor
> > > > > > > > > endorsed by it. IG is a
> trading name
> > of IG
> > > > Markets
> > > > > Limited
> > > > > > (a company
> > > > > > > > > registered in England and
> Wales,
> > company
> > > number
> > > > > 04008957)
> > > > > > and IG Index
> > > > > > > > > Limited (a company
> registered in
> > England
> > > and
> > > > Wales,
> > > > > company
> > > > > > number
> > > > > > > > > 01190902). Registered
> address at
> > Cannon
> > > Bridge
> > > > House,
> > > > > 25
> > > > > > Dowgate Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> Markets
> > Limited
> > > > (register
> > > > > number
> > > > > > 195355) and
> > > > > > > IG
> > > > > > > > > Index Limited (register
> number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > > the
> > > > > > > > > Financial Conduct
> Authority.
> > > > > > > > >
> > > > > > > > The information contained in
> this
> > email is
> > > strictly
> > > > > > confidential and for
> > > > > > > > the use of the addressee
> only, unless
> > > otherwise
> > > > > indicated. If
> > > > > > you are not
> > > > > > > > the intended recipient,
> please do not
> > read,
> > > copy,
> > > > use or
> > > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any
> attachment. Please
> > also
> > > notify
> > > > the
> > > > > sender
> > > > > > by replying
> > > > > > > > to this email or by
> telephone (+44(020
> > 7896
> > > 0011)
> > > > and
> > > > > then
> > > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it.
> Opinions,
> > conclusion
> > > (etc)
> > > > that do
> > > > > not
> > > > > > relate to
> > > > > > > the
> > > > > > > > official business of this
> company
> > shall be
> > > > understood as
> > > > > > neither given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a
> trading name
> > of IG
> > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > > registered in England and
> Wales,
> > company
> > > number
> > > > > 04008957) and
> > > > > > IG Index
> > > > > > > > Limited (a company
> registered in
> > England and
> > > Wales,
> > > > > company
> > > > > > number
> > > > > > > > 01190902). Registered
> address at Cannon
> > > Bridge
> > > > House, 25
> > > > > > Dowgate Hill,
> > > > > > > > London EC4R 2YA. Both IG
> Markets
> > Limited
> > > (register
> > > > number
> > > > > > 195355) and IG
> > > > > > > > Index Limited (register
> number 114059)
> > are
> > > > authorised and
> > > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > > The information contained in
> this email
> > is
> > > strictly
> > > > > confidential
> > > > > > and for
> > > > > > > the use of the addressee only,
> unless
> > otherwise
> > > > indicated.
> > > > > If
> > > > > > you are not
> > > > > > > the intended recipient, please
> do not
> > read,
> > > copy,
> > > > use or
> > > > > > disclose to others
> > > > > > > this message or any
> attachment. Please
> > also
> > > notify
> > > > the
> > > > > sender by
> > > > > > replying
> > > > > > > to this email or by telephone
> (+44(020
> > 7896
> > > 0011)
> > > > and then
> > > > > > delete the email
> > > > > > > and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that do
> > > > > not
> > > > > > relate to the
> > > > > > > official business of this
> company shall
> > be
> > > > understood as
> > > > > neither
> > > > > > given nor
> > > > > > > endorsed by it. IG is a
> trading name of
> > IG
> > > Markets
> > > > Limited
> > > > > (a
> > > > > > company
> > > > > > > registered in England and
> Wales, company
> > number
> > > > 04008957)
> > > > > and IG
> > > > > > Index
> > > > > > > Limited (a company registered
> in England
> > and
> > > Wales,
> > > > company
> > > > > > number
> > > > > > > 01190902). Registered address
> at Cannon
> > Bridge
> > > > House, 25
> > > > > Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG
> Markets Limited
> > > (register
> > > > number
> > > > > > 195355) and IG
> > > > > > > Index Limited (register number
> 114059)
> > are
> > > > authorised and
> > > > > > regulated by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email
> is strictly
> > > > confidential and
> > > > > for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you
> > > > > are not
> > > > > > the intended recipient, please do not
> read, copy,
> > use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please
> also notify
> > the
> > > sender
> > > > by
> > > > > replying
> > > > > > to this email or by telephone (+44(020
> 7896 0011)
> > and
> > > then
> > > > delete
> > > > > the email
> > > > > > and any copies of it. Opinions,
> conclusion (etc)
> > that do
> > > not
> > > > relate
> > > > > to the
> > > > > > official business of this company shall
> be
> > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of
> IG Markets
> > > Limited (a
> > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG
> > > > > Index
> > > > > > Limited (a company registered in England
> and Wales,
> > > company
> > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > > 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059)
> are
> > authorised and
> > > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is
> strictly
> > > confidential and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you
> > > > are not
> > > > > the intended recipient, please do not read,
> copy, use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also
> notify the
> > sender
> > > by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896
> 0011) and
> > then
> > > delete
> > > > the email
> > > > > and any copies of it. Opinions, conclusion
> (etc) that do
> > not
> > > relate
> > > > to the
> > > > > official business of this company shall be
> understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG
> > > > Index
> > > > > Limited (a company registered in England and
> Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited
> (register number
> > > 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you
> > > are not
> > > > the intended recipient, please do not read, copy,
> use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify
> the sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011)
> and then
> > delete
> > > the email
> > > > and any copies of it. Opinions, conclusion (etc)
> that do not
> > relate
> > > to the
> > > > official business of this company shall be
> understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355)
> > > and IG
> > > > Index Limited (register number 114059) are
> authorised and
> > regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential
> > and
> > > for the use of the addressee only, unless otherwise
> indicated. If
> > you are
> > > not the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > others this message or any attachment. Please also notify
> the sender
> > by
> > > replying to this email or by telephone (+44(020 7896 0011)
> and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to the official business of this company shall be
> understood
> > as
> > > neither given nor endorsed by it. IG is a trading name of
> IG Markets
> > > Limited (a company registered in England and Wales,
> company number
> > > 04008957) and IG Index Limited (a company registered in
> England and
> > Wales,
> > > company number 01190902). Registered address at Cannon
> Bridge House,
> > 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > 195355) and IG Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> > >
> >
> >
> > The information contained in this email is strictly confidential
> and for
> > the use of the addressee only, unless otherwise indicated. If
> you are not
> > the intended recipient, please do not read, copy, use or
> disclose to others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then
> delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company
> number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> > Index Limited (register number 114059) are authorised and
> regulated by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
KIP updated in response to the below comments:
> 1. Is the intent of `Headers.filter` to include or exclude the headers
> matching the key? Can you add a javadoc to clarify?
> 2. The KIP mentions that we will introduce V4 of FetchRequest and V4 of
> ProduceRequest. Can you change this to say that the changes will
> piggyback
> onto V3 of ProduceRequest and V4 of FetchRequest which were introduced
> in
> KIP-98?
On 24/02/2017, 23:20, "Michael Pearce" <Mi...@ig.com> wrote:
We’re trying to make an eco-system for people to be able to use headers, I think we want to ensure some least common features are supported and not limited.
Some examples we have already.
On consume interceptors a security interceptor may need to take the current header, decrypt the data and replace the token with the next token for the next processing, in case of a single decryption token being one time use only.
On produce it could be the interceptors add some values in the clear from the systems that supply them, but later a security header interceptor needs to encrypt some headers, as such needs to replace the current value with new one.
I note Radai already requested this in the thread, I assume he has some use case also. S
Simple add / remove is a least common feature.
Rgds,
Mike
On 24/02/2017, 23:00, "Jason Gustafson" <ja...@confluent.io> wrote:
Hey Michael,
I'm not strongly opposed to them; I just don't see a lot of benefit. One
thing it would be good to understand is why a consumer interceptor would
need to add headers and why a producer interceptor would need to remove
them. Maybe we only need the common cases?
Thanks,
Jason
On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jason,
>
> Sorry I thought this was the agreed compromise to provide an api that
> avoid boiler plate in return for immutabilty.
>
> If not then mutability will be needed as a goal is to have a single clean
> method call to append/remove a header.
>
> Cheers
> Mike
>
> On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Michael,
>
> I didn't actually comment on the new methods for ProducerRecord and
> ConsumerRecord. If they only save some boilerplate, I'd just as well
> not
> have them.
>
> Also a couple minor comments:
>
> 1. Is the intent of `Headers.filter` to include or exclude the headers
> matching the key? Can you add a javadoc to clarify?
> 2. The KIP mentions that we will introduce V4 of FetchRequest and V4 of
> ProduceRequest. Can you change this to say that the changes will
> piggyback
> onto V3 of ProduceRequest and V4 of FetchRequest which were introduced
> in
> KIP-98?
>
> The rest of the KIP looks good to me.
>
> -Jason
>
> On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > I’ve added the methods on the ProducerRecord that will return a new
> > instance of ProducerRecord with modified headers.
> >
> > On 24/02/2017, 19:22, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Pattern.compile is expensive, and even if cached String.equals is
> > faster than matched. also if we end up with an internal map in
> future for
> > performance it will be easier to be by key.
> >
> > As all that's needed is to get header by key.
> >
> > With like the other arguements of let's implement simple and
> then we
> > can always add pattern later as well if it's found it's needed. (As
> noted
> > it's easier to add methods than to take away)
> >
> > Great I'll update kip with extra methods on producerecord and a
> note
> > that new objects are returned by method calls.
> >
> >
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Friday, February 24, 2017 6:51:45 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > The APIs in the current KIP look good to me. Just a couple
> questions:
> > why
> > does filter not return Headers? Also would it be useful if the
> key is a
> > regex?
> >
> > On the point of immutability.. One option might be to use a
> mutable
> > object
> > only when passing the headers through the interceptor chain. I
> think as
> > long as we resort to mutability only when clear performance
> results
> > show
> > that it is worthwhile, I am satisfied. As Ismael noted, for
> common
> > scenarios it is possible to get reasonable performance with
> immutable
> > objects.
> >
> > -Jason
> >
> > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > Hi
> > >
> > > On 1, How can you guarantee two separate implemented clients
> would
> > add
> > > the headers in the same order we are not specifying an order
> at the
> > > protocol level (nor should we) with regards to keyA being
> ordered
> > before
> > > keyB? We shouldn’t be expecting keyA to be always set before
> keyB.
> > >
> > > On 2, I believe we have changed the naming based on feedback
> from
> > Jason
> > > already, e.g. we don’t have “get” method that inferred O(1)
> > performance,
> > > like wise nor “put” but we have an “append”
> > >
> > > On 3, in the KafkaProducer, I think we have mutability
> already, the
> > value
> > > for time is changed if it is null, at the point of send:
> > > “
> > > long timestamp = record.timestamp() == null ?
> > > time.milliseconds() : record.timestamp();
> > > “
> > >
> > > As such the timestamp is already mutable, so what’s the
> difference
> > here,
> > > we already have some mixed semantics. On timestamp.
> > > e.g. currently if I send to records with timestamp not set,
> the wire
> > > binary sent the value for the timestamp would be different, as
> such
> > we have
> > > mutation for the same record.
> > >
> > > On 4, I think we should not expect not 1 or 2 headers, but
> infact
> > 10’s of
> > > headers. This is the concern on immutable headers, whilst the
> append
> > > self-reference works nicely, what if someone needs to remove a
> > header?
> > >
> > > Trying to get this moving:
> > >
> > > If we really wanted Immutable Headers and essentially you guys
> wont
> > give
> > > +1 for it without.
> > >
> > > Whats the feeling for adding methods to ProducerRecord that
> does the
> > > boiler plate code or creating a new ProducerRecord with the
> altered
> > new
> > > headers (appended or removed) inside. E.g.
> > >
> > > ProducerRecord {
> > >
> > >
> > > ProducerRecord append(Iterable<Headers> headersToAppend){
> > > return new ProducerRecord(key, value, headers.append(
> > headersToAppend),
> > > ….)
> > > }
> > >
> > > ProducerRecord remove(Iterable<Headers> headersToAppend){
> > > return new ProducerRecord(key, value, headers.remove(
> > headersToAppend),
> > > ….)
> > > }
> > >
> > > }
> > >
> > > Were the headers methods actually returns new objects, and the
> > producer
> > > records methods create a new producer record with all the
> current
> > values,
> > > but with the new modified headers.
> > >
> > > Then interceptors / code return this new object?
> > >
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael
> Juma" <
> > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > >
> > > Hi Michael,
> > >
> > > Did you mean that you were happy to compromise to keep it
> > mutable or
> > > immutable? You wrote the former, but it sounded from the
> > sentence that
> > > it
> > > could have been a typo. So, my thoughts on this is that
> there
> > are a few
> > > things to take into account:
> > >
> > > 1. Semantics
> > > 2. Simplicity of use (the common operations should be easy
> to do)
> > > 3. If it's easy to reason about and safe (immutability
> helps
> > with this)
> > > 4. Efficiency (both memory and CPU usage)
> > >
> > > Regarding 1, I think it would be good to be very clear
> about the
> > > guarantees
> > > that we are providing. It seems that we are saying that
> keys are
> > > unordered,
> > > but what about the case where there are multiple values
> for the
> > same
> > > key?
> > > It seems that for some use cases (e.g. lineage), it may be
> > useful to
> > > add
> > > values to the same key while preserving the order.
> > >
> > > Regarding 2, I agree that it's useful to have methods in
> > `Headers` for
> > > the
> > > very common use cases although we have to be careful with
> the
> > naming to
> > > avoid giving the wrong impression. Also, when it comes to
> > `Map.Entry`,
> > > I
> > > think I'd prefer a `toMap` method that simply gives the
> user a
> > Map if
> > > that's what they want (it makes it clear that there's a
> > conversion
> > > happening).
> > >
> > > Regarding 3, the concern I have if we make the headers
> mutable
> > is that
> > > it
> > > seems to introduce some inconsistencies and potential edge
> > cases. At
> > > the
> > > moment, it's safe to keep a ProducerRecord and resend it,
> for
> > example.
> > > If
> > > the record is mutated by an interceptor, then this can
> lead to
> > weird
> > > behaviour. Also, it seems inconsistent that one has to
> create a
> > new
> > > ProducerRecord to modify the record timestamp, but that
> one has
> > to
> > > mutate
> > > the record to add headers. It seems like one should either
> > embrace
> > > immutability or mutability, but mixing both is not ideal.
> > >
> > > Regarding 4, for the cases where there are a small number
> of
> > headers
> > > and/or
> > > 1 or 2 interceptors, it doesn't seem difficult to come up
> with
> > > reasonable
> > > implementations for mutable and immutable cases that do a
> good
> > enough
> > > job.
> > > However, if the number of headers and interceptors is
> large,
> > then care
> > > is
> > > needed for the immutable case to avoid unnecessary
> copying. A
> > simple
> > > option
> > > that adds little memory overhead if we have Header
> instances is
> > to
> > > simply
> > > add a self-reference to the previous Header in the linked
> list.
> > >
> > > Ismael
> > >
> > > On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Im happy to compromise to keep it mutable but move to an
> append
> > > style api.
> > > > (as in guava interables concat)
> > > >
> > > > class Headers {
> > > > Headers append(Iterable<Header> headers);
> > > > }
> > > >
> > > >
> > > > I don’t think we’d want prepend, this would give the
> idea of
> > > guaranteed
> > > > ordering, when in actual fact we don’t provide that
> guarantee
> > (.e.g
> > > one
> > > > client can put headerA, then headerB, but another could
> put
> > headerB
> > > then
> > > > headerA, this shouldn’t cause issues), Also what if we
> changed
> > to a
> > > hashmap
> > > > for the internal implementation, its just a bucket of
> entries
> > no
> > > ordering.
> > > > I think we just need to provide an api to add/append
> headers.
> > > >
> > > > This ok? If so ill update KIP to record this.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> jason@confluent.io>
> > wrote:
> > > >
> > > > The point about usability is fair. It's also
> reasonable to
> > > expect that
> > > > common use cases such as appending headers should be
> done
> > > efficiently.
> > > >
> > > > Perhaps we could compromise with something like this?
> > > >
> > > > class Headers {
> > > > Headers append(Iterable<Header> headers);
> > > > Headers prepend(Iterable<Header> headers);
> > > > }
> > > >
> > > > That retains ease of use while still giving
> ourselves some
> > > flexibility
> > > > in
> > > > the implementation.
> > > >
> > > > -Jason
> > > >
> > > >
> > > > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > I wasn’t referring to the headers needing to be
> copied,
> > im
> > > meaning
> > > > the
> > > > > fact we’d be forcing a new producer record to be
> > created, with
> > > all
> > > > the
> > > > > contents copied.
> > > > >
> > > > > i.e what will happen is utility method will be
> created
> > or end
> > > up
> > > > being
> > > > > used, which does this, and returns the new
> ProducerRecord
> > > instance.
> > > > >
> > > > > ProducerRecord addHeader(ProducerRecord record,
> Header
> > > header){
> > > > > Return New ProducerRecord(record.key, record.value,
> > > > record.timestamp…..,
> > > > > record.headers.concat(header))
> > > > > }
> > > > >
> > > > > To me this seems ugly, but will be inevitable if we
> > don’t make
> > > adding
> > > > > headers to existing records a simple clean method
> call.
> > > > >
> > > > >
> > > > >
> > > > > On 22/02/2017, 22:57, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > Lazy init can achieve/avoid that.
> > > > >
> > > > > Re the concat, why don’t we implement that
> inside the
> > > Headers
> > > > rather
> > > > > than causing everyone to implement this as adding
> > headers in
> > > > interceptors
> > > > > will be a dominant use case. We want a user
> friendly API.
> > > Having as
> > > > a user
> > > > > having to code this instead of having the headers
> handle
> > this
> > > for me
> > > > seems
> > > > > redundant.
> > > > >
> > > > > On 22/02/2017, 22:34, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > wrote:
> > > > >
> > > > > I thought the argument was against
> creating the
> > extra
> > > objects
> > > > > unnecessarily
> > > > > (i.e. if they were not accessed). And note
> that
> > making
> > > the
> > > > Headers
> > > > > immutable doesn't necessarily mean that
> they
> > need to be
> > > > copied:
> > > > > you can do
> > > > > a trick like Guava's Iterables.concat to
> add
> > additional
> > > > headers
> > > > > without
> > > > > changing the underlying collections.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 2:22 PM, Michael
> Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > If the argument for not having a map
> holding
> > the
> > > key, value
> > > > > pairs is due
> > > > > > to garbage creation of HashMap entry's,
> > forcing the
> > > > creation of
> > > > > a whole new
> > > > > > producer record to simply add a head,
> surely is
> > > creating
> > > > a-lot
> > > > > more?
> > > > > > ________________________________________
> > > > > > From: Jason Gustafson <
> jason@confluent.io>
> > > > > > Sent: Wednesday, February 22, 2017 10:09
> PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> Record
> > Headers
> > > > > >
> > > > > > The current producer interceptor API is
> this:
> > > > > >
> > > > > > ProducerRecord<K, V>
> onSend(ProducerRecord<K,
> > V>
> > > record);
> > > > > >
> > > > > > So adding a header means creating a new
> > > ProducerRecord
> > > > with a
> > > > > new header
> > > > > > added to the current headers and
> returning it.
> > Would
> > > that
> > > > not
> > > > > work?
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > So how would you have this work if not
> > mutable
> > > where
> > > > > interceptors would
> > > > > > > add headers?
> > > > > > >
> > > > > > > Sent using OWA for iPhone
> > > > > > > ______________________________
> __________
> > > > > > > From: Jason Gustafson <
> jason@confluent.io>
> > > > > > > Sent: Wednesday, February 22, 2017
> 8:42:27 PM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> Record
> > Headers
> > > > > > >
> > > > > > > I think the point on the mutability of
> > Headers is
> > > worth
> > > > > discussing a
> > > > > > little
> > > > > > > more. As far as I can tell, once the
> > > ProducerRecord (or
> > > > > ConsumerRecord)
> > > > > > is
> > > > > > > constructed, there should be no need to
> > further
> > > change
> > > > the
> > > > > headers. Is
> > > > > > that
> > > > > > > correct? If so, then why not enforce
> that
> > that is
> > > the
> > > > case
> > > > > through the
> > > > > > API?
> > > > > > > One problem with mutability it that it
> > constrains
> > > the
> > > > > implementation of
> > > > > > > Headers. For example, if we were
> backing
> > with a
> > > byte
> > > > slice,
> > > > > would we
> > > > > > recopy
> > > > > > > the bytes if a header is added or
> would we
> > > maintain a
> > > > satellite
> > > > > > collection
> > > > > > > of added records. Seems not great
> either
> > way. If we
> > > > really
> > > > > think
> > > > > > mutability
> > > > > > > is needed, perhaps we could add a
> method to
> > > headers to
> > > > convert
> > > > > it to a
> > > > > > > mutable type (e.g. a Headers.toMap)?
> > > > > > >
> > > > > > > I'm also with Ismael about exposing
> > Headers.get().
> > > I
> > > > thought
> > > > > it might
> > > > > > make
> > > > > > > sense to have a method like this
> instead:
> > > > > > >
> > > > > > > Iterable<Header> findMatching(Pattern
> > pattern);
> > > > > > >
> > > > > > > This makes the (potential) need to
> scan the
> > headers
> > > > clear in
> > > > > the API. I'd
> > > > > > > also be fine exposing no getter at
> all. In
> > > general, Ï
> > > > think
> > > > > it's good to
> > > > > > > start with a minimalistic API and work
> from
> > there
> > > since
> > > > it's
> > > > > always
> > > > > > easier
> > > > > > > to add methods than remove them.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 9:16 AM,
> Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Hi Ismael
> > > > > > > >
> > > > > > > > On point 1,
> > > > > > > >
> > > > > > > > Sure makes sense will update shortly.
> > > > > > > >
> > > > > > > > On point 2,
> > > > > > > >
> > > > > > > > Setter/getter typical to
> > properties/headers api’s
> > > > > traditionally are map
> > > > > > > > styled interfaces and what I believe
> is
> > most
> > > expected
> > > > styled
> > > > > thus the
> > > > > > > Key,
> > > > > > > > Value setter.
> > > > > > > > Also it would mean rather than an
> > interface, we
> > > would
> > > > be
> > > > > making our
> > > > > > > > internal header impl object we have
> for the
> > > array,
> > > > exposed.
> > > > > E.g. if we
> > > > > > > had
> > > > > > > > a Map really this would be Map.Entry
> > interface,
> > > this
> > > > is the
> > > > > same
> > > > > > reasons
> > > > > > > on
> > > > > > > > the map interface I cannot actually
> make
> > the
> > > > underlying Node
> > > > > object
> > > > > > > that’s
> > > > > > > > the implementation for HashMap, so
> that
> > > internals can
> > > > be
> > > > > changed.
> > > > > > > >
> > > > > > > >
> > > > > > > > On point 3,
> > > > > > > >
> > > > > > > > I think it people do expect it to be
> > performant,
> > > thus
> > > > why
> > > > > originally
> > > > > > > > concern I raised with using an array
> for
> > to me
> > > is an
> > > > early
> > > > > memory
> > > > > > > > optimisation. I think the user
> experience
> > of
> > > > > properties/headers is on a
> > > > > > > > get/set model. This is why its
> important
> > we have
> > > > > encapsulated logic
> > > > > > that
> > > > > > > > then allows us to change this to a
> map, if
> > this
> > > > becomes and
> > > > > issue, and
> > > > > > > the
> > > > > > > > memory overhead of hashmap is less
> so.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 22/02/2017, 15:56, "
> ismaelj@gmail.com
> > on
> > > behalf of
> > > > > Ismael Juma" <
> > > > > > > > ismaelj@gmail.com on behalf of
> > ismael@juma.me.uk
> > > >
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Great to see the progress that
> has been
> > > achieved
> > > > on this
> > > > > one. :) A
> > > > > > > few
> > > > > > > > comments regarding the APIs (I'm
> still
> > > reviewing
> > > > the
> > > > > message format
> > > > > > > > changes):
> > > > > > > >
> > > > > > > > 1. Nit: `getHeaders` in
> > `ProducerRecord` and
> > > > > `ConsumerRecord`
> > > > > > should
> > > > > > > be
> > > > > > > > named `headers` (we avoid the
> `get`
> > prefix in
> > > > Kafka)
> > > > > > > >
> > > > > > > > 2. The `Headers` class is mutable
> > (there's
> > > an `add`
> > > > > method). Does
> > > > > > it
> > > > > > > > need
> > > > > > > > to be? If so, it would be good to
> > explain
> > > why.
> > > > Related
> > > > > to that, we
> > > > > > > > should
> > > > > > > > also explain the thinking around
> > > thread-safety. If
> > > > we
> > > > > keep the
> > > > > > `add`
> > > > > > > > method, it may make sense for it
> to
> > take a
> > > `Header`
> > > > > (that way we
> > > > > > can
> > > > > > > > add
> > > > > > > > things to `Header` without
> changing the
> > > interface).
> > > > > > > >
> > > > > > > > 3. Do we need the `Headers.get()`
> > method?
> > > People
> > > > usually
> > > > > assume
> > > > > > that
> > > > > > > > `get`
> > > > > > > > would be efficient, but
> depending on
> > the
> > > > implementation
> > > > > (the
> > > > > > current
> > > > > > > > proposal states that an array
> would be
> > > used), it
> > > > may not
> > > > > be. If we
> > > > > > > > expect
> > > > > > > > the number of headers to be
> small, it
> > doesn't
> > > > matter
> > > > > though.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Tue, Feb 21, 2017 at 6:38 PM,
> > Michael
> > > Pearce <
> > > > > > > Michael.Pearce@ig.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jason,
> > > > > > > > >
> > > > > > > > > Have converted the
> interface/api
> > bullets
> > > into
> > > > > interface code
> > > > > > > > snippets.
> > > > > > > > >
> > > > > > > > > Agreed implementation won’t
> take too
> > long.
> > > We
> > > > have
> > > > > early versions
> > > > > > > > already.
> > > > > > > > > Maybe a week before you think
> about
> > > merging I
> > > > would
> > > > > assume it
> > > > > > would
> > > > > > > > be more
> > > > > > > > > stabilised? I was thinking
> then we
> > could
> > > fork
> > > > from
> > > > > your confluent
> > > > > > > > branch,
> > > > > > > > > making and then holding KIP-82
> > changes in
> > > a patch
> > > > > file, that we
> > > > > > can
> > > > > > > > then
> > > > > > > > > re-fork from apache once KIP98
> final
> > is
> > > merged,
> > > > and
> > > > > apply patch
> > > > > > > with
> > > > > > > > last
> > > > > > > > > minute changes.
> > > > > > > > >
> > > > > > > > > Cheers
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 22/02/2017, 00:56, "Jason
> > Gustafson" <
> > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hey Michael,
> > > > > > > > >
> > > > > > > > > Awesome. I have a minor
> request.
> > The
> > > APIs are
> > > > > currently
> > > > > > > > documented as a
> > > > > > > > > wiki list. Would you mind
> adding
> > a code
> > > > snippet
> > > > > instead?
> > > > > > It's a
> > > > > > > > bit
> > > > > > > > > easier
> > > > > > > > > to process.
> > > > > > > > >
> > > > > > > > > How will be best to manage
> this,
> > as we
> > > will
> > > > > obviously build
> > > > > > off
> > > > > > > > your
> > > > > > > > > KIP’s
> > > > > > > > > > protocol changes, to
> avoid a
> > merge
> > > hell,
> > > > should
> > > > > we branch
> > > > > > > from
> > > > > > > > your
> > > > > > > > > branch
> > > > > > > > > > in the confluent repo or
> is it
> > worth
> > > > having a
> > > > > KIP-98
> > > > > > special
> > > > > > > > branch
> > > > > > > > > in the
> > > > > > > > > > apache git, that we can
> > branch/fork
> > > from?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I was thinking about this
> also.
> > > Ideally we'd
> > > > like
> > > > > to get the
> > > > > > > > changes
> > > > > > > > > in as
> > > > > > > > > close together as possible
> since
> > we
> > > only
> > > > want one
> > > > > magic bump
> > > > > > > and
> > > > > > > > some
> > > > > > > > > users
> > > > > > > > > deploy trunk. The level of
> > effort to
> > > change
> > > > the
> > > > > format for
> > > > > > > > headers
> > > > > > > > > seems
> > > > > > > > > not too high. Do you
> agree? My
> > guess
> > > is that
> > > > the
> > > > > KIP-98
> > > > > > message
> > > > > > > > format
> > > > > > > > > patch will take 2-3 weeks
> to
> > review
> > > before we
> > > > > merge to trunk,
> > > > > > > so
> > > > > > > > you
> > > > > > > > > could
> > > > > > > > > hold off implementing
> until that
> > patch
> > > has
> > > > somewhat
> > > > > > stabilized.
> > > > > > > > That
> > > > > > > > > would
> > > > > > > > > save some potential rebase
> pain.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information contained in
> this
> > email is
> > > > strictly
> > > > > confidential
> > > > > > > and
> > > > > > > > for
> > > > > > > > > the use of the addressee only,
> unless
> > > otherwise
> > > > > indicated. If you
> > > > > > > > are not
> > > > > > > > > the intended recipient, please
> do
> > not read,
> > > > copy, use
> > > > > or disclose
> > > > > > > to
> > > > > > > > others
> > > > > > > > > this message or any attachment.
> > Please also
> > > > notify the
> > > > > sender by
> > > > > > > > replying
> > > > > > > > > to this email or by telephone
> > (+44(020
> > > 7896 0011)
> > > > and
> > > > > then delete
> > > > > > > the
> > > > > > > > > email and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc)
> > > > > that do
> > > > > > not
> > > > > > > > relate
> > > > > > > > > to the official business of
> this
> > company
> > > shall be
> > > > > understood as
> > > > > > > > neither
> > > > > > > > > given nor endorsed by it. IG
> is a
> > trading
> > > name
> > > > of IG
> > > > > Markets
> > > > > > > Limited
> > > > > > > > (a
> > > > > > > > > company registered in England
> and
> > Wales,
> > > company
> > > > > number 04008957)
> > > > > > > > and IG
> > > > > > > > > Index Limited (a company
> registered
> > in
> > > England
> > > > and
> > > > > Wales, company
> > > > > > > > number
> > > > > > > > > 01190902). Registered address
> at
> > Cannon
> > > Bridge
> > > > House,
> > > > > 25 Dowgate
> > > > > > > > Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> Markets
> > Limited
> > > > (register
> > > > > number 195355)
> > > > > > > > and IG
> > > > > > > > > Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated
> > > > > > > > by the
> > > > > > > > > Financial Conduct Authority.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this
> email is
> > > strictly
> > > > > confidential and
> > > > > > for
> > > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated.
> > > > > If you are
> > > > > > not
> > > > > > > > the intended recipient, please do
> not read,
> > > copy, use
> > > > or
> > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please also
> > > notify the
> > > > > sender by
> > > > > > replying
> > > > > > > > to this email or by telephone
> (+44(020 7896
> > > 0011) and
> > > > then
> > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that
> > > > do not
> > > > > relate to
> > > > > > > the
> > > > > > > > official business of this company
> shall be
> > > understood
> > > > as
> > > > > neither given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a trading name
> of IG
> > > Markets
> > > > Limited
> > > > > (a company
> > > > > > > > registered in England and Wales,
> company
> > number
> > > > 04008957)
> > > > > and IG Index
> > > > > > > > Limited (a company registered in
> England
> > and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > > 01190902). Registered address at
> Cannon
> > Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > > (register
> > > > number
> > > > > 195355) and
> > > > > > IG
> > > > > > > > Index Limited (register number
> 114059) are
> > > authorised
> > > > and
> > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > > The information contained in this
> email is
> > strictly
> > > > > confidential and for
> > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If
> > > > > you are not
> > > > > > > the intended recipient, please do not
> read,
> > copy,
> > > use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please
> also
> > notify
> > > the
> > > > sender
> > > > > by replying
> > > > > > > to this email or by telephone (+44(020
> 7896
> > 0011)
> > > and
> > > > then
> > > > > delete the
> > > > > > email
> > > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to
> > > > > > the
> > > > > > > official business of this company
> shall be
> > > understood as
> > > > > neither given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name
> of IG
> > Markets
> > > > Limited (a
> > > > > company
> > > > > > > registered in England and Wales,
> company
> > number
> > > > 04008957) and
> > > > > IG Index
> > > > > > > Limited (a company registered in
> England and
> > Wales,
> > > > company
> > > > > number
> > > > > > > 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > > Dowgate Hill,
> > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > (register
> > > number
> > > > > 195355) and IG
> > > > > > > Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > > The information contained in this email
> is
> > strictly
> > > > confidential
> > > > > and for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated.
> > > > If
> > > > > you are not
> > > > > > the intended recipient, please do not
> read,
> > copy,
> > > use or
> > > > > disclose to others
> > > > > > this message or any attachment. Please
> also
> > notify
> > > the
> > > > sender by
> > > > > replying
> > > > > > to this email or by telephone (+44(020
> 7896
> > 0011)
> > > and then
> > > > > delete the email
> > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to the
> > > > > > official business of this company shall
> be
> > > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of
> IG
> > Markets
> > > Limited
> > > > (a
> > > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957)
> > > > and IG
> > > > > Index
> > > > > > Limited (a company registered in England
> and
> > Wales,
> > > company
> > > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > > 195355) and IG
> > > > > > Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > > confidential and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you
> > > > are not
> > > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify
> the
> > sender
> > > by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then
> > > delete
> > > > the email
> > > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate
> > > > to the
> > > > > official business of this company shall be
> understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG
> > > > Index
> > > > > Limited (a company registered in England and Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you
> > > are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> > > the email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> > > to the
> > > > official business of this company shall be understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential
> and
> > for the use of the addressee only, unless otherwise indicated. If
> you are
> > not the intended recipient, please do not read, copy, use or
> disclose to
> > others this message or any attachment. Please also notify the sender
> by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the official business of this company shall be understood
> as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and IG Index Limited (register number 114059) are authorised
> and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
We’re trying to make an eco-system for people to be able to use headers, I think we want to ensure some least common features are supported and not limited.
Some examples we have already.
On consume interceptors a security interceptor may need to take the current header, decrypt the data and replace the token with the next token for the next processing, in case of a single decryption token being one time use only.
On produce it could be the interceptors add some values in the clear from the systems that supply them, but later a security header interceptor needs to encrypt some headers, as such needs to replace the current value with new one.
I note Radai already requested this in the thread, I assume he has some use case also. S
Simple add / remove is a least common feature.
Rgds,
Mike
On 24/02/2017, 23:00, "Jason Gustafson" <ja...@confluent.io> wrote:
Hey Michael,
I'm not strongly opposed to them; I just don't see a lot of benefit. One
thing it would be good to understand is why a consumer interceptor would
need to add headers and why a producer interceptor would need to remove
them. Maybe we only need the common cases?
Thanks,
Jason
On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jason,
>
> Sorry I thought this was the agreed compromise to provide an api that
> avoid boiler plate in return for immutabilty.
>
> If not then mutability will be needed as a goal is to have a single clean
> method call to append/remove a header.
>
> Cheers
> Mike
>
> On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Michael,
>
> I didn't actually comment on the new methods for ProducerRecord and
> ConsumerRecord. If they only save some boilerplate, I'd just as well
> not
> have them.
>
> Also a couple minor comments:
>
> 1. Is the intent of `Headers.filter` to include or exclude the headers
> matching the key? Can you add a javadoc to clarify?
> 2. The KIP mentions that we will introduce V4 of FetchRequest and V4 of
> ProduceRequest. Can you change this to say that the changes will
> piggyback
> onto V3 of ProduceRequest and V4 of FetchRequest which were introduced
> in
> KIP-98?
>
> The rest of the KIP looks good to me.
>
> -Jason
>
> On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > I’ve added the methods on the ProducerRecord that will return a new
> > instance of ProducerRecord with modified headers.
> >
> > On 24/02/2017, 19:22, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Pattern.compile is expensive, and even if cached String.equals is
> > faster than matched. also if we end up with an internal map in
> future for
> > performance it will be easier to be by key.
> >
> > As all that's needed is to get header by key.
> >
> > With like the other arguements of let's implement simple and
> then we
> > can always add pattern later as well if it's found it's needed. (As
> noted
> > it's easier to add methods than to take away)
> >
> > Great I'll update kip with extra methods on producerecord and a
> note
> > that new objects are returned by method calls.
> >
> >
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Friday, February 24, 2017 6:51:45 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > The APIs in the current KIP look good to me. Just a couple
> questions:
> > why
> > does filter not return Headers? Also would it be useful if the
> key is a
> > regex?
> >
> > On the point of immutability.. One option might be to use a
> mutable
> > object
> > only when passing the headers through the interceptor chain. I
> think as
> > long as we resort to mutability only when clear performance
> results
> > show
> > that it is worthwhile, I am satisfied. As Ismael noted, for
> common
> > scenarios it is possible to get reasonable performance with
> immutable
> > objects.
> >
> > -Jason
> >
> > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > Hi
> > >
> > > On 1, How can you guarantee two separate implemented clients
> would
> > add
> > > the headers in the same order we are not specifying an order
> at the
> > > protocol level (nor should we) with regards to keyA being
> ordered
> > before
> > > keyB? We shouldn’t be expecting keyA to be always set before
> keyB.
> > >
> > > On 2, I believe we have changed the naming based on feedback
> from
> > Jason
> > > already, e.g. we don’t have “get” method that inferred O(1)
> > performance,
> > > like wise nor “put” but we have an “append”
> > >
> > > On 3, in the KafkaProducer, I think we have mutability
> already, the
> > value
> > > for time is changed if it is null, at the point of send:
> > > “
> > > long timestamp = record.timestamp() == null ?
> > > time.milliseconds() : record.timestamp();
> > > “
> > >
> > > As such the timestamp is already mutable, so what’s the
> difference
> > here,
> > > we already have some mixed semantics. On timestamp.
> > > e.g. currently if I send to records with timestamp not set,
> the wire
> > > binary sent the value for the timestamp would be different, as
> such
> > we have
> > > mutation for the same record.
> > >
> > > On 4, I think we should not expect not 1 or 2 headers, but
> infact
> > 10’s of
> > > headers. This is the concern on immutable headers, whilst the
> append
> > > self-reference works nicely, what if someone needs to remove a
> > header?
> > >
> > > Trying to get this moving:
> > >
> > > If we really wanted Immutable Headers and essentially you guys
> wont
> > give
> > > +1 for it without.
> > >
> > > Whats the feeling for adding methods to ProducerRecord that
> does the
> > > boiler plate code or creating a new ProducerRecord with the
> altered
> > new
> > > headers (appended or removed) inside. E.g.
> > >
> > > ProducerRecord {
> > >
> > >
> > > ProducerRecord append(Iterable<Headers> headersToAppend){
> > > return new ProducerRecord(key, value, headers.append(
> > headersToAppend),
> > > ….)
> > > }
> > >
> > > ProducerRecord remove(Iterable<Headers> headersToAppend){
> > > return new ProducerRecord(key, value, headers.remove(
> > headersToAppend),
> > > ….)
> > > }
> > >
> > > }
> > >
> > > Were the headers methods actually returns new objects, and the
> > producer
> > > records methods create a new producer record with all the
> current
> > values,
> > > but with the new modified headers.
> > >
> > > Then interceptors / code return this new object?
> > >
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael
> Juma" <
> > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > >
> > > Hi Michael,
> > >
> > > Did you mean that you were happy to compromise to keep it
> > mutable or
> > > immutable? You wrote the former, but it sounded from the
> > sentence that
> > > it
> > > could have been a typo. So, my thoughts on this is that
> there
> > are a few
> > > things to take into account:
> > >
> > > 1. Semantics
> > > 2. Simplicity of use (the common operations should be easy
> to do)
> > > 3. If it's easy to reason about and safe (immutability
> helps
> > with this)
> > > 4. Efficiency (both memory and CPU usage)
> > >
> > > Regarding 1, I think it would be good to be very clear
> about the
> > > guarantees
> > > that we are providing. It seems that we are saying that
> keys are
> > > unordered,
> > > but what about the case where there are multiple values
> for the
> > same
> > > key?
> > > It seems that for some use cases (e.g. lineage), it may be
> > useful to
> > > add
> > > values to the same key while preserving the order.
> > >
> > > Regarding 2, I agree that it's useful to have methods in
> > `Headers` for
> > > the
> > > very common use cases although we have to be careful with
> the
> > naming to
> > > avoid giving the wrong impression. Also, when it comes to
> > `Map.Entry`,
> > > I
> > > think I'd prefer a `toMap` method that simply gives the
> user a
> > Map if
> > > that's what they want (it makes it clear that there's a
> > conversion
> > > happening).
> > >
> > > Regarding 3, the concern I have if we make the headers
> mutable
> > is that
> > > it
> > > seems to introduce some inconsistencies and potential edge
> > cases. At
> > > the
> > > moment, it's safe to keep a ProducerRecord and resend it,
> for
> > example.
> > > If
> > > the record is mutated by an interceptor, then this can
> lead to
> > weird
> > > behaviour. Also, it seems inconsistent that one has to
> create a
> > new
> > > ProducerRecord to modify the record timestamp, but that
> one has
> > to
> > > mutate
> > > the record to add headers. It seems like one should either
> > embrace
> > > immutability or mutability, but mixing both is not ideal.
> > >
> > > Regarding 4, for the cases where there are a small number
> of
> > headers
> > > and/or
> > > 1 or 2 interceptors, it doesn't seem difficult to come up
> with
> > > reasonable
> > > implementations for mutable and immutable cases that do a
> good
> > enough
> > > job.
> > > However, if the number of headers and interceptors is
> large,
> > then care
> > > is
> > > needed for the immutable case to avoid unnecessary
> copying. A
> > simple
> > > option
> > > that adds little memory overhead if we have Header
> instances is
> > to
> > > simply
> > > add a self-reference to the previous Header in the linked
> list.
> > >
> > > Ismael
> > >
> > > On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Im happy to compromise to keep it mutable but move to an
> append
> > > style api.
> > > > (as in guava interables concat)
> > > >
> > > > class Headers {
> > > > Headers append(Iterable<Header> headers);
> > > > }
> > > >
> > > >
> > > > I don’t think we’d want prepend, this would give the
> idea of
> > > guaranteed
> > > > ordering, when in actual fact we don’t provide that
> guarantee
> > (.e.g
> > > one
> > > > client can put headerA, then headerB, but another could
> put
> > headerB
> > > then
> > > > headerA, this shouldn’t cause issues), Also what if we
> changed
> > to a
> > > hashmap
> > > > for the internal implementation, its just a bucket of
> entries
> > no
> > > ordering.
> > > > I think we just need to provide an api to add/append
> headers.
> > > >
> > > > This ok? If so ill update KIP to record this.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> jason@confluent.io>
> > wrote:
> > > >
> > > > The point about usability is fair. It's also
> reasonable to
> > > expect that
> > > > common use cases such as appending headers should be
> done
> > > efficiently.
> > > >
> > > > Perhaps we could compromise with something like this?
> > > >
> > > > class Headers {
> > > > Headers append(Iterable<Header> headers);
> > > > Headers prepend(Iterable<Header> headers);
> > > > }
> > > >
> > > > That retains ease of use while still giving
> ourselves some
> > > flexibility
> > > > in
> > > > the implementation.
> > > >
> > > > -Jason
> > > >
> > > >
> > > > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > I wasn’t referring to the headers needing to be
> copied,
> > im
> > > meaning
> > > > the
> > > > > fact we’d be forcing a new producer record to be
> > created, with
> > > all
> > > > the
> > > > > contents copied.
> > > > >
> > > > > i.e what will happen is utility method will be
> created
> > or end
> > > up
> > > > being
> > > > > used, which does this, and returns the new
> ProducerRecord
> > > instance.
> > > > >
> > > > > ProducerRecord addHeader(ProducerRecord record,
> Header
> > > header){
> > > > > Return New ProducerRecord(record.key, record.value,
> > > > record.timestamp…..,
> > > > > record.headers.concat(header))
> > > > > }
> > > > >
> > > > > To me this seems ugly, but will be inevitable if we
> > don’t make
> > > adding
> > > > > headers to existing records a simple clean method
> call.
> > > > >
> > > > >
> > > > >
> > > > > On 22/02/2017, 22:57, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > Lazy init can achieve/avoid that.
> > > > >
> > > > > Re the concat, why don’t we implement that
> inside the
> > > Headers
> > > > rather
> > > > > than causing everyone to implement this as adding
> > headers in
> > > > interceptors
> > > > > will be a dominant use case. We want a user
> friendly API.
> > > Having as
> > > > a user
> > > > > having to code this instead of having the headers
> handle
> > this
> > > for me
> > > > seems
> > > > > redundant.
> > > > >
> > > > > On 22/02/2017, 22:34, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > wrote:
> > > > >
> > > > > I thought the argument was against
> creating the
> > extra
> > > objects
> > > > > unnecessarily
> > > > > (i.e. if they were not accessed). And note
> that
> > making
> > > the
> > > > Headers
> > > > > immutable doesn't necessarily mean that
> they
> > need to be
> > > > copied:
> > > > > you can do
> > > > > a trick like Guava's Iterables.concat to
> add
> > additional
> > > > headers
> > > > > without
> > > > > changing the underlying collections.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 2:22 PM, Michael
> Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > If the argument for not having a map
> holding
> > the
> > > key, value
> > > > > pairs is due
> > > > > > to garbage creation of HashMap entry's,
> > forcing the
> > > > creation of
> > > > > a whole new
> > > > > > producer record to simply add a head,
> surely is
> > > creating
> > > > a-lot
> > > > > more?
> > > > > > ________________________________________
> > > > > > From: Jason Gustafson <
> jason@confluent.io>
> > > > > > Sent: Wednesday, February 22, 2017 10:09
> PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> Record
> > Headers
> > > > > >
> > > > > > The current producer interceptor API is
> this:
> > > > > >
> > > > > > ProducerRecord<K, V>
> onSend(ProducerRecord<K,
> > V>
> > > record);
> > > > > >
> > > > > > So adding a header means creating a new
> > > ProducerRecord
> > > > with a
> > > > > new header
> > > > > > added to the current headers and
> returning it.
> > Would
> > > that
> > > > not
> > > > > work?
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > So how would you have this work if not
> > mutable
> > > where
> > > > > interceptors would
> > > > > > > add headers?
> > > > > > >
> > > > > > > Sent using OWA for iPhone
> > > > > > > ______________________________
> __________
> > > > > > > From: Jason Gustafson <
> jason@confluent.io>
> > > > > > > Sent: Wednesday, February 22, 2017
> 8:42:27 PM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> Record
> > Headers
> > > > > > >
> > > > > > > I think the point on the mutability of
> > Headers is
> > > worth
> > > > > discussing a
> > > > > > little
> > > > > > > more. As far as I can tell, once the
> > > ProducerRecord (or
> > > > > ConsumerRecord)
> > > > > > is
> > > > > > > constructed, there should be no need to
> > further
> > > change
> > > > the
> > > > > headers. Is
> > > > > > that
> > > > > > > correct? If so, then why not enforce
> that
> > that is
> > > the
> > > > case
> > > > > through the
> > > > > > API?
> > > > > > > One problem with mutability it that it
> > constrains
> > > the
> > > > > implementation of
> > > > > > > Headers. For example, if we were
> backing
> > with a
> > > byte
> > > > slice,
> > > > > would we
> > > > > > recopy
> > > > > > > the bytes if a header is added or
> would we
> > > maintain a
> > > > satellite
> > > > > > collection
> > > > > > > of added records. Seems not great
> either
> > way. If we
> > > > really
> > > > > think
> > > > > > mutability
> > > > > > > is needed, perhaps we could add a
> method to
> > > headers to
> > > > convert
> > > > > it to a
> > > > > > > mutable type (e.g. a Headers.toMap)?
> > > > > > >
> > > > > > > I'm also with Ismael about exposing
> > Headers.get().
> > > I
> > > > thought
> > > > > it might
> > > > > > make
> > > > > > > sense to have a method like this
> instead:
> > > > > > >
> > > > > > > Iterable<Header> findMatching(Pattern
> > pattern);
> > > > > > >
> > > > > > > This makes the (potential) need to
> scan the
> > headers
> > > > clear in
> > > > > the API. I'd
> > > > > > > also be fine exposing no getter at
> all. In
> > > general, Ï
> > > > think
> > > > > it's good to
> > > > > > > start with a minimalistic API and work
> from
> > there
> > > since
> > > > it's
> > > > > always
> > > > > > easier
> > > > > > > to add methods than remove them.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 9:16 AM,
> Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Hi Ismael
> > > > > > > >
> > > > > > > > On point 1,
> > > > > > > >
> > > > > > > > Sure makes sense will update shortly.
> > > > > > > >
> > > > > > > > On point 2,
> > > > > > > >
> > > > > > > > Setter/getter typical to
> > properties/headers api’s
> > > > > traditionally are map
> > > > > > > > styled interfaces and what I believe
> is
> > most
> > > expected
> > > > styled
> > > > > thus the
> > > > > > > Key,
> > > > > > > > Value setter.
> > > > > > > > Also it would mean rather than an
> > interface, we
> > > would
> > > > be
> > > > > making our
> > > > > > > > internal header impl object we have
> for the
> > > array,
> > > > exposed.
> > > > > E.g. if we
> > > > > > > had
> > > > > > > > a Map really this would be Map.Entry
> > interface,
> > > this
> > > > is the
> > > > > same
> > > > > > reasons
> > > > > > > on
> > > > > > > > the map interface I cannot actually
> make
> > the
> > > > underlying Node
> > > > > object
> > > > > > > that’s
> > > > > > > > the implementation for HashMap, so
> that
> > > internals can
> > > > be
> > > > > changed.
> > > > > > > >
> > > > > > > >
> > > > > > > > On point 3,
> > > > > > > >
> > > > > > > > I think it people do expect it to be
> > performant,
> > > thus
> > > > why
> > > > > originally
> > > > > > > > concern I raised with using an array
> for
> > to me
> > > is an
> > > > early
> > > > > memory
> > > > > > > > optimisation. I think the user
> experience
> > of
> > > > > properties/headers is on a
> > > > > > > > get/set model. This is why its
> important
> > we have
> > > > > encapsulated logic
> > > > > > that
> > > > > > > > then allows us to change this to a
> map, if
> > this
> > > > becomes and
> > > > > issue, and
> > > > > > > the
> > > > > > > > memory overhead of hashmap is less
> so.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 22/02/2017, 15:56, "
> ismaelj@gmail.com
> > on
> > > behalf of
> > > > > Ismael Juma" <
> > > > > > > > ismaelj@gmail.com on behalf of
> > ismael@juma.me.uk
> > > >
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Great to see the progress that
> has been
> > > achieved
> > > > on this
> > > > > one. :) A
> > > > > > > few
> > > > > > > > comments regarding the APIs (I'm
> still
> > > reviewing
> > > > the
> > > > > message format
> > > > > > > > changes):
> > > > > > > >
> > > > > > > > 1. Nit: `getHeaders` in
> > `ProducerRecord` and
> > > > > `ConsumerRecord`
> > > > > > should
> > > > > > > be
> > > > > > > > named `headers` (we avoid the
> `get`
> > prefix in
> > > > Kafka)
> > > > > > > >
> > > > > > > > 2. The `Headers` class is mutable
> > (there's
> > > an `add`
> > > > > method). Does
> > > > > > it
> > > > > > > > need
> > > > > > > > to be? If so, it would be good to
> > explain
> > > why.
> > > > Related
> > > > > to that, we
> > > > > > > > should
> > > > > > > > also explain the thinking around
> > > thread-safety. If
> > > > we
> > > > > keep the
> > > > > > `add`
> > > > > > > > method, it may make sense for it
> to
> > take a
> > > `Header`
> > > > > (that way we
> > > > > > can
> > > > > > > > add
> > > > > > > > things to `Header` without
> changing the
> > > interface).
> > > > > > > >
> > > > > > > > 3. Do we need the `Headers.get()`
> > method?
> > > People
> > > > usually
> > > > > assume
> > > > > > that
> > > > > > > > `get`
> > > > > > > > would be efficient, but
> depending on
> > the
> > > > implementation
> > > > > (the
> > > > > > current
> > > > > > > > proposal states that an array
> would be
> > > used), it
> > > > may not
> > > > > be. If we
> > > > > > > > expect
> > > > > > > > the number of headers to be
> small, it
> > doesn't
> > > > matter
> > > > > though.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Tue, Feb 21, 2017 at 6:38 PM,
> > Michael
> > > Pearce <
> > > > > > > Michael.Pearce@ig.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jason,
> > > > > > > > >
> > > > > > > > > Have converted the
> interface/api
> > bullets
> > > into
> > > > > interface code
> > > > > > > > snippets.
> > > > > > > > >
> > > > > > > > > Agreed implementation won’t
> take too
> > long.
> > > We
> > > > have
> > > > > early versions
> > > > > > > > already.
> > > > > > > > > Maybe a week before you think
> about
> > > merging I
> > > > would
> > > > > assume it
> > > > > > would
> > > > > > > > be more
> > > > > > > > > stabilised? I was thinking
> then we
> > could
> > > fork
> > > > from
> > > > > your confluent
> > > > > > > > branch,
> > > > > > > > > making and then holding KIP-82
> > changes in
> > > a patch
> > > > > file, that we
> > > > > > can
> > > > > > > > then
> > > > > > > > > re-fork from apache once KIP98
> final
> > is
> > > merged,
> > > > and
> > > > > apply patch
> > > > > > > with
> > > > > > > > last
> > > > > > > > > minute changes.
> > > > > > > > >
> > > > > > > > > Cheers
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 22/02/2017, 00:56, "Jason
> > Gustafson" <
> > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hey Michael,
> > > > > > > > >
> > > > > > > > > Awesome. I have a minor
> request.
> > The
> > > APIs are
> > > > > currently
> > > > > > > > documented as a
> > > > > > > > > wiki list. Would you mind
> adding
> > a code
> > > > snippet
> > > > > instead?
> > > > > > It's a
> > > > > > > > bit
> > > > > > > > > easier
> > > > > > > > > to process.
> > > > > > > > >
> > > > > > > > > How will be best to manage
> this,
> > as we
> > > will
> > > > > obviously build
> > > > > > off
> > > > > > > > your
> > > > > > > > > KIP’s
> > > > > > > > > > protocol changes, to
> avoid a
> > merge
> > > hell,
> > > > should
> > > > > we branch
> > > > > > > from
> > > > > > > > your
> > > > > > > > > branch
> > > > > > > > > > in the confluent repo or
> is it
> > worth
> > > > having a
> > > > > KIP-98
> > > > > > special
> > > > > > > > branch
> > > > > > > > > in the
> > > > > > > > > > apache git, that we can
> > branch/fork
> > > from?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I was thinking about this
> also.
> > > Ideally we'd
> > > > like
> > > > > to get the
> > > > > > > > changes
> > > > > > > > > in as
> > > > > > > > > close together as possible
> since
> > we
> > > only
> > > > want one
> > > > > magic bump
> > > > > > > and
> > > > > > > > some
> > > > > > > > > users
> > > > > > > > > deploy trunk. The level of
> > effort to
> > > change
> > > > the
> > > > > format for
> > > > > > > > headers
> > > > > > > > > seems
> > > > > > > > > not too high. Do you
> agree? My
> > guess
> > > is that
> > > > the
> > > > > KIP-98
> > > > > > message
> > > > > > > > format
> > > > > > > > > patch will take 2-3 weeks
> to
> > review
> > > before we
> > > > > merge to trunk,
> > > > > > > so
> > > > > > > > you
> > > > > > > > > could
> > > > > > > > > hold off implementing
> until that
> > patch
> > > has
> > > > somewhat
> > > > > > stabilized.
> > > > > > > > That
> > > > > > > > > would
> > > > > > > > > save some potential rebase
> pain.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information contained in
> this
> > email is
> > > > strictly
> > > > > confidential
> > > > > > > and
> > > > > > > > for
> > > > > > > > > the use of the addressee only,
> unless
> > > otherwise
> > > > > indicated. If you
> > > > > > > > are not
> > > > > > > > > the intended recipient, please
> do
> > not read,
> > > > copy, use
> > > > > or disclose
> > > > > > > to
> > > > > > > > others
> > > > > > > > > this message or any attachment.
> > Please also
> > > > notify the
> > > > > sender by
> > > > > > > > replying
> > > > > > > > > to this email or by telephone
> > (+44(020
> > > 7896 0011)
> > > > and
> > > > > then delete
> > > > > > > the
> > > > > > > > > email and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc)
> > > > > that do
> > > > > > not
> > > > > > > > relate
> > > > > > > > > to the official business of
> this
> > company
> > > shall be
> > > > > understood as
> > > > > > > > neither
> > > > > > > > > given nor endorsed by it. IG
> is a
> > trading
> > > name
> > > > of IG
> > > > > Markets
> > > > > > > Limited
> > > > > > > > (a
> > > > > > > > > company registered in England
> and
> > Wales,
> > > company
> > > > > number 04008957)
> > > > > > > > and IG
> > > > > > > > > Index Limited (a company
> registered
> > in
> > > England
> > > > and
> > > > > Wales, company
> > > > > > > > number
> > > > > > > > > 01190902). Registered address
> at
> > Cannon
> > > Bridge
> > > > House,
> > > > > 25 Dowgate
> > > > > > > > Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> Markets
> > Limited
> > > > (register
> > > > > number 195355)
> > > > > > > > and IG
> > > > > > > > > Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated
> > > > > > > > by the
> > > > > > > > > Financial Conduct Authority.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this
> email is
> > > strictly
> > > > > confidential and
> > > > > > for
> > > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated.
> > > > > If you are
> > > > > > not
> > > > > > > > the intended recipient, please do
> not read,
> > > copy, use
> > > > or
> > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please also
> > > notify the
> > > > > sender by
> > > > > > replying
> > > > > > > > to this email or by telephone
> (+44(020 7896
> > > 0011) and
> > > > then
> > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that
> > > > do not
> > > > > relate to
> > > > > > > the
> > > > > > > > official business of this company
> shall be
> > > understood
> > > > as
> > > > > neither given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a trading name
> of IG
> > > Markets
> > > > Limited
> > > > > (a company
> > > > > > > > registered in England and Wales,
> company
> > number
> > > > 04008957)
> > > > > and IG Index
> > > > > > > > Limited (a company registered in
> England
> > and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > > 01190902). Registered address at
> Cannon
> > Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > > (register
> > > > number
> > > > > 195355) and
> > > > > > IG
> > > > > > > > Index Limited (register number
> 114059) are
> > > authorised
> > > > and
> > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > > The information contained in this
> email is
> > strictly
> > > > > confidential and for
> > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If
> > > > > you are not
> > > > > > > the intended recipient, please do not
> read,
> > copy,
> > > use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please
> also
> > notify
> > > the
> > > > sender
> > > > > by replying
> > > > > > > to this email or by telephone (+44(020
> 7896
> > 0011)
> > > and
> > > > then
> > > > > delete the
> > > > > > email
> > > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to
> > > > > > the
> > > > > > > official business of this company
> shall be
> > > understood as
> > > > > neither given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name
> of IG
> > Markets
> > > > Limited (a
> > > > > company
> > > > > > > registered in England and Wales,
> company
> > number
> > > > 04008957) and
> > > > > IG Index
> > > > > > > Limited (a company registered in
> England and
> > Wales,
> > > > company
> > > > > number
> > > > > > > 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > > Dowgate Hill,
> > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > (register
> > > number
> > > > > 195355) and IG
> > > > > > > Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > > The information contained in this email
> is
> > strictly
> > > > confidential
> > > > > and for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated.
> > > > If
> > > > > you are not
> > > > > > the intended recipient, please do not
> read,
> > copy,
> > > use or
> > > > > disclose to others
> > > > > > this message or any attachment. Please
> also
> > notify
> > > the
> > > > sender by
> > > > > replying
> > > > > > to this email or by telephone (+44(020
> 7896
> > 0011)
> > > and then
> > > > > delete the email
> > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to the
> > > > > > official business of this company shall
> be
> > > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of
> IG
> > Markets
> > > Limited
> > > > (a
> > > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957)
> > > > and IG
> > > > > Index
> > > > > > Limited (a company registered in England
> and
> > Wales,
> > > company
> > > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > > 195355) and IG
> > > > > > Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > > confidential and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you
> > > > are not
> > > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify
> the
> > sender
> > > by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then
> > > delete
> > > > the email
> > > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate
> > > > to the
> > > > > official business of this company shall be
> understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG
> > > > Index
> > > > > Limited (a company registered in England and Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you
> > > are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> > > the email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> > > to the
> > > > official business of this company shall be understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential
> and
> > for the use of the addressee only, unless otherwise indicated. If
> you are
> > not the intended recipient, please do not read, copy, use or
> disclose to
> > others this message or any attachment. Please also notify the sender
> by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the official business of this company shall be understood
> as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and IG Index Limited (register number 114059) are authorised
> and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Michael,
I'm not strongly opposed to them; I just don't see a lot of benefit. One
thing it would be good to understand is why a consumer interceptor would
need to add headers and why a producer interceptor would need to remove
them. Maybe we only need the common cases?
Thanks,
Jason
On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jason,
>
> Sorry I thought this was the agreed compromise to provide an api that
> avoid boiler plate in return for immutabilty.
>
> If not then mutability will be needed as a goal is to have a single clean
> method call to append/remove a header.
>
> Cheers
> Mike
>
> On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Michael,
>
> I didn't actually comment on the new methods for ProducerRecord and
> ConsumerRecord. If they only save some boilerplate, I'd just as well
> not
> have them.
>
> Also a couple minor comments:
>
> 1. Is the intent of `Headers.filter` to include or exclude the headers
> matching the key? Can you add a javadoc to clarify?
> 2. The KIP mentions that we will introduce V4 of FetchRequest and V4 of
> ProduceRequest. Can you change this to say that the changes will
> piggyback
> onto V3 of ProduceRequest and V4 of FetchRequest which were introduced
> in
> KIP-98?
>
> The rest of the KIP looks good to me.
>
> -Jason
>
> On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > I’ve added the methods on the ProducerRecord that will return a new
> > instance of ProducerRecord with modified headers.
> >
> > On 24/02/2017, 19:22, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Pattern.compile is expensive, and even if cached String.equals is
> > faster than matched. also if we end up with an internal map in
> future for
> > performance it will be easier to be by key.
> >
> > As all that's needed is to get header by key.
> >
> > With like the other arguements of let's implement simple and
> then we
> > can always add pattern later as well if it's found it's needed. (As
> noted
> > it's easier to add methods than to take away)
> >
> > Great I'll update kip with extra methods on producerecord and a
> note
> > that new objects are returned by method calls.
> >
> >
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Friday, February 24, 2017 6:51:45 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > The APIs in the current KIP look good to me. Just a couple
> questions:
> > why
> > does filter not return Headers? Also would it be useful if the
> key is a
> > regex?
> >
> > On the point of immutability.. One option might be to use a
> mutable
> > object
> > only when passing the headers through the interceptor chain. I
> think as
> > long as we resort to mutability only when clear performance
> results
> > show
> > that it is worthwhile, I am satisfied. As Ismael noted, for
> common
> > scenarios it is possible to get reasonable performance with
> immutable
> > objects.
> >
> > -Jason
> >
> > On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > Hi
> > >
> > > On 1, How can you guarantee two separate implemented clients
> would
> > add
> > > the headers in the same order we are not specifying an order
> at the
> > > protocol level (nor should we) with regards to keyA being
> ordered
> > before
> > > keyB? We shouldn’t be expecting keyA to be always set before
> keyB.
> > >
> > > On 2, I believe we have changed the naming based on feedback
> from
> > Jason
> > > already, e.g. we don’t have “get” method that inferred O(1)
> > performance,
> > > like wise nor “put” but we have an “append”
> > >
> > > On 3, in the KafkaProducer, I think we have mutability
> already, the
> > value
> > > for time is changed if it is null, at the point of send:
> > > “
> > > long timestamp = record.timestamp() == null ?
> > > time.milliseconds() : record.timestamp();
> > > “
> > >
> > > As such the timestamp is already mutable, so what’s the
> difference
> > here,
> > > we already have some mixed semantics. On timestamp.
> > > e.g. currently if I send to records with timestamp not set,
> the wire
> > > binary sent the value for the timestamp would be different, as
> such
> > we have
> > > mutation for the same record.
> > >
> > > On 4, I think we should not expect not 1 or 2 headers, but
> infact
> > 10’s of
> > > headers. This is the concern on immutable headers, whilst the
> append
> > > self-reference works nicely, what if someone needs to remove a
> > header?
> > >
> > > Trying to get this moving:
> > >
> > > If we really wanted Immutable Headers and essentially you guys
> wont
> > give
> > > +1 for it without.
> > >
> > > Whats the feeling for adding methods to ProducerRecord that
> does the
> > > boiler plate code or creating a new ProducerRecord with the
> altered
> > new
> > > headers (appended or removed) inside. E.g.
> > >
> > > ProducerRecord {
> > >
> > >
> > > ProducerRecord append(Iterable<Headers> headersToAppend){
> > > return new ProducerRecord(key, value, headers.append(
> > headersToAppend),
> > > ….)
> > > }
> > >
> > > ProducerRecord remove(Iterable<Headers> headersToAppend){
> > > return new ProducerRecord(key, value, headers.remove(
> > headersToAppend),
> > > ….)
> > > }
> > >
> > > }
> > >
> > > Were the headers methods actually returns new objects, and the
> > producer
> > > records methods create a new producer record with all the
> current
> > values,
> > > but with the new modified headers.
> > >
> > > Then interceptors / code return this new object?
> > >
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael
> Juma" <
> > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > >
> > > Hi Michael,
> > >
> > > Did you mean that you were happy to compromise to keep it
> > mutable or
> > > immutable? You wrote the former, but it sounded from the
> > sentence that
> > > it
> > > could have been a typo. So, my thoughts on this is that
> there
> > are a few
> > > things to take into account:
> > >
> > > 1. Semantics
> > > 2. Simplicity of use (the common operations should be easy
> to do)
> > > 3. If it's easy to reason about and safe (immutability
> helps
> > with this)
> > > 4. Efficiency (both memory and CPU usage)
> > >
> > > Regarding 1, I think it would be good to be very clear
> about the
> > > guarantees
> > > that we are providing. It seems that we are saying that
> keys are
> > > unordered,
> > > but what about the case where there are multiple values
> for the
> > same
> > > key?
> > > It seems that for some use cases (e.g. lineage), it may be
> > useful to
> > > add
> > > values to the same key while preserving the order.
> > >
> > > Regarding 2, I agree that it's useful to have methods in
> > `Headers` for
> > > the
> > > very common use cases although we have to be careful with
> the
> > naming to
> > > avoid giving the wrong impression. Also, when it comes to
> > `Map.Entry`,
> > > I
> > > think I'd prefer a `toMap` method that simply gives the
> user a
> > Map if
> > > that's what they want (it makes it clear that there's a
> > conversion
> > > happening).
> > >
> > > Regarding 3, the concern I have if we make the headers
> mutable
> > is that
> > > it
> > > seems to introduce some inconsistencies and potential edge
> > cases. At
> > > the
> > > moment, it's safe to keep a ProducerRecord and resend it,
> for
> > example.
> > > If
> > > the record is mutated by an interceptor, then this can
> lead to
> > weird
> > > behaviour. Also, it seems inconsistent that one has to
> create a
> > new
> > > ProducerRecord to modify the record timestamp, but that
> one has
> > to
> > > mutate
> > > the record to add headers. It seems like one should either
> > embrace
> > > immutability or mutability, but mixing both is not ideal.
> > >
> > > Regarding 4, for the cases where there are a small number
> of
> > headers
> > > and/or
> > > 1 or 2 interceptors, it doesn't seem difficult to come up
> with
> > > reasonable
> > > implementations for mutable and immutable cases that do a
> good
> > enough
> > > job.
> > > However, if the number of headers and interceptors is
> large,
> > then care
> > > is
> > > needed for the immutable case to avoid unnecessary
> copying. A
> > simple
> > > option
> > > that adds little memory overhead if we have Header
> instances is
> > to
> > > simply
> > > add a self-reference to the previous Header in the linked
> list.
> > >
> > > Ismael
> > >
> > > On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Im happy to compromise to keep it mutable but move to an
> append
> > > style api.
> > > > (as in guava interables concat)
> > > >
> > > > class Headers {
> > > > Headers append(Iterable<Header> headers);
> > > > }
> > > >
> > > >
> > > > I don’t think we’d want prepend, this would give the
> idea of
> > > guaranteed
> > > > ordering, when in actual fact we don’t provide that
> guarantee
> > (.e.g
> > > one
> > > > client can put headerA, then headerB, but another could
> put
> > headerB
> > > then
> > > > headerA, this shouldn’t cause issues), Also what if we
> changed
> > to a
> > > hashmap
> > > > for the internal implementation, its just a bucket of
> entries
> > no
> > > ordering.
> > > > I think we just need to provide an api to add/append
> headers.
> > > >
> > > > This ok? If so ill update KIP to record this.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > > On 23/02/2017, 00:37, "Jason Gustafson" <
> jason@confluent.io>
> > wrote:
> > > >
> > > > The point about usability is fair. It's also
> reasonable to
> > > expect that
> > > > common use cases such as appending headers should be
> done
> > > efficiently.
> > > >
> > > > Perhaps we could compromise with something like this?
> > > >
> > > > class Headers {
> > > > Headers append(Iterable<Header> headers);
> > > > Headers prepend(Iterable<Header> headers);
> > > > }
> > > >
> > > > That retains ease of use while still giving
> ourselves some
> > > flexibility
> > > > in
> > > > the implementation.
> > > >
> > > > -Jason
> > > >
> > > >
> > > > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > I wasn’t referring to the headers needing to be
> copied,
> > im
> > > meaning
> > > > the
> > > > > fact we’d be forcing a new producer record to be
> > created, with
> > > all
> > > > the
> > > > > contents copied.
> > > > >
> > > > > i.e what will happen is utility method will be
> created
> > or end
> > > up
> > > > being
> > > > > used, which does this, and returns the new
> ProducerRecord
> > > instance.
> > > > >
> > > > > ProducerRecord addHeader(ProducerRecord record,
> Header
> > > header){
> > > > > Return New ProducerRecord(record.key, record.value,
> > > > record.timestamp…..,
> > > > > record.headers.concat(header))
> > > > > }
> > > > >
> > > > > To me this seems ugly, but will be inevitable if we
> > don’t make
> > > adding
> > > > > headers to existing records a simple clean method
> call.
> > > > >
> > > > >
> > > > >
> > > > > On 22/02/2017, 22:57, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > Lazy init can achieve/avoid that.
> > > > >
> > > > > Re the concat, why don’t we implement that
> inside the
> > > Headers
> > > > rather
> > > > > than causing everyone to implement this as adding
> > headers in
> > > > interceptors
> > > > > will be a dominant use case. We want a user
> friendly API.
> > > Having as
> > > > a user
> > > > > having to code this instead of having the headers
> handle
> > this
> > > for me
> > > > seems
> > > > > redundant.
> > > > >
> > > > > On 22/02/2017, 22:34, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > wrote:
> > > > >
> > > > > I thought the argument was against
> creating the
> > extra
> > > objects
> > > > > unnecessarily
> > > > > (i.e. if they were not accessed). And note
> that
> > making
> > > the
> > > > Headers
> > > > > immutable doesn't necessarily mean that
> they
> > need to be
> > > > copied:
> > > > > you can do
> > > > > a trick like Guava's Iterables.concat to
> add
> > additional
> > > > headers
> > > > > without
> > > > > changing the underlying collections.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 2:22 PM, Michael
> Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > If the argument for not having a map
> holding
> > the
> > > key, value
> > > > > pairs is due
> > > > > > to garbage creation of HashMap entry's,
> > forcing the
> > > > creation of
> > > > > a whole new
> > > > > > producer record to simply add a head,
> surely is
> > > creating
> > > > a-lot
> > > > > more?
> > > > > > ________________________________________
> > > > > > From: Jason Gustafson <
> jason@confluent.io>
> > > > > > Sent: Wednesday, February 22, 2017 10:09
> PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> Record
> > Headers
> > > > > >
> > > > > > The current producer interceptor API is
> this:
> > > > > >
> > > > > > ProducerRecord<K, V>
> onSend(ProducerRecord<K,
> > V>
> > > record);
> > > > > >
> > > > > > So adding a header means creating a new
> > > ProducerRecord
> > > > with a
> > > > > new header
> > > > > > added to the current headers and
> returning it.
> > Would
> > > that
> > > > not
> > > > > work?
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > So how would you have this work if not
> > mutable
> > > where
> > > > > interceptors would
> > > > > > > add headers?
> > > > > > >
> > > > > > > Sent using OWA for iPhone
> > > > > > > ______________________________
> __________
> > > > > > > From: Jason Gustafson <
> jason@confluent.io>
> > > > > > > Sent: Wednesday, February 22, 2017
> 8:42:27 PM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> Record
> > Headers
> > > > > > >
> > > > > > > I think the point on the mutability of
> > Headers is
> > > worth
> > > > > discussing a
> > > > > > little
> > > > > > > more. As far as I can tell, once the
> > > ProducerRecord (or
> > > > > ConsumerRecord)
> > > > > > is
> > > > > > > constructed, there should be no need to
> > further
> > > change
> > > > the
> > > > > headers. Is
> > > > > > that
> > > > > > > correct? If so, then why not enforce
> that
> > that is
> > > the
> > > > case
> > > > > through the
> > > > > > API?
> > > > > > > One problem with mutability it that it
> > constrains
> > > the
> > > > > implementation of
> > > > > > > Headers. For example, if we were
> backing
> > with a
> > > byte
> > > > slice,
> > > > > would we
> > > > > > recopy
> > > > > > > the bytes if a header is added or
> would we
> > > maintain a
> > > > satellite
> > > > > > collection
> > > > > > > of added records. Seems not great
> either
> > way. If we
> > > > really
> > > > > think
> > > > > > mutability
> > > > > > > is needed, perhaps we could add a
> method to
> > > headers to
> > > > convert
> > > > > it to a
> > > > > > > mutable type (e.g. a Headers.toMap)?
> > > > > > >
> > > > > > > I'm also with Ismael about exposing
> > Headers.get().
> > > I
> > > > thought
> > > > > it might
> > > > > > make
> > > > > > > sense to have a method like this
> instead:
> > > > > > >
> > > > > > > Iterable<Header> findMatching(Pattern
> > pattern);
> > > > > > >
> > > > > > > This makes the (potential) need to
> scan the
> > headers
> > > > clear in
> > > > > the API. I'd
> > > > > > > also be fine exposing no getter at
> all. In
> > > general, Ï
> > > > think
> > > > > it's good to
> > > > > > > start with a minimalistic API and work
> from
> > there
> > > since
> > > > it's
> > > > > always
> > > > > > easier
> > > > > > > to add methods than remove them.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Wed, Feb 22, 2017 at 9:16 AM,
> Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Hi Ismael
> > > > > > > >
> > > > > > > > On point 1,
> > > > > > > >
> > > > > > > > Sure makes sense will update shortly.
> > > > > > > >
> > > > > > > > On point 2,
> > > > > > > >
> > > > > > > > Setter/getter typical to
> > properties/headers api’s
> > > > > traditionally are map
> > > > > > > > styled interfaces and what I believe
> is
> > most
> > > expected
> > > > styled
> > > > > thus the
> > > > > > > Key,
> > > > > > > > Value setter.
> > > > > > > > Also it would mean rather than an
> > interface, we
> > > would
> > > > be
> > > > > making our
> > > > > > > > internal header impl object we have
> for the
> > > array,
> > > > exposed.
> > > > > E.g. if we
> > > > > > > had
> > > > > > > > a Map really this would be Map.Entry
> > interface,
> > > this
> > > > is the
> > > > > same
> > > > > > reasons
> > > > > > > on
> > > > > > > > the map interface I cannot actually
> make
> > the
> > > > underlying Node
> > > > > object
> > > > > > > that’s
> > > > > > > > the implementation for HashMap, so
> that
> > > internals can
> > > > be
> > > > > changed.
> > > > > > > >
> > > > > > > >
> > > > > > > > On point 3,
> > > > > > > >
> > > > > > > > I think it people do expect it to be
> > performant,
> > > thus
> > > > why
> > > > > originally
> > > > > > > > concern I raised with using an array
> for
> > to me
> > > is an
> > > > early
> > > > > memory
> > > > > > > > optimisation. I think the user
> experience
> > of
> > > > > properties/headers is on a
> > > > > > > > get/set model. This is why its
> important
> > we have
> > > > > encapsulated logic
> > > > > > that
> > > > > > > > then allows us to change this to a
> map, if
> > this
> > > > becomes and
> > > > > issue, and
> > > > > > > the
> > > > > > > > memory overhead of hashmap is less
> so.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 22/02/2017, 15:56, "
> ismaelj@gmail.com
> > on
> > > behalf of
> > > > > Ismael Juma" <
> > > > > > > > ismaelj@gmail.com on behalf of
> > ismael@juma.me.uk
> > > >
> > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Great to see the progress that
> has been
> > > achieved
> > > > on this
> > > > > one. :) A
> > > > > > > few
> > > > > > > > comments regarding the APIs (I'm
> still
> > > reviewing
> > > > the
> > > > > message format
> > > > > > > > changes):
> > > > > > > >
> > > > > > > > 1. Nit: `getHeaders` in
> > `ProducerRecord` and
> > > > > `ConsumerRecord`
> > > > > > should
> > > > > > > be
> > > > > > > > named `headers` (we avoid the
> `get`
> > prefix in
> > > > Kafka)
> > > > > > > >
> > > > > > > > 2. The `Headers` class is mutable
> > (there's
> > > an `add`
> > > > > method). Does
> > > > > > it
> > > > > > > > need
> > > > > > > > to be? If so, it would be good to
> > explain
> > > why.
> > > > Related
> > > > > to that, we
> > > > > > > > should
> > > > > > > > also explain the thinking around
> > > thread-safety. If
> > > > we
> > > > > keep the
> > > > > > `add`
> > > > > > > > method, it may make sense for it
> to
> > take a
> > > `Header`
> > > > > (that way we
> > > > > > can
> > > > > > > > add
> > > > > > > > things to `Header` without
> changing the
> > > interface).
> > > > > > > >
> > > > > > > > 3. Do we need the `Headers.get()`
> > method?
> > > People
> > > > usually
> > > > > assume
> > > > > > that
> > > > > > > > `get`
> > > > > > > > would be efficient, but
> depending on
> > the
> > > > implementation
> > > > > (the
> > > > > > current
> > > > > > > > proposal states that an array
> would be
> > > used), it
> > > > may not
> > > > > be. If we
> > > > > > > > expect
> > > > > > > > the number of headers to be
> small, it
> > doesn't
> > > > matter
> > > > > though.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Tue, Feb 21, 2017 at 6:38 PM,
> > Michael
> > > Pearce <
> > > > > > > Michael.Pearce@ig.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jason,
> > > > > > > > >
> > > > > > > > > Have converted the
> interface/api
> > bullets
> > > into
> > > > > interface code
> > > > > > > > snippets.
> > > > > > > > >
> > > > > > > > > Agreed implementation won’t
> take too
> > long.
> > > We
> > > > have
> > > > > early versions
> > > > > > > > already.
> > > > > > > > > Maybe a week before you think
> about
> > > merging I
> > > > would
> > > > > assume it
> > > > > > would
> > > > > > > > be more
> > > > > > > > > stabilised? I was thinking
> then we
> > could
> > > fork
> > > > from
> > > > > your confluent
> > > > > > > > branch,
> > > > > > > > > making and then holding KIP-82
> > changes in
> > > a patch
> > > > > file, that we
> > > > > > can
> > > > > > > > then
> > > > > > > > > re-fork from apache once KIP98
> final
> > is
> > > merged,
> > > > and
> > > > > apply patch
> > > > > > > with
> > > > > > > > last
> > > > > > > > > minute changes.
> > > > > > > > >
> > > > > > > > > Cheers
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 22/02/2017, 00:56, "Jason
> > Gustafson" <
> > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hey Michael,
> > > > > > > > >
> > > > > > > > > Awesome. I have a minor
> request.
> > The
> > > APIs are
> > > > > currently
> > > > > > > > documented as a
> > > > > > > > > wiki list. Would you mind
> adding
> > a code
> > > > snippet
> > > > > instead?
> > > > > > It's a
> > > > > > > > bit
> > > > > > > > > easier
> > > > > > > > > to process.
> > > > > > > > >
> > > > > > > > > How will be best to manage
> this,
> > as we
> > > will
> > > > > obviously build
> > > > > > off
> > > > > > > > your
> > > > > > > > > KIP’s
> > > > > > > > > > protocol changes, to
> avoid a
> > merge
> > > hell,
> > > > should
> > > > > we branch
> > > > > > > from
> > > > > > > > your
> > > > > > > > > branch
> > > > > > > > > > in the confluent repo or
> is it
> > worth
> > > > having a
> > > > > KIP-98
> > > > > > special
> > > > > > > > branch
> > > > > > > > > in the
> > > > > > > > > > apache git, that we can
> > branch/fork
> > > from?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I was thinking about this
> also.
> > > Ideally we'd
> > > > like
> > > > > to get the
> > > > > > > > changes
> > > > > > > > > in as
> > > > > > > > > close together as possible
> since
> > we
> > > only
> > > > want one
> > > > > magic bump
> > > > > > > and
> > > > > > > > some
> > > > > > > > > users
> > > > > > > > > deploy trunk. The level of
> > effort to
> > > change
> > > > the
> > > > > format for
> > > > > > > > headers
> > > > > > > > > seems
> > > > > > > > > not too high. Do you
> agree? My
> > guess
> > > is that
> > > > the
> > > > > KIP-98
> > > > > > message
> > > > > > > > format
> > > > > > > > > patch will take 2-3 weeks
> to
> > review
> > > before we
> > > > > merge to trunk,
> > > > > > > so
> > > > > > > > you
> > > > > > > > > could
> > > > > > > > > hold off implementing
> until that
> > patch
> > > has
> > > > somewhat
> > > > > > stabilized.
> > > > > > > > That
> > > > > > > > > would
> > > > > > > > > save some potential rebase
> pain.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information contained in
> this
> > email is
> > > > strictly
> > > > > confidential
> > > > > > > and
> > > > > > > > for
> > > > > > > > > the use of the addressee only,
> unless
> > > otherwise
> > > > > indicated. If you
> > > > > > > > are not
> > > > > > > > > the intended recipient, please
> do
> > not read,
> > > > copy, use
> > > > > or disclose
> > > > > > > to
> > > > > > > > others
> > > > > > > > > this message or any attachment.
> > Please also
> > > > notify the
> > > > > sender by
> > > > > > > > replying
> > > > > > > > > to this email or by telephone
> > (+44(020
> > > 7896 0011)
> > > > and
> > > > > then delete
> > > > > > > the
> > > > > > > > > email and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc)
> > > > > that do
> > > > > > not
> > > > > > > > relate
> > > > > > > > > to the official business of
> this
> > company
> > > shall be
> > > > > understood as
> > > > > > > > neither
> > > > > > > > > given nor endorsed by it. IG
> is a
> > trading
> > > name
> > > > of IG
> > > > > Markets
> > > > > > > Limited
> > > > > > > > (a
> > > > > > > > > company registered in England
> and
> > Wales,
> > > company
> > > > > number 04008957)
> > > > > > > > and IG
> > > > > > > > > Index Limited (a company
> registered
> > in
> > > England
> > > > and
> > > > > Wales, company
> > > > > > > > number
> > > > > > > > > 01190902). Registered address
> at
> > Cannon
> > > Bridge
> > > > House,
> > > > > 25 Dowgate
> > > > > > > > Hill,
> > > > > > > > > London EC4R 2YA. Both IG
> Markets
> > Limited
> > > > (register
> > > > > number 195355)
> > > > > > > > and IG
> > > > > > > > > Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated
> > > > > > > > by the
> > > > > > > > > Financial Conduct Authority.
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this
> email is
> > > strictly
> > > > > confidential and
> > > > > > for
> > > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated.
> > > > > If you are
> > > > > > not
> > > > > > > > the intended recipient, please do
> not read,
> > > copy, use
> > > > or
> > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please also
> > > notify the
> > > > > sender by
> > > > > > replying
> > > > > > > > to this email or by telephone
> (+44(020 7896
> > > 0011) and
> > > > then
> > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that
> > > > do not
> > > > > relate to
> > > > > > > the
> > > > > > > > official business of this company
> shall be
> > > understood
> > > > as
> > > > > neither given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a trading name
> of IG
> > > Markets
> > > > Limited
> > > > > (a company
> > > > > > > > registered in England and Wales,
> company
> > number
> > > > 04008957)
> > > > > and IG Index
> > > > > > > > Limited (a company registered in
> England
> > and
> > > Wales,
> > > > company
> > > > > number
> > > > > > > > 01190902). Registered address at
> Cannon
> > Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > > (register
> > > > number
> > > > > 195355) and
> > > > > > IG
> > > > > > > > Index Limited (register number
> 114059) are
> > > authorised
> > > > and
> > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > > The information contained in this
> email is
> > strictly
> > > > > confidential and for
> > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If
> > > > > you are not
> > > > > > > the intended recipient, please do not
> read,
> > copy,
> > > use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please
> also
> > notify
> > > the
> > > > sender
> > > > > by replying
> > > > > > > to this email or by telephone (+44(020
> 7896
> > 0011)
> > > and
> > > > then
> > > > > delete the
> > > > > > email
> > > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to
> > > > > > the
> > > > > > > official business of this company
> shall be
> > > understood as
> > > > > neither given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name
> of IG
> > Markets
> > > > Limited (a
> > > > > company
> > > > > > > registered in England and Wales,
> company
> > number
> > > > 04008957) and
> > > > > IG Index
> > > > > > > Limited (a company registered in
> England and
> > Wales,
> > > > company
> > > > > number
> > > > > > > 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > > Dowgate Hill,
> > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > (register
> > > number
> > > > > 195355) and IG
> > > > > > > Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > > The information contained in this email
> is
> > strictly
> > > > confidential
> > > > > and for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated.
> > > > If
> > > > > you are not
> > > > > > the intended recipient, please do not
> read,
> > copy,
> > > use or
> > > > > disclose to others
> > > > > > this message or any attachment. Please
> also
> > notify
> > > the
> > > > sender by
> > > > > replying
> > > > > > to this email or by telephone (+44(020
> 7896
> > 0011)
> > > and then
> > > > > delete the email
> > > > > > and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > relate to the
> > > > > > official business of this company shall
> be
> > > understood as
> > > > neither
> > > > > given nor
> > > > > > endorsed by it. IG is a trading name of
> IG
> > Markets
> > > Limited
> > > > (a
> > > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957)
> > > > and IG
> > > > > Index
> > > > > > Limited (a company registered in England
> and
> > Wales,
> > > company
> > > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > > House, 25
> > > > Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > > 195355) and IG
> > > > > > Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > > confidential and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you
> > > > are not
> > > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify
> the
> > sender
> > > by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then
> > > delete
> > > > the email
> > > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate
> > > > to the
> > > > > official business of this company shall be
> understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG
> > > > Index
> > > > > Limited (a company registered in England and Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you
> > > are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> > > the email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> > > to the
> > > > official business of this company shall be understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential
> and
> > for the use of the addressee only, unless otherwise indicated. If
> you are
> > not the intended recipient, please do not read, copy, use or
> disclose to
> > others this message or any attachment. Please also notify the sender
> by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the official business of this company shall be understood
> as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and IG Index Limited (register number 114059) are authorised
> and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Jason,
Sorry I thought this was the agreed compromise to provide an api that avoid boiler plate in return for immutabilty.
If not then mutability will be needed as a goal is to have a single clean method call to append/remove a header.
Cheers
Mike
On 24/02/2017, 22:15, "Jason Gustafson" <ja...@confluent.io> wrote:
Hey Michael,
I didn't actually comment on the new methods for ProducerRecord and
ConsumerRecord. If they only save some boilerplate, I'd just as well not
have them.
Also a couple minor comments:
1. Is the intent of `Headers.filter` to include or exclude the headers
matching the key? Can you add a javadoc to clarify?
2. The KIP mentions that we will introduce V4 of FetchRequest and V4 of
ProduceRequest. Can you change this to say that the changes will piggyback
onto V3 of ProduceRequest and V4 of FetchRequest which were introduced in
KIP-98?
The rest of the KIP looks good to me.
-Jason
On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <Mi...@ig.com>
wrote:
> I’ve added the methods on the ProducerRecord that will return a new
> instance of ProducerRecord with modified headers.
>
> On 24/02/2017, 19:22, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Pattern.compile is expensive, and even if cached String.equals is
> faster than matched. also if we end up with an internal map in future for
> performance it will be easier to be by key.
>
> As all that's needed is to get header by key.
>
> With like the other arguements of let's implement simple and then we
> can always add pattern later as well if it's found it's needed. (As noted
> it's easier to add methods than to take away)
>
> Great I'll update kip with extra methods on producerecord and a note
> that new objects are returned by method calls.
>
>
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Friday, February 24, 2017 6:51:45 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The APIs in the current KIP look good to me. Just a couple questions:
> why
> does filter not return Headers? Also would it be useful if the key is a
> regex?
>
> On the point of immutability.. One option might be to use a mutable
> object
> only when passing the headers through the interceptor chain. I think as
> long as we resort to mutability only when clear performance results
> show
> that it is worthwhile, I am satisfied. As Ismael noted, for common
> scenarios it is possible to get reasonable performance with immutable
> objects.
>
> -Jason
>
> On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > Hi
> >
> > On 1, How can you guarantee two separate implemented clients would
> add
> > the headers in the same order we are not specifying an order at the
> > protocol level (nor should we) with regards to keyA being ordered
> before
> > keyB? We shouldn’t be expecting keyA to be always set before keyB.
> >
> > On 2, I believe we have changed the naming based on feedback from
> Jason
> > already, e.g. we don’t have “get” method that inferred O(1)
> performance,
> > like wise nor “put” but we have an “append”
> >
> > On 3, in the KafkaProducer, I think we have mutability already, the
> value
> > for time is changed if it is null, at the point of send:
> > “
> > long timestamp = record.timestamp() == null ?
> > time.milliseconds() : record.timestamp();
> > “
> >
> > As such the timestamp is already mutable, so what’s the difference
> here,
> > we already have some mixed semantics. On timestamp.
> > e.g. currently if I send to records with timestamp not set, the wire
> > binary sent the value for the timestamp would be different, as such
> we have
> > mutation for the same record.
> >
> > On 4, I think we should not expect not 1 or 2 headers, but infact
> 10’s of
> > headers. This is the concern on immutable headers, whilst the append
> > self-reference works nicely, what if someone needs to remove a
> header?
> >
> > Trying to get this moving:
> >
> > If we really wanted Immutable Headers and essentially you guys wont
> give
> > +1 for it without.
> >
> > Whats the feeling for adding methods to ProducerRecord that does the
> > boiler plate code or creating a new ProducerRecord with the altered
> new
> > headers (appended or removed) inside. E.g.
> >
> > ProducerRecord {
> >
> >
> > ProducerRecord append(Iterable<Headers> headersToAppend){
> > return new ProducerRecord(key, value, headers.append(
> headersToAppend),
> > ….)
> > }
> >
> > ProducerRecord remove(Iterable<Headers> headersToAppend){
> > return new ProducerRecord(key, value, headers.remove(
> headersToAppend),
> > ….)
> > }
> >
> > }
> >
> > Were the headers methods actually returns new objects, and the
> producer
> > records methods create a new producer record with all the current
> values,
> > but with the new modified headers.
> >
> > Then interceptors / code return this new object?
> >
> >
> > Cheers
> > Mike
> >
> >
> >
> >
> >
> >
> > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael Juma" <
> > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> >
> > Hi Michael,
> >
> > Did you mean that you were happy to compromise to keep it
> mutable or
> > immutable? You wrote the former, but it sounded from the
> sentence that
> > it
> > could have been a typo. So, my thoughts on this is that there
> are a few
> > things to take into account:
> >
> > 1. Semantics
> > 2. Simplicity of use (the common operations should be easy to do)
> > 3. If it's easy to reason about and safe (immutability helps
> with this)
> > 4. Efficiency (both memory and CPU usage)
> >
> > Regarding 1, I think it would be good to be very clear about the
> > guarantees
> > that we are providing. It seems that we are saying that keys are
> > unordered,
> > but what about the case where there are multiple values for the
> same
> > key?
> > It seems that for some use cases (e.g. lineage), it may be
> useful to
> > add
> > values to the same key while preserving the order.
> >
> > Regarding 2, I agree that it's useful to have methods in
> `Headers` for
> > the
> > very common use cases although we have to be careful with the
> naming to
> > avoid giving the wrong impression. Also, when it comes to
> `Map.Entry`,
> > I
> > think I'd prefer a `toMap` method that simply gives the user a
> Map if
> > that's what they want (it makes it clear that there's a
> conversion
> > happening).
> >
> > Regarding 3, the concern I have if we make the headers mutable
> is that
> > it
> > seems to introduce some inconsistencies and potential edge
> cases. At
> > the
> > moment, it's safe to keep a ProducerRecord and resend it, for
> example.
> > If
> > the record is mutated by an interceptor, then this can lead to
> weird
> > behaviour. Also, it seems inconsistent that one has to create a
> new
> > ProducerRecord to modify the record timestamp, but that one has
> to
> > mutate
> > the record to add headers. It seems like one should either
> embrace
> > immutability or mutability, but mixing both is not ideal.
> >
> > Regarding 4, for the cases where there are a small number of
> headers
> > and/or
> > 1 or 2 interceptors, it doesn't seem difficult to come up with
> > reasonable
> > implementations for mutable and immutable cases that do a good
> enough
> > job.
> > However, if the number of headers and interceptors is large,
> then care
> > is
> > needed for the immutable case to avoid unnecessary copying. A
> simple
> > option
> > that adds little memory overhead if we have Header instances is
> to
> > simply
> > add a self-reference to the previous Header in the linked list.
> >
> > Ismael
> >
> > On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > Im happy to compromise to keep it mutable but move to an append
> > style api.
> > > (as in guava interables concat)
> > >
> > > class Headers {
> > > Headers append(Iterable<Header> headers);
> > > }
> > >
> > >
> > > I don’t think we’d want prepend, this would give the idea of
> > guaranteed
> > > ordering, when in actual fact we don’t provide that guarantee
> (.e.g
> > one
> > > client can put headerA, then headerB, but another could put
> headerB
> > then
> > > headerA, this shouldn’t cause issues), Also what if we changed
> to a
> > hashmap
> > > for the internal implementation, its just a bucket of entries
> no
> > ordering.
> > > I think we just need to provide an api to add/append headers.
> > >
> > > This ok? If so ill update KIP to record this.
> > >
> > > Cheers
> > > Mike
> > >
> > > On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> > >
> > > The point about usability is fair. It's also reasonable to
> > expect that
> > > common use cases such as appending headers should be done
> > efficiently.
> > >
> > > Perhaps we could compromise with something like this?
> > >
> > > class Headers {
> > > Headers append(Iterable<Header> headers);
> > > Headers prepend(Iterable<Header> headers);
> > > }
> > >
> > > That retains ease of use while still giving ourselves some
> > flexibility
> > > in
> > > the implementation.
> > >
> > > -Jason
> > >
> > >
> > > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > I wasn’t referring to the headers needing to be copied,
> im
> > meaning
> > > the
> > > > fact we’d be forcing a new producer record to be
> created, with
> > all
> > > the
> > > > contents copied.
> > > >
> > > > i.e what will happen is utility method will be created
> or end
> > up
> > > being
> > > > used, which does this, and returns the new ProducerRecord
> > instance.
> > > >
> > > > ProducerRecord addHeader(ProducerRecord record, Header
> > header){
> > > > Return New ProducerRecord(record.key, record.value,
> > > record.timestamp…..,
> > > > record.headers.concat(header))
> > > > }
> > > >
> > > > To me this seems ugly, but will be inevitable if we
> don’t make
> > adding
> > > > headers to existing records a simple clean method call.
> > > >
> > > >
> > > >
> > > > On 22/02/2017, 22:57, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > Lazy init can achieve/avoid that.
> > > >
> > > > Re the concat, why don’t we implement that inside the
> > Headers
> > > rather
> > > > than causing everyone to implement this as adding
> headers in
> > > interceptors
> > > > will be a dominant use case. We want a user friendly API.
> > Having as
> > > a user
> > > > having to code this instead of having the headers handle
> this
> > for me
> > > seems
> > > > redundant.
> > > >
> > > > On 22/02/2017, 22:34, "Jason Gustafson" <
> > jason@confluent.io>
> > > wrote:
> > > >
> > > > I thought the argument was against creating the
> extra
> > objects
> > > > unnecessarily
> > > > (i.e. if they were not accessed). And note that
> making
> > the
> > > Headers
> > > > immutable doesn't necessarily mean that they
> need to be
> > > copied:
> > > > you can do
> > > > a trick like Guava's Iterables.concat to add
> additional
> > > headers
> > > > without
> > > > changing the underlying collections.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > If the argument for not having a map holding
> the
> > key, value
> > > > pairs is due
> > > > > to garbage creation of HashMap entry's,
> forcing the
> > > creation of
> > > > a whole new
> > > > > producer record to simply add a head, surely is
> > creating
> > > a-lot
> > > > more?
> > > > > ________________________________________
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > >
> > > > > The current producer interceptor API is this:
> > > > >
> > > > > ProducerRecord<K, V> onSend(ProducerRecord<K,
> V>
> > record);
> > > > >
> > > > > So adding a header means creating a new
> > ProducerRecord
> > > with a
> > > > new header
> > > > > added to the current headers and returning it.
> Would
> > that
> > > not
> > > > work?
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > So how would you have this work if not
> mutable
> > where
> > > > interceptors would
> > > > > > add headers?
> > > > > >
> > > > > > Sent using OWA for iPhone
> > > > > > ________________________________________
> > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > > >
> > > > > > I think the point on the mutability of
> Headers is
> > worth
> > > > discussing a
> > > > > little
> > > > > > more. As far as I can tell, once the
> > ProducerRecord (or
> > > > ConsumerRecord)
> > > > > is
> > > > > > constructed, there should be no need to
> further
> > change
> > > the
> > > > headers. Is
> > > > > that
> > > > > > correct? If so, then why not enforce that
> that is
> > the
> > > case
> > > > through the
> > > > > API?
> > > > > > One problem with mutability it that it
> constrains
> > the
> > > > implementation of
> > > > > > Headers. For example, if we were backing
> with a
> > byte
> > > slice,
> > > > would we
> > > > > recopy
> > > > > > the bytes if a header is added or would we
> > maintain a
> > > satellite
> > > > > collection
> > > > > > of added records. Seems not great either
> way. If we
> > > really
> > > > think
> > > > > mutability
> > > > > > is needed, perhaps we could add a method to
> > headers to
> > > convert
> > > > it to a
> > > > > > mutable type (e.g. a Headers.toMap)?
> > > > > >
> > > > > > I'm also with Ismael about exposing
> Headers.get().
> > I
> > > thought
> > > > it might
> > > > > make
> > > > > > sense to have a method like this instead:
> > > > > >
> > > > > > Iterable<Header> findMatching(Pattern
> pattern);
> > > > > >
> > > > > > This makes the (potential) need to scan the
> headers
> > > clear in
> > > > the API. I'd
> > > > > > also be fine exposing no getter at all. In
> > general, Ï
> > > think
> > > > it's good to
> > > > > > start with a minimalistic API and work from
> there
> > since
> > > it's
> > > > always
> > > > > easier
> > > > > > to add methods than remove them.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Hi Ismael
> > > > > > >
> > > > > > > On point 1,
> > > > > > >
> > > > > > > Sure makes sense will update shortly.
> > > > > > >
> > > > > > > On point 2,
> > > > > > >
> > > > > > > Setter/getter typical to
> properties/headers api’s
> > > > traditionally are map
> > > > > > > styled interfaces and what I believe is
> most
> > expected
> > > styled
> > > > thus the
> > > > > > Key,
> > > > > > > Value setter.
> > > > > > > Also it would mean rather than an
> interface, we
> > would
> > > be
> > > > making our
> > > > > > > internal header impl object we have for the
> > array,
> > > exposed.
> > > > E.g. if we
> > > > > > had
> > > > > > > a Map really this would be Map.Entry
> interface,
> > this
> > > is the
> > > > same
> > > > > reasons
> > > > > > on
> > > > > > > the map interface I cannot actually make
> the
> > > underlying Node
> > > > object
> > > > > > that’s
> > > > > > > the implementation for HashMap, so that
> > internals can
> > > be
> > > > changed.
> > > > > > >
> > > > > > >
> > > > > > > On point 3,
> > > > > > >
> > > > > > > I think it people do expect it to be
> performant,
> > thus
> > > why
> > > > originally
> > > > > > > concern I raised with using an array for
> to me
> > is an
> > > early
> > > > memory
> > > > > > > optimisation. I think the user experience
> of
> > > > properties/headers is on a
> > > > > > > get/set model. This is why its important
> we have
> > > > encapsulated logic
> > > > > that
> > > > > > > then allows us to change this to a map, if
> this
> > > becomes and
> > > > issue, and
> > > > > > the
> > > > > > > memory overhead of hashmap is less so.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com
> on
> > behalf of
> > > > Ismael Juma" <
> > > > > > > ismaelj@gmail.com on behalf of
> ismael@juma.me.uk
> > >
> > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Great to see the progress that has been
> > achieved
> > > on this
> > > > one. :) A
> > > > > > few
> > > > > > > comments regarding the APIs (I'm still
> > reviewing
> > > the
> > > > message format
> > > > > > > changes):
> > > > > > >
> > > > > > > 1. Nit: `getHeaders` in
> `ProducerRecord` and
> > > > `ConsumerRecord`
> > > > > should
> > > > > > be
> > > > > > > named `headers` (we avoid the `get`
> prefix in
> > > Kafka)
> > > > > > >
> > > > > > > 2. The `Headers` class is mutable
> (there's
> > an `add`
> > > > method). Does
> > > > > it
> > > > > > > need
> > > > > > > to be? If so, it would be good to
> explain
> > why.
> > > Related
> > > > to that, we
> > > > > > > should
> > > > > > > also explain the thinking around
> > thread-safety. If
> > > we
> > > > keep the
> > > > > `add`
> > > > > > > method, it may make sense for it to
> take a
> > `Header`
> > > > (that way we
> > > > > can
> > > > > > > add
> > > > > > > things to `Header` without changing the
> > interface).
> > > > > > >
> > > > > > > 3. Do we need the `Headers.get()`
> method?
> > People
> > > usually
> > > > assume
> > > > > that
> > > > > > > `get`
> > > > > > > would be efficient, but depending on
> the
> > > implementation
> > > > (the
> > > > > current
> > > > > > > proposal states that an array would be
> > used), it
> > > may not
> > > > be. If we
> > > > > > > expect
> > > > > > > the number of headers to be small, it
> doesn't
> > > matter
> > > > though.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Tue, Feb 21, 2017 at 6:38 PM,
> Michael
> > Pearce <
> > > > > > Michael.Pearce@ig.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > Have converted the interface/api
> bullets
> > into
> > > > interface code
> > > > > > > snippets.
> > > > > > > >
> > > > > > > > Agreed implementation won’t take too
> long.
> > We
> > > have
> > > > early versions
> > > > > > > already.
> > > > > > > > Maybe a week before you think about
> > merging I
> > > would
> > > > assume it
> > > > > would
> > > > > > > be more
> > > > > > > > stabilised? I was thinking then we
> could
> > fork
> > > from
> > > > your confluent
> > > > > > > branch,
> > > > > > > > making and then holding KIP-82
> changes in
> > a patch
> > > > file, that we
> > > > > can
> > > > > > > then
> > > > > > > > re-fork from apache once KIP98 final
> is
> > merged,
> > > and
> > > > apply patch
> > > > > > with
> > > > > > > last
> > > > > > > > minute changes.
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > > Mike
> > > > > > > >
> > > > > > > >
> > > > > > > > On 22/02/2017, 00:56, "Jason
> Gustafson" <
> > > > jason@confluent.io>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hey Michael,
> > > > > > > >
> > > > > > > > Awesome. I have a minor request.
> The
> > APIs are
> > > > currently
> > > > > > > documented as a
> > > > > > > > wiki list. Would you mind adding
> a code
> > > snippet
> > > > instead?
> > > > > It's a
> > > > > > > bit
> > > > > > > > easier
> > > > > > > > to process.
> > > > > > > >
> > > > > > > > How will be best to manage this,
> as we
> > will
> > > > obviously build
> > > > > off
> > > > > > > your
> > > > > > > > KIP’s
> > > > > > > > > protocol changes, to avoid a
> merge
> > hell,
> > > should
> > > > we branch
> > > > > > from
> > > > > > > your
> > > > > > > > branch
> > > > > > > > > in the confluent repo or is it
> worth
> > > having a
> > > > KIP-98
> > > > > special
> > > > > > > branch
> > > > > > > > in the
> > > > > > > > > apache git, that we can
> branch/fork
> > from?
> > > > > > > >
> > > > > > > >
> > > > > > > > I was thinking about this also.
> > Ideally we'd
> > > like
> > > > to get the
> > > > > > > changes
> > > > > > > > in as
> > > > > > > > close together as possible since
> we
> > only
> > > want one
> > > > magic bump
> > > > > > and
> > > > > > > some
> > > > > > > > users
> > > > > > > > deploy trunk. The level of
> effort to
> > change
> > > the
> > > > format for
> > > > > > > headers
> > > > > > > > seems
> > > > > > > > not too high. Do you agree? My
> guess
> > is that
> > > the
> > > > KIP-98
> > > > > message
> > > > > > > format
> > > > > > > > patch will take 2-3 weeks to
> review
> > before we
> > > > merge to trunk,
> > > > > > so
> > > > > > > you
> > > > > > > > could
> > > > > > > > hold off implementing until that
> patch
> > has
> > > somewhat
> > > > > stabilized.
> > > > > > > That
> > > > > > > > would
> > > > > > > > save some potential rebase pain.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this
> email is
> > > strictly
> > > > confidential
> > > > > > and
> > > > > > > for
> > > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If you
> > > > > > > are not
> > > > > > > > the intended recipient, please do
> not read,
> > > copy, use
> > > > or disclose
> > > > > > to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please also
> > > notify the
> > > > sender by
> > > > > > > replying
> > > > > > > > to this email or by telephone
> (+44(020
> > 7896 0011)
> > > and
> > > > then delete
> > > > > > the
> > > > > > > > email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that do
> > > > > not
> > > > > > > relate
> > > > > > > > to the official business of this
> company
> > shall be
> > > > understood as
> > > > > > > neither
> > > > > > > > given nor endorsed by it. IG is a
> trading
> > name
> > > of IG
> > > > Markets
> > > > > > Limited
> > > > > > > (a
> > > > > > > > company registered in England and
> Wales,
> > company
> > > > number 04008957)
> > > > > > > and IG
> > > > > > > > Index Limited (a company registered
> in
> > England
> > > and
> > > > Wales, company
> > > > > > > number
> > > > > > > > 01190902). Registered address at
> Cannon
> > Bridge
> > > House,
> > > > 25 Dowgate
> > > > > > > Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > > (register
> > > > number 195355)
> > > > > > > and IG
> > > > > > > > Index Limited (register number
> 114059) are
> > > authorised
> > > > and
> > > > > regulated
> > > > > > > by the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > strictly
> > > > confidential and
> > > > > for
> > > > > > > the use of the addressee only, unless
> otherwise
> > > indicated.
> > > > If you are
> > > > > not
> > > > > > > the intended recipient, please do not read,
> > copy, use
> > > or
> > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please also
> > notify the
> > > > sender by
> > > > > replying
> > > > > > > to this email or by telephone (+44(020 7896
> > 0011) and
> > > then
> > > > delete the
> > > > > > email
> > > > > > > and any copies of it. Opinions, conclusion
> (etc)
> > that
> > > do not
> > > > relate to
> > > > > > the
> > > > > > > official business of this company shall be
> > understood
> > > as
> > > > neither given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name of IG
> > Markets
> > > Limited
> > > > (a company
> > > > > > > registered in England and Wales, company
> number
> > > 04008957)
> > > > and IG Index
> > > > > > > Limited (a company registered in England
> and
> > Wales,
> > > company
> > > > number
> > > > > > > 01190902). Registered address at Cannon
> Bridge
> > House,
> > > 25
> > > > Dowgate Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and
> > > > > IG
> > > > > > > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > > The information contained in this email is
> strictly
> > > > confidential and for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you are not
> > > > > > the intended recipient, please do not read,
> copy,
> > use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify
> > the
> > > sender
> > > > by replying
> > > > > > to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then
> > > > delete the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > relate to
> > > > > the
> > > > > > official business of this company shall be
> > understood as
> > > > neither given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG Index
> > > > > > Limited (a company registered in England and
> Wales,
> > > company
> > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > House, 25
> > > > Dowgate Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > > 195355) and IG
> > > > > > Index Limited (register number 114059) are
> > authorised and
> > > > regulated by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is
> strictly
> > > confidential
> > > > and for
> > > > > the use of the addressee only, unless otherwise
> > indicated.
> > > If
> > > > you are not
> > > > > the intended recipient, please do not read,
> copy,
> > use or
> > > > disclose to others
> > > > > this message or any attachment. Please also
> notify
> > the
> > > sender by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896
> 0011)
> > and then
> > > > delete the email
> > > > > and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > relate to the
> > > > > official business of this company shall be
> > understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited
> > > (a
> > > > company
> > > > > registered in England and Wales, company number
> > 04008957)
> > > and IG
> > > > Index
> > > > > Limited (a company registered in England and
> Wales,
> > company
> > > > number
> > > > > 01190902). Registered address at Cannon Bridge
> > House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > > 195355) and IG
> > > > > Index Limited (register number 114059) are
> > authorised and
> > > > regulated by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you
> > > are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> > > the email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> > > to the
> > > > official business of this company shall be understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Michael,
I didn't actually comment on the new methods for ProducerRecord and
ConsumerRecord. If they only save some boilerplate, I'd just as well not
have them.
Also a couple minor comments:
1. Is the intent of `Headers.filter` to include or exclude the headers
matching the key? Can you add a javadoc to clarify?
2. The KIP mentions that we will introduce V4 of FetchRequest and V4 of
ProduceRequest. Can you change this to say that the changes will piggyback
onto V3 of ProduceRequest and V4 of FetchRequest which were introduced in
KIP-98?
The rest of the KIP looks good to me.
-Jason
On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <Mi...@ig.com>
wrote:
> I’ve added the methods on the ProducerRecord that will return a new
> instance of ProducerRecord with modified headers.
>
> On 24/02/2017, 19:22, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Pattern.compile is expensive, and even if cached String.equals is
> faster than matched. also if we end up with an internal map in future for
> performance it will be easier to be by key.
>
> As all that's needed is to get header by key.
>
> With like the other arguements of let's implement simple and then we
> can always add pattern later as well if it's found it's needed. (As noted
> it's easier to add methods than to take away)
>
> Great I'll update kip with extra methods on producerecord and a note
> that new objects are returned by method calls.
>
>
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Friday, February 24, 2017 6:51:45 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The APIs in the current KIP look good to me. Just a couple questions:
> why
> does filter not return Headers? Also would it be useful if the key is a
> regex?
>
> On the point of immutability.. One option might be to use a mutable
> object
> only when passing the headers through the interceptor chain. I think as
> long as we resort to mutability only when clear performance results
> show
> that it is worthwhile, I am satisfied. As Ismael noted, for common
> scenarios it is possible to get reasonable performance with immutable
> objects.
>
> -Jason
>
> On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > Hi
> >
> > On 1, How can you guarantee two separate implemented clients would
> add
> > the headers in the same order we are not specifying an order at the
> > protocol level (nor should we) with regards to keyA being ordered
> before
> > keyB? We shouldn’t be expecting keyA to be always set before keyB.
> >
> > On 2, I believe we have changed the naming based on feedback from
> Jason
> > already, e.g. we don’t have “get” method that inferred O(1)
> performance,
> > like wise nor “put” but we have an “append”
> >
> > On 3, in the KafkaProducer, I think we have mutability already, the
> value
> > for time is changed if it is null, at the point of send:
> > “
> > long timestamp = record.timestamp() == null ?
> > time.milliseconds() : record.timestamp();
> > “
> >
> > As such the timestamp is already mutable, so what’s the difference
> here,
> > we already have some mixed semantics. On timestamp.
> > e.g. currently if I send to records with timestamp not set, the wire
> > binary sent the value for the timestamp would be different, as such
> we have
> > mutation for the same record.
> >
> > On 4, I think we should not expect not 1 or 2 headers, but infact
> 10’s of
> > headers. This is the concern on immutable headers, whilst the append
> > self-reference works nicely, what if someone needs to remove a
> header?
> >
> > Trying to get this moving:
> >
> > If we really wanted Immutable Headers and essentially you guys wont
> give
> > +1 for it without.
> >
> > Whats the feeling for adding methods to ProducerRecord that does the
> > boiler plate code or creating a new ProducerRecord with the altered
> new
> > headers (appended or removed) inside. E.g.
> >
> > ProducerRecord {
> >
> >
> > ProducerRecord append(Iterable<Headers> headersToAppend){
> > return new ProducerRecord(key, value, headers.append(
> headersToAppend),
> > ….)
> > }
> >
> > ProducerRecord remove(Iterable<Headers> headersToAppend){
> > return new ProducerRecord(key, value, headers.remove(
> headersToAppend),
> > ….)
> > }
> >
> > }
> >
> > Were the headers methods actually returns new objects, and the
> producer
> > records methods create a new producer record with all the current
> values,
> > but with the new modified headers.
> >
> > Then interceptors / code return this new object?
> >
> >
> > Cheers
> > Mike
> >
> >
> >
> >
> >
> >
> > On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael Juma" <
> > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> >
> > Hi Michael,
> >
> > Did you mean that you were happy to compromise to keep it
> mutable or
> > immutable? You wrote the former, but it sounded from the
> sentence that
> > it
> > could have been a typo. So, my thoughts on this is that there
> are a few
> > things to take into account:
> >
> > 1. Semantics
> > 2. Simplicity of use (the common operations should be easy to do)
> > 3. If it's easy to reason about and safe (immutability helps
> with this)
> > 4. Efficiency (both memory and CPU usage)
> >
> > Regarding 1, I think it would be good to be very clear about the
> > guarantees
> > that we are providing. It seems that we are saying that keys are
> > unordered,
> > but what about the case where there are multiple values for the
> same
> > key?
> > It seems that for some use cases (e.g. lineage), it may be
> useful to
> > add
> > values to the same key while preserving the order.
> >
> > Regarding 2, I agree that it's useful to have methods in
> `Headers` for
> > the
> > very common use cases although we have to be careful with the
> naming to
> > avoid giving the wrong impression. Also, when it comes to
> `Map.Entry`,
> > I
> > think I'd prefer a `toMap` method that simply gives the user a
> Map if
> > that's what they want (it makes it clear that there's a
> conversion
> > happening).
> >
> > Regarding 3, the concern I have if we make the headers mutable
> is that
> > it
> > seems to introduce some inconsistencies and potential edge
> cases. At
> > the
> > moment, it's safe to keep a ProducerRecord and resend it, for
> example.
> > If
> > the record is mutated by an interceptor, then this can lead to
> weird
> > behaviour. Also, it seems inconsistent that one has to create a
> new
> > ProducerRecord to modify the record timestamp, but that one has
> to
> > mutate
> > the record to add headers. It seems like one should either
> embrace
> > immutability or mutability, but mixing both is not ideal.
> >
> > Regarding 4, for the cases where there are a small number of
> headers
> > and/or
> > 1 or 2 interceptors, it doesn't seem difficult to come up with
> > reasonable
> > implementations for mutable and immutable cases that do a good
> enough
> > job.
> > However, if the number of headers and interceptors is large,
> then care
> > is
> > needed for the immutable case to avoid unnecessary copying. A
> simple
> > option
> > that adds little memory overhead if we have Header instances is
> to
> > simply
> > add a self-reference to the previous Header in the linked list.
> >
> > Ismael
> >
> > On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > Im happy to compromise to keep it mutable but move to an append
> > style api.
> > > (as in guava interables concat)
> > >
> > > class Headers {
> > > Headers append(Iterable<Header> headers);
> > > }
> > >
> > >
> > > I don’t think we’d want prepend, this would give the idea of
> > guaranteed
> > > ordering, when in actual fact we don’t provide that guarantee
> (.e.g
> > one
> > > client can put headerA, then headerB, but another could put
> headerB
> > then
> > > headerA, this shouldn’t cause issues), Also what if we changed
> to a
> > hashmap
> > > for the internal implementation, its just a bucket of entries
> no
> > ordering.
> > > I think we just need to provide an api to add/append headers.
> > >
> > > This ok? If so ill update KIP to record this.
> > >
> > > Cheers
> > > Mike
> > >
> > > On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> > >
> > > The point about usability is fair. It's also reasonable to
> > expect that
> > > common use cases such as appending headers should be done
> > efficiently.
> > >
> > > Perhaps we could compromise with something like this?
> > >
> > > class Headers {
> > > Headers append(Iterable<Header> headers);
> > > Headers prepend(Iterable<Header> headers);
> > > }
> > >
> > > That retains ease of use while still giving ourselves some
> > flexibility
> > > in
> > > the implementation.
> > >
> > > -Jason
> > >
> > >
> > > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > I wasn’t referring to the headers needing to be copied,
> im
> > meaning
> > > the
> > > > fact we’d be forcing a new producer record to be
> created, with
> > all
> > > the
> > > > contents copied.
> > > >
> > > > i.e what will happen is utility method will be created
> or end
> > up
> > > being
> > > > used, which does this, and returns the new ProducerRecord
> > instance.
> > > >
> > > > ProducerRecord addHeader(ProducerRecord record, Header
> > header){
> > > > Return New ProducerRecord(record.key, record.value,
> > > record.timestamp…..,
> > > > record.headers.concat(header))
> > > > }
> > > >
> > > > To me this seems ugly, but will be inevitable if we
> don’t make
> > adding
> > > > headers to existing records a simple clean method call.
> > > >
> > > >
> > > >
> > > > On 22/02/2017, 22:57, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > Lazy init can achieve/avoid that.
> > > >
> > > > Re the concat, why don’t we implement that inside the
> > Headers
> > > rather
> > > > than causing everyone to implement this as adding
> headers in
> > > interceptors
> > > > will be a dominant use case. We want a user friendly API.
> > Having as
> > > a user
> > > > having to code this instead of having the headers handle
> this
> > for me
> > > seems
> > > > redundant.
> > > >
> > > > On 22/02/2017, 22:34, "Jason Gustafson" <
> > jason@confluent.io>
> > > wrote:
> > > >
> > > > I thought the argument was against creating the
> extra
> > objects
> > > > unnecessarily
> > > > (i.e. if they were not accessed). And note that
> making
> > the
> > > Headers
> > > > immutable doesn't necessarily mean that they
> need to be
> > > copied:
> > > > you can do
> > > > a trick like Guava's Iterables.concat to add
> additional
> > > headers
> > > > without
> > > > changing the underlying collections.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > If the argument for not having a map holding
> the
> > key, value
> > > > pairs is due
> > > > > to garbage creation of HashMap entry's,
> forcing the
> > > creation of
> > > > a whole new
> > > > > producer record to simply add a head, surely is
> > creating
> > > a-lot
> > > > more?
> > > > > ________________________________________
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > >
> > > > > The current producer interceptor API is this:
> > > > >
> > > > > ProducerRecord<K, V> onSend(ProducerRecord<K,
> V>
> > record);
> > > > >
> > > > > So adding a header means creating a new
> > ProducerRecord
> > > with a
> > > > new header
> > > > > added to the current headers and returning it.
> Would
> > that
> > > not
> > > > work?
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > So how would you have this work if not
> mutable
> > where
> > > > interceptors would
> > > > > > add headers?
> > > > > >
> > > > > > Sent using OWA for iPhone
> > > > > > ________________________________________
> > > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > > >
> > > > > > I think the point on the mutability of
> Headers is
> > worth
> > > > discussing a
> > > > > little
> > > > > > more. As far as I can tell, once the
> > ProducerRecord (or
> > > > ConsumerRecord)
> > > > > is
> > > > > > constructed, there should be no need to
> further
> > change
> > > the
> > > > headers. Is
> > > > > that
> > > > > > correct? If so, then why not enforce that
> that is
> > the
> > > case
> > > > through the
> > > > > API?
> > > > > > One problem with mutability it that it
> constrains
> > the
> > > > implementation of
> > > > > > Headers. For example, if we were backing
> with a
> > byte
> > > slice,
> > > > would we
> > > > > recopy
> > > > > > the bytes if a header is added or would we
> > maintain a
> > > satellite
> > > > > collection
> > > > > > of added records. Seems not great either
> way. If we
> > > really
> > > > think
> > > > > mutability
> > > > > > is needed, perhaps we could add a method to
> > headers to
> > > convert
> > > > it to a
> > > > > > mutable type (e.g. a Headers.toMap)?
> > > > > >
> > > > > > I'm also with Ismael about exposing
> Headers.get().
> > I
> > > thought
> > > > it might
> > > > > make
> > > > > > sense to have a method like this instead:
> > > > > >
> > > > > > Iterable<Header> findMatching(Pattern
> pattern);
> > > > > >
> > > > > > This makes the (potential) need to scan the
> headers
> > > clear in
> > > > the API. I'd
> > > > > > also be fine exposing no getter at all. In
> > general, Ï
> > > think
> > > > it's good to
> > > > > > start with a minimalistic API and work from
> there
> > since
> > > it's
> > > > always
> > > > > easier
> > > > > > to add methods than remove them.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Hi Ismael
> > > > > > >
> > > > > > > On point 1,
> > > > > > >
> > > > > > > Sure makes sense will update shortly.
> > > > > > >
> > > > > > > On point 2,
> > > > > > >
> > > > > > > Setter/getter typical to
> properties/headers api’s
> > > > traditionally are map
> > > > > > > styled interfaces and what I believe is
> most
> > expected
> > > styled
> > > > thus the
> > > > > > Key,
> > > > > > > Value setter.
> > > > > > > Also it would mean rather than an
> interface, we
> > would
> > > be
> > > > making our
> > > > > > > internal header impl object we have for the
> > array,
> > > exposed.
> > > > E.g. if we
> > > > > > had
> > > > > > > a Map really this would be Map.Entry
> interface,
> > this
> > > is the
> > > > same
> > > > > reasons
> > > > > > on
> > > > > > > the map interface I cannot actually make
> the
> > > underlying Node
> > > > object
> > > > > > that’s
> > > > > > > the implementation for HashMap, so that
> > internals can
> > > be
> > > > changed.
> > > > > > >
> > > > > > >
> > > > > > > On point 3,
> > > > > > >
> > > > > > > I think it people do expect it to be
> performant,
> > thus
> > > why
> > > > originally
> > > > > > > concern I raised with using an array for
> to me
> > is an
> > > early
> > > > memory
> > > > > > > optimisation. I think the user experience
> of
> > > > properties/headers is on a
> > > > > > > get/set model. This is why its important
> we have
> > > > encapsulated logic
> > > > > that
> > > > > > > then allows us to change this to a map, if
> this
> > > becomes and
> > > > issue, and
> > > > > > the
> > > > > > > memory overhead of hashmap is less so.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com
> on
> > behalf of
> > > > Ismael Juma" <
> > > > > > > ismaelj@gmail.com on behalf of
> ismael@juma.me.uk
> > >
> > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Great to see the progress that has been
> > achieved
> > > on this
> > > > one. :) A
> > > > > > few
> > > > > > > comments regarding the APIs (I'm still
> > reviewing
> > > the
> > > > message format
> > > > > > > changes):
> > > > > > >
> > > > > > > 1. Nit: `getHeaders` in
> `ProducerRecord` and
> > > > `ConsumerRecord`
> > > > > should
> > > > > > be
> > > > > > > named `headers` (we avoid the `get`
> prefix in
> > > Kafka)
> > > > > > >
> > > > > > > 2. The `Headers` class is mutable
> (there's
> > an `add`
> > > > method). Does
> > > > > it
> > > > > > > need
> > > > > > > to be? If so, it would be good to
> explain
> > why.
> > > Related
> > > > to that, we
> > > > > > > should
> > > > > > > also explain the thinking around
> > thread-safety. If
> > > we
> > > > keep the
> > > > > `add`
> > > > > > > method, it may make sense for it to
> take a
> > `Header`
> > > > (that way we
> > > > > can
> > > > > > > add
> > > > > > > things to `Header` without changing the
> > interface).
> > > > > > >
> > > > > > > 3. Do we need the `Headers.get()`
> method?
> > People
> > > usually
> > > > assume
> > > > > that
> > > > > > > `get`
> > > > > > > would be efficient, but depending on
> the
> > > implementation
> > > > (the
> > > > > current
> > > > > > > proposal states that an array would be
> > used), it
> > > may not
> > > > be. If we
> > > > > > > expect
> > > > > > > the number of headers to be small, it
> doesn't
> > > matter
> > > > though.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Tue, Feb 21, 2017 at 6:38 PM,
> Michael
> > Pearce <
> > > > > > Michael.Pearce@ig.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jason,
> > > > > > > >
> > > > > > > > Have converted the interface/api
> bullets
> > into
> > > > interface code
> > > > > > > snippets.
> > > > > > > >
> > > > > > > > Agreed implementation won’t take too
> long.
> > We
> > > have
> > > > early versions
> > > > > > > already.
> > > > > > > > Maybe a week before you think about
> > merging I
> > > would
> > > > assume it
> > > > > would
> > > > > > > be more
> > > > > > > > stabilised? I was thinking then we
> could
> > fork
> > > from
> > > > your confluent
> > > > > > > branch,
> > > > > > > > making and then holding KIP-82
> changes in
> > a patch
> > > > file, that we
> > > > > can
> > > > > > > then
> > > > > > > > re-fork from apache once KIP98 final
> is
> > merged,
> > > and
> > > > apply patch
> > > > > > with
> > > > > > > last
> > > > > > > > minute changes.
> > > > > > > >
> > > > > > > > Cheers
> > > > > > > > Mike
> > > > > > > >
> > > > > > > >
> > > > > > > > On 22/02/2017, 00:56, "Jason
> Gustafson" <
> > > > jason@confluent.io>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hey Michael,
> > > > > > > >
> > > > > > > > Awesome. I have a minor request.
> The
> > APIs are
> > > > currently
> > > > > > > documented as a
> > > > > > > > wiki list. Would you mind adding
> a code
> > > snippet
> > > > instead?
> > > > > It's a
> > > > > > > bit
> > > > > > > > easier
> > > > > > > > to process.
> > > > > > > >
> > > > > > > > How will be best to manage this,
> as we
> > will
> > > > obviously build
> > > > > off
> > > > > > > your
> > > > > > > > KIP’s
> > > > > > > > > protocol changes, to avoid a
> merge
> > hell,
> > > should
> > > > we branch
> > > > > > from
> > > > > > > your
> > > > > > > > branch
> > > > > > > > > in the confluent repo or is it
> worth
> > > having a
> > > > KIP-98
> > > > > special
> > > > > > > branch
> > > > > > > > in the
> > > > > > > > > apache git, that we can
> branch/fork
> > from?
> > > > > > > >
> > > > > > > >
> > > > > > > > I was thinking about this also.
> > Ideally we'd
> > > like
> > > > to get the
> > > > > > > changes
> > > > > > > > in as
> > > > > > > > close together as possible since
> we
> > only
> > > want one
> > > > magic bump
> > > > > > and
> > > > > > > some
> > > > > > > > users
> > > > > > > > deploy trunk. The level of
> effort to
> > change
> > > the
> > > > format for
> > > > > > > headers
> > > > > > > > seems
> > > > > > > > not too high. Do you agree? My
> guess
> > is that
> > > the
> > > > KIP-98
> > > > > message
> > > > > > > format
> > > > > > > > patch will take 2-3 weeks to
> review
> > before we
> > > > merge to trunk,
> > > > > > so
> > > > > > > you
> > > > > > > > could
> > > > > > > > hold off implementing until that
> patch
> > has
> > > somewhat
> > > > > stabilized.
> > > > > > > That
> > > > > > > > would
> > > > > > > > save some potential rebase pain.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in this
> email is
> > > strictly
> > > > confidential
> > > > > > and
> > > > > > > for
> > > > > > > > the use of the addressee only, unless
> > otherwise
> > > > indicated. If you
> > > > > > > are not
> > > > > > > > the intended recipient, please do
> not read,
> > > copy, use
> > > > or disclose
> > > > > > to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please also
> > > notify the
> > > > sender by
> > > > > > > replying
> > > > > > > > to this email or by telephone
> (+44(020
> > 7896 0011)
> > > and
> > > > then delete
> > > > > > the
> > > > > > > > email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that do
> > > > > not
> > > > > > > relate
> > > > > > > > to the official business of this
> company
> > shall be
> > > > understood as
> > > > > > > neither
> > > > > > > > given nor endorsed by it. IG is a
> trading
> > name
> > > of IG
> > > > Markets
> > > > > > Limited
> > > > > > > (a
> > > > > > > > company registered in England and
> Wales,
> > company
> > > > number 04008957)
> > > > > > > and IG
> > > > > > > > Index Limited (a company registered
> in
> > England
> > > and
> > > > Wales, company
> > > > > > > number
> > > > > > > > 01190902). Registered address at
> Cannon
> > Bridge
> > > House,
> > > > 25 Dowgate
> > > > > > > Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > > (register
> > > > number 195355)
> > > > > > > and IG
> > > > > > > > Index Limited (register number
> 114059) are
> > > authorised
> > > > and
> > > > > regulated
> > > > > > > by the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > strictly
> > > > confidential and
> > > > > for
> > > > > > > the use of the addressee only, unless
> otherwise
> > > indicated.
> > > > If you are
> > > > > not
> > > > > > > the intended recipient, please do not read,
> > copy, use
> > > or
> > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please also
> > notify the
> > > > sender by
> > > > > replying
> > > > > > > to this email or by telephone (+44(020 7896
> > 0011) and
> > > then
> > > > delete the
> > > > > > email
> > > > > > > and any copies of it. Opinions, conclusion
> (etc)
> > that
> > > do not
> > > > relate to
> > > > > > the
> > > > > > > official business of this company shall be
> > understood
> > > as
> > > > neither given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name of IG
> > Markets
> > > Limited
> > > > (a company
> > > > > > > registered in England and Wales, company
> number
> > > 04008957)
> > > > and IG Index
> > > > > > > Limited (a company registered in England
> and
> > Wales,
> > > company
> > > > number
> > > > > > > 01190902). Registered address at Cannon
> Bridge
> > House,
> > > 25
> > > > Dowgate Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and
> > > > > IG
> > > > > > > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > > The information contained in this email is
> strictly
> > > > confidential and for
> > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you are not
> > > > > > the intended recipient, please do not read,
> copy,
> > use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify
> > the
> > > sender
> > > > by replying
> > > > > > to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then
> > > > delete the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > relate to
> > > > > the
> > > > > > official business of this company shall be
> > understood as
> > > > neither given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG Index
> > > > > > Limited (a company registered in England and
> Wales,
> > > company
> > > > number
> > > > > > 01190902). Registered address at Cannon
> Bridge
> > House, 25
> > > > Dowgate Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > > 195355) and IG
> > > > > > Index Limited (register number 114059) are
> > authorised and
> > > > regulated by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is
> strictly
> > > confidential
> > > > and for
> > > > > the use of the addressee only, unless otherwise
> > indicated.
> > > If
> > > > you are not
> > > > > the intended recipient, please do not read,
> copy,
> > use or
> > > > disclose to others
> > > > > this message or any attachment. Please also
> notify
> > the
> > > sender by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896
> 0011)
> > and then
> > > > delete the email
> > > > > and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > relate to the
> > > > > official business of this company shall be
> > understood as
> > > neither
> > > > given nor
> > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited
> > > (a
> > > > company
> > > > > registered in England and Wales, company number
> > 04008957)
> > > and IG
> > > > Index
> > > > > Limited (a company registered in England and
> Wales,
> > company
> > > > number
> > > > > 01190902). Registered address at Cannon Bridge
> > House, 25
> > > Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > > 195355) and IG
> > > > > Index Limited (register number 114059) are
> > authorised and
> > > > regulated by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you
> > > are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> > > the email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate
> > > to the
> > > > official business of this company shall be understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
I’ve added the methods on the ProducerRecord that will return a new instance of ProducerRecord with modified headers.
On 24/02/2017, 19:22, "Michael Pearce" <Mi...@ig.com> wrote:
Pattern.compile is expensive, and even if cached String.equals is faster than matched. also if we end up with an internal map in future for performance it will be easier to be by key.
As all that's needed is to get header by key.
With like the other arguements of let's implement simple and then we can always add pattern later as well if it's found it's needed. (As noted it's easier to add methods than to take away)
Great I'll update kip with extra methods on producerecord and a note that new objects are returned by method calls.
Sent using OWA for iPhone
________________________________________
From: Jason Gustafson <ja...@confluent.io>
Sent: Friday, February 24, 2017 6:51:45 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
The APIs in the current KIP look good to me. Just a couple questions: why
does filter not return Headers? Also would it be useful if the key is a
regex?
On the point of immutability.. One option might be to use a mutable object
only when passing the headers through the interceptor chain. I think as
long as we resort to mutability only when clear performance results show
that it is worthwhile, I am satisfied. As Ismael noted, for common
scenarios it is possible to get reasonable performance with immutable
objects.
-Jason
On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi
>
> On 1, How can you guarantee two separate implemented clients would add
> the headers in the same order we are not specifying an order at the
> protocol level (nor should we) with regards to keyA being ordered before
> keyB? We shouldn’t be expecting keyA to be always set before keyB.
>
> On 2, I believe we have changed the naming based on feedback from Jason
> already, e.g. we don’t have “get” method that inferred O(1) performance,
> like wise nor “put” but we have an “append”
>
> On 3, in the KafkaProducer, I think we have mutability already, the value
> for time is changed if it is null, at the point of send:
> “
> long timestamp = record.timestamp() == null ?
> time.milliseconds() : record.timestamp();
> “
>
> As such the timestamp is already mutable, so what’s the difference here,
> we already have some mixed semantics. On timestamp.
> e.g. currently if I send to records with timestamp not set, the wire
> binary sent the value for the timestamp would be different, as such we have
> mutation for the same record.
>
> On 4, I think we should not expect not 1 or 2 headers, but infact 10’s of
> headers. This is the concern on immutable headers, whilst the append
> self-reference works nicely, what if someone needs to remove a header?
>
> Trying to get this moving:
>
> If we really wanted Immutable Headers and essentially you guys wont give
> +1 for it without.
>
> Whats the feeling for adding methods to ProducerRecord that does the
> boiler plate code or creating a new ProducerRecord with the altered new
> headers (appended or removed) inside. E.g.
>
> ProducerRecord {
>
>
> ProducerRecord append(Iterable<Headers> headersToAppend){
> return new ProducerRecord(key, value, headers.append(headersToAppend),
> ….)
> }
>
> ProducerRecord remove(Iterable<Headers> headersToAppend){
> return new ProducerRecord(key, value, headers.remove(headersToAppend),
> ….)
> }
>
> }
>
> Were the headers methods actually returns new objects, and the producer
> records methods create a new producer record with all the current values,
> but with the new modified headers.
>
> Then interceptors / code return this new object?
>
>
> Cheers
> Mike
>
>
>
>
>
>
> On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael Juma" <
> ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>
> Hi Michael,
>
> Did you mean that you were happy to compromise to keep it mutable or
> immutable? You wrote the former, but it sounded from the sentence that
> it
> could have been a typo. So, my thoughts on this is that there are a few
> things to take into account:
>
> 1. Semantics
> 2. Simplicity of use (the common operations should be easy to do)
> 3. If it's easy to reason about and safe (immutability helps with this)
> 4. Efficiency (both memory and CPU usage)
>
> Regarding 1, I think it would be good to be very clear about the
> guarantees
> that we are providing. It seems that we are saying that keys are
> unordered,
> but what about the case where there are multiple values for the same
> key?
> It seems that for some use cases (e.g. lineage), it may be useful to
> add
> values to the same key while preserving the order.
>
> Regarding 2, I agree that it's useful to have methods in `Headers` for
> the
> very common use cases although we have to be careful with the naming to
> avoid giving the wrong impression. Also, when it comes to `Map.Entry`,
> I
> think I'd prefer a `toMap` method that simply gives the user a Map if
> that's what they want (it makes it clear that there's a conversion
> happening).
>
> Regarding 3, the concern I have if we make the headers mutable is that
> it
> seems to introduce some inconsistencies and potential edge cases. At
> the
> moment, it's safe to keep a ProducerRecord and resend it, for example.
> If
> the record is mutated by an interceptor, then this can lead to weird
> behaviour. Also, it seems inconsistent that one has to create a new
> ProducerRecord to modify the record timestamp, but that one has to
> mutate
> the record to add headers. It seems like one should either embrace
> immutability or mutability, but mixing both is not ideal.
>
> Regarding 4, for the cases where there are a small number of headers
> and/or
> 1 or 2 interceptors, it doesn't seem difficult to come up with
> reasonable
> implementations for mutable and immutable cases that do a good enough
> job.
> However, if the number of headers and interceptors is large, then care
> is
> needed for the immutable case to avoid unnecessary copying. A simple
> option
> that adds little memory overhead if we have Header instances is to
> simply
> add a self-reference to the previous Header in the linked list.
>
> Ismael
>
> On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > Im happy to compromise to keep it mutable but move to an append
> style api.
> > (as in guava interables concat)
> >
> > class Headers {
> > Headers append(Iterable<Header> headers);
> > }
> >
> >
> > I don’t think we’d want prepend, this would give the idea of
> guaranteed
> > ordering, when in actual fact we don’t provide that guarantee (.e.g
> one
> > client can put headerA, then headerB, but another could put headerB
> then
> > headerA, this shouldn’t cause issues), Also what if we changed to a
> hashmap
> > for the internal implementation, its just a bucket of entries no
> ordering.
> > I think we just need to provide an api to add/append headers.
> >
> > This ok? If so ill update KIP to record this.
> >
> > Cheers
> > Mike
> >
> > On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > The point about usability is fair. It's also reasonable to
> expect that
> > common use cases such as appending headers should be done
> efficiently.
> >
> > Perhaps we could compromise with something like this?
> >
> > class Headers {
> > Headers append(Iterable<Header> headers);
> > Headers prepend(Iterable<Header> headers);
> > }
> >
> > That retains ease of use while still giving ourselves some
> flexibility
> > in
> > the implementation.
> >
> > -Jason
> >
> >
> > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > I wasn’t referring to the headers needing to be copied, im
> meaning
> > the
> > > fact we’d be forcing a new producer record to be created, with
> all
> > the
> > > contents copied.
> > >
> > > i.e what will happen is utility method will be created or end
> up
> > being
> > > used, which does this, and returns the new ProducerRecord
> instance.
> > >
> > > ProducerRecord addHeader(ProducerRecord record, Header
> header){
> > > Return New ProducerRecord(record.key, record.value,
> > record.timestamp…..,
> > > record.headers.concat(header))
> > > }
> > >
> > > To me this seems ugly, but will be inevitable if we don’t make
> adding
> > > headers to existing records a simple clean method call.
> > >
> > >
> > >
> > > On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > Lazy init can achieve/avoid that.
> > >
> > > Re the concat, why don’t we implement that inside the
> Headers
> > rather
> > > than causing everyone to implement this as adding headers in
> > interceptors
> > > will be a dominant use case. We want a user friendly API.
> Having as
> > a user
> > > having to code this instead of having the headers handle this
> for me
> > seems
> > > redundant.
> > >
> > > On 22/02/2017, 22:34, "Jason Gustafson" <
> jason@confluent.io>
> > wrote:
> > >
> > > I thought the argument was against creating the extra
> objects
> > > unnecessarily
> > > (i.e. if they were not accessed). And note that making
> the
> > Headers
> > > immutable doesn't necessarily mean that they need to be
> > copied:
> > > you can do
> > > a trick like Guava's Iterables.concat to add additional
> > headers
> > > without
> > > changing the underlying collections.
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > If the argument for not having a map holding the
> key, value
> > > pairs is due
> > > > to garbage creation of HashMap entry's, forcing the
> > creation of
> > > a whole new
> > > > producer record to simply add a head, surely is
> creating
> > a-lot
> > > more?
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > The current producer interceptor API is this:
> > > >
> > > > ProducerRecord<K, V> onSend(ProducerRecord<K, V>
> record);
> > > >
> > > > So adding a header means creating a new
> ProducerRecord
> > with a
> > > new header
> > > > added to the current headers and returning it. Would
> that
> > not
> > > work?
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > So how would you have this work if not mutable
> where
> > > interceptors would
> > > > > add headers?
> > > > >
> > > > > Sent using OWA for iPhone
> > > > > ________________________________________
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > > >
> > > > > I think the point on the mutability of Headers is
> worth
> > > discussing a
> > > > little
> > > > > more. As far as I can tell, once the
> ProducerRecord (or
> > > ConsumerRecord)
> > > > is
> > > > > constructed, there should be no need to further
> change
> > the
> > > headers. Is
> > > > that
> > > > > correct? If so, then why not enforce that that is
> the
> > case
> > > through the
> > > > API?
> > > > > One problem with mutability it that it constrains
> the
> > > implementation of
> > > > > Headers. For example, if we were backing with a
> byte
> > slice,
> > > would we
> > > > recopy
> > > > > the bytes if a header is added or would we
> maintain a
> > satellite
> > > > collection
> > > > > of added records. Seems not great either way. If we
> > really
> > > think
> > > > mutability
> > > > > is needed, perhaps we could add a method to
> headers to
> > convert
> > > it to a
> > > > > mutable type (e.g. a Headers.toMap)?
> > > > >
> > > > > I'm also with Ismael about exposing Headers.get().
> I
> > thought
> > > it might
> > > > make
> > > > > sense to have a method like this instead:
> > > > >
> > > > > Iterable<Header> findMatching(Pattern pattern);
> > > > >
> > > > > This makes the (potential) need to scan the headers
> > clear in
> > > the API. I'd
> > > > > also be fine exposing no getter at all. In
> general, Ï
> > think
> > > it's good to
> > > > > start with a minimalistic API and work from there
> since
> > it's
> > > always
> > > > easier
> > > > > to add methods than remove them.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Hi Ismael
> > > > > >
> > > > > > On point 1,
> > > > > >
> > > > > > Sure makes sense will update shortly.
> > > > > >
> > > > > > On point 2,
> > > > > >
> > > > > > Setter/getter typical to properties/headers api’s
> > > traditionally are map
> > > > > > styled interfaces and what I believe is most
> expected
> > styled
> > > thus the
> > > > > Key,
> > > > > > Value setter.
> > > > > > Also it would mean rather than an interface, we
> would
> > be
> > > making our
> > > > > > internal header impl object we have for the
> array,
> > exposed.
> > > E.g. if we
> > > > > had
> > > > > > a Map really this would be Map.Entry interface,
> this
> > is the
> > > same
> > > > reasons
> > > > > on
> > > > > > the map interface I cannot actually make the
> > underlying Node
> > > object
> > > > > that’s
> > > > > > the implementation for HashMap, so that
> internals can
> > be
> > > changed.
> > > > > >
> > > > > >
> > > > > > On point 3,
> > > > > >
> > > > > > I think it people do expect it to be performant,
> thus
> > why
> > > originally
> > > > > > concern I raised with using an array for to me
> is an
> > early
> > > memory
> > > > > > optimisation. I think the user experience of
> > > properties/headers is on a
> > > > > > get/set model. This is why its important we have
> > > encapsulated logic
> > > > that
> > > > > > then allows us to change this to a map, if this
> > becomes and
> > > issue, and
> > > > > the
> > > > > > memory overhead of hashmap is less so.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on
> behalf of
> > > Ismael Juma" <
> > > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk
> >
> > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Great to see the progress that has been
> achieved
> > on this
> > > one. :) A
> > > > > few
> > > > > > comments regarding the APIs (I'm still
> reviewing
> > the
> > > message format
> > > > > > changes):
> > > > > >
> > > > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> > > `ConsumerRecord`
> > > > should
> > > > > be
> > > > > > named `headers` (we avoid the `get` prefix in
> > Kafka)
> > > > > >
> > > > > > 2. The `Headers` class is mutable (there's
> an `add`
> > > method). Does
> > > > it
> > > > > > need
> > > > > > to be? If so, it would be good to explain
> why.
> > Related
> > > to that, we
> > > > > > should
> > > > > > also explain the thinking around
> thread-safety. If
> > we
> > > keep the
> > > > `add`
> > > > > > method, it may make sense for it to take a
> `Header`
> > > (that way we
> > > > can
> > > > > > add
> > > > > > things to `Header` without changing the
> interface).
> > > > > >
> > > > > > 3. Do we need the `Headers.get()` method?
> People
> > usually
> > > assume
> > > > that
> > > > > > `get`
> > > > > > would be efficient, but depending on the
> > implementation
> > > (the
> > > > current
> > > > > > proposal states that an array would be
> used), it
> > may not
> > > be. If we
> > > > > > expect
> > > > > > the number of headers to be small, it doesn't
> > matter
> > > though.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael
> Pearce <
> > > > > Michael.Pearce@ig.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Have converted the interface/api bullets
> into
> > > interface code
> > > > > > snippets.
> > > > > > >
> > > > > > > Agreed implementation won’t take too long.
> We
> > have
> > > early versions
> > > > > > already.
> > > > > > > Maybe a week before you think about
> merging I
> > would
> > > assume it
> > > > would
> > > > > > be more
> > > > > > > stabilised? I was thinking then we could
> fork
> > from
> > > your confluent
> > > > > > branch,
> > > > > > > making and then holding KIP-82 changes in
> a patch
> > > file, that we
> > > > can
> > > > > > then
> > > > > > > re-fork from apache once KIP98 final is
> merged,
> > and
> > > apply patch
> > > > > with
> > > > > > last
> > > > > > > minute changes.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Mike
> > > > > > >
> > > > > > >
> > > > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > Hey Michael,
> > > > > > >
> > > > > > > Awesome. I have a minor request. The
> APIs are
> > > currently
> > > > > > documented as a
> > > > > > > wiki list. Would you mind adding a code
> > snippet
> > > instead?
> > > > It's a
> > > > > > bit
> > > > > > > easier
> > > > > > > to process.
> > > > > > >
> > > > > > > How will be best to manage this, as we
> will
> > > obviously build
> > > > off
> > > > > > your
> > > > > > > KIP’s
> > > > > > > > protocol changes, to avoid a merge
> hell,
> > should
> > > we branch
> > > > > from
> > > > > > your
> > > > > > > branch
> > > > > > > > in the confluent repo or is it worth
> > having a
> > > KIP-98
> > > > special
> > > > > > branch
> > > > > > > in the
> > > > > > > > apache git, that we can branch/fork
> from?
> > > > > > >
> > > > > > >
> > > > > > > I was thinking about this also.
> Ideally we'd
> > like
> > > to get the
> > > > > > changes
> > > > > > > in as
> > > > > > > close together as possible since we
> only
> > want one
> > > magic bump
> > > > > and
> > > > > > some
> > > > > > > users
> > > > > > > deploy trunk. The level of effort to
> change
> > the
> > > format for
> > > > > > headers
> > > > > > > seems
> > > > > > > not too high. Do you agree? My guess
> is that
> > the
> > > KIP-98
> > > > message
> > > > > > format
> > > > > > > patch will take 2-3 weeks to review
> before we
> > > merge to trunk,
> > > > > so
> > > > > > you
> > > > > > > could
> > > > > > > hold off implementing until that patch
> has
> > somewhat
> > > > stabilized.
> > > > > > That
> > > > > > > would
> > > > > > > save some potential rebase pain.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > strictly
> > > confidential
> > > > > and
> > > > > > for
> > > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If you
> > > > > > are not
> > > > > > > the intended recipient, please do not read,
> > copy, use
> > > or disclose
> > > > > to
> > > > > > others
> > > > > > > this message or any attachment. Please also
> > notify the
> > > sender by
> > > > > > replying
> > > > > > > to this email or by telephone (+44(020
> 7896 0011)
> > and
> > > then delete
> > > > > the
> > > > > > > email and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > > relate
> > > > > > > to the official business of this company
> shall be
> > > understood as
> > > > > > neither
> > > > > > > given nor endorsed by it. IG is a trading
> name
> > of IG
> > > Markets
> > > > > Limited
> > > > > > (a
> > > > > > > company registered in England and Wales,
> company
> > > number 04008957)
> > > > > > and IG
> > > > > > > Index Limited (a company registered in
> England
> > and
> > > Wales, company
> > > > > > number
> > > > > > > 01190902). Registered address at Cannon
> Bridge
> > House,
> > > 25 Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number 195355)
> > > > > > and IG
> > > > > > > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated
> > > > > > by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > > confidential and
> > > > for
> > > > > > the use of the addressee only, unless otherwise
> > indicated.
> > > If you are
> > > > not
> > > > > > the intended recipient, please do not read,
> copy, use
> > or
> > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify the
> > > sender by
> > > > replying
> > > > > > to this email or by telephone (+44(020 7896
> 0011) and
> > then
> > > delete the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion (etc)
> that
> > do not
> > > relate to
> > > > > the
> > > > > > official business of this company shall be
> understood
> > as
> > > neither given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited
> > > (a company
> > > > > > registered in England and Wales, company number
> > 04008957)
> > > and IG Index
> > > > > > Limited (a company registered in England and
> Wales,
> > company
> > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > 195355) and
> > > > IG
> > > > > > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is strictly
> > > confidential and for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you are not
> > > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify
> the
> > sender
> > > by replying
> > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then
> > > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to
> > > > the
> > > > > official business of this company shall be
> understood as
> > > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG Index
> > > > > Limited (a company registered in England and Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355) and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential
> > > and for
> > > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > you are not
> > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to others
> > > > this message or any attachment. Please also notify
> the
> > sender by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011)
> and then
> > > delete the email
> > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to the
> > > > official business of this company shall be
> understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > company
> > > > registered in England and Wales, company number
> 04008957)
> > and IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > > number
> > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355) and IG
> > > > Index Limited (register number 114059) are
> authorised and
> > > regulated by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Pattern.compile is expensive, and even if cached String.equals is faster than matched. also if we end up with an internal map in future for performance it will be easier to be by key.
As all that's needed is to get header by key.
With like the other arguements of let's implement simple and then we can always add pattern later as well if it's found it's needed. (As noted it's easier to add methods than to take away)
Great I'll update kip with extra methods on producerecord and a note that new objects are returned by method calls.
Sent using OWA for iPhone
________________________________________
From: Jason Gustafson <ja...@confluent.io>
Sent: Friday, February 24, 2017 6:51:45 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
The APIs in the current KIP look good to me. Just a couple questions: why
does filter not return Headers? Also would it be useful if the key is a
regex?
On the point of immutability.. One option might be to use a mutable object
only when passing the headers through the interceptor chain. I think as
long as we resort to mutability only when clear performance results show
that it is worthwhile, I am satisfied. As Ismael noted, for common
scenarios it is possible to get reasonable performance with immutable
objects.
-Jason
On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi
>
> On 1, How can you guarantee two separate implemented clients would add
> the headers in the same order we are not specifying an order at the
> protocol level (nor should we) with regards to keyA being ordered before
> keyB? We shouldn’t be expecting keyA to be always set before keyB.
>
> On 2, I believe we have changed the naming based on feedback from Jason
> already, e.g. we don’t have “get” method that inferred O(1) performance,
> like wise nor “put” but we have an “append”
>
> On 3, in the KafkaProducer, I think we have mutability already, the value
> for time is changed if it is null, at the point of send:
> “
> long timestamp = record.timestamp() == null ?
> time.milliseconds() : record.timestamp();
> “
>
> As such the timestamp is already mutable, so what’s the difference here,
> we already have some mixed semantics. On timestamp.
> e.g. currently if I send to records with timestamp not set, the wire
> binary sent the value for the timestamp would be different, as such we have
> mutation for the same record.
>
> On 4, I think we should not expect not 1 or 2 headers, but infact 10’s of
> headers. This is the concern on immutable headers, whilst the append
> self-reference works nicely, what if someone needs to remove a header?
>
> Trying to get this moving:
>
> If we really wanted Immutable Headers and essentially you guys wont give
> +1 for it without.
>
> Whats the feeling for adding methods to ProducerRecord that does the
> boiler plate code or creating a new ProducerRecord with the altered new
> headers (appended or removed) inside. E.g.
>
> ProducerRecord {
>
>
> ProducerRecord append(Iterable<Headers> headersToAppend){
> return new ProducerRecord(key, value, headers.append(headersToAppend),
> ….)
> }
>
> ProducerRecord remove(Iterable<Headers> headersToAppend){
> return new ProducerRecord(key, value, headers.remove(headersToAppend),
> ….)
> }
>
> }
>
> Were the headers methods actually returns new objects, and the producer
> records methods create a new producer record with all the current values,
> but with the new modified headers.
>
> Then interceptors / code return this new object?
>
>
> Cheers
> Mike
>
>
>
>
>
>
> On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael Juma" <
> ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>
> Hi Michael,
>
> Did you mean that you were happy to compromise to keep it mutable or
> immutable? You wrote the former, but it sounded from the sentence that
> it
> could have been a typo. So, my thoughts on this is that there are a few
> things to take into account:
>
> 1. Semantics
> 2. Simplicity of use (the common operations should be easy to do)
> 3. If it's easy to reason about and safe (immutability helps with this)
> 4. Efficiency (both memory and CPU usage)
>
> Regarding 1, I think it would be good to be very clear about the
> guarantees
> that we are providing. It seems that we are saying that keys are
> unordered,
> but what about the case where there are multiple values for the same
> key?
> It seems that for some use cases (e.g. lineage), it may be useful to
> add
> values to the same key while preserving the order.
>
> Regarding 2, I agree that it's useful to have methods in `Headers` for
> the
> very common use cases although we have to be careful with the naming to
> avoid giving the wrong impression. Also, when it comes to `Map.Entry`,
> I
> think I'd prefer a `toMap` method that simply gives the user a Map if
> that's what they want (it makes it clear that there's a conversion
> happening).
>
> Regarding 3, the concern I have if we make the headers mutable is that
> it
> seems to introduce some inconsistencies and potential edge cases. At
> the
> moment, it's safe to keep a ProducerRecord and resend it, for example.
> If
> the record is mutated by an interceptor, then this can lead to weird
> behaviour. Also, it seems inconsistent that one has to create a new
> ProducerRecord to modify the record timestamp, but that one has to
> mutate
> the record to add headers. It seems like one should either embrace
> immutability or mutability, but mixing both is not ideal.
>
> Regarding 4, for the cases where there are a small number of headers
> and/or
> 1 or 2 interceptors, it doesn't seem difficult to come up with
> reasonable
> implementations for mutable and immutable cases that do a good enough
> job.
> However, if the number of headers and interceptors is large, then care
> is
> needed for the immutable case to avoid unnecessary copying. A simple
> option
> that adds little memory overhead if we have Header instances is to
> simply
> add a self-reference to the previous Header in the linked list.
>
> Ismael
>
> On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > Im happy to compromise to keep it mutable but move to an append
> style api.
> > (as in guava interables concat)
> >
> > class Headers {
> > Headers append(Iterable<Header> headers);
> > }
> >
> >
> > I don’t think we’d want prepend, this would give the idea of
> guaranteed
> > ordering, when in actual fact we don’t provide that guarantee (.e.g
> one
> > client can put headerA, then headerB, but another could put headerB
> then
> > headerA, this shouldn’t cause issues), Also what if we changed to a
> hashmap
> > for the internal implementation, its just a bucket of entries no
> ordering.
> > I think we just need to provide an api to add/append headers.
> >
> > This ok? If so ill update KIP to record this.
> >
> > Cheers
> > Mike
> >
> > On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > The point about usability is fair. It's also reasonable to
> expect that
> > common use cases such as appending headers should be done
> efficiently.
> >
> > Perhaps we could compromise with something like this?
> >
> > class Headers {
> > Headers append(Iterable<Header> headers);
> > Headers prepend(Iterable<Header> headers);
> > }
> >
> > That retains ease of use while still giving ourselves some
> flexibility
> > in
> > the implementation.
> >
> > -Jason
> >
> >
> > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > I wasn’t referring to the headers needing to be copied, im
> meaning
> > the
> > > fact we’d be forcing a new producer record to be created, with
> all
> > the
> > > contents copied.
> > >
> > > i.e what will happen is utility method will be created or end
> up
> > being
> > > used, which does this, and returns the new ProducerRecord
> instance.
> > >
> > > ProducerRecord addHeader(ProducerRecord record, Header
> header){
> > > Return New ProducerRecord(record.key, record.value,
> > record.timestamp…..,
> > > record.headers.concat(header))
> > > }
> > >
> > > To me this seems ugly, but will be inevitable if we don’t make
> adding
> > > headers to existing records a simple clean method call.
> > >
> > >
> > >
> > > On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > Lazy init can achieve/avoid that.
> > >
> > > Re the concat, why don’t we implement that inside the
> Headers
> > rather
> > > than causing everyone to implement this as adding headers in
> > interceptors
> > > will be a dominant use case. We want a user friendly API.
> Having as
> > a user
> > > having to code this instead of having the headers handle this
> for me
> > seems
> > > redundant.
> > >
> > > On 22/02/2017, 22:34, "Jason Gustafson" <
> jason@confluent.io>
> > wrote:
> > >
> > > I thought the argument was against creating the extra
> objects
> > > unnecessarily
> > > (i.e. if they were not accessed). And note that making
> the
> > Headers
> > > immutable doesn't necessarily mean that they need to be
> > copied:
> > > you can do
> > > a trick like Guava's Iterables.concat to add additional
> > headers
> > > without
> > > changing the underlying collections.
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > If the argument for not having a map holding the
> key, value
> > > pairs is due
> > > > to garbage creation of HashMap entry's, forcing the
> > creation of
> > > a whole new
> > > > producer record to simply add a head, surely is
> creating
> > a-lot
> > > more?
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > The current producer interceptor API is this:
> > > >
> > > > ProducerRecord<K, V> onSend(ProducerRecord<K, V>
> record);
> > > >
> > > > So adding a header means creating a new
> ProducerRecord
> > with a
> > > new header
> > > > added to the current headers and returning it. Would
> that
> > not
> > > work?
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > So how would you have this work if not mutable
> where
> > > interceptors would
> > > > > add headers?
> > > > >
> > > > > Sent using OWA for iPhone
> > > > > ________________________________________
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > > >
> > > > > I think the point on the mutability of Headers is
> worth
> > > discussing a
> > > > little
> > > > > more. As far as I can tell, once the
> ProducerRecord (or
> > > ConsumerRecord)
> > > > is
> > > > > constructed, there should be no need to further
> change
> > the
> > > headers. Is
> > > > that
> > > > > correct? If so, then why not enforce that that is
> the
> > case
> > > through the
> > > > API?
> > > > > One problem with mutability it that it constrains
> the
> > > implementation of
> > > > > Headers. For example, if we were backing with a
> byte
> > slice,
> > > would we
> > > > recopy
> > > > > the bytes if a header is added or would we
> maintain a
> > satellite
> > > > collection
> > > > > of added records. Seems not great either way. If we
> > really
> > > think
> > > > mutability
> > > > > is needed, perhaps we could add a method to
> headers to
> > convert
> > > it to a
> > > > > mutable type (e.g. a Headers.toMap)?
> > > > >
> > > > > I'm also with Ismael about exposing Headers.get().
> I
> > thought
> > > it might
> > > > make
> > > > > sense to have a method like this instead:
> > > > >
> > > > > Iterable<Header> findMatching(Pattern pattern);
> > > > >
> > > > > This makes the (potential) need to scan the headers
> > clear in
> > > the API. I'd
> > > > > also be fine exposing no getter at all. In
> general, Ï
> > think
> > > it's good to
> > > > > start with a minimalistic API and work from there
> since
> > it's
> > > always
> > > > easier
> > > > > to add methods than remove them.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Hi Ismael
> > > > > >
> > > > > > On point 1,
> > > > > >
> > > > > > Sure makes sense will update shortly.
> > > > > >
> > > > > > On point 2,
> > > > > >
> > > > > > Setter/getter typical to properties/headers api’s
> > > traditionally are map
> > > > > > styled interfaces and what I believe is most
> expected
> > styled
> > > thus the
> > > > > Key,
> > > > > > Value setter.
> > > > > > Also it would mean rather than an interface, we
> would
> > be
> > > making our
> > > > > > internal header impl object we have for the
> array,
> > exposed.
> > > E.g. if we
> > > > > had
> > > > > > a Map really this would be Map.Entry interface,
> this
> > is the
> > > same
> > > > reasons
> > > > > on
> > > > > > the map interface I cannot actually make the
> > underlying Node
> > > object
> > > > > that’s
> > > > > > the implementation for HashMap, so that
> internals can
> > be
> > > changed.
> > > > > >
> > > > > >
> > > > > > On point 3,
> > > > > >
> > > > > > I think it people do expect it to be performant,
> thus
> > why
> > > originally
> > > > > > concern I raised with using an array for to me
> is an
> > early
> > > memory
> > > > > > optimisation. I think the user experience of
> > > properties/headers is on a
> > > > > > get/set model. This is why its important we have
> > > encapsulated logic
> > > > that
> > > > > > then allows us to change this to a map, if this
> > becomes and
> > > issue, and
> > > > > the
> > > > > > memory overhead of hashmap is less so.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on
> behalf of
> > > Ismael Juma" <
> > > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk
> >
> > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Great to see the progress that has been
> achieved
> > on this
> > > one. :) A
> > > > > few
> > > > > > comments regarding the APIs (I'm still
> reviewing
> > the
> > > message format
> > > > > > changes):
> > > > > >
> > > > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> > > `ConsumerRecord`
> > > > should
> > > > > be
> > > > > > named `headers` (we avoid the `get` prefix in
> > Kafka)
> > > > > >
> > > > > > 2. The `Headers` class is mutable (there's
> an `add`
> > > method). Does
> > > > it
> > > > > > need
> > > > > > to be? If so, it would be good to explain
> why.
> > Related
> > > to that, we
> > > > > > should
> > > > > > also explain the thinking around
> thread-safety. If
> > we
> > > keep the
> > > > `add`
> > > > > > method, it may make sense for it to take a
> `Header`
> > > (that way we
> > > > can
> > > > > > add
> > > > > > things to `Header` without changing the
> interface).
> > > > > >
> > > > > > 3. Do we need the `Headers.get()` method?
> People
> > usually
> > > assume
> > > > that
> > > > > > `get`
> > > > > > would be efficient, but depending on the
> > implementation
> > > (the
> > > > current
> > > > > > proposal states that an array would be
> used), it
> > may not
> > > be. If we
> > > > > > expect
> > > > > > the number of headers to be small, it doesn't
> > matter
> > > though.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael
> Pearce <
> > > > > Michael.Pearce@ig.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Have converted the interface/api bullets
> into
> > > interface code
> > > > > > snippets.
> > > > > > >
> > > > > > > Agreed implementation won’t take too long.
> We
> > have
> > > early versions
> > > > > > already.
> > > > > > > Maybe a week before you think about
> merging I
> > would
> > > assume it
> > > > would
> > > > > > be more
> > > > > > > stabilised? I was thinking then we could
> fork
> > from
> > > your confluent
> > > > > > branch,
> > > > > > > making and then holding KIP-82 changes in
> a patch
> > > file, that we
> > > > can
> > > > > > then
> > > > > > > re-fork from apache once KIP98 final is
> merged,
> > and
> > > apply patch
> > > > > with
> > > > > > last
> > > > > > > minute changes.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Mike
> > > > > > >
> > > > > > >
> > > > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > Hey Michael,
> > > > > > >
> > > > > > > Awesome. I have a minor request. The
> APIs are
> > > currently
> > > > > > documented as a
> > > > > > > wiki list. Would you mind adding a code
> > snippet
> > > instead?
> > > > It's a
> > > > > > bit
> > > > > > > easier
> > > > > > > to process.
> > > > > > >
> > > > > > > How will be best to manage this, as we
> will
> > > obviously build
> > > > off
> > > > > > your
> > > > > > > KIP’s
> > > > > > > > protocol changes, to avoid a merge
> hell,
> > should
> > > we branch
> > > > > from
> > > > > > your
> > > > > > > branch
> > > > > > > > in the confluent repo or is it worth
> > having a
> > > KIP-98
> > > > special
> > > > > > branch
> > > > > > > in the
> > > > > > > > apache git, that we can branch/fork
> from?
> > > > > > >
> > > > > > >
> > > > > > > I was thinking about this also.
> Ideally we'd
> > like
> > > to get the
> > > > > > changes
> > > > > > > in as
> > > > > > > close together as possible since we
> only
> > want one
> > > magic bump
> > > > > and
> > > > > > some
> > > > > > > users
> > > > > > > deploy trunk. The level of effort to
> change
> > the
> > > format for
> > > > > > headers
> > > > > > > seems
> > > > > > > not too high. Do you agree? My guess
> is that
> > the
> > > KIP-98
> > > > message
> > > > > > format
> > > > > > > patch will take 2-3 weeks to review
> before we
> > > merge to trunk,
> > > > > so
> > > > > > you
> > > > > > > could
> > > > > > > hold off implementing until that patch
> has
> > somewhat
> > > > stabilized.
> > > > > > That
> > > > > > > would
> > > > > > > save some potential rebase pain.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > strictly
> > > confidential
> > > > > and
> > > > > > for
> > > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If you
> > > > > > are not
> > > > > > > the intended recipient, please do not read,
> > copy, use
> > > or disclose
> > > > > to
> > > > > > others
> > > > > > > this message or any attachment. Please also
> > notify the
> > > sender by
> > > > > > replying
> > > > > > > to this email or by telephone (+44(020
> 7896 0011)
> > and
> > > then delete
> > > > > the
> > > > > > > email and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > > relate
> > > > > > > to the official business of this company
> shall be
> > > understood as
> > > > > > neither
> > > > > > > given nor endorsed by it. IG is a trading
> name
> > of IG
> > > Markets
> > > > > Limited
> > > > > > (a
> > > > > > > company registered in England and Wales,
> company
> > > number 04008957)
> > > > > > and IG
> > > > > > > Index Limited (a company registered in
> England
> > and
> > > Wales, company
> > > > > > number
> > > > > > > 01190902). Registered address at Cannon
> Bridge
> > House,
> > > 25 Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number 195355)
> > > > > > and IG
> > > > > > > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated
> > > > > > by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > > confidential and
> > > > for
> > > > > > the use of the addressee only, unless otherwise
> > indicated.
> > > If you are
> > > > not
> > > > > > the intended recipient, please do not read,
> copy, use
> > or
> > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify the
> > > sender by
> > > > replying
> > > > > > to this email or by telephone (+44(020 7896
> 0011) and
> > then
> > > delete the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion (etc)
> that
> > do not
> > > relate to
> > > > > the
> > > > > > official business of this company shall be
> understood
> > as
> > > neither given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited
> > > (a company
> > > > > > registered in England and Wales, company number
> > 04008957)
> > > and IG Index
> > > > > > Limited (a company registered in England and
> Wales,
> > company
> > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > 195355) and
> > > > IG
> > > > > > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is strictly
> > > confidential and for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you are not
> > > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify
> the
> > sender
> > > by replying
> > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then
> > > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to
> > > > the
> > > > > official business of this company shall be
> understood as
> > > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG Index
> > > > > Limited (a company registered in England and Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355) and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential
> > > and for
> > > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > you are not
> > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to others
> > > > this message or any attachment. Please also notify
> the
> > sender by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011)
> and then
> > > delete the email
> > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to the
> > > > official business of this company shall be
> understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > company
> > > > registered in England and Wales, company number
> 04008957)
> > and IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > > number
> > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355) and IG
> > > > Index Limited (register number 114059) are
> authorised and
> > > regulated by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
The APIs in the current KIP look good to me. Just a couple questions: why
does filter not return Headers? Also would it be useful if the key is a
regex?
On the point of immutability.. One option might be to use a mutable object
only when passing the headers through the interceptor chain. I think as
long as we resort to mutability only when clear performance results show
that it is worthwhile, I am satisfied. As Ismael noted, for common
scenarios it is possible to get reasonable performance with immutable
objects.
-Jason
On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi
>
> On 1, How can you guarantee two separate implemented clients would add
> the headers in the same order we are not specifying an order at the
> protocol level (nor should we) with regards to keyA being ordered before
> keyB? We shouldn’t be expecting keyA to be always set before keyB.
>
> On 2, I believe we have changed the naming based on feedback from Jason
> already, e.g. we don’t have “get” method that inferred O(1) performance,
> like wise nor “put” but we have an “append”
>
> On 3, in the KafkaProducer, I think we have mutability already, the value
> for time is changed if it is null, at the point of send:
> “
> long timestamp = record.timestamp() == null ?
> time.milliseconds() : record.timestamp();
> “
>
> As such the timestamp is already mutable, so what’s the difference here,
> we already have some mixed semantics. On timestamp.
> e.g. currently if I send to records with timestamp not set, the wire
> binary sent the value for the timestamp would be different, as such we have
> mutation for the same record.
>
> On 4, I think we should not expect not 1 or 2 headers, but infact 10’s of
> headers. This is the concern on immutable headers, whilst the append
> self-reference works nicely, what if someone needs to remove a header?
>
> Trying to get this moving:
>
> If we really wanted Immutable Headers and essentially you guys wont give
> +1 for it without.
>
> Whats the feeling for adding methods to ProducerRecord that does the
> boiler plate code or creating a new ProducerRecord with the altered new
> headers (appended or removed) inside. E.g.
>
> ProducerRecord {
>
>
> ProducerRecord append(Iterable<Headers> headersToAppend){
> return new ProducerRecord(key, value, headers.append(headersToAppend),
> ….)
> }
>
> ProducerRecord remove(Iterable<Headers> headersToAppend){
> return new ProducerRecord(key, value, headers.remove(headersToAppend),
> ….)
> }
>
> }
>
> Were the headers methods actually returns new objects, and the producer
> records methods create a new producer record with all the current values,
> but with the new modified headers.
>
> Then interceptors / code return this new object?
>
>
> Cheers
> Mike
>
>
>
>
>
>
> On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael Juma" <
> ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>
> Hi Michael,
>
> Did you mean that you were happy to compromise to keep it mutable or
> immutable? You wrote the former, but it sounded from the sentence that
> it
> could have been a typo. So, my thoughts on this is that there are a few
> things to take into account:
>
> 1. Semantics
> 2. Simplicity of use (the common operations should be easy to do)
> 3. If it's easy to reason about and safe (immutability helps with this)
> 4. Efficiency (both memory and CPU usage)
>
> Regarding 1, I think it would be good to be very clear about the
> guarantees
> that we are providing. It seems that we are saying that keys are
> unordered,
> but what about the case where there are multiple values for the same
> key?
> It seems that for some use cases (e.g. lineage), it may be useful to
> add
> values to the same key while preserving the order.
>
> Regarding 2, I agree that it's useful to have methods in `Headers` for
> the
> very common use cases although we have to be careful with the naming to
> avoid giving the wrong impression. Also, when it comes to `Map.Entry`,
> I
> think I'd prefer a `toMap` method that simply gives the user a Map if
> that's what they want (it makes it clear that there's a conversion
> happening).
>
> Regarding 3, the concern I have if we make the headers mutable is that
> it
> seems to introduce some inconsistencies and potential edge cases. At
> the
> moment, it's safe to keep a ProducerRecord and resend it, for example.
> If
> the record is mutated by an interceptor, then this can lead to weird
> behaviour. Also, it seems inconsistent that one has to create a new
> ProducerRecord to modify the record timestamp, but that one has to
> mutate
> the record to add headers. It seems like one should either embrace
> immutability or mutability, but mixing both is not ideal.
>
> Regarding 4, for the cases where there are a small number of headers
> and/or
> 1 or 2 interceptors, it doesn't seem difficult to come up with
> reasonable
> implementations for mutable and immutable cases that do a good enough
> job.
> However, if the number of headers and interceptors is large, then care
> is
> needed for the immutable case to avoid unnecessary copying. A simple
> option
> that adds little memory overhead if we have Header instances is to
> simply
> add a self-reference to the previous Header in the linked list.
>
> Ismael
>
> On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > Im happy to compromise to keep it mutable but move to an append
> style api.
> > (as in guava interables concat)
> >
> > class Headers {
> > Headers append(Iterable<Header> headers);
> > }
> >
> >
> > I don’t think we’d want prepend, this would give the idea of
> guaranteed
> > ordering, when in actual fact we don’t provide that guarantee (.e.g
> one
> > client can put headerA, then headerB, but another could put headerB
> then
> > headerA, this shouldn’t cause issues), Also what if we changed to a
> hashmap
> > for the internal implementation, its just a bucket of entries no
> ordering.
> > I think we just need to provide an api to add/append headers.
> >
> > This ok? If so ill update KIP to record this.
> >
> > Cheers
> > Mike
> >
> > On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > The point about usability is fair. It's also reasonable to
> expect that
> > common use cases such as appending headers should be done
> efficiently.
> >
> > Perhaps we could compromise with something like this?
> >
> > class Headers {
> > Headers append(Iterable<Header> headers);
> > Headers prepend(Iterable<Header> headers);
> > }
> >
> > That retains ease of use while still giving ourselves some
> flexibility
> > in
> > the implementation.
> >
> > -Jason
> >
> >
> > On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > I wasn’t referring to the headers needing to be copied, im
> meaning
> > the
> > > fact we’d be forcing a new producer record to be created, with
> all
> > the
> > > contents copied.
> > >
> > > i.e what will happen is utility method will be created or end
> up
> > being
> > > used, which does this, and returns the new ProducerRecord
> instance.
> > >
> > > ProducerRecord addHeader(ProducerRecord record, Header
> header){
> > > Return New ProducerRecord(record.key, record.value,
> > record.timestamp…..,
> > > record.headers.concat(header))
> > > }
> > >
> > > To me this seems ugly, but will be inevitable if we don’t make
> adding
> > > headers to existing records a simple clean method call.
> > >
> > >
> > >
> > > On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > Lazy init can achieve/avoid that.
> > >
> > > Re the concat, why don’t we implement that inside the
> Headers
> > rather
> > > than causing everyone to implement this as adding headers in
> > interceptors
> > > will be a dominant use case. We want a user friendly API.
> Having as
> > a user
> > > having to code this instead of having the headers handle this
> for me
> > seems
> > > redundant.
> > >
> > > On 22/02/2017, 22:34, "Jason Gustafson" <
> jason@confluent.io>
> > wrote:
> > >
> > > I thought the argument was against creating the extra
> objects
> > > unnecessarily
> > > (i.e. if they were not accessed). And note that making
> the
> > Headers
> > > immutable doesn't necessarily mean that they need to be
> > copied:
> > > you can do
> > > a trick like Guava's Iterables.concat to add additional
> > headers
> > > without
> > > changing the underlying collections.
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > If the argument for not having a map holding the
> key, value
> > > pairs is due
> > > > to garbage creation of HashMap entry's, forcing the
> > creation of
> > > a whole new
> > > > producer record to simply add a head, surely is
> creating
> > a-lot
> > > more?
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > The current producer interceptor API is this:
> > > >
> > > > ProducerRecord<K, V> onSend(ProducerRecord<K, V>
> record);
> > > >
> > > > So adding a header means creating a new
> ProducerRecord
> > with a
> > > new header
> > > > added to the current headers and returning it. Would
> that
> > not
> > > work?
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > So how would you have this work if not mutable
> where
> > > interceptors would
> > > > > add headers?
> > > > >
> > > > > Sent using OWA for iPhone
> > > > > ________________________________________
> > > > > From: Jason Gustafson <ja...@confluent.io>
> > > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > > >
> > > > > I think the point on the mutability of Headers is
> worth
> > > discussing a
> > > > little
> > > > > more. As far as I can tell, once the
> ProducerRecord (or
> > > ConsumerRecord)
> > > > is
> > > > > constructed, there should be no need to further
> change
> > the
> > > headers. Is
> > > > that
> > > > > correct? If so, then why not enforce that that is
> the
> > case
> > > through the
> > > > API?
> > > > > One problem with mutability it that it constrains
> the
> > > implementation of
> > > > > Headers. For example, if we were backing with a
> byte
> > slice,
> > > would we
> > > > recopy
> > > > > the bytes if a header is added or would we
> maintain a
> > satellite
> > > > collection
> > > > > of added records. Seems not great either way. If we
> > really
> > > think
> > > > mutability
> > > > > is needed, perhaps we could add a method to
> headers to
> > convert
> > > it to a
> > > > > mutable type (e.g. a Headers.toMap)?
> > > > >
> > > > > I'm also with Ismael about exposing Headers.get().
> I
> > thought
> > > it might
> > > > make
> > > > > sense to have a method like this instead:
> > > > >
> > > > > Iterable<Header> findMatching(Pattern pattern);
> > > > >
> > > > > This makes the (potential) need to scan the headers
> > clear in
> > > the API. I'd
> > > > > also be fine exposing no getter at all. In
> general, Ï
> > think
> > > it's good to
> > > > > start with a minimalistic API and work from there
> since
> > it's
> > > always
> > > > easier
> > > > > to add methods than remove them.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > Hi Ismael
> > > > > >
> > > > > > On point 1,
> > > > > >
> > > > > > Sure makes sense will update shortly.
> > > > > >
> > > > > > On point 2,
> > > > > >
> > > > > > Setter/getter typical to properties/headers api’s
> > > traditionally are map
> > > > > > styled interfaces and what I believe is most
> expected
> > styled
> > > thus the
> > > > > Key,
> > > > > > Value setter.
> > > > > > Also it would mean rather than an interface, we
> would
> > be
> > > making our
> > > > > > internal header impl object we have for the
> array,
> > exposed.
> > > E.g. if we
> > > > > had
> > > > > > a Map really this would be Map.Entry interface,
> this
> > is the
> > > same
> > > > reasons
> > > > > on
> > > > > > the map interface I cannot actually make the
> > underlying Node
> > > object
> > > > > that’s
> > > > > > the implementation for HashMap, so that
> internals can
> > be
> > > changed.
> > > > > >
> > > > > >
> > > > > > On point 3,
> > > > > >
> > > > > > I think it people do expect it to be performant,
> thus
> > why
> > > originally
> > > > > > concern I raised with using an array for to me
> is an
> > early
> > > memory
> > > > > > optimisation. I think the user experience of
> > > properties/headers is on a
> > > > > > get/set model. This is why its important we have
> > > encapsulated logic
> > > > that
> > > > > > then allows us to change this to a map, if this
> > becomes and
> > > issue, and
> > > > > the
> > > > > > memory overhead of hashmap is less so.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on
> behalf of
> > > Ismael Juma" <
> > > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk
> >
> > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Great to see the progress that has been
> achieved
> > on this
> > > one. :) A
> > > > > few
> > > > > > comments regarding the APIs (I'm still
> reviewing
> > the
> > > message format
> > > > > > changes):
> > > > > >
> > > > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> > > `ConsumerRecord`
> > > > should
> > > > > be
> > > > > > named `headers` (we avoid the `get` prefix in
> > Kafka)
> > > > > >
> > > > > > 2. The `Headers` class is mutable (there's
> an `add`
> > > method). Does
> > > > it
> > > > > > need
> > > > > > to be? If so, it would be good to explain
> why.
> > Related
> > > to that, we
> > > > > > should
> > > > > > also explain the thinking around
> thread-safety. If
> > we
> > > keep the
> > > > `add`
> > > > > > method, it may make sense for it to take a
> `Header`
> > > (that way we
> > > > can
> > > > > > add
> > > > > > things to `Header` without changing the
> interface).
> > > > > >
> > > > > > 3. Do we need the `Headers.get()` method?
> People
> > usually
> > > assume
> > > > that
> > > > > > `get`
> > > > > > would be efficient, but depending on the
> > implementation
> > > (the
> > > > current
> > > > > > proposal states that an array would be
> used), it
> > may not
> > > be. If we
> > > > > > expect
> > > > > > the number of headers to be small, it doesn't
> > matter
> > > though.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael
> Pearce <
> > > > > Michael.Pearce@ig.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jason,
> > > > > > >
> > > > > > > Have converted the interface/api bullets
> into
> > > interface code
> > > > > > snippets.
> > > > > > >
> > > > > > > Agreed implementation won’t take too long.
> We
> > have
> > > early versions
> > > > > > already.
> > > > > > > Maybe a week before you think about
> merging I
> > would
> > > assume it
> > > > would
> > > > > > be more
> > > > > > > stabilised? I was thinking then we could
> fork
> > from
> > > your confluent
> > > > > > branch,
> > > > > > > making and then holding KIP-82 changes in
> a patch
> > > file, that we
> > > > can
> > > > > > then
> > > > > > > re-fork from apache once KIP98 final is
> merged,
> > and
> > > apply patch
> > > > > with
> > > > > > last
> > > > > > > minute changes.
> > > > > > >
> > > > > > > Cheers
> > > > > > > Mike
> > > > > > >
> > > > > > >
> > > > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > > wrote:
> > > > > > >
> > > > > > > Hey Michael,
> > > > > > >
> > > > > > > Awesome. I have a minor request. The
> APIs are
> > > currently
> > > > > > documented as a
> > > > > > > wiki list. Would you mind adding a code
> > snippet
> > > instead?
> > > > It's a
> > > > > > bit
> > > > > > > easier
> > > > > > > to process.
> > > > > > >
> > > > > > > How will be best to manage this, as we
> will
> > > obviously build
> > > > off
> > > > > > your
> > > > > > > KIP’s
> > > > > > > > protocol changes, to avoid a merge
> hell,
> > should
> > > we branch
> > > > > from
> > > > > > your
> > > > > > > branch
> > > > > > > > in the confluent repo or is it worth
> > having a
> > > KIP-98
> > > > special
> > > > > > branch
> > > > > > > in the
> > > > > > > > apache git, that we can branch/fork
> from?
> > > > > > >
> > > > > > >
> > > > > > > I was thinking about this also.
> Ideally we'd
> > like
> > > to get the
> > > > > > changes
> > > > > > > in as
> > > > > > > close together as possible since we
> only
> > want one
> > > magic bump
> > > > > and
> > > > > > some
> > > > > > > users
> > > > > > > deploy trunk. The level of effort to
> change
> > the
> > > format for
> > > > > > headers
> > > > > > > seems
> > > > > > > not too high. Do you agree? My guess
> is that
> > the
> > > KIP-98
> > > > message
> > > > > > format
> > > > > > > patch will take 2-3 weeks to review
> before we
> > > merge to trunk,
> > > > > so
> > > > > > you
> > > > > > > could
> > > > > > > hold off implementing until that patch
> has
> > somewhat
> > > > stabilized.
> > > > > > That
> > > > > > > would
> > > > > > > save some potential rebase pain.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email is
> > strictly
> > > confidential
> > > > > and
> > > > > > for
> > > > > > > the use of the addressee only, unless
> otherwise
> > > indicated. If you
> > > > > > are not
> > > > > > > the intended recipient, please do not read,
> > copy, use
> > > or disclose
> > > > > to
> > > > > > others
> > > > > > > this message or any attachment. Please also
> > notify the
> > > sender by
> > > > > > replying
> > > > > > > to this email or by telephone (+44(020
> 7896 0011)
> > and
> > > then delete
> > > > > the
> > > > > > > email and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that do
> > > > not
> > > > > > relate
> > > > > > > to the official business of this company
> shall be
> > > understood as
> > > > > > neither
> > > > > > > given nor endorsed by it. IG is a trading
> name
> > of IG
> > > Markets
> > > > > Limited
> > > > > > (a
> > > > > > > company registered in England and Wales,
> company
> > > number 04008957)
> > > > > > and IG
> > > > > > > Index Limited (a company registered in
> England
> > and
> > > Wales, company
> > > > > > number
> > > > > > > 01190902). Registered address at Cannon
> Bridge
> > House,
> > > 25 Dowgate
> > > > > > Hill,
> > > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number 195355)
> > > > > > and IG
> > > > > > > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated
> > > > > > by the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > > confidential and
> > > > for
> > > > > > the use of the addressee only, unless otherwise
> > indicated.
> > > If you are
> > > > not
> > > > > > the intended recipient, please do not read,
> copy, use
> > or
> > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify the
> > > sender by
> > > > replying
> > > > > > to this email or by telephone (+44(020 7896
> 0011) and
> > then
> > > delete the
> > > > > email
> > > > > > and any copies of it. Opinions, conclusion (etc)
> that
> > do not
> > > relate to
> > > > > the
> > > > > > official business of this company shall be
> understood
> > as
> > > neither given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited
> > > (a company
> > > > > > registered in England and Wales, company number
> > 04008957)
> > > and IG Index
> > > > > > Limited (a company registered in England and
> Wales,
> > company
> > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > 195355) and
> > > > IG
> > > > > > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > > The information contained in this email is strictly
> > > confidential and for
> > > > > the use of the addressee only, unless otherwise
> > indicated. If
> > > you are not
> > > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify
> the
> > sender
> > > by replying
> > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then
> > > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to
> > > > the
> > > > > official business of this company shall be
> understood as
> > > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > > registered in England and Wales, company number
> > 04008957) and
> > > IG Index
> > > > > Limited (a company registered in England and Wales,
> > company
> > > number
> > > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355) and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential
> > > and for
> > > > the use of the addressee only, unless otherwise
> indicated.
> > If
> > > you are not
> > > > the intended recipient, please do not read, copy,
> use or
> > > disclose to others
> > > > this message or any attachment. Please also notify
> the
> > sender by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011)
> and then
> > > delete the email
> > > > and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > relate to the
> > > > official business of this company shall be
> understood as
> > neither
> > > given nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > company
> > > > registered in England and Wales, company number
> 04008957)
> > and IG
> > > Index
> > > > Limited (a company registered in England and Wales,
> company
> > > number
> > > > 01190902). Registered address at Cannon Bridge
> House, 25
> > Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355) and IG
> > > > Index Limited (register number 114059) are
> authorised and
> > > regulated by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > for
> > > the use of the addressee only, unless otherwise indicated. If
> you
> > are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG
> > Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi
On 1, How can you guarantee two separate implemented clients would add the headers in the same order we are not specifying an order at the protocol level (nor should we) with regards to keyA being ordered before keyB? We shouldn’t be expecting keyA to be always set before keyB.
On 2, I believe we have changed the naming based on feedback from Jason already, e.g. we don’t have “get” method that inferred O(1) performance, like wise nor “put” but we have an “append”
On 3, in the KafkaProducer, I think we have mutability already, the value for time is changed if it is null, at the point of send:
“
long timestamp = record.timestamp() == null ? time.milliseconds() : record.timestamp();
“
As such the timestamp is already mutable, so what’s the difference here, we already have some mixed semantics. On timestamp.
e.g. currently if I send to records with timestamp not set, the wire binary sent the value for the timestamp would be different, as such we have mutation for the same record.
On 4, I think we should not expect not 1 or 2 headers, but infact 10’s of headers. This is the concern on immutable headers, whilst the append self-reference works nicely, what if someone needs to remove a header?
Trying to get this moving:
If we really wanted Immutable Headers and essentially you guys wont give +1 for it without.
Whats the feeling for adding methods to ProducerRecord that does the boiler plate code or creating a new ProducerRecord with the altered new headers (appended or removed) inside. E.g.
ProducerRecord {
ProducerRecord append(Iterable<Headers> headersToAppend){
return new ProducerRecord(key, value, headers.append(headersToAppend), ….)
}
ProducerRecord remove(Iterable<Headers> headersToAppend){
return new ProducerRecord(key, value, headers.remove(headersToAppend), ….)
}
}
Were the headers methods actually returns new objects, and the producer records methods create a new producer record with all the current values, but with the new modified headers.
Then interceptors / code return this new object?
Cheers
Mike
On 24/02/2017, 16:02, "ismaelj@gmail.com on behalf of Ismael Juma" <ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
Hi Michael,
Did you mean that you were happy to compromise to keep it mutable or
immutable? You wrote the former, but it sounded from the sentence that it
could have been a typo. So, my thoughts on this is that there are a few
things to take into account:
1. Semantics
2. Simplicity of use (the common operations should be easy to do)
3. If it's easy to reason about and safe (immutability helps with this)
4. Efficiency (both memory and CPU usage)
Regarding 1, I think it would be good to be very clear about the guarantees
that we are providing. It seems that we are saying that keys are unordered,
but what about the case where there are multiple values for the same key?
It seems that for some use cases (e.g. lineage), it may be useful to add
values to the same key while preserving the order.
Regarding 2, I agree that it's useful to have methods in `Headers` for the
very common use cases although we have to be careful with the naming to
avoid giving the wrong impression. Also, when it comes to `Map.Entry`, I
think I'd prefer a `toMap` method that simply gives the user a Map if
that's what they want (it makes it clear that there's a conversion
happening).
Regarding 3, the concern I have if we make the headers mutable is that it
seems to introduce some inconsistencies and potential edge cases. At the
moment, it's safe to keep a ProducerRecord and resend it, for example. If
the record is mutated by an interceptor, then this can lead to weird
behaviour. Also, it seems inconsistent that one has to create a new
ProducerRecord to modify the record timestamp, but that one has to mutate
the record to add headers. It seems like one should either embrace
immutability or mutability, but mixing both is not ideal.
Regarding 4, for the cases where there are a small number of headers and/or
1 or 2 interceptors, it doesn't seem difficult to come up with reasonable
implementations for mutable and immutable cases that do a good enough job.
However, if the number of headers and interceptors is large, then care is
needed for the immutable case to avoid unnecessary copying. A simple option
that adds little memory overhead if we have Header instances is to simply
add a self-reference to the previous Header in the linked list.
Ismael
On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Im happy to compromise to keep it mutable but move to an append style api.
> (as in guava interables concat)
>
> class Headers {
> Headers append(Iterable<Header> headers);
> }
>
>
> I don’t think we’d want prepend, this would give the idea of guaranteed
> ordering, when in actual fact we don’t provide that guarantee (.e.g one
> client can put headerA, then headerB, but another could put headerB then
> headerA, this shouldn’t cause issues), Also what if we changed to a hashmap
> for the internal implementation, its just a bucket of entries no ordering.
> I think we just need to provide an api to add/append headers.
>
> This ok? If so ill update KIP to record this.
>
> Cheers
> Mike
>
> On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> The point about usability is fair. It's also reasonable to expect that
> common use cases such as appending headers should be done efficiently.
>
> Perhaps we could compromise with something like this?
>
> class Headers {
> Headers append(Iterable<Header> headers);
> Headers prepend(Iterable<Header> headers);
> }
>
> That retains ease of use while still giving ourselves some flexibility
> in
> the implementation.
>
> -Jason
>
>
> On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > I wasn’t referring to the headers needing to be copied, im meaning
> the
> > fact we’d be forcing a new producer record to be created, with all
> the
> > contents copied.
> >
> > i.e what will happen is utility method will be created or end up
> being
> > used, which does this, and returns the new ProducerRecord instance.
> >
> > ProducerRecord addHeader(ProducerRecord record, Header header){
> > Return New ProducerRecord(record.key, record.value,
> record.timestamp…..,
> > record.headers.concat(header))
> > }
> >
> > To me this seems ugly, but will be inevitable if we don’t make adding
> > headers to existing records a simple clean method call.
> >
> >
> >
> > On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Lazy init can achieve/avoid that.
> >
> > Re the concat, why don’t we implement that inside the Headers
> rather
> > than causing everyone to implement this as adding headers in
> interceptors
> > will be a dominant use case. We want a user friendly API. Having as
> a user
> > having to code this instead of having the headers handle this for me
> seems
> > redundant.
> >
> > On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >
> > I thought the argument was against creating the extra objects
> > unnecessarily
> > (i.e. if they were not accessed). And note that making the
> Headers
> > immutable doesn't necessarily mean that they need to be
> copied:
> > you can do
> > a trick like Guava's Iterables.concat to add additional
> headers
> > without
> > changing the underlying collections.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > If the argument for not having a map holding the key, value
> > pairs is due
> > > to garbage creation of HashMap entry's, forcing the
> creation of
> > a whole new
> > > producer record to simply add a head, surely is creating
> a-lot
> > more?
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > The current producer interceptor API is this:
> > >
> > > ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
> > >
> > > So adding a header means creating a new ProducerRecord
> with a
> > new header
> > > added to the current headers and returning it. Would that
> not
> > work?
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > So how would you have this work if not mutable where
> > interceptors would
> > > > add headers?
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > I think the point on the mutability of Headers is worth
> > discussing a
> > > little
> > > > more. As far as I can tell, once the ProducerRecord (or
> > ConsumerRecord)
> > > is
> > > > constructed, there should be no need to further change
> the
> > headers. Is
> > > that
> > > > correct? If so, then why not enforce that that is the
> case
> > through the
> > > API?
> > > > One problem with mutability it that it constrains the
> > implementation of
> > > > Headers. For example, if we were backing with a byte
> slice,
> > would we
> > > recopy
> > > > the bytes if a header is added or would we maintain a
> satellite
> > > collection
> > > > of added records. Seems not great either way. If we
> really
> > think
> > > mutability
> > > > is needed, perhaps we could add a method to headers to
> convert
> > it to a
> > > > mutable type (e.g. a Headers.toMap)?
> > > >
> > > > I'm also with Ismael about exposing Headers.get(). I
> thought
> > it might
> > > make
> > > > sense to have a method like this instead:
> > > >
> > > > Iterable<Header> findMatching(Pattern pattern);
> > > >
> > > > This makes the (potential) need to scan the headers
> clear in
> > the API. I'd
> > > > also be fine exposing no getter at all. In general, Ï
> think
> > it's good to
> > > > start with a minimalistic API and work from there since
> it's
> > always
> > > easier
> > > > to add methods than remove them.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > >
> > > > > Hi Ismael
> > > > >
> > > > > On point 1,
> > > > >
> > > > > Sure makes sense will update shortly.
> > > > >
> > > > > On point 2,
> > > > >
> > > > > Setter/getter typical to properties/headers api’s
> > traditionally are map
> > > > > styled interfaces and what I believe is most expected
> styled
> > thus the
> > > > Key,
> > > > > Value setter.
> > > > > Also it would mean rather than an interface, we would
> be
> > making our
> > > > > internal header impl object we have for the array,
> exposed.
> > E.g. if we
> > > > had
> > > > > a Map really this would be Map.Entry interface, this
> is the
> > same
> > > reasons
> > > > on
> > > > > the map interface I cannot actually make the
> underlying Node
> > object
> > > > that’s
> > > > > the implementation for HashMap, so that internals can
> be
> > changed.
> > > > >
> > > > >
> > > > > On point 3,
> > > > >
> > > > > I think it people do expect it to be performant, thus
> why
> > originally
> > > > > concern I raised with using an array for to me is an
> early
> > memory
> > > > > optimisation. I think the user experience of
> > properties/headers is on a
> > > > > get/set model. This is why its important we have
> > encapsulated logic
> > > that
> > > > > then allows us to change this to a map, if this
> becomes and
> > issue, and
> > > > the
> > > > > memory overhead of hashmap is less so.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of
> > Ismael Juma" <
> > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk>
> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Great to see the progress that has been achieved
> on this
> > one. :) A
> > > > few
> > > > > comments regarding the APIs (I'm still reviewing
> the
> > message format
> > > > > changes):
> > > > >
> > > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> > `ConsumerRecord`
> > > should
> > > > be
> > > > > named `headers` (we avoid the `get` prefix in
> Kafka)
> > > > >
> > > > > 2. The `Headers` class is mutable (there's an `add`
> > method). Does
> > > it
> > > > > need
> > > > > to be? If so, it would be good to explain why.
> Related
> > to that, we
> > > > > should
> > > > > also explain the thinking around thread-safety. If
> we
> > keep the
> > > `add`
> > > > > method, it may make sense for it to take a `Header`
> > (that way we
> > > can
> > > > > add
> > > > > things to `Header` without changing the interface).
> > > > >
> > > > > 3. Do we need the `Headers.get()` method? People
> usually
> > assume
> > > that
> > > > > `get`
> > > > > would be efficient, but depending on the
> implementation
> > (the
> > > current
> > > > > proposal states that an array would be used), it
> may not
> > be. If we
> > > > > expect
> > > > > the number of headers to be small, it doesn't
> matter
> > though.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Have converted the interface/api bullets into
> > interface code
> > > > > snippets.
> > > > > >
> > > > > > Agreed implementation won’t take too long. We
> have
> > early versions
> > > > > already.
> > > > > > Maybe a week before you think about merging I
> would
> > assume it
> > > would
> > > > > be more
> > > > > > stabilised? I was thinking then we could fork
> from
> > your confluent
> > > > > branch,
> > > > > > making and then holding KIP-82 changes in a patch
> > file, that we
> > > can
> > > > > then
> > > > > > re-fork from apache once KIP98 final is merged,
> and
> > apply patch
> > > > with
> > > > > last
> > > > > > minute changes.
> > > > > >
> > > > > > Cheers
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > Hey Michael,
> > > > > >
> > > > > > Awesome. I have a minor request. The APIs are
> > currently
> > > > > documented as a
> > > > > > wiki list. Would you mind adding a code
> snippet
> > instead?
> > > It's a
> > > > > bit
> > > > > > easier
> > > > > > to process.
> > > > > >
> > > > > > How will be best to manage this, as we will
> > obviously build
> > > off
> > > > > your
> > > > > > KIP’s
> > > > > > > protocol changes, to avoid a merge hell,
> should
> > we branch
> > > > from
> > > > > your
> > > > > > branch
> > > > > > > in the confluent repo or is it worth
> having a
> > KIP-98
> > > special
> > > > > branch
> > > > > > in the
> > > > > > > apache git, that we can branch/fork from?
> > > > > >
> > > > > >
> > > > > > I was thinking about this also. Ideally we'd
> like
> > to get the
> > > > > changes
> > > > > > in as
> > > > > > close together as possible since we only
> want one
> > magic bump
> > > > and
> > > > > some
> > > > > > users
> > > > > > deploy trunk. The level of effort to change
> the
> > format for
> > > > > headers
> > > > > > seems
> > > > > > not too high. Do you agree? My guess is that
> the
> > KIP-98
> > > message
> > > > > format
> > > > > > patch will take 2-3 weeks to review before we
> > merge to trunk,
> > > > so
> > > > > you
> > > > > > could
> > > > > > hold off implementing until that patch has
> somewhat
> > > stabilized.
> > > > > That
> > > > > > would
> > > > > > save some potential rebase pain.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > confidential
> > > > and
> > > > > for
> > > > > > the use of the addressee only, unless otherwise
> > indicated. If you
> > > > > are not
> > > > > > the intended recipient, please do not read,
> copy, use
> > or disclose
> > > > to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify the
> > sender by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then delete
> > > > the
> > > > > > email and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > > relate
> > > > > > to the official business of this company shall be
> > understood as
> > > > > neither
> > > > > > given nor endorsed by it. IG is a trading name
> of IG
> > Markets
> > > > Limited
> > > > > (a
> > > > > > company registered in England and Wales, company
> > number 04008957)
> > > > > and IG
> > > > > > Index Limited (a company registered in England
> and
> > Wales, company
> > > > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25 Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are
> > > not
> > > > > the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify the
> > sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > > the
> > > > > official business of this company shall be understood
> as
> > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > > registered in England and Wales, company number
> 04008957)
> > and IG Index
> > > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > > 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and
> > > IG
> > > > > Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential
> > and for
> > > the use of the addressee only, unless otherwise indicated.
> If
> > you are not
> > > the intended recipient, please do not read, copy, use or
> > disclose to others
> > > this message or any attachment. Please also notify the
> sender by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the email
> > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > company
> > > registered in England and Wales, company number 04008957)
> and IG
> > Index
> > > Limited (a company registered in England and Wales, company
> > number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> > regulated by the
> > > Financial Conduct Authority.
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Michael,
Did you mean that you were happy to compromise to keep it mutable or
immutable? You wrote the former, but it sounded from the sentence that it
could have been a typo. So, my thoughts on this is that there are a few
things to take into account:
1. Semantics
2. Simplicity of use (the common operations should be easy to do)
3. If it's easy to reason about and safe (immutability helps with this)
4. Efficiency (both memory and CPU usage)
Regarding 1, I think it would be good to be very clear about the guarantees
that we are providing. It seems that we are saying that keys are unordered,
but what about the case where there are multiple values for the same key?
It seems that for some use cases (e.g. lineage), it may be useful to add
values to the same key while preserving the order.
Regarding 2, I agree that it's useful to have methods in `Headers` for the
very common use cases although we have to be careful with the naming to
avoid giving the wrong impression. Also, when it comes to `Map.Entry`, I
think I'd prefer a `toMap` method that simply gives the user a Map if
that's what they want (it makes it clear that there's a conversion
happening).
Regarding 3, the concern I have if we make the headers mutable is that it
seems to introduce some inconsistencies and potential edge cases. At the
moment, it's safe to keep a ProducerRecord and resend it, for example. If
the record is mutated by an interceptor, then this can lead to weird
behaviour. Also, it seems inconsistent that one has to create a new
ProducerRecord to modify the record timestamp, but that one has to mutate
the record to add headers. It seems like one should either embrace
immutability or mutability, but mixing both is not ideal.
Regarding 4, for the cases where there are a small number of headers and/or
1 or 2 interceptors, it doesn't seem difficult to come up with reasonable
implementations for mutable and immutable cases that do a good enough job.
However, if the number of headers and interceptors is large, then care is
needed for the immutable case to avoid unnecessary copying. A simple option
that adds little memory overhead if we have Header instances is to simply
add a self-reference to the previous Header in the linked list.
Ismael
On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Im happy to compromise to keep it mutable but move to an append style api.
> (as in guava interables concat)
>
> class Headers {
> Headers append(Iterable<Header> headers);
> }
>
>
> I don’t think we’d want prepend, this would give the idea of guaranteed
> ordering, when in actual fact we don’t provide that guarantee (.e.g one
> client can put headerA, then headerB, but another could put headerB then
> headerA, this shouldn’t cause issues), Also what if we changed to a hashmap
> for the internal implementation, its just a bucket of entries no ordering.
> I think we just need to provide an api to add/append headers.
>
> This ok? If so ill update KIP to record this.
>
> Cheers
> Mike
>
> On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> The point about usability is fair. It's also reasonable to expect that
> common use cases such as appending headers should be done efficiently.
>
> Perhaps we could compromise with something like this?
>
> class Headers {
> Headers append(Iterable<Header> headers);
> Headers prepend(Iterable<Header> headers);
> }
>
> That retains ease of use while still giving ourselves some flexibility
> in
> the implementation.
>
> -Jason
>
>
> On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > I wasn’t referring to the headers needing to be copied, im meaning
> the
> > fact we’d be forcing a new producer record to be created, with all
> the
> > contents copied.
> >
> > i.e what will happen is utility method will be created or end up
> being
> > used, which does this, and returns the new ProducerRecord instance.
> >
> > ProducerRecord addHeader(ProducerRecord record, Header header){
> > Return New ProducerRecord(record.key, record.value,
> record.timestamp…..,
> > record.headers.concat(header))
> > }
> >
> > To me this seems ugly, but will be inevitable if we don’t make adding
> > headers to existing records a simple clean method call.
> >
> >
> >
> > On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Lazy init can achieve/avoid that.
> >
> > Re the concat, why don’t we implement that inside the Headers
> rather
> > than causing everyone to implement this as adding headers in
> interceptors
> > will be a dominant use case. We want a user friendly API. Having as
> a user
> > having to code this instead of having the headers handle this for me
> seems
> > redundant.
> >
> > On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >
> > I thought the argument was against creating the extra objects
> > unnecessarily
> > (i.e. if they were not accessed). And note that making the
> Headers
> > immutable doesn't necessarily mean that they need to be
> copied:
> > you can do
> > a trick like Guava's Iterables.concat to add additional
> headers
> > without
> > changing the underlying collections.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > If the argument for not having a map holding the key, value
> > pairs is due
> > > to garbage creation of HashMap entry's, forcing the
> creation of
> > a whole new
> > > producer record to simply add a head, surely is creating
> a-lot
> > more?
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Wednesday, February 22, 2017 10:09 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > The current producer interceptor API is this:
> > >
> > > ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
> > >
> > > So adding a header means creating a new ProducerRecord
> with a
> > new header
> > > added to the current headers and returning it. Would that
> not
> > work?
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > So how would you have this work if not mutable where
> > interceptors would
> > > > add headers?
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Jason Gustafson <ja...@confluent.io>
> > > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > I think the point on the mutability of Headers is worth
> > discussing a
> > > little
> > > > more. As far as I can tell, once the ProducerRecord (or
> > ConsumerRecord)
> > > is
> > > > constructed, there should be no need to further change
> the
> > headers. Is
> > > that
> > > > correct? If so, then why not enforce that that is the
> case
> > through the
> > > API?
> > > > One problem with mutability it that it constrains the
> > implementation of
> > > > Headers. For example, if we were backing with a byte
> slice,
> > would we
> > > recopy
> > > > the bytes if a header is added or would we maintain a
> satellite
> > > collection
> > > > of added records. Seems not great either way. If we
> really
> > think
> > > mutability
> > > > is needed, perhaps we could add a method to headers to
> convert
> > it to a
> > > > mutable type (e.g. a Headers.toMap)?
> > > >
> > > > I'm also with Ismael about exposing Headers.get(). I
> thought
> > it might
> > > make
> > > > sense to have a method like this instead:
> > > >
> > > > Iterable<Header> findMatching(Pattern pattern);
> > > >
> > > > This makes the (potential) need to scan the headers
> clear in
> > the API. I'd
> > > > also be fine exposing no getter at all. In general, Ï
> think
> > it's good to
> > > > start with a minimalistic API and work from there since
> it's
> > always
> > > easier
> > > > to add methods than remove them.
> > > >
> > > > -Jason
> > > >
> > > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > >
> > > > > Hi Ismael
> > > > >
> > > > > On point 1,
> > > > >
> > > > > Sure makes sense will update shortly.
> > > > >
> > > > > On point 2,
> > > > >
> > > > > Setter/getter typical to properties/headers api’s
> > traditionally are map
> > > > > styled interfaces and what I believe is most expected
> styled
> > thus the
> > > > Key,
> > > > > Value setter.
> > > > > Also it would mean rather than an interface, we would
> be
> > making our
> > > > > internal header impl object we have for the array,
> exposed.
> > E.g. if we
> > > > had
> > > > > a Map really this would be Map.Entry interface, this
> is the
> > same
> > > reasons
> > > > on
> > > > > the map interface I cannot actually make the
> underlying Node
> > object
> > > > that’s
> > > > > the implementation for HashMap, so that internals can
> be
> > changed.
> > > > >
> > > > >
> > > > > On point 3,
> > > > >
> > > > > I think it people do expect it to be performant, thus
> why
> > originally
> > > > > concern I raised with using an array for to me is an
> early
> > memory
> > > > > optimisation. I think the user experience of
> > properties/headers is on a
> > > > > get/set model. This is why its important we have
> > encapsulated logic
> > > that
> > > > > then allows us to change this to a map, if this
> becomes and
> > issue, and
> > > > the
> > > > > memory overhead of hashmap is less so.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of
> > Ismael Juma" <
> > > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk>
> wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Great to see the progress that has been achieved
> on this
> > one. :) A
> > > > few
> > > > > comments regarding the APIs (I'm still reviewing
> the
> > message format
> > > > > changes):
> > > > >
> > > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> > `ConsumerRecord`
> > > should
> > > > be
> > > > > named `headers` (we avoid the `get` prefix in
> Kafka)
> > > > >
> > > > > 2. The `Headers` class is mutable (there's an `add`
> > method). Does
> > > it
> > > > > need
> > > > > to be? If so, it would be good to explain why.
> Related
> > to that, we
> > > > > should
> > > > > also explain the thinking around thread-safety. If
> we
> > keep the
> > > `add`
> > > > > method, it may make sense for it to take a `Header`
> > (that way we
> > > can
> > > > > add
> > > > > things to `Header` without changing the interface).
> > > > >
> > > > > 3. Do we need the `Headers.get()` method? People
> usually
> > assume
> > > that
> > > > > `get`
> > > > > would be efficient, but depending on the
> implementation
> > (the
> > > current
> > > > > proposal states that an array would be used), it
> may not
> > be. If we
> > > > > expect
> > > > > the number of headers to be small, it doesn't
> matter
> > though.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jason,
> > > > > >
> > > > > > Have converted the interface/api bullets into
> > interface code
> > > > > snippets.
> > > > > >
> > > > > > Agreed implementation won’t take too long. We
> have
> > early versions
> > > > > already.
> > > > > > Maybe a week before you think about merging I
> would
> > assume it
> > > would
> > > > > be more
> > > > > > stabilised? I was thinking then we could fork
> from
> > your confluent
> > > > > branch,
> > > > > > making and then holding KIP-82 changes in a patch
> > file, that we
> > > can
> > > > > then
> > > > > > re-fork from apache once KIP98 final is merged,
> and
> > apply patch
> > > > with
> > > > > last
> > > > > > minute changes.
> > > > > >
> > > > > > Cheers
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > Hey Michael,
> > > > > >
> > > > > > Awesome. I have a minor request. The APIs are
> > currently
> > > > > documented as a
> > > > > > wiki list. Would you mind adding a code
> snippet
> > instead?
> > > It's a
> > > > > bit
> > > > > > easier
> > > > > > to process.
> > > > > >
> > > > > > How will be best to manage this, as we will
> > obviously build
> > > off
> > > > > your
> > > > > > KIP’s
> > > > > > > protocol changes, to avoid a merge hell,
> should
> > we branch
> > > > from
> > > > > your
> > > > > > branch
> > > > > > > in the confluent repo or is it worth
> having a
> > KIP-98
> > > special
> > > > > branch
> > > > > > in the
> > > > > > > apache git, that we can branch/fork from?
> > > > > >
> > > > > >
> > > > > > I was thinking about this also. Ideally we'd
> like
> > to get the
> > > > > changes
> > > > > > in as
> > > > > > close together as possible since we only
> want one
> > magic bump
> > > > and
> > > > > some
> > > > > > users
> > > > > > deploy trunk. The level of effort to change
> the
> > format for
> > > > > headers
> > > > > > seems
> > > > > > not too high. Do you agree? My guess is that
> the
> > KIP-98
> > > message
> > > > > format
> > > > > > patch will take 2-3 weeks to review before we
> > merge to trunk,
> > > > so
> > > > > you
> > > > > > could
> > > > > > hold off implementing until that patch has
> somewhat
> > > stabilized.
> > > > > That
> > > > > > would
> > > > > > save some potential rebase pain.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > confidential
> > > > and
> > > > > for
> > > > > > the use of the addressee only, unless otherwise
> > indicated. If you
> > > > > are not
> > > > > > the intended recipient, please do not read,
> copy, use
> > or disclose
> > > > to
> > > > > others
> > > > > > this message or any attachment. Please also
> notify the
> > sender by
> > > > > replying
> > > > > > to this email or by telephone (+44(020 7896 0011)
> and
> > then delete
> > > > the
> > > > > > email and any copies of it. Opinions, conclusion
> (etc)
> > that do
> > > not
> > > > > relate
> > > > > > to the official business of this company shall be
> > understood as
> > > > > neither
> > > > > > given nor endorsed by it. IG is a trading name
> of IG
> > Markets
> > > > Limited
> > > > > (a
> > > > > > company registered in England and Wales, company
> > number 04008957)
> > > > > and IG
> > > > > > Index Limited (a company registered in England
> and
> > Wales, company
> > > > > number
> > > > > > 01190902). Registered address at Cannon Bridge
> House,
> > 25 Dowgate
> > > > > Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> (register
> > number 195355)
> > > > > and IG
> > > > > > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated
> > > > > by the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > confidential and
> > > for
> > > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are
> > > not
> > > > > the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > > others
> > > > > this message or any attachment. Please also notify the
> > sender by
> > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > > the
> > > > > official business of this company shall be understood
> as
> > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > > registered in England and Wales, company number
> 04008957)
> > and IG Index
> > > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > > 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and
> > > IG
> > > > > Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated. If
> > you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender
> > by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > > registered in England and Wales, company number
> 04008957) and
> > IG Index
> > > > Limited (a company registered in England and Wales,
> company
> > number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential
> > and for
> > > the use of the addressee only, unless otherwise indicated.
> If
> > you are not
> > > the intended recipient, please do not read, copy, use or
> > disclose to others
> > > this message or any attachment. Please also notify the
> sender by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> > delete the email
> > > and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the
> > > official business of this company shall be understood as
> neither
> > given nor
> > > endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > company
> > > registered in England and Wales, company number 04008957)
> and IG
> > Index
> > > Limited (a company registered in England and Wales, company
> > number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> > regulated by the
> > > Financial Conduct Authority.
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Im happy to compromise to keep it mutable but move to an append style api. (as in guava interables concat)
class Headers {
Headers append(Iterable<Header> headers);
}
I don’t think we’d want prepend, this would give the idea of guaranteed ordering, when in actual fact we don’t provide that guarantee (.e.g one client can put headerA, then headerB, but another could put headerB then headerA, this shouldn’t cause issues), Also what if we changed to a hashmap for the internal implementation, its just a bucket of entries no ordering. I think we just need to provide an api to add/append headers.
This ok? If so ill update KIP to record this.
Cheers
Mike
On 23/02/2017, 00:37, "Jason Gustafson" <ja...@confluent.io> wrote:
The point about usability is fair. It's also reasonable to expect that
common use cases such as appending headers should be done efficiently.
Perhaps we could compromise with something like this?
class Headers {
Headers append(Iterable<Header> headers);
Headers prepend(Iterable<Header> headers);
}
That retains ease of use while still giving ourselves some flexibility in
the implementation.
-Jason
On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <Mi...@ig.com>
wrote:
> I wasn’t referring to the headers needing to be copied, im meaning the
> fact we’d be forcing a new producer record to be created, with all the
> contents copied.
>
> i.e what will happen is utility method will be created or end up being
> used, which does this, and returns the new ProducerRecord instance.
>
> ProducerRecord addHeader(ProducerRecord record, Header header){
> Return New ProducerRecord(record.key, record.value, record.timestamp…..,
> record.headers.concat(header))
> }
>
> To me this seems ugly, but will be inevitable if we don’t make adding
> headers to existing records a simple clean method call.
>
>
>
> On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Lazy init can achieve/avoid that.
>
> Re the concat, why don’t we implement that inside the Headers rather
> than causing everyone to implement this as adding headers in interceptors
> will be a dominant use case. We want a user friendly API. Having as a user
> having to code this instead of having the headers handle this for me seems
> redundant.
>
> On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> I thought the argument was against creating the extra objects
> unnecessarily
> (i.e. if they were not accessed). And note that making the Headers
> immutable doesn't necessarily mean that they need to be copied:
> you can do
> a trick like Guava's Iterables.concat to add additional headers
> without
> changing the underlying collections.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > If the argument for not having a map holding the key, value
> pairs is due
> > to garbage creation of HashMap entry's, forcing the creation of
> a whole new
> > producer record to simply add a head, surely is creating a-lot
> more?
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Wednesday, February 22, 2017 10:09 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > The current producer interceptor API is this:
> >
> > ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
> >
> > So adding a header means creating a new ProducerRecord with a
> new header
> > added to the current headers and returning it. Would that not
> work?
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > wrote:
> >
> > > So how would you have this work if not mutable where
> interceptors would
> > > add headers?
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > I think the point on the mutability of Headers is worth
> discussing a
> > little
> > > more. As far as I can tell, once the ProducerRecord (or
> ConsumerRecord)
> > is
> > > constructed, there should be no need to further change the
> headers. Is
> > that
> > > correct? If so, then why not enforce that that is the case
> through the
> > API?
> > > One problem with mutability it that it constrains the
> implementation of
> > > Headers. For example, if we were backing with a byte slice,
> would we
> > recopy
> > > the bytes if a header is added or would we maintain a satellite
> > collection
> > > of added records. Seems not great either way. If we really
> think
> > mutability
> > > is needed, perhaps we could add a method to headers to convert
> it to a
> > > mutable type (e.g. a Headers.toMap)?
> > >
> > > I'm also with Ismael about exposing Headers.get(). I thought
> it might
> > make
> > > sense to have a method like this instead:
> > >
> > > Iterable<Header> findMatching(Pattern pattern);
> > >
> > > This makes the (potential) need to scan the headers clear in
> the API. I'd
> > > also be fine exposing no getter at all. In general, Ï think
> it's good to
> > > start with a minimalistic API and work from there since it's
> always
> > easier
> > > to add methods than remove them.
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > >
> > > > Hi Ismael
> > > >
> > > > On point 1,
> > > >
> > > > Sure makes sense will update shortly.
> > > >
> > > > On point 2,
> > > >
> > > > Setter/getter typical to properties/headers api’s
> traditionally are map
> > > > styled interfaces and what I believe is most expected styled
> thus the
> > > Key,
> > > > Value setter.
> > > > Also it would mean rather than an interface, we would be
> making our
> > > > internal header impl object we have for the array, exposed.
> E.g. if we
> > > had
> > > > a Map really this would be Map.Entry interface, this is the
> same
> > reasons
> > > on
> > > > the map interface I cannot actually make the underlying Node
> object
> > > that’s
> > > > the implementation for HashMap, so that internals can be
> changed.
> > > >
> > > >
> > > > On point 3,
> > > >
> > > > I think it people do expect it to be performant, thus why
> originally
> > > > concern I raised with using an array for to me is an early
> memory
> > > > optimisation. I think the user experience of
> properties/headers is on a
> > > > get/set model. This is why its important we have
> encapsulated logic
> > that
> > > > then allows us to change this to a map, if this becomes and
> issue, and
> > > the
> > > > memory overhead of hashmap is less so.
> > > >
> > > >
> > > >
> > > >
> > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of
> Ismael Juma" <
> > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Great to see the progress that has been achieved on this
> one. :) A
> > > few
> > > > comments regarding the APIs (I'm still reviewing the
> message format
> > > > changes):
> > > >
> > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> `ConsumerRecord`
> > should
> > > be
> > > > named `headers` (we avoid the `get` prefix in Kafka)
> > > >
> > > > 2. The `Headers` class is mutable (there's an `add`
> method). Does
> > it
> > > > need
> > > > to be? If so, it would be good to explain why. Related
> to that, we
> > > > should
> > > > also explain the thinking around thread-safety. If we
> keep the
> > `add`
> > > > method, it may make sense for it to take a `Header`
> (that way we
> > can
> > > > add
> > > > things to `Header` without changing the interface).
> > > >
> > > > 3. Do we need the `Headers.get()` method? People usually
> assume
> > that
> > > > `get`
> > > > would be efficient, but depending on the implementation
> (the
> > current
> > > > proposal states that an array would be used), it may not
> be. If we
> > > > expect
> > > > the number of headers to be small, it doesn't matter
> though.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Have converted the interface/api bullets into
> interface code
> > > > snippets.
> > > > >
> > > > > Agreed implementation won’t take too long. We have
> early versions
> > > > already.
> > > > > Maybe a week before you think about merging I would
> assume it
> > would
> > > > be more
> > > > > stabilised? I was thinking then we could fork from
> your confluent
> > > > branch,
> > > > > making and then holding KIP-82 changes in a patch
> file, that we
> > can
> > > > then
> > > > > re-fork from apache once KIP98 final is merged, and
> apply patch
> > > with
> > > > last
> > > > > minute changes.
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > >
> > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> jason@confluent.io>
> > > wrote:
> > > > >
> > > > > Hey Michael,
> > > > >
> > > > > Awesome. I have a minor request. The APIs are
> currently
> > > > documented as a
> > > > > wiki list. Would you mind adding a code snippet
> instead?
> > It's a
> > > > bit
> > > > > easier
> > > > > to process.
> > > > >
> > > > > How will be best to manage this, as we will
> obviously build
> > off
> > > > your
> > > > > KIP’s
> > > > > > protocol changes, to avoid a merge hell, should
> we branch
> > > from
> > > > your
> > > > > branch
> > > > > > in the confluent repo or is it worth having a
> KIP-98
> > special
> > > > branch
> > > > > in the
> > > > > > apache git, that we can branch/fork from?
> > > > >
> > > > >
> > > > > I was thinking about this also. Ideally we'd like
> to get the
> > > > changes
> > > > > in as
> > > > > close together as possible since we only want one
> magic bump
> > > and
> > > > some
> > > > > users
> > > > > deploy trunk. The level of effort to change the
> format for
> > > > headers
> > > > > seems
> > > > > not too high. Do you agree? My guess is that the
> KIP-98
> > message
> > > > format
> > > > > patch will take 2-3 weeks to review before we
> merge to trunk,
> > > so
> > > > you
> > > > > could
> > > > > hold off implementing until that patch has somewhat
> > stabilized.
> > > > That
> > > > > would
> > > > > save some potential rebase pain.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> confidential
> > > and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> indicated. If you
> > > > are not
> > > > > the intended recipient, please do not read, copy, use
> or disclose
> > > to
> > > > others
> > > > > this message or any attachment. Please also notify the
> sender by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and
> then delete
> > > the
> > > > > email and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > > relate
> > > > > to the official business of this company shall be
> understood as
> > > > neither
> > > > > given nor endorsed by it. IG is a trading name of IG
> Markets
> > > Limited
> > > > (a
> > > > > company registered in England and Wales, company
> number 04008957)
> > > > and IG
> > > > > Index Limited (a company registered in England and
> Wales, company
> > > > number
> > > > > 01190902). Registered address at Cannon Bridge House,
> 25 Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are authorised
> and
> > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated.
> If you are
> > not
> > > > the intended recipient, please do not read, copy, use or
> disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > > the
> > > > official business of this company shall be understood as
> neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a company
> > > > registered in England and Wales, company number 04008957)
> and IG Index
> > > > Limited (a company registered in England and Wales, company
> number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and
> regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential and for
> > > the use of the addressee only, unless otherwise indicated. If
> you are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > the
> > > official business of this company shall be understood as
> neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential
> and for
> > the use of the addressee only, unless otherwise indicated. If
> you are not
> > the intended recipient, please do not read, copy, use or
> disclose to others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then
> delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company
> number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> > Index Limited (register number 114059) are authorised and
> regulated by the
> > Financial Conduct Authority.
> >
>
>
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
The point about usability is fair. It's also reasonable to expect that
common use cases such as appending headers should be done efficiently.
Perhaps we could compromise with something like this?
class Headers {
Headers append(Iterable<Header> headers);
Headers prepend(Iterable<Header> headers);
}
That retains ease of use while still giving ourselves some flexibility in
the implementation.
-Jason
On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <Mi...@ig.com>
wrote:
> I wasn’t referring to the headers needing to be copied, im meaning the
> fact we’d be forcing a new producer record to be created, with all the
> contents copied.
>
> i.e what will happen is utility method will be created or end up being
> used, which does this, and returns the new ProducerRecord instance.
>
> ProducerRecord addHeader(ProducerRecord record, Header header){
> Return New ProducerRecord(record.key, record.value, record.timestamp…..,
> record.headers.concat(header))
> }
>
> To me this seems ugly, but will be inevitable if we don’t make adding
> headers to existing records a simple clean method call.
>
>
>
> On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Lazy init can achieve/avoid that.
>
> Re the concat, why don’t we implement that inside the Headers rather
> than causing everyone to implement this as adding headers in interceptors
> will be a dominant use case. We want a user friendly API. Having as a user
> having to code this instead of having the headers handle this for me seems
> redundant.
>
> On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> I thought the argument was against creating the extra objects
> unnecessarily
> (i.e. if they were not accessed). And note that making the Headers
> immutable doesn't necessarily mean that they need to be copied:
> you can do
> a trick like Guava's Iterables.concat to add additional headers
> without
> changing the underlying collections.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > If the argument for not having a map holding the key, value
> pairs is due
> > to garbage creation of HashMap entry's, forcing the creation of
> a whole new
> > producer record to simply add a head, surely is creating a-lot
> more?
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Wednesday, February 22, 2017 10:09 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > The current producer interceptor API is this:
> >
> > ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
> >
> > So adding a header means creating a new ProducerRecord with a
> new header
> > added to the current headers and returning it. Would that not
> work?
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > wrote:
> >
> > > So how would you have this work if not mutable where
> interceptors would
> > > add headers?
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Jason Gustafson <ja...@confluent.io>
> > > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > I think the point on the mutability of Headers is worth
> discussing a
> > little
> > > more. As far as I can tell, once the ProducerRecord (or
> ConsumerRecord)
> > is
> > > constructed, there should be no need to further change the
> headers. Is
> > that
> > > correct? If so, then why not enforce that that is the case
> through the
> > API?
> > > One problem with mutability it that it constrains the
> implementation of
> > > Headers. For example, if we were backing with a byte slice,
> would we
> > recopy
> > > the bytes if a header is added or would we maintain a satellite
> > collection
> > > of added records. Seems not great either way. If we really
> think
> > mutability
> > > is needed, perhaps we could add a method to headers to convert
> it to a
> > > mutable type (e.g. a Headers.toMap)?
> > >
> > > I'm also with Ismael about exposing Headers.get(). I thought
> it might
> > make
> > > sense to have a method like this instead:
> > >
> > > Iterable<Header> findMatching(Pattern pattern);
> > >
> > > This makes the (potential) need to scan the headers clear in
> the API. I'd
> > > also be fine exposing no getter at all. In general, Ï think
> it's good to
> > > start with a minimalistic API and work from there since it's
> always
> > easier
> > > to add methods than remove them.
> > >
> > > -Jason
> > >
> > > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > >
> > > > Hi Ismael
> > > >
> > > > On point 1,
> > > >
> > > > Sure makes sense will update shortly.
> > > >
> > > > On point 2,
> > > >
> > > > Setter/getter typical to properties/headers api’s
> traditionally are map
> > > > styled interfaces and what I believe is most expected styled
> thus the
> > > Key,
> > > > Value setter.
> > > > Also it would mean rather than an interface, we would be
> making our
> > > > internal header impl object we have for the array, exposed.
> E.g. if we
> > > had
> > > > a Map really this would be Map.Entry interface, this is the
> same
> > reasons
> > > on
> > > > the map interface I cannot actually make the underlying Node
> object
> > > that’s
> > > > the implementation for HashMap, so that internals can be
> changed.
> > > >
> > > >
> > > > On point 3,
> > > >
> > > > I think it people do expect it to be performant, thus why
> originally
> > > > concern I raised with using an array for to me is an early
> memory
> > > > optimisation. I think the user experience of
> properties/headers is on a
> > > > get/set model. This is why its important we have
> encapsulated logic
> > that
> > > > then allows us to change this to a map, if this becomes and
> issue, and
> > > the
> > > > memory overhead of hashmap is less so.
> > > >
> > > >
> > > >
> > > >
> > > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of
> Ismael Juma" <
> > > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Great to see the progress that has been achieved on this
> one. :) A
> > > few
> > > > comments regarding the APIs (I'm still reviewing the
> message format
> > > > changes):
> > > >
> > > > 1. Nit: `getHeaders` in `ProducerRecord` and
> `ConsumerRecord`
> > should
> > > be
> > > > named `headers` (we avoid the `get` prefix in Kafka)
> > > >
> > > > 2. The `Headers` class is mutable (there's an `add`
> method). Does
> > it
> > > > need
> > > > to be? If so, it would be good to explain why. Related
> to that, we
> > > > should
> > > > also explain the thinking around thread-safety. If we
> keep the
> > `add`
> > > > method, it may make sense for it to take a `Header`
> (that way we
> > can
> > > > add
> > > > things to `Header` without changing the interface).
> > > >
> > > > 3. Do we need the `Headers.get()` method? People usually
> assume
> > that
> > > > `get`
> > > > would be efficient, but depending on the implementation
> (the
> > current
> > > > proposal states that an array would be used), it may not
> be. If we
> > > > expect
> > > > the number of headers to be small, it doesn't matter
> though.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > > Michael.Pearce@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Jason,
> > > > >
> > > > > Have converted the interface/api bullets into
> interface code
> > > > snippets.
> > > > >
> > > > > Agreed implementation won’t take too long. We have
> early versions
> > > > already.
> > > > > Maybe a week before you think about merging I would
> assume it
> > would
> > > > be more
> > > > > stabilised? I was thinking then we could fork from
> your confluent
> > > > branch,
> > > > > making and then holding KIP-82 changes in a patch
> file, that we
> > can
> > > > then
> > > > > re-fork from apache once KIP98 final is merged, and
> apply patch
> > > with
> > > > last
> > > > > minute changes.
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > >
> > > > > On 22/02/2017, 00:56, "Jason Gustafson" <
> jason@confluent.io>
> > > wrote:
> > > > >
> > > > > Hey Michael,
> > > > >
> > > > > Awesome. I have a minor request. The APIs are
> currently
> > > > documented as a
> > > > > wiki list. Would you mind adding a code snippet
> instead?
> > It's a
> > > > bit
> > > > > easier
> > > > > to process.
> > > > >
> > > > > How will be best to manage this, as we will
> obviously build
> > off
> > > > your
> > > > > KIP’s
> > > > > > protocol changes, to avoid a merge hell, should
> we branch
> > > from
> > > > your
> > > > > branch
> > > > > > in the confluent repo or is it worth having a
> KIP-98
> > special
> > > > branch
> > > > > in the
> > > > > > apache git, that we can branch/fork from?
> > > > >
> > > > >
> > > > > I was thinking about this also. Ideally we'd like
> to get the
> > > > changes
> > > > > in as
> > > > > close together as possible since we only want one
> magic bump
> > > and
> > > > some
> > > > > users
> > > > > deploy trunk. The level of effort to change the
> format for
> > > > headers
> > > > > seems
> > > > > not too high. Do you agree? My guess is that the
> KIP-98
> > message
> > > > format
> > > > > patch will take 2-3 weeks to review before we
> merge to trunk,
> > > so
> > > > you
> > > > > could
> > > > > hold off implementing until that patch has somewhat
> > stabilized.
> > > > That
> > > > > would
> > > > > save some potential rebase pain.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> confidential
> > > and
> > > > for
> > > > > the use of the addressee only, unless otherwise
> indicated. If you
> > > > are not
> > > > > the intended recipient, please do not read, copy, use
> or disclose
> > > to
> > > > others
> > > > > this message or any attachment. Please also notify the
> sender by
> > > > replying
> > > > > to this email or by telephone (+44(020 7896 0011) and
> then delete
> > > the
> > > > > email and any copies of it. Opinions, conclusion (etc)
> that do
> > not
> > > > relate
> > > > > to the official business of this company shall be
> understood as
> > > > neither
> > > > > given nor endorsed by it. IG is a trading name of IG
> Markets
> > > Limited
> > > > (a
> > > > > company registered in England and Wales, company
> number 04008957)
> > > > and IG
> > > > > Index Limited (a company registered in England and
> Wales, company
> > > > number
> > > > > 01190902). Registered address at Cannon Bridge House,
> 25 Dowgate
> > > > Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited (register
> number 195355)
> > > > and IG
> > > > > Index Limited (register number 114059) are authorised
> and
> > regulated
> > > > by the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> confidential and
> > for
> > > > the use of the addressee only, unless otherwise indicated.
> If you are
> > not
> > > > the intended recipient, please do not read, copy, use or
> disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > > the
> > > > official business of this company shall be understood as
> neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited
> (a company
> > > > registered in England and Wales, company number 04008957)
> and IG Index
> > > > Limited (a company registered in England and Wales, company
> number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and
> regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > > The information contained in this email is strictly
> confidential and for
> > > the use of the addressee only, unless otherwise indicated. If
> you are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the sender
> by replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > the
> > > official business of this company shall be understood as
> neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > registered in England and Wales, company number 04008957) and
> IG Index
> > > Limited (a company registered in England and Wales, company
> number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential
> and for
> > the use of the addressee only, unless otherwise indicated. If
> you are not
> > the intended recipient, please do not read, copy, use or
> disclose to others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then
> delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to the
> > official business of this company shall be understood as neither
> given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > registered in England and Wales, company number 04008957) and IG
> Index
> > Limited (a company registered in England and Wales, company
> number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> > Index Limited (register number 114059) are authorised and
> regulated by the
> > Financial Conduct Authority.
> >
>
>
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
I wasn’t referring to the headers needing to be copied, im meaning the fact we’d be forcing a new producer record to be created, with all the contents copied.
i.e what will happen is utility method will be created or end up being used, which does this, and returns the new ProducerRecord instance.
ProducerRecord addHeader(ProducerRecord record, Header header){
Return New ProducerRecord(record.key, record.value, record.timestamp….., record.headers.concat(header))
}
To me this seems ugly, but will be inevitable if we don’t make adding headers to existing records a simple clean method call.
On 22/02/2017, 22:57, "Michael Pearce" <Mi...@ig.com> wrote:
Lazy init can achieve/avoid that.
Re the concat, why don’t we implement that inside the Headers rather than causing everyone to implement this as adding headers in interceptors will be a dominant use case. We want a user friendly API. Having as a user having to code this instead of having the headers handle this for me seems redundant.
On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io> wrote:
I thought the argument was against creating the extra objects unnecessarily
(i.e. if they were not accessed). And note that making the Headers
immutable doesn't necessarily mean that they need to be copied: you can do
a trick like Guava's Iterables.concat to add additional headers without
changing the underlying collections.
-Jason
On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <Mi...@ig.com>
wrote:
> If the argument for not having a map holding the key, value pairs is due
> to garbage creation of HashMap entry's, forcing the creation of a whole new
> producer record to simply add a head, surely is creating a-lot more?
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Wednesday, February 22, 2017 10:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The current producer interceptor API is this:
>
> ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
>
> So adding a header means creating a new ProducerRecord with a new header
> added to the current headers and returning it. Would that not work?
>
> -Jason
>
> On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > So how would you have this work if not mutable where interceptors would
> > add headers?
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > I think the point on the mutability of Headers is worth discussing a
> little
> > more. As far as I can tell, once the ProducerRecord (or ConsumerRecord)
> is
> > constructed, there should be no need to further change the headers. Is
> that
> > correct? If so, then why not enforce that that is the case through the
> API?
> > One problem with mutability it that it constrains the implementation of
> > Headers. For example, if we were backing with a byte slice, would we
> recopy
> > the bytes if a header is added or would we maintain a satellite
> collection
> > of added records. Seems not great either way. If we really think
> mutability
> > is needed, perhaps we could add a method to headers to convert it to a
> > mutable type (e.g. a Headers.toMap)?
> >
> > I'm also with Ismael about exposing Headers.get(). I thought it might
> make
> > sense to have a method like this instead:
> >
> > Iterable<Header> findMatching(Pattern pattern);
> >
> > This makes the (potential) need to scan the headers clear in the API. I'd
> > also be fine exposing no getter at all. In general, Ï think it's good to
> > start with a minimalistic API and work from there since it's always
> easier
> > to add methods than remove them.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > >
> > > Hi Ismael
> > >
> > > On point 1,
> > >
> > > Sure makes sense will update shortly.
> > >
> > > On point 2,
> > >
> > > Setter/getter typical to properties/headers api’s traditionally are map
> > > styled interfaces and what I believe is most expected styled thus the
> > Key,
> > > Value setter.
> > > Also it would mean rather than an interface, we would be making our
> > > internal header impl object we have for the array, exposed. E.g. if we
> > had
> > > a Map really this would be Map.Entry interface, this is the same
> reasons
> > on
> > > the map interface I cannot actually make the underlying Node object
> > that’s
> > > the implementation for HashMap, so that internals can be changed.
> > >
> > >
> > > On point 3,
> > >
> > > I think it people do expect it to be performant, thus why originally
> > > concern I raised with using an array for to me is an early memory
> > > optimisation. I think the user experience of properties/headers is on a
> > > get/set model. This is why its important we have encapsulated logic
> that
> > > then allows us to change this to a map, if this becomes and issue, and
> > the
> > > memory overhead of hashmap is less so.
> > >
> > >
> > >
> > >
> > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <
> > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > >
> > > Hi all,
> > >
> > > Great to see the progress that has been achieved on this one. :) A
> > few
> > > comments regarding the APIs (I'm still reviewing the message format
> > > changes):
> > >
> > > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord`
> should
> > be
> > > named `headers` (we avoid the `get` prefix in Kafka)
> > >
> > > 2. The `Headers` class is mutable (there's an `add` method). Does
> it
> > > need
> > > to be? If so, it would be good to explain why. Related to that, we
> > > should
> > > also explain the thinking around thread-safety. If we keep the
> `add`
> > > method, it may make sense for it to take a `Header` (that way we
> can
> > > add
> > > things to `Header` without changing the interface).
> > >
> > > 3. Do we need the `Headers.get()` method? People usually assume
> that
> > > `get`
> > > would be efficient, but depending on the implementation (the
> current
> > > proposal states that an array would be used), it may not be. If we
> > > expect
> > > the number of headers to be small, it doesn't matter though.
> > >
> > > Ismael
> > >
> > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Have converted the interface/api bullets into interface code
> > > snippets.
> > > >
> > > > Agreed implementation won’t take too long. We have early versions
> > > already.
> > > > Maybe a week before you think about merging I would assume it
> would
> > > be more
> > > > stabilised? I was thinking then we could fork from your confluent
> > > branch,
> > > > making and then holding KIP-82 changes in a patch file, that we
> can
> > > then
> > > > re-fork from apache once KIP98 final is merged, and apply patch
> > with
> > > last
> > > > minute changes.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > >
> > > > On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > > >
> > > > Hey Michael,
> > > >
> > > > Awesome. I have a minor request. The APIs are currently
> > > documented as a
> > > > wiki list. Would you mind adding a code snippet instead?
> It's a
> > > bit
> > > > easier
> > > > to process.
> > > >
> > > > How will be best to manage this, as we will obviously build
> off
> > > your
> > > > KIP’s
> > > > > protocol changes, to avoid a merge hell, should we branch
> > from
> > > your
> > > > branch
> > > > > in the confluent repo or is it worth having a KIP-98
> special
> > > branch
> > > > in the
> > > > > apache git, that we can branch/fork from?
> > > >
> > > >
> > > > I was thinking about this also. Ideally we'd like to get the
> > > changes
> > > > in as
> > > > close together as possible since we only want one magic bump
> > and
> > > some
> > > > users
> > > > deploy trunk. The level of effort to change the format for
> > > headers
> > > > seems
> > > > not too high. Do you agree? My guess is that the KIP-98
> message
> > > format
> > > > patch will take 2-3 weeks to review before we merge to trunk,
> > so
> > > you
> > > > could
> > > > hold off implementing until that patch has somewhat
> stabilized.
> > > That
> > > > would
> > > > save some potential rebase pain.
> > > >
> > > > -Jason
> > > >
> > > >
> > > > The information contained in this email is strictly confidential
> > and
> > > for
> > > > the use of the addressee only, unless otherwise indicated. If you
> > > are not
> > > > the intended recipient, please do not read, copy, use or disclose
> > to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete
> > the
> > > > email and any copies of it. Opinions, conclusion (etc) that do
> not
> > > relate
> > > > to the official business of this company shall be understood as
> > > neither
> > > > given nor endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > (a
> > > > company registered in England and Wales, company number 04008957)
> > > and IG
> > > > Index Limited (a company registered in England and Wales, company
> > > number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Lazy init can achieve/avoid that.
Re the concat, why don’t we implement that inside the Headers rather than causing everyone to implement this as adding headers in interceptors will be a dominant use case. We want a user friendly API. Having as a user having to code this instead of having the headers handle this for me seems redundant.
On 22/02/2017, 22:34, "Jason Gustafson" <ja...@confluent.io> wrote:
I thought the argument was against creating the extra objects unnecessarily
(i.e. if they were not accessed). And note that making the Headers
immutable doesn't necessarily mean that they need to be copied: you can do
a trick like Guava's Iterables.concat to add additional headers without
changing the underlying collections.
-Jason
On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <Mi...@ig.com>
wrote:
> If the argument for not having a map holding the key, value pairs is due
> to garbage creation of HashMap entry's, forcing the creation of a whole new
> producer record to simply add a head, surely is creating a-lot more?
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Wednesday, February 22, 2017 10:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The current producer interceptor API is this:
>
> ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
>
> So adding a header means creating a new ProducerRecord with a new header
> added to the current headers and returning it. Would that not work?
>
> -Jason
>
> On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > So how would you have this work if not mutable where interceptors would
> > add headers?
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > I think the point on the mutability of Headers is worth discussing a
> little
> > more. As far as I can tell, once the ProducerRecord (or ConsumerRecord)
> is
> > constructed, there should be no need to further change the headers. Is
> that
> > correct? If so, then why not enforce that that is the case through the
> API?
> > One problem with mutability it that it constrains the implementation of
> > Headers. For example, if we were backing with a byte slice, would we
> recopy
> > the bytes if a header is added or would we maintain a satellite
> collection
> > of added records. Seems not great either way. If we really think
> mutability
> > is needed, perhaps we could add a method to headers to convert it to a
> > mutable type (e.g. a Headers.toMap)?
> >
> > I'm also with Ismael about exposing Headers.get(). I thought it might
> make
> > sense to have a method like this instead:
> >
> > Iterable<Header> findMatching(Pattern pattern);
> >
> > This makes the (potential) need to scan the headers clear in the API. I'd
> > also be fine exposing no getter at all. In general, Ï think it's good to
> > start with a minimalistic API and work from there since it's always
> easier
> > to add methods than remove them.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > >
> > > Hi Ismael
> > >
> > > On point 1,
> > >
> > > Sure makes sense will update shortly.
> > >
> > > On point 2,
> > >
> > > Setter/getter typical to properties/headers api’s traditionally are map
> > > styled interfaces and what I believe is most expected styled thus the
> > Key,
> > > Value setter.
> > > Also it would mean rather than an interface, we would be making our
> > > internal header impl object we have for the array, exposed. E.g. if we
> > had
> > > a Map really this would be Map.Entry interface, this is the same
> reasons
> > on
> > > the map interface I cannot actually make the underlying Node object
> > that’s
> > > the implementation for HashMap, so that internals can be changed.
> > >
> > >
> > > On point 3,
> > >
> > > I think it people do expect it to be performant, thus why originally
> > > concern I raised with using an array for to me is an early memory
> > > optimisation. I think the user experience of properties/headers is on a
> > > get/set model. This is why its important we have encapsulated logic
> that
> > > then allows us to change this to a map, if this becomes and issue, and
> > the
> > > memory overhead of hashmap is less so.
> > >
> > >
> > >
> > >
> > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <
> > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > >
> > > Hi all,
> > >
> > > Great to see the progress that has been achieved on this one. :) A
> > few
> > > comments regarding the APIs (I'm still reviewing the message format
> > > changes):
> > >
> > > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord`
> should
> > be
> > > named `headers` (we avoid the `get` prefix in Kafka)
> > >
> > > 2. The `Headers` class is mutable (there's an `add` method). Does
> it
> > > need
> > > to be? If so, it would be good to explain why. Related to that, we
> > > should
> > > also explain the thinking around thread-safety. If we keep the
> `add`
> > > method, it may make sense for it to take a `Header` (that way we
> can
> > > add
> > > things to `Header` without changing the interface).
> > >
> > > 3. Do we need the `Headers.get()` method? People usually assume
> that
> > > `get`
> > > would be efficient, but depending on the implementation (the
> current
> > > proposal states that an array would be used), it may not be. If we
> > > expect
> > > the number of headers to be small, it doesn't matter though.
> > >
> > > Ismael
> > >
> > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Have converted the interface/api bullets into interface code
> > > snippets.
> > > >
> > > > Agreed implementation won’t take too long. We have early versions
> > > already.
> > > > Maybe a week before you think about merging I would assume it
> would
> > > be more
> > > > stabilised? I was thinking then we could fork from your confluent
> > > branch,
> > > > making and then holding KIP-82 changes in a patch file, that we
> can
> > > then
> > > > re-fork from apache once KIP98 final is merged, and apply patch
> > with
> > > last
> > > > minute changes.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > >
> > > > On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > > >
> > > > Hey Michael,
> > > >
> > > > Awesome. I have a minor request. The APIs are currently
> > > documented as a
> > > > wiki list. Would you mind adding a code snippet instead?
> It's a
> > > bit
> > > > easier
> > > > to process.
> > > >
> > > > How will be best to manage this, as we will obviously build
> off
> > > your
> > > > KIP’s
> > > > > protocol changes, to avoid a merge hell, should we branch
> > from
> > > your
> > > > branch
> > > > > in the confluent repo or is it worth having a KIP-98
> special
> > > branch
> > > > in the
> > > > > apache git, that we can branch/fork from?
> > > >
> > > >
> > > > I was thinking about this also. Ideally we'd like to get the
> > > changes
> > > > in as
> > > > close together as possible since we only want one magic bump
> > and
> > > some
> > > > users
> > > > deploy trunk. The level of effort to change the format for
> > > headers
> > > > seems
> > > > not too high. Do you agree? My guess is that the KIP-98
> message
> > > format
> > > > patch will take 2-3 weeks to review before we merge to trunk,
> > so
> > > you
> > > > could
> > > > hold off implementing until that patch has somewhat
> stabilized.
> > > That
> > > > would
> > > > save some potential rebase pain.
> > > >
> > > > -Jason
> > > >
> > > >
> > > > The information contained in this email is strictly confidential
> > and
> > > for
> > > > the use of the addressee only, unless otherwise indicated. If you
> > > are not
> > > > the intended recipient, please do not read, copy, use or disclose
> > to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete
> > the
> > > > email and any copies of it. Opinions, conclusion (etc) that do
> not
> > > relate
> > > > to the official business of this company shall be understood as
> > > neither
> > > > given nor endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > (a
> > > > company registered in England and Wales, company number 04008957)
> > > and IG
> > > > Index Limited (a company registered in England and Wales, company
> > > number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
I thought the argument was against creating the extra objects unnecessarily
(i.e. if they were not accessed). And note that making the Headers
immutable doesn't necessarily mean that they need to be copied: you can do
a trick like Guava's Iterables.concat to add additional headers without
changing the underlying collections.
-Jason
On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <Mi...@ig.com>
wrote:
> If the argument for not having a map holding the key, value pairs is due
> to garbage creation of HashMap entry's, forcing the creation of a whole new
> producer record to simply add a head, surely is creating a-lot more?
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Wednesday, February 22, 2017 10:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> The current producer interceptor API is this:
>
> ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
>
> So adding a header means creating a new ProducerRecord with a new header
> added to the current headers and returning it. Would that not work?
>
> -Jason
>
> On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > So how would you have this work if not mutable where interceptors would
> > add headers?
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Jason Gustafson <ja...@confluent.io>
> > Sent: Wednesday, February 22, 2017 8:42:27 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > I think the point on the mutability of Headers is worth discussing a
> little
> > more. As far as I can tell, once the ProducerRecord (or ConsumerRecord)
> is
> > constructed, there should be no need to further change the headers. Is
> that
> > correct? If so, then why not enforce that that is the case through the
> API?
> > One problem with mutability it that it constrains the implementation of
> > Headers. For example, if we were backing with a byte slice, would we
> recopy
> > the bytes if a header is added or would we maintain a satellite
> collection
> > of added records. Seems not great either way. If we really think
> mutability
> > is needed, perhaps we could add a method to headers to convert it to a
> > mutable type (e.g. a Headers.toMap)?
> >
> > I'm also with Ismael about exposing Headers.get(). I thought it might
> make
> > sense to have a method like this instead:
> >
> > Iterable<Header> findMatching(Pattern pattern);
> >
> > This makes the (potential) need to scan the headers clear in the API. I'd
> > also be fine exposing no getter at all. In general, Ï think it's good to
> > start with a minimalistic API and work from there since it's always
> easier
> > to add methods than remove them.
> >
> > -Jason
> >
> > On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > >
> > > Hi Ismael
> > >
> > > On point 1,
> > >
> > > Sure makes sense will update shortly.
> > >
> > > On point 2,
> > >
> > > Setter/getter typical to properties/headers api’s traditionally are map
> > > styled interfaces and what I believe is most expected styled thus the
> > Key,
> > > Value setter.
> > > Also it would mean rather than an interface, we would be making our
> > > internal header impl object we have for the array, exposed. E.g. if we
> > had
> > > a Map really this would be Map.Entry interface, this is the same
> reasons
> > on
> > > the map interface I cannot actually make the underlying Node object
> > that’s
> > > the implementation for HashMap, so that internals can be changed.
> > >
> > >
> > > On point 3,
> > >
> > > I think it people do expect it to be performant, thus why originally
> > > concern I raised with using an array for to me is an early memory
> > > optimisation. I think the user experience of properties/headers is on a
> > > get/set model. This is why its important we have encapsulated logic
> that
> > > then allows us to change this to a map, if this becomes and issue, and
> > the
> > > memory overhead of hashmap is less so.
> > >
> > >
> > >
> > >
> > > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <
> > > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> > >
> > > Hi all,
> > >
> > > Great to see the progress that has been achieved on this one. :) A
> > few
> > > comments regarding the APIs (I'm still reviewing the message format
> > > changes):
> > >
> > > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord`
> should
> > be
> > > named `headers` (we avoid the `get` prefix in Kafka)
> > >
> > > 2. The `Headers` class is mutable (there's an `add` method). Does
> it
> > > need
> > > to be? If so, it would be good to explain why. Related to that, we
> > > should
> > > also explain the thinking around thread-safety. If we keep the
> `add`
> > > method, it may make sense for it to take a `Header` (that way we
> can
> > > add
> > > things to `Header` without changing the interface).
> > >
> > > 3. Do we need the `Headers.get()` method? People usually assume
> that
> > > `get`
> > > would be efficient, but depending on the implementation (the
> current
> > > proposal states that an array would be used), it may not be. If we
> > > expect
> > > the number of headers to be small, it doesn't matter though.
> > >
> > > Ismael
> > >
> > > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> > Michael.Pearce@ig.com
> > > >
> > > wrote:
> > >
> > > > Hi Jason,
> > > >
> > > > Have converted the interface/api bullets into interface code
> > > snippets.
> > > >
> > > > Agreed implementation won’t take too long. We have early versions
> > > already.
> > > > Maybe a week before you think about merging I would assume it
> would
> > > be more
> > > > stabilised? I was thinking then we could fork from your confluent
> > > branch,
> > > > making and then holding KIP-82 changes in a patch file, that we
> can
> > > then
> > > > re-fork from apache once KIP98 final is merged, and apply patch
> > with
> > > last
> > > > minute changes.
> > > >
> > > > Cheers
> > > > Mike
> > > >
> > > >
> > > > On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > > >
> > > > Hey Michael,
> > > >
> > > > Awesome. I have a minor request. The APIs are currently
> > > documented as a
> > > > wiki list. Would you mind adding a code snippet instead?
> It's a
> > > bit
> > > > easier
> > > > to process.
> > > >
> > > > How will be best to manage this, as we will obviously build
> off
> > > your
> > > > KIP’s
> > > > > protocol changes, to avoid a merge hell, should we branch
> > from
> > > your
> > > > branch
> > > > > in the confluent repo or is it worth having a KIP-98
> special
> > > branch
> > > > in the
> > > > > apache git, that we can branch/fork from?
> > > >
> > > >
> > > > I was thinking about this also. Ideally we'd like to get the
> > > changes
> > > > in as
> > > > close together as possible since we only want one magic bump
> > and
> > > some
> > > > users
> > > > deploy trunk. The level of effort to change the format for
> > > headers
> > > > seems
> > > > not too high. Do you agree? My guess is that the KIP-98
> message
> > > format
> > > > patch will take 2-3 weeks to review before we merge to trunk,
> > so
> > > you
> > > > could
> > > > hold off implementing until that patch has somewhat
> stabilized.
> > > That
> > > > would
> > > > save some potential rebase pain.
> > > >
> > > > -Jason
> > > >
> > > >
> > > > The information contained in this email is strictly confidential
> > and
> > > for
> > > > the use of the addressee only, unless otherwise indicated. If you
> > > are not
> > > > the intended recipient, please do not read, copy, use or disclose
> > to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then delete
> > the
> > > > email and any copies of it. Opinions, conclusion (etc) that do
> not
> > > relate
> > > > to the official business of this company shall be understood as
> > > neither
> > > > given nor endorsed by it. IG is a trading name of IG Markets
> > Limited
> > > (a
> > > > company registered in England and Wales, company number 04008957)
> > > and IG
> > > > Index Limited (a company registered in England and Wales, company
> > > number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > > and IG
> > > > Index Limited (register number 114059) are authorised and
> regulated
> > > by the
> > > > Financial Conduct Authority.
> > > >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
If the argument for not having a map holding the key, value pairs is due to garbage creation of HashMap entry's, forcing the creation of a whole new producer record to simply add a head, surely is creating a-lot more?
________________________________________
From: Jason Gustafson <ja...@confluent.io>
Sent: Wednesday, February 22, 2017 10:09 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
The current producer interceptor API is this:
ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
So adding a header means creating a new ProducerRecord with a new header
added to the current headers and returning it. Would that not work?
-Jason
On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <Mi...@ig.com>
wrote:
> So how would you have this work if not mutable where interceptors would
> add headers?
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Wednesday, February 22, 2017 8:42:27 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> I think the point on the mutability of Headers is worth discussing a little
> more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
> constructed, there should be no need to further change the headers. Is that
> correct? If so, then why not enforce that that is the case through the API?
> One problem with mutability it that it constrains the implementation of
> Headers. For example, if we were backing with a byte slice, would we recopy
> the bytes if a header is added or would we maintain a satellite collection
> of added records. Seems not great either way. If we really think mutability
> is needed, perhaps we could add a method to headers to convert it to a
> mutable type (e.g. a Headers.toMap)?
>
> I'm also with Ismael about exposing Headers.get(). I thought it might make
> sense to have a method like this instead:
>
> Iterable<Header> findMatching(Pattern pattern);
>
> This makes the (potential) need to scan the headers clear in the API. I'd
> also be fine exposing no getter at all. In general, Ï think it's good to
> start with a minimalistic API and work from there since it's always easier
> to add methods than remove them.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> >
> > Hi Ismael
> >
> > On point 1,
> >
> > Sure makes sense will update shortly.
> >
> > On point 2,
> >
> > Setter/getter typical to properties/headers api’s traditionally are map
> > styled interfaces and what I believe is most expected styled thus the
> Key,
> > Value setter.
> > Also it would mean rather than an interface, we would be making our
> > internal header impl object we have for the array, exposed. E.g. if we
> had
> > a Map really this would be Map.Entry interface, this is the same reasons
> on
> > the map interface I cannot actually make the underlying Node object
> that’s
> > the implementation for HashMap, so that internals can be changed.
> >
> >
> > On point 3,
> >
> > I think it people do expect it to be performant, thus why originally
> > concern I raised with using an array for to me is an early memory
> > optimisation. I think the user experience of properties/headers is on a
> > get/set model. This is why its important we have encapsulated logic that
> > then allows us to change this to a map, if this becomes and issue, and
> the
> > memory overhead of hashmap is less so.
> >
> >
> >
> >
> > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <
> > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> >
> > Hi all,
> >
> > Great to see the progress that has been achieved on this one. :) A
> few
> > comments regarding the APIs (I'm still reviewing the message format
> > changes):
> >
> > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should
> be
> > named `headers` (we avoid the `get` prefix in Kafka)
> >
> > 2. The `Headers` class is mutable (there's an `add` method). Does it
> > need
> > to be? If so, it would be good to explain why. Related to that, we
> > should
> > also explain the thinking around thread-safety. If we keep the `add`
> > method, it may make sense for it to take a `Header` (that way we can
> > add
> > things to `Header` without changing the interface).
> >
> > 3. Do we need the `Headers.get()` method? People usually assume that
> > `get`
> > would be efficient, but depending on the implementation (the current
> > proposal states that an array would be used), it may not be. If we
> > expect
> > the number of headers to be small, it doesn't matter though.
> >
> > Ismael
> >
> > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Have converted the interface/api bullets into interface code
> > snippets.
> > >
> > > Agreed implementation won’t take too long. We have early versions
> > already.
> > > Maybe a week before you think about merging I would assume it would
> > be more
> > > stabilised? I was thinking then we could fork from your confluent
> > branch,
> > > making and then holding KIP-82 changes in a patch file, that we can
> > then
> > > re-fork from apache once KIP98 final is merged, and apply patch
> with
> > last
> > > minute changes.
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > > On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> > >
> > > Hey Michael,
> > >
> > > Awesome. I have a minor request. The APIs are currently
> > documented as a
> > > wiki list. Would you mind adding a code snippet instead? It's a
> > bit
> > > easier
> > > to process.
> > >
> > > How will be best to manage this, as we will obviously build off
> > your
> > > KIP’s
> > > > protocol changes, to avoid a merge hell, should we branch
> from
> > your
> > > branch
> > > > in the confluent repo or is it worth having a KIP-98 special
> > branch
> > > in the
> > > > apache git, that we can branch/fork from?
> > >
> > >
> > > I was thinking about this also. Ideally we'd like to get the
> > changes
> > > in as
> > > close together as possible since we only want one magic bump
> and
> > some
> > > users
> > > deploy trunk. The level of effort to change the format for
> > headers
> > > seems
> > > not too high. Do you agree? My guess is that the KIP-98 message
> > format
> > > patch will take 2-3 weeks to review before we merge to trunk,
> so
> > you
> > > could
> > > hold off implementing until that patch has somewhat stabilized.
> > That
> > > would
> > > save some potential rebase pain.
> > >
> > > -Jason
> > >
> > >
> > > The information contained in this email is strictly confidential
> and
> > for
> > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> > > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> > > this message or any attachment. Please also notify the sender by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > email and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > to the official business of this company shall be understood as
> > neither
> > > given nor endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > company registered in England and Wales, company number 04008957)
> > and IG
> > > Index Limited (a company registered in England and Wales, company
> > number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and regulated
> > by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
The current producer interceptor API is this:
ProducerRecord<K, V> onSend(ProducerRecord<K, V> record);
So adding a header means creating a new ProducerRecord with a new header
added to the current headers and returning it. Would that not work?
-Jason
On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <Mi...@ig.com>
wrote:
> So how would you have this work if not mutable where interceptors would
> add headers?
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Wednesday, February 22, 2017 8:42:27 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> I think the point on the mutability of Headers is worth discussing a little
> more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
> constructed, there should be no need to further change the headers. Is that
> correct? If so, then why not enforce that that is the case through the API?
> One problem with mutability it that it constrains the implementation of
> Headers. For example, if we were backing with a byte slice, would we recopy
> the bytes if a header is added or would we maintain a satellite collection
> of added records. Seems not great either way. If we really think mutability
> is needed, perhaps we could add a method to headers to convert it to a
> mutable type (e.g. a Headers.toMap)?
>
> I'm also with Ismael about exposing Headers.get(). I thought it might make
> sense to have a method like this instead:
>
> Iterable<Header> findMatching(Pattern pattern);
>
> This makes the (potential) need to scan the headers clear in the API. I'd
> also be fine exposing no getter at all. In general, Ï think it's good to
> start with a minimalistic API and work from there since it's always easier
> to add methods than remove them.
>
> -Jason
>
> On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> >
> > Hi Ismael
> >
> > On point 1,
> >
> > Sure makes sense will update shortly.
> >
> > On point 2,
> >
> > Setter/getter typical to properties/headers api’s traditionally are map
> > styled interfaces and what I believe is most expected styled thus the
> Key,
> > Value setter.
> > Also it would mean rather than an interface, we would be making our
> > internal header impl object we have for the array, exposed. E.g. if we
> had
> > a Map really this would be Map.Entry interface, this is the same reasons
> on
> > the map interface I cannot actually make the underlying Node object
> that’s
> > the implementation for HashMap, so that internals can be changed.
> >
> >
> > On point 3,
> >
> > I think it people do expect it to be performant, thus why originally
> > concern I raised with using an array for to me is an early memory
> > optimisation. I think the user experience of properties/headers is on a
> > get/set model. This is why its important we have encapsulated logic that
> > then allows us to change this to a map, if this becomes and issue, and
> the
> > memory overhead of hashmap is less so.
> >
> >
> >
> >
> > On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <
> > ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
> >
> > Hi all,
> >
> > Great to see the progress that has been achieved on this one. :) A
> few
> > comments regarding the APIs (I'm still reviewing the message format
> > changes):
> >
> > 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should
> be
> > named `headers` (we avoid the `get` prefix in Kafka)
> >
> > 2. The `Headers` class is mutable (there's an `add` method). Does it
> > need
> > to be? If so, it would be good to explain why. Related to that, we
> > should
> > also explain the thinking around thread-safety. If we keep the `add`
> > method, it may make sense for it to take a `Header` (that way we can
> > add
> > things to `Header` without changing the interface).
> >
> > 3. Do we need the `Headers.get()` method? People usually assume that
> > `get`
> > would be efficient, but depending on the implementation (the current
> > proposal states that an array would be used), it may not be. If we
> > expect
> > the number of headers to be small, it doesn't matter though.
> >
> > Ismael
> >
> > On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <
> Michael.Pearce@ig.com
> > >
> > wrote:
> >
> > > Hi Jason,
> > >
> > > Have converted the interface/api bullets into interface code
> > snippets.
> > >
> > > Agreed implementation won’t take too long. We have early versions
> > already.
> > > Maybe a week before you think about merging I would assume it would
> > be more
> > > stabilised? I was thinking then we could fork from your confluent
> > branch,
> > > making and then holding KIP-82 changes in a patch file, that we can
> > then
> > > re-fork from apache once KIP98 final is merged, and apply patch
> with
> > last
> > > minute changes.
> > >
> > > Cheers
> > > Mike
> > >
> > >
> > > On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> > >
> > > Hey Michael,
> > >
> > > Awesome. I have a minor request. The APIs are currently
> > documented as a
> > > wiki list. Would you mind adding a code snippet instead? It's a
> > bit
> > > easier
> > > to process.
> > >
> > > How will be best to manage this, as we will obviously build off
> > your
> > > KIP’s
> > > > protocol changes, to avoid a merge hell, should we branch
> from
> > your
> > > branch
> > > > in the confluent repo or is it worth having a KIP-98 special
> > branch
> > > in the
> > > > apache git, that we can branch/fork from?
> > >
> > >
> > > I was thinking about this also. Ideally we'd like to get the
> > changes
> > > in as
> > > close together as possible since we only want one magic bump
> and
> > some
> > > users
> > > deploy trunk. The level of effort to change the format for
> > headers
> > > seems
> > > not too high. Do you agree? My guess is that the KIP-98 message
> > format
> > > patch will take 2-3 weeks to review before we merge to trunk,
> so
> > you
> > > could
> > > hold off implementing until that patch has somewhat stabilized.
> > That
> > > would
> > > save some potential rebase pain.
> > >
> > > -Jason
> > >
> > >
> > > The information contained in this email is strictly confidential
> and
> > for
> > > the use of the addressee only, unless otherwise indicated. If you
> > are not
> > > the intended recipient, please do not read, copy, use or disclose
> to
> > others
> > > this message or any attachment. Please also notify the sender by
> > replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > > email and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > to the official business of this company shall be understood as
> > neither
> > > given nor endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a
> > > company registered in England and Wales, company number 04008957)
> > and IG
> > > Index Limited (a company registered in England and Wales, company
> > number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> > > Index Limited (register number 114059) are authorised and regulated
> > by the
> > > Financial Conduct Authority.
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
So how would you have this work if not mutable where interceptors would add headers?
Sent using OWA for iPhone
________________________________________
From: Jason Gustafson <ja...@confluent.io>
Sent: Wednesday, February 22, 2017 8:42:27 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
I think the point on the mutability of Headers is worth discussing a little
more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
constructed, there should be no need to further change the headers. Is that
correct? If so, then why not enforce that that is the case through the API?
One problem with mutability it that it constrains the implementation of
Headers. For example, if we were backing with a byte slice, would we recopy
the bytes if a header is added or would we maintain a satellite collection
of added records. Seems not great either way. If we really think mutability
is needed, perhaps we could add a method to headers to convert it to a
mutable type (e.g. a Headers.toMap)?
I'm also with Ismael about exposing Headers.get(). I thought it might make
sense to have a method like this instead:
Iterable<Header> findMatching(Pattern pattern);
This makes the (potential) need to scan the headers clear in the API. I'd
also be fine exposing no getter at all. In general, Ï think it's good to
start with a minimalistic API and work from there since it's always easier
to add methods than remove them.
-Jason
On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <Mi...@ig.com>
wrote:
>
> Hi Ismael
>
> On point 1,
>
> Sure makes sense will update shortly.
>
> On point 2,
>
> Setter/getter typical to properties/headers api’s traditionally are map
> styled interfaces and what I believe is most expected styled thus the Key,
> Value setter.
> Also it would mean rather than an interface, we would be making our
> internal header impl object we have for the array, exposed. E.g. if we had
> a Map really this would be Map.Entry interface, this is the same reasons on
> the map interface I cannot actually make the underlying Node object that’s
> the implementation for HashMap, so that internals can be changed.
>
>
> On point 3,
>
> I think it people do expect it to be performant, thus why originally
> concern I raised with using an array for to me is an early memory
> optimisation. I think the user experience of properties/headers is on a
> get/set model. This is why its important we have encapsulated logic that
> then allows us to change this to a map, if this becomes and issue, and the
> memory overhead of hashmap is less so.
>
>
>
>
> On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <
> ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>
> Hi all,
>
> Great to see the progress that has been achieved on this one. :) A few
> comments regarding the APIs (I'm still reviewing the message format
> changes):
>
> 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
> named `headers` (we avoid the `get` prefix in Kafka)
>
> 2. The `Headers` class is mutable (there's an `add` method). Does it
> need
> to be? If so, it would be good to explain why. Related to that, we
> should
> also explain the thinking around thread-safety. If we keep the `add`
> method, it may make sense for it to take a `Header` (that way we can
> add
> things to `Header` without changing the interface).
>
> 3. Do we need the `Headers.get()` method? People usually assume that
> `get`
> would be efficient, but depending on the implementation (the current
> proposal states that an array would be used), it may not be. If we
> expect
> the number of headers to be small, it doesn't matter though.
>
> Ismael
>
> On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > Hi Jason,
> >
> > Have converted the interface/api bullets into interface code
> snippets.
> >
> > Agreed implementation won’t take too long. We have early versions
> already.
> > Maybe a week before you think about merging I would assume it would
> be more
> > stabilised? I was thinking then we could fork from your confluent
> branch,
> > making and then holding KIP-82 changes in a patch file, that we can
> then
> > re-fork from apache once KIP98 final is merged, and apply patch with
> last
> > minute changes.
> >
> > Cheers
> > Mike
> >
> >
> > On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > Hey Michael,
> >
> > Awesome. I have a minor request. The APIs are currently
> documented as a
> > wiki list. Would you mind adding a code snippet instead? It's a
> bit
> > easier
> > to process.
> >
> > How will be best to manage this, as we will obviously build off
> your
> > KIP’s
> > > protocol changes, to avoid a merge hell, should we branch from
> your
> > branch
> > > in the confluent repo or is it worth having a KIP-98 special
> branch
> > in the
> > > apache git, that we can branch/fork from?
> >
> >
> > I was thinking about this also. Ideally we'd like to get the
> changes
> > in as
> > close together as possible since we only want one magic bump and
> some
> > users
> > deploy trunk. The level of effort to change the format for
> headers
> > seems
> > not too high. Do you agree? My guess is that the KIP-98 message
> format
> > patch will take 2-3 weeks to review before we merge to trunk, so
> you
> > could
> > hold off implementing until that patch has somewhat stabilized.
> That
> > would
> > save some potential rebase pain.
> >
> > -Jason
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the official business of this company shall be understood as
> neither
> > given nor endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > company registered in England and Wales, company number 04008957)
> and IG
> > Index Limited (a company registered in England and Wales, company
> number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
I think the point on the mutability of Headers is worth discussing a little
more. As far as I can tell, once the ProducerRecord (or ConsumerRecord) is
constructed, there should be no need to further change the headers. Is that
correct? If so, then why not enforce that that is the case through the API?
One problem with mutability it that it constrains the implementation of
Headers. For example, if we were backing with a byte slice, would we recopy
the bytes if a header is added or would we maintain a satellite collection
of added records. Seems not great either way. If we really think mutability
is needed, perhaps we could add a method to headers to convert it to a
mutable type (e.g. a Headers.toMap)?
I'm also with Ismael about exposing Headers.get(). I thought it might make
sense to have a method like this instead:
Iterable<Header> findMatching(Pattern pattern);
This makes the (potential) need to scan the headers clear in the API. I'd
also be fine exposing no getter at all. In general, Ï think it's good to
start with a minimalistic API and work from there since it's always easier
to add methods than remove them.
-Jason
On Wed, Feb 22, 2017 at 9:16 AM, Michael Pearce <Mi...@ig.com>
wrote:
>
> Hi Ismael
>
> On point 1,
>
> Sure makes sense will update shortly.
>
> On point 2,
>
> Setter/getter typical to properties/headers api’s traditionally are map
> styled interfaces and what I believe is most expected styled thus the Key,
> Value setter.
> Also it would mean rather than an interface, we would be making our
> internal header impl object we have for the array, exposed. E.g. if we had
> a Map really this would be Map.Entry interface, this is the same reasons on
> the map interface I cannot actually make the underlying Node object that’s
> the implementation for HashMap, so that internals can be changed.
>
>
> On point 3,
>
> I think it people do expect it to be performant, thus why originally
> concern I raised with using an array for to me is an early memory
> optimisation. I think the user experience of properties/headers is on a
> get/set model. This is why its important we have encapsulated logic that
> then allows us to change this to a map, if this becomes and issue, and the
> memory overhead of hashmap is less so.
>
>
>
>
> On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <
> ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
>
> Hi all,
>
> Great to see the progress that has been achieved on this one. :) A few
> comments regarding the APIs (I'm still reviewing the message format
> changes):
>
> 1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
> named `headers` (we avoid the `get` prefix in Kafka)
>
> 2. The `Headers` class is mutable (there's an `add` method). Does it
> need
> to be? If so, it would be good to explain why. Related to that, we
> should
> also explain the thinking around thread-safety. If we keep the `add`
> method, it may make sense for it to take a `Header` (that way we can
> add
> things to `Header` without changing the interface).
>
> 3. Do we need the `Headers.get()` method? People usually assume that
> `get`
> would be efficient, but depending on the implementation (the current
> proposal states that an array would be used), it may not be. If we
> expect
> the number of headers to be small, it doesn't matter though.
>
> Ismael
>
> On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> wrote:
>
> > Hi Jason,
> >
> > Have converted the interface/api bullets into interface code
> snippets.
> >
> > Agreed implementation won’t take too long. We have early versions
> already.
> > Maybe a week before you think about merging I would assume it would
> be more
> > stabilised? I was thinking then we could fork from your confluent
> branch,
> > making and then holding KIP-82 changes in a patch file, that we can
> then
> > re-fork from apache once KIP98 final is merged, and apply patch with
> last
> > minute changes.
> >
> > Cheers
> > Mike
> >
> >
> > On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > Hey Michael,
> >
> > Awesome. I have a minor request. The APIs are currently
> documented as a
> > wiki list. Would you mind adding a code snippet instead? It's a
> bit
> > easier
> > to process.
> >
> > How will be best to manage this, as we will obviously build off
> your
> > KIP’s
> > > protocol changes, to avoid a merge hell, should we branch from
> your
> > branch
> > > in the confluent repo or is it worth having a KIP-98 special
> branch
> > in the
> > > apache git, that we can branch/fork from?
> >
> >
> > I was thinking about this also. Ideally we'd like to get the
> changes
> > in as
> > close together as possible since we only want one magic bump and
> some
> > users
> > deploy trunk. The level of effort to change the format for
> headers
> > seems
> > not too high. Do you agree? My guess is that the KIP-98 message
> format
> > patch will take 2-3 weeks to review before we merge to trunk, so
> you
> > could
> > hold off implementing until that patch has somewhat stabilized.
> That
> > would
> > save some potential rebase pain.
> >
> > -Jason
> >
> >
> > The information contained in this email is strictly confidential and
> for
> > the use of the addressee only, unless otherwise indicated. If you
> are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by
> replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the official business of this company shall be understood as
> neither
> > given nor endorsed by it. IG is a trading name of IG Markets Limited
> (a
> > company registered in England and Wales, company number 04008957)
> and IG
> > Index Limited (a company registered in England and Wales, company
> number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > Index Limited (register number 114059) are authorised and regulated
> by the
> > Financial Conduct Authority.
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Ismael
On point 1,
Sure makes sense will update shortly.
On point 2,
Setter/getter typical to properties/headers api’s traditionally are map styled interfaces and what I believe is most expected styled thus the Key, Value setter.
Also it would mean rather than an interface, we would be making our internal header impl object we have for the array, exposed. E.g. if we had a Map really this would be Map.Entry interface, this is the same reasons on the map interface I cannot actually make the underlying Node object that’s the implementation for HashMap, so that internals can be changed.
On point 3,
I think it people do expect it to be performant, thus why originally concern I raised with using an array for to me is an early memory optimisation. I think the user experience of properties/headers is on a get/set model. This is why its important we have encapsulated logic that then allows us to change this to a map, if this becomes and issue, and the memory overhead of hashmap is less so.
On 22/02/2017, 15:56, "ismaelj@gmail.com on behalf of Ismael Juma" <ismaelj@gmail.com on behalf of ismael@juma.me.uk> wrote:
Hi all,
Great to see the progress that has been achieved on this one. :) A few
comments regarding the APIs (I'm still reviewing the message format
changes):
1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
named `headers` (we avoid the `get` prefix in Kafka)
2. The `Headers` class is mutable (there's an `add` method). Does it need
to be? If so, it would be good to explain why. Related to that, we should
also explain the thinking around thread-safety. If we keep the `add`
method, it may make sense for it to take a `Header` (that way we can add
things to `Header` without changing the interface).
3. Do we need the `Headers.get()` method? People usually assume that `get`
would be efficient, but depending on the implementation (the current
proposal states that an array would be used), it may not be. If we expect
the number of headers to be small, it doesn't matter though.
Ismael
On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jason,
>
> Have converted the interface/api bullets into interface code snippets.
>
> Agreed implementation won’t take too long. We have early versions already.
> Maybe a week before you think about merging I would assume it would be more
> stabilised? I was thinking then we could fork from your confluent branch,
> making and then holding KIP-82 changes in a patch file, that we can then
> re-fork from apache once KIP98 final is merged, and apply patch with last
> minute changes.
>
> Cheers
> Mike
>
>
> On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Michael,
>
> Awesome. I have a minor request. The APIs are currently documented as a
> wiki list. Would you mind adding a code snippet instead? It's a bit
> easier
> to process.
>
> How will be best to manage this, as we will obviously build off your
> KIP’s
> > protocol changes, to avoid a merge hell, should we branch from your
> branch
> > in the confluent repo or is it worth having a KIP-98 special branch
> in the
> > apache git, that we can branch/fork from?
>
>
> I was thinking about this also. Ideally we'd like to get the changes
> in as
> close together as possible since we only want one magic bump and some
> users
> deploy trunk. The level of effort to change the format for headers
> seems
> not too high. Do you agree? My guess is that the KIP-98 message format
> patch will take 2-3 weeks to review before we merge to trunk, so you
> could
> hold off implementing until that patch has somewhat stabilized. That
> would
> save some potential rebase pain.
>
> -Jason
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the
> email and any copies of it. Opinions, conclusion (etc) that do not relate
> to the official business of this company shall be understood as neither
> given nor endorsed by it. IG is a trading name of IG Markets Limited (a
> company registered in England and Wales, company number 04008957) and IG
> Index Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ismael Juma <is...@juma.me.uk>.
Hi all,
Great to see the progress that has been achieved on this one. :) A few
comments regarding the APIs (I'm still reviewing the message format
changes):
1. Nit: `getHeaders` in `ProducerRecord` and `ConsumerRecord` should be
named `headers` (we avoid the `get` prefix in Kafka)
2. The `Headers` class is mutable (there's an `add` method). Does it need
to be? If so, it would be good to explain why. Related to that, we should
also explain the thinking around thread-safety. If we keep the `add`
method, it may make sense for it to take a `Header` (that way we can add
things to `Header` without changing the interface).
3. Do we need the `Headers.get()` method? People usually assume that `get`
would be efficient, but depending on the implementation (the current
proposal states that an array would be used), it may not be. If we expect
the number of headers to be small, it doesn't matter though.
Ismael
On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jason,
>
> Have converted the interface/api bullets into interface code snippets.
>
> Agreed implementation won’t take too long. We have early versions already.
> Maybe a week before you think about merging I would assume it would be more
> stabilised? I was thinking then we could fork from your confluent branch,
> making and then holding KIP-82 changes in a patch file, that we can then
> re-fork from apache once KIP98 final is merged, and apply patch with last
> minute changes.
>
> Cheers
> Mike
>
>
> On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Michael,
>
> Awesome. I have a minor request. The APIs are currently documented as a
> wiki list. Would you mind adding a code snippet instead? It's a bit
> easier
> to process.
>
> How will be best to manage this, as we will obviously build off your
> KIP’s
> > protocol changes, to avoid a merge hell, should we branch from your
> branch
> > in the confluent repo or is it worth having a KIP-98 special branch
> in the
> > apache git, that we can branch/fork from?
>
>
> I was thinking about this also. Ideally we'd like to get the changes
> in as
> close together as possible since we only want one magic bump and some
> users
> deploy trunk. The level of effort to change the format for headers
> seems
> not too high. Do you agree? My guess is that the KIP-98 message format
> patch will take 2-3 weeks to review before we merge to trunk, so you
> could
> hold off implementing until that patch has somewhat stabilized. That
> would
> save some potential rebase pain.
>
> -Jason
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the
> email and any copies of it. Opinions, conclusion (etc) that do not relate
> to the official business of this company shall be understood as neither
> given nor endorsed by it. IG is a trading name of IG Markets Limited (a
> company registered in England and Wales, company number 04008957) and IG
> Index Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Jason,
Have converted the interface/api bullets into interface code snippets.
Agreed implementation won’t take too long. We have early versions already. Maybe a week before you think about merging I would assume it would be more stabilised? I was thinking then we could fork from your confluent branch, making and then holding KIP-82 changes in a patch file, that we can then re-fork from apache once KIP98 final is merged, and apply patch with last minute changes.
Cheers
Mike
On 22/02/2017, 00:56, "Jason Gustafson" <ja...@confluent.io> wrote:
Hey Michael,
Awesome. I have a minor request. The APIs are currently documented as a
wiki list. Would you mind adding a code snippet instead? It's a bit easier
to process.
How will be best to manage this, as we will obviously build off your KIP’s
> protocol changes, to avoid a merge hell, should we branch from your branch
> in the confluent repo or is it worth having a KIP-98 special branch in the
> apache git, that we can branch/fork from?
I was thinking about this also. Ideally we'd like to get the changes in as
close together as possible since we only want one magic bump and some users
deploy trunk. The level of effort to change the format for headers seems
not too high. Do you agree? My guess is that the KIP-98 message format
patch will take 2-3 weeks to review before we merge to trunk, so you could
hold off implementing until that patch has somewhat stabilized. That would
save some potential rebase pain.
-Jason
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Michael,
Awesome. I have a minor request. The APIs are currently documented as a
wiki list. Would you mind adding a code snippet instead? It's a bit easier
to process.
How will be best to manage this, as we will obviously build off your KIP’s
> protocol changes, to avoid a merge hell, should we branch from your branch
> in the confluent repo or is it worth having a KIP-98 special branch in the
> apache git, that we can branch/fork from?
I was thinking about this also. Ideally we'd like to get the changes in as
close together as possible since we only want one magic bump and some users
deploy trunk. The level of effort to change the format for headers seems
not too high. Do you agree? My guess is that the KIP-98 message format
patch will take 2-3 weeks to review before we merge to trunk, so you could
hold off implementing until that patch has somewhat stabilized. That would
save some potential rebase pain.
-Jason
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Great ☺
@Jason I updated this kip-82, with these bits changed, could you kindly check it over (mostly the protocol section using the newer protocol defined in KIP-98 rather than the current existing) and check it makes sense.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers
Anything else needed or wanting to discuss after this? Or we at the +1 from you stage now?
On a side note.
How will be best to manage this, as we will obviously build off your KIP’s protocol changes, to avoid a merge hell, should we branch from your branch in the confluent repo or is it worth having a KIP-98 special branch in the apache git, that we can branch/fork from?
On 22/02/2017, 00:06, "Michael Pearce" <Mi...@ig.com> wrote:
RE KIP-98 discussion,
No worries, can you add it back in on KIP-98, then we go with Headers at the end with this KIP. And I think all are in agreement right?
Cheers
Mike
On 22/02/2017, 00:02, "Michael Pearce" <Mi...@ig.com> wrote:
This is a bit of why I’d rather have headers as self-contained, e.g. it has a headersLen and headers data, as we’d have to introduce one or the other, and it seems cleaner to add the headers than add back in the valuelen.
On 21/02/2017, 23:57, "Michael Pearce" <Mi...@ig.com> wrote:
Hi Jason
I looked at the doc, it seems the length of the following data sections are being left described in your doc in some place similar to how we have on the wiki, and in others not.
E.g. your doc has this as the proposed message described (which is in blue and red colouring)
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
Value => data [OPTIONAL]
Note I notice there is a missing
ValueLen => uintVar [OPTIONAL]
Else we wouldn’t be able to put headers at the end, without its length being defined. So either we can leave ValueLen removed, as is in your new format, and keep headers
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
HeadersLen => uintVar [OPTIONAL]
Headers => data [OPTIONAL]
Value => data [OPTIONAL]
Headers => [Header]
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
OR we will need to re-introduce ValueLen which seems to be removed in Kip-98
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
ValueLen => uintVar [OPTIONAL]
Value => data [OPTIONAL]
Headers => Header1, Header2 … HeaderN
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
Cheers
Mike
On 21/02/2017, 20:45, "ignacio.solis@gmail.com on behalf of Ignacio Solis" <ignacio.solis@gmail.com on behalf of isolis@igso.net> wrote:
Short answer:
Headers at the end are fine, this proposal looks ok.
Longer answer:
At LinkedIn we considered headers at the end multiple times. We even
had a backwards compatible headers-in-V design with headers at the
end.
The main reason headers come at the beginning of messaging systems is
in case they affect the rest of the message. This is either because
of parsing differences (a header might indicate how to parse the
remainder of the message), or performance optimizations (a header
might indicate that you can discard the message and can stop parsing).
I believe that neither of these cases are common in Kafka (at this
time) so having the headers are the end shouldn't be a big issue.
From a performance perspective we may incur an extra page cache miss
but at the rate at which kafka does messaging it won't bottleneck
things.
Headers at the end have one potential advantage, that in simple
circumstances, naive systems can append headers to a message without
doing a lot of memory manipulation. Normally not that big of a deal
since we could always use scatter/gather methods if needed, but maybe
some simple systems will benefit from that. ... Not to mention most
Java programmers might not pay attention to these type of performance
issues.
Anyway...
Overall, there are many things that I would have done differently in
the proposal (int keys, total header length, no VarInts, etc. ), but
the cost of these is small in comparison to the benefit we get from
having headers at all.
If we are all ok with this design then I would be the first to +1 the format.
Nacho
Side note: Shouldn't headers at the end be called footers? We could
always call it metadata :-).
On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com> wrote:
> I think I'd like some opinions from some of the active chaps at LinkedIn so it's not just a two way decision (or any others for that matter).
>
> Any opinions on the below or no real concerns / indifferent by anyone with this change?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Tuesday, February 21, 2017 6:59:38 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Jun's suggestion gives us the best of both worlds. Because the headers are
> the last field in the record and you have the total size of the record,
> then you can easily get the size of headers in bytes. So they can be copied
> without parsing and we can do the lazy initialization without an extra cost.
>
> The message set notion has changed in KIP-98 so that the message set can
> have its own fields. See here for more detail:
> https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
> What we're proposing to change here is the schema for the message (aka
> record) itself. We'll update that document depending on the outcome of the
> discussion here.
>
> -Jason
>
>
> On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> This works whilst no other feature needs similar. I prefer to keep it
>> contained/encapsulated it then allows the bytes of the headers to also be
>> copied transferred as one without any parsing what so ever, but if you
>> really feel strongly on the below ill go with it, just want to flag this
>> concern.
>>
>> Btw, the below notation seems to be different to the notation used in
>> previous kips like kip-32 which I took the notation from, where the length
>> of the following bytes for byte arrays is clearly defined. So im 100% on
>> what you propose could you define it in this original styled notation?
>>
>>
>> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>>
>> Hey Jun, that's a good thought. Maybe then the full schema could look
>> like
>> this:
>>
>> Record =>
>> Length => uintVar
>> Attributes => int8
>> TimestampDelta => intVar
>> OffsetDelta => uintVar
>> Key => bytes
>> Value => bytes
>> Headers => [HeaderKey HeaderValue]
>> HeaderKey => string
>> HeaderValue => bytes
>>
>> (where the bytes and string type use a variable length size field)
>>
>> Does that work for you, Michael?
>>
>> -Jason
>>
>> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > Regarding the overhead of skipping the headers in the message.
>> Another way
>> > could be to add the headers at the end of the message after the
>> value part.
>> > This way, there is no need to skip the headers unless you really
>> need it.
>> > This will allow us to represent the headers explicitly as an array of
>> > key/value pairs without the skipping overhead when headers are not
>> needed.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
>> >
>> > wrote:
>> >
>> > > Michael,
>> > >
>> > > Here's a link to the branch with the message format changes:
>> > > https://github.com/confluentinc/kafka/tree/
>> exactly-once-message-format.
>> > > The
>> > > class to modify to add headers is EosLogRecord (name will
>> definitely
>> > change
>> > > prior to merging in trunk). This branch uses attributes to
>> indicate null
>> > > key and value. I really don't have a strong preference for or
>> against
>> > this
>> > > as long as we're consistent. I do feel it's kind of a pity to need
>> the
>> > > signed version of the varint just to be able to indicate null. For
>> > headers,
>> > > it's an easier call because we don't actually need to represent
>> null
>> > > headers (using size zero will work fine).
>> > >
>> > > -Jason
>> > >
>> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > We can make it, implement Iterable<Header>, no concerns on this.
>> > > >
>> > > > Ive noted this on the KIP.
>> > > >
>> > > >
>> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > > Btw Jason, can you share the branch with the VarInts
>> helper/util
>> > > class
>> > > > that KIP-98 will be introducing and also will have the current
>> modified
>> > > > (key and value lengths) message, so we can look at, thanks.
>> > > >
>> > > > On 20/02/2017, 01:11, "Michael Pearce" <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > > >
>> > > > RE “ProducerRecord and ConsumerRecord are immutable
>> > currently”. I
>> > > > beg to differ on this, if I set a byte[] as my value of my
>> record, I
>> > can
>> > > > later call value() and then alter the bytes in the array. As
>> such it is
>> > > > very much mutable.
>> > > >
>> > > > Headers also will anyhow by nature be mutable on
>> produce, as
>> > you
>> > > > add to them.
>> > > >
>> > > >
>> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
>> jason@confluent.io>
>> > > > wrote:
>> > > >
>> > > > Re; having a dedicated Headers object. I think it
>> makes
>> > > sense.
>> > > > Maybe it
>> > > > could implement Iterable<Header>?
>> > > >
>> > > > One thing I'm not too sure about is mutability. Both
>> the
>> > > > ProducerRecord and
>> > > > ConsumerRecord are immutable currently. It would be
>> nice to
>> > > > keep it that
>> > > > way. Perhaps Headers could itself be immutable and
>> passed
>> > > > through the
>> > > > constructors?
>> > > >
>> > > > Just going through how this is, I don’t believe
>> this would
>> > > > provide much
>> > > > > saving. If issue trying to resolve is around
>> > object/garbage
>> > > > creation of
>> > > > > Map.Entry objects.
>> > > >
>> > > >
>> > > > It's definitely better because we only allocate the
>> space
>> > we
>> > > > will use, but
>> > > > I assume most of the actual memory cost is in the
>> header
>> > data
>> > > > itself which
>> > > > we can't do much about. Maybe we can start with the
>> simple
>> > > > array
>> > > > representation and optimize from there after we find
>> it's
>> > > > worthwhile to do
>> > > > so? If we use the Headers object, we can change this
>> later
>> > > > without breaking
>> > > > the API.
>> > > >
>> > > > -Jason
>> > > >
>> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
>> > > > radai.rosenblatt@gmail.com> wrote:
>> > > >
>> > > > > "(By the way, doesn't it feel a bit odd that we
>> seem to
>> > be
>> > > > designing a
>> > > > > feature which is optimized for people not using?)"
>> > > > >
>> > > > > very. this (i claim :-) ) is the result of very
>> intense
>> > > > opposition to the
>> > > > > usefulness of the feature early on, and not a real
>> design
>> > > > goal
>> > > > >
>> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Whilst if having an array of a Header[] would
>> mean
>> > > getting
>> > > > this through,
>> > > > > > im happy changing this, as already done.
>> > > > > >
>> > > > > > Just going through how this is, I don’t believe
>> this
>> > > > would provide much
>> > > > > > saving. If issue trying to resolve is around
>> > > > object/garbage creation of
>> > > > > > Map.Entry objects.
>> > > > > >
>> > > > > > All you’re doing here is replacing the equiv of a
>> > > > HashMap.Node
>> > > > > (Map.Entry)
>> > > > > > which simply holds reference to key and value
>> objects
>> > > with
>> > > > a custom
>> > > > > variant.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com> wrote:
>> > > > > >
>> > > > > > On point 1 & 2 Ive updated KIP to show
>> varints (and
>> > > > removed the bit
>> > > > > > flag). (on the assumption KIP 98 is getting the
>> > agreement
>> > > > the protocol is
>> > > > > > moving from int32 to varInts as standard)
>> > > > > >
>> > > > > > On point 3 ive updated to use an array of
>> Header
>> > > > class, instead of a
>> > > > > > MultiMap in the Headers class object
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > On points 1 and 2 I agree.
>> > > > > >
>> > > > > > This also affects kip-98, I should
>> expect this
>> > > > resolved before
>> > > > > > that vote also passes. If it is accepted there
>> (I’m
>> > > > assuming this is
>> > > > > > getting discussed on that KIP? As you’re driving
>> the
>> > move
>> > > > to VarInts), I
>> > > > > am
>> > > > > > happy to make this KIP will simply follow suit to
>> > > whatever
>> > > > is agreed in
>> > > > > > KIP-98.
>> > > > > >
>> > > > > > On 3) Agreed this is a simpler form, as
>> long as
>> > > no
>> > > > one is
>> > > > > > expecting hashmap lookup performance ( O(1) ), I
>> still
>> > > > would prefer an
>> > > > > > encapsulated class, so if we find that holding
>> it as an
>> > > > array in future
>> > > > > is
>> > > > > > causing some perf issues, the internals are not
>> exposed
>> > > to
>> > > > end users,
>> > > > > > allowing the internal structure to move to a map.
>> > > > > >
>> > > > > > Can we compromise on?
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Headers headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Headers {
>> > > > > > Header[] headers;
>> > > > > >
>> > > > > > add(String key, byte[] value)
>> > > > > > Collection<byte[]> get(String key)
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
>> > > > jason@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > >
>> > > > > >
>> > > > > > This is exactly what I have been
>> trying to
>> > > get
>> > > > at. The use
>> > > > > > cases documented
>> > > > > > here are middleware:
>> > > > > > https://cwiki.apache.org/
>> > > > confluence/display/KAFKA/A+
>> > > > > > Case+for+Kafka+Headers.
>> > > > > > If headers are intended for the
>> application
>> > > > space as well,
>> > > > > the
>> > > > > > document
>> > > > > > should be updated accordingly so
>> that it is
>> > > an
>> > > > explicit
>> > > > > design
>> > > > > > goal and not
>> > > > > > an unfortunate design byproduct.
>> There may
>> > be
>> > > > some
>> > > > > > disagreement on whether
>> > > > > > it _should_ be a design goal, but it
>> may
>> > not
>> > > > matter much
>> > > > > since
>> > > > > > the current
>> > > > > > interceptor API is probably
>> insufficient
>> > for
>> > > > middleware
>> > > > > > applications (I
>> > > > > > haven't thought of a way to do this
>> that
>> > > isn't
>> > > > cumbersome).
>> > > > > >
>> > > > > > That aside, the three unresolved
>> points for
>> > > me
>> > > > are the
>> > > > > > following:
>> > > > > >
>> > > > > > 1. The use of varints. As a one-off
>> for use
>> > > > only with record
>> > > > > > headers, the
>> > > > > > case was weak, but if we are using
>> them
>> > > > throughout the
>> > > > > message
>> > > > > > format, then
>> > > > > > we should do so here as well. The
>> > additional
>> > > > complexity is
>> > > > > > minimal and
>> > > > > > early performance testing fully
>> justifies
>> > > > their use.
>> > > > > >
>> > > > > > 2. If varints are used, the case for
>> using
>> > > > attributes to
>> > > > > > indicate null
>> > > > > > headers also becomes weak. We only
>> add one
>> > > > additional byte in
>> > > > > > each message
>> > > > > > if there are no headers. Whatever
>> the case,
>> > > > let us be
>> > > > > > consistent with how
>> > > > > > we treat null keys and values.
>> > > > > >
>> > > > > > 3. We have apparently agreed that the
>> > broker
>> > > > will validate
>> > > > > > headers (please
>> > > > > > add this to the KIP). That being the
>> case,
>> > I
>> > > > would prefer to
>> > > > > > use Kafka's
>> > > > > > conventional format for arrays. The
>> > downside
>> > > > is that it takes
>> > > > > > more work to
>> > > > > > skip over the headers on the
>> consumer,
>> > though
>> > > > it's unclear if
>> > > > > > the cost
>> > > > > > would matter in practice. Some
>> concern was
>> > > > previously
>> > > > > > expressed about the
>> > > > > > allocation of maps. An alternative
>> would be
>> > > to
>> > > > use arrays,
>> > > > > i.e.
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Header[] headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > > This would work nicely with the
>> > conventional
>> > > > array format and
>> > > > > > my guess is
>> > > > > > it would obviate the need do any lazy
>> > > > initialization. If we
>> > > > > > use the map as
>> > > > > > is currently documented, then it is
>> > possible
>> > > > with either
>> > > > > > representation to
>> > > > > > slice the headers and initialize them
>> > lazily.
>> > > > Either way, it
>> > > > > > might be a
>> > > > > > good idea to use a separate object to
>> > > > represent the headers
>> > > > > in
>> > > > > > case we need
>> > > > > > to modify it in the future in some
>> way.
>> > > > > >
>> > > > > > (By the way, doesn't it feel a bit
>> odd that
>> > > we
>> > > > seem to be
>> > > > > > designing a
>> > > > > > feature which is optimized for
>> people not
>> > > > using?)
>> > > > > >
>> > > > > >
>> > > > > > If we can resolve these points, then
>> at
>> > least
>> > > > you will get my
>> > > > > > vote.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jason
>> > > > > >
>> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
>> radai <
>> > > > > > radai.rosenblatt@gmail.com> wrote:
>> > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > > > IIUC samze currently has headers
>> "in-V"
>> > and
>> > > > would just
>> > > > > > switch over to kafka
>> > > > > > > headers if they exist.
>> > > > > > > im sure plenty of other users of
>> kafka
>> > > would
>> > > > have a use for
>> > > > > > headers.
>> > > > > > > im pretty sure use cases exist
>> around
>> > > > shuffling data
>> > > > > > into/out-of kafka
>> > > > > > > (kafka connect or equivalent) where
>> > > metadata
>> > > > from one end
>> > > > > > could copied over
>> > > > > > > to the other (S3, for example uses
>> http
>> > > > headers for
>> > > > > > user-accessible
>> > > > > > > metadata). it will be kafka client
>> code
>> > > > getting/setting
>> > > > > > those headers. not
>> > > > > > > an interceptor.
>> > > > > > >
>> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
>> Michael
>> > > > Pearce <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > For APM single event tracing,
>> need
>> > access
>> > > > to the header
>> > > > > at
>> > > > > > the point of
>> > > > > > > > processing on the processing
>> thread.
>> > > > > > > >
>> > > > > > > > As such interceptors will not
>> work/be
>> > > > suitable for these,
>> > > > > > due to the fact
>> > > > > > > > they act on the ConsumerRecords
>> as a
>> > > > batch, before the
>> > > > > > handling thread
>> > > > > > > can
>> > > > > > > > split out and process per
>> message which
>> > > is
>> > > > the point
>> > > > > these
>> > > > > > tools will
>> > > > > > > need
>> > > > > > > > to continue to transaction
>> tracing.
>> > > > > > > >
>> > > > > > > > Like wise tools and infra pieces
>> will
>> > > need
>> > > > access to the
>> > > > > > message outside
>> > > > > > > > the interceptor realm.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 17/02/2017, 21:26, "Jason
>> > Gustafson" <
>> > > > > > jason@confluent.io> wrote:
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > So the broker will parse the
>> > headers
>> > > > and validate
>> > > > > > them. Good. The
>> > > > > > > only
>> > > > > > > > reason remaining that I can
>> see to
>> > > > leave the headers
>> > > > > > as a byte array
>> > > > > > > > is to
>> > > > > > > > make it easier for the
>> client to
>> > skip
>> > > > past them. Are
>> > > > > > we sure this is
>> > > > > > > > not
>> > > > > > > > premature optimization? Are
>> there
>> > any
>> > > > performance
>> > > > > > results which show
>> > > > > > > > that
>> > > > > > > > this is worthwhile?
>> > > > > > > >
>> > > > > > > > What’s the issue with
>> exposing a
>> > > > method getHeaders on
>> > > > > > the
>> > > > > > > > producer/consumer
>> > > > > > > > > record? It doesn’t break
>> > anything.
>> > > > We don’t need
>> > > > > any
>> > > > > > special
>> > > > > > > version.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > See my previous explanation.
>> What I
>> > > am
>> > > > trying to
>> > > > > > resist is the
>> > > > > > > headers
>> > > > > > > > becoming a general
>> > application-level
>> > > > facility. The
>> > > > > > primary use case
>> > > > > > > as
>> > > > > > > > far
>> > > > > > > > as I can tell is middleware,
>> which
>> > is
>> > > > the use case
>> > > > > > that the
>> > > > > > > > interceptors
>> > > > > > > > are providing.
>> > > > > > > >
>> > > > > > > > Current batch consumer model
>> and
>> > > > consumer
>> > > > > interceptors
>> > > > > > don’t work
>> > > > > > > where
>> > > > > > > > > headers need to be acted
>> on at
>> > per
>> > > > message level at
>> > > > > > time of
>> > > > > > > > processing,
>> > > > > > > > > very case is APM (the core
>> one),
>> > > > where the header
>> > > > > > value is used to
>> > > > > > > > continue
>> > > > > > > > > tracing.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I still don't understand the
>> point
>> > > > about batching.
>> > > > > The
>> > > > > > consumer
>> > > > > > > > records are
>> > > > > > > > exposed as a batch in the
>> consumer
>> > > > interceptor, but
>> > > > > > you can still
>> > > > > > > > iterate
>> > > > > > > > them individually. It is no
>> > different
>> > > > for the
>> > > > > consumer
>> > > > > > API itself.
>> > > > > > > >
>> > > > > > > > -Jason
>> > > > > > > >
>> > > > > > > > On Fri, Feb 17, 2017 at
>> 12:48 PM,
>> > > > Michael Pearce <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Re:
>> > > > > > > > >
>> > > > > > > > > “ The point about
>> creation of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a slice
>> of
>> > bytes
>> > > > until the time
>> > > > > > it is accessed.”
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > >
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:37,
>> "Michael
>> > > > Pearce" <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > What’s the issue with
>> > exposing
>> > > a
>> > > > method
>> > > > > > getHeaders on the
>> > > > > > > > > producer/consumer record?
>> It
>> > > doesn’t
>> > > > break
>> > > > > anything.
>> > > > > > We don’t need
>> > > > > > > > any
>> > > > > > > > > special version.
>> > > > > > > > >
>> > > > > > > > > Current batch consumer
>> model
>> > > and
>> > > > consumer
>> > > > > > interceptors don’t
>> > > > > > > work
>> > > > > > > > > where headers need to be
>> acted on
>> > > at
>> > > > per message
>> > > > > > level at time of
>> > > > > > > > > processing, very case is
>> APM (the
>> > > > core one), where
>> > > > > > the header value
>> > > > > > > > is used
>> > > > > > > > > to continue tracing.
>> JMS/HTTP etc
>> > > > all expose these,
>> > > > > > without issues.
>> > > > > > > > I would
>> > > > > > > > > NOT want to lock this down
>> to
>> > only
>> > > > be usable
>> > > > > > accessible via
>> > > > > > > > interceptors,
>> > > > > > > > > as we’d fail on one of the
>> main
>> > > > goals.
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Mike
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:21,
>> "Jason
>> > > > Gustafson" <
>> > > > > > jason@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > The point about
>> creation
>> > of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a
>> slice of
>> > > > bytes until the
>> > > > > > time it is
>> > > > > > > > accessed.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > Yes exactly we
>> have
>> > > access
>> > > > to the records
>> > > > > > thus why the
>> > > > > > > > header
>> > > > > > > > > should be
>> > > > > > > > > > accessible via
>> it and
>> > not
>> > > > hidden for only
>> > > > > > interceptors to
>> > > > > > > > access.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > As explained
>> above, the
>> > > > point is to make
>> > > > > the
>> > > > > > intended usage
>> > > > > > > > clear.
>> > > > > > > > > Applications should
>> > > continue
>> > > > to rely on the
>> > > > > > key/value
>> > > > > > > fields
>> > > > > > > > to
>> > > > > > > > > serialize
>> > > > > > > > > their own headers,
>> and it
>> > > > would be more
>> > > > > > ideal if we can
>> > > > > > > avoid
>> > > > > > > > > leaking
>> > > > > > > > > third-party
>> headers into
>> > > > applications. This
>> > > > > > is difficult to
>> > > > > > > > do
>> > > > > > > > > with the
>> > > > > > > > > current
>> interceptors
>> > > because
>> > > > they share the
>> > > > > > record objects
>> > > > > > > > with
>> > > > > > > > > the common
>> > > > > > > > > API. What I had in
>> mind
>> > is
>> > > > something like
>> > > > > an
>> > > > > > extension of
>> > > > > > > the
>> > > > > > > > > current
>> > > > > > > > > interceptors which
>> > exposed
>> > > a
>> > > > different
>> > > > > > object (e.g.
>> > > > > > > > > `RecordAndHeaders`).
>> > > > > > > > > The challenge is
>> for
>> > > MM-like
>> > > > use cases. Let
>> > > > > > me see if I can
>> > > > > > > > come
>> > > > > > > > > up with a
>> > > > > > > > > concrete proposal
>> for
>> > that
>> > > > problem.
>> > > > > > > > >
>> > > > > > > > > -Jason
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 17,
>> 2017 at
>> > > > 11:55 AM, Michael
>> > > > > > Pearce <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I am happy to
>> move the
>> > > > definition of the
>> > > > > > header into the
>> > > > > > > > message
>> > > > > > > > > body, but
>> > > > > > > > > > would cause us
>> not to
>> > > lazy
>> > > > > > initialise/parse the headers,
>> > > > > > > as
>> > > > > > > > > obviously, we
>> > > > > > > > > > would have to
>> traverse
>> > > > these reading the
>> > > > > > message.
>> > > > > > > > > >
>> > > > > > > > > > This was
>> actually one
>> > of
>> > > > Jay’s requests:
>> > > > > > > > > >
>> > > > > > > > > > “ 2. I think
>> we
>> > should
>> > > > think about
>> > > > > > creating the Map
>> > > > > > > > lazily to
>> > > > > > > > > avoid
>> > > > > > > > > > parsing out
>> all the
>> > > > headers into
>> > > > > > little objects.
>> > > > > > > > HashMaps
>> > > > > > > > > themselves
>> > > > > > > > > > are
>> > > > > > > > > > kind of
>> expensive
>> > and
>> > > > the consumer is
>> > > > > > very perf
>> > > > > > > > sensitive so
>> > > > > > > > > and making
>> > > > > > > > > > gazillions of
>> > > hashmaps
>> > > > that may or
>> > > > > may
>> > > > > > not get used
>> > > > > > > is
>> > > > > > > > > probably a bad
>> > > > > > > > > > idea.”
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 17/02/2017,
>> 19:44,
>> > > > "Michael Pearce" <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Yes exactly
>> we have
>> > > > access to the
>> > > > > > records thus why
>> > > > > > > the
>> > > > > > > > > header should
>> > > > > > > > > > be accessible
>> via it
>> > and
>> > > > not hidden for
>> > > > > > only interceptors
>> > > > > > > > to
>> > > > > > > > > access.
>> > > > > > > > > >
>> > > > > > > > > > Sent using
>> OWA for
>> > > > iPhone
>> > > > > > > > > >
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > From: Magnus
>> > > Edenhill <
>> > > > > > magnus@edenhill.se>
>> > > > > > > > > > Sent: Friday,
>> > > February
>> > > > 17, 2017
>> > > > > > 7:34:49 PM
>> > > > > > > > > > To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > Subject: Re:
>> > > [DISCUSS]
>> > > > KIP-82 - Add
>> > > > > > Record Headers
>> > > > > > > > > >
>> > > > > > > > > > Big +1 on
>> VarInts.
>> > > > > > > > > > CPUs are
>> fast,
>> > memory
>> > > > is slow.
>> > > > > > > > > >
>> > > > > > > > > > I agree with
>> Jason
>> > > > that we'll want to
>> > > > > > continue
>> > > > > > > > verifying
>> > > > > > > > > messages,
>> > > > > > > > > > including
>> their
>> > > > headers, so while I
>> > > > > > appreciate the
>> > > > > > > > idea of
>> > > > > > > > > the opaque
>> > > > > > > > > > header blob
>> it
>> > won't
>> > > > be useful in
>> > > > > > practice.
>> > > > > > > > > >
>> > > > > > > > > > /Magnus
>> > > > > > > > > >
>> > > > > > > > > > 2017-02-17
>> 10:41
>> > > > GMT-08:00 Jason
>> > > > > > Gustafson <
>> > > > > > > > > jason@confluent.io>:
>> > > > > > > > > >
>> > > > > > > > > > > Sorry, my
>> > mistake.
>> > > > The consumer
>> > > > > > interceptor is per
>> > > > > > > > batch,
>> > > > > > > > > though I'm
>> > > > > > > > > > not
>> > > > > > > > > > > sure
>> that's an
>> > > > actual limitation
>> > > > > > since you still
>> > > > > > > have
>> > > > > > > > > access to the
>> > > > > > > > > > > individual
>> > records.
>> > > > > > > > > > >
>> > > > > > > > > > > -Jason
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri,
>> Feb 17,
>> > > 2017
>> > > > at 10:39 AM,
>> > > > > > Jason Gustafson <
>> > > > > > > > > >
>> jason@confluent.io>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Re
>> headers as
>> > > byte
>> > > > array and
>> > > > > > future use by
>> > > > > > > broker.
>> > > > > > > > This
>> > > > > > > > > doesn't
>> > > > > > > > > > take away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yeah, I
>> didn't
>> > > say
>> > > > it was
>> > > > > > difficult, only
>> > > > > > > awkward.
>> > > > > > > > You
>> > > > > > > > > wouldn't
>> > > > > > > > > > write the
>> > > > > > > > > > > > schema
>> that way
>> > > if
>> > > > you were
>> > > > > > planning to use it on
>> > > > > > > > the
>> > > > > > > > > brokers from
>> > > > > > > > > > the
>> > > > > > > > > > > >
>> beginning. Note
>> > > > also that one of
>> > > > > > the benefits of
>> > > > > > > > letting
>> > > > > > > > > the broker
>> > > > > > > > > > > >
>> understand
>> > > headers
>> > > > is that it can
>> > > > > > validate that
>> > > > > > > > they are
>> > > > > > > > > properly
>> > > > > > > > > > > >
>> formatted. If
>> > > cost
>> > > > is the only
>> > > > > > concern, we should
>> > > > > > > > > confirm its
>> > > > > > > > > > impact
>> > > > > > > > > > > > through
>> > > > performance testing.
>> > > > > > > > > > > >
>> > > > > > > > > > > > One of
>> the key
>> > > use
>> > > > cases requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'm not
>> sure I
>> > > > understand the
>> > > > > > point about
>> > > > > > > batching.
>> > > > > > > > > Interceptors
>> > > > > > > > > > are
>> > > > > > > > > > > > applied
>> > > > per-message, right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > My
>> intent on
>> > > > interceptors is to
>> > > > > > keep the usage of
>> > > > > > > > headers
>> > > > > > > > > > well-defined so
>> > > > > > > > > > > > that
>> they don't
>> > > > start leaking
>> > > > > > unnecessarily into
>> > > > > > > > > applications. My
>> > > > > > > > > > guess
>> > > > > > > > > > > is
>> > > > > > > > > > > > that it's
>> > > probably
>> > > > inevitable,
>> > > > > but
>> > > > > > isolating it
>> > > > > > > in
>> > > > > > > > the
>> > > > > > > > > > interceptors
>> would
>> > > > > > > > > > > > at least
>> give
>> > > > people a second
>> > > > > > thought before
>> > > > > > > > deciding to
>> > > > > > > > > use it.
>> > > > > > > > > > The main
>> > > > > > > > > > > >
>> challenge in my
>> > > > mind is figuring
>> > > > > > out how an MM
>> > > > > > > use
>> > > > > > > > case
>> > > > > > > > > would
>> > > > > > > > > > work. It
>> > > > > > > > > > > > would be
>> more
>> > > > cumbersome to
>> > > > > > replicate headers
>> > > > > > > > through an
>> > > > > > > > > > interceptor,
>> > > > > > > > > > > > though
>> arguably
>> > > MM
>> > > > should be
>> > > > > > working at a lower
>> > > > > > > > level
>> > > > > > > > > anyway.
>> > > > > > > > > > > >
>> > > > > > > > > > > > -Jason
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri,
>> Feb 17,
>> > > > 2017 at 10:16 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Re
>> headers
>> > > > available on the
>> > > > > > record va
>> > > > > > > > interceptors only
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> One of
>> the key
>> > > > use cases
>> > > > > requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> This
>> anyhow is
>> > > > similar to
>> > > > > > jms/http/amqp where
>> > > > > > > > headers
>> > > > > > > > > are
>> > > > > > > > > > available to
>> > > > > > > > > > > >>
>> consuming
>> > > > applications.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Re
>> headers as
>> > > > byte array and
>> > > > > > future use by
>> > > > > > > > broker. This
>> > > > > > > > > doesn't
>> > > > > > > > > > take
>> > > > > > > > > > > away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Sent
>> using OWA
>> > > > for iPhone
>> > > > > > > > > > > >>
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > > >> From:
>> Jason
>> > > > Gustafson <
>> > > > > > jason@confluent.io>
>> > > > > > > > > > > >> Sent:
>> Friday,
>> > > > February 17, 2017
>> > > > > > 5:55:42 PM
>> > > > > > > > > > > >> To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > > >>
>> Subject: Re:
>> > > > [DISCUSS] KIP-82 -
>> > > > > > Add Record
>> > > > > > > Headers
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Yes,
>> that is
>> > > what
>> > > > we are
>> > > > > > proposing, so using
>> > > > > > > > varints
>> > > > > > > > > for headers
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >>
>> consistent
>> > with
>> > > > the rest of the
>> > > > > > message. We have
>> > > > > > > > used
>> > > > > > > > > static
>> > > > > > > > > > helper
>> > > > > > > > > > > >> methods
>> > > > > > > > > > > >> in our
>> > prototype
>> > > > implementation.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> The
>> cost of
>> > > > parsing, we want to
>> > > > > > parse/interpret
>> > > > > > > > the
>> > > > > > > > > headers
>> > > > > > > > > > lazily (this
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > a key
>> point
>> > > > brought up earlier
>> > > > > > in discussions)
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I'm a
>> bit
>> > > > skeptical of this. Has
>> > > > > > anyone done the
>> > > > > > > > > performance
>> > > > > > > > > > testing? I
>> > > > > > > > > > > >> can
>> > > > > > > > > > > >> probably
>> > > > implement it and test
>> > > > > it
>> > > > > > if no one else
>> > > > > > > > has. I
>> > > > > > > > > was also
>> > > > > > > > > > under
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> impression
>> > that
>> > > > there may be use
>> > > > > > cases down the
>> > > > > > > > road
>> > > > > > > > > where the
>> > > > > > > > > > broker
>> > > > > > > > > > > >> would
>> > > > > > > > > > > >> need to
>> > > interpret
>> > > > headers. That
>> > > > > > wouldn't be off
>> > > > > > > > the
>> > > > > > > > > table in the
>> > > > > > > > > > future
>> > > > > > > > > > > if
>> > > > > > > > > > > >> it's
>> > represented
>> > > > as bytes, but
>> > > > > it
>> > > > > > would be quite
>> > > > > > > > a bit
>> > > > > > > > > more
>> > > > > > > > > > awkward,
>> > > > > > > > > > > >> right?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> By the
>> way,
>> > one
>> > > > question I have
>> > > > > > been wondering
>> > > > > > > > about. My
>> > > > > > > > > > understanding
>> > > > > > > > > > > is
>> > > > > > > > > > > >> that
>> headers
>> > are
>> > > > primarily for
>> > > > > > use cases where a
>> > > > > > > > > third-party
>> > > > > > > > > > components
>> > > > > > > > > > > >> wants to
>> > enrich
>> > > > messages without
>> > > > > > needing to
>> > > > > > > > understand
>> > > > > > > > > or modify
>> > > > > > > > > > the
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> of the
>> message
>> > > > key and value.
>> > > > > For
>> > > > > > the
>> > > > > > > applications
>> > > > > > > > > which directly
>> > > > > > > > > > > produce
>> > > > > > > > > > > >> and
>> consume
>> > the
>> > > > messages and
>> > > > > > control the
>> > > > > > > key/value
>> > > > > > > > > schema
>> > > > > > > > > > directly, it
>> > > > > > > > > > > >> seems
>> we would
>> > > > rather have them
>> > > > > > implement
>> > > > > > > headers
>> > > > > > > > > directly in
>> > > > > > > > > > their own
>> > > > > > > > > > > >> schema.
>> > > Supposing
>> > > > for the sake
>> > > > > of
>> > > > > > argument that
>> > > > > > > > it was
>> > > > > > > > > possible,
>> > > > > > > > > > my
>> > > > > > > > > > > >>
>> question is
>> > > > whether it be
>> > > > > > sufficient to expose
>> > > > > > > the
>> > > > > > > > > headers in the
>> > > > > > > > > > > >>
>> interceptor
>> > API
>> > > > and not in the
>> > > > > > common API?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> -Jason
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Fri,
>> Feb
>> > 17,
>> > > > 2017 at 3:26 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > On
>> the point
>> > > of
>> > > > varInts
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Cheers
>> > > > > > > > > > > >> > Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > 17/02/2017,
>> > > > 11:22, "Michael
>> > > > > > Pearce" <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On the
>> > > > point re: headers
>> > > > > in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > being a
>> > > > > > > > > > byte
>> > > > > > > > > > > array
>> > > > > > > > > > > >> > and
>> not a
>> > > count
>> > > > of elements
>> > > > > > followed by the
>> > > > > > > > elements.
>> > > > > > > > > Again
>> > > > > > > > > > this was
>> > > > > > > > > > > >> >
>> > > > discussed/argued previously.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> It was
>> > > > agreed on for a few
>> > > > > > reasons some of
>> > > > > > > > which
>> > > > > > > > > you have
>> > > > > > > > > > > obviously
>> > > > > > > > > > > >> >
>> picked up
>> > on:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Broker
>> > is
>> > > > able to pass it
>> > > > > > through opaquely
>> > > > > > > > > > > >> >
>> The cost
>> > > of
>> > > > parsing, we
>> > > > > > want to
>> > > > > > > > parse/interpret
>> > > > > > > > > the headers
>> > > > > > > > > > lazily
>> > > > > > > > > > > >> > (this
>> is a
>> > key
>> > > > point brought
>> > > > > up
>> > > > > > earlier in
>> > > > > > > > > discussions)
>> > > > > > > > > > > >> >
>> Headers
>> > > can
>> > > > be copied from
>> > > > > > consumer record
>> > > > > > > > to
>> > > > > > > > > producer
>> > > > > > > > > > record (aka
>> > > > > > > > > > > >> > mirror
>> > makers
>> > > > etc) without
>> > > > > > parsing if no
>> > > > > > > > changes are
>> > > > > > > > > being made
>> > > > > > > > > > or
>> > > > > > > > > > > being
>> > > > > > > > > > > >> >
>> looked at.
>> > > > > > > > > > > >> >
>> Keeps
>> > the
>> > > > broker agnostic
>> > > > > > to the format
>> > > > > > > > > > > >> >
>> You need
>> > > an
>> > > > int32 either
>> > > > > > for the byte size
>> > > > > > > > of the
>> > > > > > > > > headers,
>> > > > > > > > > > or for
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > count
>> of
>> > > > elements, so
>> > > > > overheads
>> > > > > > are the same,
>> > > > > > > > but
>> > > > > > > > > with going
>> > > > > > > > > > with an
>> > > > > > > > > > > >> opaque
>> > > > > > > > > > > >> > byte
>> array
>> > has
>> > > > the above
>> > > > > > advantages.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Cheers
>> > > > > > > > > > > >> >
>> Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > > > 17/02/2017, 02:50,
>> > > > > > "Jason Gustafson" <
>> > > > > > > > > jason@confluent.io
>> > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > Sorry,
>> > > > should have
>> > > > > > noted that the
>> > > > > > > > performance
>> > > > > > > > > testing
>> > > > > > > > > > was done
>> > > > > > > > > > > >> > using
>> the
>> > > > > > > > > > > >> >
>> > > > producer performance
>> > > > > > tool shipped with
>> > > > > > > > Kafka.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > -Jason
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On
>> > > Thu,
>> > > > Feb 16, 2017
>> > > > > at
>> > > > > > 6:44 PM, Jason
>> > > > > > > > > Gustafson <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> >
>> > Hey
>> > > > Nacho,
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > I've
>> > > > compared
>> > > > > > performance of our
>> > > > > > > > KIP-98
>> > > > > > > > > > implementation
>> with
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> without
>> > > > > > > > > > > >> >
>> >
>> > > > varints. For
>> > > > > messages
>> > > > > > around 128
>> > > > > > > > bytes, we
>> > > > > > > > > see an
>> > > > > > > > > > increase
>> > > > > > > > > > > in
>> > > > > > > > > > > >> >
>> throughput
>> > of
>> > > > > > > > > > > >> >
>> >
>> > > about
>> > > > 30% using the
>> > > > > > default
>> > > > > > > > configuration
>> > > > > > > > > settings.
>> > > > > > > > > > At 256
>> > > > > > > > > > > >> >
>> bytes, the
>> > > > > > > > > > > >> >
>> >
>> > > > increase is around
>> > > > > > 16%. Obviously
>> > > > > > > the
>> > > > > > > > > performance
>> > > > > > > > > > converge
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> messages get
>> > > > > > > > > > > >> >
>> >
>> > > > larger, but it seems
>> > > > > > well worth the
>> > > > > > > > cost.
>> > > > > > > > > Note that
>> > > > > > > > > > we are
>> > > > > > > > > > > >> also
>> > > > > > > > > > > >> >
>> seeing a
>> > > > > > > > > > > >> >
>> >
>> > > > substantial
>> > > > > > performance increase
>> > > > > > > > against
>> > > > > > > > > trunk
>> > > > > > > > > > primarily
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >
>> > much
>> > > > more efficient
>> > > > > > packing that
>> > > > > > > > varints
>> > > > > > > > > provide us.
>> > > > > > > > > > > Anything
>> > > > > > > > > > > >> >
>> adding to
>> > > > > > > > > > > >> >
>> >
>> > > > message overhead,
>> > > > > > such as record
>> > > > > > > > headers,
>> > > > > > > > > would only
>> > > > > > > > > > > increase
>> > > > > > > > > > > >> > the
>> relative
>> > > > > > > > > > > >> >
>> >
>> > > > difference. (Of
>> > > > > > course take these
>> > > > > > > > numbers
>> > > > > > > > > with a
>> > > > > > > > > > grain of
>> > > > > > > > > > > salt
>> > > > > > > > > > > >> > since
>> I have
>> > > > > > > > > > > >> >
>> >
>> > only
>> > > > used the
>> > > > > default
>> > > > > > settings with
>> > > > > > > > both
>> > > > > > > > > the producer
>> > > > > > > > > > and
>> > > > > > > > > > > >> broker
>> > > > > > > > > > > >> > on my
>> > > > > > > > > > > >> >
>> >
>> > > local
>> > > > machine. We
>> > > > > > intend to provide
>> > > > > > > > more
>> > > > > > > > > extensive
>> > > > > > > > > > > performance
>> > > > > > > > > > > >> >
>> details as
>> > > > > > > > > > > >> >
>> >
>> > part
>> > > > of the work for
>> > > > > > KIP-98.)
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > The
>> > > > implementation
>> > > > > we
>> > > > > > are using is
>> > > > > > > > from
>> > > > > > > > > protobuf (
>> > > > > > > > > > > >> >
>> >
>> > > > > > https://developers.google.com/
>> > > > > > > > > > >
>> > > > protocol-buffers/docs/encoding
>> > > > > > > > > > > >> ),
>> > > > > > > > > > > >> > which
>> is
>> > > > > > > > > > > >> >
>> >
>> > also
>> > > > used in HBase.
>> > > > > > It is trivial to
>> > > > > > > > > implement and as
>> > > > > > > > > > far
>> > > > > > > > > > > as I
>> > > > > > > > > > > >> > know
>> doesn't
>> > > > > > > > > > > >> >
>> >
>> > > > suffer from the
>> > > > > > aliasing problem you
>> > > > > > > > are
>> > > > > > > > > describing. I
>> > > > > > > > > > > checked
>> > > > > > > > > > > >> > with
>> Magnus
>> > > > > > > > > > > >> >
>> >
>> > (the
>> > > > author of
>> > > > > > librdkafka) and he
>> > > > > > > > agreed
>> > > > > > > > > that the
>> > > > > > > > > > savings
>> > > > > > > > > > > >> seemed
>> > > > > > > > > > > >> > worth
>> the
>> > > > > > > > > > > >> >
>> >
>> > cost
>> > > > of
>> > > > > > implementation.
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > -Jason
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > On
>> > > > Thu, Feb 16, 2017
>> > > > > > at 4:32 PM,
>> > > > > > > > Ignacio
>> > > > > > > > > Solis <
>> > > > > > > > > > > >>
>> > isolis@igso.net
>> > > >
>> > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >>
>> > > > -VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > I'm
>> > > > one of the
>> > > > > > people (if not the
>> > > > > > > > most)
>> > > > > > > > > opposed to
>> > > > > > > > > > VarInts.
>> > > > > > > > > > > >> >
>> VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > have
>> > > > a place, but
>> > > > > > this is not it.
>> > > > > > > > (We
>> > > > > > > > > had a large
>> > > > > > > > > > > >>
>> discussion
>> > > > > > > > > > > >> > about
>> > > > > > > > > > > >> >
>> >>
>> > > them
>> > > > at the
>> > > > > > beginning of KIP-82
>> > > > > > > time)
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > anybody has real
>> > > > > > life
>> > > > > > > performance
>> > > > > > > > > numbers of
>> > > > > > > > > > VarInts
>> > > > > > > > > > > >> >
>> improving
>> > > > > > > > > > > >> >
>> >>
>> > > > things or
>> > > > > > significantly reducing
>> > > > > > > > resources
>> > > > > > > > > I would
>> > > > > > > > > > like to
>> > > > > > > > > > > >> know
>> > > > > > > > > > > >> > what
>> > > > > > > > > > > >> >
>> >>
>> > > that
>> > > > case may be.
>> > > > > > Yes, you can save
>> > > > > > > > some
>> > > > > > > > > bytes here
>> > > > > > > > > > and
>> > > > > > > > > > > >> there,
>> > > > > > > > > > > >> > but
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > is probably
>> > > > > > insignificant to
>> > > > > > > the
>> > > > > > > > > overall system
>> > > > > > > > > > > behavior
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> >
>> >>
>> > > > storage
>> > > > > > requirements. -- I say
>> > > > > > > this
>> > > > > > > > with
>> > > > > > > > > respect to
>> > > > > > > > > > using
>> > > > > > > > > > > >> >
>> VarInts in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > protocol
>> > > > > itself,
>> > > > > > not as part of
>> > > > > > > > the
>> > > > > > > > > data.
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > VarInts require you
>> > > > > > to parse the
>> > > > > > > Int
>> > > > > > > > > before using it
>> > > > > > > > > > and
>> > > > > > > > > > > >> >
>> depending on
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > encoding they
>> > > > > > can suffer from
>> > > > > > > > aliasing
>> > > > > > > > > (multiple
>> > > > > > > > > > > >> >
>> > > representations
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > the same
>> > > > > value).
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > Why
>> > > > add complexity?
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > Nacho
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > On
>> > > > Thu, Feb 16,
>> > > > > 2017
>> > > > > > at 10:29 AM,
>> > > > > > > > Colin
>> > > > > > > > > McCabe <
>> > > > > > > > > > > >> >
>> > > > cmccabe@apache.org>
>> > > > > > > > > > > >> >
>> >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >> >
>> > > +1
>> > > > for varints
>> > > > > > here-- it would
>> > > > > > > save
>> > > > > > > > > quite a bit of
>> > > > > > > > > > space.
>> > > > > > > > > > > >> > They
>> are
>> > > > > > > > > > > >> >
>> >> >
>> > > > pretty quick to
>> > > > > > implement as
>> > > > > > > well.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > > I
>> > > > think it makes
>> > > > > > sense for values
>> > > > > > > > to be
>> > > > > > > > > byte
>> > > > > > > > > > arrays.
>> > > > > > > > > > > Users
>> > > > > > > > > > > >> > might
>> want
>> > > > > > > > > > > >> >
>> >> >
>> > > to
>> > > > attach
>> > > > > > arbitrary payloads;
>> > > > > > > they
>> > > > > > > > > shouldn't be
>> > > > > > > > > > forced to
>> > > > > > > > > > > >> >
>> serialize
>> > > > > > > > > > > >> >
>> >> >
>> > > > everything to
>> > > > > Java
>> > > > > > strings.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > best,
>> > > > > > > > > > > >> >
>> >> >
>> > > > Colin
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > On
>> > > > Thu, Feb 16,
>> > > > > > 2017, at 09:52,
>> > > > > > > > Jason
>> > > > > > > > > Gustafson
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hey Michael,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hmm, I guess the
>> > > > > > point of
>> > > > > > > > representing
>> > > > > > > > > it as
>> > > > > > > > > > bytes is to
>> > > > > > > > > > > >> > allow
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > broker
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > to pass it
>> > > > > > through opaquely? Is
>> > > > > > > > the
>> > > > > > > > > cost of
>> > > > > > > > > > parsing
>> > > > > > > > > > > them a
>> > > > > > > > > > > >> >
>> concern, or
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > are
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > we simply trying
>> > > > > > to ensure that
>> > > > > > > > the
>> > > > > > > > > broker stays
>> > > > > > > > > > > agnostic
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > format?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On varints, I
>> > > > > > think adding
>> > > > > > > > support for
>> > > > > > > > > them makes
>> > > > > > > > > > less
>> > > > > > > > > > > >> sense
>> > > > > > > > > > > >> > for an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > isolated use
>> > > > > > case, but as part
>> > > > > > > of
>> > > > > > > > a
>> > > > > > > > > more holistic
>> > > > > > > > > > change
>> > > > > > > > > > > >> > (such
>> as
>> > what
>> > > > > > > > > > > >> >
>> >>
>> > we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > have proposed in
>> > > > > > KIP-98), I
>> > > > > > > think
>> > > > > > > > they
>> > > > > > > > > are
>> > > > > > > > > > justifiable.
>> > > > > > > > > > > If
>> > > > > > > > > > > >> > we
>> add them,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > then the need to
>> > > > > > use attributes
>> > > > > > > > becomes
>> > > > > > > > > quite a
>> > > > > > > > > > bit
>> > > > > > > > > > > >> weaker,
>> > > > > > > > > > > >> >
>> right? The
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > other thing I
>> > > > > > find slightly odd
>> > > > > > > > is the
>> > > > > > > > > fact that
>> > > > > > > > > > null
>> > > > > > > > > > > >> >
>> headers has
>> > no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > actual
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > semantic meaning
>> > > > > > for the message
>> > > > > > > > > (unlike null
>> > > > > > > > > > keys and
>> > > > > > > > > > > >> >
>> values). It
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > a
>> > > > space
>> > > > > > optimization. It seems a
>> > > > > > > > bit
>> > > > > > > > > better to
>> > > > > > > > > > always
>> > > > > > > > > > > use
>> > > > > > > > > > > >> > size
>> 0 to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > indicate having
>> > > > > > no headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Overall, the
>> > > > > main
>> > > > > > point is
>> > > > > > > > ensuring
>> > > > > > > > > that the
>> > > > > > > > > > message
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> >
>> remains
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > consistent,
>> > > > > > either within the
>> > > > > > > > larger
>> > > > > > > > > protocol, or
>> > > > > > > > > > at a
>> > > > > > > > > > > >> >
>> minimum
>> > within
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > message itself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On Thu, Feb 16,
>> > > > > > 2017 at 6:39 AM,
>> > > > > > > > > Michael Pearce <
>> > > > > > > > > > > >> >
>> > > > Michael.Pearce@ig.com
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Hi Jason,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 1) in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > the headers
>> > > > > > > > > > are
>> > > > > > > > > > > >> simply
>> > > > > > > > > > > >> > a byte
>> > > > > > > > > > > >> >
>> >>
>> > > > array,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > as like the
>> > > > > key
>> > > > > > or value, this
>> > > > > > > > is to
>> > > > > > > > > clearly
>> > > > > > > > > > demarcate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> header in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > core message.
>> > > > > > Then the header
>> > > > > > > > byte
>> > > > > > > > > array in the
>> > > > > > > > > > core
>> > > > > > > > > > > >> >
>> message is
>> > an
>> > > > > > > > > > > >> >
>> >>
>> > > > array of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key, value
>> > > > > > pairs. This is what
>> > > > > > > > it is
>> > > > > > > > > denoting.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Then this
>> > > > > would
>> > > > > > be I guess in
>> > > > > > > > the
>> > > > > > > > > given
>> > > > > > > > > > notation:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Headers =>
>> > > > > > [KeyLength, Key,
>> > > > > > > > > ValueLength, Value]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > KeyLength
>> > > > > > => int32
>> > > > > > > > > <-----------------NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised key
>> > > > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Key =>
>> > > > > bytes
>> > > > > > > > > <---------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > string
>> > (UTF8)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > bytes of the
>> > > > > > header key
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > ValueLength
>> > > > > > => int32
>> > > > > > > > > <-------------- NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised
>> > > > > > header value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Value =>
>> > > > > > bytes
>> > > > > > > > > <-------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > form
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The key length
>> > > > > > and value
>> > > > > > > length
>> > > > > > > > is
>> > > > > > > > > matching the
>> > > > > > > > > > way
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> protocol is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > defined in the
>> > > > > > core message
>> > > > > > > > currently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 2)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Var sized
>> > > > > ints,
>> > > > > > this was
>> > > > > > > > discussed
>> > > > > > > > > much earlier
>> > > > > > > > > > on, in
>> > > > > > > > > > > >> > fact
>> I had
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > suggested it
>> > > > > > myself (with
>> > > > > > > Hadoop
>> > > > > > > > > references),
>> > > > > > > > > > the
>> > > > > > > > > > > >> >
>> complexity
>> > of
>> > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compared to
>> > > > > > having a simpler
>> > > > > > > > protocol
>> > > > > > > > > was
>> > > > > > > > > > argued and
>> > > > > > > > > > > >> >
>> agreed it
>> > > > > > > > > > > >> >
>> >>
>> > > > wasn’t worth
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the complexity
>> > > > > > as all other
>> > > > > > > > clients
>> > > > > > > > > in other
>> > > > > > > > > > languages
>> > > > > > > > > > > >> > would
>> need
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > > > ensure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > theyre using
>> > > > > > the right var
>> > > > > > > size
>> > > > > > > > > algorithm, as
>> > > > > > > > > > there
>> > > > > > > > > > > is a
>> > > > > > > > > > > >> > few.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 3)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > We did the
>> > > > > > attributes,
>> > > > > > > optional
>> > > > > > > > > approach as
>> > > > > > > > > > originally
>> > > > > > > > > > > >> > there
>> was
>> > > > > > > > > > > >> >
>> >>
>> > > > marked
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > concern that
>> > > > > > headers would
>> > > > > > > > cause a
>> > > > > > > > > message size
>> > > > > > > > > > > overhead
>> > > > > > > > > > > >> > for
>> others,
>> > > > > > > > > > > >> >
>> >>
>> > who
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > don’t want
>> > > > > > them. As such this
>> > > > > > > > is the
>> > > > > > > > > clean
>> > > > > > > > > > solution to
>> > > > > > > > > > > >> >
>> achieve
>> > that.
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that no longer
>> > > > > > holds, and we
>> > > > > > > > don’t
>> > > > > > > > > care that we
>> > > > > > > > > > add
>> > > > > > > > > > > >> 4bytes
>> > > > > > > > > > > >> >
>> overhead,
>> > > > > > > > > > > >> >
>> >>
>> > > then
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > im happy to
>> > > > > > remove.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > I’m personally
>> > > > > > in favour of
>> > > > > > > > keeping
>> > > > > > > > > the message
>> > > > > > > > > > as
>> > > > > > > > > > > small
>> > > > > > > > > > > >> > as
>> possible
>> > > > > > > > > > > >> >
>> >>
>> > so
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > people don’t
>> > > > > > get shocks in
>> > > > > > > perf
>> > > > > > > > and
>> > > > > > > > > throughputs
>> > > > > > > > > > dues
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> message
>> > size,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > unless they
>> > > > > > actively use the
>> > > > > > > > feature,
>> > > > > > > > > as such I
>> > > > > > > > > > do
>> > > > > > > > > > > >> prefer
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > attribute bit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wise feature
>> > > > > > flag approach
>> > > > > > > > myself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On 16/02/2017,
>> > > > > > 05:40, "Jason
>> > > > > > > > > Gustafson" <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > We have
>> > > > > > proposed a few
>> > > > > > > > > significant changes
>> > > > > > > > > > to the
>> > > > > > > > > > > >> >
>> message
>> > format
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > KIP-98
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > which now
>> > > > > > seems likely to
>> > > > > > > > pass
>> > > > > > > > > (perhaps
>> > > > > > > > > > with some
>> > > > > > > > > > > >> >
>> iterations
>> > on
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > implementation details).
>> > > > > > > It
>> > > > > > > > would
>> > > > > > > > > be good
>> > > > > > > > > > to try
>> > > > > > > > > > > and
>> > > > > > > > > > > >> >
>> coordinate
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > changes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > in both of
>> > > > > > the proposals
>> > > > > > > to
>> > > > > > > > make
>> > > > > > > > > sure they
>> > > > > > > > > > are
>> > > > > > > > > > > >> >
>> consistent
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compatible.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I think
>> > > > > > using the
>> > > > > > > > attributes to
>> > > > > > > > > indicate
>> > > > > > > > > > null
>> > > > > > > > > > > >> headers
>> > > > > > > > > > > >> > is a
>> > > > > > > > > > > >> >
>> >>
>> > > > reasonable
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > approach.
>> > > > > > We have proposed
>> > > > > > > > to do
>> > > > > > > > > the same
>> > > > > > > > > > thing
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> message
>> > > > > > > > > > > >> >
>> >>
>> > key
>> > > > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > value.
>> > > > > That
>> > > > > > said, I
>> > > > > > > > sympathize
>> > > > > > > > > with Jay's
>> > > > > > > > > > > argument.
>> > > > > > > > > > > >> > Having
>> > > > > > > > > > > >> >
>> >>
>> > > > multiple
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > ways to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > specify a
>> > > > > > null value
>> > > > > > > > increases
>> > > > > > > > > the overall
>> > > > > > > > > > > >>
>> complexity
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > You
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > can see
>> > > > > > this just from the
>> > > > > > > > fact
>> > > > > > > > > that you
>> > > > > > > > > > need the
>> > > > > > > > > > > >> > extra
>> > verbiage
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > protocol
>> > > > > > specification in
>> > > > > > > > this
>> > > > > > > > > KIP and in
>> > > > > > > > > > KIP-98
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> describe the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > dependence
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > between
>> > > > > the
>> > > > > > fields and the
>> > > > > > > > > attributes. It
>> > > > > > > > > > seems
>> > > > > > > > > > > >> like a
>> > > > > > > > > > > >> >
>> slippery
>> > > > > > > > > > > >> >
>> >>
>> > > slope
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > if
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > you start
>> > > > > > allowing
>> > > > > > > different
>> > > > > > > > > request types
>> > > > > > > > > > to
>> > > > > > > > > > > >> >
>> implement
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification differently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > You can
>> > > > > > also argue that
>> > > > > > > the
>> > > > > > > > > messages
>> > > > > > > > > > already are
>> > > > > > > > > > > and
>> > > > > > > > > > > >> > are
>> likely
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > remain a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > special
>> > > > > > case. For example,
>> > > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > currently no
>> > > > > > > > > > > >> >
>> generality
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > how
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > compressed
>> > > > > > message sets
>> > > > > > > are
>> > > > > > > > > represented
>> > > > > > > > > > that would
>> > > > > > > > > > > >> be
>> > > > > > > > > > > >> >
>> applicable
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > other
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > request
>> > > > > > types. Some might
>> > > > > > > > see this
>> > > > > > > > > > divergence as
>> > > > > > > > > > > an
>> > > > > > > > > > > >> >
>> unfortunate
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > deficiency
>> > > > > > which should be
>> > > > > > > > fixed;
>> > > > > > > > > others
>> > > > > > > > > > might see
>> > > > > > > > > > > >> it
>> > > > > > > > > > > >> > as
>> sort of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > inevitability of needing
>> > > > > > > to
>> > > > > > > > > optimize where
>> > > > > > > > > > it
>> > > > > > > > > > > counts
>> > > > > > > > > > > >> > most.
>> I'm
>> > > > > > > > > > > >> >
>> >>
>> > > > probably
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > somewhere
>> > > > > > in between, but
>> > > > > > > I
>> > > > > > > > think
>> > > > > > > > > we
>> > > > > > > > > > probably all
>> > > > > > > > > > > >> > share
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > intuition
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol should be
>> > > > > > > kept
>> > > > > > > > as
>> > > > > > > > > consistent as
>> > > > > > > > > > > >>
>> possible.
>> > > > > > > > > > > >> > With
>> that
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > mind,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > here are a
>> > > > > > few comments:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 1. One
>> > > > > > thing I found a
>> > > > > > > > little odd
>> > > > > > > > > when
>> > > > > > > > > > reading the
>> > > > > > > > > > > >> >
>> current
>> > > > > > > > > > > >> >
>> >>
>> > > > proposal is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > headers
>> > > > > > are both
>> > > > > > > > represented
>> > > > > > > > > as an
>> > > > > > > > > > array of
>> > > > > > > > > > > >> bytes
>> > > > > > > > > > > >> > and
>> as an
>> > > > > > > > > > > >> >
>> >>
>> > > array
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > key/value
>> > > > > > pairs. I'd
>> > > > > > > > probably
>> > > > > > > > > suggest
>> > > > > > > > > > something
>> > > > > > > > > > > like
>> > > > > > > > > > > >> > this:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > [HeaderKey
>> > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > HeaderKey
>> > > > > > => String
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue => Bytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > An array
>> > > > > in
>> > > > > > the Kafka
>> > > > > > > > protocol is
>> > > > > > > > > > represented as a
>> > > > > > > > > > > >> > 4-byte
>> > integer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > indicating
>> > > > > > the number of
>> > > > > > > > elements
>> > > > > > > > > in the
>> > > > > > > > > > array
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > serialization of the
>> > > > > > > > elements.
>> > > > > > > > > Unless I'm
>> > > > > > > > > > > >> >
>> > > > misunderstanding, what
>> > > > > > > > > > > >> >
>> >>
>> > you
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > instead is
>> > > > > > the total size
>> > > > > > > > of the
>> > > > > > > > > headers in
>> > > > > > > > > > bytes
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > elements.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I'm not
>> > > > > > sure I see any
>> > > > > > > > reason for
>> > > > > > > > > this
>> > > > > > > > > > > >>
>> inconsistency.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 2. In
>> > > > > > KIP-98, we've
>> > > > > > > > introduced
>> > > > > > > > > > variable-length
>> > > > > > > > > > > >> integer
>> > > > > > > > > > > >> >
>> fields.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Effectively,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > we've
>> > > > > > enriched (or
>> > > > > > > > "complicated"
>> > > > > > > > > as Jay
>> > > > > > > > > > might say
>> > > > > > > > > > > ;)
>> > > > > > > > > > > >> > the
>> protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification to include
>> > > > > > > the
>> > > > > > > > > following
>> > > > > > > > > > types:
>> > > > > > > > > > > >> VarInt,
>> > > > > > > > > > > >> >
>> VarLong,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > UnsignedVarInt and
>> > > > > > > > > UnsignedVarLong.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Along with
>> > > > > > these
>> > > > > > > > primitives, we
>> > > > > > > > > could
>> > > > > > > > > > introduce
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> following
>> > > > > > > > > > > >> >
>> >>
>> > > > types:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeArray =>
>> > > > > > > > NumberOfItems
>> > > > > > > > > Item1 Item2
>> > > > > > > > > > .. ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItems =>
>> > > > > > > > UnsignedVarInt
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeNullableArray =>
>> > > > > > > > > NumberOfItemsOrNull
>> > > > > > > > > > Item1
>> > > > > > > > > > > >> > Item2
>> ..
>> > ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItemsOrNull =>
>> > > > > > > > VarInt
>> > > > > > > > > (-1 means
>> > > > > > > > > > null)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > And
>> > > > > > similarly for the
>> > > > > > > > `String`
>> > > > > > > > > and `Bytes`
>> > > > > > > > > > types.
>> > > > > > > > > > > >> > These
>> types
>> > > > > > > > > > > >> >
>> >>
>> > can
>> > > > save
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > considerable amount of
>> > > > > > > > space in
>> > > > > > > > > this
>> > > > > > > > > > proposal
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > they
>> can
>> > > > > > > > > > > >> >
>> >>
>> > be
>> > > > used
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > both the
>> > > > > > number of headers
>> > > > > > > > > included in the
>> > > > > > > > > > message
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > lengths of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > header
>> > > > > keys
>> > > > > > and values. We
>> > > > > > > > could
>> > > > > > > > > do this
>> > > > > > > > > > instead:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > > > VarSizeArray[HeaderKey
>> > > > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > HeaderKey
>> > > > > > =>
>> > > > > > > VarSizeString
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue =>
>> > > > > > > > VarSizeBytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Combining
>> > > > > > the savings from
>> > > > > > > > the
>> > > > > > > > > use of
>> > > > > > > > > > variable
>> > > > > > > > > > > >> length
>> > > > > > > > > > > >> >
>> fields, the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > benefit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > of using
>> > > > > > the attributes to
>> > > > > > > > > represent null
>> > > > > > > > > > seems
>> > > > > > > > > > > >> pretty
>> > > > > > > > > > > >> > small.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 3.
>> > > > > > Whichever way we go
>> > > > > > > > (whether
>> > > > > > > > > we use the
>> > > > > > > > > > > >>
>> attributes
>> > > > > > > > > > > >> > or
>> not), we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > should at
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > least be
>> > > > > > consistent
>> > > > > > > between
>> > > > > > > > this
>> > > > > > > > > KIP and
>> > > > > > > > > > KIP-98.
>> > > > > > > > > > > It
>> > > > > > > > > > > >> > would
>> be
>> > very
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > strange
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > to have
>> > > > > two
>> > > > > > ways to
>> > > > > > > > represent
>> > > > > > > > > null values
>> > > > > > > > > > in the
>> > > > > > > > > > > >> same
>> > > > > > > > > > > >> >
>> schema.
>> > > > > > > > > > > >> >
>> >>
>> > > > Either
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > way is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > OK with
>> > > > > me.
>> > > > > > I think some
>> > > > > > > > > message-level
>> > > > > > > > > > >
>> optimizations
>> > > > > > > > > > > >> > are
>> > > > > > > > > > > >> >
>> >>
>> > > > justifiable,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > but
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > savings
>> > > > > > here seem
>> > > > > > > > minimal (a
>> > > > > > > > > few bytes
>> > > > > > > > > > per
>> > > > > > > > > > > >> >
>> message), so
>> > > > > > > > > > > >> >
>> >>
>> > > > maybe it's
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > worth the
>> > > > > > cost of letting
>> > > > > > > > the
>> > > > > > > > > message
>> > > > > > > > > > diverge even
>> > > > > > > > > > > >> >
>> further from
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > rest of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > On Wed,
>> > > > > Feb
>> > > > > > 15, 2017 at
>> > > > > > > > 8:52 AM,
>> > > > > > > > > radai <
>> > > > > > > > > > > >> >
>> >>
>> > > > > > radai.rosenblatt@gmail.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > I've
>> > > > > > trimmed the inline
>> > > > > > > > > contents as this
>> > > > > > > > > > mail is
>> > > > > > > > > > > >> >
>> getting too
>> > > > > > > > > > > >> >
>> >>
>> > big
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > apache
>> > > > > > mailing list
>> > > > > > > > software to
>> > > > > > > > > deliver
>> > > > > > > > > > :-(
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 1. the
>> > > > > > important thing
>> > > > > > > for
>> > > > > > > > > > interoperability
>> is
>> > > > > > > > > > > for
>> > > > > > > > > > > >> >
>> different
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "interested
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > parties"
>> > > > > > (plugins, infra
>> > > > > > > > > layers/wrappers,
>> > > > > > > > > > > >>
>> user-code)
>> > > > > > > > > > > >> > to be
>> > > > > > > > > > > >> >
>> >>
>> > > able
>> > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > stick
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > pieces
>> > > > > of
>> > > > > > metadata onto
>> > > > > > > > msgs
>> > > > > > > > > without
>> > > > > > > > > > getting in
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> >
>> other's
>> > > > > > > > > > > >> >
>> >>
>> > > way.
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > common
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > key
>> > > > > > scheme (Strings, as
>> > > > > > > > of the
>> > > > > > > > > time of
>> > > > > > > > > > this
>> > > > > > > > > > > >> >
>> writing?) is
>> > > all
>> > > > > > > > > > > >> >
>> >>
>> > > thats
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > required
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > for
>> > > > > that.
>> > > > > > it is assumed
>> > > > > > > > that
>> > > > > > > > > the other end
>> > > > > > > > > > > >> >
>> interested
>> > in
>> > > > any
>> > > > > > > > > > > >> >
>> >>
>> > > such
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > piece of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > metadata
>> > > > > > knows the
>> > > > > > > > encoding,
>> > > > > > > > > and byte[]
>> > > > > > > > > > provides
>> > > > > > > > > > > >> for
>> > > > > > > > > > > >> > the
>> most
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > flexibility.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > i
>> > > > > believe
>> > > > > > this is the
>> > > > > > > same
>> > > > > > > > > logic behind
>> > > > > > > > > > core
>> > > > > > > > > > > kafka
>> > > > > > > > > > > >> > being
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > byte[]/byte[]
>> > > > > -
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Strings
>> > > > > > are more
>> > > > > > > "usable"
>> > > > > > > > but
>> > > > > > > > > bytes are
>> > > > > > > > > > flexible
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > so
>> were
>> > > > > > > > > > > >> >
>> >>
>> > > > chosen.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Also -
>> > > > > > core kafka doesnt
>> > > > > > > > even
>> > > > > > > > > do that
>> > > > > > > > > > good of a
>> > > > > > > > > > > >> job
>> > > > > > > > > > > >> > on
>> > > > > > > > > > > >> >
>> >>
>> > > > usability of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > payload
>> > > > > > (example - i
>> > > > > > > have
>> > > > > > > > to
>> > > > > > > > > specify the
>> > > > > > > > > > nop
>> > > > > > > > > > > >> byte[]
>> > > > > > > > > > > >> >
>> "decoders"
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > explicitly
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > in
>> > > > > conf),
>> > > > > > and again
>> > > > > > > > sacrificies
>> > > > > > > > > usability
>> > > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > sake
>> of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > performance
>> > > > > (no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > convenient single-record
>> > > > > > > > > processing as
>> > > > > > > > > > poll is a
>> > > > > > > > > > > >> >
>> batch, lots
>> > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > obscure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > little
>> > > > > > config details
>> > > > > > > > exposing
>> > > > > > > > > internals
>> > > > > > > > > > of the
>> > > > > > > > > > > >> >
>> batching
>> > > > > > > > > > > >> >
>> >>
>> > > > mechanism,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > etc)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > this is
>> > > > > > also why i
>> > > > > > > really
>> > > > > > > > > dislike the
>> > > > > > > > > > idea of a
>> > > > > > > > > > > >> > "type
>> > system"
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > values,
>> > > > > > it further
>> > > > > > > > degrades the
>> > > > > > > > > > usability, adds
>> > > > > > > > > > > >> >
>> complexity
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > eventually get in
>> > > > > > > > people's way,
>> > > > > > > > > also, it
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >> > the
>> 2nd/3rd
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > home-group
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization mechanism
>> > > > > > > > in core
>> > > > > > > > > kafka
>> > > > > > > > > > (counting
>> > > > > > > > > > > 2
>> > > > > > > > > > > >> >
>> iterations
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "type
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > definition DSL")
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 2. this
>> > > > > > is an
>> > > > > > > > implementation
>> > > > > > > > > detail, and
>> > > > > > > > > > not
>> > > > > > > > > > > even
>> > > > > > > > > > > >> a
>> > > > > > > > > > > >> > very
>> "user
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > facing" one?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to the
>> > > > > > best of my
>> > > > > > > > understanding
>> > > > > > > > > the vote
>> > > > > > > > > > process
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > on
>> proposed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > API/behaviour. also -
>> > > > > > > > since
>> > > > > > > > > we're willing
>> > > > > > > > > > to go
>> > > > > > > > > > > >> with
>> > > > > > > > > > > >> >
>> strings
>> > > > > > > > > > > >> >
>> >>
>> > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialize
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > a
>> > > > > 0-sized
>> > > > > > header blob
>> > > > > > > and
>> > > > > > > > IIUC
>> > > > > > > > > you dont
>> > > > > > > > > > need any
>> > > > > > > > > > > >> >
>> optionals
>> > > > > > > > > > > >> >
>> >>
>> > > > anymore.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 3. yes,
>> > > > > > we can :-)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > On Tue,
>> > > > > > Feb 14, 2017 at
>> > > > > > > > 11:56
>> > > > > > > > > PM, Michael
>> > > > > > > > > > > Pearce <
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Hi
>> > > > > Jay,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1)
>> > > > > > There was some
>> > > > > > > > initial
>> > > > > > > > > debate on the
>> > > > > > > > > > value
>> > > > > > > > > > > >> > part,
>> as
>> > youll
>> > > > > > > > > > > >> >
>> >>
>> > > note
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > String,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > String
>> > > > > > headers were
>> > > > > > > > > discounted early
>> > > > > > > > > > on. The
>> > > > > > > > > > > >> >
>> reason for
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > flexibility
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > keeping in line
>> > > > > > > > with the
>> > > > > > > > > > flexibility of
>> > > > > > > > > > > key,
>> > > > > > > > > > > >> > value
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > message
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > object
>> > > > > > itself. I don’t
>> > > > > > > > think
>> > > > > > > > > it takes
>> > > > > > > > > > away
>> > > > > > > > > > > from
>> > > > > > > > > > > >> an
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > as
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > each
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > plugin
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > will
>> > > > > > care for their
>> > > > > > > own
>> > > > > > > > key,
>> > > > > > > > > this way
>> > > > > > > > > > ints,
>> > > > > > > > > > > >> >
>> booleans ,
>> > > > exotic
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > binary
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > can
>> > > > > all
>> > > > > > be catered
>> > > > > > > for=.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > wanted
>> > > > > > > > to
>> > > > > > > > > push for a
>> > > > > > > > > > typed
>> > > > > > > > > > > >> value
>> > > > > > > > > > > >> >
>> interface,
>> > > > > > > > > > > >> >
>> >> I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wouldn’t
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > want
>> > > > > > just String
>> > > > > > > values
>> > > > > > > > > supported, but
>> > > > > > > > > > the the
>> > > > > > > > > > > >> >
>> primatives
>> > > > > > > > > > > >> >
>> >>
>> > > plus
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > string and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > also
>> > > > > > still keeping the
>> > > > > > > > > ability to have a
>> > > > > > > > > > > binary
>> > > > > > > > > > > >> > for
>> custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > binaries that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > some
>> > > > > > organisations may
>> > > > > > > > have.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > i. I
>> > > > > > have written this
>> > > > > > > > slight
>> > > > > > > > > > alternative
>> > > > > > > > > > > here,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > https://cwiki.apache.org/
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > > >> >
>> > > > 82+-+Add+Record+Headers+-+
>> > > > > Typed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > ii.
>> > > > > > Essentially the
>> > > > > > > > value
>> > > > > > > > > bytes, has a
>> > > > > > > > > > leading
>> > > > > > > > > > > >> > byte
>> > overhead.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > > This tells you
>> > > > > > > what
>> > > > > > > > type
>> > > > > > > > > the value
>> > > > > > > > > > is,
>> > > > > > > > > > > >> before
>> > > > > > > > > > > >> >
>> reading
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > rest
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes,
>> > > > > > allowing
>> > > > > > > > > >
>> > > > serialisation/deserialization
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > and
>> from the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > primitives,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > string
>> > > > > > and byte[].
>> > > > > > > This
>> > > > > > > > is
>> > > > > > > > > akin to some
>> > > > > > > > > > other
>> > > > > > > > > > > >> >
>> messaging
>> > > > > > > > > > > >> >
>> >>
>> > > > systems.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2) We
>> > > > > > are making it
>> > > > > > > > optional,
>> > > > > > > > > so that
>> > > > > > > > > > for
>> > > > > > > > > > > those
>> > > > > > > > > > > >> > not
>> wanting
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > headers have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 0
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes
>> > > > > > overhead (think
>> > > > > > > > of it
>> > > > > > > > > as a feature
>> > > > > > > > > > > flag),
>> > > > > > > > > > > >> I
>> > > > > > > > > > > >> > don’t
>> > > > > > > > > > > >> >
>> >>
>> > > > think this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > complex, especially if
>> > > > > > > > > comparing to
>> > > > > > > > > > changes
>> > > > > > > > > > > >> >
>> proposed in
>> > > > > > > > > > > >> >
>> >>
>> > > > other kips
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > like
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > kip-98.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > really
>> > > > > > > > don’t
>> > > > > > > > > like
>> > > > > > > > > > this, we
>> > > > > > > > > > > can
>> > > > > > > > > > > >> > drop
>> it, but
>> > > > > > > > > > > >> >
>> >>
>> > it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > would
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > mean
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > buying
>> > > > > > into 4 bytes
>> > > > > > > > extra
>> > > > > > > > > overhead for
>> > > > > > > > > > users
>> > > > > > > > > > > who
>> > > > > > > > > > > >> > do
>> not want
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > use
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3) In
>> > > > > > the summary yes,
>> > > > > > > > it is
>> > > > > > > > > at a higher
>> > > > > > > > > > > level,
>> > > > > > > > > > > >> > but I
>> think
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is well
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > documented in the
>> > > > > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > section.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a.
>> > > > > > Added getHeaders
>> > > > > > > > method to
>> > > > > > > > > > >
>> Producer/Consumer
>> > > > > > > > > > > >> >
>> record (that
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > it)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > b.
>> > > > > > We’ve also detailed
>> > > > > > > > the
>> > > > > > > > > new Headers
>> > > > > > > > > > class
>> > > > > > > > > > > >> that
>> > > > > > > > > > > >> > this
>> method
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > returns
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > encapsulates the
>> > > > > > > headers
>> > > > > > > > > protocol and
>> > > > > > > > > > logic.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Best,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Mike
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ==Original questions
>> > > > > > > > from the
>> > > > > > > > > vote
>> > > > > > > > > > thread from
>> > > > > > > > > > > >> > Jay.==
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Couple
>> > > > > > of things I
>> > > > > > > > think we
>> > > > > > > > > still need
>> > > > > > > > > > to work
>> > > > > > > > > > > >> out:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > I
>> > > > > > think we agree
>> > > > > > > > about
>> > > > > > > > > the key,
>> > > > > > > > > > but I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > we
>> haven't
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > talked about
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > value yet. I
>> > > > > > > > think if
>> > > > > > > > > our goal
>> > > > > > > > > > is an
>> > > > > > > > > > > open
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > these
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > spread across many
>> > > > > > > > plugins
>> > > > > > > > > from many
>> > > > > > > > > > > systems
>> > > > > > > > > > > >> we
>> > > > > > > > > > > >> > should
>> > > > > > > > > > > >> >
>> >>
>> > > > consider
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > making
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > string as well so
>> > > > > > > > it can
>> > > > > > > > > be
>> > > > > > > > > > printed, set
>> > > > > > > > > > > >> via
>> > > > > > > > > > > >> > a UI,
>> set
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > config,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > etc.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > Basically
>> > > > > > > encouraging
>> > > > > > > > > pluggable
>> > > > > > > > > > >
>> serialization
>> > > > > > > > > > > >> >
>> formats
>> > > > > > > > > > > >> >
>> >>
>> > > here
>> > > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > lead
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bit
>> > > > > > of a tower of
>> > > > > > > > babel.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2.
>> > > > > > This proposal
>> > > > > > > > still
>> > > > > > > > > includes a
>> > > > > > > > > > pretty
>> > > > > > > > > > > big
>> > > > > > > > > > > >> >
>> change to
>> > our
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > protocol
>> > > > > > > > definition
>> > > > > > > > > layer.
>> > > > > > > > > > Essentially
>> > > > > > > > > > > >> it is
>> > > > > > > > > > > >> >
>> >>
>> > > > introducing an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > optional
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > type, where the
>> > > > > > > > format is
>> > > > > > > > > data
>> > > > > > > > > > dependent. I
>> > > > > > > > > > > >> > think
>> this
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > actually a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > big
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > change though it
>> > > > > > > > doesn't
>> > > > > > > > > seem like
>> > > > > > > > > > it. It
>> > > > > > > > > > > >> means
>> > > > > > > > > > > >> > you
>> can no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > longer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > specify
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > this
>> > > > > > type with our
>> > > > > > > > type
>> > > > > > > > > definition
>> > > > > > > > > > DSL, and
>> > > > > > > > > > > >> >
>> likewise it
>> > > > > > > > > > > >> >
>> >>
>> > > > requires
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > handling in client
>> > > > > > > > libs.
>> > > > > > > > > This isn't
>> > > > > > > > > > a huge
>> > > > > > > > > > > >> >
>> thing, since
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Record
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > definition is
>> > > > > > > custom
>> > > > > > > > > anyway, but I
>> > > > > > > > > > think
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > kind
>> of
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > inconsistency is
>> > > > > > > very
>> > > > > > > > > non-desirable
>> > > > > > > > > > and
>> > > > > > > > > > > ties
>> > > > > > > > > > > >> > you to
>> > > > > > > > > > > >> >
>> >>
>> > > > hand-coding
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > things.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > think the type
>> > > > > > > should
>> > > > > > > > > instead by [Key
>> > > > > > > > > > > Value]
>> > > > > > > > > > > >> in
>> > > > > > > > > > > >> > our
>> BNF,
>> > > > > > > > > > > >> >
>> >>
>> > > where
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > value are both
>> > > > > > > short
>> > > > > > > > > strings as used
>> > > > > > > > > > > >>
>> elsewhere.
>> > > > > > > > > > > >> > This
>> > > > > > > > > > > >> >
>> >>
>> > > > brings it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > in line
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > with
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > rest of the
>> > > > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3.
>> > > > > > Could we get
>> > > > > > > more
>> > > > > > > > > specific about
>> > > > > > > > > > the
>> > > > > > > > > > > exact
>> > > > > > > > > > > >> > Java
>> API
>> > > > > > > > > > > >> >
>> >>
>> > > > change to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ProducerRecord,
>> > > > > > > > > ConsumerRecord,
>> > > > > > > > > > Record,
>> > > > > > > > > > > etc?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > -Jay
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The
>> > > > > information
>> > > > > > contained in
>> > > > > > > > this
>> > > > > > > > > email is
>> > > > > > > > > > strictly
>> > > > > > > > > > > >> >
>> confidential
>> > > and
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the use of the
>> > > > > > addressee only,
>> > > > > > > > unless
>> > > > > > > > > otherwise
>> > > > > > > > > > > >>
>> indicated.
>> > > > > > > > > > > >> > If you
>> > > > > > > > > > > >> >
>> >>
>> > are
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the intended
>> > > > > > recipient, please
>> > > > > > > > do not
>> > > > > > > > > read,
>> > > > > > > > > > copy, use
>> > > > > > > > > > > or
>> > > > > > > > > > > >> >
>> disclose to
>> > > > > > > > > > > >> >
>> >>
>> > > > others
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > this message
>> > > > > or
>> > > > > > any
>> > > > > > > attachment.
>> > > > > > > > > Please also
>> > > > > > > > > > notify the
>> > > > > > > > > > > >> >
>> sender by
>> > > > > > > > > > > >> >
>> >>
>> > > > replying
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > to this email
>> > > > > > or by telephone
>> > > > > > > > > (+44(020 7896
>> > > > > > > > > > 0011) and
>> > > > > > > > > > > >> then
>> > > > > > > > > > > >> > delete
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > email
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > and any copies
>> > > > > > of it.
>> > > > > > > Opinions,
>> > > > > > > > > conclusion
>> > > > > > > > > > (etc) that
>> > > > > > > > > > > do
>> > > > > > > > > > > >> > not
>> relate
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > official
>> > > > > > business of this
>> > > > > > > > company
>> > > > > > > > > shall be
>> > > > > > > > > > understood
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> neither
>> > > > > > > > > > > >> >
>> >>
>> > > > given nor
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > endorsed by
>> > > > > it.
>> > > > > > IG is a
>> > > > > > > trading
>> > > > > > > > name
>> > > > > > > > > of IG
>> > > > > > > > > > Markets
>> > > > > > > > > > > >> Limited
>> > > > > > > > > > > >> > (a
>> company
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > registered in
>> > > > > > England and
>> > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 04008957)
>> > > > > > > > > > > >> > and IG
>> > > > > > > > > > > >> >
>> >>
>> > > Index
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Limited (a
>> > > > > > company registered
>> > > > > > > in
>> > > > > > > > > England and
>> > > > > > > > > > Wales,
>> > > > > > > > > > > >> >
>> company
>> > number
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > 01190902).
>> > > > > > Registered address
>> > > > > > > at
>> > > > > > > > > Cannon Bridge
>> > > > > > > > > > House,
>> > > > > > > > > > > 25
>> > > > > > > > > > > >> >
>> Dowgate
>> > > > > > > > > > > >> >
>> >>
>> > > Hill,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > London EC4R
>> > > > > > 2YA. Both IG
>> > > > > > > Markets
>> > > > > > > > > Limited
>> > > > > > > > > > (register
>> > > > > > > > > > > >> number
>> > > > > > > > > > > >> >
>> 195355)
>> > > > > > > > > > > >> >
>> >>
>> > and
>> > > > IG
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Index Limited
>> > > > > > (register number
>> > > > > > > > > 114059) are
>> > > > > > > > > > authorised
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> regulated
>> > > > > > > > > > > >> >
>> >>
>> > by
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Financial
>> > > > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > --
>> > > > > > > > > > > >> >
>> >>
>> > > > Nacho - Ignacio
>> > > > > > Solis -
>> > > > > > > > isolis@igso.net
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > The
>> > > information
>> > > > contained in
>> > > > > > this email is
>> > > > > > > > strictly
>> > > > > > > > > > confidential and
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> use of
>> > the
>> > > > addressee only,
>> > > > > > unless
>> > > > > > > otherwise
>> > > > > > > > > indicated. If
>> > > > > > > > > > you are
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > the
>> intended
>> > > > recipient, please
>> > > > > > do not read,
>> > > > > > > > copy, use
>> > > > > > > > > or
>> > > > > > > > > > disclose to
>> > > > > > > > > > > >> others
>> > > > > > > > > > > >> > this
>> message
>> > > or
>> > > > any
>> > > > > attachment.
>> > > > > > Please also
>> > > > > > > > notify
>> > > > > > > > > the sender by
>> > > > > > > > > > > >> replying
>> > > > > > > > > > > >> > to
>> this
>> > email
>> > > > or by telephone
>> > > > > > (+44(020 7896
>> > > > > > > > 0011) and
>> > > > > > > > > then
>> > > > > > > > > > delete the
>> > > > > > > > > > > >> email
>> > > > > > > > > > > >> > and
>> any
>> > copies
>> > > > of it.
>> > > > > Opinions,
>> > > > > > conclusion
>> > > > > > > > (etc) that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > understood
>> > > > > > > > > as
>> > > > > > > > > > neither given
>> > > > > > > > > > > >> nor
>> > > > > > > > > > > >> >
>> endorsed by
>> > > it.
>> > > > IG is a
>> > > > > trading
>> > > > > > name of IG
>> > > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >> >
>> registered
>> > in
>> > > > England and
>> > > > > > Wales, company
>> > > > > > > number
>> > > > > > > > > 04008957) and
>> > > > > > > > > > IG Index
>> > > > > > > > > > > >> >
>> Limited (a
>> > > > company registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > >> >
>> 01190902).
>> > > > Registered address
>> > > > > > at Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25
>> > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > >> >
>> London EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and
>> > > > > > > > > > > IG
>> > > > > > > > > > > >> > Index
>> > Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> Financial
>> > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> The
>> > information
>> > > > contained in
>> > > > > this
>> > > > > > email is
>> > > > > > > > strictly
>> > > > > > > > > confidential
>> > > > > > > > > > and for
>> > > > > > > > > > > >> the use
>> of the
>> > > > addressee only,
>> > > > > > unless otherwise
>> > > > > > > > > indicated. If you
>> > > > > > > > > > are
>> > > > > > > > > > > not
>> > > > > > > > > > > >> the
>> intended
>> > > > recipient, please
>> > > > > do
>> > > > > > not read,
>> > > > > > > copy,
>> > > > > > > > use
>> > > > > > > > > or disclose
>> > > > > > > > > > to
>> > > > > > > > > > > others
>> > > > > > > > > > > >> this
>> message
>> > or
>> > > > any attachment.
>> > > > > > Please also
>> > > > > > > > notify the
>> > > > > > > > > sender by
>> > > > > > > > > > > replying
>> > > > > > > > > > > >> to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the
>> > > > > > > > > > > email
>> > > > > > > > > > > >> and any
>> copies
>> > > of
>> > > > it. Opinions,
>> > > > > > conclusion (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > the
>> > > > > > > > > > > >> official
>> > > business
>> > > > of this
>> > > > > company
>> > > > > > shall be
>> > > > > > > > understood
>> > > > > > > > > as neither
>> > > > > > > > > > given
>> > > > > > > > > > > nor
>> > > > > > > > > > > >>
>> endorsed by
>> > it.
>> > > > IG is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >>
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG
>> > > > > > > > > > Index
>> > > > > > > > > > > >> Limited
>> (a
>> > > > company registered in
>> > > > > > England and
>> > > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 01190902).
>> > > > Registered address at
>> > > > > > Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25 Dowgate
>> > > > > > > > > > Hill,
>> > > > > > > > > > > >> London
>> EC4R
>> > 2YA.
>> > > > Both IG Markets
>> > > > > > Limited
>> > > > > > > (register
>> > > > > > > > > number 195355)
>> > > > > > > > > > and IG
>> > > > > > > > > > > >> Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> Financial
>> > > Conduct
>> > > > Authority.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > The
>> information
>> > > > contained in this
>> > > > > > email is strictly
>> > > > > > > > > confidential and
>> > > > > > > > > > for the use of
>> the
>> > > > addressee only, unless
>> > > > > > otherwise
>> > > > > > > > indicated.
>> > > > > > > > > If you are
>> > > > > > > > > > not the intended
>> > > > recipient, please do not
>> > > > > > read, copy, use
>> > > > > > > > or
>> > > > > > > > > disclose to
>> > > > > > > > > > others this
>> message or
>> > > any
>> > > > attachment.
>> > > > > > Please also notify
>> > > > > > > > the
>> > > > > > > > > sender by
>> > > > > > > > > > replying to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the email and any
>> > copies
>> > > > of it. Opinions,
>> > > > > > conclusion
>> > > > > > > (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to the
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > > understood as
>> > > > > > > > > > neither given nor
>> > > endorsed
>> > > > by it. IG is a
>> > > > > > trading name of
>> > > > > > > > IG
>> > > > > > > > > Markets
>> > > > > > > > > > Limited (a
>> company
>> > > > registered in England
>> > > > > > and Wales,
>> > > > > > > company
>> > > > > > > > > number
>> > > > > > > > > > 04008957) and IG
>> Index
>> > > > Limited (a company
>> > > > > > registered in
>> > > > > > > > England
>> > > > > > > > > and Wales,
>> > > > > > > > > > company number
>> > 01190902).
>> > > > Registered
>> > > > > > address at Cannon
>> > > > > > > > Bridge
>> > > > > > > > > House, 25
>> > > > > > > > > > Dowgate Hill,
>> London
>> > EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and IG
>> Index
>> > > > Limited (register
>> > > > > > number 114059) are
>> > > > > > > > > authorised and
>> > > > > > > > > > regulated by the
>> > > Financial
>> > > > Conduct
>> > > > > > Authority.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The information
>> contained in
>> > > > this email is
>> > > > > > strictly
>> > > > > > > confidential
>> > > > > > > > and
>> > > > > > > > > for the use of the
>> addressee
>> > only,
>> > > > unless otherwise
>> > > > > > indicated. If
>> > > > > > > > you are
>> > > > > > > > > not the intended recipient,
>> > please
>> > > > do not read,
>> > > > > > copy, use or
>> > > > > > > > disclose to
>> > > > > > > > > others this message or any
>> > > > attachment. Please also
>> > > > > > notify the
>> > > > > > > sender
>> > > > > > > > by
>> > > > > > > > > replying to this email or
>> by
>> > > > telephone (+44(020
>> > > > > 7896
>> > > > > > 0011) and then
>> > > > > > > > delete
>> > > > > > > > > the email and any copies
>> of it.
>> > > > Opinions,
>> > > > > conclusion
>> > > > > > (etc) that do
>> > > > > > > > not
>> > > > > > > > > relate to the official
>> business
>> > of
>> > > > this company
>> > > > > > shall be understood
>> > > > > > > > as
>> > > > > > > > > neither given nor endorsed
>> by it.
>> > > IG
>> > > > is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a company
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG Index
>> Limited (a
>> > > > company
>> > > > > registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company number 01190902).
>> > > Registered
>> > > > address at
>> > > > > > Cannon Bridge
>> > > > > > > House,
>> > > > > > > > 25
>> > > > > > > > > Dowgate Hill, London EC4R
>> 2YA.
>> > Both
>> > > > IG Markets
>> > > > > > Limited (register
>> > > > > > > > number
>> > > > > > > > > 195355) and IG Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > authorised
>> > > > > > > > and
>> > > > > > > > > regulated by the Financial
>> > Conduct
>> > > > Authority.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > The information contained in
>> this email
>> > > is
>> > > > strictly
>> > > > > > confidential and for
>> > > > > > > > the use of the addressee only,
>> unless
>> > > > otherwise
>> > > > > indicated.
>> > > > > > If you are not
>> > > > > > > > the intended recipient, please
>> do not
>> > > > read, copy, use or
>> > > > > > disclose to
>> > > > > > > others
>> > > > > > > > this message or any attachment.
>> Please
>> > > > also notify the
>> > > > > > sender by replying
>> > > > > > > > to this email or by telephone
>> (+44(020
>> > > > 7896 0011) and
>> > > > > then
>> > > > > > delete the
>> > > > > > > email
>> > > > > > > > and any copies of it. Opinions,
>> > > conclusion
>> > > > (etc) that do
>> > > > > > not relate to
>> > > > > > > the
>> > > > > > > > official business of this
>> company shall
>> > > be
>> > > > understood as
>> > > > > > neither given
>> > > > > > > nor
>> > > > > > > > endorsed by it. IG is a trading
>> name of
>> > > IG
>> > > > Markets
>> > > > > Limited
>> > > > > > (a company
>> > > > > > > > registered in England and Wales,
>> > company
>> > > > number 04008957)
>> > > > > > and IG Index
>> > > > > > > > Limited (a company registered in
>> > England
>> > > > and Wales,
>> > > > > > company number
>> > > > > > > > 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill,
>> > > > > > > > London EC4R 2YA. Both IG Markets
>> > Limited
>> > > > (register number
>> > > > > > 195355) and IG
>> > > > > > > > Index Limited (register number
>> 114059)
>> > > are
>> > > > authorised and
>> > > > > > regulated by
>> > > > > > > the
>> > > > > > > > Financial Conduct Authority.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > The information contained in this email is
>> strictly
>> > > > confidential and
>> > > > > > for the use of the addressee only, unless
>> otherwise
>> > > > indicated. If you are
>> > > > > > not the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or disclose to
>> > > > > > others this message or any attachment. Please
>> also
>> > notify
>> > > > the sender by
>> > > > > > replying to this email or by telephone (+44(020
>> 7896
>> > > 0011)
>> > > > and then
>> > > > > delete
>> > > > > > the email and any copies of it. Opinions,
>> conclusion
>> > > (etc)
>> > > > that do not
>> > > > > > relate to the official business of this company
>> shall
>> > be
>> > > > understood as
>> > > > > > neither given nor endorsed by it. IG is a
>> trading name
>> > of
>> > > > IG Markets
>> > > > > > Limited (a company registered in England and
>> Wales,
>> > > > company number
>> > > > > > 04008957) and IG Index Limited (a company
>> registered in
>> > > > England and
>> > > > > Wales,
>> > > > > > company number 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register number
>> > > > > > 195355) and IG Index Limited (register number
>> 114059)
>> > are
>> > > > authorised and
>> > > > > > regulated by the Financial Conduct Authority.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential
>> and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If
>> you are
>> > not
>> > > > the intended recipient, please do not read, copy, use or
>> disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then
>> delete the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company
>> number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number
>> 195355) and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and
>> regulated by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > >
>> >
>>
>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
--
Nacho - Ignacio Solis - isolis@igso.net
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
RE KIP-98 discussion,
No worries, can you add it back in on KIP-98, then we go with Headers at the end with this KIP. And I think all are in agreement right?
Cheers
Mike
On 22/02/2017, 00:02, "Michael Pearce" <Mi...@ig.com> wrote:
This is a bit of why I’d rather have headers as self-contained, e.g. it has a headersLen and headers data, as we’d have to introduce one or the other, and it seems cleaner to add the headers than add back in the valuelen.
On 21/02/2017, 23:57, "Michael Pearce" <Mi...@ig.com> wrote:
Hi Jason
I looked at the doc, it seems the length of the following data sections are being left described in your doc in some place similar to how we have on the wiki, and in others not.
E.g. your doc has this as the proposed message described (which is in blue and red colouring)
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
Value => data [OPTIONAL]
Note I notice there is a missing
ValueLen => uintVar [OPTIONAL]
Else we wouldn’t be able to put headers at the end, without its length being defined. So either we can leave ValueLen removed, as is in your new format, and keep headers
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
HeadersLen => uintVar [OPTIONAL]
Headers => data [OPTIONAL]
Value => data [OPTIONAL]
Headers => [Header]
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
OR we will need to re-introduce ValueLen which seems to be removed in Kip-98
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
ValueLen => uintVar [OPTIONAL]
Value => data [OPTIONAL]
Headers => Header1, Header2 … HeaderN
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
Cheers
Mike
On 21/02/2017, 20:45, "ignacio.solis@gmail.com on behalf of Ignacio Solis" <ignacio.solis@gmail.com on behalf of isolis@igso.net> wrote:
Short answer:
Headers at the end are fine, this proposal looks ok.
Longer answer:
At LinkedIn we considered headers at the end multiple times. We even
had a backwards compatible headers-in-V design with headers at the
end.
The main reason headers come at the beginning of messaging systems is
in case they affect the rest of the message. This is either because
of parsing differences (a header might indicate how to parse the
remainder of the message), or performance optimizations (a header
might indicate that you can discard the message and can stop parsing).
I believe that neither of these cases are common in Kafka (at this
time) so having the headers are the end shouldn't be a big issue.
From a performance perspective we may incur an extra page cache miss
but at the rate at which kafka does messaging it won't bottleneck
things.
Headers at the end have one potential advantage, that in simple
circumstances, naive systems can append headers to a message without
doing a lot of memory manipulation. Normally not that big of a deal
since we could always use scatter/gather methods if needed, but maybe
some simple systems will benefit from that. ... Not to mention most
Java programmers might not pay attention to these type of performance
issues.
Anyway...
Overall, there are many things that I would have done differently in
the proposal (int keys, total header length, no VarInts, etc. ), but
the cost of these is small in comparison to the benefit we get from
having headers at all.
If we are all ok with this design then I would be the first to +1 the format.
Nacho
Side note: Shouldn't headers at the end be called footers? We could
always call it metadata :-).
On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com> wrote:
> I think I'd like some opinions from some of the active chaps at LinkedIn so it's not just a two way decision (or any others for that matter).
>
> Any opinions on the below or no real concerns / indifferent by anyone with this change?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Tuesday, February 21, 2017 6:59:38 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Jun's suggestion gives us the best of both worlds. Because the headers are
> the last field in the record and you have the total size of the record,
> then you can easily get the size of headers in bytes. So they can be copied
> without parsing and we can do the lazy initialization without an extra cost.
>
> The message set notion has changed in KIP-98 so that the message set can
> have its own fields. See here for more detail:
> https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
> What we're proposing to change here is the schema for the message (aka
> record) itself. We'll update that document depending on the outcome of the
> discussion here.
>
> -Jason
>
>
> On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> This works whilst no other feature needs similar. I prefer to keep it
>> contained/encapsulated it then allows the bytes of the headers to also be
>> copied transferred as one without any parsing what so ever, but if you
>> really feel strongly on the below ill go with it, just want to flag this
>> concern.
>>
>> Btw, the below notation seems to be different to the notation used in
>> previous kips like kip-32 which I took the notation from, where the length
>> of the following bytes for byte arrays is clearly defined. So im 100% on
>> what you propose could you define it in this original styled notation?
>>
>>
>> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>>
>> Hey Jun, that's a good thought. Maybe then the full schema could look
>> like
>> this:
>>
>> Record =>
>> Length => uintVar
>> Attributes => int8
>> TimestampDelta => intVar
>> OffsetDelta => uintVar
>> Key => bytes
>> Value => bytes
>> Headers => [HeaderKey HeaderValue]
>> HeaderKey => string
>> HeaderValue => bytes
>>
>> (where the bytes and string type use a variable length size field)
>>
>> Does that work for you, Michael?
>>
>> -Jason
>>
>> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > Regarding the overhead of skipping the headers in the message.
>> Another way
>> > could be to add the headers at the end of the message after the
>> value part.
>> > This way, there is no need to skip the headers unless you really
>> need it.
>> > This will allow us to represent the headers explicitly as an array of
>> > key/value pairs without the skipping overhead when headers are not
>> needed.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
>> >
>> > wrote:
>> >
>> > > Michael,
>> > >
>> > > Here's a link to the branch with the message format changes:
>> > > https://github.com/confluentinc/kafka/tree/
>> exactly-once-message-format.
>> > > The
>> > > class to modify to add headers is EosLogRecord (name will
>> definitely
>> > change
>> > > prior to merging in trunk). This branch uses attributes to
>> indicate null
>> > > key and value. I really don't have a strong preference for or
>> against
>> > this
>> > > as long as we're consistent. I do feel it's kind of a pity to need
>> the
>> > > signed version of the varint just to be able to indicate null. For
>> > headers,
>> > > it's an easier call because we don't actually need to represent
>> null
>> > > headers (using size zero will work fine).
>> > >
>> > > -Jason
>> > >
>> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > We can make it, implement Iterable<Header>, no concerns on this.
>> > > >
>> > > > Ive noted this on the KIP.
>> > > >
>> > > >
>> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > > Btw Jason, can you share the branch with the VarInts
>> helper/util
>> > > class
>> > > > that KIP-98 will be introducing and also will have the current
>> modified
>> > > > (key and value lengths) message, so we can look at, thanks.
>> > > >
>> > > > On 20/02/2017, 01:11, "Michael Pearce" <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > > >
>> > > > RE “ProducerRecord and ConsumerRecord are immutable
>> > currently”. I
>> > > > beg to differ on this, if I set a byte[] as my value of my
>> record, I
>> > can
>> > > > later call value() and then alter the bytes in the array. As
>> such it is
>> > > > very much mutable.
>> > > >
>> > > > Headers also will anyhow by nature be mutable on
>> produce, as
>> > you
>> > > > add to them.
>> > > >
>> > > >
>> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
>> jason@confluent.io>
>> > > > wrote:
>> > > >
>> > > > Re; having a dedicated Headers object. I think it
>> makes
>> > > sense.
>> > > > Maybe it
>> > > > could implement Iterable<Header>?
>> > > >
>> > > > One thing I'm not too sure about is mutability. Both
>> the
>> > > > ProducerRecord and
>> > > > ConsumerRecord are immutable currently. It would be
>> nice to
>> > > > keep it that
>> > > > way. Perhaps Headers could itself be immutable and
>> passed
>> > > > through the
>> > > > constructors?
>> > > >
>> > > > Just going through how this is, I don’t believe
>> this would
>> > > > provide much
>> > > > > saving. If issue trying to resolve is around
>> > object/garbage
>> > > > creation of
>> > > > > Map.Entry objects.
>> > > >
>> > > >
>> > > > It's definitely better because we only allocate the
>> space
>> > we
>> > > > will use, but
>> > > > I assume most of the actual memory cost is in the
>> header
>> > data
>> > > > itself which
>> > > > we can't do much about. Maybe we can start with the
>> simple
>> > > > array
>> > > > representation and optimize from there after we find
>> it's
>> > > > worthwhile to do
>> > > > so? If we use the Headers object, we can change this
>> later
>> > > > without breaking
>> > > > the API.
>> > > >
>> > > > -Jason
>> > > >
>> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
>> > > > radai.rosenblatt@gmail.com> wrote:
>> > > >
>> > > > > "(By the way, doesn't it feel a bit odd that we
>> seem to
>> > be
>> > > > designing a
>> > > > > feature which is optimized for people not using?)"
>> > > > >
>> > > > > very. this (i claim :-) ) is the result of very
>> intense
>> > > > opposition to the
>> > > > > usefulness of the feature early on, and not a real
>> design
>> > > > goal
>> > > > >
>> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Whilst if having an array of a Header[] would
>> mean
>> > > getting
>> > > > this through,
>> > > > > > im happy changing this, as already done.
>> > > > > >
>> > > > > > Just going through how this is, I don’t believe
>> this
>> > > > would provide much
>> > > > > > saving. If issue trying to resolve is around
>> > > > object/garbage creation of
>> > > > > > Map.Entry objects.
>> > > > > >
>> > > > > > All you’re doing here is replacing the equiv of a
>> > > > HashMap.Node
>> > > > > (Map.Entry)
>> > > > > > which simply holds reference to key and value
>> objects
>> > > with
>> > > > a custom
>> > > > > variant.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com> wrote:
>> > > > > >
>> > > > > > On point 1 & 2 Ive updated KIP to show
>> varints (and
>> > > > removed the bit
>> > > > > > flag). (on the assumption KIP 98 is getting the
>> > agreement
>> > > > the protocol is
>> > > > > > moving from int32 to varInts as standard)
>> > > > > >
>> > > > > > On point 3 ive updated to use an array of
>> Header
>> > > > class, instead of a
>> > > > > > MultiMap in the Headers class object
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > On points 1 and 2 I agree.
>> > > > > >
>> > > > > > This also affects kip-98, I should
>> expect this
>> > > > resolved before
>> > > > > > that vote also passes. If it is accepted there
>> (I’m
>> > > > assuming this is
>> > > > > > getting discussed on that KIP? As you’re driving
>> the
>> > move
>> > > > to VarInts), I
>> > > > > am
>> > > > > > happy to make this KIP will simply follow suit to
>> > > whatever
>> > > > is agreed in
>> > > > > > KIP-98.
>> > > > > >
>> > > > > > On 3) Agreed this is a simpler form, as
>> long as
>> > > no
>> > > > one is
>> > > > > > expecting hashmap lookup performance ( O(1) ), I
>> still
>> > > > would prefer an
>> > > > > > encapsulated class, so if we find that holding
>> it as an
>> > > > array in future
>> > > > > is
>> > > > > > causing some perf issues, the internals are not
>> exposed
>> > > to
>> > > > end users,
>> > > > > > allowing the internal structure to move to a map.
>> > > > > >
>> > > > > > Can we compromise on?
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Headers headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Headers {
>> > > > > > Header[] headers;
>> > > > > >
>> > > > > > add(String key, byte[] value)
>> > > > > > Collection<byte[]> get(String key)
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
>> > > > jason@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > >
>> > > > > >
>> > > > > > This is exactly what I have been
>> trying to
>> > > get
>> > > > at. The use
>> > > > > > cases documented
>> > > > > > here are middleware:
>> > > > > > https://cwiki.apache.org/
>> > > > confluence/display/KAFKA/A+
>> > > > > > Case+for+Kafka+Headers.
>> > > > > > If headers are intended for the
>> application
>> > > > space as well,
>> > > > > the
>> > > > > > document
>> > > > > > should be updated accordingly so
>> that it is
>> > > an
>> > > > explicit
>> > > > > design
>> > > > > > goal and not
>> > > > > > an unfortunate design byproduct.
>> There may
>> > be
>> > > > some
>> > > > > > disagreement on whether
>> > > > > > it _should_ be a design goal, but it
>> may
>> > not
>> > > > matter much
>> > > > > since
>> > > > > > the current
>> > > > > > interceptor API is probably
>> insufficient
>> > for
>> > > > middleware
>> > > > > > applications (I
>> > > > > > haven't thought of a way to do this
>> that
>> > > isn't
>> > > > cumbersome).
>> > > > > >
>> > > > > > That aside, the three unresolved
>> points for
>> > > me
>> > > > are the
>> > > > > > following:
>> > > > > >
>> > > > > > 1. The use of varints. As a one-off
>> for use
>> > > > only with record
>> > > > > > headers, the
>> > > > > > case was weak, but if we are using
>> them
>> > > > throughout the
>> > > > > message
>> > > > > > format, then
>> > > > > > we should do so here as well. The
>> > additional
>> > > > complexity is
>> > > > > > minimal and
>> > > > > > early performance testing fully
>> justifies
>> > > > their use.
>> > > > > >
>> > > > > > 2. If varints are used, the case for
>> using
>> > > > attributes to
>> > > > > > indicate null
>> > > > > > headers also becomes weak. We only
>> add one
>> > > > additional byte in
>> > > > > > each message
>> > > > > > if there are no headers. Whatever
>> the case,
>> > > > let us be
>> > > > > > consistent with how
>> > > > > > we treat null keys and values.
>> > > > > >
>> > > > > > 3. We have apparently agreed that the
>> > broker
>> > > > will validate
>> > > > > > headers (please
>> > > > > > add this to the KIP). That being the
>> case,
>> > I
>> > > > would prefer to
>> > > > > > use Kafka's
>> > > > > > conventional format for arrays. The
>> > downside
>> > > > is that it takes
>> > > > > > more work to
>> > > > > > skip over the headers on the
>> consumer,
>> > though
>> > > > it's unclear if
>> > > > > > the cost
>> > > > > > would matter in practice. Some
>> concern was
>> > > > previously
>> > > > > > expressed about the
>> > > > > > allocation of maps. An alternative
>> would be
>> > > to
>> > > > use arrays,
>> > > > > i.e.
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Header[] headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > > This would work nicely with the
>> > conventional
>> > > > array format and
>> > > > > > my guess is
>> > > > > > it would obviate the need do any lazy
>> > > > initialization. If we
>> > > > > > use the map as
>> > > > > > is currently documented, then it is
>> > possible
>> > > > with either
>> > > > > > representation to
>> > > > > > slice the headers and initialize them
>> > lazily.
>> > > > Either way, it
>> > > > > > might be a
>> > > > > > good idea to use a separate object to
>> > > > represent the headers
>> > > > > in
>> > > > > > case we need
>> > > > > > to modify it in the future in some
>> way.
>> > > > > >
>> > > > > > (By the way, doesn't it feel a bit
>> odd that
>> > > we
>> > > > seem to be
>> > > > > > designing a
>> > > > > > feature which is optimized for
>> people not
>> > > > using?)
>> > > > > >
>> > > > > >
>> > > > > > If we can resolve these points, then
>> at
>> > least
>> > > > you will get my
>> > > > > > vote.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jason
>> > > > > >
>> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
>> radai <
>> > > > > > radai.rosenblatt@gmail.com> wrote:
>> > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > > > IIUC samze currently has headers
>> "in-V"
>> > and
>> > > > would just
>> > > > > > switch over to kafka
>> > > > > > > headers if they exist.
>> > > > > > > im sure plenty of other users of
>> kafka
>> > > would
>> > > > have a use for
>> > > > > > headers.
>> > > > > > > im pretty sure use cases exist
>> around
>> > > > shuffling data
>> > > > > > into/out-of kafka
>> > > > > > > (kafka connect or equivalent) where
>> > > metadata
>> > > > from one end
>> > > > > > could copied over
>> > > > > > > to the other (S3, for example uses
>> http
>> > > > headers for
>> > > > > > user-accessible
>> > > > > > > metadata). it will be kafka client
>> code
>> > > > getting/setting
>> > > > > > those headers. not
>> > > > > > > an interceptor.
>> > > > > > >
>> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
>> Michael
>> > > > Pearce <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > For APM single event tracing,
>> need
>> > access
>> > > > to the header
>> > > > > at
>> > > > > > the point of
>> > > > > > > > processing on the processing
>> thread.
>> > > > > > > >
>> > > > > > > > As such interceptors will not
>> work/be
>> > > > suitable for these,
>> > > > > > due to the fact
>> > > > > > > > they act on the ConsumerRecords
>> as a
>> > > > batch, before the
>> > > > > > handling thread
>> > > > > > > can
>> > > > > > > > split out and process per
>> message which
>> > > is
>> > > > the point
>> > > > > these
>> > > > > > tools will
>> > > > > > > need
>> > > > > > > > to continue to transaction
>> tracing.
>> > > > > > > >
>> > > > > > > > Like wise tools and infra pieces
>> will
>> > > need
>> > > > access to the
>> > > > > > message outside
>> > > > > > > > the interceptor realm.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 17/02/2017, 21:26, "Jason
>> > Gustafson" <
>> > > > > > jason@confluent.io> wrote:
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > So the broker will parse the
>> > headers
>> > > > and validate
>> > > > > > them. Good. The
>> > > > > > > only
>> > > > > > > > reason remaining that I can
>> see to
>> > > > leave the headers
>> > > > > > as a byte array
>> > > > > > > > is to
>> > > > > > > > make it easier for the
>> client to
>> > skip
>> > > > past them. Are
>> > > > > > we sure this is
>> > > > > > > > not
>> > > > > > > > premature optimization? Are
>> there
>> > any
>> > > > performance
>> > > > > > results which show
>> > > > > > > > that
>> > > > > > > > this is worthwhile?
>> > > > > > > >
>> > > > > > > > What’s the issue with
>> exposing a
>> > > > method getHeaders on
>> > > > > > the
>> > > > > > > > producer/consumer
>> > > > > > > > > record? It doesn’t break
>> > anything.
>> > > > We don’t need
>> > > > > any
>> > > > > > special
>> > > > > > > version.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > See my previous explanation.
>> What I
>> > > am
>> > > > trying to
>> > > > > > resist is the
>> > > > > > > headers
>> > > > > > > > becoming a general
>> > application-level
>> > > > facility. The
>> > > > > > primary use case
>> > > > > > > as
>> > > > > > > > far
>> > > > > > > > as I can tell is middleware,
>> which
>> > is
>> > > > the use case
>> > > > > > that the
>> > > > > > > > interceptors
>> > > > > > > > are providing.
>> > > > > > > >
>> > > > > > > > Current batch consumer model
>> and
>> > > > consumer
>> > > > > interceptors
>> > > > > > don’t work
>> > > > > > > where
>> > > > > > > > > headers need to be acted
>> on at
>> > per
>> > > > message level at
>> > > > > > time of
>> > > > > > > > processing,
>> > > > > > > > > very case is APM (the core
>> one),
>> > > > where the header
>> > > > > > value is used to
>> > > > > > > > continue
>> > > > > > > > > tracing.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I still don't understand the
>> point
>> > > > about batching.
>> > > > > The
>> > > > > > consumer
>> > > > > > > > records are
>> > > > > > > > exposed as a batch in the
>> consumer
>> > > > interceptor, but
>> > > > > > you can still
>> > > > > > > > iterate
>> > > > > > > > them individually. It is no
>> > different
>> > > > for the
>> > > > > consumer
>> > > > > > API itself.
>> > > > > > > >
>> > > > > > > > -Jason
>> > > > > > > >
>> > > > > > > > On Fri, Feb 17, 2017 at
>> 12:48 PM,
>> > > > Michael Pearce <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Re:
>> > > > > > > > >
>> > > > > > > > > “ The point about
>> creation of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a slice
>> of
>> > bytes
>> > > > until the time
>> > > > > > it is accessed.”
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > >
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:37,
>> "Michael
>> > > > Pearce" <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > What’s the issue with
>> > exposing
>> > > a
>> > > > method
>> > > > > > getHeaders on the
>> > > > > > > > > producer/consumer record?
>> It
>> > > doesn’t
>> > > > break
>> > > > > anything.
>> > > > > > We don’t need
>> > > > > > > > any
>> > > > > > > > > special version.
>> > > > > > > > >
>> > > > > > > > > Current batch consumer
>> model
>> > > and
>> > > > consumer
>> > > > > > interceptors don’t
>> > > > > > > work
>> > > > > > > > > where headers need to be
>> acted on
>> > > at
>> > > > per message
>> > > > > > level at time of
>> > > > > > > > > processing, very case is
>> APM (the
>> > > > core one), where
>> > > > > > the header value
>> > > > > > > > is used
>> > > > > > > > > to continue tracing.
>> JMS/HTTP etc
>> > > > all expose these,
>> > > > > > without issues.
>> > > > > > > > I would
>> > > > > > > > > NOT want to lock this down
>> to
>> > only
>> > > > be usable
>> > > > > > accessible via
>> > > > > > > > interceptors,
>> > > > > > > > > as we’d fail on one of the
>> main
>> > > > goals.
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Mike
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:21,
>> "Jason
>> > > > Gustafson" <
>> > > > > > jason@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > The point about
>> creation
>> > of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a
>> slice of
>> > > > bytes until the
>> > > > > > time it is
>> > > > > > > > accessed.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > Yes exactly we
>> have
>> > > access
>> > > > to the records
>> > > > > > thus why the
>> > > > > > > > header
>> > > > > > > > > should be
>> > > > > > > > > > accessible via
>> it and
>> > not
>> > > > hidden for only
>> > > > > > interceptors to
>> > > > > > > > access.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > As explained
>> above, the
>> > > > point is to make
>> > > > > the
>> > > > > > intended usage
>> > > > > > > > clear.
>> > > > > > > > > Applications should
>> > > continue
>> > > > to rely on the
>> > > > > > key/value
>> > > > > > > fields
>> > > > > > > > to
>> > > > > > > > > serialize
>> > > > > > > > > their own headers,
>> and it
>> > > > would be more
>> > > > > > ideal if we can
>> > > > > > > avoid
>> > > > > > > > > leaking
>> > > > > > > > > third-party
>> headers into
>> > > > applications. This
>> > > > > > is difficult to
>> > > > > > > > do
>> > > > > > > > > with the
>> > > > > > > > > current
>> interceptors
>> > > because
>> > > > they share the
>> > > > > > record objects
>> > > > > > > > with
>> > > > > > > > > the common
>> > > > > > > > > API. What I had in
>> mind
>> > is
>> > > > something like
>> > > > > an
>> > > > > > extension of
>> > > > > > > the
>> > > > > > > > > current
>> > > > > > > > > interceptors which
>> > exposed
>> > > a
>> > > > different
>> > > > > > object (e.g.
>> > > > > > > > > `RecordAndHeaders`).
>> > > > > > > > > The challenge is
>> for
>> > > MM-like
>> > > > use cases. Let
>> > > > > > me see if I can
>> > > > > > > > come
>> > > > > > > > > up with a
>> > > > > > > > > concrete proposal
>> for
>> > that
>> > > > problem.
>> > > > > > > > >
>> > > > > > > > > -Jason
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 17,
>> 2017 at
>> > > > 11:55 AM, Michael
>> > > > > > Pearce <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I am happy to
>> move the
>> > > > definition of the
>> > > > > > header into the
>> > > > > > > > message
>> > > > > > > > > body, but
>> > > > > > > > > > would cause us
>> not to
>> > > lazy
>> > > > > > initialise/parse the headers,
>> > > > > > > as
>> > > > > > > > > obviously, we
>> > > > > > > > > > would have to
>> traverse
>> > > > these reading the
>> > > > > > message.
>> > > > > > > > > >
>> > > > > > > > > > This was
>> actually one
>> > of
>> > > > Jay’s requests:
>> > > > > > > > > >
>> > > > > > > > > > “ 2. I think
>> we
>> > should
>> > > > think about
>> > > > > > creating the Map
>> > > > > > > > lazily to
>> > > > > > > > > avoid
>> > > > > > > > > > parsing out
>> all the
>> > > > headers into
>> > > > > > little objects.
>> > > > > > > > HashMaps
>> > > > > > > > > themselves
>> > > > > > > > > > are
>> > > > > > > > > > kind of
>> expensive
>> > and
>> > > > the consumer is
>> > > > > > very perf
>> > > > > > > > sensitive so
>> > > > > > > > > and making
>> > > > > > > > > > gazillions of
>> > > hashmaps
>> > > > that may or
>> > > > > may
>> > > > > > not get used
>> > > > > > > is
>> > > > > > > > > probably a bad
>> > > > > > > > > > idea.”
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 17/02/2017,
>> 19:44,
>> > > > "Michael Pearce" <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Yes exactly
>> we have
>> > > > access to the
>> > > > > > records thus why
>> > > > > > > the
>> > > > > > > > > header should
>> > > > > > > > > > be accessible
>> via it
>> > and
>> > > > not hidden for
>> > > > > > only interceptors
>> > > > > > > > to
>> > > > > > > > > access.
>> > > > > > > > > >
>> > > > > > > > > > Sent using
>> OWA for
>> > > > iPhone
>> > > > > > > > > >
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > From: Magnus
>> > > Edenhill <
>> > > > > > magnus@edenhill.se>
>> > > > > > > > > > Sent: Friday,
>> > > February
>> > > > 17, 2017
>> > > > > > 7:34:49 PM
>> > > > > > > > > > To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > Subject: Re:
>> > > [DISCUSS]
>> > > > KIP-82 - Add
>> > > > > > Record Headers
>> > > > > > > > > >
>> > > > > > > > > > Big +1 on
>> VarInts.
>> > > > > > > > > > CPUs are
>> fast,
>> > memory
>> > > > is slow.
>> > > > > > > > > >
>> > > > > > > > > > I agree with
>> Jason
>> > > > that we'll want to
>> > > > > > continue
>> > > > > > > > verifying
>> > > > > > > > > messages,
>> > > > > > > > > > including
>> their
>> > > > headers, so while I
>> > > > > > appreciate the
>> > > > > > > > idea of
>> > > > > > > > > the opaque
>> > > > > > > > > > header blob
>> it
>> > won't
>> > > > be useful in
>> > > > > > practice.
>> > > > > > > > > >
>> > > > > > > > > > /Magnus
>> > > > > > > > > >
>> > > > > > > > > > 2017-02-17
>> 10:41
>> > > > GMT-08:00 Jason
>> > > > > > Gustafson <
>> > > > > > > > > jason@confluent.io>:
>> > > > > > > > > >
>> > > > > > > > > > > Sorry, my
>> > mistake.
>> > > > The consumer
>> > > > > > interceptor is per
>> > > > > > > > batch,
>> > > > > > > > > though I'm
>> > > > > > > > > > not
>> > > > > > > > > > > sure
>> that's an
>> > > > actual limitation
>> > > > > > since you still
>> > > > > > > have
>> > > > > > > > > access to the
>> > > > > > > > > > > individual
>> > records.
>> > > > > > > > > > >
>> > > > > > > > > > > -Jason
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri,
>> Feb 17,
>> > > 2017
>> > > > at 10:39 AM,
>> > > > > > Jason Gustafson <
>> > > > > > > > > >
>> jason@confluent.io>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Re
>> headers as
>> > > byte
>> > > > array and
>> > > > > > future use by
>> > > > > > > broker.
>> > > > > > > > This
>> > > > > > > > > doesn't
>> > > > > > > > > > take away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yeah, I
>> didn't
>> > > say
>> > > > it was
>> > > > > > difficult, only
>> > > > > > > awkward.
>> > > > > > > > You
>> > > > > > > > > wouldn't
>> > > > > > > > > > write the
>> > > > > > > > > > > > schema
>> that way
>> > > if
>> > > > you were
>> > > > > > planning to use it on
>> > > > > > > > the
>> > > > > > > > > brokers from
>> > > > > > > > > > the
>> > > > > > > > > > > >
>> beginning. Note
>> > > > also that one of
>> > > > > > the benefits of
>> > > > > > > > letting
>> > > > > > > > > the broker
>> > > > > > > > > > > >
>> understand
>> > > headers
>> > > > is that it can
>> > > > > > validate that
>> > > > > > > > they are
>> > > > > > > > > properly
>> > > > > > > > > > > >
>> formatted. If
>> > > cost
>> > > > is the only
>> > > > > > concern, we should
>> > > > > > > > > confirm its
>> > > > > > > > > > impact
>> > > > > > > > > > > > through
>> > > > performance testing.
>> > > > > > > > > > > >
>> > > > > > > > > > > > One of
>> the key
>> > > use
>> > > > cases requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'm not
>> sure I
>> > > > understand the
>> > > > > > point about
>> > > > > > > batching.
>> > > > > > > > > Interceptors
>> > > > > > > > > > are
>> > > > > > > > > > > > applied
>> > > > per-message, right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > My
>> intent on
>> > > > interceptors is to
>> > > > > > keep the usage of
>> > > > > > > > headers
>> > > > > > > > > > well-defined so
>> > > > > > > > > > > > that
>> they don't
>> > > > start leaking
>> > > > > > unnecessarily into
>> > > > > > > > > applications. My
>> > > > > > > > > > guess
>> > > > > > > > > > > is
>> > > > > > > > > > > > that it's
>> > > probably
>> > > > inevitable,
>> > > > > but
>> > > > > > isolating it
>> > > > > > > in
>> > > > > > > > the
>> > > > > > > > > > interceptors
>> would
>> > > > > > > > > > > > at least
>> give
>> > > > people a second
>> > > > > > thought before
>> > > > > > > > deciding to
>> > > > > > > > > use it.
>> > > > > > > > > > The main
>> > > > > > > > > > > >
>> challenge in my
>> > > > mind is figuring
>> > > > > > out how an MM
>> > > > > > > use
>> > > > > > > > case
>> > > > > > > > > would
>> > > > > > > > > > work. It
>> > > > > > > > > > > > would be
>> more
>> > > > cumbersome to
>> > > > > > replicate headers
>> > > > > > > > through an
>> > > > > > > > > > interceptor,
>> > > > > > > > > > > > though
>> arguably
>> > > MM
>> > > > should be
>> > > > > > working at a lower
>> > > > > > > > level
>> > > > > > > > > anyway.
>> > > > > > > > > > > >
>> > > > > > > > > > > > -Jason
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri,
>> Feb 17,
>> > > > 2017 at 10:16 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Re
>> headers
>> > > > available on the
>> > > > > > record va
>> > > > > > > > interceptors only
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> One of
>> the key
>> > > > use cases
>> > > > > requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> This
>> anyhow is
>> > > > similar to
>> > > > > > jms/http/amqp where
>> > > > > > > > headers
>> > > > > > > > > are
>> > > > > > > > > > available to
>> > > > > > > > > > > >>
>> consuming
>> > > > applications.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Re
>> headers as
>> > > > byte array and
>> > > > > > future use by
>> > > > > > > > broker. This
>> > > > > > > > > doesn't
>> > > > > > > > > > take
>> > > > > > > > > > > away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Sent
>> using OWA
>> > > > for iPhone
>> > > > > > > > > > > >>
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > > >> From:
>> Jason
>> > > > Gustafson <
>> > > > > > jason@confluent.io>
>> > > > > > > > > > > >> Sent:
>> Friday,
>> > > > February 17, 2017
>> > > > > > 5:55:42 PM
>> > > > > > > > > > > >> To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > > >>
>> Subject: Re:
>> > > > [DISCUSS] KIP-82 -
>> > > > > > Add Record
>> > > > > > > Headers
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Yes,
>> that is
>> > > what
>> > > > we are
>> > > > > > proposing, so using
>> > > > > > > > varints
>> > > > > > > > > for headers
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >>
>> consistent
>> > with
>> > > > the rest of the
>> > > > > > message. We have
>> > > > > > > > used
>> > > > > > > > > static
>> > > > > > > > > > helper
>> > > > > > > > > > > >> methods
>> > > > > > > > > > > >> in our
>> > prototype
>> > > > implementation.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> The
>> cost of
>> > > > parsing, we want to
>> > > > > > parse/interpret
>> > > > > > > > the
>> > > > > > > > > headers
>> > > > > > > > > > lazily (this
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > a key
>> point
>> > > > brought up earlier
>> > > > > > in discussions)
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I'm a
>> bit
>> > > > skeptical of this. Has
>> > > > > > anyone done the
>> > > > > > > > > performance
>> > > > > > > > > > testing? I
>> > > > > > > > > > > >> can
>> > > > > > > > > > > >> probably
>> > > > implement it and test
>> > > > > it
>> > > > > > if no one else
>> > > > > > > > has. I
>> > > > > > > > > was also
>> > > > > > > > > > under
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> impression
>> > that
>> > > > there may be use
>> > > > > > cases down the
>> > > > > > > > road
>> > > > > > > > > where the
>> > > > > > > > > > broker
>> > > > > > > > > > > >> would
>> > > > > > > > > > > >> need to
>> > > interpret
>> > > > headers. That
>> > > > > > wouldn't be off
>> > > > > > > > the
>> > > > > > > > > table in the
>> > > > > > > > > > future
>> > > > > > > > > > > if
>> > > > > > > > > > > >> it's
>> > represented
>> > > > as bytes, but
>> > > > > it
>> > > > > > would be quite
>> > > > > > > > a bit
>> > > > > > > > > more
>> > > > > > > > > > awkward,
>> > > > > > > > > > > >> right?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> By the
>> way,
>> > one
>> > > > question I have
>> > > > > > been wondering
>> > > > > > > > about. My
>> > > > > > > > > > understanding
>> > > > > > > > > > > is
>> > > > > > > > > > > >> that
>> headers
>> > are
>> > > > primarily for
>> > > > > > use cases where a
>> > > > > > > > > third-party
>> > > > > > > > > > components
>> > > > > > > > > > > >> wants to
>> > enrich
>> > > > messages without
>> > > > > > needing to
>> > > > > > > > understand
>> > > > > > > > > or modify
>> > > > > > > > > > the
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> of the
>> message
>> > > > key and value.
>> > > > > For
>> > > > > > the
>> > > > > > > applications
>> > > > > > > > > which directly
>> > > > > > > > > > > produce
>> > > > > > > > > > > >> and
>> consume
>> > the
>> > > > messages and
>> > > > > > control the
>> > > > > > > key/value
>> > > > > > > > > schema
>> > > > > > > > > > directly, it
>> > > > > > > > > > > >> seems
>> we would
>> > > > rather have them
>> > > > > > implement
>> > > > > > > headers
>> > > > > > > > > directly in
>> > > > > > > > > > their own
>> > > > > > > > > > > >> schema.
>> > > Supposing
>> > > > for the sake
>> > > > > of
>> > > > > > argument that
>> > > > > > > > it was
>> > > > > > > > > possible,
>> > > > > > > > > > my
>> > > > > > > > > > > >>
>> question is
>> > > > whether it be
>> > > > > > sufficient to expose
>> > > > > > > the
>> > > > > > > > > headers in the
>> > > > > > > > > > > >>
>> interceptor
>> > API
>> > > > and not in the
>> > > > > > common API?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> -Jason
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Fri,
>> Feb
>> > 17,
>> > > > 2017 at 3:26 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > On
>> the point
>> > > of
>> > > > varInts
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Cheers
>> > > > > > > > > > > >> > Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > 17/02/2017,
>> > > > 11:22, "Michael
>> > > > > > Pearce" <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On the
>> > > > point re: headers
>> > > > > in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > being a
>> > > > > > > > > > byte
>> > > > > > > > > > > array
>> > > > > > > > > > > >> > and
>> not a
>> > > count
>> > > > of elements
>> > > > > > followed by the
>> > > > > > > > elements.
>> > > > > > > > > Again
>> > > > > > > > > > this was
>> > > > > > > > > > > >> >
>> > > > discussed/argued previously.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> It was
>> > > > agreed on for a few
>> > > > > > reasons some of
>> > > > > > > > which
>> > > > > > > > > you have
>> > > > > > > > > > > obviously
>> > > > > > > > > > > >> >
>> picked up
>> > on:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Broker
>> > is
>> > > > able to pass it
>> > > > > > through opaquely
>> > > > > > > > > > > >> >
>> The cost
>> > > of
>> > > > parsing, we
>> > > > > > want to
>> > > > > > > > parse/interpret
>> > > > > > > > > the headers
>> > > > > > > > > > lazily
>> > > > > > > > > > > >> > (this
>> is a
>> > key
>> > > > point brought
>> > > > > up
>> > > > > > earlier in
>> > > > > > > > > discussions)
>> > > > > > > > > > > >> >
>> Headers
>> > > can
>> > > > be copied from
>> > > > > > consumer record
>> > > > > > > > to
>> > > > > > > > > producer
>> > > > > > > > > > record (aka
>> > > > > > > > > > > >> > mirror
>> > makers
>> > > > etc) without
>> > > > > > parsing if no
>> > > > > > > > changes are
>> > > > > > > > > being made
>> > > > > > > > > > or
>> > > > > > > > > > > being
>> > > > > > > > > > > >> >
>> looked at.
>> > > > > > > > > > > >> >
>> Keeps
>> > the
>> > > > broker agnostic
>> > > > > > to the format
>> > > > > > > > > > > >> >
>> You need
>> > > an
>> > > > int32 either
>> > > > > > for the byte size
>> > > > > > > > of the
>> > > > > > > > > headers,
>> > > > > > > > > > or for
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > count
>> of
>> > > > elements, so
>> > > > > overheads
>> > > > > > are the same,
>> > > > > > > > but
>> > > > > > > > > with going
>> > > > > > > > > > with an
>> > > > > > > > > > > >> opaque
>> > > > > > > > > > > >> > byte
>> array
>> > has
>> > > > the above
>> > > > > > advantages.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Cheers
>> > > > > > > > > > > >> >
>> Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > > > 17/02/2017, 02:50,
>> > > > > > "Jason Gustafson" <
>> > > > > > > > > jason@confluent.io
>> > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > Sorry,
>> > > > should have
>> > > > > > noted that the
>> > > > > > > > performance
>> > > > > > > > > testing
>> > > > > > > > > > was done
>> > > > > > > > > > > >> > using
>> the
>> > > > > > > > > > > >> >
>> > > > producer performance
>> > > > > > tool shipped with
>> > > > > > > > Kafka.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > -Jason
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On
>> > > Thu,
>> > > > Feb 16, 2017
>> > > > > at
>> > > > > > 6:44 PM, Jason
>> > > > > > > > > Gustafson <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> >
>> > Hey
>> > > > Nacho,
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > I've
>> > > > compared
>> > > > > > performance of our
>> > > > > > > > KIP-98
>> > > > > > > > > > implementation
>> with
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> without
>> > > > > > > > > > > >> >
>> >
>> > > > varints. For
>> > > > > messages
>> > > > > > around 128
>> > > > > > > > bytes, we
>> > > > > > > > > see an
>> > > > > > > > > > increase
>> > > > > > > > > > > in
>> > > > > > > > > > > >> >
>> throughput
>> > of
>> > > > > > > > > > > >> >
>> >
>> > > about
>> > > > 30% using the
>> > > > > > default
>> > > > > > > > configuration
>> > > > > > > > > settings.
>> > > > > > > > > > At 256
>> > > > > > > > > > > >> >
>> bytes, the
>> > > > > > > > > > > >> >
>> >
>> > > > increase is around
>> > > > > > 16%. Obviously
>> > > > > > > the
>> > > > > > > > > performance
>> > > > > > > > > > converge
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> messages get
>> > > > > > > > > > > >> >
>> >
>> > > > larger, but it seems
>> > > > > > well worth the
>> > > > > > > > cost.
>> > > > > > > > > Note that
>> > > > > > > > > > we are
>> > > > > > > > > > > >> also
>> > > > > > > > > > > >> >
>> seeing a
>> > > > > > > > > > > >> >
>> >
>> > > > substantial
>> > > > > > performance increase
>> > > > > > > > against
>> > > > > > > > > trunk
>> > > > > > > > > > primarily
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >
>> > much
>> > > > more efficient
>> > > > > > packing that
>> > > > > > > > varints
>> > > > > > > > > provide us.
>> > > > > > > > > > > Anything
>> > > > > > > > > > > >> >
>> adding to
>> > > > > > > > > > > >> >
>> >
>> > > > message overhead,
>> > > > > > such as record
>> > > > > > > > headers,
>> > > > > > > > > would only
>> > > > > > > > > > > increase
>> > > > > > > > > > > >> > the
>> relative
>> > > > > > > > > > > >> >
>> >
>> > > > difference. (Of
>> > > > > > course take these
>> > > > > > > > numbers
>> > > > > > > > > with a
>> > > > > > > > > > grain of
>> > > > > > > > > > > salt
>> > > > > > > > > > > >> > since
>> I have
>> > > > > > > > > > > >> >
>> >
>> > only
>> > > > used the
>> > > > > default
>> > > > > > settings with
>> > > > > > > > both
>> > > > > > > > > the producer
>> > > > > > > > > > and
>> > > > > > > > > > > >> broker
>> > > > > > > > > > > >> > on my
>> > > > > > > > > > > >> >
>> >
>> > > local
>> > > > machine. We
>> > > > > > intend to provide
>> > > > > > > > more
>> > > > > > > > > extensive
>> > > > > > > > > > > performance
>> > > > > > > > > > > >> >
>> details as
>> > > > > > > > > > > >> >
>> >
>> > part
>> > > > of the work for
>> > > > > > KIP-98.)
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > The
>> > > > implementation
>> > > > > we
>> > > > > > are using is
>> > > > > > > > from
>> > > > > > > > > protobuf (
>> > > > > > > > > > > >> >
>> >
>> > > > > > https://developers.google.com/
>> > > > > > > > > > >
>> > > > protocol-buffers/docs/encoding
>> > > > > > > > > > > >> ),
>> > > > > > > > > > > >> > which
>> is
>> > > > > > > > > > > >> >
>> >
>> > also
>> > > > used in HBase.
>> > > > > > It is trivial to
>> > > > > > > > > implement and as
>> > > > > > > > > > far
>> > > > > > > > > > > as I
>> > > > > > > > > > > >> > know
>> doesn't
>> > > > > > > > > > > >> >
>> >
>> > > > suffer from the
>> > > > > > aliasing problem you
>> > > > > > > > are
>> > > > > > > > > describing. I
>> > > > > > > > > > > checked
>> > > > > > > > > > > >> > with
>> Magnus
>> > > > > > > > > > > >> >
>> >
>> > (the
>> > > > author of
>> > > > > > librdkafka) and he
>> > > > > > > > agreed
>> > > > > > > > > that the
>> > > > > > > > > > savings
>> > > > > > > > > > > >> seemed
>> > > > > > > > > > > >> > worth
>> the
>> > > > > > > > > > > >> >
>> >
>> > cost
>> > > > of
>> > > > > > implementation.
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > -Jason
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > On
>> > > > Thu, Feb 16, 2017
>> > > > > > at 4:32 PM,
>> > > > > > > > Ignacio
>> > > > > > > > > Solis <
>> > > > > > > > > > > >>
>> > isolis@igso.net
>> > > >
>> > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >>
>> > > > -VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > I'm
>> > > > one of the
>> > > > > > people (if not the
>> > > > > > > > most)
>> > > > > > > > > opposed to
>> > > > > > > > > > VarInts.
>> > > > > > > > > > > >> >
>> VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > have
>> > > > a place, but
>> > > > > > this is not it.
>> > > > > > > > (We
>> > > > > > > > > had a large
>> > > > > > > > > > > >>
>> discussion
>> > > > > > > > > > > >> > about
>> > > > > > > > > > > >> >
>> >>
>> > > them
>> > > > at the
>> > > > > > beginning of KIP-82
>> > > > > > > time)
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > anybody has real
>> > > > > > life
>> > > > > > > performance
>> > > > > > > > > numbers of
>> > > > > > > > > > VarInts
>> > > > > > > > > > > >> >
>> improving
>> > > > > > > > > > > >> >
>> >>
>> > > > things or
>> > > > > > significantly reducing
>> > > > > > > > resources
>> > > > > > > > > I would
>> > > > > > > > > > like to
>> > > > > > > > > > > >> know
>> > > > > > > > > > > >> > what
>> > > > > > > > > > > >> >
>> >>
>> > > that
>> > > > case may be.
>> > > > > > Yes, you can save
>> > > > > > > > some
>> > > > > > > > > bytes here
>> > > > > > > > > > and
>> > > > > > > > > > > >> there,
>> > > > > > > > > > > >> > but
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > is probably
>> > > > > > insignificant to
>> > > > > > > the
>> > > > > > > > > overall system
>> > > > > > > > > > > behavior
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> >
>> >>
>> > > > storage
>> > > > > > requirements. -- I say
>> > > > > > > this
>> > > > > > > > with
>> > > > > > > > > respect to
>> > > > > > > > > > using
>> > > > > > > > > > > >> >
>> VarInts in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > protocol
>> > > > > itself,
>> > > > > > not as part of
>> > > > > > > > the
>> > > > > > > > > data.
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > VarInts require you
>> > > > > > to parse the
>> > > > > > > Int
>> > > > > > > > > before using it
>> > > > > > > > > > and
>> > > > > > > > > > > >> >
>> depending on
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > encoding they
>> > > > > > can suffer from
>> > > > > > > > aliasing
>> > > > > > > > > (multiple
>> > > > > > > > > > > >> >
>> > > representations
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > the same
>> > > > > value).
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > Why
>> > > > add complexity?
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > Nacho
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > On
>> > > > Thu, Feb 16,
>> > > > > 2017
>> > > > > > at 10:29 AM,
>> > > > > > > > Colin
>> > > > > > > > > McCabe <
>> > > > > > > > > > > >> >
>> > > > cmccabe@apache.org>
>> > > > > > > > > > > >> >
>> >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >> >
>> > > +1
>> > > > for varints
>> > > > > > here-- it would
>> > > > > > > save
>> > > > > > > > > quite a bit of
>> > > > > > > > > > space.
>> > > > > > > > > > > >> > They
>> are
>> > > > > > > > > > > >> >
>> >> >
>> > > > pretty quick to
>> > > > > > implement as
>> > > > > > > well.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > > I
>> > > > think it makes
>> > > > > > sense for values
>> > > > > > > > to be
>> > > > > > > > > byte
>> > > > > > > > > > arrays.
>> > > > > > > > > > > Users
>> > > > > > > > > > > >> > might
>> want
>> > > > > > > > > > > >> >
>> >> >
>> > > to
>> > > > attach
>> > > > > > arbitrary payloads;
>> > > > > > > they
>> > > > > > > > > shouldn't be
>> > > > > > > > > > forced to
>> > > > > > > > > > > >> >
>> serialize
>> > > > > > > > > > > >> >
>> >> >
>> > > > everything to
>> > > > > Java
>> > > > > > strings.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > best,
>> > > > > > > > > > > >> >
>> >> >
>> > > > Colin
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > On
>> > > > Thu, Feb 16,
>> > > > > > 2017, at 09:52,
>> > > > > > > > Jason
>> > > > > > > > > Gustafson
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hey Michael,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hmm, I guess the
>> > > > > > point of
>> > > > > > > > representing
>> > > > > > > > > it as
>> > > > > > > > > > bytes is to
>> > > > > > > > > > > >> > allow
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > broker
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > to pass it
>> > > > > > through opaquely? Is
>> > > > > > > > the
>> > > > > > > > > cost of
>> > > > > > > > > > parsing
>> > > > > > > > > > > them a
>> > > > > > > > > > > >> >
>> concern, or
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > are
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > we simply trying
>> > > > > > to ensure that
>> > > > > > > > the
>> > > > > > > > > broker stays
>> > > > > > > > > > > agnostic
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > format?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On varints, I
>> > > > > > think adding
>> > > > > > > > support for
>> > > > > > > > > them makes
>> > > > > > > > > > less
>> > > > > > > > > > > >> sense
>> > > > > > > > > > > >> > for an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > isolated use
>> > > > > > case, but as part
>> > > > > > > of
>> > > > > > > > a
>> > > > > > > > > more holistic
>> > > > > > > > > > change
>> > > > > > > > > > > >> > (such
>> as
>> > what
>> > > > > > > > > > > >> >
>> >>
>> > we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > have proposed in
>> > > > > > KIP-98), I
>> > > > > > > think
>> > > > > > > > they
>> > > > > > > > > are
>> > > > > > > > > > justifiable.
>> > > > > > > > > > > If
>> > > > > > > > > > > >> > we
>> add them,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > then the need to
>> > > > > > use attributes
>> > > > > > > > becomes
>> > > > > > > > > quite a
>> > > > > > > > > > bit
>> > > > > > > > > > > >> weaker,
>> > > > > > > > > > > >> >
>> right? The
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > other thing I
>> > > > > > find slightly odd
>> > > > > > > > is the
>> > > > > > > > > fact that
>> > > > > > > > > > null
>> > > > > > > > > > > >> >
>> headers has
>> > no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > actual
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > semantic meaning
>> > > > > > for the message
>> > > > > > > > > (unlike null
>> > > > > > > > > > keys and
>> > > > > > > > > > > >> >
>> values). It
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > a
>> > > > space
>> > > > > > optimization. It seems a
>> > > > > > > > bit
>> > > > > > > > > better to
>> > > > > > > > > > always
>> > > > > > > > > > > use
>> > > > > > > > > > > >> > size
>> 0 to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > indicate having
>> > > > > > no headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Overall, the
>> > > > > main
>> > > > > > point is
>> > > > > > > > ensuring
>> > > > > > > > > that the
>> > > > > > > > > > message
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> >
>> remains
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > consistent,
>> > > > > > either within the
>> > > > > > > > larger
>> > > > > > > > > protocol, or
>> > > > > > > > > > at a
>> > > > > > > > > > > >> >
>> minimum
>> > within
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > message itself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On Thu, Feb 16,
>> > > > > > 2017 at 6:39 AM,
>> > > > > > > > > Michael Pearce <
>> > > > > > > > > > > >> >
>> > > > Michael.Pearce@ig.com
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Hi Jason,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 1) in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > the headers
>> > > > > > > > > > are
>> > > > > > > > > > > >> simply
>> > > > > > > > > > > >> > a byte
>> > > > > > > > > > > >> >
>> >>
>> > > > array,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > as like the
>> > > > > key
>> > > > > > or value, this
>> > > > > > > > is to
>> > > > > > > > > clearly
>> > > > > > > > > > demarcate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> header in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > core message.
>> > > > > > Then the header
>> > > > > > > > byte
>> > > > > > > > > array in the
>> > > > > > > > > > core
>> > > > > > > > > > > >> >
>> message is
>> > an
>> > > > > > > > > > > >> >
>> >>
>> > > > array of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key, value
>> > > > > > pairs. This is what
>> > > > > > > > it is
>> > > > > > > > > denoting.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Then this
>> > > > > would
>> > > > > > be I guess in
>> > > > > > > > the
>> > > > > > > > > given
>> > > > > > > > > > notation:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Headers =>
>> > > > > > [KeyLength, Key,
>> > > > > > > > > ValueLength, Value]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > KeyLength
>> > > > > > => int32
>> > > > > > > > > <-----------------NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised key
>> > > > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Key =>
>> > > > > bytes
>> > > > > > > > > <---------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > string
>> > (UTF8)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > bytes of the
>> > > > > > header key
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > ValueLength
>> > > > > > => int32
>> > > > > > > > > <-------------- NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised
>> > > > > > header value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Value =>
>> > > > > > bytes
>> > > > > > > > > <-------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > form
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The key length
>> > > > > > and value
>> > > > > > > length
>> > > > > > > > is
>> > > > > > > > > matching the
>> > > > > > > > > > way
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> protocol is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > defined in the
>> > > > > > core message
>> > > > > > > > currently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 2)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Var sized
>> > > > > ints,
>> > > > > > this was
>> > > > > > > > discussed
>> > > > > > > > > much earlier
>> > > > > > > > > > on, in
>> > > > > > > > > > > >> > fact
>> I had
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > suggested it
>> > > > > > myself (with
>> > > > > > > Hadoop
>> > > > > > > > > references),
>> > > > > > > > > > the
>> > > > > > > > > > > >> >
>> complexity
>> > of
>> > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compared to
>> > > > > > having a simpler
>> > > > > > > > protocol
>> > > > > > > > > was
>> > > > > > > > > > argued and
>> > > > > > > > > > > >> >
>> agreed it
>> > > > > > > > > > > >> >
>> >>
>> > > > wasn’t worth
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the complexity
>> > > > > > as all other
>> > > > > > > > clients
>> > > > > > > > > in other
>> > > > > > > > > > languages
>> > > > > > > > > > > >> > would
>> need
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > > > ensure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > theyre using
>> > > > > > the right var
>> > > > > > > size
>> > > > > > > > > algorithm, as
>> > > > > > > > > > there
>> > > > > > > > > > > is a
>> > > > > > > > > > > >> > few.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 3)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > We did the
>> > > > > > attributes,
>> > > > > > > optional
>> > > > > > > > > approach as
>> > > > > > > > > > originally
>> > > > > > > > > > > >> > there
>> was
>> > > > > > > > > > > >> >
>> >>
>> > > > marked
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > concern that
>> > > > > > headers would
>> > > > > > > > cause a
>> > > > > > > > > message size
>> > > > > > > > > > > overhead
>> > > > > > > > > > > >> > for
>> others,
>> > > > > > > > > > > >> >
>> >>
>> > who
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > don’t want
>> > > > > > them. As such this
>> > > > > > > > is the
>> > > > > > > > > clean
>> > > > > > > > > > solution to
>> > > > > > > > > > > >> >
>> achieve
>> > that.
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that no longer
>> > > > > > holds, and we
>> > > > > > > > don’t
>> > > > > > > > > care that we
>> > > > > > > > > > add
>> > > > > > > > > > > >> 4bytes
>> > > > > > > > > > > >> >
>> overhead,
>> > > > > > > > > > > >> >
>> >>
>> > > then
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > im happy to
>> > > > > > remove.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > I’m personally
>> > > > > > in favour of
>> > > > > > > > keeping
>> > > > > > > > > the message
>> > > > > > > > > > as
>> > > > > > > > > > > small
>> > > > > > > > > > > >> > as
>> possible
>> > > > > > > > > > > >> >
>> >>
>> > so
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > people don’t
>> > > > > > get shocks in
>> > > > > > > perf
>> > > > > > > > and
>> > > > > > > > > throughputs
>> > > > > > > > > > dues
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> message
>> > size,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > unless they
>> > > > > > actively use the
>> > > > > > > > feature,
>> > > > > > > > > as such I
>> > > > > > > > > > do
>> > > > > > > > > > > >> prefer
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > attribute bit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wise feature
>> > > > > > flag approach
>> > > > > > > > myself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On 16/02/2017,
>> > > > > > 05:40, "Jason
>> > > > > > > > > Gustafson" <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > We have
>> > > > > > proposed a few
>> > > > > > > > > significant changes
>> > > > > > > > > > to the
>> > > > > > > > > > > >> >
>> message
>> > format
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > KIP-98
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > which now
>> > > > > > seems likely to
>> > > > > > > > pass
>> > > > > > > > > (perhaps
>> > > > > > > > > > with some
>> > > > > > > > > > > >> >
>> iterations
>> > on
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > implementation details).
>> > > > > > > It
>> > > > > > > > would
>> > > > > > > > > be good
>> > > > > > > > > > to try
>> > > > > > > > > > > and
>> > > > > > > > > > > >> >
>> coordinate
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > changes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > in both of
>> > > > > > the proposals
>> > > > > > > to
>> > > > > > > > make
>> > > > > > > > > sure they
>> > > > > > > > > > are
>> > > > > > > > > > > >> >
>> consistent
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compatible.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I think
>> > > > > > using the
>> > > > > > > > attributes to
>> > > > > > > > > indicate
>> > > > > > > > > > null
>> > > > > > > > > > > >> headers
>> > > > > > > > > > > >> > is a
>> > > > > > > > > > > >> >
>> >>
>> > > > reasonable
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > approach.
>> > > > > > We have proposed
>> > > > > > > > to do
>> > > > > > > > > the same
>> > > > > > > > > > thing
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> message
>> > > > > > > > > > > >> >
>> >>
>> > key
>> > > > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > value.
>> > > > > That
>> > > > > > said, I
>> > > > > > > > sympathize
>> > > > > > > > > with Jay's
>> > > > > > > > > > > argument.
>> > > > > > > > > > > >> > Having
>> > > > > > > > > > > >> >
>> >>
>> > > > multiple
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > ways to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > specify a
>> > > > > > null value
>> > > > > > > > increases
>> > > > > > > > > the overall
>> > > > > > > > > > > >>
>> complexity
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > You
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > can see
>> > > > > > this just from the
>> > > > > > > > fact
>> > > > > > > > > that you
>> > > > > > > > > > need the
>> > > > > > > > > > > >> > extra
>> > verbiage
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > protocol
>> > > > > > specification in
>> > > > > > > > this
>> > > > > > > > > KIP and in
>> > > > > > > > > > KIP-98
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> describe the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > dependence
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > between
>> > > > > the
>> > > > > > fields and the
>> > > > > > > > > attributes. It
>> > > > > > > > > > seems
>> > > > > > > > > > > >> like a
>> > > > > > > > > > > >> >
>> slippery
>> > > > > > > > > > > >> >
>> >>
>> > > slope
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > if
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > you start
>> > > > > > allowing
>> > > > > > > different
>> > > > > > > > > request types
>> > > > > > > > > > to
>> > > > > > > > > > > >> >
>> implement
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification differently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > You can
>> > > > > > also argue that
>> > > > > > > the
>> > > > > > > > > messages
>> > > > > > > > > > already are
>> > > > > > > > > > > and
>> > > > > > > > > > > >> > are
>> likely
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > remain a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > special
>> > > > > > case. For example,
>> > > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > currently no
>> > > > > > > > > > > >> >
>> generality
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > how
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > compressed
>> > > > > > message sets
>> > > > > > > are
>> > > > > > > > > represented
>> > > > > > > > > > that would
>> > > > > > > > > > > >> be
>> > > > > > > > > > > >> >
>> applicable
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > other
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > request
>> > > > > > types. Some might
>> > > > > > > > see this
>> > > > > > > > > > divergence as
>> > > > > > > > > > > an
>> > > > > > > > > > > >> >
>> unfortunate
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > deficiency
>> > > > > > which should be
>> > > > > > > > fixed;
>> > > > > > > > > others
>> > > > > > > > > > might see
>> > > > > > > > > > > >> it
>> > > > > > > > > > > >> > as
>> sort of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > inevitability of needing
>> > > > > > > to
>> > > > > > > > > optimize where
>> > > > > > > > > > it
>> > > > > > > > > > > counts
>> > > > > > > > > > > >> > most.
>> I'm
>> > > > > > > > > > > >> >
>> >>
>> > > > probably
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > somewhere
>> > > > > > in between, but
>> > > > > > > I
>> > > > > > > > think
>> > > > > > > > > we
>> > > > > > > > > > probably all
>> > > > > > > > > > > >> > share
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > intuition
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol should be
>> > > > > > > kept
>> > > > > > > > as
>> > > > > > > > > consistent as
>> > > > > > > > > > > >>
>> possible.
>> > > > > > > > > > > >> > With
>> that
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > mind,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > here are a
>> > > > > > few comments:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 1. One
>> > > > > > thing I found a
>> > > > > > > > little odd
>> > > > > > > > > when
>> > > > > > > > > > reading the
>> > > > > > > > > > > >> >
>> current
>> > > > > > > > > > > >> >
>> >>
>> > > > proposal is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > headers
>> > > > > > are both
>> > > > > > > > represented
>> > > > > > > > > as an
>> > > > > > > > > > array of
>> > > > > > > > > > > >> bytes
>> > > > > > > > > > > >> > and
>> as an
>> > > > > > > > > > > >> >
>> >>
>> > > array
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > key/value
>> > > > > > pairs. I'd
>> > > > > > > > probably
>> > > > > > > > > suggest
>> > > > > > > > > > something
>> > > > > > > > > > > like
>> > > > > > > > > > > >> > this:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > [HeaderKey
>> > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > HeaderKey
>> > > > > > => String
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue => Bytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > An array
>> > > > > in
>> > > > > > the Kafka
>> > > > > > > > protocol is
>> > > > > > > > > > represented as a
>> > > > > > > > > > > >> > 4-byte
>> > integer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > indicating
>> > > > > > the number of
>> > > > > > > > elements
>> > > > > > > > > in the
>> > > > > > > > > > array
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > serialization of the
>> > > > > > > > elements.
>> > > > > > > > > Unless I'm
>> > > > > > > > > > > >> >
>> > > > misunderstanding, what
>> > > > > > > > > > > >> >
>> >>
>> > you
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > instead is
>> > > > > > the total size
>> > > > > > > > of the
>> > > > > > > > > headers in
>> > > > > > > > > > bytes
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > elements.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I'm not
>> > > > > > sure I see any
>> > > > > > > > reason for
>> > > > > > > > > this
>> > > > > > > > > > > >>
>> inconsistency.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 2. In
>> > > > > > KIP-98, we've
>> > > > > > > > introduced
>> > > > > > > > > > variable-length
>> > > > > > > > > > > >> integer
>> > > > > > > > > > > >> >
>> fields.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Effectively,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > we've
>> > > > > > enriched (or
>> > > > > > > > "complicated"
>> > > > > > > > > as Jay
>> > > > > > > > > > might say
>> > > > > > > > > > > ;)
>> > > > > > > > > > > >> > the
>> protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification to include
>> > > > > > > the
>> > > > > > > > > following
>> > > > > > > > > > types:
>> > > > > > > > > > > >> VarInt,
>> > > > > > > > > > > >> >
>> VarLong,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > UnsignedVarInt and
>> > > > > > > > > UnsignedVarLong.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Along with
>> > > > > > these
>> > > > > > > > primitives, we
>> > > > > > > > > could
>> > > > > > > > > > introduce
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> following
>> > > > > > > > > > > >> >
>> >>
>> > > > types:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeArray =>
>> > > > > > > > NumberOfItems
>> > > > > > > > > Item1 Item2
>> > > > > > > > > > .. ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItems =>
>> > > > > > > > UnsignedVarInt
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeNullableArray =>
>> > > > > > > > > NumberOfItemsOrNull
>> > > > > > > > > > Item1
>> > > > > > > > > > > >> > Item2
>> ..
>> > ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItemsOrNull =>
>> > > > > > > > VarInt
>> > > > > > > > > (-1 means
>> > > > > > > > > > null)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > And
>> > > > > > similarly for the
>> > > > > > > > `String`
>> > > > > > > > > and `Bytes`
>> > > > > > > > > > types.
>> > > > > > > > > > > >> > These
>> types
>> > > > > > > > > > > >> >
>> >>
>> > can
>> > > > save
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > considerable amount of
>> > > > > > > > space in
>> > > > > > > > > this
>> > > > > > > > > > proposal
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > they
>> can
>> > > > > > > > > > > >> >
>> >>
>> > be
>> > > > used
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > both the
>> > > > > > number of headers
>> > > > > > > > > included in the
>> > > > > > > > > > message
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > lengths of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > header
>> > > > > keys
>> > > > > > and values. We
>> > > > > > > > could
>> > > > > > > > > do this
>> > > > > > > > > > instead:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > > > VarSizeArray[HeaderKey
>> > > > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > HeaderKey
>> > > > > > =>
>> > > > > > > VarSizeString
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue =>
>> > > > > > > > VarSizeBytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Combining
>> > > > > > the savings from
>> > > > > > > > the
>> > > > > > > > > use of
>> > > > > > > > > > variable
>> > > > > > > > > > > >> length
>> > > > > > > > > > > >> >
>> fields, the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > benefit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > of using
>> > > > > > the attributes to
>> > > > > > > > > represent null
>> > > > > > > > > > seems
>> > > > > > > > > > > >> pretty
>> > > > > > > > > > > >> > small.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 3.
>> > > > > > Whichever way we go
>> > > > > > > > (whether
>> > > > > > > > > we use the
>> > > > > > > > > > > >>
>> attributes
>> > > > > > > > > > > >> > or
>> not), we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > should at
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > least be
>> > > > > > consistent
>> > > > > > > between
>> > > > > > > > this
>> > > > > > > > > KIP and
>> > > > > > > > > > KIP-98.
>> > > > > > > > > > > It
>> > > > > > > > > > > >> > would
>> be
>> > very
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > strange
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > to have
>> > > > > two
>> > > > > > ways to
>> > > > > > > > represent
>> > > > > > > > > null values
>> > > > > > > > > > in the
>> > > > > > > > > > > >> same
>> > > > > > > > > > > >> >
>> schema.
>> > > > > > > > > > > >> >
>> >>
>> > > > Either
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > way is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > OK with
>> > > > > me.
>> > > > > > I think some
>> > > > > > > > > message-level
>> > > > > > > > > > >
>> optimizations
>> > > > > > > > > > > >> > are
>> > > > > > > > > > > >> >
>> >>
>> > > > justifiable,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > but
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > savings
>> > > > > > here seem
>> > > > > > > > minimal (a
>> > > > > > > > > few bytes
>> > > > > > > > > > per
>> > > > > > > > > > > >> >
>> message), so
>> > > > > > > > > > > >> >
>> >>
>> > > > maybe it's
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > worth the
>> > > > > > cost of letting
>> > > > > > > > the
>> > > > > > > > > message
>> > > > > > > > > > diverge even
>> > > > > > > > > > > >> >
>> further from
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > rest of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > On Wed,
>> > > > > Feb
>> > > > > > 15, 2017 at
>> > > > > > > > 8:52 AM,
>> > > > > > > > > radai <
>> > > > > > > > > > > >> >
>> >>
>> > > > > > radai.rosenblatt@gmail.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > I've
>> > > > > > trimmed the inline
>> > > > > > > > > contents as this
>> > > > > > > > > > mail is
>> > > > > > > > > > > >> >
>> getting too
>> > > > > > > > > > > >> >
>> >>
>> > big
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > apache
>> > > > > > mailing list
>> > > > > > > > software to
>> > > > > > > > > deliver
>> > > > > > > > > > :-(
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 1. the
>> > > > > > important thing
>> > > > > > > for
>> > > > > > > > > > interoperability
>> is
>> > > > > > > > > > > for
>> > > > > > > > > > > >> >
>> different
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "interested
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > parties"
>> > > > > > (plugins, infra
>> > > > > > > > > layers/wrappers,
>> > > > > > > > > > > >>
>> user-code)
>> > > > > > > > > > > >> > to be
>> > > > > > > > > > > >> >
>> >>
>> > > able
>> > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > stick
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > pieces
>> > > > > of
>> > > > > > metadata onto
>> > > > > > > > msgs
>> > > > > > > > > without
>> > > > > > > > > > getting in
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> >
>> other's
>> > > > > > > > > > > >> >
>> >>
>> > > way.
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > common
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > key
>> > > > > > scheme (Strings, as
>> > > > > > > > of the
>> > > > > > > > > time of
>> > > > > > > > > > this
>> > > > > > > > > > > >> >
>> writing?) is
>> > > all
>> > > > > > > > > > > >> >
>> >>
>> > > thats
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > required
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > for
>> > > > > that.
>> > > > > > it is assumed
>> > > > > > > > that
>> > > > > > > > > the other end
>> > > > > > > > > > > >> >
>> interested
>> > in
>> > > > any
>> > > > > > > > > > > >> >
>> >>
>> > > such
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > piece of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > metadata
>> > > > > > knows the
>> > > > > > > > encoding,
>> > > > > > > > > and byte[]
>> > > > > > > > > > provides
>> > > > > > > > > > > >> for
>> > > > > > > > > > > >> > the
>> most
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > flexibility.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > i
>> > > > > believe
>> > > > > > this is the
>> > > > > > > same
>> > > > > > > > > logic behind
>> > > > > > > > > > core
>> > > > > > > > > > > kafka
>> > > > > > > > > > > >> > being
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > byte[]/byte[]
>> > > > > -
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Strings
>> > > > > > are more
>> > > > > > > "usable"
>> > > > > > > > but
>> > > > > > > > > bytes are
>> > > > > > > > > > flexible
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > so
>> were
>> > > > > > > > > > > >> >
>> >>
>> > > > chosen.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Also -
>> > > > > > core kafka doesnt
>> > > > > > > > even
>> > > > > > > > > do that
>> > > > > > > > > > good of a
>> > > > > > > > > > > >> job
>> > > > > > > > > > > >> > on
>> > > > > > > > > > > >> >
>> >>
>> > > > usability of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > payload
>> > > > > > (example - i
>> > > > > > > have
>> > > > > > > > to
>> > > > > > > > > specify the
>> > > > > > > > > > nop
>> > > > > > > > > > > >> byte[]
>> > > > > > > > > > > >> >
>> "decoders"
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > explicitly
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > in
>> > > > > conf),
>> > > > > > and again
>> > > > > > > > sacrificies
>> > > > > > > > > usability
>> > > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > sake
>> of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > performance
>> > > > > (no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > convenient single-record
>> > > > > > > > > processing as
>> > > > > > > > > > poll is a
>> > > > > > > > > > > >> >
>> batch, lots
>> > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > obscure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > little
>> > > > > > config details
>> > > > > > > > exposing
>> > > > > > > > > internals
>> > > > > > > > > > of the
>> > > > > > > > > > > >> >
>> batching
>> > > > > > > > > > > >> >
>> >>
>> > > > mechanism,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > etc)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > this is
>> > > > > > also why i
>> > > > > > > really
>> > > > > > > > > dislike the
>> > > > > > > > > > idea of a
>> > > > > > > > > > > >> > "type
>> > system"
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > values,
>> > > > > > it further
>> > > > > > > > degrades the
>> > > > > > > > > > usability, adds
>> > > > > > > > > > > >> >
>> complexity
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > eventually get in
>> > > > > > > > people's way,
>> > > > > > > > > also, it
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >> > the
>> 2nd/3rd
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > home-group
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization mechanism
>> > > > > > > > in core
>> > > > > > > > > kafka
>> > > > > > > > > > (counting
>> > > > > > > > > > > 2
>> > > > > > > > > > > >> >
>> iterations
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "type
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > definition DSL")
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 2. this
>> > > > > > is an
>> > > > > > > > implementation
>> > > > > > > > > detail, and
>> > > > > > > > > > not
>> > > > > > > > > > > even
>> > > > > > > > > > > >> a
>> > > > > > > > > > > >> > very
>> "user
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > facing" one?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to the
>> > > > > > best of my
>> > > > > > > > understanding
>> > > > > > > > > the vote
>> > > > > > > > > > process
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > on
>> proposed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > API/behaviour. also -
>> > > > > > > > since
>> > > > > > > > > we're willing
>> > > > > > > > > > to go
>> > > > > > > > > > > >> with
>> > > > > > > > > > > >> >
>> strings
>> > > > > > > > > > > >> >
>> >>
>> > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialize
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > a
>> > > > > 0-sized
>> > > > > > header blob
>> > > > > > > and
>> > > > > > > > IIUC
>> > > > > > > > > you dont
>> > > > > > > > > > need any
>> > > > > > > > > > > >> >
>> optionals
>> > > > > > > > > > > >> >
>> >>
>> > > > anymore.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 3. yes,
>> > > > > > we can :-)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > On Tue,
>> > > > > > Feb 14, 2017 at
>> > > > > > > > 11:56
>> > > > > > > > > PM, Michael
>> > > > > > > > > > > Pearce <
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Hi
>> > > > > Jay,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1)
>> > > > > > There was some
>> > > > > > > > initial
>> > > > > > > > > debate on the
>> > > > > > > > > > value
>> > > > > > > > > > > >> > part,
>> as
>> > youll
>> > > > > > > > > > > >> >
>> >>
>> > > note
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > String,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > String
>> > > > > > headers were
>> > > > > > > > > discounted early
>> > > > > > > > > > on. The
>> > > > > > > > > > > >> >
>> reason for
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > flexibility
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > keeping in line
>> > > > > > > > with the
>> > > > > > > > > > flexibility of
>> > > > > > > > > > > key,
>> > > > > > > > > > > >> > value
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > message
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > object
>> > > > > > itself. I don’t
>> > > > > > > > think
>> > > > > > > > > it takes
>> > > > > > > > > > away
>> > > > > > > > > > > from
>> > > > > > > > > > > >> an
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > as
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > each
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > plugin
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > will
>> > > > > > care for their
>> > > > > > > own
>> > > > > > > > key,
>> > > > > > > > > this way
>> > > > > > > > > > ints,
>> > > > > > > > > > > >> >
>> booleans ,
>> > > > exotic
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > binary
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > can
>> > > > > all
>> > > > > > be catered
>> > > > > > > for=.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > wanted
>> > > > > > > > to
>> > > > > > > > > push for a
>> > > > > > > > > > typed
>> > > > > > > > > > > >> value
>> > > > > > > > > > > >> >
>> interface,
>> > > > > > > > > > > >> >
>> >> I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wouldn’t
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > want
>> > > > > > just String
>> > > > > > > values
>> > > > > > > > > supported, but
>> > > > > > > > > > the the
>> > > > > > > > > > > >> >
>> primatives
>> > > > > > > > > > > >> >
>> >>
>> > > plus
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > string and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > also
>> > > > > > still keeping the
>> > > > > > > > > ability to have a
>> > > > > > > > > > > binary
>> > > > > > > > > > > >> > for
>> custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > binaries that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > some
>> > > > > > organisations may
>> > > > > > > > have.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > i. I
>> > > > > > have written this
>> > > > > > > > slight
>> > > > > > > > > > alternative
>> > > > > > > > > > > here,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > https://cwiki.apache.org/
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > > >> >
>> > > > 82+-+Add+Record+Headers+-+
>> > > > > Typed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > ii.
>> > > > > > Essentially the
>> > > > > > > > value
>> > > > > > > > > bytes, has a
>> > > > > > > > > > leading
>> > > > > > > > > > > >> > byte
>> > overhead.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > > This tells you
>> > > > > > > what
>> > > > > > > > type
>> > > > > > > > > the value
>> > > > > > > > > > is,
>> > > > > > > > > > > >> before
>> > > > > > > > > > > >> >
>> reading
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > rest
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes,
>> > > > > > allowing
>> > > > > > > > > >
>> > > > serialisation/deserialization
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > and
>> from the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > primitives,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > string
>> > > > > > and byte[].
>> > > > > > > This
>> > > > > > > > is
>> > > > > > > > > akin to some
>> > > > > > > > > > other
>> > > > > > > > > > > >> >
>> messaging
>> > > > > > > > > > > >> >
>> >>
>> > > > systems.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2) We
>> > > > > > are making it
>> > > > > > > > optional,
>> > > > > > > > > so that
>> > > > > > > > > > for
>> > > > > > > > > > > those
>> > > > > > > > > > > >> > not
>> wanting
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > headers have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 0
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes
>> > > > > > overhead (think
>> > > > > > > > of it
>> > > > > > > > > as a feature
>> > > > > > > > > > > flag),
>> > > > > > > > > > > >> I
>> > > > > > > > > > > >> > don’t
>> > > > > > > > > > > >> >
>> >>
>> > > > think this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > complex, especially if
>> > > > > > > > > comparing to
>> > > > > > > > > > changes
>> > > > > > > > > > > >> >
>> proposed in
>> > > > > > > > > > > >> >
>> >>
>> > > > other kips
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > like
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > kip-98.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > really
>> > > > > > > > don’t
>> > > > > > > > > like
>> > > > > > > > > > this, we
>> > > > > > > > > > > can
>> > > > > > > > > > > >> > drop
>> it, but
>> > > > > > > > > > > >> >
>> >>
>> > it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > would
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > mean
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > buying
>> > > > > > into 4 bytes
>> > > > > > > > extra
>> > > > > > > > > overhead for
>> > > > > > > > > > users
>> > > > > > > > > > > who
>> > > > > > > > > > > >> > do
>> not want
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > use
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3) In
>> > > > > > the summary yes,
>> > > > > > > > it is
>> > > > > > > > > at a higher
>> > > > > > > > > > > level,
>> > > > > > > > > > > >> > but I
>> think
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is well
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > documented in the
>> > > > > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > section.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a.
>> > > > > > Added getHeaders
>> > > > > > > > method to
>> > > > > > > > > > >
>> Producer/Consumer
>> > > > > > > > > > > >> >
>> record (that
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > it)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > b.
>> > > > > > We’ve also detailed
>> > > > > > > > the
>> > > > > > > > > new Headers
>> > > > > > > > > > class
>> > > > > > > > > > > >> that
>> > > > > > > > > > > >> > this
>> method
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > returns
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > encapsulates the
>> > > > > > > headers
>> > > > > > > > > protocol and
>> > > > > > > > > > logic.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Best,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Mike
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ==Original questions
>> > > > > > > > from the
>> > > > > > > > > vote
>> > > > > > > > > > thread from
>> > > > > > > > > > > >> > Jay.==
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Couple
>> > > > > > of things I
>> > > > > > > > think we
>> > > > > > > > > still need
>> > > > > > > > > > to work
>> > > > > > > > > > > >> out:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > I
>> > > > > > think we agree
>> > > > > > > > about
>> > > > > > > > > the key,
>> > > > > > > > > > but I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > we
>> haven't
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > talked about
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > value yet. I
>> > > > > > > > think if
>> > > > > > > > > our goal
>> > > > > > > > > > is an
>> > > > > > > > > > > open
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > these
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > spread across many
>> > > > > > > > plugins
>> > > > > > > > > from many
>> > > > > > > > > > > systems
>> > > > > > > > > > > >> we
>> > > > > > > > > > > >> > should
>> > > > > > > > > > > >> >
>> >>
>> > > > consider
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > making
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > string as well so
>> > > > > > > > it can
>> > > > > > > > > be
>> > > > > > > > > > printed, set
>> > > > > > > > > > > >> via
>> > > > > > > > > > > >> > a UI,
>> set
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > config,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > etc.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > Basically
>> > > > > > > encouraging
>> > > > > > > > > pluggable
>> > > > > > > > > > >
>> serialization
>> > > > > > > > > > > >> >
>> formats
>> > > > > > > > > > > >> >
>> >>
>> > > here
>> > > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > lead
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bit
>> > > > > > of a tower of
>> > > > > > > > babel.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2.
>> > > > > > This proposal
>> > > > > > > > still
>> > > > > > > > > includes a
>> > > > > > > > > > pretty
>> > > > > > > > > > > big
>> > > > > > > > > > > >> >
>> change to
>> > our
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > protocol
>> > > > > > > > definition
>> > > > > > > > > layer.
>> > > > > > > > > > Essentially
>> > > > > > > > > > > >> it is
>> > > > > > > > > > > >> >
>> >>
>> > > > introducing an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > optional
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > type, where the
>> > > > > > > > format is
>> > > > > > > > > data
>> > > > > > > > > > dependent. I
>> > > > > > > > > > > >> > think
>> this
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > actually a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > big
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > change though it
>> > > > > > > > doesn't
>> > > > > > > > > seem like
>> > > > > > > > > > it. It
>> > > > > > > > > > > >> means
>> > > > > > > > > > > >> > you
>> can no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > longer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > specify
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > this
>> > > > > > type with our
>> > > > > > > > type
>> > > > > > > > > definition
>> > > > > > > > > > DSL, and
>> > > > > > > > > > > >> >
>> likewise it
>> > > > > > > > > > > >> >
>> >>
>> > > > requires
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > handling in client
>> > > > > > > > libs.
>> > > > > > > > > This isn't
>> > > > > > > > > > a huge
>> > > > > > > > > > > >> >
>> thing, since
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Record
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > definition is
>> > > > > > > custom
>> > > > > > > > > anyway, but I
>> > > > > > > > > > think
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > kind
>> of
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > inconsistency is
>> > > > > > > very
>> > > > > > > > > non-desirable
>> > > > > > > > > > and
>> > > > > > > > > > > ties
>> > > > > > > > > > > >> > you to
>> > > > > > > > > > > >> >
>> >>
>> > > > hand-coding
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > things.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > think the type
>> > > > > > > should
>> > > > > > > > > instead by [Key
>> > > > > > > > > > > Value]
>> > > > > > > > > > > >> in
>> > > > > > > > > > > >> > our
>> BNF,
>> > > > > > > > > > > >> >
>> >>
>> > > where
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > value are both
>> > > > > > > short
>> > > > > > > > > strings as used
>> > > > > > > > > > > >>
>> elsewhere.
>> > > > > > > > > > > >> > This
>> > > > > > > > > > > >> >
>> >>
>> > > > brings it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > in line
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > with
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > rest of the
>> > > > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3.
>> > > > > > Could we get
>> > > > > > > more
>> > > > > > > > > specific about
>> > > > > > > > > > the
>> > > > > > > > > > > exact
>> > > > > > > > > > > >> > Java
>> API
>> > > > > > > > > > > >> >
>> >>
>> > > > change to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ProducerRecord,
>> > > > > > > > > ConsumerRecord,
>> > > > > > > > > > Record,
>> > > > > > > > > > > etc?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > -Jay
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The
>> > > > > information
>> > > > > > contained in
>> > > > > > > > this
>> > > > > > > > > email is
>> > > > > > > > > > strictly
>> > > > > > > > > > > >> >
>> confidential
>> > > and
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the use of the
>> > > > > > addressee only,
>> > > > > > > > unless
>> > > > > > > > > otherwise
>> > > > > > > > > > > >>
>> indicated.
>> > > > > > > > > > > >> > If you
>> > > > > > > > > > > >> >
>> >>
>> > are
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the intended
>> > > > > > recipient, please
>> > > > > > > > do not
>> > > > > > > > > read,
>> > > > > > > > > > copy, use
>> > > > > > > > > > > or
>> > > > > > > > > > > >> >
>> disclose to
>> > > > > > > > > > > >> >
>> >>
>> > > > others
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > this message
>> > > > > or
>> > > > > > any
>> > > > > > > attachment.
>> > > > > > > > > Please also
>> > > > > > > > > > notify the
>> > > > > > > > > > > >> >
>> sender by
>> > > > > > > > > > > >> >
>> >>
>> > > > replying
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > to this email
>> > > > > > or by telephone
>> > > > > > > > > (+44(020 7896
>> > > > > > > > > > 0011) and
>> > > > > > > > > > > >> then
>> > > > > > > > > > > >> > delete
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > email
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > and any copies
>> > > > > > of it.
>> > > > > > > Opinions,
>> > > > > > > > > conclusion
>> > > > > > > > > > (etc) that
>> > > > > > > > > > > do
>> > > > > > > > > > > >> > not
>> relate
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > official
>> > > > > > business of this
>> > > > > > > > company
>> > > > > > > > > shall be
>> > > > > > > > > > understood
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> neither
>> > > > > > > > > > > >> >
>> >>
>> > > > given nor
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > endorsed by
>> > > > > it.
>> > > > > > IG is a
>> > > > > > > trading
>> > > > > > > > name
>> > > > > > > > > of IG
>> > > > > > > > > > Markets
>> > > > > > > > > > > >> Limited
>> > > > > > > > > > > >> > (a
>> company
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > registered in
>> > > > > > England and
>> > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 04008957)
>> > > > > > > > > > > >> > and IG
>> > > > > > > > > > > >> >
>> >>
>> > > Index
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Limited (a
>> > > > > > company registered
>> > > > > > > in
>> > > > > > > > > England and
>> > > > > > > > > > Wales,
>> > > > > > > > > > > >> >
>> company
>> > number
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > 01190902).
>> > > > > > Registered address
>> > > > > > > at
>> > > > > > > > > Cannon Bridge
>> > > > > > > > > > House,
>> > > > > > > > > > > 25
>> > > > > > > > > > > >> >
>> Dowgate
>> > > > > > > > > > > >> >
>> >>
>> > > Hill,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > London EC4R
>> > > > > > 2YA. Both IG
>> > > > > > > Markets
>> > > > > > > > > Limited
>> > > > > > > > > > (register
>> > > > > > > > > > > >> number
>> > > > > > > > > > > >> >
>> 195355)
>> > > > > > > > > > > >> >
>> >>
>> > and
>> > > > IG
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Index Limited
>> > > > > > (register number
>> > > > > > > > > 114059) are
>> > > > > > > > > > authorised
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> regulated
>> > > > > > > > > > > >> >
>> >>
>> > by
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Financial
>> > > > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > --
>> > > > > > > > > > > >> >
>> >>
>> > > > Nacho - Ignacio
>> > > > > > Solis -
>> > > > > > > > isolis@igso.net
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > The
>> > > information
>> > > > contained in
>> > > > > > this email is
>> > > > > > > > strictly
>> > > > > > > > > > confidential and
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> use of
>> > the
>> > > > addressee only,
>> > > > > > unless
>> > > > > > > otherwise
>> > > > > > > > > indicated. If
>> > > > > > > > > > you are
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > the
>> intended
>> > > > recipient, please
>> > > > > > do not read,
>> > > > > > > > copy, use
>> > > > > > > > > or
>> > > > > > > > > > disclose to
>> > > > > > > > > > > >> others
>> > > > > > > > > > > >> > this
>> message
>> > > or
>> > > > any
>> > > > > attachment.
>> > > > > > Please also
>> > > > > > > > notify
>> > > > > > > > > the sender by
>> > > > > > > > > > > >> replying
>> > > > > > > > > > > >> > to
>> this
>> > email
>> > > > or by telephone
>> > > > > > (+44(020 7896
>> > > > > > > > 0011) and
>> > > > > > > > > then
>> > > > > > > > > > delete the
>> > > > > > > > > > > >> email
>> > > > > > > > > > > >> > and
>> any
>> > copies
>> > > > of it.
>> > > > > Opinions,
>> > > > > > conclusion
>> > > > > > > > (etc) that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > understood
>> > > > > > > > > as
>> > > > > > > > > > neither given
>> > > > > > > > > > > >> nor
>> > > > > > > > > > > >> >
>> endorsed by
>> > > it.
>> > > > IG is a
>> > > > > trading
>> > > > > > name of IG
>> > > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >> >
>> registered
>> > in
>> > > > England and
>> > > > > > Wales, company
>> > > > > > > number
>> > > > > > > > > 04008957) and
>> > > > > > > > > > IG Index
>> > > > > > > > > > > >> >
>> Limited (a
>> > > > company registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > >> >
>> 01190902).
>> > > > Registered address
>> > > > > > at Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25
>> > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > >> >
>> London EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and
>> > > > > > > > > > > IG
>> > > > > > > > > > > >> > Index
>> > Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> Financial
>> > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> The
>> > information
>> > > > contained in
>> > > > > this
>> > > > > > email is
>> > > > > > > > strictly
>> > > > > > > > > confidential
>> > > > > > > > > > and for
>> > > > > > > > > > > >> the use
>> of the
>> > > > addressee only,
>> > > > > > unless otherwise
>> > > > > > > > > indicated. If you
>> > > > > > > > > > are
>> > > > > > > > > > > not
>> > > > > > > > > > > >> the
>> intended
>> > > > recipient, please
>> > > > > do
>> > > > > > not read,
>> > > > > > > copy,
>> > > > > > > > use
>> > > > > > > > > or disclose
>> > > > > > > > > > to
>> > > > > > > > > > > others
>> > > > > > > > > > > >> this
>> message
>> > or
>> > > > any attachment.
>> > > > > > Please also
>> > > > > > > > notify the
>> > > > > > > > > sender by
>> > > > > > > > > > > replying
>> > > > > > > > > > > >> to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the
>> > > > > > > > > > > email
>> > > > > > > > > > > >> and any
>> copies
>> > > of
>> > > > it. Opinions,
>> > > > > > conclusion (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > the
>> > > > > > > > > > > >> official
>> > > business
>> > > > of this
>> > > > > company
>> > > > > > shall be
>> > > > > > > > understood
>> > > > > > > > > as neither
>> > > > > > > > > > given
>> > > > > > > > > > > nor
>> > > > > > > > > > > >>
>> endorsed by
>> > it.
>> > > > IG is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >>
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG
>> > > > > > > > > > Index
>> > > > > > > > > > > >> Limited
>> (a
>> > > > company registered in
>> > > > > > England and
>> > > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 01190902).
>> > > > Registered address at
>> > > > > > Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25 Dowgate
>> > > > > > > > > > Hill,
>> > > > > > > > > > > >> London
>> EC4R
>> > 2YA.
>> > > > Both IG Markets
>> > > > > > Limited
>> > > > > > > (register
>> > > > > > > > > number 195355)
>> > > > > > > > > > and IG
>> > > > > > > > > > > >> Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> Financial
>> > > Conduct
>> > > > Authority.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > The
>> information
>> > > > contained in this
>> > > > > > email is strictly
>> > > > > > > > > confidential and
>> > > > > > > > > > for the use of
>> the
>> > > > addressee only, unless
>> > > > > > otherwise
>> > > > > > > > indicated.
>> > > > > > > > > If you are
>> > > > > > > > > > not the intended
>> > > > recipient, please do not
>> > > > > > read, copy, use
>> > > > > > > > or
>> > > > > > > > > disclose to
>> > > > > > > > > > others this
>> message or
>> > > any
>> > > > attachment.
>> > > > > > Please also notify
>> > > > > > > > the
>> > > > > > > > > sender by
>> > > > > > > > > > replying to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the email and any
>> > copies
>> > > > of it. Opinions,
>> > > > > > conclusion
>> > > > > > > (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to the
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > > understood as
>> > > > > > > > > > neither given nor
>> > > endorsed
>> > > > by it. IG is a
>> > > > > > trading name of
>> > > > > > > > IG
>> > > > > > > > > Markets
>> > > > > > > > > > Limited (a
>> company
>> > > > registered in England
>> > > > > > and Wales,
>> > > > > > > company
>> > > > > > > > > number
>> > > > > > > > > > 04008957) and IG
>> Index
>> > > > Limited (a company
>> > > > > > registered in
>> > > > > > > > England
>> > > > > > > > > and Wales,
>> > > > > > > > > > company number
>> > 01190902).
>> > > > Registered
>> > > > > > address at Cannon
>> > > > > > > > Bridge
>> > > > > > > > > House, 25
>> > > > > > > > > > Dowgate Hill,
>> London
>> > EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and IG
>> Index
>> > > > Limited (register
>> > > > > > number 114059) are
>> > > > > > > > > authorised and
>> > > > > > > > > > regulated by the
>> > > Financial
>> > > > Conduct
>> > > > > > Authority.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The information
>> contained in
>> > > > this email is
>> > > > > > strictly
>> > > > > > > confidential
>> > > > > > > > and
>> > > > > > > > > for the use of the
>> addressee
>> > only,
>> > > > unless otherwise
>> > > > > > indicated. If
>> > > > > > > > you are
>> > > > > > > > > not the intended recipient,
>> > please
>> > > > do not read,
>> > > > > > copy, use or
>> > > > > > > > disclose to
>> > > > > > > > > others this message or any
>> > > > attachment. Please also
>> > > > > > notify the
>> > > > > > > sender
>> > > > > > > > by
>> > > > > > > > > replying to this email or
>> by
>> > > > telephone (+44(020
>> > > > > 7896
>> > > > > > 0011) and then
>> > > > > > > > delete
>> > > > > > > > > the email and any copies
>> of it.
>> > > > Opinions,
>> > > > > conclusion
>> > > > > > (etc) that do
>> > > > > > > > not
>> > > > > > > > > relate to the official
>> business
>> > of
>> > > > this company
>> > > > > > shall be understood
>> > > > > > > > as
>> > > > > > > > > neither given nor endorsed
>> by it.
>> > > IG
>> > > > is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a company
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG Index
>> Limited (a
>> > > > company
>> > > > > registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company number 01190902).
>> > > Registered
>> > > > address at
>> > > > > > Cannon Bridge
>> > > > > > > House,
>> > > > > > > > 25
>> > > > > > > > > Dowgate Hill, London EC4R
>> 2YA.
>> > Both
>> > > > IG Markets
>> > > > > > Limited (register
>> > > > > > > > number
>> > > > > > > > > 195355) and IG Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > authorised
>> > > > > > > > and
>> > > > > > > > > regulated by the Financial
>> > Conduct
>> > > > Authority.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > The information contained in
>> this email
>> > > is
>> > > > strictly
>> > > > > > confidential and for
>> > > > > > > > the use of the addressee only,
>> unless
>> > > > otherwise
>> > > > > indicated.
>> > > > > > If you are not
>> > > > > > > > the intended recipient, please
>> do not
>> > > > read, copy, use or
>> > > > > > disclose to
>> > > > > > > others
>> > > > > > > > this message or any attachment.
>> Please
>> > > > also notify the
>> > > > > > sender by replying
>> > > > > > > > to this email or by telephone
>> (+44(020
>> > > > 7896 0011) and
>> > > > > then
>> > > > > > delete the
>> > > > > > > email
>> > > > > > > > and any copies of it. Opinions,
>> > > conclusion
>> > > > (etc) that do
>> > > > > > not relate to
>> > > > > > > the
>> > > > > > > > official business of this
>> company shall
>> > > be
>> > > > understood as
>> > > > > > neither given
>> > > > > > > nor
>> > > > > > > > endorsed by it. IG is a trading
>> name of
>> > > IG
>> > > > Markets
>> > > > > Limited
>> > > > > > (a company
>> > > > > > > > registered in England and Wales,
>> > company
>> > > > number 04008957)
>> > > > > > and IG Index
>> > > > > > > > Limited (a company registered in
>> > England
>> > > > and Wales,
>> > > > > > company number
>> > > > > > > > 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill,
>> > > > > > > > London EC4R 2YA. Both IG Markets
>> > Limited
>> > > > (register number
>> > > > > > 195355) and IG
>> > > > > > > > Index Limited (register number
>> 114059)
>> > > are
>> > > > authorised and
>> > > > > > regulated by
>> > > > > > > the
>> > > > > > > > Financial Conduct Authority.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > The information contained in this email is
>> strictly
>> > > > confidential and
>> > > > > > for the use of the addressee only, unless
>> otherwise
>> > > > indicated. If you are
>> > > > > > not the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or disclose to
>> > > > > > others this message or any attachment. Please
>> also
>> > notify
>> > > > the sender by
>> > > > > > replying to this email or by telephone (+44(020
>> 7896
>> > > 0011)
>> > > > and then
>> > > > > delete
>> > > > > > the email and any copies of it. Opinions,
>> conclusion
>> > > (etc)
>> > > > that do not
>> > > > > > relate to the official business of this company
>> shall
>> > be
>> > > > understood as
>> > > > > > neither given nor endorsed by it. IG is a
>> trading name
>> > of
>> > > > IG Markets
>> > > > > > Limited (a company registered in England and
>> Wales,
>> > > > company number
>> > > > > > 04008957) and IG Index Limited (a company
>> registered in
>> > > > England and
>> > > > > Wales,
>> > > > > > company number 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register number
>> > > > > > 195355) and IG Index Limited (register number
>> 114059)
>> > are
>> > > > authorised and
>> > > > > > regulated by the Financial Conduct Authority.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential
>> and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If
>> you are
>> > not
>> > > > the intended recipient, please do not read, copy, use or
>> disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then
>> delete the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company
>> number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number
>> 195355) and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and
>> regulated by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > >
>> >
>>
>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
--
Nacho - Ignacio Solis - isolis@igso.net
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
This is a bit of why I’d rather have headers as self-contained, e.g. it has a headersLen and headers data, as we’d have to introduce one or the other, and it seems cleaner to add the headers than add back in the valuelen.
On 21/02/2017, 23:57, "Michael Pearce" <Mi...@ig.com> wrote:
Hi Jason
I looked at the doc, it seems the length of the following data sections are being left described in your doc in some place similar to how we have on the wiki, and in others not.
E.g. your doc has this as the proposed message described (which is in blue and red colouring)
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
Value => data [OPTIONAL]
Note I notice there is a missing
ValueLen => uintVar [OPTIONAL]
Else we wouldn’t be able to put headers at the end, without its length being defined. So either we can leave ValueLen removed, as is in your new format, and keep headers
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
HeadersLen => uintVar [OPTIONAL]
Headers => data [OPTIONAL]
Value => data [OPTIONAL]
Headers => [Header]
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
OR we will need to re-introduce ValueLen which seems to be removed in Kip-98
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
ValueLen => uintVar [OPTIONAL]
Value => data [OPTIONAL]
Headers => Header1, Header2 … HeaderN
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
Cheers
Mike
On 21/02/2017, 20:45, "ignacio.solis@gmail.com on behalf of Ignacio Solis" <ignacio.solis@gmail.com on behalf of isolis@igso.net> wrote:
Short answer:
Headers at the end are fine, this proposal looks ok.
Longer answer:
At LinkedIn we considered headers at the end multiple times. We even
had a backwards compatible headers-in-V design with headers at the
end.
The main reason headers come at the beginning of messaging systems is
in case they affect the rest of the message. This is either because
of parsing differences (a header might indicate how to parse the
remainder of the message), or performance optimizations (a header
might indicate that you can discard the message and can stop parsing).
I believe that neither of these cases are common in Kafka (at this
time) so having the headers are the end shouldn't be a big issue.
From a performance perspective we may incur an extra page cache miss
but at the rate at which kafka does messaging it won't bottleneck
things.
Headers at the end have one potential advantage, that in simple
circumstances, naive systems can append headers to a message without
doing a lot of memory manipulation. Normally not that big of a deal
since we could always use scatter/gather methods if needed, but maybe
some simple systems will benefit from that. ... Not to mention most
Java programmers might not pay attention to these type of performance
issues.
Anyway...
Overall, there are many things that I would have done differently in
the proposal (int keys, total header length, no VarInts, etc. ), but
the cost of these is small in comparison to the benefit we get from
having headers at all.
If we are all ok with this design then I would be the first to +1 the format.
Nacho
Side note: Shouldn't headers at the end be called footers? We could
always call it metadata :-).
On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com> wrote:
> I think I'd like some opinions from some of the active chaps at LinkedIn so it's not just a two way decision (or any others for that matter).
>
> Any opinions on the below or no real concerns / indifferent by anyone with this change?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Tuesday, February 21, 2017 6:59:38 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Jun's suggestion gives us the best of both worlds. Because the headers are
> the last field in the record and you have the total size of the record,
> then you can easily get the size of headers in bytes. So they can be copied
> without parsing and we can do the lazy initialization without an extra cost.
>
> The message set notion has changed in KIP-98 so that the message set can
> have its own fields. See here for more detail:
> https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
> What we're proposing to change here is the schema for the message (aka
> record) itself. We'll update that document depending on the outcome of the
> discussion here.
>
> -Jason
>
>
> On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> This works whilst no other feature needs similar. I prefer to keep it
>> contained/encapsulated it then allows the bytes of the headers to also be
>> copied transferred as one without any parsing what so ever, but if you
>> really feel strongly on the below ill go with it, just want to flag this
>> concern.
>>
>> Btw, the below notation seems to be different to the notation used in
>> previous kips like kip-32 which I took the notation from, where the length
>> of the following bytes for byte arrays is clearly defined. So im 100% on
>> what you propose could you define it in this original styled notation?
>>
>>
>> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>>
>> Hey Jun, that's a good thought. Maybe then the full schema could look
>> like
>> this:
>>
>> Record =>
>> Length => uintVar
>> Attributes => int8
>> TimestampDelta => intVar
>> OffsetDelta => uintVar
>> Key => bytes
>> Value => bytes
>> Headers => [HeaderKey HeaderValue]
>> HeaderKey => string
>> HeaderValue => bytes
>>
>> (where the bytes and string type use a variable length size field)
>>
>> Does that work for you, Michael?
>>
>> -Jason
>>
>> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > Regarding the overhead of skipping the headers in the message.
>> Another way
>> > could be to add the headers at the end of the message after the
>> value part.
>> > This way, there is no need to skip the headers unless you really
>> need it.
>> > This will allow us to represent the headers explicitly as an array of
>> > key/value pairs without the skipping overhead when headers are not
>> needed.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
>> >
>> > wrote:
>> >
>> > > Michael,
>> > >
>> > > Here's a link to the branch with the message format changes:
>> > > https://github.com/confluentinc/kafka/tree/
>> exactly-once-message-format.
>> > > The
>> > > class to modify to add headers is EosLogRecord (name will
>> definitely
>> > change
>> > > prior to merging in trunk). This branch uses attributes to
>> indicate null
>> > > key and value. I really don't have a strong preference for or
>> against
>> > this
>> > > as long as we're consistent. I do feel it's kind of a pity to need
>> the
>> > > signed version of the varint just to be able to indicate null. For
>> > headers,
>> > > it's an easier call because we don't actually need to represent
>> null
>> > > headers (using size zero will work fine).
>> > >
>> > > -Jason
>> > >
>> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > We can make it, implement Iterable<Header>, no concerns on this.
>> > > >
>> > > > Ive noted this on the KIP.
>> > > >
>> > > >
>> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > > Btw Jason, can you share the branch with the VarInts
>> helper/util
>> > > class
>> > > > that KIP-98 will be introducing and also will have the current
>> modified
>> > > > (key and value lengths) message, so we can look at, thanks.
>> > > >
>> > > > On 20/02/2017, 01:11, "Michael Pearce" <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > > >
>> > > > RE “ProducerRecord and ConsumerRecord are immutable
>> > currently”. I
>> > > > beg to differ on this, if I set a byte[] as my value of my
>> record, I
>> > can
>> > > > later call value() and then alter the bytes in the array. As
>> such it is
>> > > > very much mutable.
>> > > >
>> > > > Headers also will anyhow by nature be mutable on
>> produce, as
>> > you
>> > > > add to them.
>> > > >
>> > > >
>> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
>> jason@confluent.io>
>> > > > wrote:
>> > > >
>> > > > Re; having a dedicated Headers object. I think it
>> makes
>> > > sense.
>> > > > Maybe it
>> > > > could implement Iterable<Header>?
>> > > >
>> > > > One thing I'm not too sure about is mutability. Both
>> the
>> > > > ProducerRecord and
>> > > > ConsumerRecord are immutable currently. It would be
>> nice to
>> > > > keep it that
>> > > > way. Perhaps Headers could itself be immutable and
>> passed
>> > > > through the
>> > > > constructors?
>> > > >
>> > > > Just going through how this is, I don’t believe
>> this would
>> > > > provide much
>> > > > > saving. If issue trying to resolve is around
>> > object/garbage
>> > > > creation of
>> > > > > Map.Entry objects.
>> > > >
>> > > >
>> > > > It's definitely better because we only allocate the
>> space
>> > we
>> > > > will use, but
>> > > > I assume most of the actual memory cost is in the
>> header
>> > data
>> > > > itself which
>> > > > we can't do much about. Maybe we can start with the
>> simple
>> > > > array
>> > > > representation and optimize from there after we find
>> it's
>> > > > worthwhile to do
>> > > > so? If we use the Headers object, we can change this
>> later
>> > > > without breaking
>> > > > the API.
>> > > >
>> > > > -Jason
>> > > >
>> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
>> > > > radai.rosenblatt@gmail.com> wrote:
>> > > >
>> > > > > "(By the way, doesn't it feel a bit odd that we
>> seem to
>> > be
>> > > > designing a
>> > > > > feature which is optimized for people not using?)"
>> > > > >
>> > > > > very. this (i claim :-) ) is the result of very
>> intense
>> > > > opposition to the
>> > > > > usefulness of the feature early on, and not a real
>> design
>> > > > goal
>> > > > >
>> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Whilst if having an array of a Header[] would
>> mean
>> > > getting
>> > > > this through,
>> > > > > > im happy changing this, as already done.
>> > > > > >
>> > > > > > Just going through how this is, I don’t believe
>> this
>> > > > would provide much
>> > > > > > saving. If issue trying to resolve is around
>> > > > object/garbage creation of
>> > > > > > Map.Entry objects.
>> > > > > >
>> > > > > > All you’re doing here is replacing the equiv of a
>> > > > HashMap.Node
>> > > > > (Map.Entry)
>> > > > > > which simply holds reference to key and value
>> objects
>> > > with
>> > > > a custom
>> > > > > variant.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com> wrote:
>> > > > > >
>> > > > > > On point 1 & 2 Ive updated KIP to show
>> varints (and
>> > > > removed the bit
>> > > > > > flag). (on the assumption KIP 98 is getting the
>> > agreement
>> > > > the protocol is
>> > > > > > moving from int32 to varInts as standard)
>> > > > > >
>> > > > > > On point 3 ive updated to use an array of
>> Header
>> > > > class, instead of a
>> > > > > > MultiMap in the Headers class object
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > On points 1 and 2 I agree.
>> > > > > >
>> > > > > > This also affects kip-98, I should
>> expect this
>> > > > resolved before
>> > > > > > that vote also passes. If it is accepted there
>> (I’m
>> > > > assuming this is
>> > > > > > getting discussed on that KIP? As you’re driving
>> the
>> > move
>> > > > to VarInts), I
>> > > > > am
>> > > > > > happy to make this KIP will simply follow suit to
>> > > whatever
>> > > > is agreed in
>> > > > > > KIP-98.
>> > > > > >
>> > > > > > On 3) Agreed this is a simpler form, as
>> long as
>> > > no
>> > > > one is
>> > > > > > expecting hashmap lookup performance ( O(1) ), I
>> still
>> > > > would prefer an
>> > > > > > encapsulated class, so if we find that holding
>> it as an
>> > > > array in future
>> > > > > is
>> > > > > > causing some perf issues, the internals are not
>> exposed
>> > > to
>> > > > end users,
>> > > > > > allowing the internal structure to move to a map.
>> > > > > >
>> > > > > > Can we compromise on?
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Headers headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Headers {
>> > > > > > Header[] headers;
>> > > > > >
>> > > > > > add(String key, byte[] value)
>> > > > > > Collection<byte[]> get(String key)
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
>> > > > jason@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > >
>> > > > > >
>> > > > > > This is exactly what I have been
>> trying to
>> > > get
>> > > > at. The use
>> > > > > > cases documented
>> > > > > > here are middleware:
>> > > > > > https://cwiki.apache.org/
>> > > > confluence/display/KAFKA/A+
>> > > > > > Case+for+Kafka+Headers.
>> > > > > > If headers are intended for the
>> application
>> > > > space as well,
>> > > > > the
>> > > > > > document
>> > > > > > should be updated accordingly so
>> that it is
>> > > an
>> > > > explicit
>> > > > > design
>> > > > > > goal and not
>> > > > > > an unfortunate design byproduct.
>> There may
>> > be
>> > > > some
>> > > > > > disagreement on whether
>> > > > > > it _should_ be a design goal, but it
>> may
>> > not
>> > > > matter much
>> > > > > since
>> > > > > > the current
>> > > > > > interceptor API is probably
>> insufficient
>> > for
>> > > > middleware
>> > > > > > applications (I
>> > > > > > haven't thought of a way to do this
>> that
>> > > isn't
>> > > > cumbersome).
>> > > > > >
>> > > > > > That aside, the three unresolved
>> points for
>> > > me
>> > > > are the
>> > > > > > following:
>> > > > > >
>> > > > > > 1. The use of varints. As a one-off
>> for use
>> > > > only with record
>> > > > > > headers, the
>> > > > > > case was weak, but if we are using
>> them
>> > > > throughout the
>> > > > > message
>> > > > > > format, then
>> > > > > > we should do so here as well. The
>> > additional
>> > > > complexity is
>> > > > > > minimal and
>> > > > > > early performance testing fully
>> justifies
>> > > > their use.
>> > > > > >
>> > > > > > 2. If varints are used, the case for
>> using
>> > > > attributes to
>> > > > > > indicate null
>> > > > > > headers also becomes weak. We only
>> add one
>> > > > additional byte in
>> > > > > > each message
>> > > > > > if there are no headers. Whatever
>> the case,
>> > > > let us be
>> > > > > > consistent with how
>> > > > > > we treat null keys and values.
>> > > > > >
>> > > > > > 3. We have apparently agreed that the
>> > broker
>> > > > will validate
>> > > > > > headers (please
>> > > > > > add this to the KIP). That being the
>> case,
>> > I
>> > > > would prefer to
>> > > > > > use Kafka's
>> > > > > > conventional format for arrays. The
>> > downside
>> > > > is that it takes
>> > > > > > more work to
>> > > > > > skip over the headers on the
>> consumer,
>> > though
>> > > > it's unclear if
>> > > > > > the cost
>> > > > > > would matter in practice. Some
>> concern was
>> > > > previously
>> > > > > > expressed about the
>> > > > > > allocation of maps. An alternative
>> would be
>> > > to
>> > > > use arrays,
>> > > > > i.e.
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Header[] headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > > This would work nicely with the
>> > conventional
>> > > > array format and
>> > > > > > my guess is
>> > > > > > it would obviate the need do any lazy
>> > > > initialization. If we
>> > > > > > use the map as
>> > > > > > is currently documented, then it is
>> > possible
>> > > > with either
>> > > > > > representation to
>> > > > > > slice the headers and initialize them
>> > lazily.
>> > > > Either way, it
>> > > > > > might be a
>> > > > > > good idea to use a separate object to
>> > > > represent the headers
>> > > > > in
>> > > > > > case we need
>> > > > > > to modify it in the future in some
>> way.
>> > > > > >
>> > > > > > (By the way, doesn't it feel a bit
>> odd that
>> > > we
>> > > > seem to be
>> > > > > > designing a
>> > > > > > feature which is optimized for
>> people not
>> > > > using?)
>> > > > > >
>> > > > > >
>> > > > > > If we can resolve these points, then
>> at
>> > least
>> > > > you will get my
>> > > > > > vote.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jason
>> > > > > >
>> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
>> radai <
>> > > > > > radai.rosenblatt@gmail.com> wrote:
>> > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > > > IIUC samze currently has headers
>> "in-V"
>> > and
>> > > > would just
>> > > > > > switch over to kafka
>> > > > > > > headers if they exist.
>> > > > > > > im sure plenty of other users of
>> kafka
>> > > would
>> > > > have a use for
>> > > > > > headers.
>> > > > > > > im pretty sure use cases exist
>> around
>> > > > shuffling data
>> > > > > > into/out-of kafka
>> > > > > > > (kafka connect or equivalent) where
>> > > metadata
>> > > > from one end
>> > > > > > could copied over
>> > > > > > > to the other (S3, for example uses
>> http
>> > > > headers for
>> > > > > > user-accessible
>> > > > > > > metadata). it will be kafka client
>> code
>> > > > getting/setting
>> > > > > > those headers. not
>> > > > > > > an interceptor.
>> > > > > > >
>> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
>> Michael
>> > > > Pearce <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > For APM single event tracing,
>> need
>> > access
>> > > > to the header
>> > > > > at
>> > > > > > the point of
>> > > > > > > > processing on the processing
>> thread.
>> > > > > > > >
>> > > > > > > > As such interceptors will not
>> work/be
>> > > > suitable for these,
>> > > > > > due to the fact
>> > > > > > > > they act on the ConsumerRecords
>> as a
>> > > > batch, before the
>> > > > > > handling thread
>> > > > > > > can
>> > > > > > > > split out and process per
>> message which
>> > > is
>> > > > the point
>> > > > > these
>> > > > > > tools will
>> > > > > > > need
>> > > > > > > > to continue to transaction
>> tracing.
>> > > > > > > >
>> > > > > > > > Like wise tools and infra pieces
>> will
>> > > need
>> > > > access to the
>> > > > > > message outside
>> > > > > > > > the interceptor realm.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 17/02/2017, 21:26, "Jason
>> > Gustafson" <
>> > > > > > jason@confluent.io> wrote:
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > So the broker will parse the
>> > headers
>> > > > and validate
>> > > > > > them. Good. The
>> > > > > > > only
>> > > > > > > > reason remaining that I can
>> see to
>> > > > leave the headers
>> > > > > > as a byte array
>> > > > > > > > is to
>> > > > > > > > make it easier for the
>> client to
>> > skip
>> > > > past them. Are
>> > > > > > we sure this is
>> > > > > > > > not
>> > > > > > > > premature optimization? Are
>> there
>> > any
>> > > > performance
>> > > > > > results which show
>> > > > > > > > that
>> > > > > > > > this is worthwhile?
>> > > > > > > >
>> > > > > > > > What’s the issue with
>> exposing a
>> > > > method getHeaders on
>> > > > > > the
>> > > > > > > > producer/consumer
>> > > > > > > > > record? It doesn’t break
>> > anything.
>> > > > We don’t need
>> > > > > any
>> > > > > > special
>> > > > > > > version.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > See my previous explanation.
>> What I
>> > > am
>> > > > trying to
>> > > > > > resist is the
>> > > > > > > headers
>> > > > > > > > becoming a general
>> > application-level
>> > > > facility. The
>> > > > > > primary use case
>> > > > > > > as
>> > > > > > > > far
>> > > > > > > > as I can tell is middleware,
>> which
>> > is
>> > > > the use case
>> > > > > > that the
>> > > > > > > > interceptors
>> > > > > > > > are providing.
>> > > > > > > >
>> > > > > > > > Current batch consumer model
>> and
>> > > > consumer
>> > > > > interceptors
>> > > > > > don’t work
>> > > > > > > where
>> > > > > > > > > headers need to be acted
>> on at
>> > per
>> > > > message level at
>> > > > > > time of
>> > > > > > > > processing,
>> > > > > > > > > very case is APM (the core
>> one),
>> > > > where the header
>> > > > > > value is used to
>> > > > > > > > continue
>> > > > > > > > > tracing.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I still don't understand the
>> point
>> > > > about batching.
>> > > > > The
>> > > > > > consumer
>> > > > > > > > records are
>> > > > > > > > exposed as a batch in the
>> consumer
>> > > > interceptor, but
>> > > > > > you can still
>> > > > > > > > iterate
>> > > > > > > > them individually. It is no
>> > different
>> > > > for the
>> > > > > consumer
>> > > > > > API itself.
>> > > > > > > >
>> > > > > > > > -Jason
>> > > > > > > >
>> > > > > > > > On Fri, Feb 17, 2017 at
>> 12:48 PM,
>> > > > Michael Pearce <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Re:
>> > > > > > > > >
>> > > > > > > > > “ The point about
>> creation of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a slice
>> of
>> > bytes
>> > > > until the time
>> > > > > > it is accessed.”
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > >
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:37,
>> "Michael
>> > > > Pearce" <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > What’s the issue with
>> > exposing
>> > > a
>> > > > method
>> > > > > > getHeaders on the
>> > > > > > > > > producer/consumer record?
>> It
>> > > doesn’t
>> > > > break
>> > > > > anything.
>> > > > > > We don’t need
>> > > > > > > > any
>> > > > > > > > > special version.
>> > > > > > > > >
>> > > > > > > > > Current batch consumer
>> model
>> > > and
>> > > > consumer
>> > > > > > interceptors don’t
>> > > > > > > work
>> > > > > > > > > where headers need to be
>> acted on
>> > > at
>> > > > per message
>> > > > > > level at time of
>> > > > > > > > > processing, very case is
>> APM (the
>> > > > core one), where
>> > > > > > the header value
>> > > > > > > > is used
>> > > > > > > > > to continue tracing.
>> JMS/HTTP etc
>> > > > all expose these,
>> > > > > > without issues.
>> > > > > > > > I would
>> > > > > > > > > NOT want to lock this down
>> to
>> > only
>> > > > be usable
>> > > > > > accessible via
>> > > > > > > > interceptors,
>> > > > > > > > > as we’d fail on one of the
>> main
>> > > > goals.
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Mike
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:21,
>> "Jason
>> > > > Gustafson" <
>> > > > > > jason@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > The point about
>> creation
>> > of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a
>> slice of
>> > > > bytes until the
>> > > > > > time it is
>> > > > > > > > accessed.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > Yes exactly we
>> have
>> > > access
>> > > > to the records
>> > > > > > thus why the
>> > > > > > > > header
>> > > > > > > > > should be
>> > > > > > > > > > accessible via
>> it and
>> > not
>> > > > hidden for only
>> > > > > > interceptors to
>> > > > > > > > access.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > As explained
>> above, the
>> > > > point is to make
>> > > > > the
>> > > > > > intended usage
>> > > > > > > > clear.
>> > > > > > > > > Applications should
>> > > continue
>> > > > to rely on the
>> > > > > > key/value
>> > > > > > > fields
>> > > > > > > > to
>> > > > > > > > > serialize
>> > > > > > > > > their own headers,
>> and it
>> > > > would be more
>> > > > > > ideal if we can
>> > > > > > > avoid
>> > > > > > > > > leaking
>> > > > > > > > > third-party
>> headers into
>> > > > applications. This
>> > > > > > is difficult to
>> > > > > > > > do
>> > > > > > > > > with the
>> > > > > > > > > current
>> interceptors
>> > > because
>> > > > they share the
>> > > > > > record objects
>> > > > > > > > with
>> > > > > > > > > the common
>> > > > > > > > > API. What I had in
>> mind
>> > is
>> > > > something like
>> > > > > an
>> > > > > > extension of
>> > > > > > > the
>> > > > > > > > > current
>> > > > > > > > > interceptors which
>> > exposed
>> > > a
>> > > > different
>> > > > > > object (e.g.
>> > > > > > > > > `RecordAndHeaders`).
>> > > > > > > > > The challenge is
>> for
>> > > MM-like
>> > > > use cases. Let
>> > > > > > me see if I can
>> > > > > > > > come
>> > > > > > > > > up with a
>> > > > > > > > > concrete proposal
>> for
>> > that
>> > > > problem.
>> > > > > > > > >
>> > > > > > > > > -Jason
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 17,
>> 2017 at
>> > > > 11:55 AM, Michael
>> > > > > > Pearce <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I am happy to
>> move the
>> > > > definition of the
>> > > > > > header into the
>> > > > > > > > message
>> > > > > > > > > body, but
>> > > > > > > > > > would cause us
>> not to
>> > > lazy
>> > > > > > initialise/parse the headers,
>> > > > > > > as
>> > > > > > > > > obviously, we
>> > > > > > > > > > would have to
>> traverse
>> > > > these reading the
>> > > > > > message.
>> > > > > > > > > >
>> > > > > > > > > > This was
>> actually one
>> > of
>> > > > Jay’s requests:
>> > > > > > > > > >
>> > > > > > > > > > “ 2. I think
>> we
>> > should
>> > > > think about
>> > > > > > creating the Map
>> > > > > > > > lazily to
>> > > > > > > > > avoid
>> > > > > > > > > > parsing out
>> all the
>> > > > headers into
>> > > > > > little objects.
>> > > > > > > > HashMaps
>> > > > > > > > > themselves
>> > > > > > > > > > are
>> > > > > > > > > > kind of
>> expensive
>> > and
>> > > > the consumer is
>> > > > > > very perf
>> > > > > > > > sensitive so
>> > > > > > > > > and making
>> > > > > > > > > > gazillions of
>> > > hashmaps
>> > > > that may or
>> > > > > may
>> > > > > > not get used
>> > > > > > > is
>> > > > > > > > > probably a bad
>> > > > > > > > > > idea.”
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 17/02/2017,
>> 19:44,
>> > > > "Michael Pearce" <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Yes exactly
>> we have
>> > > > access to the
>> > > > > > records thus why
>> > > > > > > the
>> > > > > > > > > header should
>> > > > > > > > > > be accessible
>> via it
>> > and
>> > > > not hidden for
>> > > > > > only interceptors
>> > > > > > > > to
>> > > > > > > > > access.
>> > > > > > > > > >
>> > > > > > > > > > Sent using
>> OWA for
>> > > > iPhone
>> > > > > > > > > >
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > From: Magnus
>> > > Edenhill <
>> > > > > > magnus@edenhill.se>
>> > > > > > > > > > Sent: Friday,
>> > > February
>> > > > 17, 2017
>> > > > > > 7:34:49 PM
>> > > > > > > > > > To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > Subject: Re:
>> > > [DISCUSS]
>> > > > KIP-82 - Add
>> > > > > > Record Headers
>> > > > > > > > > >
>> > > > > > > > > > Big +1 on
>> VarInts.
>> > > > > > > > > > CPUs are
>> fast,
>> > memory
>> > > > is slow.
>> > > > > > > > > >
>> > > > > > > > > > I agree with
>> Jason
>> > > > that we'll want to
>> > > > > > continue
>> > > > > > > > verifying
>> > > > > > > > > messages,
>> > > > > > > > > > including
>> their
>> > > > headers, so while I
>> > > > > > appreciate the
>> > > > > > > > idea of
>> > > > > > > > > the opaque
>> > > > > > > > > > header blob
>> it
>> > won't
>> > > > be useful in
>> > > > > > practice.
>> > > > > > > > > >
>> > > > > > > > > > /Magnus
>> > > > > > > > > >
>> > > > > > > > > > 2017-02-17
>> 10:41
>> > > > GMT-08:00 Jason
>> > > > > > Gustafson <
>> > > > > > > > > jason@confluent.io>:
>> > > > > > > > > >
>> > > > > > > > > > > Sorry, my
>> > mistake.
>> > > > The consumer
>> > > > > > interceptor is per
>> > > > > > > > batch,
>> > > > > > > > > though I'm
>> > > > > > > > > > not
>> > > > > > > > > > > sure
>> that's an
>> > > > actual limitation
>> > > > > > since you still
>> > > > > > > have
>> > > > > > > > > access to the
>> > > > > > > > > > > individual
>> > records.
>> > > > > > > > > > >
>> > > > > > > > > > > -Jason
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri,
>> Feb 17,
>> > > 2017
>> > > > at 10:39 AM,
>> > > > > > Jason Gustafson <
>> > > > > > > > > >
>> jason@confluent.io>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Re
>> headers as
>> > > byte
>> > > > array and
>> > > > > > future use by
>> > > > > > > broker.
>> > > > > > > > This
>> > > > > > > > > doesn't
>> > > > > > > > > > take away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yeah, I
>> didn't
>> > > say
>> > > > it was
>> > > > > > difficult, only
>> > > > > > > awkward.
>> > > > > > > > You
>> > > > > > > > > wouldn't
>> > > > > > > > > > write the
>> > > > > > > > > > > > schema
>> that way
>> > > if
>> > > > you were
>> > > > > > planning to use it on
>> > > > > > > > the
>> > > > > > > > > brokers from
>> > > > > > > > > > the
>> > > > > > > > > > > >
>> beginning. Note
>> > > > also that one of
>> > > > > > the benefits of
>> > > > > > > > letting
>> > > > > > > > > the broker
>> > > > > > > > > > > >
>> understand
>> > > headers
>> > > > is that it can
>> > > > > > validate that
>> > > > > > > > they are
>> > > > > > > > > properly
>> > > > > > > > > > > >
>> formatted. If
>> > > cost
>> > > > is the only
>> > > > > > concern, we should
>> > > > > > > > > confirm its
>> > > > > > > > > > impact
>> > > > > > > > > > > > through
>> > > > performance testing.
>> > > > > > > > > > > >
>> > > > > > > > > > > > One of
>> the key
>> > > use
>> > > > cases requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'm not
>> sure I
>> > > > understand the
>> > > > > > point about
>> > > > > > > batching.
>> > > > > > > > > Interceptors
>> > > > > > > > > > are
>> > > > > > > > > > > > applied
>> > > > per-message, right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > My
>> intent on
>> > > > interceptors is to
>> > > > > > keep the usage of
>> > > > > > > > headers
>> > > > > > > > > > well-defined so
>> > > > > > > > > > > > that
>> they don't
>> > > > start leaking
>> > > > > > unnecessarily into
>> > > > > > > > > applications. My
>> > > > > > > > > > guess
>> > > > > > > > > > > is
>> > > > > > > > > > > > that it's
>> > > probably
>> > > > inevitable,
>> > > > > but
>> > > > > > isolating it
>> > > > > > > in
>> > > > > > > > the
>> > > > > > > > > > interceptors
>> would
>> > > > > > > > > > > > at least
>> give
>> > > > people a second
>> > > > > > thought before
>> > > > > > > > deciding to
>> > > > > > > > > use it.
>> > > > > > > > > > The main
>> > > > > > > > > > > >
>> challenge in my
>> > > > mind is figuring
>> > > > > > out how an MM
>> > > > > > > use
>> > > > > > > > case
>> > > > > > > > > would
>> > > > > > > > > > work. It
>> > > > > > > > > > > > would be
>> more
>> > > > cumbersome to
>> > > > > > replicate headers
>> > > > > > > > through an
>> > > > > > > > > > interceptor,
>> > > > > > > > > > > > though
>> arguably
>> > > MM
>> > > > should be
>> > > > > > working at a lower
>> > > > > > > > level
>> > > > > > > > > anyway.
>> > > > > > > > > > > >
>> > > > > > > > > > > > -Jason
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri,
>> Feb 17,
>> > > > 2017 at 10:16 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Re
>> headers
>> > > > available on the
>> > > > > > record va
>> > > > > > > > interceptors only
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> One of
>> the key
>> > > > use cases
>> > > > > requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> This
>> anyhow is
>> > > > similar to
>> > > > > > jms/http/amqp where
>> > > > > > > > headers
>> > > > > > > > > are
>> > > > > > > > > > available to
>> > > > > > > > > > > >>
>> consuming
>> > > > applications.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Re
>> headers as
>> > > > byte array and
>> > > > > > future use by
>> > > > > > > > broker. This
>> > > > > > > > > doesn't
>> > > > > > > > > > take
>> > > > > > > > > > > away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Sent
>> using OWA
>> > > > for iPhone
>> > > > > > > > > > > >>
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > > >> From:
>> Jason
>> > > > Gustafson <
>> > > > > > jason@confluent.io>
>> > > > > > > > > > > >> Sent:
>> Friday,
>> > > > February 17, 2017
>> > > > > > 5:55:42 PM
>> > > > > > > > > > > >> To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > > >>
>> Subject: Re:
>> > > > [DISCUSS] KIP-82 -
>> > > > > > Add Record
>> > > > > > > Headers
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Yes,
>> that is
>> > > what
>> > > > we are
>> > > > > > proposing, so using
>> > > > > > > > varints
>> > > > > > > > > for headers
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >>
>> consistent
>> > with
>> > > > the rest of the
>> > > > > > message. We have
>> > > > > > > > used
>> > > > > > > > > static
>> > > > > > > > > > helper
>> > > > > > > > > > > >> methods
>> > > > > > > > > > > >> in our
>> > prototype
>> > > > implementation.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> The
>> cost of
>> > > > parsing, we want to
>> > > > > > parse/interpret
>> > > > > > > > the
>> > > > > > > > > headers
>> > > > > > > > > > lazily (this
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > a key
>> point
>> > > > brought up earlier
>> > > > > > in discussions)
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I'm a
>> bit
>> > > > skeptical of this. Has
>> > > > > > anyone done the
>> > > > > > > > > performance
>> > > > > > > > > > testing? I
>> > > > > > > > > > > >> can
>> > > > > > > > > > > >> probably
>> > > > implement it and test
>> > > > > it
>> > > > > > if no one else
>> > > > > > > > has. I
>> > > > > > > > > was also
>> > > > > > > > > > under
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> impression
>> > that
>> > > > there may be use
>> > > > > > cases down the
>> > > > > > > > road
>> > > > > > > > > where the
>> > > > > > > > > > broker
>> > > > > > > > > > > >> would
>> > > > > > > > > > > >> need to
>> > > interpret
>> > > > headers. That
>> > > > > > wouldn't be off
>> > > > > > > > the
>> > > > > > > > > table in the
>> > > > > > > > > > future
>> > > > > > > > > > > if
>> > > > > > > > > > > >> it's
>> > represented
>> > > > as bytes, but
>> > > > > it
>> > > > > > would be quite
>> > > > > > > > a bit
>> > > > > > > > > more
>> > > > > > > > > > awkward,
>> > > > > > > > > > > >> right?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> By the
>> way,
>> > one
>> > > > question I have
>> > > > > > been wondering
>> > > > > > > > about. My
>> > > > > > > > > > understanding
>> > > > > > > > > > > is
>> > > > > > > > > > > >> that
>> headers
>> > are
>> > > > primarily for
>> > > > > > use cases where a
>> > > > > > > > > third-party
>> > > > > > > > > > components
>> > > > > > > > > > > >> wants to
>> > enrich
>> > > > messages without
>> > > > > > needing to
>> > > > > > > > understand
>> > > > > > > > > or modify
>> > > > > > > > > > the
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> of the
>> message
>> > > > key and value.
>> > > > > For
>> > > > > > the
>> > > > > > > applications
>> > > > > > > > > which directly
>> > > > > > > > > > > produce
>> > > > > > > > > > > >> and
>> consume
>> > the
>> > > > messages and
>> > > > > > control the
>> > > > > > > key/value
>> > > > > > > > > schema
>> > > > > > > > > > directly, it
>> > > > > > > > > > > >> seems
>> we would
>> > > > rather have them
>> > > > > > implement
>> > > > > > > headers
>> > > > > > > > > directly in
>> > > > > > > > > > their own
>> > > > > > > > > > > >> schema.
>> > > Supposing
>> > > > for the sake
>> > > > > of
>> > > > > > argument that
>> > > > > > > > it was
>> > > > > > > > > possible,
>> > > > > > > > > > my
>> > > > > > > > > > > >>
>> question is
>> > > > whether it be
>> > > > > > sufficient to expose
>> > > > > > > the
>> > > > > > > > > headers in the
>> > > > > > > > > > > >>
>> interceptor
>> > API
>> > > > and not in the
>> > > > > > common API?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> -Jason
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Fri,
>> Feb
>> > 17,
>> > > > 2017 at 3:26 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > On
>> the point
>> > > of
>> > > > varInts
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Cheers
>> > > > > > > > > > > >> > Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > 17/02/2017,
>> > > > 11:22, "Michael
>> > > > > > Pearce" <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On the
>> > > > point re: headers
>> > > > > in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > being a
>> > > > > > > > > > byte
>> > > > > > > > > > > array
>> > > > > > > > > > > >> > and
>> not a
>> > > count
>> > > > of elements
>> > > > > > followed by the
>> > > > > > > > elements.
>> > > > > > > > > Again
>> > > > > > > > > > this was
>> > > > > > > > > > > >> >
>> > > > discussed/argued previously.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> It was
>> > > > agreed on for a few
>> > > > > > reasons some of
>> > > > > > > > which
>> > > > > > > > > you have
>> > > > > > > > > > > obviously
>> > > > > > > > > > > >> >
>> picked up
>> > on:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Broker
>> > is
>> > > > able to pass it
>> > > > > > through opaquely
>> > > > > > > > > > > >> >
>> The cost
>> > > of
>> > > > parsing, we
>> > > > > > want to
>> > > > > > > > parse/interpret
>> > > > > > > > > the headers
>> > > > > > > > > > lazily
>> > > > > > > > > > > >> > (this
>> is a
>> > key
>> > > > point brought
>> > > > > up
>> > > > > > earlier in
>> > > > > > > > > discussions)
>> > > > > > > > > > > >> >
>> Headers
>> > > can
>> > > > be copied from
>> > > > > > consumer record
>> > > > > > > > to
>> > > > > > > > > producer
>> > > > > > > > > > record (aka
>> > > > > > > > > > > >> > mirror
>> > makers
>> > > > etc) without
>> > > > > > parsing if no
>> > > > > > > > changes are
>> > > > > > > > > being made
>> > > > > > > > > > or
>> > > > > > > > > > > being
>> > > > > > > > > > > >> >
>> looked at.
>> > > > > > > > > > > >> >
>> Keeps
>> > the
>> > > > broker agnostic
>> > > > > > to the format
>> > > > > > > > > > > >> >
>> You need
>> > > an
>> > > > int32 either
>> > > > > > for the byte size
>> > > > > > > > of the
>> > > > > > > > > headers,
>> > > > > > > > > > or for
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > count
>> of
>> > > > elements, so
>> > > > > overheads
>> > > > > > are the same,
>> > > > > > > > but
>> > > > > > > > > with going
>> > > > > > > > > > with an
>> > > > > > > > > > > >> opaque
>> > > > > > > > > > > >> > byte
>> array
>> > has
>> > > > the above
>> > > > > > advantages.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Cheers
>> > > > > > > > > > > >> >
>> Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > > > 17/02/2017, 02:50,
>> > > > > > "Jason Gustafson" <
>> > > > > > > > > jason@confluent.io
>> > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > Sorry,
>> > > > should have
>> > > > > > noted that the
>> > > > > > > > performance
>> > > > > > > > > testing
>> > > > > > > > > > was done
>> > > > > > > > > > > >> > using
>> the
>> > > > > > > > > > > >> >
>> > > > producer performance
>> > > > > > tool shipped with
>> > > > > > > > Kafka.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > -Jason
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On
>> > > Thu,
>> > > > Feb 16, 2017
>> > > > > at
>> > > > > > 6:44 PM, Jason
>> > > > > > > > > Gustafson <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> >
>> > Hey
>> > > > Nacho,
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > I've
>> > > > compared
>> > > > > > performance of our
>> > > > > > > > KIP-98
>> > > > > > > > > > implementation
>> with
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> without
>> > > > > > > > > > > >> >
>> >
>> > > > varints. For
>> > > > > messages
>> > > > > > around 128
>> > > > > > > > bytes, we
>> > > > > > > > > see an
>> > > > > > > > > > increase
>> > > > > > > > > > > in
>> > > > > > > > > > > >> >
>> throughput
>> > of
>> > > > > > > > > > > >> >
>> >
>> > > about
>> > > > 30% using the
>> > > > > > default
>> > > > > > > > configuration
>> > > > > > > > > settings.
>> > > > > > > > > > At 256
>> > > > > > > > > > > >> >
>> bytes, the
>> > > > > > > > > > > >> >
>> >
>> > > > increase is around
>> > > > > > 16%. Obviously
>> > > > > > > the
>> > > > > > > > > performance
>> > > > > > > > > > converge
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> messages get
>> > > > > > > > > > > >> >
>> >
>> > > > larger, but it seems
>> > > > > > well worth the
>> > > > > > > > cost.
>> > > > > > > > > Note that
>> > > > > > > > > > we are
>> > > > > > > > > > > >> also
>> > > > > > > > > > > >> >
>> seeing a
>> > > > > > > > > > > >> >
>> >
>> > > > substantial
>> > > > > > performance increase
>> > > > > > > > against
>> > > > > > > > > trunk
>> > > > > > > > > > primarily
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >
>> > much
>> > > > more efficient
>> > > > > > packing that
>> > > > > > > > varints
>> > > > > > > > > provide us.
>> > > > > > > > > > > Anything
>> > > > > > > > > > > >> >
>> adding to
>> > > > > > > > > > > >> >
>> >
>> > > > message overhead,
>> > > > > > such as record
>> > > > > > > > headers,
>> > > > > > > > > would only
>> > > > > > > > > > > increase
>> > > > > > > > > > > >> > the
>> relative
>> > > > > > > > > > > >> >
>> >
>> > > > difference. (Of
>> > > > > > course take these
>> > > > > > > > numbers
>> > > > > > > > > with a
>> > > > > > > > > > grain of
>> > > > > > > > > > > salt
>> > > > > > > > > > > >> > since
>> I have
>> > > > > > > > > > > >> >
>> >
>> > only
>> > > > used the
>> > > > > default
>> > > > > > settings with
>> > > > > > > > both
>> > > > > > > > > the producer
>> > > > > > > > > > and
>> > > > > > > > > > > >> broker
>> > > > > > > > > > > >> > on my
>> > > > > > > > > > > >> >
>> >
>> > > local
>> > > > machine. We
>> > > > > > intend to provide
>> > > > > > > > more
>> > > > > > > > > extensive
>> > > > > > > > > > > performance
>> > > > > > > > > > > >> >
>> details as
>> > > > > > > > > > > >> >
>> >
>> > part
>> > > > of the work for
>> > > > > > KIP-98.)
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > The
>> > > > implementation
>> > > > > we
>> > > > > > are using is
>> > > > > > > > from
>> > > > > > > > > protobuf (
>> > > > > > > > > > > >> >
>> >
>> > > > > > https://developers.google.com/
>> > > > > > > > > > >
>> > > > protocol-buffers/docs/encoding
>> > > > > > > > > > > >> ),
>> > > > > > > > > > > >> > which
>> is
>> > > > > > > > > > > >> >
>> >
>> > also
>> > > > used in HBase.
>> > > > > > It is trivial to
>> > > > > > > > > implement and as
>> > > > > > > > > > far
>> > > > > > > > > > > as I
>> > > > > > > > > > > >> > know
>> doesn't
>> > > > > > > > > > > >> >
>> >
>> > > > suffer from the
>> > > > > > aliasing problem you
>> > > > > > > > are
>> > > > > > > > > describing. I
>> > > > > > > > > > > checked
>> > > > > > > > > > > >> > with
>> Magnus
>> > > > > > > > > > > >> >
>> >
>> > (the
>> > > > author of
>> > > > > > librdkafka) and he
>> > > > > > > > agreed
>> > > > > > > > > that the
>> > > > > > > > > > savings
>> > > > > > > > > > > >> seemed
>> > > > > > > > > > > >> > worth
>> the
>> > > > > > > > > > > >> >
>> >
>> > cost
>> > > > of
>> > > > > > implementation.
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > -Jason
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > On
>> > > > Thu, Feb 16, 2017
>> > > > > > at 4:32 PM,
>> > > > > > > > Ignacio
>> > > > > > > > > Solis <
>> > > > > > > > > > > >>
>> > isolis@igso.net
>> > > >
>> > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >>
>> > > > -VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > I'm
>> > > > one of the
>> > > > > > people (if not the
>> > > > > > > > most)
>> > > > > > > > > opposed to
>> > > > > > > > > > VarInts.
>> > > > > > > > > > > >> >
>> VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > have
>> > > > a place, but
>> > > > > > this is not it.
>> > > > > > > > (We
>> > > > > > > > > had a large
>> > > > > > > > > > > >>
>> discussion
>> > > > > > > > > > > >> > about
>> > > > > > > > > > > >> >
>> >>
>> > > them
>> > > > at the
>> > > > > > beginning of KIP-82
>> > > > > > > time)
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > anybody has real
>> > > > > > life
>> > > > > > > performance
>> > > > > > > > > numbers of
>> > > > > > > > > > VarInts
>> > > > > > > > > > > >> >
>> improving
>> > > > > > > > > > > >> >
>> >>
>> > > > things or
>> > > > > > significantly reducing
>> > > > > > > > resources
>> > > > > > > > > I would
>> > > > > > > > > > like to
>> > > > > > > > > > > >> know
>> > > > > > > > > > > >> > what
>> > > > > > > > > > > >> >
>> >>
>> > > that
>> > > > case may be.
>> > > > > > Yes, you can save
>> > > > > > > > some
>> > > > > > > > > bytes here
>> > > > > > > > > > and
>> > > > > > > > > > > >> there,
>> > > > > > > > > > > >> > but
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > is probably
>> > > > > > insignificant to
>> > > > > > > the
>> > > > > > > > > overall system
>> > > > > > > > > > > behavior
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> >
>> >>
>> > > > storage
>> > > > > > requirements. -- I say
>> > > > > > > this
>> > > > > > > > with
>> > > > > > > > > respect to
>> > > > > > > > > > using
>> > > > > > > > > > > >> >
>> VarInts in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > protocol
>> > > > > itself,
>> > > > > > not as part of
>> > > > > > > > the
>> > > > > > > > > data.
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > VarInts require you
>> > > > > > to parse the
>> > > > > > > Int
>> > > > > > > > > before using it
>> > > > > > > > > > and
>> > > > > > > > > > > >> >
>> depending on
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > encoding they
>> > > > > > can suffer from
>> > > > > > > > aliasing
>> > > > > > > > > (multiple
>> > > > > > > > > > > >> >
>> > > representations
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > the same
>> > > > > value).
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > Why
>> > > > add complexity?
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > Nacho
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > On
>> > > > Thu, Feb 16,
>> > > > > 2017
>> > > > > > at 10:29 AM,
>> > > > > > > > Colin
>> > > > > > > > > McCabe <
>> > > > > > > > > > > >> >
>> > > > cmccabe@apache.org>
>> > > > > > > > > > > >> >
>> >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >> >
>> > > +1
>> > > > for varints
>> > > > > > here-- it would
>> > > > > > > save
>> > > > > > > > > quite a bit of
>> > > > > > > > > > space.
>> > > > > > > > > > > >> > They
>> are
>> > > > > > > > > > > >> >
>> >> >
>> > > > pretty quick to
>> > > > > > implement as
>> > > > > > > well.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > > I
>> > > > think it makes
>> > > > > > sense for values
>> > > > > > > > to be
>> > > > > > > > > byte
>> > > > > > > > > > arrays.
>> > > > > > > > > > > Users
>> > > > > > > > > > > >> > might
>> want
>> > > > > > > > > > > >> >
>> >> >
>> > > to
>> > > > attach
>> > > > > > arbitrary payloads;
>> > > > > > > they
>> > > > > > > > > shouldn't be
>> > > > > > > > > > forced to
>> > > > > > > > > > > >> >
>> serialize
>> > > > > > > > > > > >> >
>> >> >
>> > > > everything to
>> > > > > Java
>> > > > > > strings.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > best,
>> > > > > > > > > > > >> >
>> >> >
>> > > > Colin
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > On
>> > > > Thu, Feb 16,
>> > > > > > 2017, at 09:52,
>> > > > > > > > Jason
>> > > > > > > > > Gustafson
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hey Michael,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hmm, I guess the
>> > > > > > point of
>> > > > > > > > representing
>> > > > > > > > > it as
>> > > > > > > > > > bytes is to
>> > > > > > > > > > > >> > allow
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > broker
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > to pass it
>> > > > > > through opaquely? Is
>> > > > > > > > the
>> > > > > > > > > cost of
>> > > > > > > > > > parsing
>> > > > > > > > > > > them a
>> > > > > > > > > > > >> >
>> concern, or
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > are
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > we simply trying
>> > > > > > to ensure that
>> > > > > > > > the
>> > > > > > > > > broker stays
>> > > > > > > > > > > agnostic
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > format?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On varints, I
>> > > > > > think adding
>> > > > > > > > support for
>> > > > > > > > > them makes
>> > > > > > > > > > less
>> > > > > > > > > > > >> sense
>> > > > > > > > > > > >> > for an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > isolated use
>> > > > > > case, but as part
>> > > > > > > of
>> > > > > > > > a
>> > > > > > > > > more holistic
>> > > > > > > > > > change
>> > > > > > > > > > > >> > (such
>> as
>> > what
>> > > > > > > > > > > >> >
>> >>
>> > we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > have proposed in
>> > > > > > KIP-98), I
>> > > > > > > think
>> > > > > > > > they
>> > > > > > > > > are
>> > > > > > > > > > justifiable.
>> > > > > > > > > > > If
>> > > > > > > > > > > >> > we
>> add them,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > then the need to
>> > > > > > use attributes
>> > > > > > > > becomes
>> > > > > > > > > quite a
>> > > > > > > > > > bit
>> > > > > > > > > > > >> weaker,
>> > > > > > > > > > > >> >
>> right? The
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > other thing I
>> > > > > > find slightly odd
>> > > > > > > > is the
>> > > > > > > > > fact that
>> > > > > > > > > > null
>> > > > > > > > > > > >> >
>> headers has
>> > no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > actual
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > semantic meaning
>> > > > > > for the message
>> > > > > > > > > (unlike null
>> > > > > > > > > > keys and
>> > > > > > > > > > > >> >
>> values). It
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > a
>> > > > space
>> > > > > > optimization. It seems a
>> > > > > > > > bit
>> > > > > > > > > better to
>> > > > > > > > > > always
>> > > > > > > > > > > use
>> > > > > > > > > > > >> > size
>> 0 to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > indicate having
>> > > > > > no headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Overall, the
>> > > > > main
>> > > > > > point is
>> > > > > > > > ensuring
>> > > > > > > > > that the
>> > > > > > > > > > message
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> >
>> remains
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > consistent,
>> > > > > > either within the
>> > > > > > > > larger
>> > > > > > > > > protocol, or
>> > > > > > > > > > at a
>> > > > > > > > > > > >> >
>> minimum
>> > within
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > message itself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On Thu, Feb 16,
>> > > > > > 2017 at 6:39 AM,
>> > > > > > > > > Michael Pearce <
>> > > > > > > > > > > >> >
>> > > > Michael.Pearce@ig.com
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Hi Jason,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 1) in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > the headers
>> > > > > > > > > > are
>> > > > > > > > > > > >> simply
>> > > > > > > > > > > >> > a byte
>> > > > > > > > > > > >> >
>> >>
>> > > > array,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > as like the
>> > > > > key
>> > > > > > or value, this
>> > > > > > > > is to
>> > > > > > > > > clearly
>> > > > > > > > > > demarcate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> header in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > core message.
>> > > > > > Then the header
>> > > > > > > > byte
>> > > > > > > > > array in the
>> > > > > > > > > > core
>> > > > > > > > > > > >> >
>> message is
>> > an
>> > > > > > > > > > > >> >
>> >>
>> > > > array of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key, value
>> > > > > > pairs. This is what
>> > > > > > > > it is
>> > > > > > > > > denoting.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Then this
>> > > > > would
>> > > > > > be I guess in
>> > > > > > > > the
>> > > > > > > > > given
>> > > > > > > > > > notation:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Headers =>
>> > > > > > [KeyLength, Key,
>> > > > > > > > > ValueLength, Value]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > KeyLength
>> > > > > > => int32
>> > > > > > > > > <-----------------NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised key
>> > > > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Key =>
>> > > > > bytes
>> > > > > > > > > <---------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > string
>> > (UTF8)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > bytes of the
>> > > > > > header key
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > ValueLength
>> > > > > > => int32
>> > > > > > > > > <-------------- NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised
>> > > > > > header value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Value =>
>> > > > > > bytes
>> > > > > > > > > <-------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > form
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The key length
>> > > > > > and value
>> > > > > > > length
>> > > > > > > > is
>> > > > > > > > > matching the
>> > > > > > > > > > way
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> protocol is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > defined in the
>> > > > > > core message
>> > > > > > > > currently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 2)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Var sized
>> > > > > ints,
>> > > > > > this was
>> > > > > > > > discussed
>> > > > > > > > > much earlier
>> > > > > > > > > > on, in
>> > > > > > > > > > > >> > fact
>> I had
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > suggested it
>> > > > > > myself (with
>> > > > > > > Hadoop
>> > > > > > > > > references),
>> > > > > > > > > > the
>> > > > > > > > > > > >> >
>> complexity
>> > of
>> > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compared to
>> > > > > > having a simpler
>> > > > > > > > protocol
>> > > > > > > > > was
>> > > > > > > > > > argued and
>> > > > > > > > > > > >> >
>> agreed it
>> > > > > > > > > > > >> >
>> >>
>> > > > wasn’t worth
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the complexity
>> > > > > > as all other
>> > > > > > > > clients
>> > > > > > > > > in other
>> > > > > > > > > > languages
>> > > > > > > > > > > >> > would
>> need
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > > > ensure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > theyre using
>> > > > > > the right var
>> > > > > > > size
>> > > > > > > > > algorithm, as
>> > > > > > > > > > there
>> > > > > > > > > > > is a
>> > > > > > > > > > > >> > few.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 3)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > We did the
>> > > > > > attributes,
>> > > > > > > optional
>> > > > > > > > > approach as
>> > > > > > > > > > originally
>> > > > > > > > > > > >> > there
>> was
>> > > > > > > > > > > >> >
>> >>
>> > > > marked
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > concern that
>> > > > > > headers would
>> > > > > > > > cause a
>> > > > > > > > > message size
>> > > > > > > > > > > overhead
>> > > > > > > > > > > >> > for
>> others,
>> > > > > > > > > > > >> >
>> >>
>> > who
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > don’t want
>> > > > > > them. As such this
>> > > > > > > > is the
>> > > > > > > > > clean
>> > > > > > > > > > solution to
>> > > > > > > > > > > >> >
>> achieve
>> > that.
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that no longer
>> > > > > > holds, and we
>> > > > > > > > don’t
>> > > > > > > > > care that we
>> > > > > > > > > > add
>> > > > > > > > > > > >> 4bytes
>> > > > > > > > > > > >> >
>> overhead,
>> > > > > > > > > > > >> >
>> >>
>> > > then
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > im happy to
>> > > > > > remove.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > I’m personally
>> > > > > > in favour of
>> > > > > > > > keeping
>> > > > > > > > > the message
>> > > > > > > > > > as
>> > > > > > > > > > > small
>> > > > > > > > > > > >> > as
>> possible
>> > > > > > > > > > > >> >
>> >>
>> > so
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > people don’t
>> > > > > > get shocks in
>> > > > > > > perf
>> > > > > > > > and
>> > > > > > > > > throughputs
>> > > > > > > > > > dues
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> message
>> > size,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > unless they
>> > > > > > actively use the
>> > > > > > > > feature,
>> > > > > > > > > as such I
>> > > > > > > > > > do
>> > > > > > > > > > > >> prefer
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > attribute bit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wise feature
>> > > > > > flag approach
>> > > > > > > > myself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On 16/02/2017,
>> > > > > > 05:40, "Jason
>> > > > > > > > > Gustafson" <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > We have
>> > > > > > proposed a few
>> > > > > > > > > significant changes
>> > > > > > > > > > to the
>> > > > > > > > > > > >> >
>> message
>> > format
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > KIP-98
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > which now
>> > > > > > seems likely to
>> > > > > > > > pass
>> > > > > > > > > (perhaps
>> > > > > > > > > > with some
>> > > > > > > > > > > >> >
>> iterations
>> > on
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > implementation details).
>> > > > > > > It
>> > > > > > > > would
>> > > > > > > > > be good
>> > > > > > > > > > to try
>> > > > > > > > > > > and
>> > > > > > > > > > > >> >
>> coordinate
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > changes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > in both of
>> > > > > > the proposals
>> > > > > > > to
>> > > > > > > > make
>> > > > > > > > > sure they
>> > > > > > > > > > are
>> > > > > > > > > > > >> >
>> consistent
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compatible.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I think
>> > > > > > using the
>> > > > > > > > attributes to
>> > > > > > > > > indicate
>> > > > > > > > > > null
>> > > > > > > > > > > >> headers
>> > > > > > > > > > > >> > is a
>> > > > > > > > > > > >> >
>> >>
>> > > > reasonable
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > approach.
>> > > > > > We have proposed
>> > > > > > > > to do
>> > > > > > > > > the same
>> > > > > > > > > > thing
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> message
>> > > > > > > > > > > >> >
>> >>
>> > key
>> > > > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > value.
>> > > > > That
>> > > > > > said, I
>> > > > > > > > sympathize
>> > > > > > > > > with Jay's
>> > > > > > > > > > > argument.
>> > > > > > > > > > > >> > Having
>> > > > > > > > > > > >> >
>> >>
>> > > > multiple
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > ways to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > specify a
>> > > > > > null value
>> > > > > > > > increases
>> > > > > > > > > the overall
>> > > > > > > > > > > >>
>> complexity
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > You
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > can see
>> > > > > > this just from the
>> > > > > > > > fact
>> > > > > > > > > that you
>> > > > > > > > > > need the
>> > > > > > > > > > > >> > extra
>> > verbiage
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > protocol
>> > > > > > specification in
>> > > > > > > > this
>> > > > > > > > > KIP and in
>> > > > > > > > > > KIP-98
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> describe the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > dependence
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > between
>> > > > > the
>> > > > > > fields and the
>> > > > > > > > > attributes. It
>> > > > > > > > > > seems
>> > > > > > > > > > > >> like a
>> > > > > > > > > > > >> >
>> slippery
>> > > > > > > > > > > >> >
>> >>
>> > > slope
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > if
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > you start
>> > > > > > allowing
>> > > > > > > different
>> > > > > > > > > request types
>> > > > > > > > > > to
>> > > > > > > > > > > >> >
>> implement
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification differently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > You can
>> > > > > > also argue that
>> > > > > > > the
>> > > > > > > > > messages
>> > > > > > > > > > already are
>> > > > > > > > > > > and
>> > > > > > > > > > > >> > are
>> likely
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > remain a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > special
>> > > > > > case. For example,
>> > > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > currently no
>> > > > > > > > > > > >> >
>> generality
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > how
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > compressed
>> > > > > > message sets
>> > > > > > > are
>> > > > > > > > > represented
>> > > > > > > > > > that would
>> > > > > > > > > > > >> be
>> > > > > > > > > > > >> >
>> applicable
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > other
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > request
>> > > > > > types. Some might
>> > > > > > > > see this
>> > > > > > > > > > divergence as
>> > > > > > > > > > > an
>> > > > > > > > > > > >> >
>> unfortunate
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > deficiency
>> > > > > > which should be
>> > > > > > > > fixed;
>> > > > > > > > > others
>> > > > > > > > > > might see
>> > > > > > > > > > > >> it
>> > > > > > > > > > > >> > as
>> sort of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > inevitability of needing
>> > > > > > > to
>> > > > > > > > > optimize where
>> > > > > > > > > > it
>> > > > > > > > > > > counts
>> > > > > > > > > > > >> > most.
>> I'm
>> > > > > > > > > > > >> >
>> >>
>> > > > probably
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > somewhere
>> > > > > > in between, but
>> > > > > > > I
>> > > > > > > > think
>> > > > > > > > > we
>> > > > > > > > > > probably all
>> > > > > > > > > > > >> > share
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > intuition
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol should be
>> > > > > > > kept
>> > > > > > > > as
>> > > > > > > > > consistent as
>> > > > > > > > > > > >>
>> possible.
>> > > > > > > > > > > >> > With
>> that
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > mind,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > here are a
>> > > > > > few comments:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 1. One
>> > > > > > thing I found a
>> > > > > > > > little odd
>> > > > > > > > > when
>> > > > > > > > > > reading the
>> > > > > > > > > > > >> >
>> current
>> > > > > > > > > > > >> >
>> >>
>> > > > proposal is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > headers
>> > > > > > are both
>> > > > > > > > represented
>> > > > > > > > > as an
>> > > > > > > > > > array of
>> > > > > > > > > > > >> bytes
>> > > > > > > > > > > >> > and
>> as an
>> > > > > > > > > > > >> >
>> >>
>> > > array
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > key/value
>> > > > > > pairs. I'd
>> > > > > > > > probably
>> > > > > > > > > suggest
>> > > > > > > > > > something
>> > > > > > > > > > > like
>> > > > > > > > > > > >> > this:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > [HeaderKey
>> > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > HeaderKey
>> > > > > > => String
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue => Bytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > An array
>> > > > > in
>> > > > > > the Kafka
>> > > > > > > > protocol is
>> > > > > > > > > > represented as a
>> > > > > > > > > > > >> > 4-byte
>> > integer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > indicating
>> > > > > > the number of
>> > > > > > > > elements
>> > > > > > > > > in the
>> > > > > > > > > > array
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > serialization of the
>> > > > > > > > elements.
>> > > > > > > > > Unless I'm
>> > > > > > > > > > > >> >
>> > > > misunderstanding, what
>> > > > > > > > > > > >> >
>> >>
>> > you
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > instead is
>> > > > > > the total size
>> > > > > > > > of the
>> > > > > > > > > headers in
>> > > > > > > > > > bytes
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > elements.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I'm not
>> > > > > > sure I see any
>> > > > > > > > reason for
>> > > > > > > > > this
>> > > > > > > > > > > >>
>> inconsistency.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 2. In
>> > > > > > KIP-98, we've
>> > > > > > > > introduced
>> > > > > > > > > > variable-length
>> > > > > > > > > > > >> integer
>> > > > > > > > > > > >> >
>> fields.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Effectively,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > we've
>> > > > > > enriched (or
>> > > > > > > > "complicated"
>> > > > > > > > > as Jay
>> > > > > > > > > > might say
>> > > > > > > > > > > ;)
>> > > > > > > > > > > >> > the
>> protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification to include
>> > > > > > > the
>> > > > > > > > > following
>> > > > > > > > > > types:
>> > > > > > > > > > > >> VarInt,
>> > > > > > > > > > > >> >
>> VarLong,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > UnsignedVarInt and
>> > > > > > > > > UnsignedVarLong.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Along with
>> > > > > > these
>> > > > > > > > primitives, we
>> > > > > > > > > could
>> > > > > > > > > > introduce
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> following
>> > > > > > > > > > > >> >
>> >>
>> > > > types:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeArray =>
>> > > > > > > > NumberOfItems
>> > > > > > > > > Item1 Item2
>> > > > > > > > > > .. ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItems =>
>> > > > > > > > UnsignedVarInt
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeNullableArray =>
>> > > > > > > > > NumberOfItemsOrNull
>> > > > > > > > > > Item1
>> > > > > > > > > > > >> > Item2
>> ..
>> > ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItemsOrNull =>
>> > > > > > > > VarInt
>> > > > > > > > > (-1 means
>> > > > > > > > > > null)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > And
>> > > > > > similarly for the
>> > > > > > > > `String`
>> > > > > > > > > and `Bytes`
>> > > > > > > > > > types.
>> > > > > > > > > > > >> > These
>> types
>> > > > > > > > > > > >> >
>> >>
>> > can
>> > > > save
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > considerable amount of
>> > > > > > > > space in
>> > > > > > > > > this
>> > > > > > > > > > proposal
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > they
>> can
>> > > > > > > > > > > >> >
>> >>
>> > be
>> > > > used
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > both the
>> > > > > > number of headers
>> > > > > > > > > included in the
>> > > > > > > > > > message
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > lengths of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > header
>> > > > > keys
>> > > > > > and values. We
>> > > > > > > > could
>> > > > > > > > > do this
>> > > > > > > > > > instead:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > > > VarSizeArray[HeaderKey
>> > > > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > HeaderKey
>> > > > > > =>
>> > > > > > > VarSizeString
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue =>
>> > > > > > > > VarSizeBytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Combining
>> > > > > > the savings from
>> > > > > > > > the
>> > > > > > > > > use of
>> > > > > > > > > > variable
>> > > > > > > > > > > >> length
>> > > > > > > > > > > >> >
>> fields, the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > benefit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > of using
>> > > > > > the attributes to
>> > > > > > > > > represent null
>> > > > > > > > > > seems
>> > > > > > > > > > > >> pretty
>> > > > > > > > > > > >> > small.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 3.
>> > > > > > Whichever way we go
>> > > > > > > > (whether
>> > > > > > > > > we use the
>> > > > > > > > > > > >>
>> attributes
>> > > > > > > > > > > >> > or
>> not), we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > should at
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > least be
>> > > > > > consistent
>> > > > > > > between
>> > > > > > > > this
>> > > > > > > > > KIP and
>> > > > > > > > > > KIP-98.
>> > > > > > > > > > > It
>> > > > > > > > > > > >> > would
>> be
>> > very
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > strange
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > to have
>> > > > > two
>> > > > > > ways to
>> > > > > > > > represent
>> > > > > > > > > null values
>> > > > > > > > > > in the
>> > > > > > > > > > > >> same
>> > > > > > > > > > > >> >
>> schema.
>> > > > > > > > > > > >> >
>> >>
>> > > > Either
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > way is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > OK with
>> > > > > me.
>> > > > > > I think some
>> > > > > > > > > message-level
>> > > > > > > > > > >
>> optimizations
>> > > > > > > > > > > >> > are
>> > > > > > > > > > > >> >
>> >>
>> > > > justifiable,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > but
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > savings
>> > > > > > here seem
>> > > > > > > > minimal (a
>> > > > > > > > > few bytes
>> > > > > > > > > > per
>> > > > > > > > > > > >> >
>> message), so
>> > > > > > > > > > > >> >
>> >>
>> > > > maybe it's
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > worth the
>> > > > > > cost of letting
>> > > > > > > > the
>> > > > > > > > > message
>> > > > > > > > > > diverge even
>> > > > > > > > > > > >> >
>> further from
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > rest of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > On Wed,
>> > > > > Feb
>> > > > > > 15, 2017 at
>> > > > > > > > 8:52 AM,
>> > > > > > > > > radai <
>> > > > > > > > > > > >> >
>> >>
>> > > > > > radai.rosenblatt@gmail.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > I've
>> > > > > > trimmed the inline
>> > > > > > > > > contents as this
>> > > > > > > > > > mail is
>> > > > > > > > > > > >> >
>> getting too
>> > > > > > > > > > > >> >
>> >>
>> > big
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > apache
>> > > > > > mailing list
>> > > > > > > > software to
>> > > > > > > > > deliver
>> > > > > > > > > > :-(
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 1. the
>> > > > > > important thing
>> > > > > > > for
>> > > > > > > > > > interoperability
>> is
>> > > > > > > > > > > for
>> > > > > > > > > > > >> >
>> different
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "interested
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > parties"
>> > > > > > (plugins, infra
>> > > > > > > > > layers/wrappers,
>> > > > > > > > > > > >>
>> user-code)
>> > > > > > > > > > > >> > to be
>> > > > > > > > > > > >> >
>> >>
>> > > able
>> > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > stick
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > pieces
>> > > > > of
>> > > > > > metadata onto
>> > > > > > > > msgs
>> > > > > > > > > without
>> > > > > > > > > > getting in
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> >
>> other's
>> > > > > > > > > > > >> >
>> >>
>> > > way.
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > common
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > key
>> > > > > > scheme (Strings, as
>> > > > > > > > of the
>> > > > > > > > > time of
>> > > > > > > > > > this
>> > > > > > > > > > > >> >
>> writing?) is
>> > > all
>> > > > > > > > > > > >> >
>> >>
>> > > thats
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > required
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > for
>> > > > > that.
>> > > > > > it is assumed
>> > > > > > > > that
>> > > > > > > > > the other end
>> > > > > > > > > > > >> >
>> interested
>> > in
>> > > > any
>> > > > > > > > > > > >> >
>> >>
>> > > such
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > piece of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > metadata
>> > > > > > knows the
>> > > > > > > > encoding,
>> > > > > > > > > and byte[]
>> > > > > > > > > > provides
>> > > > > > > > > > > >> for
>> > > > > > > > > > > >> > the
>> most
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > flexibility.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > i
>> > > > > believe
>> > > > > > this is the
>> > > > > > > same
>> > > > > > > > > logic behind
>> > > > > > > > > > core
>> > > > > > > > > > > kafka
>> > > > > > > > > > > >> > being
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > byte[]/byte[]
>> > > > > -
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Strings
>> > > > > > are more
>> > > > > > > "usable"
>> > > > > > > > but
>> > > > > > > > > bytes are
>> > > > > > > > > > flexible
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > so
>> were
>> > > > > > > > > > > >> >
>> >>
>> > > > chosen.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Also -
>> > > > > > core kafka doesnt
>> > > > > > > > even
>> > > > > > > > > do that
>> > > > > > > > > > good of a
>> > > > > > > > > > > >> job
>> > > > > > > > > > > >> > on
>> > > > > > > > > > > >> >
>> >>
>> > > > usability of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > payload
>> > > > > > (example - i
>> > > > > > > have
>> > > > > > > > to
>> > > > > > > > > specify the
>> > > > > > > > > > nop
>> > > > > > > > > > > >> byte[]
>> > > > > > > > > > > >> >
>> "decoders"
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > explicitly
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > in
>> > > > > conf),
>> > > > > > and again
>> > > > > > > > sacrificies
>> > > > > > > > > usability
>> > > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > sake
>> of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > performance
>> > > > > (no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > convenient single-record
>> > > > > > > > > processing as
>> > > > > > > > > > poll is a
>> > > > > > > > > > > >> >
>> batch, lots
>> > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > obscure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > little
>> > > > > > config details
>> > > > > > > > exposing
>> > > > > > > > > internals
>> > > > > > > > > > of the
>> > > > > > > > > > > >> >
>> batching
>> > > > > > > > > > > >> >
>> >>
>> > > > mechanism,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > etc)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > this is
>> > > > > > also why i
>> > > > > > > really
>> > > > > > > > > dislike the
>> > > > > > > > > > idea of a
>> > > > > > > > > > > >> > "type
>> > system"
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > values,
>> > > > > > it further
>> > > > > > > > degrades the
>> > > > > > > > > > usability, adds
>> > > > > > > > > > > >> >
>> complexity
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > eventually get in
>> > > > > > > > people's way,
>> > > > > > > > > also, it
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >> > the
>> 2nd/3rd
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > home-group
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization mechanism
>> > > > > > > > in core
>> > > > > > > > > kafka
>> > > > > > > > > > (counting
>> > > > > > > > > > > 2
>> > > > > > > > > > > >> >
>> iterations
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "type
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > definition DSL")
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 2. this
>> > > > > > is an
>> > > > > > > > implementation
>> > > > > > > > > detail, and
>> > > > > > > > > > not
>> > > > > > > > > > > even
>> > > > > > > > > > > >> a
>> > > > > > > > > > > >> > very
>> "user
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > facing" one?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to the
>> > > > > > best of my
>> > > > > > > > understanding
>> > > > > > > > > the vote
>> > > > > > > > > > process
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > on
>> proposed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > API/behaviour. also -
>> > > > > > > > since
>> > > > > > > > > we're willing
>> > > > > > > > > > to go
>> > > > > > > > > > > >> with
>> > > > > > > > > > > >> >
>> strings
>> > > > > > > > > > > >> >
>> >>
>> > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialize
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > a
>> > > > > 0-sized
>> > > > > > header blob
>> > > > > > > and
>> > > > > > > > IIUC
>> > > > > > > > > you dont
>> > > > > > > > > > need any
>> > > > > > > > > > > >> >
>> optionals
>> > > > > > > > > > > >> >
>> >>
>> > > > anymore.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 3. yes,
>> > > > > > we can :-)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > On Tue,
>> > > > > > Feb 14, 2017 at
>> > > > > > > > 11:56
>> > > > > > > > > PM, Michael
>> > > > > > > > > > > Pearce <
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Hi
>> > > > > Jay,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1)
>> > > > > > There was some
>> > > > > > > > initial
>> > > > > > > > > debate on the
>> > > > > > > > > > value
>> > > > > > > > > > > >> > part,
>> as
>> > youll
>> > > > > > > > > > > >> >
>> >>
>> > > note
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > String,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > String
>> > > > > > headers were
>> > > > > > > > > discounted early
>> > > > > > > > > > on. The
>> > > > > > > > > > > >> >
>> reason for
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > flexibility
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > keeping in line
>> > > > > > > > with the
>> > > > > > > > > > flexibility of
>> > > > > > > > > > > key,
>> > > > > > > > > > > >> > value
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > message
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > object
>> > > > > > itself. I don’t
>> > > > > > > > think
>> > > > > > > > > it takes
>> > > > > > > > > > away
>> > > > > > > > > > > from
>> > > > > > > > > > > >> an
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > as
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > each
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > plugin
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > will
>> > > > > > care for their
>> > > > > > > own
>> > > > > > > > key,
>> > > > > > > > > this way
>> > > > > > > > > > ints,
>> > > > > > > > > > > >> >
>> booleans ,
>> > > > exotic
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > binary
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > can
>> > > > > all
>> > > > > > be catered
>> > > > > > > for=.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > wanted
>> > > > > > > > to
>> > > > > > > > > push for a
>> > > > > > > > > > typed
>> > > > > > > > > > > >> value
>> > > > > > > > > > > >> >
>> interface,
>> > > > > > > > > > > >> >
>> >> I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wouldn’t
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > want
>> > > > > > just String
>> > > > > > > values
>> > > > > > > > > supported, but
>> > > > > > > > > > the the
>> > > > > > > > > > > >> >
>> primatives
>> > > > > > > > > > > >> >
>> >>
>> > > plus
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > string and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > also
>> > > > > > still keeping the
>> > > > > > > > > ability to have a
>> > > > > > > > > > > binary
>> > > > > > > > > > > >> > for
>> custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > binaries that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > some
>> > > > > > organisations may
>> > > > > > > > have.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > i. I
>> > > > > > have written this
>> > > > > > > > slight
>> > > > > > > > > > alternative
>> > > > > > > > > > > here,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > https://cwiki.apache.org/
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > > >> >
>> > > > 82+-+Add+Record+Headers+-+
>> > > > > Typed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > ii.
>> > > > > > Essentially the
>> > > > > > > > value
>> > > > > > > > > bytes, has a
>> > > > > > > > > > leading
>> > > > > > > > > > > >> > byte
>> > overhead.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > > This tells you
>> > > > > > > what
>> > > > > > > > type
>> > > > > > > > > the value
>> > > > > > > > > > is,
>> > > > > > > > > > > >> before
>> > > > > > > > > > > >> >
>> reading
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > rest
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes,
>> > > > > > allowing
>> > > > > > > > > >
>> > > > serialisation/deserialization
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > and
>> from the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > primitives,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > string
>> > > > > > and byte[].
>> > > > > > > This
>> > > > > > > > is
>> > > > > > > > > akin to some
>> > > > > > > > > > other
>> > > > > > > > > > > >> >
>> messaging
>> > > > > > > > > > > >> >
>> >>
>> > > > systems.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2) We
>> > > > > > are making it
>> > > > > > > > optional,
>> > > > > > > > > so that
>> > > > > > > > > > for
>> > > > > > > > > > > those
>> > > > > > > > > > > >> > not
>> wanting
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > headers have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 0
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes
>> > > > > > overhead (think
>> > > > > > > > of it
>> > > > > > > > > as a feature
>> > > > > > > > > > > flag),
>> > > > > > > > > > > >> I
>> > > > > > > > > > > >> > don’t
>> > > > > > > > > > > >> >
>> >>
>> > > > think this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > complex, especially if
>> > > > > > > > > comparing to
>> > > > > > > > > > changes
>> > > > > > > > > > > >> >
>> proposed in
>> > > > > > > > > > > >> >
>> >>
>> > > > other kips
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > like
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > kip-98.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > really
>> > > > > > > > don’t
>> > > > > > > > > like
>> > > > > > > > > > this, we
>> > > > > > > > > > > can
>> > > > > > > > > > > >> > drop
>> it, but
>> > > > > > > > > > > >> >
>> >>
>> > it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > would
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > mean
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > buying
>> > > > > > into 4 bytes
>> > > > > > > > extra
>> > > > > > > > > overhead for
>> > > > > > > > > > users
>> > > > > > > > > > > who
>> > > > > > > > > > > >> > do
>> not want
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > use
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3) In
>> > > > > > the summary yes,
>> > > > > > > > it is
>> > > > > > > > > at a higher
>> > > > > > > > > > > level,
>> > > > > > > > > > > >> > but I
>> think
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is well
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > documented in the
>> > > > > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > section.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a.
>> > > > > > Added getHeaders
>> > > > > > > > method to
>> > > > > > > > > > >
>> Producer/Consumer
>> > > > > > > > > > > >> >
>> record (that
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > it)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > b.
>> > > > > > We’ve also detailed
>> > > > > > > > the
>> > > > > > > > > new Headers
>> > > > > > > > > > class
>> > > > > > > > > > > >> that
>> > > > > > > > > > > >> > this
>> method
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > returns
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > encapsulates the
>> > > > > > > headers
>> > > > > > > > > protocol and
>> > > > > > > > > > logic.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Best,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Mike
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ==Original questions
>> > > > > > > > from the
>> > > > > > > > > vote
>> > > > > > > > > > thread from
>> > > > > > > > > > > >> > Jay.==
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Couple
>> > > > > > of things I
>> > > > > > > > think we
>> > > > > > > > > still need
>> > > > > > > > > > to work
>> > > > > > > > > > > >> out:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > I
>> > > > > > think we agree
>> > > > > > > > about
>> > > > > > > > > the key,
>> > > > > > > > > > but I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > we
>> haven't
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > talked about
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > value yet. I
>> > > > > > > > think if
>> > > > > > > > > our goal
>> > > > > > > > > > is an
>> > > > > > > > > > > open
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > these
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > spread across many
>> > > > > > > > plugins
>> > > > > > > > > from many
>> > > > > > > > > > > systems
>> > > > > > > > > > > >> we
>> > > > > > > > > > > >> > should
>> > > > > > > > > > > >> >
>> >>
>> > > > consider
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > making
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > string as well so
>> > > > > > > > it can
>> > > > > > > > > be
>> > > > > > > > > > printed, set
>> > > > > > > > > > > >> via
>> > > > > > > > > > > >> > a UI,
>> set
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > config,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > etc.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > Basically
>> > > > > > > encouraging
>> > > > > > > > > pluggable
>> > > > > > > > > > >
>> serialization
>> > > > > > > > > > > >> >
>> formats
>> > > > > > > > > > > >> >
>> >>
>> > > here
>> > > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > lead
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bit
>> > > > > > of a tower of
>> > > > > > > > babel.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2.
>> > > > > > This proposal
>> > > > > > > > still
>> > > > > > > > > includes a
>> > > > > > > > > > pretty
>> > > > > > > > > > > big
>> > > > > > > > > > > >> >
>> change to
>> > our
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > protocol
>> > > > > > > > definition
>> > > > > > > > > layer.
>> > > > > > > > > > Essentially
>> > > > > > > > > > > >> it is
>> > > > > > > > > > > >> >
>> >>
>> > > > introducing an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > optional
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > type, where the
>> > > > > > > > format is
>> > > > > > > > > data
>> > > > > > > > > > dependent. I
>> > > > > > > > > > > >> > think
>> this
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > actually a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > big
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > change though it
>> > > > > > > > doesn't
>> > > > > > > > > seem like
>> > > > > > > > > > it. It
>> > > > > > > > > > > >> means
>> > > > > > > > > > > >> > you
>> can no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > longer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > specify
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > this
>> > > > > > type with our
>> > > > > > > > type
>> > > > > > > > > definition
>> > > > > > > > > > DSL, and
>> > > > > > > > > > > >> >
>> likewise it
>> > > > > > > > > > > >> >
>> >>
>> > > > requires
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > handling in client
>> > > > > > > > libs.
>> > > > > > > > > This isn't
>> > > > > > > > > > a huge
>> > > > > > > > > > > >> >
>> thing, since
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Record
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > definition is
>> > > > > > > custom
>> > > > > > > > > anyway, but I
>> > > > > > > > > > think
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > kind
>> of
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > inconsistency is
>> > > > > > > very
>> > > > > > > > > non-desirable
>> > > > > > > > > > and
>> > > > > > > > > > > ties
>> > > > > > > > > > > >> > you to
>> > > > > > > > > > > >> >
>> >>
>> > > > hand-coding
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > things.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > think the type
>> > > > > > > should
>> > > > > > > > > instead by [Key
>> > > > > > > > > > > Value]
>> > > > > > > > > > > >> in
>> > > > > > > > > > > >> > our
>> BNF,
>> > > > > > > > > > > >> >
>> >>
>> > > where
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > value are both
>> > > > > > > short
>> > > > > > > > > strings as used
>> > > > > > > > > > > >>
>> elsewhere.
>> > > > > > > > > > > >> > This
>> > > > > > > > > > > >> >
>> >>
>> > > > brings it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > in line
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > with
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > rest of the
>> > > > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3.
>> > > > > > Could we get
>> > > > > > > more
>> > > > > > > > > specific about
>> > > > > > > > > > the
>> > > > > > > > > > > exact
>> > > > > > > > > > > >> > Java
>> API
>> > > > > > > > > > > >> >
>> >>
>> > > > change to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ProducerRecord,
>> > > > > > > > > ConsumerRecord,
>> > > > > > > > > > Record,
>> > > > > > > > > > > etc?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > -Jay
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The
>> > > > > information
>> > > > > > contained in
>> > > > > > > > this
>> > > > > > > > > email is
>> > > > > > > > > > strictly
>> > > > > > > > > > > >> >
>> confidential
>> > > and
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the use of the
>> > > > > > addressee only,
>> > > > > > > > unless
>> > > > > > > > > otherwise
>> > > > > > > > > > > >>
>> indicated.
>> > > > > > > > > > > >> > If you
>> > > > > > > > > > > >> >
>> >>
>> > are
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the intended
>> > > > > > recipient, please
>> > > > > > > > do not
>> > > > > > > > > read,
>> > > > > > > > > > copy, use
>> > > > > > > > > > > or
>> > > > > > > > > > > >> >
>> disclose to
>> > > > > > > > > > > >> >
>> >>
>> > > > others
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > this message
>> > > > > or
>> > > > > > any
>> > > > > > > attachment.
>> > > > > > > > > Please also
>> > > > > > > > > > notify the
>> > > > > > > > > > > >> >
>> sender by
>> > > > > > > > > > > >> >
>> >>
>> > > > replying
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > to this email
>> > > > > > or by telephone
>> > > > > > > > > (+44(020 7896
>> > > > > > > > > > 0011) and
>> > > > > > > > > > > >> then
>> > > > > > > > > > > >> > delete
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > email
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > and any copies
>> > > > > > of it.
>> > > > > > > Opinions,
>> > > > > > > > > conclusion
>> > > > > > > > > > (etc) that
>> > > > > > > > > > > do
>> > > > > > > > > > > >> > not
>> relate
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > official
>> > > > > > business of this
>> > > > > > > > company
>> > > > > > > > > shall be
>> > > > > > > > > > understood
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> neither
>> > > > > > > > > > > >> >
>> >>
>> > > > given nor
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > endorsed by
>> > > > > it.
>> > > > > > IG is a
>> > > > > > > trading
>> > > > > > > > name
>> > > > > > > > > of IG
>> > > > > > > > > > Markets
>> > > > > > > > > > > >> Limited
>> > > > > > > > > > > >> > (a
>> company
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > registered in
>> > > > > > England and
>> > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 04008957)
>> > > > > > > > > > > >> > and IG
>> > > > > > > > > > > >> >
>> >>
>> > > Index
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Limited (a
>> > > > > > company registered
>> > > > > > > in
>> > > > > > > > > England and
>> > > > > > > > > > Wales,
>> > > > > > > > > > > >> >
>> company
>> > number
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > 01190902).
>> > > > > > Registered address
>> > > > > > > at
>> > > > > > > > > Cannon Bridge
>> > > > > > > > > > House,
>> > > > > > > > > > > 25
>> > > > > > > > > > > >> >
>> Dowgate
>> > > > > > > > > > > >> >
>> >>
>> > > Hill,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > London EC4R
>> > > > > > 2YA. Both IG
>> > > > > > > Markets
>> > > > > > > > > Limited
>> > > > > > > > > > (register
>> > > > > > > > > > > >> number
>> > > > > > > > > > > >> >
>> 195355)
>> > > > > > > > > > > >> >
>> >>
>> > and
>> > > > IG
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Index Limited
>> > > > > > (register number
>> > > > > > > > > 114059) are
>> > > > > > > > > > authorised
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> regulated
>> > > > > > > > > > > >> >
>> >>
>> > by
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Financial
>> > > > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > --
>> > > > > > > > > > > >> >
>> >>
>> > > > Nacho - Ignacio
>> > > > > > Solis -
>> > > > > > > > isolis@igso.net
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > The
>> > > information
>> > > > contained in
>> > > > > > this email is
>> > > > > > > > strictly
>> > > > > > > > > > confidential and
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> use of
>> > the
>> > > > addressee only,
>> > > > > > unless
>> > > > > > > otherwise
>> > > > > > > > > indicated. If
>> > > > > > > > > > you are
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > the
>> intended
>> > > > recipient, please
>> > > > > > do not read,
>> > > > > > > > copy, use
>> > > > > > > > > or
>> > > > > > > > > > disclose to
>> > > > > > > > > > > >> others
>> > > > > > > > > > > >> > this
>> message
>> > > or
>> > > > any
>> > > > > attachment.
>> > > > > > Please also
>> > > > > > > > notify
>> > > > > > > > > the sender by
>> > > > > > > > > > > >> replying
>> > > > > > > > > > > >> > to
>> this
>> > email
>> > > > or by telephone
>> > > > > > (+44(020 7896
>> > > > > > > > 0011) and
>> > > > > > > > > then
>> > > > > > > > > > delete the
>> > > > > > > > > > > >> email
>> > > > > > > > > > > >> > and
>> any
>> > copies
>> > > > of it.
>> > > > > Opinions,
>> > > > > > conclusion
>> > > > > > > > (etc) that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > understood
>> > > > > > > > > as
>> > > > > > > > > > neither given
>> > > > > > > > > > > >> nor
>> > > > > > > > > > > >> >
>> endorsed by
>> > > it.
>> > > > IG is a
>> > > > > trading
>> > > > > > name of IG
>> > > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >> >
>> registered
>> > in
>> > > > England and
>> > > > > > Wales, company
>> > > > > > > number
>> > > > > > > > > 04008957) and
>> > > > > > > > > > IG Index
>> > > > > > > > > > > >> >
>> Limited (a
>> > > > company registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > >> >
>> 01190902).
>> > > > Registered address
>> > > > > > at Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25
>> > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > >> >
>> London EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and
>> > > > > > > > > > > IG
>> > > > > > > > > > > >> > Index
>> > Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> Financial
>> > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> The
>> > information
>> > > > contained in
>> > > > > this
>> > > > > > email is
>> > > > > > > > strictly
>> > > > > > > > > confidential
>> > > > > > > > > > and for
>> > > > > > > > > > > >> the use
>> of the
>> > > > addressee only,
>> > > > > > unless otherwise
>> > > > > > > > > indicated. If you
>> > > > > > > > > > are
>> > > > > > > > > > > not
>> > > > > > > > > > > >> the
>> intended
>> > > > recipient, please
>> > > > > do
>> > > > > > not read,
>> > > > > > > copy,
>> > > > > > > > use
>> > > > > > > > > or disclose
>> > > > > > > > > > to
>> > > > > > > > > > > others
>> > > > > > > > > > > >> this
>> message
>> > or
>> > > > any attachment.
>> > > > > > Please also
>> > > > > > > > notify the
>> > > > > > > > > sender by
>> > > > > > > > > > > replying
>> > > > > > > > > > > >> to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the
>> > > > > > > > > > > email
>> > > > > > > > > > > >> and any
>> copies
>> > > of
>> > > > it. Opinions,
>> > > > > > conclusion (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > the
>> > > > > > > > > > > >> official
>> > > business
>> > > > of this
>> > > > > company
>> > > > > > shall be
>> > > > > > > > understood
>> > > > > > > > > as neither
>> > > > > > > > > > given
>> > > > > > > > > > > nor
>> > > > > > > > > > > >>
>> endorsed by
>> > it.
>> > > > IG is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >>
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG
>> > > > > > > > > > Index
>> > > > > > > > > > > >> Limited
>> (a
>> > > > company registered in
>> > > > > > England and
>> > > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 01190902).
>> > > > Registered address at
>> > > > > > Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25 Dowgate
>> > > > > > > > > > Hill,
>> > > > > > > > > > > >> London
>> EC4R
>> > 2YA.
>> > > > Both IG Markets
>> > > > > > Limited
>> > > > > > > (register
>> > > > > > > > > number 195355)
>> > > > > > > > > > and IG
>> > > > > > > > > > > >> Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> Financial
>> > > Conduct
>> > > > Authority.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > The
>> information
>> > > > contained in this
>> > > > > > email is strictly
>> > > > > > > > > confidential and
>> > > > > > > > > > for the use of
>> the
>> > > > addressee only, unless
>> > > > > > otherwise
>> > > > > > > > indicated.
>> > > > > > > > > If you are
>> > > > > > > > > > not the intended
>> > > > recipient, please do not
>> > > > > > read, copy, use
>> > > > > > > > or
>> > > > > > > > > disclose to
>> > > > > > > > > > others this
>> message or
>> > > any
>> > > > attachment.
>> > > > > > Please also notify
>> > > > > > > > the
>> > > > > > > > > sender by
>> > > > > > > > > > replying to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the email and any
>> > copies
>> > > > of it. Opinions,
>> > > > > > conclusion
>> > > > > > > (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to the
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > > understood as
>> > > > > > > > > > neither given nor
>> > > endorsed
>> > > > by it. IG is a
>> > > > > > trading name of
>> > > > > > > > IG
>> > > > > > > > > Markets
>> > > > > > > > > > Limited (a
>> company
>> > > > registered in England
>> > > > > > and Wales,
>> > > > > > > company
>> > > > > > > > > number
>> > > > > > > > > > 04008957) and IG
>> Index
>> > > > Limited (a company
>> > > > > > registered in
>> > > > > > > > England
>> > > > > > > > > and Wales,
>> > > > > > > > > > company number
>> > 01190902).
>> > > > Registered
>> > > > > > address at Cannon
>> > > > > > > > Bridge
>> > > > > > > > > House, 25
>> > > > > > > > > > Dowgate Hill,
>> London
>> > EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and IG
>> Index
>> > > > Limited (register
>> > > > > > number 114059) are
>> > > > > > > > > authorised and
>> > > > > > > > > > regulated by the
>> > > Financial
>> > > > Conduct
>> > > > > > Authority.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The information
>> contained in
>> > > > this email is
>> > > > > > strictly
>> > > > > > > confidential
>> > > > > > > > and
>> > > > > > > > > for the use of the
>> addressee
>> > only,
>> > > > unless otherwise
>> > > > > > indicated. If
>> > > > > > > > you are
>> > > > > > > > > not the intended recipient,
>> > please
>> > > > do not read,
>> > > > > > copy, use or
>> > > > > > > > disclose to
>> > > > > > > > > others this message or any
>> > > > attachment. Please also
>> > > > > > notify the
>> > > > > > > sender
>> > > > > > > > by
>> > > > > > > > > replying to this email or
>> by
>> > > > telephone (+44(020
>> > > > > 7896
>> > > > > > 0011) and then
>> > > > > > > > delete
>> > > > > > > > > the email and any copies
>> of it.
>> > > > Opinions,
>> > > > > conclusion
>> > > > > > (etc) that do
>> > > > > > > > not
>> > > > > > > > > relate to the official
>> business
>> > of
>> > > > this company
>> > > > > > shall be understood
>> > > > > > > > as
>> > > > > > > > > neither given nor endorsed
>> by it.
>> > > IG
>> > > > is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a company
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG Index
>> Limited (a
>> > > > company
>> > > > > registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company number 01190902).
>> > > Registered
>> > > > address at
>> > > > > > Cannon Bridge
>> > > > > > > House,
>> > > > > > > > 25
>> > > > > > > > > Dowgate Hill, London EC4R
>> 2YA.
>> > Both
>> > > > IG Markets
>> > > > > > Limited (register
>> > > > > > > > number
>> > > > > > > > > 195355) and IG Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > authorised
>> > > > > > > > and
>> > > > > > > > > regulated by the Financial
>> > Conduct
>> > > > Authority.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > The information contained in
>> this email
>> > > is
>> > > > strictly
>> > > > > > confidential and for
>> > > > > > > > the use of the addressee only,
>> unless
>> > > > otherwise
>> > > > > indicated.
>> > > > > > If you are not
>> > > > > > > > the intended recipient, please
>> do not
>> > > > read, copy, use or
>> > > > > > disclose to
>> > > > > > > others
>> > > > > > > > this message or any attachment.
>> Please
>> > > > also notify the
>> > > > > > sender by replying
>> > > > > > > > to this email or by telephone
>> (+44(020
>> > > > 7896 0011) and
>> > > > > then
>> > > > > > delete the
>> > > > > > > email
>> > > > > > > > and any copies of it. Opinions,
>> > > conclusion
>> > > > (etc) that do
>> > > > > > not relate to
>> > > > > > > the
>> > > > > > > > official business of this
>> company shall
>> > > be
>> > > > understood as
>> > > > > > neither given
>> > > > > > > nor
>> > > > > > > > endorsed by it. IG is a trading
>> name of
>> > > IG
>> > > > Markets
>> > > > > Limited
>> > > > > > (a company
>> > > > > > > > registered in England and Wales,
>> > company
>> > > > number 04008957)
>> > > > > > and IG Index
>> > > > > > > > Limited (a company registered in
>> > England
>> > > > and Wales,
>> > > > > > company number
>> > > > > > > > 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill,
>> > > > > > > > London EC4R 2YA. Both IG Markets
>> > Limited
>> > > > (register number
>> > > > > > 195355) and IG
>> > > > > > > > Index Limited (register number
>> 114059)
>> > > are
>> > > > authorised and
>> > > > > > regulated by
>> > > > > > > the
>> > > > > > > > Financial Conduct Authority.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > The information contained in this email is
>> strictly
>> > > > confidential and
>> > > > > > for the use of the addressee only, unless
>> otherwise
>> > > > indicated. If you are
>> > > > > > not the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or disclose to
>> > > > > > others this message or any attachment. Please
>> also
>> > notify
>> > > > the sender by
>> > > > > > replying to this email or by telephone (+44(020
>> 7896
>> > > 0011)
>> > > > and then
>> > > > > delete
>> > > > > > the email and any copies of it. Opinions,
>> conclusion
>> > > (etc)
>> > > > that do not
>> > > > > > relate to the official business of this company
>> shall
>> > be
>> > > > understood as
>> > > > > > neither given nor endorsed by it. IG is a
>> trading name
>> > of
>> > > > IG Markets
>> > > > > > Limited (a company registered in England and
>> Wales,
>> > > > company number
>> > > > > > 04008957) and IG Index Limited (a company
>> registered in
>> > > > England and
>> > > > > Wales,
>> > > > > > company number 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register number
>> > > > > > 195355) and IG Index Limited (register number
>> 114059)
>> > are
>> > > > authorised and
>> > > > > > regulated by the Financial Conduct Authority.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential
>> and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If
>> you are
>> > not
>> > > > the intended recipient, please do not read, copy, use or
>> disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then
>> delete the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company
>> number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number
>> 195355) and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and
>> regulated by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > >
>> >
>>
>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
--
Nacho - Ignacio Solis - isolis@igso.net
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Jason
I looked at the doc, it seems the length of the following data sections are being left described in your doc in some place similar to how we have on the wiki, and in others not.
E.g. your doc has this as the proposed message described (which is in blue and red colouring)
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
Value => data [OPTIONAL]
Note I notice there is a missing
ValueLen => uintVar [OPTIONAL]
Else we wouldn’t be able to put headers at the end, without its length being defined. So either we can leave ValueLen removed, as is in your new format, and keep headers
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
HeadersLen => uintVar [OPTIONAL]
Headers => data [OPTIONAL]
Value => data [OPTIONAL]
Headers => [Header]
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
OR we will need to re-introduce ValueLen which seems to be removed in Kip-98
Message =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
KeyLen => uintVar [OPTIONAL]
Key => data [OPTIONAL]
ValueLen => uintVar [OPTIONAL]
Value => data [OPTIONAL]
Headers => Header1, Header2 … HeaderN
Header =>
KeyLen => uintVar
Key => data (utf8)
ValueLen => uintVar
Value => data
Cheers
Mike
On 21/02/2017, 20:45, "ignacio.solis@gmail.com on behalf of Ignacio Solis" <ignacio.solis@gmail.com on behalf of isolis@igso.net> wrote:
Short answer:
Headers at the end are fine, this proposal looks ok.
Longer answer:
At LinkedIn we considered headers at the end multiple times. We even
had a backwards compatible headers-in-V design with headers at the
end.
The main reason headers come at the beginning of messaging systems is
in case they affect the rest of the message. This is either because
of parsing differences (a header might indicate how to parse the
remainder of the message), or performance optimizations (a header
might indicate that you can discard the message and can stop parsing).
I believe that neither of these cases are common in Kafka (at this
time) so having the headers are the end shouldn't be a big issue.
From a performance perspective we may incur an extra page cache miss
but at the rate at which kafka does messaging it won't bottleneck
things.
Headers at the end have one potential advantage, that in simple
circumstances, naive systems can append headers to a message without
doing a lot of memory manipulation. Normally not that big of a deal
since we could always use scatter/gather methods if needed, but maybe
some simple systems will benefit from that. ... Not to mention most
Java programmers might not pay attention to these type of performance
issues.
Anyway...
Overall, there are many things that I would have done differently in
the proposal (int keys, total header length, no VarInts, etc. ), but
the cost of these is small in comparison to the benefit we get from
having headers at all.
If we are all ok with this design then I would be the first to +1 the format.
Nacho
Side note: Shouldn't headers at the end be called footers? We could
always call it metadata :-).
On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com> wrote:
> I think I'd like some opinions from some of the active chaps at LinkedIn so it's not just a two way decision (or any others for that matter).
>
> Any opinions on the below or no real concerns / indifferent by anyone with this change?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Tuesday, February 21, 2017 6:59:38 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Jun's suggestion gives us the best of both worlds. Because the headers are
> the last field in the record and you have the total size of the record,
> then you can easily get the size of headers in bytes. So they can be copied
> without parsing and we can do the lazy initialization without an extra cost.
>
> The message set notion has changed in KIP-98 so that the message set can
> have its own fields. See here for more detail:
> https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
> What we're proposing to change here is the schema for the message (aka
> record) itself. We'll update that document depending on the outcome of the
> discussion here.
>
> -Jason
>
>
> On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> This works whilst no other feature needs similar. I prefer to keep it
>> contained/encapsulated it then allows the bytes of the headers to also be
>> copied transferred as one without any parsing what so ever, but if you
>> really feel strongly on the below ill go with it, just want to flag this
>> concern.
>>
>> Btw, the below notation seems to be different to the notation used in
>> previous kips like kip-32 which I took the notation from, where the length
>> of the following bytes for byte arrays is clearly defined. So im 100% on
>> what you propose could you define it in this original styled notation?
>>
>>
>> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>>
>> Hey Jun, that's a good thought. Maybe then the full schema could look
>> like
>> this:
>>
>> Record =>
>> Length => uintVar
>> Attributes => int8
>> TimestampDelta => intVar
>> OffsetDelta => uintVar
>> Key => bytes
>> Value => bytes
>> Headers => [HeaderKey HeaderValue]
>> HeaderKey => string
>> HeaderValue => bytes
>>
>> (where the bytes and string type use a variable length size field)
>>
>> Does that work for you, Michael?
>>
>> -Jason
>>
>> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > Regarding the overhead of skipping the headers in the message.
>> Another way
>> > could be to add the headers at the end of the message after the
>> value part.
>> > This way, there is no need to skip the headers unless you really
>> need it.
>> > This will allow us to represent the headers explicitly as an array of
>> > key/value pairs without the skipping overhead when headers are not
>> needed.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
>> >
>> > wrote:
>> >
>> > > Michael,
>> > >
>> > > Here's a link to the branch with the message format changes:
>> > > https://github.com/confluentinc/kafka/tree/
>> exactly-once-message-format.
>> > > The
>> > > class to modify to add headers is EosLogRecord (name will
>> definitely
>> > change
>> > > prior to merging in trunk). This branch uses attributes to
>> indicate null
>> > > key and value. I really don't have a strong preference for or
>> against
>> > this
>> > > as long as we're consistent. I do feel it's kind of a pity to need
>> the
>> > > signed version of the varint just to be able to indicate null. For
>> > headers,
>> > > it's an easier call because we don't actually need to represent
>> null
>> > > headers (using size zero will work fine).
>> > >
>> > > -Jason
>> > >
>> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > We can make it, implement Iterable<Header>, no concerns on this.
>> > > >
>> > > > Ive noted this on the KIP.
>> > > >
>> > > >
>> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > > Btw Jason, can you share the branch with the VarInts
>> helper/util
>> > > class
>> > > > that KIP-98 will be introducing and also will have the current
>> modified
>> > > > (key and value lengths) message, so we can look at, thanks.
>> > > >
>> > > > On 20/02/2017, 01:11, "Michael Pearce" <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > > >
>> > > > RE “ProducerRecord and ConsumerRecord are immutable
>> > currently”. I
>> > > > beg to differ on this, if I set a byte[] as my value of my
>> record, I
>> > can
>> > > > later call value() and then alter the bytes in the array. As
>> such it is
>> > > > very much mutable.
>> > > >
>> > > > Headers also will anyhow by nature be mutable on
>> produce, as
>> > you
>> > > > add to them.
>> > > >
>> > > >
>> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
>> jason@confluent.io>
>> > > > wrote:
>> > > >
>> > > > Re; having a dedicated Headers object. I think it
>> makes
>> > > sense.
>> > > > Maybe it
>> > > > could implement Iterable<Header>?
>> > > >
>> > > > One thing I'm not too sure about is mutability. Both
>> the
>> > > > ProducerRecord and
>> > > > ConsumerRecord are immutable currently. It would be
>> nice to
>> > > > keep it that
>> > > > way. Perhaps Headers could itself be immutable and
>> passed
>> > > > through the
>> > > > constructors?
>> > > >
>> > > > Just going through how this is, I don’t believe
>> this would
>> > > > provide much
>> > > > > saving. If issue trying to resolve is around
>> > object/garbage
>> > > > creation of
>> > > > > Map.Entry objects.
>> > > >
>> > > >
>> > > > It's definitely better because we only allocate the
>> space
>> > we
>> > > > will use, but
>> > > > I assume most of the actual memory cost is in the
>> header
>> > data
>> > > > itself which
>> > > > we can't do much about. Maybe we can start with the
>> simple
>> > > > array
>> > > > representation and optimize from there after we find
>> it's
>> > > > worthwhile to do
>> > > > so? If we use the Headers object, we can change this
>> later
>> > > > without breaking
>> > > > the API.
>> > > >
>> > > > -Jason
>> > > >
>> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
>> > > > radai.rosenblatt@gmail.com> wrote:
>> > > >
>> > > > > "(By the way, doesn't it feel a bit odd that we
>> seem to
>> > be
>> > > > designing a
>> > > > > feature which is optimized for people not using?)"
>> > > > >
>> > > > > very. this (i claim :-) ) is the result of very
>> intense
>> > > > opposition to the
>> > > > > usefulness of the feature early on, and not a real
>> design
>> > > > goal
>> > > > >
>> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Whilst if having an array of a Header[] would
>> mean
>> > > getting
>> > > > this through,
>> > > > > > im happy changing this, as already done.
>> > > > > >
>> > > > > > Just going through how this is, I don’t believe
>> this
>> > > > would provide much
>> > > > > > saving. If issue trying to resolve is around
>> > > > object/garbage creation of
>> > > > > > Map.Entry objects.
>> > > > > >
>> > > > > > All you’re doing here is replacing the equiv of a
>> > > > HashMap.Node
>> > > > > (Map.Entry)
>> > > > > > which simply holds reference to key and value
>> objects
>> > > with
>> > > > a custom
>> > > > > variant.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com> wrote:
>> > > > > >
>> > > > > > On point 1 & 2 Ive updated KIP to show
>> varints (and
>> > > > removed the bit
>> > > > > > flag). (on the assumption KIP 98 is getting the
>> > agreement
>> > > > the protocol is
>> > > > > > moving from int32 to varInts as standard)
>> > > > > >
>> > > > > > On point 3 ive updated to use an array of
>> Header
>> > > > class, instead of a
>> > > > > > MultiMap in the Headers class object
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > On points 1 and 2 I agree.
>> > > > > >
>> > > > > > This also affects kip-98, I should
>> expect this
>> > > > resolved before
>> > > > > > that vote also passes. If it is accepted there
>> (I’m
>> > > > assuming this is
>> > > > > > getting discussed on that KIP? As you’re driving
>> the
>> > move
>> > > > to VarInts), I
>> > > > > am
>> > > > > > happy to make this KIP will simply follow suit to
>> > > whatever
>> > > > is agreed in
>> > > > > > KIP-98.
>> > > > > >
>> > > > > > On 3) Agreed this is a simpler form, as
>> long as
>> > > no
>> > > > one is
>> > > > > > expecting hashmap lookup performance ( O(1) ), I
>> still
>> > > > would prefer an
>> > > > > > encapsulated class, so if we find that holding
>> it as an
>> > > > array in future
>> > > > > is
>> > > > > > causing some perf issues, the internals are not
>> exposed
>> > > to
>> > > > end users,
>> > > > > > allowing the internal structure to move to a map.
>> > > > > >
>> > > > > > Can we compromise on?
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Headers headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Headers {
>> > > > > > Header[] headers;
>> > > > > >
>> > > > > > add(String key, byte[] value)
>> > > > > > Collection<byte[]> get(String key)
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
>> > > > jason@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > >
>> > > > > >
>> > > > > > This is exactly what I have been
>> trying to
>> > > get
>> > > > at. The use
>> > > > > > cases documented
>> > > > > > here are middleware:
>> > > > > > https://cwiki.apache.org/
>> > > > confluence/display/KAFKA/A+
>> > > > > > Case+for+Kafka+Headers.
>> > > > > > If headers are intended for the
>> application
>> > > > space as well,
>> > > > > the
>> > > > > > document
>> > > > > > should be updated accordingly so
>> that it is
>> > > an
>> > > > explicit
>> > > > > design
>> > > > > > goal and not
>> > > > > > an unfortunate design byproduct.
>> There may
>> > be
>> > > > some
>> > > > > > disagreement on whether
>> > > > > > it _should_ be a design goal, but it
>> may
>> > not
>> > > > matter much
>> > > > > since
>> > > > > > the current
>> > > > > > interceptor API is probably
>> insufficient
>> > for
>> > > > middleware
>> > > > > > applications (I
>> > > > > > haven't thought of a way to do this
>> that
>> > > isn't
>> > > > cumbersome).
>> > > > > >
>> > > > > > That aside, the three unresolved
>> points for
>> > > me
>> > > > are the
>> > > > > > following:
>> > > > > >
>> > > > > > 1. The use of varints. As a one-off
>> for use
>> > > > only with record
>> > > > > > headers, the
>> > > > > > case was weak, but if we are using
>> them
>> > > > throughout the
>> > > > > message
>> > > > > > format, then
>> > > > > > we should do so here as well. The
>> > additional
>> > > > complexity is
>> > > > > > minimal and
>> > > > > > early performance testing fully
>> justifies
>> > > > their use.
>> > > > > >
>> > > > > > 2. If varints are used, the case for
>> using
>> > > > attributes to
>> > > > > > indicate null
>> > > > > > headers also becomes weak. We only
>> add one
>> > > > additional byte in
>> > > > > > each message
>> > > > > > if there are no headers. Whatever
>> the case,
>> > > > let us be
>> > > > > > consistent with how
>> > > > > > we treat null keys and values.
>> > > > > >
>> > > > > > 3. We have apparently agreed that the
>> > broker
>> > > > will validate
>> > > > > > headers (please
>> > > > > > add this to the KIP). That being the
>> case,
>> > I
>> > > > would prefer to
>> > > > > > use Kafka's
>> > > > > > conventional format for arrays. The
>> > downside
>> > > > is that it takes
>> > > > > > more work to
>> > > > > > skip over the headers on the
>> consumer,
>> > though
>> > > > it's unclear if
>> > > > > > the cost
>> > > > > > would matter in practice. Some
>> concern was
>> > > > previously
>> > > > > > expressed about the
>> > > > > > allocation of maps. An alternative
>> would be
>> > > to
>> > > > use arrays,
>> > > > > i.e.
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Header[] headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > > This would work nicely with the
>> > conventional
>> > > > array format and
>> > > > > > my guess is
>> > > > > > it would obviate the need do any lazy
>> > > > initialization. If we
>> > > > > > use the map as
>> > > > > > is currently documented, then it is
>> > possible
>> > > > with either
>> > > > > > representation to
>> > > > > > slice the headers and initialize them
>> > lazily.
>> > > > Either way, it
>> > > > > > might be a
>> > > > > > good idea to use a separate object to
>> > > > represent the headers
>> > > > > in
>> > > > > > case we need
>> > > > > > to modify it in the future in some
>> way.
>> > > > > >
>> > > > > > (By the way, doesn't it feel a bit
>> odd that
>> > > we
>> > > > seem to be
>> > > > > > designing a
>> > > > > > feature which is optimized for
>> people not
>> > > > using?)
>> > > > > >
>> > > > > >
>> > > > > > If we can resolve these points, then
>> at
>> > least
>> > > > you will get my
>> > > > > > vote.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jason
>> > > > > >
>> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
>> radai <
>> > > > > > radai.rosenblatt@gmail.com> wrote:
>> > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > > > IIUC samze currently has headers
>> "in-V"
>> > and
>> > > > would just
>> > > > > > switch over to kafka
>> > > > > > > headers if they exist.
>> > > > > > > im sure plenty of other users of
>> kafka
>> > > would
>> > > > have a use for
>> > > > > > headers.
>> > > > > > > im pretty sure use cases exist
>> around
>> > > > shuffling data
>> > > > > > into/out-of kafka
>> > > > > > > (kafka connect or equivalent) where
>> > > metadata
>> > > > from one end
>> > > > > > could copied over
>> > > > > > > to the other (S3, for example uses
>> http
>> > > > headers for
>> > > > > > user-accessible
>> > > > > > > metadata). it will be kafka client
>> code
>> > > > getting/setting
>> > > > > > those headers. not
>> > > > > > > an interceptor.
>> > > > > > >
>> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
>> Michael
>> > > > Pearce <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > For APM single event tracing,
>> need
>> > access
>> > > > to the header
>> > > > > at
>> > > > > > the point of
>> > > > > > > > processing on the processing
>> thread.
>> > > > > > > >
>> > > > > > > > As such interceptors will not
>> work/be
>> > > > suitable for these,
>> > > > > > due to the fact
>> > > > > > > > they act on the ConsumerRecords
>> as a
>> > > > batch, before the
>> > > > > > handling thread
>> > > > > > > can
>> > > > > > > > split out and process per
>> message which
>> > > is
>> > > > the point
>> > > > > these
>> > > > > > tools will
>> > > > > > > need
>> > > > > > > > to continue to transaction
>> tracing.
>> > > > > > > >
>> > > > > > > > Like wise tools and infra pieces
>> will
>> > > need
>> > > > access to the
>> > > > > > message outside
>> > > > > > > > the interceptor realm.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 17/02/2017, 21:26, "Jason
>> > Gustafson" <
>> > > > > > jason@confluent.io> wrote:
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > So the broker will parse the
>> > headers
>> > > > and validate
>> > > > > > them. Good. The
>> > > > > > > only
>> > > > > > > > reason remaining that I can
>> see to
>> > > > leave the headers
>> > > > > > as a byte array
>> > > > > > > > is to
>> > > > > > > > make it easier for the
>> client to
>> > skip
>> > > > past them. Are
>> > > > > > we sure this is
>> > > > > > > > not
>> > > > > > > > premature optimization? Are
>> there
>> > any
>> > > > performance
>> > > > > > results which show
>> > > > > > > > that
>> > > > > > > > this is worthwhile?
>> > > > > > > >
>> > > > > > > > What’s the issue with
>> exposing a
>> > > > method getHeaders on
>> > > > > > the
>> > > > > > > > producer/consumer
>> > > > > > > > > record? It doesn’t break
>> > anything.
>> > > > We don’t need
>> > > > > any
>> > > > > > special
>> > > > > > > version.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > See my previous explanation.
>> What I
>> > > am
>> > > > trying to
>> > > > > > resist is the
>> > > > > > > headers
>> > > > > > > > becoming a general
>> > application-level
>> > > > facility. The
>> > > > > > primary use case
>> > > > > > > as
>> > > > > > > > far
>> > > > > > > > as I can tell is middleware,
>> which
>> > is
>> > > > the use case
>> > > > > > that the
>> > > > > > > > interceptors
>> > > > > > > > are providing.
>> > > > > > > >
>> > > > > > > > Current batch consumer model
>> and
>> > > > consumer
>> > > > > interceptors
>> > > > > > don’t work
>> > > > > > > where
>> > > > > > > > > headers need to be acted
>> on at
>> > per
>> > > > message level at
>> > > > > > time of
>> > > > > > > > processing,
>> > > > > > > > > very case is APM (the core
>> one),
>> > > > where the header
>> > > > > > value is used to
>> > > > > > > > continue
>> > > > > > > > > tracing.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I still don't understand the
>> point
>> > > > about batching.
>> > > > > The
>> > > > > > consumer
>> > > > > > > > records are
>> > > > > > > > exposed as a batch in the
>> consumer
>> > > > interceptor, but
>> > > > > > you can still
>> > > > > > > > iterate
>> > > > > > > > them individually. It is no
>> > different
>> > > > for the
>> > > > > consumer
>> > > > > > API itself.
>> > > > > > > >
>> > > > > > > > -Jason
>> > > > > > > >
>> > > > > > > > On Fri, Feb 17, 2017 at
>> 12:48 PM,
>> > > > Michael Pearce <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Re:
>> > > > > > > > >
>> > > > > > > > > “ The point about
>> creation of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a slice
>> of
>> > bytes
>> > > > until the time
>> > > > > > it is accessed.”
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > >
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:37,
>> "Michael
>> > > > Pearce" <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > What’s the issue with
>> > exposing
>> > > a
>> > > > method
>> > > > > > getHeaders on the
>> > > > > > > > > producer/consumer record?
>> It
>> > > doesn’t
>> > > > break
>> > > > > anything.
>> > > > > > We don’t need
>> > > > > > > > any
>> > > > > > > > > special version.
>> > > > > > > > >
>> > > > > > > > > Current batch consumer
>> model
>> > > and
>> > > > consumer
>> > > > > > interceptors don’t
>> > > > > > > work
>> > > > > > > > > where headers need to be
>> acted on
>> > > at
>> > > > per message
>> > > > > > level at time of
>> > > > > > > > > processing, very case is
>> APM (the
>> > > > core one), where
>> > > > > > the header value
>> > > > > > > > is used
>> > > > > > > > > to continue tracing.
>> JMS/HTTP etc
>> > > > all expose these,
>> > > > > > without issues.
>> > > > > > > > I would
>> > > > > > > > > NOT want to lock this down
>> to
>> > only
>> > > > be usable
>> > > > > > accessible via
>> > > > > > > > interceptors,
>> > > > > > > > > as we’d fail on one of the
>> main
>> > > > goals.
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Mike
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:21,
>> "Jason
>> > > > Gustafson" <
>> > > > > > jason@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > The point about
>> creation
>> > of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a
>> slice of
>> > > > bytes until the
>> > > > > > time it is
>> > > > > > > > accessed.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > Yes exactly we
>> have
>> > > access
>> > > > to the records
>> > > > > > thus why the
>> > > > > > > > header
>> > > > > > > > > should be
>> > > > > > > > > > accessible via
>> it and
>> > not
>> > > > hidden for only
>> > > > > > interceptors to
>> > > > > > > > access.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > As explained
>> above, the
>> > > > point is to make
>> > > > > the
>> > > > > > intended usage
>> > > > > > > > clear.
>> > > > > > > > > Applications should
>> > > continue
>> > > > to rely on the
>> > > > > > key/value
>> > > > > > > fields
>> > > > > > > > to
>> > > > > > > > > serialize
>> > > > > > > > > their own headers,
>> and it
>> > > > would be more
>> > > > > > ideal if we can
>> > > > > > > avoid
>> > > > > > > > > leaking
>> > > > > > > > > third-party
>> headers into
>> > > > applications. This
>> > > > > > is difficult to
>> > > > > > > > do
>> > > > > > > > > with the
>> > > > > > > > > current
>> interceptors
>> > > because
>> > > > they share the
>> > > > > > record objects
>> > > > > > > > with
>> > > > > > > > > the common
>> > > > > > > > > API. What I had in
>> mind
>> > is
>> > > > something like
>> > > > > an
>> > > > > > extension of
>> > > > > > > the
>> > > > > > > > > current
>> > > > > > > > > interceptors which
>> > exposed
>> > > a
>> > > > different
>> > > > > > object (e.g.
>> > > > > > > > > `RecordAndHeaders`).
>> > > > > > > > > The challenge is
>> for
>> > > MM-like
>> > > > use cases. Let
>> > > > > > me see if I can
>> > > > > > > > come
>> > > > > > > > > up with a
>> > > > > > > > > concrete proposal
>> for
>> > that
>> > > > problem.
>> > > > > > > > >
>> > > > > > > > > -Jason
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 17,
>> 2017 at
>> > > > 11:55 AM, Michael
>> > > > > > Pearce <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I am happy to
>> move the
>> > > > definition of the
>> > > > > > header into the
>> > > > > > > > message
>> > > > > > > > > body, but
>> > > > > > > > > > would cause us
>> not to
>> > > lazy
>> > > > > > initialise/parse the headers,
>> > > > > > > as
>> > > > > > > > > obviously, we
>> > > > > > > > > > would have to
>> traverse
>> > > > these reading the
>> > > > > > message.
>> > > > > > > > > >
>> > > > > > > > > > This was
>> actually one
>> > of
>> > > > Jay’s requests:
>> > > > > > > > > >
>> > > > > > > > > > “ 2. I think
>> we
>> > should
>> > > > think about
>> > > > > > creating the Map
>> > > > > > > > lazily to
>> > > > > > > > > avoid
>> > > > > > > > > > parsing out
>> all the
>> > > > headers into
>> > > > > > little objects.
>> > > > > > > > HashMaps
>> > > > > > > > > themselves
>> > > > > > > > > > are
>> > > > > > > > > > kind of
>> expensive
>> > and
>> > > > the consumer is
>> > > > > > very perf
>> > > > > > > > sensitive so
>> > > > > > > > > and making
>> > > > > > > > > > gazillions of
>> > > hashmaps
>> > > > that may or
>> > > > > may
>> > > > > > not get used
>> > > > > > > is
>> > > > > > > > > probably a bad
>> > > > > > > > > > idea.”
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 17/02/2017,
>> 19:44,
>> > > > "Michael Pearce" <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Yes exactly
>> we have
>> > > > access to the
>> > > > > > records thus why
>> > > > > > > the
>> > > > > > > > > header should
>> > > > > > > > > > be accessible
>> via it
>> > and
>> > > > not hidden for
>> > > > > > only interceptors
>> > > > > > > > to
>> > > > > > > > > access.
>> > > > > > > > > >
>> > > > > > > > > > Sent using
>> OWA for
>> > > > iPhone
>> > > > > > > > > >
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > From: Magnus
>> > > Edenhill <
>> > > > > > magnus@edenhill.se>
>> > > > > > > > > > Sent: Friday,
>> > > February
>> > > > 17, 2017
>> > > > > > 7:34:49 PM
>> > > > > > > > > > To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > Subject: Re:
>> > > [DISCUSS]
>> > > > KIP-82 - Add
>> > > > > > Record Headers
>> > > > > > > > > >
>> > > > > > > > > > Big +1 on
>> VarInts.
>> > > > > > > > > > CPUs are
>> fast,
>> > memory
>> > > > is slow.
>> > > > > > > > > >
>> > > > > > > > > > I agree with
>> Jason
>> > > > that we'll want to
>> > > > > > continue
>> > > > > > > > verifying
>> > > > > > > > > messages,
>> > > > > > > > > > including
>> their
>> > > > headers, so while I
>> > > > > > appreciate the
>> > > > > > > > idea of
>> > > > > > > > > the opaque
>> > > > > > > > > > header blob
>> it
>> > won't
>> > > > be useful in
>> > > > > > practice.
>> > > > > > > > > >
>> > > > > > > > > > /Magnus
>> > > > > > > > > >
>> > > > > > > > > > 2017-02-17
>> 10:41
>> > > > GMT-08:00 Jason
>> > > > > > Gustafson <
>> > > > > > > > > jason@confluent.io>:
>> > > > > > > > > >
>> > > > > > > > > > > Sorry, my
>> > mistake.
>> > > > The consumer
>> > > > > > interceptor is per
>> > > > > > > > batch,
>> > > > > > > > > though I'm
>> > > > > > > > > > not
>> > > > > > > > > > > sure
>> that's an
>> > > > actual limitation
>> > > > > > since you still
>> > > > > > > have
>> > > > > > > > > access to the
>> > > > > > > > > > > individual
>> > records.
>> > > > > > > > > > >
>> > > > > > > > > > > -Jason
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri,
>> Feb 17,
>> > > 2017
>> > > > at 10:39 AM,
>> > > > > > Jason Gustafson <
>> > > > > > > > > >
>> jason@confluent.io>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Re
>> headers as
>> > > byte
>> > > > array and
>> > > > > > future use by
>> > > > > > > broker.
>> > > > > > > > This
>> > > > > > > > > doesn't
>> > > > > > > > > > take away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yeah, I
>> didn't
>> > > say
>> > > > it was
>> > > > > > difficult, only
>> > > > > > > awkward.
>> > > > > > > > You
>> > > > > > > > > wouldn't
>> > > > > > > > > > write the
>> > > > > > > > > > > > schema
>> that way
>> > > if
>> > > > you were
>> > > > > > planning to use it on
>> > > > > > > > the
>> > > > > > > > > brokers from
>> > > > > > > > > > the
>> > > > > > > > > > > >
>> beginning. Note
>> > > > also that one of
>> > > > > > the benefits of
>> > > > > > > > letting
>> > > > > > > > > the broker
>> > > > > > > > > > > >
>> understand
>> > > headers
>> > > > is that it can
>> > > > > > validate that
>> > > > > > > > they are
>> > > > > > > > > properly
>> > > > > > > > > > > >
>> formatted. If
>> > > cost
>> > > > is the only
>> > > > > > concern, we should
>> > > > > > > > > confirm its
>> > > > > > > > > > impact
>> > > > > > > > > > > > through
>> > > > performance testing.
>> > > > > > > > > > > >
>> > > > > > > > > > > > One of
>> the key
>> > > use
>> > > > cases requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'm not
>> sure I
>> > > > understand the
>> > > > > > point about
>> > > > > > > batching.
>> > > > > > > > > Interceptors
>> > > > > > > > > > are
>> > > > > > > > > > > > applied
>> > > > per-message, right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > My
>> intent on
>> > > > interceptors is to
>> > > > > > keep the usage of
>> > > > > > > > headers
>> > > > > > > > > > well-defined so
>> > > > > > > > > > > > that
>> they don't
>> > > > start leaking
>> > > > > > unnecessarily into
>> > > > > > > > > applications. My
>> > > > > > > > > > guess
>> > > > > > > > > > > is
>> > > > > > > > > > > > that it's
>> > > probably
>> > > > inevitable,
>> > > > > but
>> > > > > > isolating it
>> > > > > > > in
>> > > > > > > > the
>> > > > > > > > > > interceptors
>> would
>> > > > > > > > > > > > at least
>> give
>> > > > people a second
>> > > > > > thought before
>> > > > > > > > deciding to
>> > > > > > > > > use it.
>> > > > > > > > > > The main
>> > > > > > > > > > > >
>> challenge in my
>> > > > mind is figuring
>> > > > > > out how an MM
>> > > > > > > use
>> > > > > > > > case
>> > > > > > > > > would
>> > > > > > > > > > work. It
>> > > > > > > > > > > > would be
>> more
>> > > > cumbersome to
>> > > > > > replicate headers
>> > > > > > > > through an
>> > > > > > > > > > interceptor,
>> > > > > > > > > > > > though
>> arguably
>> > > MM
>> > > > should be
>> > > > > > working at a lower
>> > > > > > > > level
>> > > > > > > > > anyway.
>> > > > > > > > > > > >
>> > > > > > > > > > > > -Jason
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri,
>> Feb 17,
>> > > > 2017 at 10:16 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Re
>> headers
>> > > > available on the
>> > > > > > record va
>> > > > > > > > interceptors only
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> One of
>> the key
>> > > > use cases
>> > > > > requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> This
>> anyhow is
>> > > > similar to
>> > > > > > jms/http/amqp where
>> > > > > > > > headers
>> > > > > > > > > are
>> > > > > > > > > > available to
>> > > > > > > > > > > >>
>> consuming
>> > > > applications.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Re
>> headers as
>> > > > byte array and
>> > > > > > future use by
>> > > > > > > > broker. This
>> > > > > > > > > doesn't
>> > > > > > > > > > take
>> > > > > > > > > > > away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Sent
>> using OWA
>> > > > for iPhone
>> > > > > > > > > > > >>
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > > >> From:
>> Jason
>> > > > Gustafson <
>> > > > > > jason@confluent.io>
>> > > > > > > > > > > >> Sent:
>> Friday,
>> > > > February 17, 2017
>> > > > > > 5:55:42 PM
>> > > > > > > > > > > >> To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > > >>
>> Subject: Re:
>> > > > [DISCUSS] KIP-82 -
>> > > > > > Add Record
>> > > > > > > Headers
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Yes,
>> that is
>> > > what
>> > > > we are
>> > > > > > proposing, so using
>> > > > > > > > varints
>> > > > > > > > > for headers
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >>
>> consistent
>> > with
>> > > > the rest of the
>> > > > > > message. We have
>> > > > > > > > used
>> > > > > > > > > static
>> > > > > > > > > > helper
>> > > > > > > > > > > >> methods
>> > > > > > > > > > > >> in our
>> > prototype
>> > > > implementation.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> The
>> cost of
>> > > > parsing, we want to
>> > > > > > parse/interpret
>> > > > > > > > the
>> > > > > > > > > headers
>> > > > > > > > > > lazily (this
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > a key
>> point
>> > > > brought up earlier
>> > > > > > in discussions)
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I'm a
>> bit
>> > > > skeptical of this. Has
>> > > > > > anyone done the
>> > > > > > > > > performance
>> > > > > > > > > > testing? I
>> > > > > > > > > > > >> can
>> > > > > > > > > > > >> probably
>> > > > implement it and test
>> > > > > it
>> > > > > > if no one else
>> > > > > > > > has. I
>> > > > > > > > > was also
>> > > > > > > > > > under
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> impression
>> > that
>> > > > there may be use
>> > > > > > cases down the
>> > > > > > > > road
>> > > > > > > > > where the
>> > > > > > > > > > broker
>> > > > > > > > > > > >> would
>> > > > > > > > > > > >> need to
>> > > interpret
>> > > > headers. That
>> > > > > > wouldn't be off
>> > > > > > > > the
>> > > > > > > > > table in the
>> > > > > > > > > > future
>> > > > > > > > > > > if
>> > > > > > > > > > > >> it's
>> > represented
>> > > > as bytes, but
>> > > > > it
>> > > > > > would be quite
>> > > > > > > > a bit
>> > > > > > > > > more
>> > > > > > > > > > awkward,
>> > > > > > > > > > > >> right?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> By the
>> way,
>> > one
>> > > > question I have
>> > > > > > been wondering
>> > > > > > > > about. My
>> > > > > > > > > > understanding
>> > > > > > > > > > > is
>> > > > > > > > > > > >> that
>> headers
>> > are
>> > > > primarily for
>> > > > > > use cases where a
>> > > > > > > > > third-party
>> > > > > > > > > > components
>> > > > > > > > > > > >> wants to
>> > enrich
>> > > > messages without
>> > > > > > needing to
>> > > > > > > > understand
>> > > > > > > > > or modify
>> > > > > > > > > > the
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> of the
>> message
>> > > > key and value.
>> > > > > For
>> > > > > > the
>> > > > > > > applications
>> > > > > > > > > which directly
>> > > > > > > > > > > produce
>> > > > > > > > > > > >> and
>> consume
>> > the
>> > > > messages and
>> > > > > > control the
>> > > > > > > key/value
>> > > > > > > > > schema
>> > > > > > > > > > directly, it
>> > > > > > > > > > > >> seems
>> we would
>> > > > rather have them
>> > > > > > implement
>> > > > > > > headers
>> > > > > > > > > directly in
>> > > > > > > > > > their own
>> > > > > > > > > > > >> schema.
>> > > Supposing
>> > > > for the sake
>> > > > > of
>> > > > > > argument that
>> > > > > > > > it was
>> > > > > > > > > possible,
>> > > > > > > > > > my
>> > > > > > > > > > > >>
>> question is
>> > > > whether it be
>> > > > > > sufficient to expose
>> > > > > > > the
>> > > > > > > > > headers in the
>> > > > > > > > > > > >>
>> interceptor
>> > API
>> > > > and not in the
>> > > > > > common API?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> -Jason
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Fri,
>> Feb
>> > 17,
>> > > > 2017 at 3:26 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > On
>> the point
>> > > of
>> > > > varInts
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Cheers
>> > > > > > > > > > > >> > Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > 17/02/2017,
>> > > > 11:22, "Michael
>> > > > > > Pearce" <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On the
>> > > > point re: headers
>> > > > > in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > being a
>> > > > > > > > > > byte
>> > > > > > > > > > > array
>> > > > > > > > > > > >> > and
>> not a
>> > > count
>> > > > of elements
>> > > > > > followed by the
>> > > > > > > > elements.
>> > > > > > > > > Again
>> > > > > > > > > > this was
>> > > > > > > > > > > >> >
>> > > > discussed/argued previously.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> It was
>> > > > agreed on for a few
>> > > > > > reasons some of
>> > > > > > > > which
>> > > > > > > > > you have
>> > > > > > > > > > > obviously
>> > > > > > > > > > > >> >
>> picked up
>> > on:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Broker
>> > is
>> > > > able to pass it
>> > > > > > through opaquely
>> > > > > > > > > > > >> >
>> The cost
>> > > of
>> > > > parsing, we
>> > > > > > want to
>> > > > > > > > parse/interpret
>> > > > > > > > > the headers
>> > > > > > > > > > lazily
>> > > > > > > > > > > >> > (this
>> is a
>> > key
>> > > > point brought
>> > > > > up
>> > > > > > earlier in
>> > > > > > > > > discussions)
>> > > > > > > > > > > >> >
>> Headers
>> > > can
>> > > > be copied from
>> > > > > > consumer record
>> > > > > > > > to
>> > > > > > > > > producer
>> > > > > > > > > > record (aka
>> > > > > > > > > > > >> > mirror
>> > makers
>> > > > etc) without
>> > > > > > parsing if no
>> > > > > > > > changes are
>> > > > > > > > > being made
>> > > > > > > > > > or
>> > > > > > > > > > > being
>> > > > > > > > > > > >> >
>> looked at.
>> > > > > > > > > > > >> >
>> Keeps
>> > the
>> > > > broker agnostic
>> > > > > > to the format
>> > > > > > > > > > > >> >
>> You need
>> > > an
>> > > > int32 either
>> > > > > > for the byte size
>> > > > > > > > of the
>> > > > > > > > > headers,
>> > > > > > > > > > or for
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > count
>> of
>> > > > elements, so
>> > > > > overheads
>> > > > > > are the same,
>> > > > > > > > but
>> > > > > > > > > with going
>> > > > > > > > > > with an
>> > > > > > > > > > > >> opaque
>> > > > > > > > > > > >> > byte
>> array
>> > has
>> > > > the above
>> > > > > > advantages.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Cheers
>> > > > > > > > > > > >> >
>> Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > > > 17/02/2017, 02:50,
>> > > > > > "Jason Gustafson" <
>> > > > > > > > > jason@confluent.io
>> > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > Sorry,
>> > > > should have
>> > > > > > noted that the
>> > > > > > > > performance
>> > > > > > > > > testing
>> > > > > > > > > > was done
>> > > > > > > > > > > >> > using
>> the
>> > > > > > > > > > > >> >
>> > > > producer performance
>> > > > > > tool shipped with
>> > > > > > > > Kafka.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > -Jason
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On
>> > > Thu,
>> > > > Feb 16, 2017
>> > > > > at
>> > > > > > 6:44 PM, Jason
>> > > > > > > > > Gustafson <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> >
>> > Hey
>> > > > Nacho,
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > I've
>> > > > compared
>> > > > > > performance of our
>> > > > > > > > KIP-98
>> > > > > > > > > > implementation
>> with
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> without
>> > > > > > > > > > > >> >
>> >
>> > > > varints. For
>> > > > > messages
>> > > > > > around 128
>> > > > > > > > bytes, we
>> > > > > > > > > see an
>> > > > > > > > > > increase
>> > > > > > > > > > > in
>> > > > > > > > > > > >> >
>> throughput
>> > of
>> > > > > > > > > > > >> >
>> >
>> > > about
>> > > > 30% using the
>> > > > > > default
>> > > > > > > > configuration
>> > > > > > > > > settings.
>> > > > > > > > > > At 256
>> > > > > > > > > > > >> >
>> bytes, the
>> > > > > > > > > > > >> >
>> >
>> > > > increase is around
>> > > > > > 16%. Obviously
>> > > > > > > the
>> > > > > > > > > performance
>> > > > > > > > > > converge
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> messages get
>> > > > > > > > > > > >> >
>> >
>> > > > larger, but it seems
>> > > > > > well worth the
>> > > > > > > > cost.
>> > > > > > > > > Note that
>> > > > > > > > > > we are
>> > > > > > > > > > > >> also
>> > > > > > > > > > > >> >
>> seeing a
>> > > > > > > > > > > >> >
>> >
>> > > > substantial
>> > > > > > performance increase
>> > > > > > > > against
>> > > > > > > > > trunk
>> > > > > > > > > > primarily
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >
>> > much
>> > > > more efficient
>> > > > > > packing that
>> > > > > > > > varints
>> > > > > > > > > provide us.
>> > > > > > > > > > > Anything
>> > > > > > > > > > > >> >
>> adding to
>> > > > > > > > > > > >> >
>> >
>> > > > message overhead,
>> > > > > > such as record
>> > > > > > > > headers,
>> > > > > > > > > would only
>> > > > > > > > > > > increase
>> > > > > > > > > > > >> > the
>> relative
>> > > > > > > > > > > >> >
>> >
>> > > > difference. (Of
>> > > > > > course take these
>> > > > > > > > numbers
>> > > > > > > > > with a
>> > > > > > > > > > grain of
>> > > > > > > > > > > salt
>> > > > > > > > > > > >> > since
>> I have
>> > > > > > > > > > > >> >
>> >
>> > only
>> > > > used the
>> > > > > default
>> > > > > > settings with
>> > > > > > > > both
>> > > > > > > > > the producer
>> > > > > > > > > > and
>> > > > > > > > > > > >> broker
>> > > > > > > > > > > >> > on my
>> > > > > > > > > > > >> >
>> >
>> > > local
>> > > > machine. We
>> > > > > > intend to provide
>> > > > > > > > more
>> > > > > > > > > extensive
>> > > > > > > > > > > performance
>> > > > > > > > > > > >> >
>> details as
>> > > > > > > > > > > >> >
>> >
>> > part
>> > > > of the work for
>> > > > > > KIP-98.)
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > The
>> > > > implementation
>> > > > > we
>> > > > > > are using is
>> > > > > > > > from
>> > > > > > > > > protobuf (
>> > > > > > > > > > > >> >
>> >
>> > > > > > https://developers.google.com/
>> > > > > > > > > > >
>> > > > protocol-buffers/docs/encoding
>> > > > > > > > > > > >> ),
>> > > > > > > > > > > >> > which
>> is
>> > > > > > > > > > > >> >
>> >
>> > also
>> > > > used in HBase.
>> > > > > > It is trivial to
>> > > > > > > > > implement and as
>> > > > > > > > > > far
>> > > > > > > > > > > as I
>> > > > > > > > > > > >> > know
>> doesn't
>> > > > > > > > > > > >> >
>> >
>> > > > suffer from the
>> > > > > > aliasing problem you
>> > > > > > > > are
>> > > > > > > > > describing. I
>> > > > > > > > > > > checked
>> > > > > > > > > > > >> > with
>> Magnus
>> > > > > > > > > > > >> >
>> >
>> > (the
>> > > > author of
>> > > > > > librdkafka) and he
>> > > > > > > > agreed
>> > > > > > > > > that the
>> > > > > > > > > > savings
>> > > > > > > > > > > >> seemed
>> > > > > > > > > > > >> > worth
>> the
>> > > > > > > > > > > >> >
>> >
>> > cost
>> > > > of
>> > > > > > implementation.
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > -Jason
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > On
>> > > > Thu, Feb 16, 2017
>> > > > > > at 4:32 PM,
>> > > > > > > > Ignacio
>> > > > > > > > > Solis <
>> > > > > > > > > > > >>
>> > isolis@igso.net
>> > > >
>> > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >>
>> > > > -VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > I'm
>> > > > one of the
>> > > > > > people (if not the
>> > > > > > > > most)
>> > > > > > > > > opposed to
>> > > > > > > > > > VarInts.
>> > > > > > > > > > > >> >
>> VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > have
>> > > > a place, but
>> > > > > > this is not it.
>> > > > > > > > (We
>> > > > > > > > > had a large
>> > > > > > > > > > > >>
>> discussion
>> > > > > > > > > > > >> > about
>> > > > > > > > > > > >> >
>> >>
>> > > them
>> > > > at the
>> > > > > > beginning of KIP-82
>> > > > > > > time)
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > anybody has real
>> > > > > > life
>> > > > > > > performance
>> > > > > > > > > numbers of
>> > > > > > > > > > VarInts
>> > > > > > > > > > > >> >
>> improving
>> > > > > > > > > > > >> >
>> >>
>> > > > things or
>> > > > > > significantly reducing
>> > > > > > > > resources
>> > > > > > > > > I would
>> > > > > > > > > > like to
>> > > > > > > > > > > >> know
>> > > > > > > > > > > >> > what
>> > > > > > > > > > > >> >
>> >>
>> > > that
>> > > > case may be.
>> > > > > > Yes, you can save
>> > > > > > > > some
>> > > > > > > > > bytes here
>> > > > > > > > > > and
>> > > > > > > > > > > >> there,
>> > > > > > > > > > > >> > but
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > is probably
>> > > > > > insignificant to
>> > > > > > > the
>> > > > > > > > > overall system
>> > > > > > > > > > > behavior
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> >
>> >>
>> > > > storage
>> > > > > > requirements. -- I say
>> > > > > > > this
>> > > > > > > > with
>> > > > > > > > > respect to
>> > > > > > > > > > using
>> > > > > > > > > > > >> >
>> VarInts in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > protocol
>> > > > > itself,
>> > > > > > not as part of
>> > > > > > > > the
>> > > > > > > > > data.
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > VarInts require you
>> > > > > > to parse the
>> > > > > > > Int
>> > > > > > > > > before using it
>> > > > > > > > > > and
>> > > > > > > > > > > >> >
>> depending on
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > encoding they
>> > > > > > can suffer from
>> > > > > > > > aliasing
>> > > > > > > > > (multiple
>> > > > > > > > > > > >> >
>> > > representations
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > the same
>> > > > > value).
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > Why
>> > > > add complexity?
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > Nacho
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > On
>> > > > Thu, Feb 16,
>> > > > > 2017
>> > > > > > at 10:29 AM,
>> > > > > > > > Colin
>> > > > > > > > > McCabe <
>> > > > > > > > > > > >> >
>> > > > cmccabe@apache.org>
>> > > > > > > > > > > >> >
>> >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >> >
>> > > +1
>> > > > for varints
>> > > > > > here-- it would
>> > > > > > > save
>> > > > > > > > > quite a bit of
>> > > > > > > > > > space.
>> > > > > > > > > > > >> > They
>> are
>> > > > > > > > > > > >> >
>> >> >
>> > > > pretty quick to
>> > > > > > implement as
>> > > > > > > well.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > > I
>> > > > think it makes
>> > > > > > sense for values
>> > > > > > > > to be
>> > > > > > > > > byte
>> > > > > > > > > > arrays.
>> > > > > > > > > > > Users
>> > > > > > > > > > > >> > might
>> want
>> > > > > > > > > > > >> >
>> >> >
>> > > to
>> > > > attach
>> > > > > > arbitrary payloads;
>> > > > > > > they
>> > > > > > > > > shouldn't be
>> > > > > > > > > > forced to
>> > > > > > > > > > > >> >
>> serialize
>> > > > > > > > > > > >> >
>> >> >
>> > > > everything to
>> > > > > Java
>> > > > > > strings.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > best,
>> > > > > > > > > > > >> >
>> >> >
>> > > > Colin
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > On
>> > > > Thu, Feb 16,
>> > > > > > 2017, at 09:52,
>> > > > > > > > Jason
>> > > > > > > > > Gustafson
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hey Michael,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hmm, I guess the
>> > > > > > point of
>> > > > > > > > representing
>> > > > > > > > > it as
>> > > > > > > > > > bytes is to
>> > > > > > > > > > > >> > allow
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > broker
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > to pass it
>> > > > > > through opaquely? Is
>> > > > > > > > the
>> > > > > > > > > cost of
>> > > > > > > > > > parsing
>> > > > > > > > > > > them a
>> > > > > > > > > > > >> >
>> concern, or
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > are
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > we simply trying
>> > > > > > to ensure that
>> > > > > > > > the
>> > > > > > > > > broker stays
>> > > > > > > > > > > agnostic
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > format?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On varints, I
>> > > > > > think adding
>> > > > > > > > support for
>> > > > > > > > > them makes
>> > > > > > > > > > less
>> > > > > > > > > > > >> sense
>> > > > > > > > > > > >> > for an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > isolated use
>> > > > > > case, but as part
>> > > > > > > of
>> > > > > > > > a
>> > > > > > > > > more holistic
>> > > > > > > > > > change
>> > > > > > > > > > > >> > (such
>> as
>> > what
>> > > > > > > > > > > >> >
>> >>
>> > we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > have proposed in
>> > > > > > KIP-98), I
>> > > > > > > think
>> > > > > > > > they
>> > > > > > > > > are
>> > > > > > > > > > justifiable.
>> > > > > > > > > > > If
>> > > > > > > > > > > >> > we
>> add them,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > then the need to
>> > > > > > use attributes
>> > > > > > > > becomes
>> > > > > > > > > quite a
>> > > > > > > > > > bit
>> > > > > > > > > > > >> weaker,
>> > > > > > > > > > > >> >
>> right? The
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > other thing I
>> > > > > > find slightly odd
>> > > > > > > > is the
>> > > > > > > > > fact that
>> > > > > > > > > > null
>> > > > > > > > > > > >> >
>> headers has
>> > no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > actual
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > semantic meaning
>> > > > > > for the message
>> > > > > > > > > (unlike null
>> > > > > > > > > > keys and
>> > > > > > > > > > > >> >
>> values). It
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > a
>> > > > space
>> > > > > > optimization. It seems a
>> > > > > > > > bit
>> > > > > > > > > better to
>> > > > > > > > > > always
>> > > > > > > > > > > use
>> > > > > > > > > > > >> > size
>> 0 to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > indicate having
>> > > > > > no headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Overall, the
>> > > > > main
>> > > > > > point is
>> > > > > > > > ensuring
>> > > > > > > > > that the
>> > > > > > > > > > message
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> >
>> remains
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > consistent,
>> > > > > > either within the
>> > > > > > > > larger
>> > > > > > > > > protocol, or
>> > > > > > > > > > at a
>> > > > > > > > > > > >> >
>> minimum
>> > within
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > message itself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On Thu, Feb 16,
>> > > > > > 2017 at 6:39 AM,
>> > > > > > > > > Michael Pearce <
>> > > > > > > > > > > >> >
>> > > > Michael.Pearce@ig.com
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Hi Jason,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 1) in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > the headers
>> > > > > > > > > > are
>> > > > > > > > > > > >> simply
>> > > > > > > > > > > >> > a byte
>> > > > > > > > > > > >> >
>> >>
>> > > > array,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > as like the
>> > > > > key
>> > > > > > or value, this
>> > > > > > > > is to
>> > > > > > > > > clearly
>> > > > > > > > > > demarcate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> header in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > core message.
>> > > > > > Then the header
>> > > > > > > > byte
>> > > > > > > > > array in the
>> > > > > > > > > > core
>> > > > > > > > > > > >> >
>> message is
>> > an
>> > > > > > > > > > > >> >
>> >>
>> > > > array of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key, value
>> > > > > > pairs. This is what
>> > > > > > > > it is
>> > > > > > > > > denoting.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Then this
>> > > > > would
>> > > > > > be I guess in
>> > > > > > > > the
>> > > > > > > > > given
>> > > > > > > > > > notation:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Headers =>
>> > > > > > [KeyLength, Key,
>> > > > > > > > > ValueLength, Value]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > KeyLength
>> > > > > > => int32
>> > > > > > > > > <-----------------NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised key
>> > > > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Key =>
>> > > > > bytes
>> > > > > > > > > <---------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > string
>> > (UTF8)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > bytes of the
>> > > > > > header key
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > ValueLength
>> > > > > > => int32
>> > > > > > > > > <-------------- NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised
>> > > > > > header value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Value =>
>> > > > > > bytes
>> > > > > > > > > <-------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > form
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The key length
>> > > > > > and value
>> > > > > > > length
>> > > > > > > > is
>> > > > > > > > > matching the
>> > > > > > > > > > way
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> protocol is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > defined in the
>> > > > > > core message
>> > > > > > > > currently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 2)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Var sized
>> > > > > ints,
>> > > > > > this was
>> > > > > > > > discussed
>> > > > > > > > > much earlier
>> > > > > > > > > > on, in
>> > > > > > > > > > > >> > fact
>> I had
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > suggested it
>> > > > > > myself (with
>> > > > > > > Hadoop
>> > > > > > > > > references),
>> > > > > > > > > > the
>> > > > > > > > > > > >> >
>> complexity
>> > of
>> > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compared to
>> > > > > > having a simpler
>> > > > > > > > protocol
>> > > > > > > > > was
>> > > > > > > > > > argued and
>> > > > > > > > > > > >> >
>> agreed it
>> > > > > > > > > > > >> >
>> >>
>> > > > wasn’t worth
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the complexity
>> > > > > > as all other
>> > > > > > > > clients
>> > > > > > > > > in other
>> > > > > > > > > > languages
>> > > > > > > > > > > >> > would
>> need
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > > > ensure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > theyre using
>> > > > > > the right var
>> > > > > > > size
>> > > > > > > > > algorithm, as
>> > > > > > > > > > there
>> > > > > > > > > > > is a
>> > > > > > > > > > > >> > few.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 3)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > We did the
>> > > > > > attributes,
>> > > > > > > optional
>> > > > > > > > > approach as
>> > > > > > > > > > originally
>> > > > > > > > > > > >> > there
>> was
>> > > > > > > > > > > >> >
>> >>
>> > > > marked
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > concern that
>> > > > > > headers would
>> > > > > > > > cause a
>> > > > > > > > > message size
>> > > > > > > > > > > overhead
>> > > > > > > > > > > >> > for
>> others,
>> > > > > > > > > > > >> >
>> >>
>> > who
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > don’t want
>> > > > > > them. As such this
>> > > > > > > > is the
>> > > > > > > > > clean
>> > > > > > > > > > solution to
>> > > > > > > > > > > >> >
>> achieve
>> > that.
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that no longer
>> > > > > > holds, and we
>> > > > > > > > don’t
>> > > > > > > > > care that we
>> > > > > > > > > > add
>> > > > > > > > > > > >> 4bytes
>> > > > > > > > > > > >> >
>> overhead,
>> > > > > > > > > > > >> >
>> >>
>> > > then
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > im happy to
>> > > > > > remove.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > I’m personally
>> > > > > > in favour of
>> > > > > > > > keeping
>> > > > > > > > > the message
>> > > > > > > > > > as
>> > > > > > > > > > > small
>> > > > > > > > > > > >> > as
>> possible
>> > > > > > > > > > > >> >
>> >>
>> > so
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > people don’t
>> > > > > > get shocks in
>> > > > > > > perf
>> > > > > > > > and
>> > > > > > > > > throughputs
>> > > > > > > > > > dues
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> message
>> > size,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > unless they
>> > > > > > actively use the
>> > > > > > > > feature,
>> > > > > > > > > as such I
>> > > > > > > > > > do
>> > > > > > > > > > > >> prefer
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > attribute bit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wise feature
>> > > > > > flag approach
>> > > > > > > > myself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On 16/02/2017,
>> > > > > > 05:40, "Jason
>> > > > > > > > > Gustafson" <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > We have
>> > > > > > proposed a few
>> > > > > > > > > significant changes
>> > > > > > > > > > to the
>> > > > > > > > > > > >> >
>> message
>> > format
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > KIP-98
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > which now
>> > > > > > seems likely to
>> > > > > > > > pass
>> > > > > > > > > (perhaps
>> > > > > > > > > > with some
>> > > > > > > > > > > >> >
>> iterations
>> > on
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > implementation details).
>> > > > > > > It
>> > > > > > > > would
>> > > > > > > > > be good
>> > > > > > > > > > to try
>> > > > > > > > > > > and
>> > > > > > > > > > > >> >
>> coordinate
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > changes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > in both of
>> > > > > > the proposals
>> > > > > > > to
>> > > > > > > > make
>> > > > > > > > > sure they
>> > > > > > > > > > are
>> > > > > > > > > > > >> >
>> consistent
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compatible.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I think
>> > > > > > using the
>> > > > > > > > attributes to
>> > > > > > > > > indicate
>> > > > > > > > > > null
>> > > > > > > > > > > >> headers
>> > > > > > > > > > > >> > is a
>> > > > > > > > > > > >> >
>> >>
>> > > > reasonable
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > approach.
>> > > > > > We have proposed
>> > > > > > > > to do
>> > > > > > > > > the same
>> > > > > > > > > > thing
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> message
>> > > > > > > > > > > >> >
>> >>
>> > key
>> > > > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > value.
>> > > > > That
>> > > > > > said, I
>> > > > > > > > sympathize
>> > > > > > > > > with Jay's
>> > > > > > > > > > > argument.
>> > > > > > > > > > > >> > Having
>> > > > > > > > > > > >> >
>> >>
>> > > > multiple
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > ways to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > specify a
>> > > > > > null value
>> > > > > > > > increases
>> > > > > > > > > the overall
>> > > > > > > > > > > >>
>> complexity
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > You
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > can see
>> > > > > > this just from the
>> > > > > > > > fact
>> > > > > > > > > that you
>> > > > > > > > > > need the
>> > > > > > > > > > > >> > extra
>> > verbiage
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > protocol
>> > > > > > specification in
>> > > > > > > > this
>> > > > > > > > > KIP and in
>> > > > > > > > > > KIP-98
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> describe the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > dependence
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > between
>> > > > > the
>> > > > > > fields and the
>> > > > > > > > > attributes. It
>> > > > > > > > > > seems
>> > > > > > > > > > > >> like a
>> > > > > > > > > > > >> >
>> slippery
>> > > > > > > > > > > >> >
>> >>
>> > > slope
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > if
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > you start
>> > > > > > allowing
>> > > > > > > different
>> > > > > > > > > request types
>> > > > > > > > > > to
>> > > > > > > > > > > >> >
>> implement
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification differently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > You can
>> > > > > > also argue that
>> > > > > > > the
>> > > > > > > > > messages
>> > > > > > > > > > already are
>> > > > > > > > > > > and
>> > > > > > > > > > > >> > are
>> likely
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > remain a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > special
>> > > > > > case. For example,
>> > > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > currently no
>> > > > > > > > > > > >> >
>> generality
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > how
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > compressed
>> > > > > > message sets
>> > > > > > > are
>> > > > > > > > > represented
>> > > > > > > > > > that would
>> > > > > > > > > > > >> be
>> > > > > > > > > > > >> >
>> applicable
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > other
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > request
>> > > > > > types. Some might
>> > > > > > > > see this
>> > > > > > > > > > divergence as
>> > > > > > > > > > > an
>> > > > > > > > > > > >> >
>> unfortunate
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > deficiency
>> > > > > > which should be
>> > > > > > > > fixed;
>> > > > > > > > > others
>> > > > > > > > > > might see
>> > > > > > > > > > > >> it
>> > > > > > > > > > > >> > as
>> sort of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > inevitability of needing
>> > > > > > > to
>> > > > > > > > > optimize where
>> > > > > > > > > > it
>> > > > > > > > > > > counts
>> > > > > > > > > > > >> > most.
>> I'm
>> > > > > > > > > > > >> >
>> >>
>> > > > probably
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > somewhere
>> > > > > > in between, but
>> > > > > > > I
>> > > > > > > > think
>> > > > > > > > > we
>> > > > > > > > > > probably all
>> > > > > > > > > > > >> > share
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > intuition
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol should be
>> > > > > > > kept
>> > > > > > > > as
>> > > > > > > > > consistent as
>> > > > > > > > > > > >>
>> possible.
>> > > > > > > > > > > >> > With
>> that
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > mind,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > here are a
>> > > > > > few comments:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 1. One
>> > > > > > thing I found a
>> > > > > > > > little odd
>> > > > > > > > > when
>> > > > > > > > > > reading the
>> > > > > > > > > > > >> >
>> current
>> > > > > > > > > > > >> >
>> >>
>> > > > proposal is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > headers
>> > > > > > are both
>> > > > > > > > represented
>> > > > > > > > > as an
>> > > > > > > > > > array of
>> > > > > > > > > > > >> bytes
>> > > > > > > > > > > >> > and
>> as an
>> > > > > > > > > > > >> >
>> >>
>> > > array
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > key/value
>> > > > > > pairs. I'd
>> > > > > > > > probably
>> > > > > > > > > suggest
>> > > > > > > > > > something
>> > > > > > > > > > > like
>> > > > > > > > > > > >> > this:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > [HeaderKey
>> > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > HeaderKey
>> > > > > > => String
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue => Bytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > An array
>> > > > > in
>> > > > > > the Kafka
>> > > > > > > > protocol is
>> > > > > > > > > > represented as a
>> > > > > > > > > > > >> > 4-byte
>> > integer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > indicating
>> > > > > > the number of
>> > > > > > > > elements
>> > > > > > > > > in the
>> > > > > > > > > > array
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > serialization of the
>> > > > > > > > elements.
>> > > > > > > > > Unless I'm
>> > > > > > > > > > > >> >
>> > > > misunderstanding, what
>> > > > > > > > > > > >> >
>> >>
>> > you
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > instead is
>> > > > > > the total size
>> > > > > > > > of the
>> > > > > > > > > headers in
>> > > > > > > > > > bytes
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > elements.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I'm not
>> > > > > > sure I see any
>> > > > > > > > reason for
>> > > > > > > > > this
>> > > > > > > > > > > >>
>> inconsistency.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 2. In
>> > > > > > KIP-98, we've
>> > > > > > > > introduced
>> > > > > > > > > > variable-length
>> > > > > > > > > > > >> integer
>> > > > > > > > > > > >> >
>> fields.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Effectively,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > we've
>> > > > > > enriched (or
>> > > > > > > > "complicated"
>> > > > > > > > > as Jay
>> > > > > > > > > > might say
>> > > > > > > > > > > ;)
>> > > > > > > > > > > >> > the
>> protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification to include
>> > > > > > > the
>> > > > > > > > > following
>> > > > > > > > > > types:
>> > > > > > > > > > > >> VarInt,
>> > > > > > > > > > > >> >
>> VarLong,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > UnsignedVarInt and
>> > > > > > > > > UnsignedVarLong.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Along with
>> > > > > > these
>> > > > > > > > primitives, we
>> > > > > > > > > could
>> > > > > > > > > > introduce
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> following
>> > > > > > > > > > > >> >
>> >>
>> > > > types:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeArray =>
>> > > > > > > > NumberOfItems
>> > > > > > > > > Item1 Item2
>> > > > > > > > > > .. ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItems =>
>> > > > > > > > UnsignedVarInt
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeNullableArray =>
>> > > > > > > > > NumberOfItemsOrNull
>> > > > > > > > > > Item1
>> > > > > > > > > > > >> > Item2
>> ..
>> > ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItemsOrNull =>
>> > > > > > > > VarInt
>> > > > > > > > > (-1 means
>> > > > > > > > > > null)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > And
>> > > > > > similarly for the
>> > > > > > > > `String`
>> > > > > > > > > and `Bytes`
>> > > > > > > > > > types.
>> > > > > > > > > > > >> > These
>> types
>> > > > > > > > > > > >> >
>> >>
>> > can
>> > > > save
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > considerable amount of
>> > > > > > > > space in
>> > > > > > > > > this
>> > > > > > > > > > proposal
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > they
>> can
>> > > > > > > > > > > >> >
>> >>
>> > be
>> > > > used
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > both the
>> > > > > > number of headers
>> > > > > > > > > included in the
>> > > > > > > > > > message
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > lengths of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > header
>> > > > > keys
>> > > > > > and values. We
>> > > > > > > > could
>> > > > > > > > > do this
>> > > > > > > > > > instead:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > > > VarSizeArray[HeaderKey
>> > > > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > HeaderKey
>> > > > > > =>
>> > > > > > > VarSizeString
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue =>
>> > > > > > > > VarSizeBytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Combining
>> > > > > > the savings from
>> > > > > > > > the
>> > > > > > > > > use of
>> > > > > > > > > > variable
>> > > > > > > > > > > >> length
>> > > > > > > > > > > >> >
>> fields, the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > benefit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > of using
>> > > > > > the attributes to
>> > > > > > > > > represent null
>> > > > > > > > > > seems
>> > > > > > > > > > > >> pretty
>> > > > > > > > > > > >> > small.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 3.
>> > > > > > Whichever way we go
>> > > > > > > > (whether
>> > > > > > > > > we use the
>> > > > > > > > > > > >>
>> attributes
>> > > > > > > > > > > >> > or
>> not), we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > should at
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > least be
>> > > > > > consistent
>> > > > > > > between
>> > > > > > > > this
>> > > > > > > > > KIP and
>> > > > > > > > > > KIP-98.
>> > > > > > > > > > > It
>> > > > > > > > > > > >> > would
>> be
>> > very
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > strange
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > to have
>> > > > > two
>> > > > > > ways to
>> > > > > > > > represent
>> > > > > > > > > null values
>> > > > > > > > > > in the
>> > > > > > > > > > > >> same
>> > > > > > > > > > > >> >
>> schema.
>> > > > > > > > > > > >> >
>> >>
>> > > > Either
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > way is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > OK with
>> > > > > me.
>> > > > > > I think some
>> > > > > > > > > message-level
>> > > > > > > > > > >
>> optimizations
>> > > > > > > > > > > >> > are
>> > > > > > > > > > > >> >
>> >>
>> > > > justifiable,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > but
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > savings
>> > > > > > here seem
>> > > > > > > > minimal (a
>> > > > > > > > > few bytes
>> > > > > > > > > > per
>> > > > > > > > > > > >> >
>> message), so
>> > > > > > > > > > > >> >
>> >>
>> > > > maybe it's
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > worth the
>> > > > > > cost of letting
>> > > > > > > > the
>> > > > > > > > > message
>> > > > > > > > > > diverge even
>> > > > > > > > > > > >> >
>> further from
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > rest of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > On Wed,
>> > > > > Feb
>> > > > > > 15, 2017 at
>> > > > > > > > 8:52 AM,
>> > > > > > > > > radai <
>> > > > > > > > > > > >> >
>> >>
>> > > > > > radai.rosenblatt@gmail.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > I've
>> > > > > > trimmed the inline
>> > > > > > > > > contents as this
>> > > > > > > > > > mail is
>> > > > > > > > > > > >> >
>> getting too
>> > > > > > > > > > > >> >
>> >>
>> > big
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > apache
>> > > > > > mailing list
>> > > > > > > > software to
>> > > > > > > > > deliver
>> > > > > > > > > > :-(
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 1. the
>> > > > > > important thing
>> > > > > > > for
>> > > > > > > > > > interoperability
>> is
>> > > > > > > > > > > for
>> > > > > > > > > > > >> >
>> different
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "interested
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > parties"
>> > > > > > (plugins, infra
>> > > > > > > > > layers/wrappers,
>> > > > > > > > > > > >>
>> user-code)
>> > > > > > > > > > > >> > to be
>> > > > > > > > > > > >> >
>> >>
>> > > able
>> > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > stick
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > pieces
>> > > > > of
>> > > > > > metadata onto
>> > > > > > > > msgs
>> > > > > > > > > without
>> > > > > > > > > > getting in
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> >
>> other's
>> > > > > > > > > > > >> >
>> >>
>> > > way.
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > common
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > key
>> > > > > > scheme (Strings, as
>> > > > > > > > of the
>> > > > > > > > > time of
>> > > > > > > > > > this
>> > > > > > > > > > > >> >
>> writing?) is
>> > > all
>> > > > > > > > > > > >> >
>> >>
>> > > thats
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > required
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > for
>> > > > > that.
>> > > > > > it is assumed
>> > > > > > > > that
>> > > > > > > > > the other end
>> > > > > > > > > > > >> >
>> interested
>> > in
>> > > > any
>> > > > > > > > > > > >> >
>> >>
>> > > such
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > piece of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > metadata
>> > > > > > knows the
>> > > > > > > > encoding,
>> > > > > > > > > and byte[]
>> > > > > > > > > > provides
>> > > > > > > > > > > >> for
>> > > > > > > > > > > >> > the
>> most
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > flexibility.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > i
>> > > > > believe
>> > > > > > this is the
>> > > > > > > same
>> > > > > > > > > logic behind
>> > > > > > > > > > core
>> > > > > > > > > > > kafka
>> > > > > > > > > > > >> > being
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > byte[]/byte[]
>> > > > > -
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Strings
>> > > > > > are more
>> > > > > > > "usable"
>> > > > > > > > but
>> > > > > > > > > bytes are
>> > > > > > > > > > flexible
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > so
>> were
>> > > > > > > > > > > >> >
>> >>
>> > > > chosen.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Also -
>> > > > > > core kafka doesnt
>> > > > > > > > even
>> > > > > > > > > do that
>> > > > > > > > > > good of a
>> > > > > > > > > > > >> job
>> > > > > > > > > > > >> > on
>> > > > > > > > > > > >> >
>> >>
>> > > > usability of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > payload
>> > > > > > (example - i
>> > > > > > > have
>> > > > > > > > to
>> > > > > > > > > specify the
>> > > > > > > > > > nop
>> > > > > > > > > > > >> byte[]
>> > > > > > > > > > > >> >
>> "decoders"
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > explicitly
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > in
>> > > > > conf),
>> > > > > > and again
>> > > > > > > > sacrificies
>> > > > > > > > > usability
>> > > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > sake
>> of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > performance
>> > > > > (no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > convenient single-record
>> > > > > > > > > processing as
>> > > > > > > > > > poll is a
>> > > > > > > > > > > >> >
>> batch, lots
>> > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > obscure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > little
>> > > > > > config details
>> > > > > > > > exposing
>> > > > > > > > > internals
>> > > > > > > > > > of the
>> > > > > > > > > > > >> >
>> batching
>> > > > > > > > > > > >> >
>> >>
>> > > > mechanism,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > etc)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > this is
>> > > > > > also why i
>> > > > > > > really
>> > > > > > > > > dislike the
>> > > > > > > > > > idea of a
>> > > > > > > > > > > >> > "type
>> > system"
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > values,
>> > > > > > it further
>> > > > > > > > degrades the
>> > > > > > > > > > usability, adds
>> > > > > > > > > > > >> >
>> complexity
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > eventually get in
>> > > > > > > > people's way,
>> > > > > > > > > also, it
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >> > the
>> 2nd/3rd
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > home-group
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization mechanism
>> > > > > > > > in core
>> > > > > > > > > kafka
>> > > > > > > > > > (counting
>> > > > > > > > > > > 2
>> > > > > > > > > > > >> >
>> iterations
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "type
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > definition DSL")
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 2. this
>> > > > > > is an
>> > > > > > > > implementation
>> > > > > > > > > detail, and
>> > > > > > > > > > not
>> > > > > > > > > > > even
>> > > > > > > > > > > >> a
>> > > > > > > > > > > >> > very
>> "user
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > facing" one?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to the
>> > > > > > best of my
>> > > > > > > > understanding
>> > > > > > > > > the vote
>> > > > > > > > > > process
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > on
>> proposed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > API/behaviour. also -
>> > > > > > > > since
>> > > > > > > > > we're willing
>> > > > > > > > > > to go
>> > > > > > > > > > > >> with
>> > > > > > > > > > > >> >
>> strings
>> > > > > > > > > > > >> >
>> >>
>> > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialize
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > a
>> > > > > 0-sized
>> > > > > > header blob
>> > > > > > > and
>> > > > > > > > IIUC
>> > > > > > > > > you dont
>> > > > > > > > > > need any
>> > > > > > > > > > > >> >
>> optionals
>> > > > > > > > > > > >> >
>> >>
>> > > > anymore.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 3. yes,
>> > > > > > we can :-)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > On Tue,
>> > > > > > Feb 14, 2017 at
>> > > > > > > > 11:56
>> > > > > > > > > PM, Michael
>> > > > > > > > > > > Pearce <
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Hi
>> > > > > Jay,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1)
>> > > > > > There was some
>> > > > > > > > initial
>> > > > > > > > > debate on the
>> > > > > > > > > > value
>> > > > > > > > > > > >> > part,
>> as
>> > youll
>> > > > > > > > > > > >> >
>> >>
>> > > note
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > String,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > String
>> > > > > > headers were
>> > > > > > > > > discounted early
>> > > > > > > > > > on. The
>> > > > > > > > > > > >> >
>> reason for
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > flexibility
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > keeping in line
>> > > > > > > > with the
>> > > > > > > > > > flexibility of
>> > > > > > > > > > > key,
>> > > > > > > > > > > >> > value
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > message
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > object
>> > > > > > itself. I don’t
>> > > > > > > > think
>> > > > > > > > > it takes
>> > > > > > > > > > away
>> > > > > > > > > > > from
>> > > > > > > > > > > >> an
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > as
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > each
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > plugin
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > will
>> > > > > > care for their
>> > > > > > > own
>> > > > > > > > key,
>> > > > > > > > > this way
>> > > > > > > > > > ints,
>> > > > > > > > > > > >> >
>> booleans ,
>> > > > exotic
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > binary
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > can
>> > > > > all
>> > > > > > be catered
>> > > > > > > for=.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > wanted
>> > > > > > > > to
>> > > > > > > > > push for a
>> > > > > > > > > > typed
>> > > > > > > > > > > >> value
>> > > > > > > > > > > >> >
>> interface,
>> > > > > > > > > > > >> >
>> >> I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wouldn’t
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > want
>> > > > > > just String
>> > > > > > > values
>> > > > > > > > > supported, but
>> > > > > > > > > > the the
>> > > > > > > > > > > >> >
>> primatives
>> > > > > > > > > > > >> >
>> >>
>> > > plus
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > string and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > also
>> > > > > > still keeping the
>> > > > > > > > > ability to have a
>> > > > > > > > > > > binary
>> > > > > > > > > > > >> > for
>> custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > binaries that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > some
>> > > > > > organisations may
>> > > > > > > > have.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > i. I
>> > > > > > have written this
>> > > > > > > > slight
>> > > > > > > > > > alternative
>> > > > > > > > > > > here,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > https://cwiki.apache.org/
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > > >> >
>> > > > 82+-+Add+Record+Headers+-+
>> > > > > Typed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > ii.
>> > > > > > Essentially the
>> > > > > > > > value
>> > > > > > > > > bytes, has a
>> > > > > > > > > > leading
>> > > > > > > > > > > >> > byte
>> > overhead.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > > This tells you
>> > > > > > > what
>> > > > > > > > type
>> > > > > > > > > the value
>> > > > > > > > > > is,
>> > > > > > > > > > > >> before
>> > > > > > > > > > > >> >
>> reading
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > rest
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes,
>> > > > > > allowing
>> > > > > > > > > >
>> > > > serialisation/deserialization
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > and
>> from the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > primitives,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > string
>> > > > > > and byte[].
>> > > > > > > This
>> > > > > > > > is
>> > > > > > > > > akin to some
>> > > > > > > > > > other
>> > > > > > > > > > > >> >
>> messaging
>> > > > > > > > > > > >> >
>> >>
>> > > > systems.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2) We
>> > > > > > are making it
>> > > > > > > > optional,
>> > > > > > > > > so that
>> > > > > > > > > > for
>> > > > > > > > > > > those
>> > > > > > > > > > > >> > not
>> wanting
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > headers have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 0
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes
>> > > > > > overhead (think
>> > > > > > > > of it
>> > > > > > > > > as a feature
>> > > > > > > > > > > flag),
>> > > > > > > > > > > >> I
>> > > > > > > > > > > >> > don’t
>> > > > > > > > > > > >> >
>> >>
>> > > > think this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > complex, especially if
>> > > > > > > > > comparing to
>> > > > > > > > > > changes
>> > > > > > > > > > > >> >
>> proposed in
>> > > > > > > > > > > >> >
>> >>
>> > > > other kips
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > like
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > kip-98.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > really
>> > > > > > > > don’t
>> > > > > > > > > like
>> > > > > > > > > > this, we
>> > > > > > > > > > > can
>> > > > > > > > > > > >> > drop
>> it, but
>> > > > > > > > > > > >> >
>> >>
>> > it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > would
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > mean
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > buying
>> > > > > > into 4 bytes
>> > > > > > > > extra
>> > > > > > > > > overhead for
>> > > > > > > > > > users
>> > > > > > > > > > > who
>> > > > > > > > > > > >> > do
>> not want
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > use
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3) In
>> > > > > > the summary yes,
>> > > > > > > > it is
>> > > > > > > > > at a higher
>> > > > > > > > > > > level,
>> > > > > > > > > > > >> > but I
>> think
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is well
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > documented in the
>> > > > > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > section.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a.
>> > > > > > Added getHeaders
>> > > > > > > > method to
>> > > > > > > > > > >
>> Producer/Consumer
>> > > > > > > > > > > >> >
>> record (that
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > it)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > b.
>> > > > > > We’ve also detailed
>> > > > > > > > the
>> > > > > > > > > new Headers
>> > > > > > > > > > class
>> > > > > > > > > > > >> that
>> > > > > > > > > > > >> > this
>> method
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > returns
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > encapsulates the
>> > > > > > > headers
>> > > > > > > > > protocol and
>> > > > > > > > > > logic.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Best,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Mike
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ==Original questions
>> > > > > > > > from the
>> > > > > > > > > vote
>> > > > > > > > > > thread from
>> > > > > > > > > > > >> > Jay.==
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Couple
>> > > > > > of things I
>> > > > > > > > think we
>> > > > > > > > > still need
>> > > > > > > > > > to work
>> > > > > > > > > > > >> out:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > I
>> > > > > > think we agree
>> > > > > > > > about
>> > > > > > > > > the key,
>> > > > > > > > > > but I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > we
>> haven't
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > talked about
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > value yet. I
>> > > > > > > > think if
>> > > > > > > > > our goal
>> > > > > > > > > > is an
>> > > > > > > > > > > open
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > these
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > spread across many
>> > > > > > > > plugins
>> > > > > > > > > from many
>> > > > > > > > > > > systems
>> > > > > > > > > > > >> we
>> > > > > > > > > > > >> > should
>> > > > > > > > > > > >> >
>> >>
>> > > > consider
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > making
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > string as well so
>> > > > > > > > it can
>> > > > > > > > > be
>> > > > > > > > > > printed, set
>> > > > > > > > > > > >> via
>> > > > > > > > > > > >> > a UI,
>> set
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > config,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > etc.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > Basically
>> > > > > > > encouraging
>> > > > > > > > > pluggable
>> > > > > > > > > > >
>> serialization
>> > > > > > > > > > > >> >
>> formats
>> > > > > > > > > > > >> >
>> >>
>> > > here
>> > > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > lead
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bit
>> > > > > > of a tower of
>> > > > > > > > babel.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2.
>> > > > > > This proposal
>> > > > > > > > still
>> > > > > > > > > includes a
>> > > > > > > > > > pretty
>> > > > > > > > > > > big
>> > > > > > > > > > > >> >
>> change to
>> > our
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > protocol
>> > > > > > > > definition
>> > > > > > > > > layer.
>> > > > > > > > > > Essentially
>> > > > > > > > > > > >> it is
>> > > > > > > > > > > >> >
>> >>
>> > > > introducing an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > optional
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > type, where the
>> > > > > > > > format is
>> > > > > > > > > data
>> > > > > > > > > > dependent. I
>> > > > > > > > > > > >> > think
>> this
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > actually a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > big
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > change though it
>> > > > > > > > doesn't
>> > > > > > > > > seem like
>> > > > > > > > > > it. It
>> > > > > > > > > > > >> means
>> > > > > > > > > > > >> > you
>> can no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > longer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > specify
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > this
>> > > > > > type with our
>> > > > > > > > type
>> > > > > > > > > definition
>> > > > > > > > > > DSL, and
>> > > > > > > > > > > >> >
>> likewise it
>> > > > > > > > > > > >> >
>> >>
>> > > > requires
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > handling in client
>> > > > > > > > libs.
>> > > > > > > > > This isn't
>> > > > > > > > > > a huge
>> > > > > > > > > > > >> >
>> thing, since
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Record
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > definition is
>> > > > > > > custom
>> > > > > > > > > anyway, but I
>> > > > > > > > > > think
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > kind
>> of
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > inconsistency is
>> > > > > > > very
>> > > > > > > > > non-desirable
>> > > > > > > > > > and
>> > > > > > > > > > > ties
>> > > > > > > > > > > >> > you to
>> > > > > > > > > > > >> >
>> >>
>> > > > hand-coding
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > things.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > think the type
>> > > > > > > should
>> > > > > > > > > instead by [Key
>> > > > > > > > > > > Value]
>> > > > > > > > > > > >> in
>> > > > > > > > > > > >> > our
>> BNF,
>> > > > > > > > > > > >> >
>> >>
>> > > where
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > value are both
>> > > > > > > short
>> > > > > > > > > strings as used
>> > > > > > > > > > > >>
>> elsewhere.
>> > > > > > > > > > > >> > This
>> > > > > > > > > > > >> >
>> >>
>> > > > brings it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > in line
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > with
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > rest of the
>> > > > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3.
>> > > > > > Could we get
>> > > > > > > more
>> > > > > > > > > specific about
>> > > > > > > > > > the
>> > > > > > > > > > > exact
>> > > > > > > > > > > >> > Java
>> API
>> > > > > > > > > > > >> >
>> >>
>> > > > change to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ProducerRecord,
>> > > > > > > > > ConsumerRecord,
>> > > > > > > > > > Record,
>> > > > > > > > > > > etc?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > -Jay
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The
>> > > > > information
>> > > > > > contained in
>> > > > > > > > this
>> > > > > > > > > email is
>> > > > > > > > > > strictly
>> > > > > > > > > > > >> >
>> confidential
>> > > and
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the use of the
>> > > > > > addressee only,
>> > > > > > > > unless
>> > > > > > > > > otherwise
>> > > > > > > > > > > >>
>> indicated.
>> > > > > > > > > > > >> > If you
>> > > > > > > > > > > >> >
>> >>
>> > are
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the intended
>> > > > > > recipient, please
>> > > > > > > > do not
>> > > > > > > > > read,
>> > > > > > > > > > copy, use
>> > > > > > > > > > > or
>> > > > > > > > > > > >> >
>> disclose to
>> > > > > > > > > > > >> >
>> >>
>> > > > others
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > this message
>> > > > > or
>> > > > > > any
>> > > > > > > attachment.
>> > > > > > > > > Please also
>> > > > > > > > > > notify the
>> > > > > > > > > > > >> >
>> sender by
>> > > > > > > > > > > >> >
>> >>
>> > > > replying
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > to this email
>> > > > > > or by telephone
>> > > > > > > > > (+44(020 7896
>> > > > > > > > > > 0011) and
>> > > > > > > > > > > >> then
>> > > > > > > > > > > >> > delete
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > email
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > and any copies
>> > > > > > of it.
>> > > > > > > Opinions,
>> > > > > > > > > conclusion
>> > > > > > > > > > (etc) that
>> > > > > > > > > > > do
>> > > > > > > > > > > >> > not
>> relate
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > official
>> > > > > > business of this
>> > > > > > > > company
>> > > > > > > > > shall be
>> > > > > > > > > > understood
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> neither
>> > > > > > > > > > > >> >
>> >>
>> > > > given nor
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > endorsed by
>> > > > > it.
>> > > > > > IG is a
>> > > > > > > trading
>> > > > > > > > name
>> > > > > > > > > of IG
>> > > > > > > > > > Markets
>> > > > > > > > > > > >> Limited
>> > > > > > > > > > > >> > (a
>> company
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > registered in
>> > > > > > England and
>> > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 04008957)
>> > > > > > > > > > > >> > and IG
>> > > > > > > > > > > >> >
>> >>
>> > > Index
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Limited (a
>> > > > > > company registered
>> > > > > > > in
>> > > > > > > > > England and
>> > > > > > > > > > Wales,
>> > > > > > > > > > > >> >
>> company
>> > number
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > 01190902).
>> > > > > > Registered address
>> > > > > > > at
>> > > > > > > > > Cannon Bridge
>> > > > > > > > > > House,
>> > > > > > > > > > > 25
>> > > > > > > > > > > >> >
>> Dowgate
>> > > > > > > > > > > >> >
>> >>
>> > > Hill,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > London EC4R
>> > > > > > 2YA. Both IG
>> > > > > > > Markets
>> > > > > > > > > Limited
>> > > > > > > > > > (register
>> > > > > > > > > > > >> number
>> > > > > > > > > > > >> >
>> 195355)
>> > > > > > > > > > > >> >
>> >>
>> > and
>> > > > IG
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Index Limited
>> > > > > > (register number
>> > > > > > > > > 114059) are
>> > > > > > > > > > authorised
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> regulated
>> > > > > > > > > > > >> >
>> >>
>> > by
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Financial
>> > > > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > --
>> > > > > > > > > > > >> >
>> >>
>> > > > Nacho - Ignacio
>> > > > > > Solis -
>> > > > > > > > isolis@igso.net
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > The
>> > > information
>> > > > contained in
>> > > > > > this email is
>> > > > > > > > strictly
>> > > > > > > > > > confidential and
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> use of
>> > the
>> > > > addressee only,
>> > > > > > unless
>> > > > > > > otherwise
>> > > > > > > > > indicated. If
>> > > > > > > > > > you are
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > the
>> intended
>> > > > recipient, please
>> > > > > > do not read,
>> > > > > > > > copy, use
>> > > > > > > > > or
>> > > > > > > > > > disclose to
>> > > > > > > > > > > >> others
>> > > > > > > > > > > >> > this
>> message
>> > > or
>> > > > any
>> > > > > attachment.
>> > > > > > Please also
>> > > > > > > > notify
>> > > > > > > > > the sender by
>> > > > > > > > > > > >> replying
>> > > > > > > > > > > >> > to
>> this
>> > email
>> > > > or by telephone
>> > > > > > (+44(020 7896
>> > > > > > > > 0011) and
>> > > > > > > > > then
>> > > > > > > > > > delete the
>> > > > > > > > > > > >> email
>> > > > > > > > > > > >> > and
>> any
>> > copies
>> > > > of it.
>> > > > > Opinions,
>> > > > > > conclusion
>> > > > > > > > (etc) that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > understood
>> > > > > > > > > as
>> > > > > > > > > > neither given
>> > > > > > > > > > > >> nor
>> > > > > > > > > > > >> >
>> endorsed by
>> > > it.
>> > > > IG is a
>> > > > > trading
>> > > > > > name of IG
>> > > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >> >
>> registered
>> > in
>> > > > England and
>> > > > > > Wales, company
>> > > > > > > number
>> > > > > > > > > 04008957) and
>> > > > > > > > > > IG Index
>> > > > > > > > > > > >> >
>> Limited (a
>> > > > company registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > >> >
>> 01190902).
>> > > > Registered address
>> > > > > > at Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25
>> > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > >> >
>> London EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and
>> > > > > > > > > > > IG
>> > > > > > > > > > > >> > Index
>> > Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> Financial
>> > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> The
>> > information
>> > > > contained in
>> > > > > this
>> > > > > > email is
>> > > > > > > > strictly
>> > > > > > > > > confidential
>> > > > > > > > > > and for
>> > > > > > > > > > > >> the use
>> of the
>> > > > addressee only,
>> > > > > > unless otherwise
>> > > > > > > > > indicated. If you
>> > > > > > > > > > are
>> > > > > > > > > > > not
>> > > > > > > > > > > >> the
>> intended
>> > > > recipient, please
>> > > > > do
>> > > > > > not read,
>> > > > > > > copy,
>> > > > > > > > use
>> > > > > > > > > or disclose
>> > > > > > > > > > to
>> > > > > > > > > > > others
>> > > > > > > > > > > >> this
>> message
>> > or
>> > > > any attachment.
>> > > > > > Please also
>> > > > > > > > notify the
>> > > > > > > > > sender by
>> > > > > > > > > > > replying
>> > > > > > > > > > > >> to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the
>> > > > > > > > > > > email
>> > > > > > > > > > > >> and any
>> copies
>> > > of
>> > > > it. Opinions,
>> > > > > > conclusion (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > the
>> > > > > > > > > > > >> official
>> > > business
>> > > > of this
>> > > > > company
>> > > > > > shall be
>> > > > > > > > understood
>> > > > > > > > > as neither
>> > > > > > > > > > given
>> > > > > > > > > > > nor
>> > > > > > > > > > > >>
>> endorsed by
>> > it.
>> > > > IG is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >>
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG
>> > > > > > > > > > Index
>> > > > > > > > > > > >> Limited
>> (a
>> > > > company registered in
>> > > > > > England and
>> > > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 01190902).
>> > > > Registered address at
>> > > > > > Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25 Dowgate
>> > > > > > > > > > Hill,
>> > > > > > > > > > > >> London
>> EC4R
>> > 2YA.
>> > > > Both IG Markets
>> > > > > > Limited
>> > > > > > > (register
>> > > > > > > > > number 195355)
>> > > > > > > > > > and IG
>> > > > > > > > > > > >> Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> Financial
>> > > Conduct
>> > > > Authority.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > The
>> information
>> > > > contained in this
>> > > > > > email is strictly
>> > > > > > > > > confidential and
>> > > > > > > > > > for the use of
>> the
>> > > > addressee only, unless
>> > > > > > otherwise
>> > > > > > > > indicated.
>> > > > > > > > > If you are
>> > > > > > > > > > not the intended
>> > > > recipient, please do not
>> > > > > > read, copy, use
>> > > > > > > > or
>> > > > > > > > > disclose to
>> > > > > > > > > > others this
>> message or
>> > > any
>> > > > attachment.
>> > > > > > Please also notify
>> > > > > > > > the
>> > > > > > > > > sender by
>> > > > > > > > > > replying to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the email and any
>> > copies
>> > > > of it. Opinions,
>> > > > > > conclusion
>> > > > > > > (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to the
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > > understood as
>> > > > > > > > > > neither given nor
>> > > endorsed
>> > > > by it. IG is a
>> > > > > > trading name of
>> > > > > > > > IG
>> > > > > > > > > Markets
>> > > > > > > > > > Limited (a
>> company
>> > > > registered in England
>> > > > > > and Wales,
>> > > > > > > company
>> > > > > > > > > number
>> > > > > > > > > > 04008957) and IG
>> Index
>> > > > Limited (a company
>> > > > > > registered in
>> > > > > > > > England
>> > > > > > > > > and Wales,
>> > > > > > > > > > company number
>> > 01190902).
>> > > > Registered
>> > > > > > address at Cannon
>> > > > > > > > Bridge
>> > > > > > > > > House, 25
>> > > > > > > > > > Dowgate Hill,
>> London
>> > EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and IG
>> Index
>> > > > Limited (register
>> > > > > > number 114059) are
>> > > > > > > > > authorised and
>> > > > > > > > > > regulated by the
>> > > Financial
>> > > > Conduct
>> > > > > > Authority.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The information
>> contained in
>> > > > this email is
>> > > > > > strictly
>> > > > > > > confidential
>> > > > > > > > and
>> > > > > > > > > for the use of the
>> addressee
>> > only,
>> > > > unless otherwise
>> > > > > > indicated. If
>> > > > > > > > you are
>> > > > > > > > > not the intended recipient,
>> > please
>> > > > do not read,
>> > > > > > copy, use or
>> > > > > > > > disclose to
>> > > > > > > > > others this message or any
>> > > > attachment. Please also
>> > > > > > notify the
>> > > > > > > sender
>> > > > > > > > by
>> > > > > > > > > replying to this email or
>> by
>> > > > telephone (+44(020
>> > > > > 7896
>> > > > > > 0011) and then
>> > > > > > > > delete
>> > > > > > > > > the email and any copies
>> of it.
>> > > > Opinions,
>> > > > > conclusion
>> > > > > > (etc) that do
>> > > > > > > > not
>> > > > > > > > > relate to the official
>> business
>> > of
>> > > > this company
>> > > > > > shall be understood
>> > > > > > > > as
>> > > > > > > > > neither given nor endorsed
>> by it.
>> > > IG
>> > > > is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a company
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG Index
>> Limited (a
>> > > > company
>> > > > > registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company number 01190902).
>> > > Registered
>> > > > address at
>> > > > > > Cannon Bridge
>> > > > > > > House,
>> > > > > > > > 25
>> > > > > > > > > Dowgate Hill, London EC4R
>> 2YA.
>> > Both
>> > > > IG Markets
>> > > > > > Limited (register
>> > > > > > > > number
>> > > > > > > > > 195355) and IG Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > authorised
>> > > > > > > > and
>> > > > > > > > > regulated by the Financial
>> > Conduct
>> > > > Authority.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > The information contained in
>> this email
>> > > is
>> > > > strictly
>> > > > > > confidential and for
>> > > > > > > > the use of the addressee only,
>> unless
>> > > > otherwise
>> > > > > indicated.
>> > > > > > If you are not
>> > > > > > > > the intended recipient, please
>> do not
>> > > > read, copy, use or
>> > > > > > disclose to
>> > > > > > > others
>> > > > > > > > this message or any attachment.
>> Please
>> > > > also notify the
>> > > > > > sender by replying
>> > > > > > > > to this email or by telephone
>> (+44(020
>> > > > 7896 0011) and
>> > > > > then
>> > > > > > delete the
>> > > > > > > email
>> > > > > > > > and any copies of it. Opinions,
>> > > conclusion
>> > > > (etc) that do
>> > > > > > not relate to
>> > > > > > > the
>> > > > > > > > official business of this
>> company shall
>> > > be
>> > > > understood as
>> > > > > > neither given
>> > > > > > > nor
>> > > > > > > > endorsed by it. IG is a trading
>> name of
>> > > IG
>> > > > Markets
>> > > > > Limited
>> > > > > > (a company
>> > > > > > > > registered in England and Wales,
>> > company
>> > > > number 04008957)
>> > > > > > and IG Index
>> > > > > > > > Limited (a company registered in
>> > England
>> > > > and Wales,
>> > > > > > company number
>> > > > > > > > 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill,
>> > > > > > > > London EC4R 2YA. Both IG Markets
>> > Limited
>> > > > (register number
>> > > > > > 195355) and IG
>> > > > > > > > Index Limited (register number
>> 114059)
>> > > are
>> > > > authorised and
>> > > > > > regulated by
>> > > > > > > the
>> > > > > > > > Financial Conduct Authority.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > The information contained in this email is
>> strictly
>> > > > confidential and
>> > > > > > for the use of the addressee only, unless
>> otherwise
>> > > > indicated. If you are
>> > > > > > not the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or disclose to
>> > > > > > others this message or any attachment. Please
>> also
>> > notify
>> > > > the sender by
>> > > > > > replying to this email or by telephone (+44(020
>> 7896
>> > > 0011)
>> > > > and then
>> > > > > delete
>> > > > > > the email and any copies of it. Opinions,
>> conclusion
>> > > (etc)
>> > > > that do not
>> > > > > > relate to the official business of this company
>> shall
>> > be
>> > > > understood as
>> > > > > > neither given nor endorsed by it. IG is a
>> trading name
>> > of
>> > > > IG Markets
>> > > > > > Limited (a company registered in England and
>> Wales,
>> > > > company number
>> > > > > > 04008957) and IG Index Limited (a company
>> registered in
>> > > > England and
>> > > > > Wales,
>> > > > > > company number 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register number
>> > > > > > 195355) and IG Index Limited (register number
>> 114059)
>> > are
>> > > > authorised and
>> > > > > > regulated by the Financial Conduct Authority.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential
>> and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If
>> you are
>> > not
>> > > > the intended recipient, please do not read, copy, use or
>> disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then
>> delete the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company
>> number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number
>> 195355) and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and
>> regulated by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > >
>> >
>>
>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
--
Nacho - Ignacio Solis - isolis@igso.net
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ignacio Solis <is...@igso.net>.
Short answer:
Headers at the end are fine, this proposal looks ok.
Longer answer:
At LinkedIn we considered headers at the end multiple times. We even
had a backwards compatible headers-in-V design with headers at the
end.
The main reason headers come at the beginning of messaging systems is
in case they affect the rest of the message. This is either because
of parsing differences (a header might indicate how to parse the
remainder of the message), or performance optimizations (a header
might indicate that you can discard the message and can stop parsing).
I believe that neither of these cases are common in Kafka (at this
time) so having the headers are the end shouldn't be a big issue.
From a performance perspective we may incur an extra page cache miss
but at the rate at which kafka does messaging it won't bottleneck
things.
Headers at the end have one potential advantage, that in simple
circumstances, naive systems can append headers to a message without
doing a lot of memory manipulation. Normally not that big of a deal
since we could always use scatter/gather methods if needed, but maybe
some simple systems will benefit from that. ... Not to mention most
Java programmers might not pay attention to these type of performance
issues.
Anyway...
Overall, there are many things that I would have done differently in
the proposal (int keys, total header length, no VarInts, etc. ), but
the cost of these is small in comparison to the benefit we get from
having headers at all.
If we are all ok with this design then I would be the first to +1 the format.
Nacho
Side note: Shouldn't headers at the end be called footers? We could
always call it metadata :-).
On Tue, Feb 21, 2017 at 11:14 AM, Michael Pearce <Mi...@ig.com> wrote:
> I think I'd like some opinions from some of the active chaps at LinkedIn so it's not just a two way decision (or any others for that matter).
>
> Any opinions on the below or no real concerns / indifferent by anyone with this change?
>
> Cheers
> Mike
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Tuesday, February 21, 2017 6:59:38 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Jun's suggestion gives us the best of both worlds. Because the headers are
> the last field in the record and you have the total size of the record,
> then you can easily get the size of headers in bytes. So they can be copied
> without parsing and we can do the lazy initialization without an extra cost.
>
> The message set notion has changed in KIP-98 so that the message set can
> have its own fields. See here for more detail:
> https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
> What we're proposing to change here is the schema for the message (aka
> record) itself. We'll update that document depending on the outcome of the
> discussion here.
>
> -Jason
>
>
> On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> This works whilst no other feature needs similar. I prefer to keep it
>> contained/encapsulated it then allows the bytes of the headers to also be
>> copied transferred as one without any parsing what so ever, but if you
>> really feel strongly on the below ill go with it, just want to flag this
>> concern.
>>
>> Btw, the below notation seems to be different to the notation used in
>> previous kips like kip-32 which I took the notation from, where the length
>> of the following bytes for byte arrays is clearly defined. So im 100% on
>> what you propose could you define it in this original styled notation?
>>
>>
>> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>>
>> Hey Jun, that's a good thought. Maybe then the full schema could look
>> like
>> this:
>>
>> Record =>
>> Length => uintVar
>> Attributes => int8
>> TimestampDelta => intVar
>> OffsetDelta => uintVar
>> Key => bytes
>> Value => bytes
>> Headers => [HeaderKey HeaderValue]
>> HeaderKey => string
>> HeaderValue => bytes
>>
>> (where the bytes and string type use a variable length size field)
>>
>> Does that work for you, Michael?
>>
>> -Jason
>>
>> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>> > Regarding the overhead of skipping the headers in the message.
>> Another way
>> > could be to add the headers at the end of the message after the
>> value part.
>> > This way, there is no need to skip the headers unless you really
>> need it.
>> > This will allow us to represent the headers explicitly as an array of
>> > key/value pairs without the skipping overhead when headers are not
>> needed.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
>> >
>> > wrote:
>> >
>> > > Michael,
>> > >
>> > > Here's a link to the branch with the message format changes:
>> > > https://github.com/confluentinc/kafka/tree/
>> exactly-once-message-format.
>> > > The
>> > > class to modify to add headers is EosLogRecord (name will
>> definitely
>> > change
>> > > prior to merging in trunk). This branch uses attributes to
>> indicate null
>> > > key and value. I really don't have a strong preference for or
>> against
>> > this
>> > > as long as we're consistent. I do feel it's kind of a pity to need
>> the
>> > > signed version of the varint just to be able to indicate null. For
>> > headers,
>> > > it's an easier call because we don't actually need to represent
>> null
>> > > headers (using size zero will work fine).
>> > >
>> > > -Jason
>> > >
>> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > We can make it, implement Iterable<Header>, no concerns on this.
>> > > >
>> > > > Ive noted this on the KIP.
>> > > >
>> > > >
>> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
>> wrote:
>> > > >
>> > > > Btw Jason, can you share the branch with the VarInts
>> helper/util
>> > > class
>> > > > that KIP-98 will be introducing and also will have the current
>> modified
>> > > > (key and value lengths) message, so we can look at, thanks.
>> > > >
>> > > > On 20/02/2017, 01:11, "Michael Pearce" <
>> Michael.Pearce@ig.com>
>> > > wrote:
>> > > >
>> > > > RE “ProducerRecord and ConsumerRecord are immutable
>> > currently”. I
>> > > > beg to differ on this, if I set a byte[] as my value of my
>> record, I
>> > can
>> > > > later call value() and then alter the bytes in the array. As
>> such it is
>> > > > very much mutable.
>> > > >
>> > > > Headers also will anyhow by nature be mutable on
>> produce, as
>> > you
>> > > > add to them.
>> > > >
>> > > >
>> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
>> jason@confluent.io>
>> > > > wrote:
>> > > >
>> > > > Re; having a dedicated Headers object. I think it
>> makes
>> > > sense.
>> > > > Maybe it
>> > > > could implement Iterable<Header>?
>> > > >
>> > > > One thing I'm not too sure about is mutability. Both
>> the
>> > > > ProducerRecord and
>> > > > ConsumerRecord are immutable currently. It would be
>> nice to
>> > > > keep it that
>> > > > way. Perhaps Headers could itself be immutable and
>> passed
>> > > > through the
>> > > > constructors?
>> > > >
>> > > > Just going through how this is, I don’t believe
>> this would
>> > > > provide much
>> > > > > saving. If issue trying to resolve is around
>> > object/garbage
>> > > > creation of
>> > > > > Map.Entry objects.
>> > > >
>> > > >
>> > > > It's definitely better because we only allocate the
>> space
>> > we
>> > > > will use, but
>> > > > I assume most of the actual memory cost is in the
>> header
>> > data
>> > > > itself which
>> > > > we can't do much about. Maybe we can start with the
>> simple
>> > > > array
>> > > > representation and optimize from there after we find
>> it's
>> > > > worthwhile to do
>> > > > so? If we use the Headers object, we can change this
>> later
>> > > > without breaking
>> > > > the API.
>> > > >
>> > > > -Jason
>> > > >
>> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
>> > > > radai.rosenblatt@gmail.com> wrote:
>> > > >
>> > > > > "(By the way, doesn't it feel a bit odd that we
>> seem to
>> > be
>> > > > designing a
>> > > > > feature which is optimized for people not using?)"
>> > > > >
>> > > > > very. this (i claim :-) ) is the result of very
>> intense
>> > > > opposition to the
>> > > > > usefulness of the feature early on, and not a real
>> design
>> > > > goal
>> > > > >
>> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Whilst if having an array of a Header[] would
>> mean
>> > > getting
>> > > > this through,
>> > > > > > im happy changing this, as already done.
>> > > > > >
>> > > > > > Just going through how this is, I don’t believe
>> this
>> > > > would provide much
>> > > > > > saving. If issue trying to resolve is around
>> > > > object/garbage creation of
>> > > > > > Map.Entry objects.
>> > > > > >
>> > > > > > All you’re doing here is replacing the equiv of a
>> > > > HashMap.Node
>> > > > > (Map.Entry)
>> > > > > > which simply holds reference to key and value
>> objects
>> > > with
>> > > > a custom
>> > > > > variant.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com> wrote:
>> > > > > >
>> > > > > > On point 1 & 2 Ive updated KIP to show
>> varints (and
>> > > > removed the bit
>> > > > > > flag). (on the assumption KIP 98 is getting the
>> > agreement
>> > > > the protocol is
>> > > > > > moving from int32 to varInts as standard)
>> > > > > >
>> > > > > > On point 3 ive updated to use an array of
>> Header
>> > > > class, instead of a
>> > > > > > MultiMap in the Headers class object
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
>> > > > Michael.Pearce@ig.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > On points 1 and 2 I agree.
>> > > > > >
>> > > > > > This also affects kip-98, I should
>> expect this
>> > > > resolved before
>> > > > > > that vote also passes. If it is accepted there
>> (I’m
>> > > > assuming this is
>> > > > > > getting discussed on that KIP? As you’re driving
>> the
>> > move
>> > > > to VarInts), I
>> > > > > am
>> > > > > > happy to make this KIP will simply follow suit to
>> > > whatever
>> > > > is agreed in
>> > > > > > KIP-98.
>> > > > > >
>> > > > > > On 3) Agreed this is a simpler form, as
>> long as
>> > > no
>> > > > one is
>> > > > > > expecting hashmap lookup performance ( O(1) ), I
>> still
>> > > > would prefer an
>> > > > > > encapsulated class, so if we find that holding
>> it as an
>> > > > array in future
>> > > > > is
>> > > > > > causing some perf issues, the internals are not
>> exposed
>> > > to
>> > > > end users,
>> > > > > > allowing the internal structure to move to a map.
>> > > > > >
>> > > > > > Can we compromise on?
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Headers headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Headers {
>> > > > > > Header[] headers;
>> > > > > >
>> > > > > > add(String key, byte[] value)
>> > > > > > Collection<byte[]> get(String key)
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
>> > > > jason@confluent.io>
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > >
>> > > > > >
>> > > > > > This is exactly what I have been
>> trying to
>> > > get
>> > > > at. The use
>> > > > > > cases documented
>> > > > > > here are middleware:
>> > > > > > https://cwiki.apache.org/
>> > > > confluence/display/KAFKA/A+
>> > > > > > Case+for+Kafka+Headers.
>> > > > > > If headers are intended for the
>> application
>> > > > space as well,
>> > > > > the
>> > > > > > document
>> > > > > > should be updated accordingly so
>> that it is
>> > > an
>> > > > explicit
>> > > > > design
>> > > > > > goal and not
>> > > > > > an unfortunate design byproduct.
>> There may
>> > be
>> > > > some
>> > > > > > disagreement on whether
>> > > > > > it _should_ be a design goal, but it
>> may
>> > not
>> > > > matter much
>> > > > > since
>> > > > > > the current
>> > > > > > interceptor API is probably
>> insufficient
>> > for
>> > > > middleware
>> > > > > > applications (I
>> > > > > > haven't thought of a way to do this
>> that
>> > > isn't
>> > > > cumbersome).
>> > > > > >
>> > > > > > That aside, the three unresolved
>> points for
>> > > me
>> > > > are the
>> > > > > > following:
>> > > > > >
>> > > > > > 1. The use of varints. As a one-off
>> for use
>> > > > only with record
>> > > > > > headers, the
>> > > > > > case was weak, but if we are using
>> them
>> > > > throughout the
>> > > > > message
>> > > > > > format, then
>> > > > > > we should do so here as well. The
>> > additional
>> > > > complexity is
>> > > > > > minimal and
>> > > > > > early performance testing fully
>> justifies
>> > > > their use.
>> > > > > >
>> > > > > > 2. If varints are used, the case for
>> using
>> > > > attributes to
>> > > > > > indicate null
>> > > > > > headers also becomes weak. We only
>> add one
>> > > > additional byte in
>> > > > > > each message
>> > > > > > if there are no headers. Whatever
>> the case,
>> > > > let us be
>> > > > > > consistent with how
>> > > > > > we treat null keys and values.
>> > > > > >
>> > > > > > 3. We have apparently agreed that the
>> > broker
>> > > > will validate
>> > > > > > headers (please
>> > > > > > add this to the KIP). That being the
>> case,
>> > I
>> > > > would prefer to
>> > > > > > use Kafka's
>> > > > > > conventional format for arrays. The
>> > downside
>> > > > is that it takes
>> > > > > > more work to
>> > > > > > skip over the headers on the
>> consumer,
>> > though
>> > > > it's unclear if
>> > > > > > the cost
>> > > > > > would matter in practice. Some
>> concern was
>> > > > previously
>> > > > > > expressed about the
>> > > > > > allocation of maps. An alternative
>> would be
>> > > to
>> > > > use arrays,
>> > > > > i.e.
>> > > > > >
>> > > > > > class ConsumerRecord<K, V> {
>> > > > > > K key;
>> > > > > > V value;
>> > > > > > Header[] headers;
>> > > > > > }
>> > > > > >
>> > > > > > class Header {
>> > > > > > String key;
>> > > > > > byte[] value;
>> > > > > > }
>> > > > > >
>> > > > > > This would work nicely with the
>> > conventional
>> > > > array format and
>> > > > > > my guess is
>> > > > > > it would obviate the need do any lazy
>> > > > initialization. If we
>> > > > > > use the map as
>> > > > > > is currently documented, then it is
>> > possible
>> > > > with either
>> > > > > > representation to
>> > > > > > slice the headers and initialize them
>> > lazily.
>> > > > Either way, it
>> > > > > > might be a
>> > > > > > good idea to use a separate object to
>> > > > represent the headers
>> > > > > in
>> > > > > > case we need
>> > > > > > to modify it in the future in some
>> way.
>> > > > > >
>> > > > > > (By the way, doesn't it feel a bit
>> odd that
>> > > we
>> > > > seem to be
>> > > > > > designing a
>> > > > > > feature which is optimized for
>> people not
>> > > > using?)
>> > > > > >
>> > > > > >
>> > > > > > If we can resolve these points, then
>> at
>> > least
>> > > > you will get my
>> > > > > > vote.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Jason
>> > > > > >
>> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
>> radai <
>> > > > > > radai.rosenblatt@gmail.com> wrote:
>> > > > > >
>> > > > > > > headers dont "leak" into
>> application
>> > code.
>> > > > they are useful
>> > > > > > to application
>> > > > > > > code as well.
>> > > > > > > IIUC samze currently has headers
>> "in-V"
>> > and
>> > > > would just
>> > > > > > switch over to kafka
>> > > > > > > headers if they exist.
>> > > > > > > im sure plenty of other users of
>> kafka
>> > > would
>> > > > have a use for
>> > > > > > headers.
>> > > > > > > im pretty sure use cases exist
>> around
>> > > > shuffling data
>> > > > > > into/out-of kafka
>> > > > > > > (kafka connect or equivalent) where
>> > > metadata
>> > > > from one end
>> > > > > > could copied over
>> > > > > > > to the other (S3, for example uses
>> http
>> > > > headers for
>> > > > > > user-accessible
>> > > > > > > metadata). it will be kafka client
>> code
>> > > > getting/setting
>> > > > > > those headers. not
>> > > > > > > an interceptor.
>> > > > > > >
>> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
>> Michael
>> > > > Pearce <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > For APM single event tracing,
>> need
>> > access
>> > > > to the header
>> > > > > at
>> > > > > > the point of
>> > > > > > > > processing on the processing
>> thread.
>> > > > > > > >
>> > > > > > > > As such interceptors will not
>> work/be
>> > > > suitable for these,
>> > > > > > due to the fact
>> > > > > > > > they act on the ConsumerRecords
>> as a
>> > > > batch, before the
>> > > > > > handling thread
>> > > > > > > can
>> > > > > > > > split out and process per
>> message which
>> > > is
>> > > > the point
>> > > > > these
>> > > > > > tools will
>> > > > > > > need
>> > > > > > > > to continue to transaction
>> tracing.
>> > > > > > > >
>> > > > > > > > Like wise tools and infra pieces
>> will
>> > > need
>> > > > access to the
>> > > > > > message outside
>> > > > > > > > the interceptor realm.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 17/02/2017, 21:26, "Jason
>> > Gustafson" <
>> > > > > > jason@confluent.io> wrote:
>> > > > > > > >
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > So the broker will parse the
>> > headers
>> > > > and validate
>> > > > > > them. Good. The
>> > > > > > > only
>> > > > > > > > reason remaining that I can
>> see to
>> > > > leave the headers
>> > > > > > as a byte array
>> > > > > > > > is to
>> > > > > > > > make it easier for the
>> client to
>> > skip
>> > > > past them. Are
>> > > > > > we sure this is
>> > > > > > > > not
>> > > > > > > > premature optimization? Are
>> there
>> > any
>> > > > performance
>> > > > > > results which show
>> > > > > > > > that
>> > > > > > > > this is worthwhile?
>> > > > > > > >
>> > > > > > > > What’s the issue with
>> exposing a
>> > > > method getHeaders on
>> > > > > > the
>> > > > > > > > producer/consumer
>> > > > > > > > > record? It doesn’t break
>> > anything.
>> > > > We don’t need
>> > > > > any
>> > > > > > special
>> > > > > > > version.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > See my previous explanation.
>> What I
>> > > am
>> > > > trying to
>> > > > > > resist is the
>> > > > > > > headers
>> > > > > > > > becoming a general
>> > application-level
>> > > > facility. The
>> > > > > > primary use case
>> > > > > > > as
>> > > > > > > > far
>> > > > > > > > as I can tell is middleware,
>> which
>> > is
>> > > > the use case
>> > > > > > that the
>> > > > > > > > interceptors
>> > > > > > > > are providing.
>> > > > > > > >
>> > > > > > > > Current batch consumer model
>> and
>> > > > consumer
>> > > > > interceptors
>> > > > > > don’t work
>> > > > > > > where
>> > > > > > > > > headers need to be acted
>> on at
>> > per
>> > > > message level at
>> > > > > > time of
>> > > > > > > > processing,
>> > > > > > > > > very case is APM (the core
>> one),
>> > > > where the header
>> > > > > > value is used to
>> > > > > > > > continue
>> > > > > > > > > tracing.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > I still don't understand the
>> point
>> > > > about batching.
>> > > > > The
>> > > > > > consumer
>> > > > > > > > records are
>> > > > > > > > exposed as a batch in the
>> consumer
>> > > > interceptor, but
>> > > > > > you can still
>> > > > > > > > iterate
>> > > > > > > > them individually. It is no
>> > different
>> > > > for the
>> > > > > consumer
>> > > > > > API itself.
>> > > > > > > >
>> > > > > > > > -Jason
>> > > > > > > >
>> > > > > > > > On Fri, Feb 17, 2017 at
>> 12:48 PM,
>> > > > Michael Pearce <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Re:
>> > > > > > > > >
>> > > > > > > > > “ The point about
>> creation of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a slice
>> of
>> > bytes
>> > > > until the time
>> > > > > > it is accessed.”
>> > > > > > > > >
>> > > > > > > > > That’s exactly what we’re
>> doing
>> > the
>> > > > headers are a
>> > > > > > slice of bytes,
>> > > > > > > > which
>> > > > > > > > > then gets parsed later if
>> needed,
>> > > or
>> > > > can be parsed
>> > > > > > right away, the
>> > > > > > > > headers
>> > > > > > > > > is part of the protocol,
>> so can
>> > > > still be validated
>> > > > > > if wanted.
>> > > > > > > > >
>> > > > > > > > > If you had a header count
>> then
>> > you
>> > > > would have to go
>> > > > > > through each
>> > > > > > > > header
>> > > > > > > > > key and value length value
>> to
>> > work
>> > > > out how much to
>> > > > > > skip to get to
>> > > > > > > > say the
>> > > > > > > > > value or any future
>> component in
>> > > the
>> > > > message after
>> > > > > > the headers.
>> > > > > > > > Having it
>> > > > > > > > > as a byte[] with length
>> value
>> > makes
>> > > > this a lot
>> > > > > > easier to skip.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:37,
>> "Michael
>> > > > Pearce" <
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > What’s the issue with
>> > exposing
>> > > a
>> > > > method
>> > > > > > getHeaders on the
>> > > > > > > > > producer/consumer record?
>> It
>> > > doesn’t
>> > > > break
>> > > > > anything.
>> > > > > > We don’t need
>> > > > > > > > any
>> > > > > > > > > special version.
>> > > > > > > > >
>> > > > > > > > > Current batch consumer
>> model
>> > > and
>> > > > consumer
>> > > > > > interceptors don’t
>> > > > > > > work
>> > > > > > > > > where headers need to be
>> acted on
>> > > at
>> > > > per message
>> > > > > > level at time of
>> > > > > > > > > processing, very case is
>> APM (the
>> > > > core one), where
>> > > > > > the header value
>> > > > > > > > is used
>> > > > > > > > > to continue tracing.
>> JMS/HTTP etc
>> > > > all expose these,
>> > > > > > without issues.
>> > > > > > > > I would
>> > > > > > > > > NOT want to lock this down
>> to
>> > only
>> > > > be usable
>> > > > > > accessible via
>> > > > > > > > interceptors,
>> > > > > > > > > as we’d fail on one of the
>> main
>> > > > goals.
>> > > > > > > > >
>> > > > > > > > > Regards
>> > > > > > > > > Mike
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 17/02/2017, 20:21,
>> "Jason
>> > > > Gustafson" <
>> > > > > > jason@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > The point about
>> creation
>> > of
>> > > > maps seems
>> > > > > > orthogonal. We can
>> > > > > > > > still
>> > > > > > > > > represent
>> > > > > > > > > the headers as a
>> slice of
>> > > > bytes until the
>> > > > > > time it is
>> > > > > > > > accessed.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > > Yes exactly we
>> have
>> > > access
>> > > > to the records
>> > > > > > thus why the
>> > > > > > > > header
>> > > > > > > > > should be
>> > > > > > > > > > accessible via
>> it and
>> > not
>> > > > hidden for only
>> > > > > > interceptors to
>> > > > > > > > access.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > As explained
>> above, the
>> > > > point is to make
>> > > > > the
>> > > > > > intended usage
>> > > > > > > > clear.
>> > > > > > > > > Applications should
>> > > continue
>> > > > to rely on the
>> > > > > > key/value
>> > > > > > > fields
>> > > > > > > > to
>> > > > > > > > > serialize
>> > > > > > > > > their own headers,
>> and it
>> > > > would be more
>> > > > > > ideal if we can
>> > > > > > > avoid
>> > > > > > > > > leaking
>> > > > > > > > > third-party
>> headers into
>> > > > applications. This
>> > > > > > is difficult to
>> > > > > > > > do
>> > > > > > > > > with the
>> > > > > > > > > current
>> interceptors
>> > > because
>> > > > they share the
>> > > > > > record objects
>> > > > > > > > with
>> > > > > > > > > the common
>> > > > > > > > > API. What I had in
>> mind
>> > is
>> > > > something like
>> > > > > an
>> > > > > > extension of
>> > > > > > > the
>> > > > > > > > > current
>> > > > > > > > > interceptors which
>> > exposed
>> > > a
>> > > > different
>> > > > > > object (e.g.
>> > > > > > > > > `RecordAndHeaders`).
>> > > > > > > > > The challenge is
>> for
>> > > MM-like
>> > > > use cases. Let
>> > > > > > me see if I can
>> > > > > > > > come
>> > > > > > > > > up with a
>> > > > > > > > > concrete proposal
>> for
>> > that
>> > > > problem.
>> > > > > > > > >
>> > > > > > > > > -Jason
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Feb 17,
>> 2017 at
>> > > > 11:55 AM, Michael
>> > > > > > Pearce <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > I am happy to
>> move the
>> > > > definition of the
>> > > > > > header into the
>> > > > > > > > message
>> > > > > > > > > body, but
>> > > > > > > > > > would cause us
>> not to
>> > > lazy
>> > > > > > initialise/parse the headers,
>> > > > > > > as
>> > > > > > > > > obviously, we
>> > > > > > > > > > would have to
>> traverse
>> > > > these reading the
>> > > > > > message.
>> > > > > > > > > >
>> > > > > > > > > > This was
>> actually one
>> > of
>> > > > Jay’s requests:
>> > > > > > > > > >
>> > > > > > > > > > “ 2. I think
>> we
>> > should
>> > > > think about
>> > > > > > creating the Map
>> > > > > > > > lazily to
>> > > > > > > > > avoid
>> > > > > > > > > > parsing out
>> all the
>> > > > headers into
>> > > > > > little objects.
>> > > > > > > > HashMaps
>> > > > > > > > > themselves
>> > > > > > > > > > are
>> > > > > > > > > > kind of
>> expensive
>> > and
>> > > > the consumer is
>> > > > > > very perf
>> > > > > > > > sensitive so
>> > > > > > > > > and making
>> > > > > > > > > > gazillions of
>> > > hashmaps
>> > > > that may or
>> > > > > may
>> > > > > > not get used
>> > > > > > > is
>> > > > > > > > > probably a bad
>> > > > > > > > > > idea.”
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 17/02/2017,
>> 19:44,
>> > > > "Michael Pearce" <
>> > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Yes exactly
>> we have
>> > > > access to the
>> > > > > > records thus why
>> > > > > > > the
>> > > > > > > > > header should
>> > > > > > > > > > be accessible
>> via it
>> > and
>> > > > not hidden for
>> > > > > > only interceptors
>> > > > > > > > to
>> > > > > > > > > access.
>> > > > > > > > > >
>> > > > > > > > > > Sent using
>> OWA for
>> > > > iPhone
>> > > > > > > > > >
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > From: Magnus
>> > > Edenhill <
>> > > > > > magnus@edenhill.se>
>> > > > > > > > > > Sent: Friday,
>> > > February
>> > > > 17, 2017
>> > > > > > 7:34:49 PM
>> > > > > > > > > > To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > Subject: Re:
>> > > [DISCUSS]
>> > > > KIP-82 - Add
>> > > > > > Record Headers
>> > > > > > > > > >
>> > > > > > > > > > Big +1 on
>> VarInts.
>> > > > > > > > > > CPUs are
>> fast,
>> > memory
>> > > > is slow.
>> > > > > > > > > >
>> > > > > > > > > > I agree with
>> Jason
>> > > > that we'll want to
>> > > > > > continue
>> > > > > > > > verifying
>> > > > > > > > > messages,
>> > > > > > > > > > including
>> their
>> > > > headers, so while I
>> > > > > > appreciate the
>> > > > > > > > idea of
>> > > > > > > > > the opaque
>> > > > > > > > > > header blob
>> it
>> > won't
>> > > > be useful in
>> > > > > > practice.
>> > > > > > > > > >
>> > > > > > > > > > /Magnus
>> > > > > > > > > >
>> > > > > > > > > > 2017-02-17
>> 10:41
>> > > > GMT-08:00 Jason
>> > > > > > Gustafson <
>> > > > > > > > > jason@confluent.io>:
>> > > > > > > > > >
>> > > > > > > > > > > Sorry, my
>> > mistake.
>> > > > The consumer
>> > > > > > interceptor is per
>> > > > > > > > batch,
>> > > > > > > > > though I'm
>> > > > > > > > > > not
>> > > > > > > > > > > sure
>> that's an
>> > > > actual limitation
>> > > > > > since you still
>> > > > > > > have
>> > > > > > > > > access to the
>> > > > > > > > > > > individual
>> > records.
>> > > > > > > > > > >
>> > > > > > > > > > > -Jason
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri,
>> Feb 17,
>> > > 2017
>> > > > at 10:39 AM,
>> > > > > > Jason Gustafson <
>> > > > > > > > > >
>> jason@confluent.io>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Re
>> headers as
>> > > byte
>> > > > array and
>> > > > > > future use by
>> > > > > > > broker.
>> > > > > > > > This
>> > > > > > > > > doesn't
>> > > > > > > > > > take away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Yeah, I
>> didn't
>> > > say
>> > > > it was
>> > > > > > difficult, only
>> > > > > > > awkward.
>> > > > > > > > You
>> > > > > > > > > wouldn't
>> > > > > > > > > > write the
>> > > > > > > > > > > > schema
>> that way
>> > > if
>> > > > you were
>> > > > > > planning to use it on
>> > > > > > > > the
>> > > > > > > > > brokers from
>> > > > > > > > > > the
>> > > > > > > > > > > >
>> beginning. Note
>> > > > also that one of
>> > > > > > the benefits of
>> > > > > > > > letting
>> > > > > > > > > the broker
>> > > > > > > > > > > >
>> understand
>> > > headers
>> > > > is that it can
>> > > > > > validate that
>> > > > > > > > they are
>> > > > > > > > > properly
>> > > > > > > > > > > >
>> formatted. If
>> > > cost
>> > > > is the only
>> > > > > > concern, we should
>> > > > > > > > > confirm its
>> > > > > > > > > > impact
>> > > > > > > > > > > > through
>> > > > performance testing.
>> > > > > > > > > > > >
>> > > > > > > > > > > > One of
>> the key
>> > > use
>> > > > cases requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > I'm not
>> sure I
>> > > > understand the
>> > > > > > point about
>> > > > > > > batching.
>> > > > > > > > > Interceptors
>> > > > > > > > > > are
>> > > > > > > > > > > > applied
>> > > > per-message, right?
>> > > > > > > > > > > >
>> > > > > > > > > > > > My
>> intent on
>> > > > interceptors is to
>> > > > > > keep the usage of
>> > > > > > > > headers
>> > > > > > > > > > well-defined so
>> > > > > > > > > > > > that
>> they don't
>> > > > start leaking
>> > > > > > unnecessarily into
>> > > > > > > > > applications. My
>> > > > > > > > > > guess
>> > > > > > > > > > > is
>> > > > > > > > > > > > that it's
>> > > probably
>> > > > inevitable,
>> > > > > but
>> > > > > > isolating it
>> > > > > > > in
>> > > > > > > > the
>> > > > > > > > > > interceptors
>> would
>> > > > > > > > > > > > at least
>> give
>> > > > people a second
>> > > > > > thought before
>> > > > > > > > deciding to
>> > > > > > > > > use it.
>> > > > > > > > > > The main
>> > > > > > > > > > > >
>> challenge in my
>> > > > mind is figuring
>> > > > > > out how an MM
>> > > > > > > use
>> > > > > > > > case
>> > > > > > > > > would
>> > > > > > > > > > work. It
>> > > > > > > > > > > > would be
>> more
>> > > > cumbersome to
>> > > > > > replicate headers
>> > > > > > > > through an
>> > > > > > > > > > interceptor,
>> > > > > > > > > > > > though
>> arguably
>> > > MM
>> > > > should be
>> > > > > > working at a lower
>> > > > > > > > level
>> > > > > > > > > anyway.
>> > > > > > > > > > > >
>> > > > > > > > > > > > -Jason
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri,
>> Feb 17,
>> > > > 2017 at 10:16 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > >> Re
>> headers
>> > > > available on the
>> > > > > > record va
>> > > > > > > > interceptors only
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> One of
>> the key
>> > > > use cases
>> > > > > requires
>> > > > > > access on
>> > > > > > > > consume at
>> > > > > > > > > per
>> > > > > > > > > > event/message
>> > > > > > > > > > > >> level
>> at the
>> > > > point that message
>> > > > > > is being
>> > > > > > > > processed, as
>> > > > > > > > > such the
>> > > > > > > > > > batch
>> > > > > > > > > > > >>
>> interceptors
>> > and
>> > > > batch consume
>> > > > > > api isn't
>> > > > > > > > suitable. It
>> > > > > > > > > needs to be
>> > > > > > > > > > at the
>> > > > > > > > > > > >> record
>> level.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> This
>> anyhow is
>> > > > similar to
>> > > > > > jms/http/amqp where
>> > > > > > > > headers
>> > > > > > > > > are
>> > > > > > > > > > available to
>> > > > > > > > > > > >>
>> consuming
>> > > > applications.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Re
>> headers as
>> > > > byte array and
>> > > > > > future use by
>> > > > > > > > broker. This
>> > > > > > > > > doesn't
>> > > > > > > > > > take
>> > > > > > > > > > > away
>> > > > > > > > > > > >> from
>> that at
>> > > all.
>> > > > Nor makes it
>> > > > > > difficult at all
>> > > > > > > > in my
>> > > > > > > > > opinion.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Sent
>> using OWA
>> > > > for iPhone
>> > > > > > > > > > > >>
>> > > > ______________________________
>> > > > > > __________
>> > > > > > > > > > > >> From:
>> Jason
>> > > > Gustafson <
>> > > > > > jason@confluent.io>
>> > > > > > > > > > > >> Sent:
>> Friday,
>> > > > February 17, 2017
>> > > > > > 5:55:42 PM
>> > > > > > > > > > > >> To:
>> > > > dev@kafka.apache.org
>> > > > > > > > > > > >>
>> Subject: Re:
>> > > > [DISCUSS] KIP-82 -
>> > > > > > Add Record
>> > > > > > > Headers
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> Yes,
>> that is
>> > > what
>> > > > we are
>> > > > > > proposing, so using
>> > > > > > > > varints
>> > > > > > > > > for headers
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >>
>> consistent
>> > with
>> > > > the rest of the
>> > > > > > message. We have
>> > > > > > > > used
>> > > > > > > > > static
>> > > > > > > > > > helper
>> > > > > > > > > > > >> methods
>> > > > > > > > > > > >> in our
>> > prototype
>> > > > implementation.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> The
>> cost of
>> > > > parsing, we want to
>> > > > > > parse/interpret
>> > > > > > > > the
>> > > > > > > > > headers
>> > > > > > > > > > lazily (this
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > a key
>> point
>> > > > brought up earlier
>> > > > > > in discussions)
>> > > > > > > > > > > >>
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> I'm a
>> bit
>> > > > skeptical of this. Has
>> > > > > > anyone done the
>> > > > > > > > > performance
>> > > > > > > > > > testing? I
>> > > > > > > > > > > >> can
>> > > > > > > > > > > >> probably
>> > > > implement it and test
>> > > > > it
>> > > > > > if no one else
>> > > > > > > > has. I
>> > > > > > > > > was also
>> > > > > > > > > > under
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> impression
>> > that
>> > > > there may be use
>> > > > > > cases down the
>> > > > > > > > road
>> > > > > > > > > where the
>> > > > > > > > > > broker
>> > > > > > > > > > > >> would
>> > > > > > > > > > > >> need to
>> > > interpret
>> > > > headers. That
>> > > > > > wouldn't be off
>> > > > > > > > the
>> > > > > > > > > table in the
>> > > > > > > > > > future
>> > > > > > > > > > > if
>> > > > > > > > > > > >> it's
>> > represented
>> > > > as bytes, but
>> > > > > it
>> > > > > > would be quite
>> > > > > > > > a bit
>> > > > > > > > > more
>> > > > > > > > > > awkward,
>> > > > > > > > > > > >> right?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> By the
>> way,
>> > one
>> > > > question I have
>> > > > > > been wondering
>> > > > > > > > about. My
>> > > > > > > > > > understanding
>> > > > > > > > > > > is
>> > > > > > > > > > > >> that
>> headers
>> > are
>> > > > primarily for
>> > > > > > use cases where a
>> > > > > > > > > third-party
>> > > > > > > > > > components
>> > > > > > > > > > > >> wants to
>> > enrich
>> > > > messages without
>> > > > > > needing to
>> > > > > > > > understand
>> > > > > > > > > or modify
>> > > > > > > > > > the
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> of the
>> message
>> > > > key and value.
>> > > > > For
>> > > > > > the
>> > > > > > > applications
>> > > > > > > > > which directly
>> > > > > > > > > > > produce
>> > > > > > > > > > > >> and
>> consume
>> > the
>> > > > messages and
>> > > > > > control the
>> > > > > > > key/value
>> > > > > > > > > schema
>> > > > > > > > > > directly, it
>> > > > > > > > > > > >> seems
>> we would
>> > > > rather have them
>> > > > > > implement
>> > > > > > > headers
>> > > > > > > > > directly in
>> > > > > > > > > > their own
>> > > > > > > > > > > >> schema.
>> > > Supposing
>> > > > for the sake
>> > > > > of
>> > > > > > argument that
>> > > > > > > > it was
>> > > > > > > > > possible,
>> > > > > > > > > > my
>> > > > > > > > > > > >>
>> question is
>> > > > whether it be
>> > > > > > sufficient to expose
>> > > > > > > the
>> > > > > > > > > headers in the
>> > > > > > > > > > > >>
>> interceptor
>> > API
>> > > > and not in the
>> > > > > > common API?
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> -Jason
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> On Fri,
>> Feb
>> > 17,
>> > > > 2017 at 3:26 AM,
>> > > > > > Michael Pearce
>> > > > > > > <
>> > > > > > > > > >
>> Michael.Pearce@ig.com>
>> > > > > > > > > > > >> wrote:
>> > > > > > > > > > > >>
>> > > > > > > > > > > >> > On
>> the point
>> > > of
>> > > > varInts
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Would
>> you be
>> > > > proposing in
>> > > > > > KIP-98 to convert
>> > > > > > > the
>> > > > > > > > other
>> > > > > > > > > message
>> > > > > > > > > > int’s
>> > > > > > > > > > > (key
>> > > > > > > > > > > >> >
>> length,
>> > value
>> > > > length) also to
>> > > > > > varint to keep
>> > > > > > > it
>> > > > > > > > > uniform.
>> > > > > > > > > > > >> > Also I
>> > assume
>> > > > there will be a
>> > > > > > static or helper
>> > > > > > > > method
>> > > > > > > > > made to
>> > > > > > > > > > > write/read
>> > > > > > > > > > > >> > these
>> in the
>> > > > client and
>> > > > > server.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > Cheers
>> > > > > > > > > > > >> > Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > 17/02/2017,
>> > > > 11:22, "Michael
>> > > > > > Pearce" <
>> > > > > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On the
>> > > > point re: headers
>> > > > > in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > being a
>> > > > > > > > > > byte
>> > > > > > > > > > > array
>> > > > > > > > > > > >> > and
>> not a
>> > > count
>> > > > of elements
>> > > > > > followed by the
>> > > > > > > > elements.
>> > > > > > > > > Again
>> > > > > > > > > > this was
>> > > > > > > > > > > >> >
>> > > > discussed/argued previously.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> It was
>> > > > agreed on for a few
>> > > > > > reasons some of
>> > > > > > > > which
>> > > > > > > > > you have
>> > > > > > > > > > > obviously
>> > > > > > > > > > > >> >
>> picked up
>> > on:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Broker
>> > is
>> > > > able to pass it
>> > > > > > through opaquely
>> > > > > > > > > > > >> >
>> The cost
>> > > of
>> > > > parsing, we
>> > > > > > want to
>> > > > > > > > parse/interpret
>> > > > > > > > > the headers
>> > > > > > > > > > lazily
>> > > > > > > > > > > >> > (this
>> is a
>> > key
>> > > > point brought
>> > > > > up
>> > > > > > earlier in
>> > > > > > > > > discussions)
>> > > > > > > > > > > >> >
>> Headers
>> > > can
>> > > > be copied from
>> > > > > > consumer record
>> > > > > > > > to
>> > > > > > > > > producer
>> > > > > > > > > > record (aka
>> > > > > > > > > > > >> > mirror
>> > makers
>> > > > etc) without
>> > > > > > parsing if no
>> > > > > > > > changes are
>> > > > > > > > > being made
>> > > > > > > > > > or
>> > > > > > > > > > > being
>> > > > > > > > > > > >> >
>> looked at.
>> > > > > > > > > > > >> >
>> Keeps
>> > the
>> > > > broker agnostic
>> > > > > > to the format
>> > > > > > > > > > > >> >
>> You need
>> > > an
>> > > > int32 either
>> > > > > > for the byte size
>> > > > > > > > of the
>> > > > > > > > > headers,
>> > > > > > > > > > or for
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> > count
>> of
>> > > > elements, so
>> > > > > overheads
>> > > > > > are the same,
>> > > > > > > > but
>> > > > > > > > > with going
>> > > > > > > > > > with an
>> > > > > > > > > > > >> opaque
>> > > > > > > > > > > >> > byte
>> array
>> > has
>> > > > the above
>> > > > > > advantages.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> Cheers
>> > > > > > > > > > > >> >
>> Mike
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > On
>> > > > 17/02/2017, 02:50,
>> > > > > > "Jason Gustafson" <
>> > > > > > > > > jason@confluent.io
>> > > > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > Sorry,
>> > > > should have
>> > > > > > noted that the
>> > > > > > > > performance
>> > > > > > > > > testing
>> > > > > > > > > > was done
>> > > > > > > > > > > >> > using
>> the
>> > > > > > > > > > > >> >
>> > > > producer performance
>> > > > > > tool shipped with
>> > > > > > > > Kafka.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > -Jason
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> On
>> > > Thu,
>> > > > Feb 16, 2017
>> > > > > at
>> > > > > > 6:44 PM, Jason
>> > > > > > > > > Gustafson <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> >
>> > Hey
>> > > > Nacho,
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > I've
>> > > > compared
>> > > > > > performance of our
>> > > > > > > > KIP-98
>> > > > > > > > > > implementation
>> with
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> without
>> > > > > > > > > > > >> >
>> >
>> > > > varints. For
>> > > > > messages
>> > > > > > around 128
>> > > > > > > > bytes, we
>> > > > > > > > > see an
>> > > > > > > > > > increase
>> > > > > > > > > > > in
>> > > > > > > > > > > >> >
>> throughput
>> > of
>> > > > > > > > > > > >> >
>> >
>> > > about
>> > > > 30% using the
>> > > > > > default
>> > > > > > > > configuration
>> > > > > > > > > settings.
>> > > > > > > > > > At 256
>> > > > > > > > > > > >> >
>> bytes, the
>> > > > > > > > > > > >> >
>> >
>> > > > increase is around
>> > > > > > 16%. Obviously
>> > > > > > > the
>> > > > > > > > > performance
>> > > > > > > > > > converge
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> messages get
>> > > > > > > > > > > >> >
>> >
>> > > > larger, but it seems
>> > > > > > well worth the
>> > > > > > > > cost.
>> > > > > > > > > Note that
>> > > > > > > > > > we are
>> > > > > > > > > > > >> also
>> > > > > > > > > > > >> >
>> seeing a
>> > > > > > > > > > > >> >
>> >
>> > > > substantial
>> > > > > > performance increase
>> > > > > > > > against
>> > > > > > > > > trunk
>> > > > > > > > > > primarily
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >
>> > much
>> > > > more efficient
>> > > > > > packing that
>> > > > > > > > varints
>> > > > > > > > > provide us.
>> > > > > > > > > > > Anything
>> > > > > > > > > > > >> >
>> adding to
>> > > > > > > > > > > >> >
>> >
>> > > > message overhead,
>> > > > > > such as record
>> > > > > > > > headers,
>> > > > > > > > > would only
>> > > > > > > > > > > increase
>> > > > > > > > > > > >> > the
>> relative
>> > > > > > > > > > > >> >
>> >
>> > > > difference. (Of
>> > > > > > course take these
>> > > > > > > > numbers
>> > > > > > > > > with a
>> > > > > > > > > > grain of
>> > > > > > > > > > > salt
>> > > > > > > > > > > >> > since
>> I have
>> > > > > > > > > > > >> >
>> >
>> > only
>> > > > used the
>> > > > > default
>> > > > > > settings with
>> > > > > > > > both
>> > > > > > > > > the producer
>> > > > > > > > > > and
>> > > > > > > > > > > >> broker
>> > > > > > > > > > > >> > on my
>> > > > > > > > > > > >> >
>> >
>> > > local
>> > > > machine. We
>> > > > > > intend to provide
>> > > > > > > > more
>> > > > > > > > > extensive
>> > > > > > > > > > > performance
>> > > > > > > > > > > >> >
>> details as
>> > > > > > > > > > > >> >
>> >
>> > part
>> > > > of the work for
>> > > > > > KIP-98.)
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > The
>> > > > implementation
>> > > > > we
>> > > > > > are using is
>> > > > > > > > from
>> > > > > > > > > protobuf (
>> > > > > > > > > > > >> >
>> >
>> > > > > > https://developers.google.com/
>> > > > > > > > > > >
>> > > > protocol-buffers/docs/encoding
>> > > > > > > > > > > >> ),
>> > > > > > > > > > > >> > which
>> is
>> > > > > > > > > > > >> >
>> >
>> > also
>> > > > used in HBase.
>> > > > > > It is trivial to
>> > > > > > > > > implement and as
>> > > > > > > > > > far
>> > > > > > > > > > > as I
>> > > > > > > > > > > >> > know
>> doesn't
>> > > > > > > > > > > >> >
>> >
>> > > > suffer from the
>> > > > > > aliasing problem you
>> > > > > > > > are
>> > > > > > > > > describing. I
>> > > > > > > > > > > checked
>> > > > > > > > > > > >> > with
>> Magnus
>> > > > > > > > > > > >> >
>> >
>> > (the
>> > > > author of
>> > > > > > librdkafka) and he
>> > > > > > > > agreed
>> > > > > > > > > that the
>> > > > > > > > > > savings
>> > > > > > > > > > > >> seemed
>> > > > > > > > > > > >> > worth
>> the
>> > > > > > > > > > > >> >
>> >
>> > cost
>> > > > of
>> > > > > > implementation.
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > -Jason
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > On
>> > > > Thu, Feb 16, 2017
>> > > > > > at 4:32 PM,
>> > > > > > > > Ignacio
>> > > > > > > > > Solis <
>> > > > > > > > > > > >>
>> > isolis@igso.net
>> > > >
>> > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >>
>> > > > -VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > I'm
>> > > > one of the
>> > > > > > people (if not the
>> > > > > > > > most)
>> > > > > > > > > opposed to
>> > > > > > > > > > VarInts.
>> > > > > > > > > > > >> >
>> VarInts
>> > > > > > > > > > > >> >
>> >>
>> > > have
>> > > > a place, but
>> > > > > > this is not it.
>> > > > > > > > (We
>> > > > > > > > > had a large
>> > > > > > > > > > > >>
>> discussion
>> > > > > > > > > > > >> > about
>> > > > > > > > > > > >> >
>> >>
>> > > them
>> > > > at the
>> > > > > > beginning of KIP-82
>> > > > > > > time)
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > anybody has real
>> > > > > > life
>> > > > > > > performance
>> > > > > > > > > numbers of
>> > > > > > > > > > VarInts
>> > > > > > > > > > > >> >
>> improving
>> > > > > > > > > > > >> >
>> >>
>> > > > things or
>> > > > > > significantly reducing
>> > > > > > > > resources
>> > > > > > > > > I would
>> > > > > > > > > > like to
>> > > > > > > > > > > >> know
>> > > > > > > > > > > >> > what
>> > > > > > > > > > > >> >
>> >>
>> > > that
>> > > > case may be.
>> > > > > > Yes, you can save
>> > > > > > > > some
>> > > > > > > > > bytes here
>> > > > > > > > > > and
>> > > > > > > > > > > >> there,
>> > > > > > > > > > > >> > but
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > is probably
>> > > > > > insignificant to
>> > > > > > > the
>> > > > > > > > > overall system
>> > > > > > > > > > > behavior
>> > > > > > > > > > > >> > and
>> > > > > > > > > > > >> >
>> >>
>> > > > storage
>> > > > > > requirements. -- I say
>> > > > > > > this
>> > > > > > > > with
>> > > > > > > > > respect to
>> > > > > > > > > > using
>> > > > > > > > > > > >> >
>> VarInts in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > protocol
>> > > > > itself,
>> > > > > > not as part of
>> > > > > > > > the
>> > > > > > > > > data.
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > VarInts require you
>> > > > > > to parse the
>> > > > > > > Int
>> > > > > > > > > before using it
>> > > > > > > > > > and
>> > > > > > > > > > > >> >
>> depending on
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > encoding they
>> > > > > > can suffer from
>> > > > > > > > aliasing
>> > > > > > > > > (multiple
>> > > > > > > > > > > >> >
>> > > representations
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > the same
>> > > > > value).
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > Why
>> > > > add complexity?
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > Nacho
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > On
>> > > > Thu, Feb 16,
>> > > > > 2017
>> > > > > > at 10:29 AM,
>> > > > > > > > Colin
>> > > > > > > > > McCabe <
>> > > > > > > > > > > >> >
>> > > > cmccabe@apache.org>
>> > > > > > > > > > > >> >
>> >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >> >
>> > > +1
>> > > > for varints
>> > > > > > here-- it would
>> > > > > > > save
>> > > > > > > > > quite a bit of
>> > > > > > > > > > space.
>> > > > > > > > > > > >> > They
>> are
>> > > > > > > > > > > >> >
>> >> >
>> > > > pretty quick to
>> > > > > > implement as
>> > > > > > > well.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > > I
>> > > > think it makes
>> > > > > > sense for values
>> > > > > > > > to be
>> > > > > > > > > byte
>> > > > > > > > > > arrays.
>> > > > > > > > > > > Users
>> > > > > > > > > > > >> > might
>> want
>> > > > > > > > > > > >> >
>> >> >
>> > > to
>> > > > attach
>> > > > > > arbitrary payloads;
>> > > > > > > they
>> > > > > > > > > shouldn't be
>> > > > > > > > > > forced to
>> > > > > > > > > > > >> >
>> serialize
>> > > > > > > > > > > >> >
>> >> >
>> > > > everything to
>> > > > > Java
>> > > > > > strings.
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > best,
>> > > > > > > > > > > >> >
>> >> >
>> > > > Colin
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >> >
>> > > On
>> > > > Thu, Feb 16,
>> > > > > > 2017, at 09:52,
>> > > > > > > > Jason
>> > > > > > > > > Gustafson
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hey Michael,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Hmm, I guess the
>> > > > > > point of
>> > > > > > > > representing
>> > > > > > > > > it as
>> > > > > > > > > > bytes is to
>> > > > > > > > > > > >> > allow
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > broker
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > to pass it
>> > > > > > through opaquely? Is
>> > > > > > > > the
>> > > > > > > > > cost of
>> > > > > > > > > > parsing
>> > > > > > > > > > > them a
>> > > > > > > > > > > >> >
>> concern, or
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > are
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > we simply trying
>> > > > > > to ensure that
>> > > > > > > > the
>> > > > > > > > > broker stays
>> > > > > > > > > > > agnostic
>> > > > > > > > > > > >> to
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > format?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On varints, I
>> > > > > > think adding
>> > > > > > > > support for
>> > > > > > > > > them makes
>> > > > > > > > > > less
>> > > > > > > > > > > >> sense
>> > > > > > > > > > > >> > for an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > isolated use
>> > > > > > case, but as part
>> > > > > > > of
>> > > > > > > > a
>> > > > > > > > > more holistic
>> > > > > > > > > > change
>> > > > > > > > > > > >> > (such
>> as
>> > what
>> > > > > > > > > > > >> >
>> >>
>> > we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > have proposed in
>> > > > > > KIP-98), I
>> > > > > > > think
>> > > > > > > > they
>> > > > > > > > > are
>> > > > > > > > > > justifiable.
>> > > > > > > > > > > If
>> > > > > > > > > > > >> > we
>> add them,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > then the need to
>> > > > > > use attributes
>> > > > > > > > becomes
>> > > > > > > > > quite a
>> > > > > > > > > > bit
>> > > > > > > > > > > >> weaker,
>> > > > > > > > > > > >> >
>> right? The
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > other thing I
>> > > > > > find slightly odd
>> > > > > > > > is the
>> > > > > > > > > fact that
>> > > > > > > > > > null
>> > > > > > > > > > > >> >
>> headers has
>> > no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > actual
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > semantic meaning
>> > > > > > for the message
>> > > > > > > > > (unlike null
>> > > > > > > > > > keys and
>> > > > > > > > > > > >> >
>> values). It
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > a
>> > > > space
>> > > > > > optimization. It seems a
>> > > > > > > > bit
>> > > > > > > > > better to
>> > > > > > > > > > always
>> > > > > > > > > > > use
>> > > > > > > > > > > >> > size
>> 0 to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > indicate having
>> > > > > > no headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > Overall, the
>> > > > > main
>> > > > > > point is
>> > > > > > > > ensuring
>> > > > > > > > > that the
>> > > > > > > > > > message
>> > > > > > > > > > > >> schema
>> > > > > > > > > > > >> >
>> remains
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > consistent,
>> > > > > > either within the
>> > > > > > > > larger
>> > > > > > > > > protocol, or
>> > > > > > > > > > at a
>> > > > > > > > > > > >> >
>> minimum
>> > within
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > message itself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > On Thu, Feb 16,
>> > > > > > 2017 at 6:39 AM,
>> > > > > > > > > Michael Pearce <
>> > > > > > > > > > > >> >
>> > > > Michael.Pearce@ig.com
>> > > > > > > > > > > >> >
>> >> >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Hi Jason,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 1) in
>> > > > > > the message
>> > > > > > > > protocol
>> > > > > > > > > the headers
>> > > > > > > > > > are
>> > > > > > > > > > > >> simply
>> > > > > > > > > > > >> > a byte
>> > > > > > > > > > > >> >
>> >>
>> > > > array,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > as like the
>> > > > > key
>> > > > > > or value, this
>> > > > > > > > is to
>> > > > > > > > > clearly
>> > > > > > > > > > demarcate
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> header in
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > core message.
>> > > > > > Then the header
>> > > > > > > > byte
>> > > > > > > > > array in the
>> > > > > > > > > > core
>> > > > > > > > > > > >> >
>> message is
>> > an
>> > > > > > > > > > > >> >
>> >>
>> > > > array of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key, value
>> > > > > > pairs. This is what
>> > > > > > > > it is
>> > > > > > > > > denoting.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Then this
>> > > > > would
>> > > > > > be I guess in
>> > > > > > > > the
>> > > > > > > > > given
>> > > > > > > > > > notation:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Headers =>
>> > > > > > [KeyLength, Key,
>> > > > > > > > > ValueLength, Value]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > KeyLength
>> > > > > > => int32
>> > > > > > > > > <-----------------NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised key
>> > > > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Key =>
>> > > > > bytes
>> > > > > > > > > <---------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > string
>> > (UTF8)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > bytes of the
>> > > > > > header key
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > ValueLength
>> > > > > > => int32
>> > > > > > > > > <-------------- NEW
>> > > > > > > > > > size of
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> byte[] of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialised
>> > > > > > header value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Value =>
>> > > > > > bytes
>> > > > > > > > > <-------------------- NEW
>> > > > > > > > > > > serialised
>> > > > > > > > > > > >> > form
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > value
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The key length
>> > > > > > and value
>> > > > > > > length
>> > > > > > > > is
>> > > > > > > > > matching the
>> > > > > > > > > > way
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> protocol is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > defined in the
>> > > > > > core message
>> > > > > > > > currently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 2)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Var sized
>> > > > > ints,
>> > > > > > this was
>> > > > > > > > discussed
>> > > > > > > > > much earlier
>> > > > > > > > > > on, in
>> > > > > > > > > > > >> > fact
>> I had
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > suggested it
>> > > > > > myself (with
>> > > > > > > Hadoop
>> > > > > > > > > references),
>> > > > > > > > > > the
>> > > > > > > > > > > >> >
>> complexity
>> > of
>> > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compared to
>> > > > > > having a simpler
>> > > > > > > > protocol
>> > > > > > > > > was
>> > > > > > > > > > argued and
>> > > > > > > > > > > >> >
>> agreed it
>> > > > > > > > > > > >> >
>> >>
>> > > > wasn’t worth
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the complexity
>> > > > > > as all other
>> > > > > > > > clients
>> > > > > > > > > in other
>> > > > > > > > > > languages
>> > > > > > > > > > > >> > would
>> need
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > > > ensure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > theyre using
>> > > > > > the right var
>> > > > > > > size
>> > > > > > > > > algorithm, as
>> > > > > > > > > > there
>> > > > > > > > > > > is a
>> > > > > > > > > > > >> > few.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On point 3)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > We did the
>> > > > > > attributes,
>> > > > > > > optional
>> > > > > > > > > approach as
>> > > > > > > > > > originally
>> > > > > > > > > > > >> > there
>> was
>> > > > > > > > > > > >> >
>> >>
>> > > > marked
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > concern that
>> > > > > > headers would
>> > > > > > > > cause a
>> > > > > > > > > message size
>> > > > > > > > > > > overhead
>> > > > > > > > > > > >> > for
>> others,
>> > > > > > > > > > > >> >
>> >>
>> > who
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > don’t want
>> > > > > > them. As such this
>> > > > > > > > is the
>> > > > > > > > > clean
>> > > > > > > > > > solution to
>> > > > > > > > > > > >> >
>> achieve
>> > that.
>> > > > > > > > > > > >> >
>> >>
>> > If
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that no longer
>> > > > > > holds, and we
>> > > > > > > > don’t
>> > > > > > > > > care that we
>> > > > > > > > > > add
>> > > > > > > > > > > >> 4bytes
>> > > > > > > > > > > >> >
>> overhead,
>> > > > > > > > > > > >> >
>> >>
>> > > then
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > im happy to
>> > > > > > remove.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > I’m personally
>> > > > > > in favour of
>> > > > > > > > keeping
>> > > > > > > > > the message
>> > > > > > > > > > as
>> > > > > > > > > > > small
>> > > > > > > > > > > >> > as
>> possible
>> > > > > > > > > > > >> >
>> >>
>> > so
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > people don’t
>> > > > > > get shocks in
>> > > > > > > perf
>> > > > > > > > and
>> > > > > > > > > throughputs
>> > > > > > > > > > dues
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> message
>> > size,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > unless they
>> > > > > > actively use the
>> > > > > > > > feature,
>> > > > > > > > > as such I
>> > > > > > > > > > do
>> > > > > > > > > > > >> prefer
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > attribute bit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wise feature
>> > > > > > flag approach
>> > > > > > > > myself.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > On 16/02/2017,
>> > > > > > 05:40, "Jason
>> > > > > > > > > Gustafson" <
>> > > > > > > > > > > >> >
>> > > > jason@confluent.io> wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > We have
>> > > > > > proposed a few
>> > > > > > > > > significant changes
>> > > > > > > > > > to the
>> > > > > > > > > > > >> >
>> message
>> > format
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > KIP-98
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > which now
>> > > > > > seems likely to
>> > > > > > > > pass
>> > > > > > > > > (perhaps
>> > > > > > > > > > with some
>> > > > > > > > > > > >> >
>> iterations
>> > on
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > implementation details).
>> > > > > > > It
>> > > > > > > > would
>> > > > > > > > > be good
>> > > > > > > > > > to try
>> > > > > > > > > > > and
>> > > > > > > > > > > >> >
>> coordinate
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > changes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > in both of
>> > > > > > the proposals
>> > > > > > > to
>> > > > > > > > make
>> > > > > > > > > sure they
>> > > > > > > > > > are
>> > > > > > > > > > > >> >
>> consistent
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > compatible.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I think
>> > > > > > using the
>> > > > > > > > attributes to
>> > > > > > > > > indicate
>> > > > > > > > > > null
>> > > > > > > > > > > >> headers
>> > > > > > > > > > > >> > is a
>> > > > > > > > > > > >> >
>> >>
>> > > > reasonable
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > approach.
>> > > > > > We have proposed
>> > > > > > > > to do
>> > > > > > > > > the same
>> > > > > > > > > > thing
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> message
>> > > > > > > > > > > >> >
>> >>
>> > key
>> > > > and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > value.
>> > > > > That
>> > > > > > said, I
>> > > > > > > > sympathize
>> > > > > > > > > with Jay's
>> > > > > > > > > > > argument.
>> > > > > > > > > > > >> > Having
>> > > > > > > > > > > >> >
>> >>
>> > > > multiple
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > ways to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > specify a
>> > > > > > null value
>> > > > > > > > increases
>> > > > > > > > > the overall
>> > > > > > > > > > > >>
>> complexity
>> > > > > > > > > > > >> > of the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > You
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > can see
>> > > > > > this just from the
>> > > > > > > > fact
>> > > > > > > > > that you
>> > > > > > > > > > need the
>> > > > > > > > > > > >> > extra
>> > verbiage
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > protocol
>> > > > > > specification in
>> > > > > > > > this
>> > > > > > > > > KIP and in
>> > > > > > > > > > KIP-98
>> > > > > > > > > > > to
>> > > > > > > > > > > >> >
>> describe the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > dependence
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > between
>> > > > > the
>> > > > > > fields and the
>> > > > > > > > > attributes. It
>> > > > > > > > > > seems
>> > > > > > > > > > > >> like a
>> > > > > > > > > > > >> >
>> slippery
>> > > > > > > > > > > >> >
>> >>
>> > > slope
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > if
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > you start
>> > > > > > allowing
>> > > > > > > different
>> > > > > > > > > request types
>> > > > > > > > > > to
>> > > > > > > > > > > >> >
>> implement
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification differently.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > You can
>> > > > > > also argue that
>> > > > > > > the
>> > > > > > > > > messages
>> > > > > > > > > > already are
>> > > > > > > > > > > and
>> > > > > > > > > > > >> > are
>> likely
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > remain a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > special
>> > > > > > case. For example,
>> > > > > > > > there
>> > > > > > > > > is
>> > > > > > > > > > currently no
>> > > > > > > > > > > >> >
>> generality
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > how
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > compressed
>> > > > > > message sets
>> > > > > > > are
>> > > > > > > > > represented
>> > > > > > > > > > that would
>> > > > > > > > > > > >> be
>> > > > > > > > > > > >> >
>> applicable
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > other
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > request
>> > > > > > types. Some might
>> > > > > > > > see this
>> > > > > > > > > > divergence as
>> > > > > > > > > > > an
>> > > > > > > > > > > >> >
>> unfortunate
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > deficiency
>> > > > > > which should be
>> > > > > > > > fixed;
>> > > > > > > > > others
>> > > > > > > > > > might see
>> > > > > > > > > > > >> it
>> > > > > > > > > > > >> > as
>> sort of
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > inevitability of needing
>> > > > > > > to
>> > > > > > > > > optimize where
>> > > > > > > > > > it
>> > > > > > > > > > > counts
>> > > > > > > > > > > >> > most.
>> I'm
>> > > > > > > > > > > >> >
>> >>
>> > > > probably
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > somewhere
>> > > > > > in between, but
>> > > > > > > I
>> > > > > > > > think
>> > > > > > > > > we
>> > > > > > > > > > probably all
>> > > > > > > > > > > >> > share
>> the
>> > > > > > > > > > > >> >
>> >>
>> > > > intuition
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol should be
>> > > > > > > kept
>> > > > > > > > as
>> > > > > > > > > consistent as
>> > > > > > > > > > > >>
>> possible.
>> > > > > > > > > > > >> > With
>> that
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > mind,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > here are a
>> > > > > > few comments:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 1. One
>> > > > > > thing I found a
>> > > > > > > > little odd
>> > > > > > > > > when
>> > > > > > > > > > reading the
>> > > > > > > > > > > >> >
>> current
>> > > > > > > > > > > >> >
>> >>
>> > > > proposal is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > headers
>> > > > > > are both
>> > > > > > > > represented
>> > > > > > > > > as an
>> > > > > > > > > > array of
>> > > > > > > > > > > >> bytes
>> > > > > > > > > > > >> > and
>> as an
>> > > > > > > > > > > >> >
>> >>
>> > > array
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > key/value
>> > > > > > pairs. I'd
>> > > > > > > > probably
>> > > > > > > > > suggest
>> > > > > > > > > > something
>> > > > > > > > > > > like
>> > > > > > > > > > > >> > this:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > [HeaderKey
>> > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > HeaderKey
>> > > > > > => String
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue => Bytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > An array
>> > > > > in
>> > > > > > the Kafka
>> > > > > > > > protocol is
>> > > > > > > > > > represented as a
>> > > > > > > > > > > >> > 4-byte
>> > integer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > indicating
>> > > > > > the number of
>> > > > > > > > elements
>> > > > > > > > > in the
>> > > > > > > > > > array
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > serialization of the
>> > > > > > > > elements.
>> > > > > > > > > Unless I'm
>> > > > > > > > > > > >> >
>> > > > misunderstanding, what
>> > > > > > > > > > > >> >
>> >>
>> > you
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > instead is
>> > > > > > the total size
>> > > > > > > > of the
>> > > > > > > > > headers in
>> > > > > > > > > > bytes
>> > > > > > > > > > > >> >
>> followed by
>> > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > elements.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > I'm not
>> > > > > > sure I see any
>> > > > > > > > reason for
>> > > > > > > > > this
>> > > > > > > > > > > >>
>> inconsistency.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 2. In
>> > > > > > KIP-98, we've
>> > > > > > > > introduced
>> > > > > > > > > > variable-length
>> > > > > > > > > > > >> integer
>> > > > > > > > > > > >> >
>> fields.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Effectively,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > we've
>> > > > > > enriched (or
>> > > > > > > > "complicated"
>> > > > > > > > > as Jay
>> > > > > > > > > > might say
>> > > > > > > > > > > ;)
>> > > > > > > > > > > >> > the
>> protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > specification to include
>> > > > > > > the
>> > > > > > > > > following
>> > > > > > > > > > types:
>> > > > > > > > > > > >> VarInt,
>> > > > > > > > > > > >> >
>> VarLong,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > UnsignedVarInt and
>> > > > > > > > > UnsignedVarLong.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Along with
>> > > > > > these
>> > > > > > > > primitives, we
>> > > > > > > > > could
>> > > > > > > > > > introduce
>> > > > > > > > > > > the
>> > > > > > > > > > > >> >
>> following
>> > > > > > > > > > > >> >
>> >>
>> > > > types:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeArray =>
>> > > > > > > > NumberOfItems
>> > > > > > > > > Item1 Item2
>> > > > > > > > > > .. ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItems =>
>> > > > > > > > UnsignedVarInt
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > VarSizeNullableArray =>
>> > > > > > > > > NumberOfItemsOrNull
>> > > > > > > > > > Item1
>> > > > > > > > > > > >> > Item2
>> ..
>> > ItemN
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > NumberOfItemsOrNull =>
>> > > > > > > > VarInt
>> > > > > > > > > (-1 means
>> > > > > > > > > > null)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > And
>> > > > > > similarly for the
>> > > > > > > > `String`
>> > > > > > > > > and `Bytes`
>> > > > > > > > > > types.
>> > > > > > > > > > > >> > These
>> types
>> > > > > > > > > > > >> >
>> >>
>> > can
>> > > > save
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > considerable amount of
>> > > > > > > > space in
>> > > > > > > > > this
>> > > > > > > > > > proposal
>> > > > > > > > > > > >> because
>> > > > > > > > > > > >> > they
>> can
>> > > > > > > > > > > >> >
>> >>
>> > be
>> > > > used
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > both the
>> > > > > > number of headers
>> > > > > > > > > included in the
>> > > > > > > > > > message
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > the
>> > > > > > > > > > > >> >
>> >>
>> > > > lengths of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > header
>> > > > > keys
>> > > > > > and values. We
>> > > > > > > > could
>> > > > > > > > > do this
>> > > > > > > > > > instead:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Headers =>
>> > > > > > > > VarSizeArray[HeaderKey
>> > > > > > > > > > HeaderValue]
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > HeaderKey
>> > > > > > =>
>> > > > > > > VarSizeString
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > HeaderValue =>
>> > > > > > > > VarSizeBytes
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > Combining
>> > > > > > the savings from
>> > > > > > > > the
>> > > > > > > > > use of
>> > > > > > > > > > variable
>> > > > > > > > > > > >> length
>> > > > > > > > > > > >> >
>> fields, the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > benefit
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > of using
>> > > > > > the attributes to
>> > > > > > > > > represent null
>> > > > > > > > > > seems
>> > > > > > > > > > > >> pretty
>> > > > > > > > > > > >> > small.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > 3.
>> > > > > > Whichever way we go
>> > > > > > > > (whether
>> > > > > > > > > we use the
>> > > > > > > > > > > >>
>> attributes
>> > > > > > > > > > > >> > or
>> not), we
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > should at
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > least be
>> > > > > > consistent
>> > > > > > > between
>> > > > > > > > this
>> > > > > > > > > KIP and
>> > > > > > > > > > KIP-98.
>> > > > > > > > > > > It
>> > > > > > > > > > > >> > would
>> be
>> > very
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > strange
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > to have
>> > > > > two
>> > > > > > ways to
>> > > > > > > > represent
>> > > > > > > > > null values
>> > > > > > > > > > in the
>> > > > > > > > > > > >> same
>> > > > > > > > > > > >> >
>> schema.
>> > > > > > > > > > > >> >
>> >>
>> > > > Either
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > way is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > OK with
>> > > > > me.
>> > > > > > I think some
>> > > > > > > > > message-level
>> > > > > > > > > > >
>> optimizations
>> > > > > > > > > > > >> > are
>> > > > > > > > > > > >> >
>> >>
>> > > > justifiable,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > but
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > savings
>> > > > > > here seem
>> > > > > > > > minimal (a
>> > > > > > > > > few bytes
>> > > > > > > > > > per
>> > > > > > > > > > > >> >
>> message), so
>> > > > > > > > > > > >> >
>> >>
>> > > > maybe it's
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > worth the
>> > > > > > cost of letting
>> > > > > > > > the
>> > > > > > > > > message
>> > > > > > > > > > diverge even
>> > > > > > > > > > > >> >
>> further from
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > rest of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > the
>> > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > -Jason
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > On Wed,
>> > > > > Feb
>> > > > > > 15, 2017 at
>> > > > > > > > 8:52 AM,
>> > > > > > > > > radai <
>> > > > > > > > > > > >> >
>> >>
>> > > > > > radai.rosenblatt@gmail.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > I've
>> > > > > > trimmed the inline
>> > > > > > > > > contents as this
>> > > > > > > > > > mail is
>> > > > > > > > > > > >> >
>> getting too
>> > > > > > > > > > > >> >
>> >>
>> > big
>> > > > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > apache
>> > > > > > mailing list
>> > > > > > > > software to
>> > > > > > > > > deliver
>> > > > > > > > > > :-(
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 1. the
>> > > > > > important thing
>> > > > > > > for
>> > > > > > > > > > interoperability
>> is
>> > > > > > > > > > > for
>> > > > > > > > > > > >> >
>> different
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "interested
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > parties"
>> > > > > > (plugins, infra
>> > > > > > > > > layers/wrappers,
>> > > > > > > > > > > >>
>> user-code)
>> > > > > > > > > > > >> > to be
>> > > > > > > > > > > >> >
>> >>
>> > > able
>> > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > stick
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > pieces
>> > > > > of
>> > > > > > metadata onto
>> > > > > > > > msgs
>> > > > > > > > > without
>> > > > > > > > > > getting in
>> > > > > > > > > > > >> each
>> > > > > > > > > > > >> >
>> other's
>> > > > > > > > > > > >> >
>> >>
>> > > way.
>> > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > common
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > key
>> > > > > > scheme (Strings, as
>> > > > > > > > of the
>> > > > > > > > > time of
>> > > > > > > > > > this
>> > > > > > > > > > > >> >
>> writing?) is
>> > > all
>> > > > > > > > > > > >> >
>> >>
>> > > thats
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > required
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > for
>> > > > > that.
>> > > > > > it is assumed
>> > > > > > > > that
>> > > > > > > > > the other end
>> > > > > > > > > > > >> >
>> interested
>> > in
>> > > > any
>> > > > > > > > > > > >> >
>> >>
>> > > such
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > piece of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > metadata
>> > > > > > knows the
>> > > > > > > > encoding,
>> > > > > > > > > and byte[]
>> > > > > > > > > > provides
>> > > > > > > > > > > >> for
>> > > > > > > > > > > >> > the
>> most
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > flexibility.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > i
>> > > > > believe
>> > > > > > this is the
>> > > > > > > same
>> > > > > > > > > logic behind
>> > > > > > > > > > core
>> > > > > > > > > > > kafka
>> > > > > > > > > > > >> > being
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > byte[]/byte[]
>> > > > > -
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Strings
>> > > > > > are more
>> > > > > > > "usable"
>> > > > > > > > but
>> > > > > > > > > bytes are
>> > > > > > > > > > flexible
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> > so
>> were
>> > > > > > > > > > > >> >
>> >>
>> > > > chosen.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > Also -
>> > > > > > core kafka doesnt
>> > > > > > > > even
>> > > > > > > > > do that
>> > > > > > > > > > good of a
>> > > > > > > > > > > >> job
>> > > > > > > > > > > >> > on
>> > > > > > > > > > > >> >
>> >>
>> > > > usability of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > payload
>> > > > > > (example - i
>> > > > > > > have
>> > > > > > > > to
>> > > > > > > > > specify the
>> > > > > > > > > > nop
>> > > > > > > > > > > >> byte[]
>> > > > > > > > > > > >> >
>> "decoders"
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > explicitly
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > in
>> > > > > conf),
>> > > > > > and again
>> > > > > > > > sacrificies
>> > > > > > > > > usability
>> > > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > >> > sake
>> of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > performance
>> > > > > (no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > convenient single-record
>> > > > > > > > > processing as
>> > > > > > > > > > poll is a
>> > > > > > > > > > > >> >
>> batch, lots
>> > of
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > obscure
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > little
>> > > > > > config details
>> > > > > > > > exposing
>> > > > > > > > > internals
>> > > > > > > > > > of the
>> > > > > > > > > > > >> >
>> batching
>> > > > > > > > > > > >> >
>> >>
>> > > > mechanism,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > etc)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > this is
>> > > > > > also why i
>> > > > > > > really
>> > > > > > > > > dislike the
>> > > > > > > > > > idea of a
>> > > > > > > > > > > >> > "type
>> > system"
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > values,
>> > > > > > it further
>> > > > > > > > degrades the
>> > > > > > > > > > usability, adds
>> > > > > > > > > > > >> >
>> complexity
>> > and
>> > > > > > > > > > > >> >
>> >>
>> > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > eventually get in
>> > > > > > > > people's way,
>> > > > > > > > > also, it
>> > > > > > > > > > would
>> > > > > > > > > > > be
>> > > > > > > > > > > >> > the
>> 2nd/3rd
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > home-group
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization mechanism
>> > > > > > > > in core
>> > > > > > > > > kafka
>> > > > > > > > > > (counting
>> > > > > > > > > > > 2
>> > > > > > > > > > > >> >
>> iterations
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > "type
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > definition DSL")
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 2. this
>> > > > > > is an
>> > > > > > > > implementation
>> > > > > > > > > detail, and
>> > > > > > > > > > not
>> > > > > > > > > > > even
>> > > > > > > > > > > >> a
>> > > > > > > > > > > >> > very
>> "user
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > facing" one?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to the
>> > > > > > best of my
>> > > > > > > > understanding
>> > > > > > > > > the vote
>> > > > > > > > > > process
>> > > > > > > > > > > >> is
>> > > > > > > > > > > >> > on
>> proposed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > API/behaviour. also -
>> > > > > > > > since
>> > > > > > > > > we're willing
>> > > > > > > > > > to go
>> > > > > > > > > > > >> with
>> > > > > > > > > > > >> >
>> strings
>> > > > > > > > > > > >> >
>> >>
>> > > just
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > serialize
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > a
>> > > > > 0-sized
>> > > > > > header blob
>> > > > > > > and
>> > > > > > > > IIUC
>> > > > > > > > > you dont
>> > > > > > > > > > need any
>> > > > > > > > > > > >> >
>> optionals
>> > > > > > > > > > > >> >
>> >>
>> > > > anymore.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 3. yes,
>> > > > > > we can :-)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > On Tue,
>> > > > > > Feb 14, 2017 at
>> > > > > > > > 11:56
>> > > > > > > > > PM, Michael
>> > > > > > > > > > > Pearce <
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > Michael.Pearce@ig.com>
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > wrote:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Hi
>> > > > > Jay,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1)
>> > > > > > There was some
>> > > > > > > > initial
>> > > > > > > > > debate on the
>> > > > > > > > > > value
>> > > > > > > > > > > >> > part,
>> as
>> > youll
>> > > > > > > > > > > >> >
>> >>
>> > > note
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > String,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > String
>> > > > > > headers were
>> > > > > > > > > discounted early
>> > > > > > > > > > on. The
>> > > > > > > > > > > >> >
>> reason for
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > flexibility
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > keeping in line
>> > > > > > > > with the
>> > > > > > > > > > flexibility of
>> > > > > > > > > > > key,
>> > > > > > > > > > > >> > value
>> of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > message
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > object
>> > > > > > itself. I don’t
>> > > > > > > > think
>> > > > > > > > > it takes
>> > > > > > > > > > away
>> > > > > > > > > > > from
>> > > > > > > > > > > >> an
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > as
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > each
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > plugin
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > will
>> > > > > > care for their
>> > > > > > > own
>> > > > > > > > key,
>> > > > > > > > > this way
>> > > > > > > > > > ints,
>> > > > > > > > > > > >> >
>> booleans ,
>> > > > exotic
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > binary
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > can
>> > > > > all
>> > > > > > be catered
>> > > > > > > for=.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > wanted
>> > > > > > > > to
>> > > > > > > > > push for a
>> > > > > > > > > > typed
>> > > > > > > > > > > >> value
>> > > > > > > > > > > >> >
>> interface,
>> > > > > > > > > > > >> >
>> >> I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > wouldn’t
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > want
>> > > > > > just String
>> > > > > > > values
>> > > > > > > > > supported, but
>> > > > > > > > > > the the
>> > > > > > > > > > > >> >
>> primatives
>> > > > > > > > > > > >> >
>> >>
>> > > plus
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > string and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > also
>> > > > > > still keeping the
>> > > > > > > > > ability to have a
>> > > > > > > > > > > binary
>> > > > > > > > > > > >> > for
>> custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > binaries that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > some
>> > > > > > organisations may
>> > > > > > > > have.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > i. I
>> > > > > > have written this
>> > > > > > > > slight
>> > > > > > > > > > alternative
>> > > > > > > > > > > here,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > https://cwiki.apache.org/
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > > >> >
>> > > > 82+-+Add+Record+Headers+-+
>> > > > > Typed
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > ii.
>> > > > > > Essentially the
>> > > > > > > > value
>> > > > > > > > > bytes, has a
>> > > > > > > > > > leading
>> > > > > > > > > > > >> > byte
>> > overhead.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > > This tells you
>> > > > > > > what
>> > > > > > > > type
>> > > > > > > > > the value
>> > > > > > > > > > is,
>> > > > > > > > > > > >> before
>> > > > > > > > > > > >> >
>> reading
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > rest
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > of the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes,
>> > > > > > allowing
>> > > > > > > > > >
>> > > > serialisation/deserialization
>> > > > > > > > > > > to
>> > > > > > > > > > > >> > and
>> from the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > primitives,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > string
>> > > > > > and byte[].
>> > > > > > > This
>> > > > > > > > is
>> > > > > > > > > akin to some
>> > > > > > > > > > other
>> > > > > > > > > > > >> >
>> messaging
>> > > > > > > > > > > >> >
>> >>
>> > > > systems.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2) We
>> > > > > > are making it
>> > > > > > > > optional,
>> > > > > > > > > so that
>> > > > > > > > > > for
>> > > > > > > > > > > those
>> > > > > > > > > > > >> > not
>> wanting
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > headers have
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > 0
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bytes
>> > > > > > overhead (think
>> > > > > > > > of it
>> > > > > > > > > as a feature
>> > > > > > > > > > > flag),
>> > > > > > > > > > > >> I
>> > > > > > > > > > > >> > don’t
>> > > > > > > > > > > >> >
>> >>
>> > > > think this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > complex, especially if
>> > > > > > > > > comparing to
>> > > > > > > > > > changes
>> > > > > > > > > > > >> >
>> proposed in
>> > > > > > > > > > > >> >
>> >>
>> > > > other kips
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > like
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > kip-98.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a. If
>> > > > > > you really
>> > > > > > > really
>> > > > > > > > don’t
>> > > > > > > > > like
>> > > > > > > > > > this, we
>> > > > > > > > > > > can
>> > > > > > > > > > > >> > drop
>> it, but
>> > > > > > > > > > > >> >
>> >>
>> > it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > would
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > mean
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > buying
>> > > > > > into 4 bytes
>> > > > > > > > extra
>> > > > > > > > > overhead for
>> > > > > > > > > > users
>> > > > > > > > > > > who
>> > > > > > > > > > > >> > do
>> not want
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > use
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > headers.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3) In
>> > > > > > the summary yes,
>> > > > > > > > it is
>> > > > > > > > > at a higher
>> > > > > > > > > > > level,
>> > > > > > > > > > > >> > but I
>> think
>> > > > > > > > > > > >> >
>> >>
>> > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > is well
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > documented in the
>> > > > > > > > proposed
>> > > > > > > > > changes
>> > > > > > > > > > section.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a.
>> > > > > > Added getHeaders
>> > > > > > > > method to
>> > > > > > > > > > >
>> Producer/Consumer
>> > > > > > > > > > > >> >
>> record (that
>> > > > > > > > > > > >> >
>> >>
>> > is
>> > > > it)
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > b.
>> > > > > > We’ve also detailed
>> > > > > > > > the
>> > > > > > > > > new Headers
>> > > > > > > > > > class
>> > > > > > > > > > > >> that
>> > > > > > > > > > > >> > this
>> method
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > returns
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > that
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > encapsulates the
>> > > > > > > headers
>> > > > > > > > > protocol and
>> > > > > > > > > > logic.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Best,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Mike
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ==Original questions
>> > > > > > > > from the
>> > > > > > > > > vote
>> > > > > > > > > > thread from
>> > > > > > > > > > > >> > Jay.==
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > Couple
>> > > > > > of things I
>> > > > > > > > think we
>> > > > > > > > > still need
>> > > > > > > > > > to work
>> > > > > > > > > > > >> out:
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 1.
>> > > > > I
>> > > > > > think we agree
>> > > > > > > > about
>> > > > > > > > > the key,
>> > > > > > > > > > but I
>> > > > > > > > > > > >> think
>> > > > > > > > > > > >> > we
>> haven't
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > talked about
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > value yet. I
>> > > > > > > > think if
>> > > > > > > > > our goal
>> > > > > > > > > > is an
>> > > > > > > > > > > open
>> > > > > > > > > > > >> >
>> ecosystem
>> > > > > > > > > > > >> >
>> >>
>> > of
>> > > > these
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > header
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > spread across many
>> > > > > > > > plugins
>> > > > > > > > > from many
>> > > > > > > > > > > systems
>> > > > > > > > > > > >> we
>> > > > > > > > > > > >> > should
>> > > > > > > > > > > >> >
>> >>
>> > > > consider
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > making
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > this
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > string as well so
>> > > > > > > > it can
>> > > > > > > > > be
>> > > > > > > > > > printed, set
>> > > > > > > > > > > >> via
>> > > > > > > > > > > >> > a UI,
>> set
>> > > > > > > > > > > >> >
>> >>
>> > in
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > config,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > etc.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > Basically
>> > > > > > > encouraging
>> > > > > > > > > pluggable
>> > > > > > > > > > >
>> serialization
>> > > > > > > > > > > >> >
>> formats
>> > > > > > > > > > > >> >
>> >>
>> > > here
>> > > > will
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > lead
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > bit
>> > > > > > of a tower of
>> > > > > > > > babel.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 2.
>> > > > > > This proposal
>> > > > > > > > still
>> > > > > > > > > includes a
>> > > > > > > > > > pretty
>> > > > > > > > > > > big
>> > > > > > > > > > > >> >
>> change to
>> > our
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > serialization
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > and
>> > > > > > protocol
>> > > > > > > > definition
>> > > > > > > > > layer.
>> > > > > > > > > > Essentially
>> > > > > > > > > > > >> it is
>> > > > > > > > > > > >> >
>> >>
>> > > > introducing an
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > optional
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > type, where the
>> > > > > > > > format is
>> > > > > > > > > data
>> > > > > > > > > > dependent. I
>> > > > > > > > > > > >> > think
>> this
>> > is
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > actually a
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > big
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > change though it
>> > > > > > > > doesn't
>> > > > > > > > > seem like
>> > > > > > > > > > it. It
>> > > > > > > > > > > >> means
>> > > > > > > > > > > >> > you
>> can no
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > longer
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > specify
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > this
>> > > > > > type with our
>> > > > > > > > type
>> > > > > > > > > definition
>> > > > > > > > > > DSL, and
>> > > > > > > > > > > >> >
>> likewise it
>> > > > > > > > > > > >> >
>> >>
>> > > > requires
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > custom
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > handling in client
>> > > > > > > > libs.
>> > > > > > > > > This isn't
>> > > > > > > > > > a huge
>> > > > > > > > > > > >> >
>> thing, since
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Record
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > definition is
>> > > > > > > custom
>> > > > > > > > > anyway, but I
>> > > > > > > > > > think
>> > > > > > > > > > > this
>> > > > > > > > > > > >> > kind
>> of
>> > > > > > > > > > > >> >
>> >>
>> > > > protocol
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > inconsistency is
>> > > > > > > very
>> > > > > > > > > non-desirable
>> > > > > > > > > > and
>> > > > > > > > > > > ties
>> > > > > > > > > > > >> > you to
>> > > > > > > > > > > >> >
>> >>
>> > > > hand-coding
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > things.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > I
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > think the type
>> > > > > > > should
>> > > > > > > > > instead by [Key
>> > > > > > > > > > > Value]
>> > > > > > > > > > > >> in
>> > > > > > > > > > > >> > our
>> BNF,
>> > > > > > > > > > > >> >
>> >>
>> > > where
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > key and
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > value are both
>> > > > > > > short
>> > > > > > > > > strings as used
>> > > > > > > > > > > >>
>> elsewhere.
>> > > > > > > > > > > >> > This
>> > > > > > > > > > > >> >
>> >>
>> > > > brings it
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > in line
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > with
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > the
>> > > > > > rest of the
>> > > > > > > > protocol.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > 3.
>> > > > > > Could we get
>> > > > > > > more
>> > > > > > > > > specific about
>> > > > > > > > > > the
>> > > > > > > > > > > exact
>> > > > > > > > > > > >> > Java
>> API
>> > > > > > > > > > > >> >
>> >>
>> > > > change to
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > ProducerRecord,
>> > > > > > > > > ConsumerRecord,
>> > > > > > > > > > Record,
>> > > > > > > > > > > etc?
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > > -Jay
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > The
>> > > > > information
>> > > > > > contained in
>> > > > > > > > this
>> > > > > > > > > email is
>> > > > > > > > > > strictly
>> > > > > > > > > > > >> >
>> confidential
>> > > and
>> > > > > > > > > > > >> >
>> >>
>> > for
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the use of the
>> > > > > > addressee only,
>> > > > > > > > unless
>> > > > > > > > > otherwise
>> > > > > > > > > > > >>
>> indicated.
>> > > > > > > > > > > >> > If you
>> > > > > > > > > > > >> >
>> >>
>> > are
>> > > > not
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > the intended
>> > > > > > recipient, please
>> > > > > > > > do not
>> > > > > > > > > read,
>> > > > > > > > > > copy, use
>> > > > > > > > > > > or
>> > > > > > > > > > > >> >
>> disclose to
>> > > > > > > > > > > >> >
>> >>
>> > > > others
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > this message
>> > > > > or
>> > > > > > any
>> > > > > > > attachment.
>> > > > > > > > > Please also
>> > > > > > > > > > notify the
>> > > > > > > > > > > >> >
>> sender by
>> > > > > > > > > > > >> >
>> >>
>> > > > replying
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > to this email
>> > > > > > or by telephone
>> > > > > > > > > (+44(020 7896
>> > > > > > > > > > 0011) and
>> > > > > > > > > > > >> then
>> > > > > > > > > > > >> > delete
>> > > > > > > > > > > >> >
>> >>
>> > the
>> > > > email
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > and any copies
>> > > > > > of it.
>> > > > > > > Opinions,
>> > > > > > > > > conclusion
>> > > > > > > > > > (etc) that
>> > > > > > > > > > > do
>> > > > > > > > > > > >> > not
>> relate
>> > > > > > > > > > > >> >
>> >>
>> > to
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > official
>> > > > > > business of this
>> > > > > > > > company
>> > > > > > > > > shall be
>> > > > > > > > > > understood
>> > > > > > > > > > > as
>> > > > > > > > > > > >> >
>> neither
>> > > > > > > > > > > >> >
>> >>
>> > > > given nor
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > endorsed by
>> > > > > it.
>> > > > > > IG is a
>> > > > > > > trading
>> > > > > > > > name
>> > > > > > > > > of IG
>> > > > > > > > > > Markets
>> > > > > > > > > > > >> Limited
>> > > > > > > > > > > >> > (a
>> company
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > registered in
>> > > > > > England and
>> > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 04008957)
>> > > > > > > > > > > >> > and IG
>> > > > > > > > > > > >> >
>> >>
>> > > Index
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Limited (a
>> > > > > > company registered
>> > > > > > > in
>> > > > > > > > > England and
>> > > > > > > > > > Wales,
>> > > > > > > > > > > >> >
>> company
>> > number
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > 01190902).
>> > > > > > Registered address
>> > > > > > > at
>> > > > > > > > > Cannon Bridge
>> > > > > > > > > > House,
>> > > > > > > > > > > 25
>> > > > > > > > > > > >> >
>> Dowgate
>> > > > > > > > > > > >> >
>> >>
>> > > Hill,
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > London EC4R
>> > > > > > 2YA. Both IG
>> > > > > > > Markets
>> > > > > > > > > Limited
>> > > > > > > > > > (register
>> > > > > > > > > > > >> number
>> > > > > > > > > > > >> >
>> 195355)
>> > > > > > > > > > > >> >
>> >>
>> > and
>> > > > IG
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Index Limited
>> > > > > > (register number
>> > > > > > > > > 114059) are
>> > > > > > > > > > authorised
>> > > > > > > > > > > >> and
>> > > > > > > > > > > >> >
>> regulated
>> > > > > > > > > > > >> >
>> >>
>> > by
>> > > > the
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > Financial
>> > > > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> >>
>> > >>
>> > > >
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >>
>> > --
>> > > > > > > > > > > >> >
>> >>
>> > > > Nacho - Ignacio
>> > > > > > Solis -
>> > > > > > > > isolis@igso.net
>> > > > > > > > > > > >> >
>> >>
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> > The
>> > > information
>> > > > contained in
>> > > > > > this email is
>> > > > > > > > strictly
>> > > > > > > > > > confidential and
>> > > > > > > > > > > for
>> > > > > > > > > > > >> > the
>> use of
>> > the
>> > > > addressee only,
>> > > > > > unless
>> > > > > > > otherwise
>> > > > > > > > > indicated. If
>> > > > > > > > > > you are
>> > > > > > > > > > > >> not
>> > > > > > > > > > > >> > the
>> intended
>> > > > recipient, please
>> > > > > > do not read,
>> > > > > > > > copy, use
>> > > > > > > > > or
>> > > > > > > > > > disclose to
>> > > > > > > > > > > >> others
>> > > > > > > > > > > >> > this
>> message
>> > > or
>> > > > any
>> > > > > attachment.
>> > > > > > Please also
>> > > > > > > > notify
>> > > > > > > > > the sender by
>> > > > > > > > > > > >> replying
>> > > > > > > > > > > >> > to
>> this
>> > email
>> > > > or by telephone
>> > > > > > (+44(020 7896
>> > > > > > > > 0011) and
>> > > > > > > > > then
>> > > > > > > > > > delete the
>> > > > > > > > > > > >> email
>> > > > > > > > > > > >> > and
>> any
>> > copies
>> > > > of it.
>> > > > > Opinions,
>> > > > > > conclusion
>> > > > > > > > (etc) that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > understood
>> > > > > > > > > as
>> > > > > > > > > > neither given
>> > > > > > > > > > > >> nor
>> > > > > > > > > > > >> >
>> endorsed by
>> > > it.
>> > > > IG is a
>> > > > > trading
>> > > > > > name of IG
>> > > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >> >
>> registered
>> > in
>> > > > England and
>> > > > > > Wales, company
>> > > > > > > number
>> > > > > > > > > 04008957) and
>> > > > > > > > > > IG Index
>> > > > > > > > > > > >> >
>> Limited (a
>> > > > company registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > >> >
>> 01190902).
>> > > > Registered address
>> > > > > > at Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25
>> > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > >> >
>> London EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and
>> > > > > > > > > > > IG
>> > > > > > > > > > > >> > Index
>> > Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > >> the
>> > > > > > > > > > > >> >
>> Financial
>> > > > Conduct Authority.
>> > > > > > > > > > > >> >
>> > > > > > > > > > > >> The
>> > information
>> > > > contained in
>> > > > > this
>> > > > > > email is
>> > > > > > > > strictly
>> > > > > > > > > confidential
>> > > > > > > > > > and for
>> > > > > > > > > > > >> the use
>> of the
>> > > > addressee only,
>> > > > > > unless otherwise
>> > > > > > > > > indicated. If you
>> > > > > > > > > > are
>> > > > > > > > > > > not
>> > > > > > > > > > > >> the
>> intended
>> > > > recipient, please
>> > > > > do
>> > > > > > not read,
>> > > > > > > copy,
>> > > > > > > > use
>> > > > > > > > > or disclose
>> > > > > > > > > > to
>> > > > > > > > > > > others
>> > > > > > > > > > > >> this
>> message
>> > or
>> > > > any attachment.
>> > > > > > Please also
>> > > > > > > > notify the
>> > > > > > > > > sender by
>> > > > > > > > > > > replying
>> > > > > > > > > > > >> to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the
>> > > > > > > > > > > email
>> > > > > > > > > > > >> and any
>> copies
>> > > of
>> > > > it. Opinions,
>> > > > > > conclusion (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to
>> > > > > > > > > > > the
>> > > > > > > > > > > >> official
>> > > business
>> > > > of this
>> > > > > company
>> > > > > > shall be
>> > > > > > > > understood
>> > > > > > > > > as neither
>> > > > > > > > > > given
>> > > > > > > > > > > nor
>> > > > > > > > > > > >>
>> endorsed by
>> > it.
>> > > > IG is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a
>> > > > > > > > > > company
>> > > > > > > > > > > >>
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG
>> > > > > > > > > > Index
>> > > > > > > > > > > >> Limited
>> (a
>> > > > company registered in
>> > > > > > England and
>> > > > > > > > Wales,
>> > > > > > > > > company number
>> > > > > > > > > > > >>
>> 01190902).
>> > > > Registered address at
>> > > > > > Cannon Bridge
>> > > > > > > > House,
>> > > > > > > > > 25 Dowgate
>> > > > > > > > > > Hill,
>> > > > > > > > > > > >> London
>> EC4R
>> > 2YA.
>> > > > Both IG Markets
>> > > > > > Limited
>> > > > > > > (register
>> > > > > > > > > number 195355)
>> > > > > > > > > > and IG
>> > > > > > > > > > > >> Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > > authorised
>> > > > > > > > > and
>> > > > > > > > > > regulated by
>> > > > > > > > > > > the
>> > > > > > > > > > > >>
>> Financial
>> > > Conduct
>> > > > Authority.
>> > > > > > > > > > > >>
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > The
>> information
>> > > > contained in this
>> > > > > > email is strictly
>> > > > > > > > > confidential and
>> > > > > > > > > > for the use of
>> the
>> > > > addressee only, unless
>> > > > > > otherwise
>> > > > > > > > indicated.
>> > > > > > > > > If you are
>> > > > > > > > > > not the intended
>> > > > recipient, please do not
>> > > > > > read, copy, use
>> > > > > > > > or
>> > > > > > > > > disclose to
>> > > > > > > > > > others this
>> message or
>> > > any
>> > > > attachment.
>> > > > > > Please also notify
>> > > > > > > > the
>> > > > > > > > > sender by
>> > > > > > > > > > replying to this
>> email
>> > or
>> > > > by telephone
>> > > > > > (+44(020 7896
>> > > > > > > 0011)
>> > > > > > > > and
>> > > > > > > > > then delete
>> > > > > > > > > > the email and any
>> > copies
>> > > > of it. Opinions,
>> > > > > > conclusion
>> > > > > > > (etc)
>> > > > > > > > that
>> > > > > > > > > do not
>> > > > > > > > > > relate to the
>> official
>> > > > business of this
>> > > > > > company shall be
>> > > > > > > > > understood as
>> > > > > > > > > > neither given nor
>> > > endorsed
>> > > > by it. IG is a
>> > > > > > trading name of
>> > > > > > > > IG
>> > > > > > > > > Markets
>> > > > > > > > > > Limited (a
>> company
>> > > > registered in England
>> > > > > > and Wales,
>> > > > > > > company
>> > > > > > > > > number
>> > > > > > > > > > 04008957) and IG
>> Index
>> > > > Limited (a company
>> > > > > > registered in
>> > > > > > > > England
>> > > > > > > > > and Wales,
>> > > > > > > > > > company number
>> > 01190902).
>> > > > Registered
>> > > > > > address at Cannon
>> > > > > > > > Bridge
>> > > > > > > > > House, 25
>> > > > > > > > > > Dowgate Hill,
>> London
>> > EC4R
>> > > > 2YA. Both IG
>> > > > > > Markets Limited
>> > > > > > > > (register
>> > > > > > > > > number
>> > > > > > > > > > 195355) and IG
>> Index
>> > > > Limited (register
>> > > > > > number 114059) are
>> > > > > > > > > authorised and
>> > > > > > > > > > regulated by the
>> > > Financial
>> > > > Conduct
>> > > > > > Authority.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The information
>> contained in
>> > > > this email is
>> > > > > > strictly
>> > > > > > > confidential
>> > > > > > > > and
>> > > > > > > > > for the use of the
>> addressee
>> > only,
>> > > > unless otherwise
>> > > > > > indicated. If
>> > > > > > > > you are
>> > > > > > > > > not the intended recipient,
>> > please
>> > > > do not read,
>> > > > > > copy, use or
>> > > > > > > > disclose to
>> > > > > > > > > others this message or any
>> > > > attachment. Please also
>> > > > > > notify the
>> > > > > > > sender
>> > > > > > > > by
>> > > > > > > > > replying to this email or
>> by
>> > > > telephone (+44(020
>> > > > > 7896
>> > > > > > 0011) and then
>> > > > > > > > delete
>> > > > > > > > > the email and any copies
>> of it.
>> > > > Opinions,
>> > > > > conclusion
>> > > > > > (etc) that do
>> > > > > > > > not
>> > > > > > > > > relate to the official
>> business
>> > of
>> > > > this company
>> > > > > > shall be understood
>> > > > > > > > as
>> > > > > > > > > neither given nor endorsed
>> by it.
>> > > IG
>> > > > is a trading
>> > > > > > name of IG
>> > > > > > > Markets
>> > > > > > > > > Limited (a company
>> registered in
>> > > > England and Wales,
>> > > > > > company number
>> > > > > > > > > 04008957) and IG Index
>> Limited (a
>> > > > company
>> > > > > registered
>> > > > > > in England and
>> > > > > > > > Wales,
>> > > > > > > > > company number 01190902).
>> > > Registered
>> > > > address at
>> > > > > > Cannon Bridge
>> > > > > > > House,
>> > > > > > > > 25
>> > > > > > > > > Dowgate Hill, London EC4R
>> 2YA.
>> > Both
>> > > > IG Markets
>> > > > > > Limited (register
>> > > > > > > > number
>> > > > > > > > > 195355) and IG Index
>> Limited
>> > > > (register number
>> > > > > > 114059) are
>> > > > > > > authorised
>> > > > > > > > and
>> > > > > > > > > regulated by the Financial
>> > Conduct
>> > > > Authority.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > The information contained in
>> this email
>> > > is
>> > > > strictly
>> > > > > > confidential and for
>> > > > > > > > the use of the addressee only,
>> unless
>> > > > otherwise
>> > > > > indicated.
>> > > > > > If you are not
>> > > > > > > > the intended recipient, please
>> do not
>> > > > read, copy, use or
>> > > > > > disclose to
>> > > > > > > others
>> > > > > > > > this message or any attachment.
>> Please
>> > > > also notify the
>> > > > > > sender by replying
>> > > > > > > > to this email or by telephone
>> (+44(020
>> > > > 7896 0011) and
>> > > > > then
>> > > > > > delete the
>> > > > > > > email
>> > > > > > > > and any copies of it. Opinions,
>> > > conclusion
>> > > > (etc) that do
>> > > > > > not relate to
>> > > > > > > the
>> > > > > > > > official business of this
>> company shall
>> > > be
>> > > > understood as
>> > > > > > neither given
>> > > > > > > nor
>> > > > > > > > endorsed by it. IG is a trading
>> name of
>> > > IG
>> > > > Markets
>> > > > > Limited
>> > > > > > (a company
>> > > > > > > > registered in England and Wales,
>> > company
>> > > > number 04008957)
>> > > > > > and IG Index
>> > > > > > > > Limited (a company registered in
>> > England
>> > > > and Wales,
>> > > > > > company number
>> > > > > > > > 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill,
>> > > > > > > > London EC4R 2YA. Both IG Markets
>> > Limited
>> > > > (register number
>> > > > > > 195355) and IG
>> > > > > > > > Index Limited (register number
>> 114059)
>> > > are
>> > > > authorised and
>> > > > > > regulated by
>> > > > > > > the
>> > > > > > > > Financial Conduct Authority.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > The information contained in this email is
>> strictly
>> > > > confidential and
>> > > > > > for the use of the addressee only, unless
>> otherwise
>> > > > indicated. If you are
>> > > > > > not the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or disclose to
>> > > > > > others this message or any attachment. Please
>> also
>> > notify
>> > > > the sender by
>> > > > > > replying to this email or by telephone (+44(020
>> 7896
>> > > 0011)
>> > > > and then
>> > > > > delete
>> > > > > > the email and any copies of it. Opinions,
>> conclusion
>> > > (etc)
>> > > > that do not
>> > > > > > relate to the official business of this company
>> shall
>> > be
>> > > > understood as
>> > > > > > neither given nor endorsed by it. IG is a
>> trading name
>> > of
>> > > > IG Markets
>> > > > > > Limited (a company registered in England and
>> Wales,
>> > > > company number
>> > > > > > 04008957) and IG Index Limited (a company
>> registered in
>> > > > England and
>> > > > > Wales,
>> > > > > > company number 01190902). Registered address at
>> Cannon
>> > > > Bridge House, 25
>> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register number
>> > > > > > 195355) and IG Index Limited (register number
>> 114059)
>> > are
>> > > > authorised and
>> > > > > > regulated by the Financial Conduct Authority.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential
>> and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If
>> you are
>> > not
>> > > > the intended recipient, please do not read, copy, use or
>> disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then
>> delete the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company
>> number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number
>> 195355) and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and
>> regulated by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > >
>> >
>>
>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
> The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
--
Nacho - Ignacio Solis - isolis@igso.net
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
I think I'd like some opinions from some of the active chaps at LinkedIn so it's not just a two way decision (or any others for that matter).
Any opinions on the below or no real concerns / indifferent by anyone with this change?
Cheers
Mike
Sent using OWA for iPhone
________________________________________
From: Jason Gustafson <ja...@confluent.io>
Sent: Tuesday, February 21, 2017 6:59:38 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Jun's suggestion gives us the best of both worlds. Because the headers are
the last field in the record and you have the total size of the record,
then you can easily get the size of headers in bytes. So they can be copied
without parsing and we can do the lazy initialization without an extra cost.
The message set notion has changed in KIP-98 so that the message set can
have its own fields. See here for more detail:
https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
What we're proposing to change here is the schema for the message (aka
record) itself. We'll update that document depending on the outcome of the
discussion here.
-Jason
On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
wrote:
> This works whilst no other feature needs similar. I prefer to keep it
> contained/encapsulated it then allows the bytes of the headers to also be
> copied transferred as one without any parsing what so ever, but if you
> really feel strongly on the below ill go with it, just want to flag this
> concern.
>
> Btw, the below notation seems to be different to the notation used in
> previous kips like kip-32 which I took the notation from, where the length
> of the following bytes for byte arrays is clearly defined. So im 100% on
> what you propose could you define it in this original styled notation?
>
>
> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Jun, that's a good thought. Maybe then the full schema could look
> like
> this:
>
> Record =>
> Length => uintVar
> Attributes => int8
> TimestampDelta => intVar
> OffsetDelta => uintVar
> Key => bytes
> Value => bytes
> Headers => [HeaderKey HeaderValue]
> HeaderKey => string
> HeaderValue => bytes
>
> (where the bytes and string type use a variable length size field)
>
> Does that work for you, Michael?
>
> -Jason
>
> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Regarding the overhead of skipping the headers in the message.
> Another way
> > could be to add the headers at the end of the message after the
> value part.
> > This way, there is no need to skip the headers unless you really
> need it.
> > This will allow us to represent the headers explicitly as an array of
> > key/value pairs without the skipping overhead when headers are not
> needed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
> >
> > wrote:
> >
> > > Michael,
> > >
> > > Here's a link to the branch with the message format changes:
> > > https://github.com/confluentinc/kafka/tree/
> exactly-once-message-format.
> > > The
> > > class to modify to add headers is EosLogRecord (name will
> definitely
> > change
> > > prior to merging in trunk). This branch uses attributes to
> indicate null
> > > key and value. I really don't have a strong preference for or
> against
> > this
> > > as long as we're consistent. I do feel it's kind of a pity to need
> the
> > > signed version of the varint just to be able to indicate null. For
> > headers,
> > > it's an easier call because we don't actually need to represent
> null
> > > headers (using size zero will work fine).
> > >
> > > -Jason
> > >
> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > We can make it, implement Iterable<Header>, no concerns on this.
> > > >
> > > > Ive noted this on the KIP.
> > > >
> > > >
> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
> wrote:
> > > >
> > > > Btw Jason, can you share the branch with the VarInts
> helper/util
> > > class
> > > > that KIP-98 will be introducing and also will have the current
> modified
> > > > (key and value lengths) message, so we can look at, thanks.
> > > >
> > > > On 20/02/2017, 01:11, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > RE “ProducerRecord and ConsumerRecord are immutable
> > currently”. I
> > > > beg to differ on this, if I set a byte[] as my value of my
> record, I
> > can
> > > > later call value() and then alter the bytes in the array. As
> such it is
> > > > very much mutable.
> > > >
> > > > Headers also will anyhow by nature be mutable on
> produce, as
> > you
> > > > add to them.
> > > >
> > > >
> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
> jason@confluent.io>
> > > > wrote:
> > > >
> > > > Re; having a dedicated Headers object. I think it
> makes
> > > sense.
> > > > Maybe it
> > > > could implement Iterable<Header>?
> > > >
> > > > One thing I'm not too sure about is mutability. Both
> the
> > > > ProducerRecord and
> > > > ConsumerRecord are immutable currently. It would be
> nice to
> > > > keep it that
> > > > way. Perhaps Headers could itself be immutable and
> passed
> > > > through the
> > > > constructors?
> > > >
> > > > Just going through how this is, I don’t believe
> this would
> > > > provide much
> > > > > saving. If issue trying to resolve is around
> > object/garbage
> > > > creation of
> > > > > Map.Entry objects.
> > > >
> > > >
> > > > It's definitely better because we only allocate the
> space
> > we
> > > > will use, but
> > > > I assume most of the actual memory cost is in the
> header
> > data
> > > > itself which
> > > > we can't do much about. Maybe we can start with the
> simple
> > > > array
> > > > representation and optimize from there after we find
> it's
> > > > worthwhile to do
> > > > so? If we use the Headers object, we can change this
> later
> > > > without breaking
> > > > the API.
> > > >
> > > > -Jason
> > > >
> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> > > > radai.rosenblatt@gmail.com> wrote:
> > > >
> > > > > "(By the way, doesn't it feel a bit odd that we
> seem to
> > be
> > > > designing a
> > > > > feature which is optimized for people not using?)"
> > > > >
> > > > > very. this (i claim :-) ) is the result of very
> intense
> > > > opposition to the
> > > > > usefulness of the feature early on, and not a real
> design
> > > > goal
> > > > >
> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > Whilst if having an array of a Header[] would
> mean
> > > getting
> > > > this through,
> > > > > > im happy changing this, as already done.
> > > > > >
> > > > > > Just going through how this is, I don’t believe
> this
> > > > would provide much
> > > > > > saving. If issue trying to resolve is around
> > > > object/garbage creation of
> > > > > > Map.Entry objects.
> > > > > >
> > > > > > All you’re doing here is replacing the equiv of a
> > > > HashMap.Node
> > > > > (Map.Entry)
> > > > > > which simply holds reference to key and value
> objects
> > > with
> > > > a custom
> > > > > variant.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
> > > > Michael.Pearce@ig.com> wrote:
> > > > > >
> > > > > > On point 1 & 2 Ive updated KIP to show
> varints (and
> > > > removed the bit
> > > > > > flag). (on the assumption KIP 98 is getting the
> > agreement
> > > > the protocol is
> > > > > > moving from int32 to varInts as standard)
> > > > > >
> > > > > > On point 3 ive updated to use an array of
> Header
> > > > class, instead of a
> > > > > > MultiMap in the Headers class object
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > On points 1 and 2 I agree.
> > > > > >
> > > > > > This also affects kip-98, I should
> expect this
> > > > resolved before
> > > > > > that vote also passes. If it is accepted there
> (I’m
> > > > assuming this is
> > > > > > getting discussed on that KIP? As you’re driving
> the
> > move
> > > > to VarInts), I
> > > > > am
> > > > > > happy to make this KIP will simply follow suit to
> > > whatever
> > > > is agreed in
> > > > > > KIP-98.
> > > > > >
> > > > > > On 3) Agreed this is a simpler form, as
> long as
> > > no
> > > > one is
> > > > > > expecting hashmap lookup performance ( O(1) ), I
> still
> > > > would prefer an
> > > > > > encapsulated class, so if we find that holding
> it as an
> > > > array in future
> > > > > is
> > > > > > causing some perf issues, the internals are not
> exposed
> > > to
> > > > end users,
> > > > > > allowing the internal structure to move to a map.
> > > > > >
> > > > > > Can we compromise on?
> > > > > >
> > > > > > class ConsumerRecord<K, V> {
> > > > > > K key;
> > > > > > V value;
> > > > > > Headers headers;
> > > > > > }
> > > > > >
> > > > > > class Headers {
> > > > > > Header[] headers;
> > > > > >
> > > > > > add(String key, byte[] value)
> > > > > > Collection<byte[]> get(String key)
> > > > > > }
> > > > > >
> > > > > > class Header {
> > > > > > String key;
> > > > > > byte[] value;
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
> > > > jason@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > headers dont "leak" into
> application
> > code.
> > > > they are useful
> > > > > > to application
> > > > > > > code as well.
> > > > > >
> > > > > >
> > > > > > This is exactly what I have been
> trying to
> > > get
> > > > at. The use
> > > > > > cases documented
> > > > > > here are middleware:
> > > > > > https://cwiki.apache.org/
> > > > confluence/display/KAFKA/A+
> > > > > > Case+for+Kafka+Headers.
> > > > > > If headers are intended for the
> application
> > > > space as well,
> > > > > the
> > > > > > document
> > > > > > should be updated accordingly so
> that it is
> > > an
> > > > explicit
> > > > > design
> > > > > > goal and not
> > > > > > an unfortunate design byproduct.
> There may
> > be
> > > > some
> > > > > > disagreement on whether
> > > > > > it _should_ be a design goal, but it
> may
> > not
> > > > matter much
> > > > > since
> > > > > > the current
> > > > > > interceptor API is probably
> insufficient
> > for
> > > > middleware
> > > > > > applications (I
> > > > > > haven't thought of a way to do this
> that
> > > isn't
> > > > cumbersome).
> > > > > >
> > > > > > That aside, the three unresolved
> points for
> > > me
> > > > are the
> > > > > > following:
> > > > > >
> > > > > > 1. The use of varints. As a one-off
> for use
> > > > only with record
> > > > > > headers, the
> > > > > > case was weak, but if we are using
> them
> > > > throughout the
> > > > > message
> > > > > > format, then
> > > > > > we should do so here as well. The
> > additional
> > > > complexity is
> > > > > > minimal and
> > > > > > early performance testing fully
> justifies
> > > > their use.
> > > > > >
> > > > > > 2. If varints are used, the case for
> using
> > > > attributes to
> > > > > > indicate null
> > > > > > headers also becomes weak. We only
> add one
> > > > additional byte in
> > > > > > each message
> > > > > > if there are no headers. Whatever
> the case,
> > > > let us be
> > > > > > consistent with how
> > > > > > we treat null keys and values.
> > > > > >
> > > > > > 3. We have apparently agreed that the
> > broker
> > > > will validate
> > > > > > headers (please
> > > > > > add this to the KIP). That being the
> case,
> > I
> > > > would prefer to
> > > > > > use Kafka's
> > > > > > conventional format for arrays. The
> > downside
> > > > is that it takes
> > > > > > more work to
> > > > > > skip over the headers on the
> consumer,
> > though
> > > > it's unclear if
> > > > > > the cost
> > > > > > would matter in practice. Some
> concern was
> > > > previously
> > > > > > expressed about the
> > > > > > allocation of maps. An alternative
> would be
> > > to
> > > > use arrays,
> > > > > i.e.
> > > > > >
> > > > > > class ConsumerRecord<K, V> {
> > > > > > K key;
> > > > > > V value;
> > > > > > Header[] headers;
> > > > > > }
> > > > > >
> > > > > > class Header {
> > > > > > String key;
> > > > > > byte[] value;
> > > > > > }
> > > > > >
> > > > > > This would work nicely with the
> > conventional
> > > > array format and
> > > > > > my guess is
> > > > > > it would obviate the need do any lazy
> > > > initialization. If we
> > > > > > use the map as
> > > > > > is currently documented, then it is
> > possible
> > > > with either
> > > > > > representation to
> > > > > > slice the headers and initialize them
> > lazily.
> > > > Either way, it
> > > > > > might be a
> > > > > > good idea to use a separate object to
> > > > represent the headers
> > > > > in
> > > > > > case we need
> > > > > > to modify it in the future in some
> way.
> > > > > >
> > > > > > (By the way, doesn't it feel a bit
> odd that
> > > we
> > > > seem to be
> > > > > > designing a
> > > > > > feature which is optimized for
> people not
> > > > using?)
> > > > > >
> > > > > >
> > > > > > If we can resolve these points, then
> at
> > least
> > > > you will get my
> > > > > > vote.
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
> radai <
> > > > > > radai.rosenblatt@gmail.com> wrote:
> > > > > >
> > > > > > > headers dont "leak" into
> application
> > code.
> > > > they are useful
> > > > > > to application
> > > > > > > code as well.
> > > > > > > IIUC samze currently has headers
> "in-V"
> > and
> > > > would just
> > > > > > switch over to kafka
> > > > > > > headers if they exist.
> > > > > > > im sure plenty of other users of
> kafka
> > > would
> > > > have a use for
> > > > > > headers.
> > > > > > > im pretty sure use cases exist
> around
> > > > shuffling data
> > > > > > into/out-of kafka
> > > > > > > (kafka connect or equivalent) where
> > > metadata
> > > > from one end
> > > > > > could copied over
> > > > > > > to the other (S3, for example uses
> http
> > > > headers for
> > > > > > user-accessible
> > > > > > > metadata). it will be kafka client
> code
> > > > getting/setting
> > > > > > those headers. not
> > > > > > > an interceptor.
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
> Michael
> > > > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > For APM single event tracing,
> need
> > access
> > > > to the header
> > > > > at
> > > > > > the point of
> > > > > > > > processing on the processing
> thread.
> > > > > > > >
> > > > > > > > As such interceptors will not
> work/be
> > > > suitable for these,
> > > > > > due to the fact
> > > > > > > > they act on the ConsumerRecords
> as a
> > > > batch, before the
> > > > > > handling thread
> > > > > > > can
> > > > > > > > split out and process per
> message which
> > > is
> > > > the point
> > > > > these
> > > > > > tools will
> > > > > > > need
> > > > > > > > to continue to transaction
> tracing.
> > > > > > > >
> > > > > > > > Like wise tools and infra pieces
> will
> > > need
> > > > access to the
> > > > > > message outside
> > > > > > > > the interceptor realm.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 17/02/2017, 21:26, "Jason
> > Gustafson" <
> > > > > > jason@confluent.io> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > That’s exactly what we’re
> doing
> > the
> > > > headers are a
> > > > > > slice of bytes,
> > > > > > > > which
> > > > > > > > > then gets parsed later if
> needed,
> > > or
> > > > can be parsed
> > > > > > right away, the
> > > > > > > > headers
> > > > > > > > > is part of the protocol,
> so can
> > > > still be validated
> > > > > > if wanted.
> > > > > > > > > If you had a header count
> then
> > you
> > > > would have to go
> > > > > > through each
> > > > > > > > header
> > > > > > > > > key and value length value
> to
> > work
> > > > out how much to
> > > > > > skip to get to
> > > > > > > > say the
> > > > > > > > > value or any future
> component in
> > > the
> > > > message after
> > > > > > the headers.
> > > > > > > > Having it
> > > > > > > > > as a byte[] with length
> value
> > makes
> > > > this a lot
> > > > > > easier to skip.
> > > > > > > >
> > > > > > > >
> > > > > > > > So the broker will parse the
> > headers
> > > > and validate
> > > > > > them. Good. The
> > > > > > > only
> > > > > > > > reason remaining that I can
> see to
> > > > leave the headers
> > > > > > as a byte array
> > > > > > > > is to
> > > > > > > > make it easier for the
> client to
> > skip
> > > > past them. Are
> > > > > > we sure this is
> > > > > > > > not
> > > > > > > > premature optimization? Are
> there
> > any
> > > > performance
> > > > > > results which show
> > > > > > > > that
> > > > > > > > this is worthwhile?
> > > > > > > >
> > > > > > > > What’s the issue with
> exposing a
> > > > method getHeaders on
> > > > > > the
> > > > > > > > producer/consumer
> > > > > > > > > record? It doesn’t break
> > anything.
> > > > We don’t need
> > > > > any
> > > > > > special
> > > > > > > version.
> > > > > > > >
> > > > > > > >
> > > > > > > > See my previous explanation.
> What I
> > > am
> > > > trying to
> > > > > > resist is the
> > > > > > > headers
> > > > > > > > becoming a general
> > application-level
> > > > facility. The
> > > > > > primary use case
> > > > > > > as
> > > > > > > > far
> > > > > > > > as I can tell is middleware,
> which
> > is
> > > > the use case
> > > > > > that the
> > > > > > > > interceptors
> > > > > > > > are providing.
> > > > > > > >
> > > > > > > > Current batch consumer model
> and
> > > > consumer
> > > > > interceptors
> > > > > > don’t work
> > > > > > > where
> > > > > > > > > headers need to be acted
> on at
> > per
> > > > message level at
> > > > > > time of
> > > > > > > > processing,
> > > > > > > > > very case is APM (the core
> one),
> > > > where the header
> > > > > > value is used to
> > > > > > > > continue
> > > > > > > > > tracing.
> > > > > > > >
> > > > > > > >
> > > > > > > > I still don't understand the
> point
> > > > about batching.
> > > > > The
> > > > > > consumer
> > > > > > > > records are
> > > > > > > > exposed as a batch in the
> consumer
> > > > interceptor, but
> > > > > > you can still
> > > > > > > > iterate
> > > > > > > > them individually. It is no
> > different
> > > > for the
> > > > > consumer
> > > > > > API itself.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at
> 12:48 PM,
> > > > Michael Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Re:
> > > > > > > > >
> > > > > > > > > “ The point about
> creation of
> > > > maps seems
> > > > > > orthogonal. We can
> > > > > > > still
> > > > > > > > > represent
> > > > > > > > > the headers as a slice
> of
> > bytes
> > > > until the time
> > > > > > it is accessed.”
> > > > > > > > >
> > > > > > > > > That’s exactly what we’re
> doing
> > the
> > > > headers are a
> > > > > > slice of bytes,
> > > > > > > > which
> > > > > > > > > then gets parsed later if
> needed,
> > > or
> > > > can be parsed
> > > > > > right away, the
> > > > > > > > headers
> > > > > > > > > is part of the protocol,
> so can
> > > > still be validated
> > > > > > if wanted.
> > > > > > > > >
> > > > > > > > > If you had a header count
> then
> > you
> > > > would have to go
> > > > > > through each
> > > > > > > > header
> > > > > > > > > key and value length value
> to
> > work
> > > > out how much to
> > > > > > skip to get to
> > > > > > > > say the
> > > > > > > > > value or any future
> component in
> > > the
> > > > message after
> > > > > > the headers.
> > > > > > > > Having it
> > > > > > > > > as a byte[] with length
> value
> > makes
> > > > this a lot
> > > > > > easier to skip.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 17/02/2017, 20:37,
> "Michael
> > > > Pearce" <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > What’s the issue with
> > exposing
> > > a
> > > > method
> > > > > > getHeaders on the
> > > > > > > > > producer/consumer record?
> It
> > > doesn’t
> > > > break
> > > > > anything.
> > > > > > We don’t need
> > > > > > > > any
> > > > > > > > > special version.
> > > > > > > > >
> > > > > > > > > Current batch consumer
> model
> > > and
> > > > consumer
> > > > > > interceptors don’t
> > > > > > > work
> > > > > > > > > where headers need to be
> acted on
> > > at
> > > > per message
> > > > > > level at time of
> > > > > > > > > processing, very case is
> APM (the
> > > > core one), where
> > > > > > the header value
> > > > > > > > is used
> > > > > > > > > to continue tracing.
> JMS/HTTP etc
> > > > all expose these,
> > > > > > without issues.
> > > > > > > > I would
> > > > > > > > > NOT want to lock this down
> to
> > only
> > > > be usable
> > > > > > accessible via
> > > > > > > > interceptors,
> > > > > > > > > as we’d fail on one of the
> main
> > > > goals.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 17/02/2017, 20:21,
> "Jason
> > > > Gustafson" <
> > > > > > jason@confluent.io>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > The point about
> creation
> > of
> > > > maps seems
> > > > > > orthogonal. We can
> > > > > > > > still
> > > > > > > > > represent
> > > > > > > > > the headers as a
> slice of
> > > > bytes until the
> > > > > > time it is
> > > > > > > > accessed.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Yes exactly we
> have
> > > access
> > > > to the records
> > > > > > thus why the
> > > > > > > > header
> > > > > > > > > should be
> > > > > > > > > > accessible via
> it and
> > not
> > > > hidden for only
> > > > > > interceptors to
> > > > > > > > access.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > As explained
> above, the
> > > > point is to make
> > > > > the
> > > > > > intended usage
> > > > > > > > clear.
> > > > > > > > > Applications should
> > > continue
> > > > to rely on the
> > > > > > key/value
> > > > > > > fields
> > > > > > > > to
> > > > > > > > > serialize
> > > > > > > > > their own headers,
> and it
> > > > would be more
> > > > > > ideal if we can
> > > > > > > avoid
> > > > > > > > > leaking
> > > > > > > > > third-party
> headers into
> > > > applications. This
> > > > > > is difficult to
> > > > > > > > do
> > > > > > > > > with the
> > > > > > > > > current
> interceptors
> > > because
> > > > they share the
> > > > > > record objects
> > > > > > > > with
> > > > > > > > > the common
> > > > > > > > > API. What I had in
> mind
> > is
> > > > something like
> > > > > an
> > > > > > extension of
> > > > > > > the
> > > > > > > > > current
> > > > > > > > > interceptors which
> > exposed
> > > a
> > > > different
> > > > > > object (e.g.
> > > > > > > > > `RecordAndHeaders`).
> > > > > > > > > The challenge is
> for
> > > MM-like
> > > > use cases. Let
> > > > > > me see if I can
> > > > > > > > come
> > > > > > > > > up with a
> > > > > > > > > concrete proposal
> for
> > that
> > > > problem.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 17,
> 2017 at
> > > > 11:55 AM, Michael
> > > > > > Pearce <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I am happy to
> move the
> > > > definition of the
> > > > > > header into the
> > > > > > > > message
> > > > > > > > > body, but
> > > > > > > > > > would cause us
> not to
> > > lazy
> > > > > > initialise/parse the headers,
> > > > > > > as
> > > > > > > > > obviously, we
> > > > > > > > > > would have to
> traverse
> > > > these reading the
> > > > > > message.
> > > > > > > > > >
> > > > > > > > > > This was
> actually one
> > of
> > > > Jay’s requests:
> > > > > > > > > >
> > > > > > > > > > “ 2. I think
> we
> > should
> > > > think about
> > > > > > creating the Map
> > > > > > > > lazily to
> > > > > > > > > avoid
> > > > > > > > > > parsing out
> all the
> > > > headers into
> > > > > > little objects.
> > > > > > > > HashMaps
> > > > > > > > > themselves
> > > > > > > > > > are
> > > > > > > > > > kind of
> expensive
> > and
> > > > the consumer is
> > > > > > very perf
> > > > > > > > sensitive so
> > > > > > > > > and making
> > > > > > > > > > gazillions of
> > > hashmaps
> > > > that may or
> > > > > may
> > > > > > not get used
> > > > > > > is
> > > > > > > > > probably a bad
> > > > > > > > > > idea.”
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 17/02/2017,
> 19:44,
> > > > "Michael Pearce" <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Yes exactly
> we have
> > > > access to the
> > > > > > records thus why
> > > > > > > the
> > > > > > > > > header should
> > > > > > > > > > be accessible
> via it
> > and
> > > > not hidden for
> > > > > > only interceptors
> > > > > > > > to
> > > > > > > > > access.
> > > > > > > > > >
> > > > > > > > > > Sent using
> OWA for
> > > > iPhone
> > > > > > > > > >
> > > > ______________________________
> > > > > > __________
> > > > > > > > > > From: Magnus
> > > Edenhill <
> > > > > > magnus@edenhill.se>
> > > > > > > > > > Sent: Friday,
> > > February
> > > > 17, 2017
> > > > > > 7:34:49 PM
> > > > > > > > > > To:
> > > > dev@kafka.apache.org
> > > > > > > > > > Subject: Re:
> > > [DISCUSS]
> > > > KIP-82 - Add
> > > > > > Record Headers
> > > > > > > > > >
> > > > > > > > > > Big +1 on
> VarInts.
> > > > > > > > > > CPUs are
> fast,
> > memory
> > > > is slow.
> > > > > > > > > >
> > > > > > > > > > I agree with
> Jason
> > > > that we'll want to
> > > > > > continue
> > > > > > > > verifying
> > > > > > > > > messages,
> > > > > > > > > > including
> their
> > > > headers, so while I
> > > > > > appreciate the
> > > > > > > > idea of
> > > > > > > > > the opaque
> > > > > > > > > > header blob
> it
> > won't
> > > > be useful in
> > > > > > practice.
> > > > > > > > > >
> > > > > > > > > > /Magnus
> > > > > > > > > >
> > > > > > > > > > 2017-02-17
> 10:41
> > > > GMT-08:00 Jason
> > > > > > Gustafson <
> > > > > > > > > jason@confluent.io>:
> > > > > > > > > >
> > > > > > > > > > > Sorry, my
> > mistake.
> > > > The consumer
> > > > > > interceptor is per
> > > > > > > > batch,
> > > > > > > > > though I'm
> > > > > > > > > > not
> > > > > > > > > > > sure
> that's an
> > > > actual limitation
> > > > > > since you still
> > > > > > > have
> > > > > > > > > access to the
> > > > > > > > > > > individual
> > records.
> > > > > > > > > > >
> > > > > > > > > > > -Jason
> > > > > > > > > > >
> > > > > > > > > > > On Fri,
> Feb 17,
> > > 2017
> > > > at 10:39 AM,
> > > > > > Jason Gustafson <
> > > > > > > > > >
> jason@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Re
> headers as
> > > byte
> > > > array and
> > > > > > future use by
> > > > > > > broker.
> > > > > > > > This
> > > > > > > > > doesn't
> > > > > > > > > > take away
> > > > > > > > > > > >> from
> that at
> > > all.
> > > > Nor makes it
> > > > > > difficult at all
> > > > > > > > in my
> > > > > > > > > opinion.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yeah, I
> didn't
> > > say
> > > > it was
> > > > > > difficult, only
> > > > > > > awkward.
> > > > > > > > You
> > > > > > > > > wouldn't
> > > > > > > > > > write the
> > > > > > > > > > > > schema
> that way
> > > if
> > > > you were
> > > > > > planning to use it on
> > > > > > > > the
> > > > > > > > > brokers from
> > > > > > > > > > the
> > > > > > > > > > > >
> beginning. Note
> > > > also that one of
> > > > > > the benefits of
> > > > > > > > letting
> > > > > > > > > the broker
> > > > > > > > > > > >
> understand
> > > headers
> > > > is that it can
> > > > > > validate that
> > > > > > > > they are
> > > > > > > > > properly
> > > > > > > > > > > >
> formatted. If
> > > cost
> > > > is the only
> > > > > > concern, we should
> > > > > > > > > confirm its
> > > > > > > > > > impact
> > > > > > > > > > > > through
> > > > performance testing.
> > > > > > > > > > > >
> > > > > > > > > > > > One of
> the key
> > > use
> > > > cases requires
> > > > > > access on
> > > > > > > > consume at
> > > > > > > > > per
> > > > > > > > > > event/message
> > > > > > > > > > > >> level
> at the
> > > > point that message
> > > > > > is being
> > > > > > > > processed, as
> > > > > > > > > such the
> > > > > > > > > > batch
> > > > > > > > > > > >>
> interceptors
> > and
> > > > batch consume
> > > > > > api isn't
> > > > > > > > suitable. It
> > > > > > > > > needs to be
> > > > > > > > > > at the
> > > > > > > > > > > >> record
> level.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I'm not
> sure I
> > > > understand the
> > > > > > point about
> > > > > > > batching.
> > > > > > > > > Interceptors
> > > > > > > > > > are
> > > > > > > > > > > > applied
> > > > per-message, right?
> > > > > > > > > > > >
> > > > > > > > > > > > My
> intent on
> > > > interceptors is to
> > > > > > keep the usage of
> > > > > > > > headers
> > > > > > > > > > well-defined so
> > > > > > > > > > > > that
> they don't
> > > > start leaking
> > > > > > unnecessarily into
> > > > > > > > > applications. My
> > > > > > > > > > guess
> > > > > > > > > > > is
> > > > > > > > > > > > that it's
> > > probably
> > > > inevitable,
> > > > > but
> > > > > > isolating it
> > > > > > > in
> > > > > > > > the
> > > > > > > > > > interceptors
> would
> > > > > > > > > > > > at least
> give
> > > > people a second
> > > > > > thought before
> > > > > > > > deciding to
> > > > > > > > > use it.
> > > > > > > > > > The main
> > > > > > > > > > > >
> challenge in my
> > > > mind is figuring
> > > > > > out how an MM
> > > > > > > use
> > > > > > > > case
> > > > > > > > > would
> > > > > > > > > > work. It
> > > > > > > > > > > > would be
> more
> > > > cumbersome to
> > > > > > replicate headers
> > > > > > > > through an
> > > > > > > > > > interceptor,
> > > > > > > > > > > > though
> arguably
> > > MM
> > > > should be
> > > > > > working at a lower
> > > > > > > > level
> > > > > > > > > anyway.
> > > > > > > > > > > >
> > > > > > > > > > > > -Jason
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri,
> Feb 17,
> > > > 2017 at 10:16 AM,
> > > > > > Michael Pearce
> > > > > > > <
> > > > > > > > > >
> Michael.Pearce@ig.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Re
> headers
> > > > available on the
> > > > > > record va
> > > > > > > > interceptors only
> > > > > > > > > > > >>
> > > > > > > > > > > >> One of
> the key
> > > > use cases
> > > > > requires
> > > > > > access on
> > > > > > > > consume at
> > > > > > > > > per
> > > > > > > > > > event/message
> > > > > > > > > > > >> level
> at the
> > > > point that message
> > > > > > is being
> > > > > > > > processed, as
> > > > > > > > > such the
> > > > > > > > > > batch
> > > > > > > > > > > >>
> interceptors
> > and
> > > > batch consume
> > > > > > api isn't
> > > > > > > > suitable. It
> > > > > > > > > needs to be
> > > > > > > > > > at the
> > > > > > > > > > > >> record
> level.
> > > > > > > > > > > >>
> > > > > > > > > > > >> This
> anyhow is
> > > > similar to
> > > > > > jms/http/amqp where
> > > > > > > > headers
> > > > > > > > > are
> > > > > > > > > > available to
> > > > > > > > > > > >>
> consuming
> > > > applications.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Re
> headers as
> > > > byte array and
> > > > > > future use by
> > > > > > > > broker. This
> > > > > > > > > doesn't
> > > > > > > > > > take
> > > > > > > > > > > away
> > > > > > > > > > > >> from
> that at
> > > all.
> > > > Nor makes it
> > > > > > difficult at all
> > > > > > > > in my
> > > > > > > > > opinion.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> Sent
> using OWA
> > > > for iPhone
> > > > > > > > > > > >>
> > > > ______________________________
> > > > > > __________
> > > > > > > > > > > >> From:
> Jason
> > > > Gustafson <
> > > > > > jason@confluent.io>
> > > > > > > > > > > >> Sent:
> Friday,
> > > > February 17, 2017
> > > > > > 5:55:42 PM
> > > > > > > > > > > >> To:
> > > > dev@kafka.apache.org
> > > > > > > > > > > >>
> Subject: Re:
> > > > [DISCUSS] KIP-82 -
> > > > > > Add Record
> > > > > > > Headers
> > > > > > > > > > > >>
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Would
> you be
> > > > proposing in
> > > > > > KIP-98 to convert
> > > > > > > the
> > > > > > > > other
> > > > > > > > > message
> > > > > > > > > > int’s
> > > > > > > > > > > (key
> > > > > > > > > > > >> >
> length,
> > value
> > > > length) also to
> > > > > > varint to keep
> > > > > > > it
> > > > > > > > > uniform.
> > > > > > > > > > > >> > Also I
> > assume
> > > > there will be a
> > > > > > static or helper
> > > > > > > > method
> > > > > > > > > made to
> > > > > > > > > > > write/read
> > > > > > > > > > > >> > these
> in the
> > > > client and
> > > > > server.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> Yes,
> that is
> > > what
> > > > we are
> > > > > > proposing, so using
> > > > > > > > varints
> > > > > > > > > for headers
> > > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > >>
> consistent
> > with
> > > > the rest of the
> > > > > > message. We have
> > > > > > > > used
> > > > > > > > > static
> > > > > > > > > > helper
> > > > > > > > > > > >> methods
> > > > > > > > > > > >> in our
> > prototype
> > > > implementation.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The
> cost of
> > > > parsing, we want to
> > > > > > parse/interpret
> > > > > > > > the
> > > > > > > > > headers
> > > > > > > > > > lazily (this
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > a key
> point
> > > > brought up earlier
> > > > > > in discussions)
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> I'm a
> bit
> > > > skeptical of this. Has
> > > > > > anyone done the
> > > > > > > > > performance
> > > > > > > > > > testing? I
> > > > > > > > > > > >> can
> > > > > > > > > > > >> probably
> > > > implement it and test
> > > > > it
> > > > > > if no one else
> > > > > > > > has. I
> > > > > > > > > was also
> > > > > > > > > > under
> > > > > > > > > > > the
> > > > > > > > > > > >>
> impression
> > that
> > > > there may be use
> > > > > > cases down the
> > > > > > > > road
> > > > > > > > > where the
> > > > > > > > > > broker
> > > > > > > > > > > >> would
> > > > > > > > > > > >> need to
> > > interpret
> > > > headers. That
> > > > > > wouldn't be off
> > > > > > > > the
> > > > > > > > > table in the
> > > > > > > > > > future
> > > > > > > > > > > if
> > > > > > > > > > > >> it's
> > represented
> > > > as bytes, but
> > > > > it
> > > > > > would be quite
> > > > > > > > a bit
> > > > > > > > > more
> > > > > > > > > > awkward,
> > > > > > > > > > > >> right?
> > > > > > > > > > > >>
> > > > > > > > > > > >> By the
> way,
> > one
> > > > question I have
> > > > > > been wondering
> > > > > > > > about. My
> > > > > > > > > > understanding
> > > > > > > > > > > is
> > > > > > > > > > > >> that
> headers
> > are
> > > > primarily for
> > > > > > use cases where a
> > > > > > > > > third-party
> > > > > > > > > > components
> > > > > > > > > > > >> wants to
> > enrich
> > > > messages without
> > > > > > needing to
> > > > > > > > understand
> > > > > > > > > or modify
> > > > > > > > > > the
> > > > > > > > > > > >> schema
> > > > > > > > > > > >> of the
> message
> > > > key and value.
> > > > > For
> > > > > > the
> > > > > > > applications
> > > > > > > > > which directly
> > > > > > > > > > > produce
> > > > > > > > > > > >> and
> consume
> > the
> > > > messages and
> > > > > > control the
> > > > > > > key/value
> > > > > > > > > schema
> > > > > > > > > > directly, it
> > > > > > > > > > > >> seems
> we would
> > > > rather have them
> > > > > > implement
> > > > > > > headers
> > > > > > > > > directly in
> > > > > > > > > > their own
> > > > > > > > > > > >> schema.
> > > Supposing
> > > > for the sake
> > > > > of
> > > > > > argument that
> > > > > > > > it was
> > > > > > > > > possible,
> > > > > > > > > > my
> > > > > > > > > > > >>
> question is
> > > > whether it be
> > > > > > sufficient to expose
> > > > > > > the
> > > > > > > > > headers in the
> > > > > > > > > > > >>
> interceptor
> > API
> > > > and not in the
> > > > > > common API?
> > > > > > > > > > > >>
> > > > > > > > > > > >> -Jason
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Fri,
> Feb
> > 17,
> > > > 2017 at 3:26 AM,
> > > > > > Michael Pearce
> > > > > > > <
> > > > > > > > > >
> Michael.Pearce@ig.com>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > On
> the point
> > > of
> > > > varInts
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Would
> you be
> > > > proposing in
> > > > > > KIP-98 to convert
> > > > > > > the
> > > > > > > > other
> > > > > > > > > message
> > > > > > > > > > int’s
> > > > > > > > > > > (key
> > > > > > > > > > > >> >
> length,
> > value
> > > > length) also to
> > > > > > varint to keep
> > > > > > > it
> > > > > > > > > uniform.
> > > > > > > > > > > >> > Also I
> > assume
> > > > there will be a
> > > > > > static or helper
> > > > > > > > method
> > > > > > > > > made to
> > > > > > > > > > > write/read
> > > > > > > > > > > >> > these
> in the
> > > > client and
> > > > > server.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Cheers
> > > > > > > > > > > >> > Mike
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On
> > 17/02/2017,
> > > > 11:22, "Michael
> > > > > > Pearce" <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> On the
> > > > point re: headers
> > > > > in
> > > > > > the message
> > > > > > > > protocol
> > > > > > > > > being a
> > > > > > > > > > byte
> > > > > > > > > > > array
> > > > > > > > > > > >> > and
> not a
> > > count
> > > > of elements
> > > > > > followed by the
> > > > > > > > elements.
> > > > > > > > > Again
> > > > > > > > > > this was
> > > > > > > > > > > >> >
> > > > discussed/argued previously.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> It was
> > > > agreed on for a few
> > > > > > reasons some of
> > > > > > > > which
> > > > > > > > > you have
> > > > > > > > > > > obviously
> > > > > > > > > > > >> >
> picked up
> > on:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> Broker
> > is
> > > > able to pass it
> > > > > > through opaquely
> > > > > > > > > > > >> >
> The cost
> > > of
> > > > parsing, we
> > > > > > want to
> > > > > > > > parse/interpret
> > > > > > > > > the headers
> > > > > > > > > > lazily
> > > > > > > > > > > >> > (this
> is a
> > key
> > > > point brought
> > > > > up
> > > > > > earlier in
> > > > > > > > > discussions)
> > > > > > > > > > > >> >
> Headers
> > > can
> > > > be copied from
> > > > > > consumer record
> > > > > > > > to
> > > > > > > > > producer
> > > > > > > > > > record (aka
> > > > > > > > > > > >> > mirror
> > makers
> > > > etc) without
> > > > > > parsing if no
> > > > > > > > changes are
> > > > > > > > > being made
> > > > > > > > > > or
> > > > > > > > > > > being
> > > > > > > > > > > >> >
> looked at.
> > > > > > > > > > > >> >
> Keeps
> > the
> > > > broker agnostic
> > > > > > to the format
> > > > > > > > > > > >> >
> You need
> > > an
> > > > int32 either
> > > > > > for the byte size
> > > > > > > > of the
> > > > > > > > > headers,
> > > > > > > > > > or for
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > count
> of
> > > > elements, so
> > > > > overheads
> > > > > > are the same,
> > > > > > > > but
> > > > > > > > > with going
> > > > > > > > > > with an
> > > > > > > > > > > >> opaque
> > > > > > > > > > > >> > byte
> array
> > has
> > > > the above
> > > > > > advantages.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> Cheers
> > > > > > > > > > > >> >
> Mike
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On
> > > > 17/02/2017, 02:50,
> > > > > > "Jason Gustafson" <
> > > > > > > > > jason@confluent.io
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > Sorry,
> > > > should have
> > > > > > noted that the
> > > > > > > > performance
> > > > > > > > > testing
> > > > > > > > > > was done
> > > > > > > > > > > >> > using
> the
> > > > > > > > > > > >> >
> > > > producer performance
> > > > > > tool shipped with
> > > > > > > > Kafka.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > -Jason
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> On
> > > Thu,
> > > > Feb 16, 2017
> > > > > at
> > > > > > 6:44 PM, Jason
> > > > > > > > > Gustafson <
> > > > > > > > > > > >> >
> > > > jason@confluent.io> wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> >
> > Hey
> > > > Nacho,
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > I've
> > > > compared
> > > > > > performance of our
> > > > > > > > KIP-98
> > > > > > > > > > implementation
> with
> > > > > > > > > > > >> and
> > > > > > > > > > > >> >
> without
> > > > > > > > > > > >> >
> >
> > > > varints. For
> > > > > messages
> > > > > > around 128
> > > > > > > > bytes, we
> > > > > > > > > see an
> > > > > > > > > > increase
> > > > > > > > > > > in
> > > > > > > > > > > >> >
> throughput
> > of
> > > > > > > > > > > >> >
> >
> > > about
> > > > 30% using the
> > > > > > default
> > > > > > > > configuration
> > > > > > > > > settings.
> > > > > > > > > > At 256
> > > > > > > > > > > >> >
> bytes, the
> > > > > > > > > > > >> >
> >
> > > > increase is around
> > > > > > 16%. Obviously
> > > > > > > the
> > > > > > > > > performance
> > > > > > > > > > converge
> > > > > > > > > > > as
> > > > > > > > > > > >> >
> messages get
> > > > > > > > > > > >> >
> >
> > > > larger, but it seems
> > > > > > well worth the
> > > > > > > > cost.
> > > > > > > > > Note that
> > > > > > > > > > we are
> > > > > > > > > > > >> also
> > > > > > > > > > > >> >
> seeing a
> > > > > > > > > > > >> >
> >
> > > > substantial
> > > > > > performance increase
> > > > > > > > against
> > > > > > > > > trunk
> > > > > > > > > > primarily
> > > > > > > > > > > >> because
> > > > > > > > > > > >> > of the
> > > > > > > > > > > >> >
> >
> > much
> > > > more efficient
> > > > > > packing that
> > > > > > > > varints
> > > > > > > > > provide us.
> > > > > > > > > > > Anything
> > > > > > > > > > > >> >
> adding to
> > > > > > > > > > > >> >
> >
> > > > message overhead,
> > > > > > such as record
> > > > > > > > headers,
> > > > > > > > > would only
> > > > > > > > > > > increase
> > > > > > > > > > > >> > the
> relative
> > > > > > > > > > > >> >
> >
> > > > difference. (Of
> > > > > > course take these
> > > > > > > > numbers
> > > > > > > > > with a
> > > > > > > > > > grain of
> > > > > > > > > > > salt
> > > > > > > > > > > >> > since
> I have
> > > > > > > > > > > >> >
> >
> > only
> > > > used the
> > > > > default
> > > > > > settings with
> > > > > > > > both
> > > > > > > > > the producer
> > > > > > > > > > and
> > > > > > > > > > > >> broker
> > > > > > > > > > > >> > on my
> > > > > > > > > > > >> >
> >
> > > local
> > > > machine. We
> > > > > > intend to provide
> > > > > > > > more
> > > > > > > > > extensive
> > > > > > > > > > > performance
> > > > > > > > > > > >> >
> details as
> > > > > > > > > > > >> >
> >
> > part
> > > > of the work for
> > > > > > KIP-98.)
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > The
> > > > implementation
> > > > > we
> > > > > > are using is
> > > > > > > > from
> > > > > > > > > protobuf (
> > > > > > > > > > > >> >
> >
> > > > > > https://developers.google.com/
> > > > > > > > > > >
> > > > protocol-buffers/docs/encoding
> > > > > > > > > > > >> ),
> > > > > > > > > > > >> > which
> is
> > > > > > > > > > > >> >
> >
> > also
> > > > used in HBase.
> > > > > > It is trivial to
> > > > > > > > > implement and as
> > > > > > > > > > far
> > > > > > > > > > > as I
> > > > > > > > > > > >> > know
> doesn't
> > > > > > > > > > > >> >
> >
> > > > suffer from the
> > > > > > aliasing problem you
> > > > > > > > are
> > > > > > > > > describing. I
> > > > > > > > > > > checked
> > > > > > > > > > > >> > with
> Magnus
> > > > > > > > > > > >> >
> >
> > (the
> > > > author of
> > > > > > librdkafka) and he
> > > > > > > > agreed
> > > > > > > > > that the
> > > > > > > > > > savings
> > > > > > > > > > > >> seemed
> > > > > > > > > > > >> > worth
> the
> > > > > > > > > > > >> >
> >
> > cost
> > > > of
> > > > > > implementation.
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > > -Jason
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> > On
> > > > Thu, Feb 16, 2017
> > > > > > at 4:32 PM,
> > > > > > > > Ignacio
> > > > > > > > > Solis <
> > > > > > > > > > > >>
> > isolis@igso.net
> > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >>
> > > > -VarInts
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > I'm
> > > > one of the
> > > > > > people (if not the
> > > > > > > > most)
> > > > > > > > > opposed to
> > > > > > > > > > VarInts.
> > > > > > > > > > > >> >
> VarInts
> > > > > > > > > > > >> >
> >>
> > > have
> > > > a place, but
> > > > > > this is not it.
> > > > > > > > (We
> > > > > > > > > had a large
> > > > > > > > > > > >>
> discussion
> > > > > > > > > > > >> > about
> > > > > > > > > > > >> >
> >>
> > > them
> > > > at the
> > > > > > beginning of KIP-82
> > > > > > > time)
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > If
> > > > anybody has real
> > > > > > life
> > > > > > > performance
> > > > > > > > > numbers of
> > > > > > > > > > VarInts
> > > > > > > > > > > >> >
> improving
> > > > > > > > > > > >> >
> >>
> > > > things or
> > > > > > significantly reducing
> > > > > > > > resources
> > > > > > > > > I would
> > > > > > > > > > like to
> > > > > > > > > > > >> know
> > > > > > > > > > > >> > what
> > > > > > > > > > > >> >
> >>
> > > that
> > > > case may be.
> > > > > > Yes, you can save
> > > > > > > > some
> > > > > > > > > bytes here
> > > > > > > > > > and
> > > > > > > > > > > >> there,
> > > > > > > > > > > >> > but
> > > > > > > > > > > >> >
> >>
> > > this
> > > > is probably
> > > > > > insignificant to
> > > > > > > the
> > > > > > > > > overall system
> > > > > > > > > > > behavior
> > > > > > > > > > > >> > and
> > > > > > > > > > > >> >
> >>
> > > > storage
> > > > > > requirements. -- I say
> > > > > > > this
> > > > > > > > with
> > > > > > > > > respect to
> > > > > > > > > > using
> > > > > > > > > > > >> >
> VarInts in
> > > > > > > > > > > >> >
> >>
> > the
> > > > protocol
> > > > > itself,
> > > > > > not as part of
> > > > > > > > the
> > > > > > > > > data.
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > VarInts require you
> > > > > > to parse the
> > > > > > > Int
> > > > > > > > > before using it
> > > > > > > > > > and
> > > > > > > > > > > >> >
> depending on
> > > > > > > > > > > >> >
> >>
> > the
> > > > encoding they
> > > > > > can suffer from
> > > > > > > > aliasing
> > > > > > > > > (multiple
> > > > > > > > > > > >> >
> > > representations
> > > > > > > > > > > >> >
> >>
> > for
> > > > the same
> > > > > value).
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > Why
> > > > add complexity?
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > Nacho
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > On
> > > > Thu, Feb 16,
> > > > > 2017
> > > > > > at 10:29 AM,
> > > > > > > > Colin
> > > > > > > > > McCabe <
> > > > > > > > > > > >> >
> > > > cmccabe@apache.org>
> > > > > > > > > > > >> >
> >>
> > > > wrote:
> > > > > > > > > > > >> >
> >> >
> > > +1
> > > > for varints
> > > > > > here-- it would
> > > > > > > save
> > > > > > > > > quite a bit of
> > > > > > > > > > space.
> > > > > > > > > > > >> > They
> are
> > > > > > > > > > > >> >
> >> >
> > > > pretty quick to
> > > > > > implement as
> > > > > > > well.
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >>
> > > I
> > > > think it makes
> > > > > > sense for values
> > > > > > > > to be
> > > > > > > > > byte
> > > > > > > > > > arrays.
> > > > > > > > > > > Users
> > > > > > > > > > > >> > might
> want
> > > > > > > > > > > >> >
> >> >
> > > to
> > > > attach
> > > > > > arbitrary payloads;
> > > > > > > they
> > > > > > > > > shouldn't be
> > > > > > > > > > forced to
> > > > > > > > > > > >> >
> serialize
> > > > > > > > > > > >> >
> >> >
> > > > everything to
> > > > > Java
> > > > > > strings.
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >> >
> > > > best,
> > > > > > > > > > > >> >
> >> >
> > > > Colin
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >> >
> > > On
> > > > Thu, Feb 16,
> > > > > > 2017, at 09:52,
> > > > > > > > Jason
> > > > > > > > > Gustafson
> > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > Hey Michael,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > Hmm, I guess the
> > > > > > point of
> > > > > > > > representing
> > > > > > > > > it as
> > > > > > > > > > bytes is to
> > > > > > > > > > > >> > allow
> the
> > > > > > > > > > > >> >
> >>
> > > > broker
> > > > > > > > > > > >> >
> >>
> > >>
> > > > to pass it
> > > > > > through opaquely? Is
> > > > > > > > the
> > > > > > > > > cost of
> > > > > > > > > > parsing
> > > > > > > > > > > them a
> > > > > > > > > > > >> >
> concern, or
> > > > > > > > > > > >> >
> >>
> > >>
> > > > are
> > > > > > > > > > > >> >
> >>
> > >>
> > > > we simply trying
> > > > > > to ensure that
> > > > > > > > the
> > > > > > > > > broker stays
> > > > > > > > > > > agnostic
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> >
> >>
> > > > format?
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > On varints, I
> > > > > > think adding
> > > > > > > > support for
> > > > > > > > > them makes
> > > > > > > > > > less
> > > > > > > > > > > >> sense
> > > > > > > > > > > >> > for an
> > > > > > > > > > > >> >
> >>
> > >>
> > > > isolated use
> > > > > > case, but as part
> > > > > > > of
> > > > > > > > a
> > > > > > > > > more holistic
> > > > > > > > > > change
> > > > > > > > > > > >> > (such
> as
> > what
> > > > > > > > > > > >> >
> >>
> > we
> > > > > > > > > > > >> >
> >>
> > >>
> > > > have proposed in
> > > > > > KIP-98), I
> > > > > > > think
> > > > > > > > they
> > > > > > > > > are
> > > > > > > > > > justifiable.
> > > > > > > > > > > If
> > > > > > > > > > > >> > we
> add them,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > then the need to
> > > > > > use attributes
> > > > > > > > becomes
> > > > > > > > > quite a
> > > > > > > > > > bit
> > > > > > > > > > > >> weaker,
> > > > > > > > > > > >> >
> right? The
> > > > > > > > > > > >> >
> >>
> > >>
> > > > other thing I
> > > > > > find slightly odd
> > > > > > > > is the
> > > > > > > > > fact that
> > > > > > > > > > null
> > > > > > > > > > > >> >
> headers has
> > no
> > > > > > > > > > > >> >
> >>
> > >>
> > > > actual
> > > > > > > > > > > >> >
> >>
> > >>
> > > > semantic meaning
> > > > > > for the message
> > > > > > > > > (unlike null
> > > > > > > > > > keys and
> > > > > > > > > > > >> >
> values). It
> > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > just
> > > > > > > > > > > >> >
> >>
> > >>
> > > a
> > > > space
> > > > > > optimization. It seems a
> > > > > > > > bit
> > > > > > > > > better to
> > > > > > > > > > always
> > > > > > > > > > > use
> > > > > > > > > > > >> > size
> 0 to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > indicate having
> > > > > > no headers.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > Overall, the
> > > > > main
> > > > > > point is
> > > > > > > > ensuring
> > > > > > > > > that the
> > > > > > > > > > message
> > > > > > > > > > > >> schema
> > > > > > > > > > > >> >
> remains
> > > > > > > > > > > >> >
> >>
> > >>
> > > > consistent,
> > > > > > either within the
> > > > > > > > larger
> > > > > > > > > protocol, or
> > > > > > > > > > at a
> > > > > > > > > > > >> >
> minimum
> > within
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > message itself.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > -Jason
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > On Thu, Feb 16,
> > > > > > 2017 at 6:39 AM,
> > > > > > > > > Michael Pearce <
> > > > > > > > > > > >> >
> > > > Michael.Pearce@ig.com
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Hi Jason,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On point 1) in
> > > > > > the message
> > > > > > > > protocol
> > > > > > > > > the headers
> > > > > > > > > > are
> > > > > > > > > > > >> simply
> > > > > > > > > > > >> > a byte
> > > > > > > > > > > >> >
> >>
> > > > array,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > as like the
> > > > > key
> > > > > > or value, this
> > > > > > > > is to
> > > > > > > > > clearly
> > > > > > > > > > demarcate
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >
> header in
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > core message.
> > > > > > Then the header
> > > > > > > > byte
> > > > > > > > > array in the
> > > > > > > > > > core
> > > > > > > > > > > >> >
> message is
> > an
> > > > > > > > > > > >> >
> >>
> > > > array of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > key, value
> > > > > > pairs. This is what
> > > > > > > > it is
> > > > > > > > > denoting.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Then this
> > > > > would
> > > > > > be I guess in
> > > > > > > > the
> > > > > > > > > given
> > > > > > > > > > notation:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Headers =>
> > > > > > [KeyLength, Key,
> > > > > > > > > ValueLength, Value]
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > KeyLength
> > > > > > => int32
> > > > > > > > > <-----------------NEW
> > > > > > > > > > size of
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> byte[] of
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > serialised key
> > > > > > value
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Key =>
> > > > > bytes
> > > > > > > > > <---------------------- NEW
> > > > > > > > > > > serialised
> > > > > > > > > > > >> > string
> > (UTF8)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > bytes of the
> > > > > > header key
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > ValueLength
> > > > > > => int32
> > > > > > > > > <-------------- NEW
> > > > > > > > > > size of
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> byte[] of
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > serialised
> > > > > > header value
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Value =>
> > > > > > bytes
> > > > > > > > > <-------------------- NEW
> > > > > > > > > > > serialised
> > > > > > > > > > > >> > form
> of the
> > > > > > > > > > > >> >
> >>
> > > > header
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > value
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > The key length
> > > > > > and value
> > > > > > > length
> > > > > > > > is
> > > > > > > > > matching the
> > > > > > > > > > way
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> protocol is
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > defined in the
> > > > > > core message
> > > > > > > > currently.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On point 2)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Var sized
> > > > > ints,
> > > > > > this was
> > > > > > > > discussed
> > > > > > > > > much earlier
> > > > > > > > > > on, in
> > > > > > > > > > > >> > fact
> I had
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > suggested it
> > > > > > myself (with
> > > > > > > Hadoop
> > > > > > > > > references),
> > > > > > > > > > the
> > > > > > > > > > > >> >
> complexity
> > of
> > > > this
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > compared to
> > > > > > having a simpler
> > > > > > > > protocol
> > > > > > > > > was
> > > > > > > > > > argued and
> > > > > > > > > > > >> >
> agreed it
> > > > > > > > > > > >> >
> >>
> > > > wasn’t worth
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the complexity
> > > > > > as all other
> > > > > > > > clients
> > > > > > > > > in other
> > > > > > > > > > languages
> > > > > > > > > > > >> > would
> need
> > to
> > > > > > > > > > > >> >
> >>
> > > > ensure
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > theyre using
> > > > > > the right var
> > > > > > > size
> > > > > > > > > algorithm, as
> > > > > > > > > > there
> > > > > > > > > > > is a
> > > > > > > > > > > >> > few.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On point 3)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > We did the
> > > > > > attributes,
> > > > > > > optional
> > > > > > > > > approach as
> > > > > > > > > > originally
> > > > > > > > > > > >> > there
> was
> > > > > > > > > > > >> >
> >>
> > > > marked
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > concern that
> > > > > > headers would
> > > > > > > > cause a
> > > > > > > > > message size
> > > > > > > > > > > overhead
> > > > > > > > > > > >> > for
> others,
> > > > > > > > > > > >> >
> >>
> > who
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > don’t want
> > > > > > them. As such this
> > > > > > > > is the
> > > > > > > > > clean
> > > > > > > > > > solution to
> > > > > > > > > > > >> >
> achieve
> > that.
> > > > > > > > > > > >> >
> >>
> > If
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > that no longer
> > > > > > holds, and we
> > > > > > > > don’t
> > > > > > > > > care that we
> > > > > > > > > > add
> > > > > > > > > > > >> 4bytes
> > > > > > > > > > > >> >
> overhead,
> > > > > > > > > > > >> >
> >>
> > > then
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > im happy to
> > > > > > remove.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > I’m personally
> > > > > > in favour of
> > > > > > > > keeping
> > > > > > > > > the message
> > > > > > > > > > as
> > > > > > > > > > > small
> > > > > > > > > > > >> > as
> possible
> > > > > > > > > > > >> >
> >>
> > so
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > people don’t
> > > > > > get shocks in
> > > > > > > perf
> > > > > > > > and
> > > > > > > > > throughputs
> > > > > > > > > > dues
> > > > > > > > > > > to
> > > > > > > > > > > >> >
> message
> > size,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > unless they
> > > > > > actively use the
> > > > > > > > feature,
> > > > > > > > > as such I
> > > > > > > > > > do
> > > > > > > > > > > >> prefer
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> >
> >>
> > > > attribute bit
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > wise feature
> > > > > > flag approach
> > > > > > > > myself.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On 16/02/2017,
> > > > > > 05:40, "Jason
> > > > > > > > > Gustafson" <
> > > > > > > > > > > >> >
> > > > jason@confluent.io> wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > We have
> > > > > > proposed a few
> > > > > > > > > significant changes
> > > > > > > > > > to the
> > > > > > > > > > > >> >
> message
> > format
> > > > > > > > > > > >> >
> >>
> > in
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > KIP-98
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > which now
> > > > > > seems likely to
> > > > > > > > pass
> > > > > > > > > (perhaps
> > > > > > > > > > with some
> > > > > > > > > > > >> >
> iterations
> > on
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > implementation details).
> > > > > > > It
> > > > > > > > would
> > > > > > > > > be good
> > > > > > > > > > to try
> > > > > > > > > > > and
> > > > > > > > > > > >> >
> coordinate
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > changes
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > in both of
> > > > > > the proposals
> > > > > > > to
> > > > > > > > make
> > > > > > > > > sure they
> > > > > > > > > > are
> > > > > > > > > > > >> >
> consistent
> > and
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > compatible.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > I think
> > > > > > using the
> > > > > > > > attributes to
> > > > > > > > > indicate
> > > > > > > > > > null
> > > > > > > > > > > >> headers
> > > > > > > > > > > >> > is a
> > > > > > > > > > > >> >
> >>
> > > > reasonable
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > approach.
> > > > > > We have proposed
> > > > > > > > to do
> > > > > > > > > the same
> > > > > > > > > > thing
> > > > > > > > > > > for
> > > > > > > > > > > >> > the
> message
> > > > > > > > > > > >> >
> >>
> > key
> > > > and
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > value.
> > > > > That
> > > > > > said, I
> > > > > > > > sympathize
> > > > > > > > > with Jay's
> > > > > > > > > > > argument.
> > > > > > > > > > > >> > Having
> > > > > > > > > > > >> >
> >>
> > > > multiple
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > ways to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > specify a
> > > > > > null value
> > > > > > > > increases
> > > > > > > > > the overall
> > > > > > > > > > > >>
> complexity
> > > > > > > > > > > >> > of the
> > > > > > > > > > > >> >
> >>
> > > > protocol.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > You
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > can see
> > > > > > this just from the
> > > > > > > > fact
> > > > > > > > > that you
> > > > > > > > > > need the
> > > > > > > > > > > >> > extra
> > verbiage
> > > > > > > > > > > >> >
> >>
> > in
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > protocol
> > > > > > specification in
> > > > > > > > this
> > > > > > > > > KIP and in
> > > > > > > > > > KIP-98
> > > > > > > > > > > to
> > > > > > > > > > > >> >
> describe the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > dependence
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > between
> > > > > the
> > > > > > fields and the
> > > > > > > > > attributes. It
> > > > > > > > > > seems
> > > > > > > > > > > >> like a
> > > > > > > > > > > >> >
> slippery
> > > > > > > > > > > >> >
> >>
> > > slope
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > if
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > you start
> > > > > > allowing
> > > > > > > different
> > > > > > > > > request types
> > > > > > > > > > to
> > > > > > > > > > > >> >
> implement
> > the
> > > > > > > > > > > >> >
> >>
> > > > protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > specification differently.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > You can
> > > > > > also argue that
> > > > > > > the
> > > > > > > > > messages
> > > > > > > > > > already are
> > > > > > > > > > > and
> > > > > > > > > > > >> > are
> likely
> > > > > > > > > > > >> >
> >>
> > to
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > remain a
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > special
> > > > > > case. For example,
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > currently no
> > > > > > > > > > > >> >
> generality
> > in
> > > > > > > > > > > >> >
> >>
> > how
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > compressed
> > > > > > message sets
> > > > > > > are
> > > > > > > > > represented
> > > > > > > > > > that would
> > > > > > > > > > > >> be
> > > > > > > > > > > >> >
> applicable
> > > > > > > > > > > >> >
> >>
> > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > other
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > request
> > > > > > types. Some might
> > > > > > > > see this
> > > > > > > > > > divergence as
> > > > > > > > > > > an
> > > > > > > > > > > >> >
> unfortunate
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > deficiency
> > > > > > which should be
> > > > > > > > fixed;
> > > > > > > > > others
> > > > > > > > > > might see
> > > > > > > > > > > >> it
> > > > > > > > > > > >> > as
> sort of
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > inevitability of needing
> > > > > > > to
> > > > > > > > > optimize where
> > > > > > > > > > it
> > > > > > > > > > > counts
> > > > > > > > > > > >> > most.
> I'm
> > > > > > > > > > > >> >
> >>
> > > > probably
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > somewhere
> > > > > > in between, but
> > > > > > > I
> > > > > > > > think
> > > > > > > > > we
> > > > > > > > > > probably all
> > > > > > > > > > > >> > share
> the
> > > > > > > > > > > >> >
> >>
> > > > intuition
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > > protocol should be
> > > > > > > kept
> > > > > > > > as
> > > > > > > > > consistent as
> > > > > > > > > > > >>
> possible.
> > > > > > > > > > > >> > With
> that
> > > > > > > > > > > >> >
> >>
> > in
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > mind,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > here are a
> > > > > > few comments:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > 1. One
> > > > > > thing I found a
> > > > > > > > little odd
> > > > > > > > > when
> > > > > > > > > > reading the
> > > > > > > > > > > >> >
> current
> > > > > > > > > > > >> >
> >>
> > > > proposal is
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > headers
> > > > > > are both
> > > > > > > > represented
> > > > > > > > > as an
> > > > > > > > > > array of
> > > > > > > > > > > >> bytes
> > > > > > > > > > > >> > and
> as an
> > > > > > > > > > > >> >
> >>
> > > array
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > of
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > key/value
> > > > > > pairs. I'd
> > > > > > > > probably
> > > > > > > > > suggest
> > > > > > > > > > something
> > > > > > > > > > > like
> > > > > > > > > > > >> > this:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Headers =>
> > > > > > [HeaderKey
> > > > > > > > HeaderValue]
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > HeaderKey
> > > > > > => String
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > HeaderValue => Bytes
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > An array
> > > > > in
> > > > > > the Kafka
> > > > > > > > protocol is
> > > > > > > > > > represented as a
> > > > > > > > > > > >> > 4-byte
> > integer
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > indicating
> > > > > > the number of
> > > > > > > > elements
> > > > > > > > > in the
> > > > > > > > > > array
> > > > > > > > > > > >> >
> followed by
> > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > serialization of the
> > > > > > > > elements.
> > > > > > > > > Unless I'm
> > > > > > > > > > > >> >
> > > > misunderstanding, what
> > > > > > > > > > > >> >
> >>
> > you
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > have
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > instead is
> > > > > > the total size
> > > > > > > > of the
> > > > > > > > > headers in
> > > > > > > > > > bytes
> > > > > > > > > > > >> >
> followed by
> > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > elements.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > I'm not
> > > > > > sure I see any
> > > > > > > > reason for
> > > > > > > > > this
> > > > > > > > > > > >>
> inconsistency.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > 2. In
> > > > > > KIP-98, we've
> > > > > > > > introduced
> > > > > > > > > > variable-length
> > > > > > > > > > > >> integer
> > > > > > > > > > > >> >
> fields.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Effectively,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > we've
> > > > > > enriched (or
> > > > > > > > "complicated"
> > > > > > > > > as Jay
> > > > > > > > > > might say
> > > > > > > > > > > ;)
> > > > > > > > > > > >> > the
> protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > specification to include
> > > > > > > the
> > > > > > > > > following
> > > > > > > > > > types:
> > > > > > > > > > > >> VarInt,
> > > > > > > > > > > >> >
> VarLong,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > UnsignedVarInt and
> > > > > > > > > UnsignedVarLong.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Along with
> > > > > > these
> > > > > > > > primitives, we
> > > > > > > > > could
> > > > > > > > > > introduce
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> following
> > > > > > > > > > > >> >
> >>
> > > > types:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > VarSizeArray =>
> > > > > > > > NumberOfItems
> > > > > > > > > Item1 Item2
> > > > > > > > > > .. ItemN
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > NumberOfItems =>
> > > > > > > > UnsignedVarInt
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > VarSizeNullableArray =>
> > > > > > > > > NumberOfItemsOrNull
> > > > > > > > > > Item1
> > > > > > > > > > > >> > Item2
> ..
> > ItemN
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > NumberOfItemsOrNull =>
> > > > > > > > VarInt
> > > > > > > > > (-1 means
> > > > > > > > > > null)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > And
> > > > > > similarly for the
> > > > > > > > `String`
> > > > > > > > > and `Bytes`
> > > > > > > > > > types.
> > > > > > > > > > > >> > These
> types
> > > > > > > > > > > >> >
> >>
> > can
> > > > save
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > a
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > considerable amount of
> > > > > > > > space in
> > > > > > > > > this
> > > > > > > > > > proposal
> > > > > > > > > > > >> because
> > > > > > > > > > > >> > they
> can
> > > > > > > > > > > >> >
> >>
> > be
> > > > used
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > both the
> > > > > > number of headers
> > > > > > > > > included in the
> > > > > > > > > > message
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> >
> >>
> > > > lengths of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > header
> > > > > keys
> > > > > > and values. We
> > > > > > > > could
> > > > > > > > > do this
> > > > > > > > > > instead:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Headers =>
> > > > > > > > VarSizeArray[HeaderKey
> > > > > > > > > > HeaderValue]
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > HeaderKey
> > > > > > =>
> > > > > > > VarSizeString
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > HeaderValue =>
> > > > > > > > VarSizeBytes
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Combining
> > > > > > the savings from
> > > > > > > > the
> > > > > > > > > use of
> > > > > > > > > > variable
> > > > > > > > > > > >> length
> > > > > > > > > > > >> >
> fields, the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > benefit
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > of using
> > > > > > the attributes to
> > > > > > > > > represent null
> > > > > > > > > > seems
> > > > > > > > > > > >> pretty
> > > > > > > > > > > >> > small.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > 3.
> > > > > > Whichever way we go
> > > > > > > > (whether
> > > > > > > > > we use the
> > > > > > > > > > > >>
> attributes
> > > > > > > > > > > >> > or
> not), we
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > should at
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > least be
> > > > > > consistent
> > > > > > > between
> > > > > > > > this
> > > > > > > > > KIP and
> > > > > > > > > > KIP-98.
> > > > > > > > > > > It
> > > > > > > > > > > >> > would
> be
> > very
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > strange
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > to have
> > > > > two
> > > > > > ways to
> > > > > > > > represent
> > > > > > > > > null values
> > > > > > > > > > in the
> > > > > > > > > > > >> same
> > > > > > > > > > > >> >
> schema.
> > > > > > > > > > > >> >
> >>
> > > > Either
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > way is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > OK with
> > > > > me.
> > > > > > I think some
> > > > > > > > > message-level
> > > > > > > > > > >
> optimizations
> > > > > > > > > > > >> > are
> > > > > > > > > > > >> >
> >>
> > > > justifiable,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > but
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > savings
> > > > > > here seem
> > > > > > > > minimal (a
> > > > > > > > > few bytes
> > > > > > > > > > per
> > > > > > > > > > > >> >
> message), so
> > > > > > > > > > > >> >
> >>
> > > > maybe it's
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > not
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > worth the
> > > > > > cost of letting
> > > > > > > > the
> > > > > > > > > message
> > > > > > > > > > diverge even
> > > > > > > > > > > >> >
> further from
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > rest of
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > > protocol.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > -Jason
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > On Wed,
> > > > > Feb
> > > > > > 15, 2017 at
> > > > > > > > 8:52 AM,
> > > > > > > > > radai <
> > > > > > > > > > > >> >
> >>
> > > > > > radai.rosenblatt@gmail.com>
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > I've
> > > > > > trimmed the inline
> > > > > > > > > contents as this
> > > > > > > > > > mail is
> > > > > > > > > > > >> >
> getting too
> > > > > > > > > > > >> >
> >>
> > big
> > > > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > apache
> > > > > > mailing list
> > > > > > > > software to
> > > > > > > > > deliver
> > > > > > > > > > :-(
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 1. the
> > > > > > important thing
> > > > > > > for
> > > > > > > > > > interoperability
> is
> > > > > > > > > > > for
> > > > > > > > > > > >> >
> different
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > "interested
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > parties"
> > > > > > (plugins, infra
> > > > > > > > > layers/wrappers,
> > > > > > > > > > > >>
> user-code)
> > > > > > > > > > > >> > to be
> > > > > > > > > > > >> >
> >>
> > > able
> > > > to
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > stick
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > pieces
> > > > > of
> > > > > > metadata onto
> > > > > > > > msgs
> > > > > > > > > without
> > > > > > > > > > getting in
> > > > > > > > > > > >> each
> > > > > > > > > > > >> >
> other's
> > > > > > > > > > > >> >
> >>
> > > way.
> > > > a
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > common
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > key
> > > > > > scheme (Strings, as
> > > > > > > > of the
> > > > > > > > > time of
> > > > > > > > > > this
> > > > > > > > > > > >> >
> writing?) is
> > > all
> > > > > > > > > > > >> >
> >>
> > > thats
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > required
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > for
> > > > > that.
> > > > > > it is assumed
> > > > > > > > that
> > > > > > > > > the other end
> > > > > > > > > > > >> >
> interested
> > in
> > > > any
> > > > > > > > > > > >> >
> >>
> > > such
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > piece of
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > metadata
> > > > > > knows the
> > > > > > > > encoding,
> > > > > > > > > and byte[]
> > > > > > > > > > provides
> > > > > > > > > > > >> for
> > > > > > > > > > > >> > the
> most
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > flexibility.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > i
> > > > > believe
> > > > > > this is the
> > > > > > > same
> > > > > > > > > logic behind
> > > > > > > > > > core
> > > > > > > > > > > kafka
> > > > > > > > > > > >> > being
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > byte[]/byte[]
> > > > > -
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > Strings
> > > > > > are more
> > > > > > > "usable"
> > > > > > > > but
> > > > > > > > > bytes are
> > > > > > > > > > flexible
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > so
> were
> > > > > > > > > > > >> >
> >>
> > > > chosen.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > Also -
> > > > > > core kafka doesnt
> > > > > > > > even
> > > > > > > > > do that
> > > > > > > > > > good of a
> > > > > > > > > > > >> job
> > > > > > > > > > > >> > on
> > > > > > > > > > > >> >
> >>
> > > > usability of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > payload
> > > > > > (example - i
> > > > > > > have
> > > > > > > > to
> > > > > > > > > specify the
> > > > > > > > > > nop
> > > > > > > > > > > >> byte[]
> > > > > > > > > > > >> >
> "decoders"
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > explicitly
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > in
> > > > > conf),
> > > > > > and again
> > > > > > > > sacrificies
> > > > > > > > > usability
> > > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > >> > sake
> of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > performance
> > > > > (no
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > convenient single-record
> > > > > > > > > processing as
> > > > > > > > > > poll is a
> > > > > > > > > > > >> >
> batch, lots
> > of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > obscure
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > little
> > > > > > config details
> > > > > > > > exposing
> > > > > > > > > internals
> > > > > > > > > > of the
> > > > > > > > > > > >> >
> batching
> > > > > > > > > > > >> >
> >>
> > > > mechanism,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > etc)
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > this is
> > > > > > also why i
> > > > > > > really
> > > > > > > > > dislike the
> > > > > > > > > > idea of a
> > > > > > > > > > > >> > "type
> > system"
> > > > > > > > > > > >> >
> >>
> > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > header
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > values,
> > > > > > it further
> > > > > > > > degrades the
> > > > > > > > > > usability, adds
> > > > > > > > > > > >> >
> complexity
> > and
> > > > > > > > > > > >> >
> >>
> > > will
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > eventually get in
> > > > > > > > people's way,
> > > > > > > > > also, it
> > > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > >> > the
> 2nd/3rd
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > home-group
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > serialization mechanism
> > > > > > > > in core
> > > > > > > > > kafka
> > > > > > > > > > (counting
> > > > > > > > > > > 2
> > > > > > > > > > > >> >
> iterations
> > > > > > > > > > > >> >
> >>
> > of
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > "type
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > definition DSL")
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 2. this
> > > > > > is an
> > > > > > > > implementation
> > > > > > > > > detail, and
> > > > > > > > > > not
> > > > > > > > > > > even
> > > > > > > > > > > >> a
> > > > > > > > > > > >> > very
> "user
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > facing" one?
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > to the
> > > > > > best of my
> > > > > > > > understanding
> > > > > > > > > the vote
> > > > > > > > > > process
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > on
> proposed
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > API/behaviour. also -
> > > > > > > > since
> > > > > > > > > we're willing
> > > > > > > > > > to go
> > > > > > > > > > > >> with
> > > > > > > > > > > >> >
> strings
> > > > > > > > > > > >> >
> >>
> > > just
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > serialize
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > a
> > > > > 0-sized
> > > > > > header blob
> > > > > > > and
> > > > > > > > IIUC
> > > > > > > > > you dont
> > > > > > > > > > need any
> > > > > > > > > > > >> >
> optionals
> > > > > > > > > > > >> >
> >>
> > > > anymore.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 3. yes,
> > > > > > we can :-)
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > On Tue,
> > > > > > Feb 14, 2017 at
> > > > > > > > 11:56
> > > > > > > > > PM, Michael
> > > > > > > > > > > Pearce <
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Hi
> > > > > Jay,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 1)
> > > > > > There was some
> > > > > > > > initial
> > > > > > > > > debate on the
> > > > > > > > > > value
> > > > > > > > > > > >> > part,
> as
> > youll
> > > > > > > > > > > >> >
> >>
> > > note
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > String,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > String
> > > > > > headers were
> > > > > > > > > discounted early
> > > > > > > > > > on. The
> > > > > > > > > > > >> >
> reason for
> > > this
> > > > > > > > > > > >> >
> >>
> > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > flexibility
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > and
> > > > > > keeping in line
> > > > > > > > with the
> > > > > > > > > > flexibility of
> > > > > > > > > > > key,
> > > > > > > > > > > >> > value
> of the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > message
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > object
> > > > > > itself. I don’t
> > > > > > > > think
> > > > > > > > > it takes
> > > > > > > > > > away
> > > > > > > > > > > from
> > > > > > > > > > > >> an
> > > > > > > > > > > >> >
> ecosystem
> > > > > > > > > > > >> >
> >>
> > as
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > each
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > plugin
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > will
> > > > > > care for their
> > > > > > > own
> > > > > > > > key,
> > > > > > > > > this way
> > > > > > > > > > ints,
> > > > > > > > > > > >> >
> booleans ,
> > > > exotic
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > custom
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > binary
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > can
> > > > > all
> > > > > > be catered
> > > > > > > for=.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a. If
> > > > > > you really
> > > > > > > wanted
> > > > > > > > to
> > > > > > > > > push for a
> > > > > > > > > > typed
> > > > > > > > > > > >> value
> > > > > > > > > > > >> >
> interface,
> > > > > > > > > > > >> >
> >> I
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > wouldn’t
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > want
> > > > > > just String
> > > > > > > values
> > > > > > > > > supported, but
> > > > > > > > > > the the
> > > > > > > > > > > >> >
> primatives
> > > > > > > > > > > >> >
> >>
> > > plus
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > string and
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > also
> > > > > > still keeping the
> > > > > > > > > ability to have a
> > > > > > > > > > > binary
> > > > > > > > > > > >> > for
> custom
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > binaries that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > some
> > > > > > organisations may
> > > > > > > > have.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > i. I
> > > > > > have written this
> > > > > > > > slight
> > > > > > > > > > alternative
> > > > > > > > > > > here,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > https://cwiki.apache.org/
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > > >> >
> > > > 82+-+Add+Record+Headers+-+
> > > > > Typed
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > ii.
> > > > > > Essentially the
> > > > > > > > value
> > > > > > > > > bytes, has a
> > > > > > > > > > leading
> > > > > > > > > > > >> > byte
> > overhead.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 1.
> > > > > > This tells you
> > > > > > > what
> > > > > > > > type
> > > > > > > > > the value
> > > > > > > > > > is,
> > > > > > > > > > > >> before
> > > > > > > > > > > >> >
> reading
> > > > > > > > > > > >> >
> >>
> > the
> > > > rest
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > of the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > bytes,
> > > > > > allowing
> > > > > > > > > >
> > > > serialisation/deserialization
> > > > > > > > > > > to
> > > > > > > > > > > >> > and
> from the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > primitives,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > string
> > > > > > and byte[].
> > > > > > > This
> > > > > > > > is
> > > > > > > > > akin to some
> > > > > > > > > > other
> > > > > > > > > > > >> >
> messaging
> > > > > > > > > > > >> >
> >>
> > > > systems.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 2) We
> > > > > > are making it
> > > > > > > > optional,
> > > > > > > > > so that
> > > > > > > > > > for
> > > > > > > > > > > those
> > > > > > > > > > > >> > not
> wanting
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > headers have
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 0
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > bytes
> > > > > > overhead (think
> > > > > > > > of it
> > > > > > > > > as a feature
> > > > > > > > > > > flag),
> > > > > > > > > > > >> I
> > > > > > > > > > > >> > don’t
> > > > > > > > > > > >> >
> >>
> > > > think this
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > complex, especially if
> > > > > > > > > comparing to
> > > > > > > > > > changes
> > > > > > > > > > > >> >
> proposed in
> > > > > > > > > > > >> >
> >>
> > > > other kips
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > like
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > kip-98.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a. If
> > > > > > you really
> > > > > > > really
> > > > > > > > don’t
> > > > > > > > > like
> > > > > > > > > > this, we
> > > > > > > > > > > can
> > > > > > > > > > > >> > drop
> it, but
> > > > > > > > > > > >> >
> >>
> > it
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > would
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > mean
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > buying
> > > > > > into 4 bytes
> > > > > > > > extra
> > > > > > > > > overhead for
> > > > > > > > > > users
> > > > > > > > > > > who
> > > > > > > > > > > >> > do
> not want
> > > > > > > > > > > >> >
> >>
> > to
> > > > use
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > headers.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 3) In
> > > > > > the summary yes,
> > > > > > > > it is
> > > > > > > > > at a higher
> > > > > > > > > > > level,
> > > > > > > > > > > >> > but I
> think
> > > > > > > > > > > >> >
> >>
> > > this
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > is well
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > documented in the
> > > > > > > > proposed
> > > > > > > > > changes
> > > > > > > > > > section.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a.
> > > > > > Added getHeaders
> > > > > > > > method to
> > > > > > > > > > >
> Producer/Consumer
> > > > > > > > > > > >> >
> record (that
> > > > > > > > > > > >> >
> >>
> > is
> > > > it)
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > b.
> > > > > > We’ve also detailed
> > > > > > > > the
> > > > > > > > > new Headers
> > > > > > > > > > class
> > > > > > > > > > > >> that
> > > > > > > > > > > >> > this
> method
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > returns
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > encapsulates the
> > > > > > > headers
> > > > > > > > > protocol and
> > > > > > > > > > logic.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Best,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Mike
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > ==Original questions
> > > > > > > > from the
> > > > > > > > > vote
> > > > > > > > > > thread from
> > > > > > > > > > > >> > Jay.==
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Couple
> > > > > > of things I
> > > > > > > > think we
> > > > > > > > > still need
> > > > > > > > > > to work
> > > > > > > > > > > >> out:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 1.
> > > > > I
> > > > > > think we agree
> > > > > > > > about
> > > > > > > > > the key,
> > > > > > > > > > but I
> > > > > > > > > > > >> think
> > > > > > > > > > > >> > we
> haven't
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > talked about
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > the
> > > > > > value yet. I
> > > > > > > > think if
> > > > > > > > > our goal
> > > > > > > > > > is an
> > > > > > > > > > > open
> > > > > > > > > > > >> >
> ecosystem
> > > > > > > > > > > >> >
> >>
> > of
> > > > these
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > header
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > spread across many
> > > > > > > > plugins
> > > > > > > > > from many
> > > > > > > > > > > systems
> > > > > > > > > > > >> we
> > > > > > > > > > > >> > should
> > > > > > > > > > > >> >
> >>
> > > > consider
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > making
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > this
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a
> > > > > > string as well so
> > > > > > > > it can
> > > > > > > > > be
> > > > > > > > > > printed, set
> > > > > > > > > > > >> via
> > > > > > > > > > > >> > a UI,
> set
> > > > > > > > > > > >> >
> >>
> > in
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > config,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > etc.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > Basically
> > > > > > > encouraging
> > > > > > > > > pluggable
> > > > > > > > > > >
> serialization
> > > > > > > > > > > >> >
> formats
> > > > > > > > > > > >> >
> >>
> > > here
> > > > will
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > lead
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > bit
> > > > > > of a tower of
> > > > > > > > babel.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 2.
> > > > > > This proposal
> > > > > > > > still
> > > > > > > > > includes a
> > > > > > > > > > pretty
> > > > > > > > > > > big
> > > > > > > > > > > >> >
> change to
> > our
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > serialization
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > and
> > > > > > protocol
> > > > > > > > definition
> > > > > > > > > layer.
> > > > > > > > > > Essentially
> > > > > > > > > > > >> it is
> > > > > > > > > > > >> >
> >>
> > > > introducing an
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > optional
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > type, where the
> > > > > > > > format is
> > > > > > > > > data
> > > > > > > > > > dependent. I
> > > > > > > > > > > >> > think
> this
> > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > actually a
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > big
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > change though it
> > > > > > > > doesn't
> > > > > > > > > seem like
> > > > > > > > > > it. It
> > > > > > > > > > > >> means
> > > > > > > > > > > >> > you
> can no
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > longer
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > specify
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > this
> > > > > > type with our
> > > > > > > > type
> > > > > > > > > definition
> > > > > > > > > > DSL, and
> > > > > > > > > > > >> >
> likewise it
> > > > > > > > > > > >> >
> >>
> > > > requires
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > custom
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > handling in client
> > > > > > > > libs.
> > > > > > > > > This isn't
> > > > > > > > > > a huge
> > > > > > > > > > > >> >
> thing, since
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Record
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > definition is
> > > > > > > custom
> > > > > > > > > anyway, but I
> > > > > > > > > > think
> > > > > > > > > > > this
> > > > > > > > > > > >> > kind
> of
> > > > > > > > > > > >> >
> >>
> > > > protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > inconsistency is
> > > > > > > very
> > > > > > > > > non-desirable
> > > > > > > > > > and
> > > > > > > > > > > ties
> > > > > > > > > > > >> > you to
> > > > > > > > > > > >> >
> >>
> > > > hand-coding
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > things.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > I
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > think the type
> > > > > > > should
> > > > > > > > > instead by [Key
> > > > > > > > > > > Value]
> > > > > > > > > > > >> in
> > > > > > > > > > > >> > our
> BNF,
> > > > > > > > > > > >> >
> >>
> > > where
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > key and
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > value are both
> > > > > > > short
> > > > > > > > > strings as used
> > > > > > > > > > > >>
> elsewhere.
> > > > > > > > > > > >> > This
> > > > > > > > > > > >> >
> >>
> > > > brings it
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > in line
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > with
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > the
> > > > > > rest of the
> > > > > > > > protocol.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 3.
> > > > > > Could we get
> > > > > > > more
> > > > > > > > > specific about
> > > > > > > > > > the
> > > > > > > > > > > exact
> > > > > > > > > > > >> > Java
> API
> > > > > > > > > > > >> >
> >>
> > > > change to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > ProducerRecord,
> > > > > > > > > ConsumerRecord,
> > > > > > > > > > Record,
> > > > > > > > > > > etc?
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > -Jay
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > The
> > > > > information
> > > > > > contained in
> > > > > > > > this
> > > > > > > > > email is
> > > > > > > > > > strictly
> > > > > > > > > > > >> >
> confidential
> > > and
> > > > > > > > > > > >> >
> >>
> > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the use of the
> > > > > > addressee only,
> > > > > > > > unless
> > > > > > > > > otherwise
> > > > > > > > > > > >>
> indicated.
> > > > > > > > > > > >> > If you
> > > > > > > > > > > >> >
> >>
> > are
> > > > not
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the intended
> > > > > > recipient, please
> > > > > > > > do not
> > > > > > > > > read,
> > > > > > > > > > copy, use
> > > > > > > > > > > or
> > > > > > > > > > > >> >
> disclose to
> > > > > > > > > > > >> >
> >>
> > > > others
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > this message
> > > > > or
> > > > > > any
> > > > > > > attachment.
> > > > > > > > > Please also
> > > > > > > > > > notify the
> > > > > > > > > > > >> >
> sender by
> > > > > > > > > > > >> >
> >>
> > > > replying
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > to this email
> > > > > > or by telephone
> > > > > > > > > (+44(020 7896
> > > > > > > > > > 0011) and
> > > > > > > > > > > >> then
> > > > > > > > > > > >> > delete
> > > > > > > > > > > >> >
> >>
> > the
> > > > email
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > and any copies
> > > > > > of it.
> > > > > > > Opinions,
> > > > > > > > > conclusion
> > > > > > > > > > (etc) that
> > > > > > > > > > > do
> > > > > > > > > > > >> > not
> relate
> > > > > > > > > > > >> >
> >>
> > to
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > official
> > > > > > business of this
> > > > > > > > company
> > > > > > > > > shall be
> > > > > > > > > > understood
> > > > > > > > > > > as
> > > > > > > > > > > >> >
> neither
> > > > > > > > > > > >> >
> >>
> > > > given nor
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > endorsed by
> > > > > it.
> > > > > > IG is a
> > > > > > > trading
> > > > > > > > name
> > > > > > > > > of IG
> > > > > > > > > > Markets
> > > > > > > > > > > >> Limited
> > > > > > > > > > > >> > (a
> company
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > registered in
> > > > > > England and
> > > > > > > Wales,
> > > > > > > > > company number
> > > > > > > > > > > >>
> 04008957)
> > > > > > > > > > > >> > and IG
> > > > > > > > > > > >> >
> >>
> > > Index
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Limited (a
> > > > > > company registered
> > > > > > > in
> > > > > > > > > England and
> > > > > > > > > > Wales,
> > > > > > > > > > > >> >
> company
> > number
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > 01190902).
> > > > > > Registered address
> > > > > > > at
> > > > > > > > > Cannon Bridge
> > > > > > > > > > House,
> > > > > > > > > > > 25
> > > > > > > > > > > >> >
> Dowgate
> > > > > > > > > > > >> >
> >>
> > > Hill,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > London EC4R
> > > > > > 2YA. Both IG
> > > > > > > Markets
> > > > > > > > > Limited
> > > > > > > > > > (register
> > > > > > > > > > > >> number
> > > > > > > > > > > >> >
> 195355)
> > > > > > > > > > > >> >
> >>
> > and
> > > > IG
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Index Limited
> > > > > > (register number
> > > > > > > > > 114059) are
> > > > > > > > > > authorised
> > > > > > > > > > > >> and
> > > > > > > > > > > >> >
> regulated
> > > > > > > > > > > >> >
> >>
> > by
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Financial
> > > > > > Conduct Authority.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > --
> > > > > > > > > > > >> >
> >>
> > > > Nacho - Ignacio
> > > > > > Solis -
> > > > > > > > isolis@igso.net
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > The
> > > information
> > > > contained in
> > > > > > this email is
> > > > > > > > strictly
> > > > > > > > > > confidential and
> > > > > > > > > > > for
> > > > > > > > > > > >> > the
> use of
> > the
> > > > addressee only,
> > > > > > unless
> > > > > > > otherwise
> > > > > > > > > indicated. If
> > > > > > > > > > you are
> > > > > > > > > > > >> not
> > > > > > > > > > > >> > the
> intended
> > > > recipient, please
> > > > > > do not read,
> > > > > > > > copy, use
> > > > > > > > > or
> > > > > > > > > > disclose to
> > > > > > > > > > > >> others
> > > > > > > > > > > >> > this
> message
> > > or
> > > > any
> > > > > attachment.
> > > > > > Please also
> > > > > > > > notify
> > > > > > > > > the sender by
> > > > > > > > > > > >> replying
> > > > > > > > > > > >> > to
> this
> > email
> > > > or by telephone
> > > > > > (+44(020 7896
> > > > > > > > 0011) and
> > > > > > > > > then
> > > > > > > > > > delete the
> > > > > > > > > > > >> email
> > > > > > > > > > > >> > and
> any
> > copies
> > > > of it.
> > > > > Opinions,
> > > > > > conclusion
> > > > > > > > (etc) that
> > > > > > > > > do not
> > > > > > > > > > relate to
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >
> official
> > > > business of this
> > > > > > company shall be
> > > > > > > > understood
> > > > > > > > > as
> > > > > > > > > > neither given
> > > > > > > > > > > >> nor
> > > > > > > > > > > >> >
> endorsed by
> > > it.
> > > > IG is a
> > > > > trading
> > > > > > name of IG
> > > > > > > > Markets
> > > > > > > > > Limited (a
> > > > > > > > > > company
> > > > > > > > > > > >> >
> registered
> > in
> > > > England and
> > > > > > Wales, company
> > > > > > > number
> > > > > > > > > 04008957) and
> > > > > > > > > > IG Index
> > > > > > > > > > > >> >
> Limited (a
> > > > company registered
> > > > > > in England and
> > > > > > > > Wales,
> > > > > > > > > company
> > > > > > > > > > number
> > > > > > > > > > > >> >
> 01190902).
> > > > Registered address
> > > > > > at Cannon Bridge
> > > > > > > > House,
> > > > > > > > > 25
> > > > > > > > > > Dowgate Hill,
> > > > > > > > > > > >> >
> London EC4R
> > > > 2YA. Both IG
> > > > > > Markets Limited
> > > > > > > > (register
> > > > > > > > > number
> > > > > > > > > > 195355) and
> > > > > > > > > > > IG
> > > > > > > > > > > >> > Index
> > Limited
> > > > (register number
> > > > > > 114059) are
> > > > > > > > authorised
> > > > > > > > > and
> > > > > > > > > > regulated by
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >
> Financial
> > > > Conduct Authority.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> The
> > information
> > > > contained in
> > > > > this
> > > > > > email is
> > > > > > > > strictly
> > > > > > > > > confidential
> > > > > > > > > > and for
> > > > > > > > > > > >> the use
> of the
> > > > addressee only,
> > > > > > unless otherwise
> > > > > > > > > indicated. If you
> > > > > > > > > > are
> > > > > > > > > > > not
> > > > > > > > > > > >> the
> intended
> > > > recipient, please
> > > > > do
> > > > > > not read,
> > > > > > > copy,
> > > > > > > > use
> > > > > > > > > or disclose
> > > > > > > > > > to
> > > > > > > > > > > others
> > > > > > > > > > > >> this
> message
> > or
> > > > any attachment.
> > > > > > Please also
> > > > > > > > notify the
> > > > > > > > > sender by
> > > > > > > > > > > replying
> > > > > > > > > > > >> to this
> email
> > or
> > > > by telephone
> > > > > > (+44(020 7896
> > > > > > > 0011)
> > > > > > > > and
> > > > > > > > > then delete
> > > > > > > > > > the
> > > > > > > > > > > email
> > > > > > > > > > > >> and any
> copies
> > > of
> > > > it. Opinions,
> > > > > > conclusion (etc)
> > > > > > > > that
> > > > > > > > > do not
> > > > > > > > > > relate to
> > > > > > > > > > > the
> > > > > > > > > > > >> official
> > > business
> > > > of this
> > > > > company
> > > > > > shall be
> > > > > > > > understood
> > > > > > > > > as neither
> > > > > > > > > > given
> > > > > > > > > > > nor
> > > > > > > > > > > >>
> endorsed by
> > it.
> > > > IG is a trading
> > > > > > name of IG
> > > > > > > Markets
> > > > > > > > > Limited (a
> > > > > > > > > > company
> > > > > > > > > > > >>
> registered in
> > > > England and Wales,
> > > > > > company number
> > > > > > > > > 04008957) and IG
> > > > > > > > > > Index
> > > > > > > > > > > >> Limited
> (a
> > > > company registered in
> > > > > > England and
> > > > > > > > Wales,
> > > > > > > > > company number
> > > > > > > > > > > >>
> 01190902).
> > > > Registered address at
> > > > > > Cannon Bridge
> > > > > > > > House,
> > > > > > > > > 25 Dowgate
> > > > > > > > > > Hill,
> > > > > > > > > > > >> London
> EC4R
> > 2YA.
> > > > Both IG Markets
> > > > > > Limited
> > > > > > > (register
> > > > > > > > > number 195355)
> > > > > > > > > > and IG
> > > > > > > > > > > >> Index
> Limited
> > > > (register number
> > > > > > 114059) are
> > > > > > > > authorised
> > > > > > > > > and
> > > > > > > > > > regulated by
> > > > > > > > > > > the
> > > > > > > > > > > >>
> Financial
> > > Conduct
> > > > Authority.
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > The
> information
> > > > contained in this
> > > > > > email is strictly
> > > > > > > > > confidential and
> > > > > > > > > > for the use of
> the
> > > > addressee only, unless
> > > > > > otherwise
> > > > > > > > indicated.
> > > > > > > > > If you are
> > > > > > > > > > not the intended
> > > > recipient, please do not
> > > > > > read, copy, use
> > > > > > > > or
> > > > > > > > > disclose to
> > > > > > > > > > others this
> message or
> > > any
> > > > attachment.
> > > > > > Please also notify
> > > > > > > > the
> > > > > > > > > sender by
> > > > > > > > > > replying to this
> email
> > or
> > > > by telephone
> > > > > > (+44(020 7896
> > > > > > > 0011)
> > > > > > > > and
> > > > > > > > > then delete
> > > > > > > > > > the email and any
> > copies
> > > > of it. Opinions,
> > > > > > conclusion
> > > > > > > (etc)
> > > > > > > > that
> > > > > > > > > do not
> > > > > > > > > > relate to the
> official
> > > > business of this
> > > > > > company shall be
> > > > > > > > > understood as
> > > > > > > > > > neither given nor
> > > endorsed
> > > > by it. IG is a
> > > > > > trading name of
> > > > > > > > IG
> > > > > > > > > Markets
> > > > > > > > > > Limited (a
> company
> > > > registered in England
> > > > > > and Wales,
> > > > > > > company
> > > > > > > > > number
> > > > > > > > > > 04008957) and IG
> Index
> > > > Limited (a company
> > > > > > registered in
> > > > > > > > England
> > > > > > > > > and Wales,
> > > > > > > > > > company number
> > 01190902).
> > > > Registered
> > > > > > address at Cannon
> > > > > > > > Bridge
> > > > > > > > > House, 25
> > > > > > > > > > Dowgate Hill,
> London
> > EC4R
> > > > 2YA. Both IG
> > > > > > Markets Limited
> > > > > > > > (register
> > > > > > > > > number
> > > > > > > > > > 195355) and IG
> Index
> > > > Limited (register
> > > > > > number 114059) are
> > > > > > > > > authorised and
> > > > > > > > > > regulated by the
> > > Financial
> > > > Conduct
> > > > > > Authority.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information
> contained in
> > > > this email is
> > > > > > strictly
> > > > > > > confidential
> > > > > > > > and
> > > > > > > > > for the use of the
> addressee
> > only,
> > > > unless otherwise
> > > > > > indicated. If
> > > > > > > > you are
> > > > > > > > > not the intended recipient,
> > please
> > > > do not read,
> > > > > > copy, use or
> > > > > > > > disclose to
> > > > > > > > > others this message or any
> > > > attachment. Please also
> > > > > > notify the
> > > > > > > sender
> > > > > > > > by
> > > > > > > > > replying to this email or
> by
> > > > telephone (+44(020
> > > > > 7896
> > > > > > 0011) and then
> > > > > > > > delete
> > > > > > > > > the email and any copies
> of it.
> > > > Opinions,
> > > > > conclusion
> > > > > > (etc) that do
> > > > > > > > not
> > > > > > > > > relate to the official
> business
> > of
> > > > this company
> > > > > > shall be understood
> > > > > > > > as
> > > > > > > > > neither given nor endorsed
> by it.
> > > IG
> > > > is a trading
> > > > > > name of IG
> > > > > > > Markets
> > > > > > > > > Limited (a company
> registered in
> > > > England and Wales,
> > > > > > company number
> > > > > > > > > 04008957) and IG Index
> Limited (a
> > > > company
> > > > > registered
> > > > > > in England and
> > > > > > > > Wales,
> > > > > > > > > company number 01190902).
> > > Registered
> > > > address at
> > > > > > Cannon Bridge
> > > > > > > House,
> > > > > > > > 25
> > > > > > > > > Dowgate Hill, London EC4R
> 2YA.
> > Both
> > > > IG Markets
> > > > > > Limited (register
> > > > > > > > number
> > > > > > > > > 195355) and IG Index
> Limited
> > > > (register number
> > > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated by the Financial
> > Conduct
> > > > Authority.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in
> this email
> > > is
> > > > strictly
> > > > > > confidential and for
> > > > > > > > the use of the addressee only,
> unless
> > > > otherwise
> > > > > indicated.
> > > > > > If you are not
> > > > > > > > the intended recipient, please
> do not
> > > > read, copy, use or
> > > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please
> > > > also notify the
> > > > > > sender by replying
> > > > > > > > to this email or by telephone
> (+44(020
> > > > 7896 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it. Opinions,
> > > conclusion
> > > > (etc) that do
> > > > > > not relate to
> > > > > > > the
> > > > > > > > official business of this
> company shall
> > > be
> > > > understood as
> > > > > > neither given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a trading
> name of
> > > IG
> > > > Markets
> > > > > Limited
> > > > > > (a company
> > > > > > > > registered in England and Wales,
> > company
> > > > number 04008957)
> > > > > > and IG Index
> > > > > > > > Limited (a company registered in
> > England
> > > > and Wales,
> > > > > > company number
> > > > > > > > 01190902). Registered address at
> Cannon
> > > > Bridge House, 25
> > > > > > Dowgate Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> > Limited
> > > > (register number
> > > > > > 195355) and IG
> > > > > > > > Index Limited (register number
> 114059)
> > > are
> > > > authorised and
> > > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > > > confidential and
> > > > > > for the use of the addressee only, unless
> otherwise
> > > > indicated. If you are
> > > > > > not the intended recipient, please do not read,
> copy,
> > use
> > > > or disclose to
> > > > > > others this message or any attachment. Please
> also
> > notify
> > > > the sender by
> > > > > > replying to this email or by telephone (+44(020
> 7896
> > > 0011)
> > > > and then
> > > > > delete
> > > > > > the email and any copies of it. Opinions,
> conclusion
> > > (etc)
> > > > that do not
> > > > > > relate to the official business of this company
> shall
> > be
> > > > understood as
> > > > > > neither given nor endorsed by it. IG is a
> trading name
> > of
> > > > IG Markets
> > > > > > Limited (a company registered in England and
> Wales,
> > > > company number
> > > > > > 04008957) and IG Index Limited (a company
> registered in
> > > > England and
> > > > > Wales,
> > > > > > company number 01190902). Registered address at
> Cannon
> > > > Bridge House, 25
> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> Limited
> > > > (register number
> > > > > > 195355) and IG Index Limited (register number
> 114059)
> > are
> > > > authorised and
> > > > > > regulated by the Financial Conduct Authority.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > The information contained in this email is strictly confidential
> and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If
> you are
> > not
> > > > the intended recipient, please do not read, copy, use or
> disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company
> number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and
> regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Jun's suggestion gives us the best of both worlds. Because the headers are
the last field in the record and you have the total size of the record,
then you can easily get the size of headers in bytes. So they can be copied
without parsing and we can do the lazy initialization without an extra cost.
The message set notion has changed in KIP-98 so that the message set can
have its own fields. See here for more detail:
https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.pau6n6buvoee.
What we're proposing to change here is the schema for the message (aka
record) itself. We'll update that document depending on the outcome of the
discussion here.
-Jason
On Tue, Feb 21, 2017 at 9:10 AM, Michael Pearce <Mi...@ig.com>
wrote:
> This works whilst no other feature needs similar. I prefer to keep it
> contained/encapsulated it then allows the bytes of the headers to also be
> copied transferred as one without any parsing what so ever, but if you
> really feel strongly on the below ill go with it, just want to flag this
> concern.
>
> Btw, the below notation seems to be different to the notation used in
> previous kips like kip-32 which I took the notation from, where the length
> of the following bytes for byte arrays is clearly defined. So im 100% on
> what you propose could you define it in this original styled notation?
>
>
> On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Hey Jun, that's a good thought. Maybe then the full schema could look
> like
> this:
>
> Record =>
> Length => uintVar
> Attributes => int8
> TimestampDelta => intVar
> OffsetDelta => uintVar
> Key => bytes
> Value => bytes
> Headers => [HeaderKey HeaderValue]
> HeaderKey => string
> HeaderValue => bytes
>
> (where the bytes and string type use a variable length size field)
>
> Does that work for you, Michael?
>
> -Jason
>
> On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
>
> > Regarding the overhead of skipping the headers in the message.
> Another way
> > could be to add the headers at the end of the message after the
> value part.
> > This way, there is no need to skip the headers unless you really
> need it.
> > This will allow us to represent the headers explicitly as an array of
> > key/value pairs without the skipping overhead when headers are not
> needed.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <jason@confluent.io
> >
> > wrote:
> >
> > > Michael,
> > >
> > > Here's a link to the branch with the message format changes:
> > > https://github.com/confluentinc/kafka/tree/
> exactly-once-message-format.
> > > The
> > > class to modify to add headers is EosLogRecord (name will
> definitely
> > change
> > > prior to merging in trunk). This branch uses attributes to
> indicate null
> > > key and value. I really don't have a strong preference for or
> against
> > this
> > > as long as we're consistent. I do feel it's kind of a pity to need
> the
> > > signed version of the varint just to be able to indicate null. For
> > headers,
> > > it's an easier call because we don't actually need to represent
> null
> > > headers (using size zero will work fine).
> > >
> > > -Jason
> > >
> > > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > We can make it, implement Iterable<Header>, no concerns on this.
> > > >
> > > > Ive noted this on the KIP.
> > > >
> > > >
> > > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com>
> wrote:
> > > >
> > > > Btw Jason, can you share the branch with the VarInts
> helper/util
> > > class
> > > > that KIP-98 will be introducing and also will have the current
> modified
> > > > (key and value lengths) message, so we can look at, thanks.
> > > >
> > > > On 20/02/2017, 01:11, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > RE “ProducerRecord and ConsumerRecord are immutable
> > currently”. I
> > > > beg to differ on this, if I set a byte[] as my value of my
> record, I
> > can
> > > > later call value() and then alter the bytes in the array. As
> such it is
> > > > very much mutable.
> > > >
> > > > Headers also will anyhow by nature be mutable on
> produce, as
> > you
> > > > add to them.
> > > >
> > > >
> > > > On 20/02/2017, 00:22, "Jason Gustafson" <
> jason@confluent.io>
> > > > wrote:
> > > >
> > > > Re; having a dedicated Headers object. I think it
> makes
> > > sense.
> > > > Maybe it
> > > > could implement Iterable<Header>?
> > > >
> > > > One thing I'm not too sure about is mutability. Both
> the
> > > > ProducerRecord and
> > > > ConsumerRecord are immutable currently. It would be
> nice to
> > > > keep it that
> > > > way. Perhaps Headers could itself be immutable and
> passed
> > > > through the
> > > > constructors?
> > > >
> > > > Just going through how this is, I don’t believe
> this would
> > > > provide much
> > > > > saving. If issue trying to resolve is around
> > object/garbage
> > > > creation of
> > > > > Map.Entry objects.
> > > >
> > > >
> > > > It's definitely better because we only allocate the
> space
> > we
> > > > will use, but
> > > > I assume most of the actual memory cost is in the
> header
> > data
> > > > itself which
> > > > we can't do much about. Maybe we can start with the
> simple
> > > > array
> > > > representation and optimize from there after we find
> it's
> > > > worthwhile to do
> > > > so? If we use the Headers object, we can change this
> later
> > > > without breaking
> > > > the API.
> > > >
> > > > -Jason
> > > >
> > > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> > > > radai.rosenblatt@gmail.com> wrote:
> > > >
> > > > > "(By the way, doesn't it feel a bit odd that we
> seem to
> > be
> > > > designing a
> > > > > feature which is optimized for people not using?)"
> > > > >
> > > > > very. this (i claim :-) ) is the result of very
> intense
> > > > opposition to the
> > > > > usefulness of the feature early on, and not a real
> design
> > > > goal
> > > > >
> > > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > Whilst if having an array of a Header[] would
> mean
> > > getting
> > > > this through,
> > > > > > im happy changing this, as already done.
> > > > > >
> > > > > > Just going through how this is, I don’t believe
> this
> > > > would provide much
> > > > > > saving. If issue trying to resolve is around
> > > > object/garbage creation of
> > > > > > Map.Entry objects.
> > > > > >
> > > > > > All you’re doing here is replacing the equiv of a
> > > > HashMap.Node
> > > > > (Map.Entry)
> > > > > > which simply holds reference to key and value
> objects
> > > with
> > > > a custom
> > > > > variant.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 19/02/2017, 20:22, "Michael Pearce" <
> > > > Michael.Pearce@ig.com> wrote:
> > > > > >
> > > > > > On point 1 & 2 Ive updated KIP to show
> varints (and
> > > > removed the bit
> > > > > > flag). (on the assumption KIP 98 is getting the
> > agreement
> > > > the protocol is
> > > > > > moving from int32 to varInts as standard)
> > > > > >
> > > > > > On point 3 ive updated to use an array of
> Header
> > > > class, instead of a
> > > > > > MultiMap in the Headers class object
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 19/02/2017, 20:06, "Michael Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > On points 1 and 2 I agree.
> > > > > >
> > > > > > This also affects kip-98, I should
> expect this
> > > > resolved before
> > > > > > that vote also passes. If it is accepted there
> (I’m
> > > > assuming this is
> > > > > > getting discussed on that KIP? As you’re driving
> the
> > move
> > > > to VarInts), I
> > > > > am
> > > > > > happy to make this KIP will simply follow suit to
> > > whatever
> > > > is agreed in
> > > > > > KIP-98.
> > > > > >
> > > > > > On 3) Agreed this is a simpler form, as
> long as
> > > no
> > > > one is
> > > > > > expecting hashmap lookup performance ( O(1) ), I
> still
> > > > would prefer an
> > > > > > encapsulated class, so if we find that holding
> it as an
> > > > array in future
> > > > > is
> > > > > > causing some perf issues, the internals are not
> exposed
> > > to
> > > > end users,
> > > > > > allowing the internal structure to move to a map.
> > > > > >
> > > > > > Can we compromise on?
> > > > > >
> > > > > > class ConsumerRecord<K, V> {
> > > > > > K key;
> > > > > > V value;
> > > > > > Headers headers;
> > > > > > }
> > > > > >
> > > > > > class Headers {
> > > > > > Header[] headers;
> > > > > >
> > > > > > add(String key, byte[] value)
> > > > > > Collection<byte[]> get(String key)
> > > > > > }
> > > > > >
> > > > > > class Header {
> > > > > > String key;
> > > > > > byte[] value;
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
> > > > jason@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > headers dont "leak" into
> application
> > code.
> > > > they are useful
> > > > > > to application
> > > > > > > code as well.
> > > > > >
> > > > > >
> > > > > > This is exactly what I have been
> trying to
> > > get
> > > > at. The use
> > > > > > cases documented
> > > > > > here are middleware:
> > > > > > https://cwiki.apache.org/
> > > > confluence/display/KAFKA/A+
> > > > > > Case+for+Kafka+Headers.
> > > > > > If headers are intended for the
> application
> > > > space as well,
> > > > > the
> > > > > > document
> > > > > > should be updated accordingly so
> that it is
> > > an
> > > > explicit
> > > > > design
> > > > > > goal and not
> > > > > > an unfortunate design byproduct.
> There may
> > be
> > > > some
> > > > > > disagreement on whether
> > > > > > it _should_ be a design goal, but it
> may
> > not
> > > > matter much
> > > > > since
> > > > > > the current
> > > > > > interceptor API is probably
> insufficient
> > for
> > > > middleware
> > > > > > applications (I
> > > > > > haven't thought of a way to do this
> that
> > > isn't
> > > > cumbersome).
> > > > > >
> > > > > > That aside, the three unresolved
> points for
> > > me
> > > > are the
> > > > > > following:
> > > > > >
> > > > > > 1. The use of varints. As a one-off
> for use
> > > > only with record
> > > > > > headers, the
> > > > > > case was weak, but if we are using
> them
> > > > throughout the
> > > > > message
> > > > > > format, then
> > > > > > we should do so here as well. The
> > additional
> > > > complexity is
> > > > > > minimal and
> > > > > > early performance testing fully
> justifies
> > > > their use.
> > > > > >
> > > > > > 2. If varints are used, the case for
> using
> > > > attributes to
> > > > > > indicate null
> > > > > > headers also becomes weak. We only
> add one
> > > > additional byte in
> > > > > > each message
> > > > > > if there are no headers. Whatever
> the case,
> > > > let us be
> > > > > > consistent with how
> > > > > > we treat null keys and values.
> > > > > >
> > > > > > 3. We have apparently agreed that the
> > broker
> > > > will validate
> > > > > > headers (please
> > > > > > add this to the KIP). That being the
> case,
> > I
> > > > would prefer to
> > > > > > use Kafka's
> > > > > > conventional format for arrays. The
> > downside
> > > > is that it takes
> > > > > > more work to
> > > > > > skip over the headers on the
> consumer,
> > though
> > > > it's unclear if
> > > > > > the cost
> > > > > > would matter in practice. Some
> concern was
> > > > previously
> > > > > > expressed about the
> > > > > > allocation of maps. An alternative
> would be
> > > to
> > > > use arrays,
> > > > > i.e.
> > > > > >
> > > > > > class ConsumerRecord<K, V> {
> > > > > > K key;
> > > > > > V value;
> > > > > > Header[] headers;
> > > > > > }
> > > > > >
> > > > > > class Header {
> > > > > > String key;
> > > > > > byte[] value;
> > > > > > }
> > > > > >
> > > > > > This would work nicely with the
> > conventional
> > > > array format and
> > > > > > my guess is
> > > > > > it would obviate the need do any lazy
> > > > initialization. If we
> > > > > > use the map as
> > > > > > is currently documented, then it is
> > possible
> > > > with either
> > > > > > representation to
> > > > > > slice the headers and initialize them
> > lazily.
> > > > Either way, it
> > > > > > might be a
> > > > > > good idea to use a separate object to
> > > > represent the headers
> > > > > in
> > > > > > case we need
> > > > > > to modify it in the future in some
> way.
> > > > > >
> > > > > > (By the way, doesn't it feel a bit
> odd that
> > > we
> > > > seem to be
> > > > > > designing a
> > > > > > feature which is optimized for
> people not
> > > > using?)
> > > > > >
> > > > > >
> > > > > > If we can resolve these points, then
> at
> > least
> > > > you will get my
> > > > > > vote.
> > > > > >
> > > > > > Thanks,
> > > > > > Jason
> > > > > >
> > > > > > On Sun, Feb 19, 2017 at 7:30 AM,
> radai <
> > > > > > radai.rosenblatt@gmail.com> wrote:
> > > > > >
> > > > > > > headers dont "leak" into
> application
> > code.
> > > > they are useful
> > > > > > to application
> > > > > > > code as well.
> > > > > > > IIUC samze currently has headers
> "in-V"
> > and
> > > > would just
> > > > > > switch over to kafka
> > > > > > > headers if they exist.
> > > > > > > im sure plenty of other users of
> kafka
> > > would
> > > > have a use for
> > > > > > headers.
> > > > > > > im pretty sure use cases exist
> around
> > > > shuffling data
> > > > > > into/out-of kafka
> > > > > > > (kafka connect or equivalent) where
> > > metadata
> > > > from one end
> > > > > > could copied over
> > > > > > > to the other (S3, for example uses
> http
> > > > headers for
> > > > > > user-accessible
> > > > > > > metadata). it will be kafka client
> code
> > > > getting/setting
> > > > > > those headers. not
> > > > > > > an interceptor.
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 1:41 PM,
> Michael
> > > > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > For APM single event tracing,
> need
> > access
> > > > to the header
> > > > > at
> > > > > > the point of
> > > > > > > > processing on the processing
> thread.
> > > > > > > >
> > > > > > > > As such interceptors will not
> work/be
> > > > suitable for these,
> > > > > > due to the fact
> > > > > > > > they act on the ConsumerRecords
> as a
> > > > batch, before the
> > > > > > handling thread
> > > > > > > can
> > > > > > > > split out and process per
> message which
> > > is
> > > > the point
> > > > > these
> > > > > > tools will
> > > > > > > need
> > > > > > > > to continue to transaction
> tracing.
> > > > > > > >
> > > > > > > > Like wise tools and infra pieces
> will
> > > need
> > > > access to the
> > > > > > message outside
> > > > > > > > the interceptor realm.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 17/02/2017, 21:26, "Jason
> > Gustafson" <
> > > > > > jason@confluent.io> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > That’s exactly what we’re
> doing
> > the
> > > > headers are a
> > > > > > slice of bytes,
> > > > > > > > which
> > > > > > > > > then gets parsed later if
> needed,
> > > or
> > > > can be parsed
> > > > > > right away, the
> > > > > > > > headers
> > > > > > > > > is part of the protocol,
> so can
> > > > still be validated
> > > > > > if wanted.
> > > > > > > > > If you had a header count
> then
> > you
> > > > would have to go
> > > > > > through each
> > > > > > > > header
> > > > > > > > > key and value length value
> to
> > work
> > > > out how much to
> > > > > > skip to get to
> > > > > > > > say the
> > > > > > > > > value or any future
> component in
> > > the
> > > > message after
> > > > > > the headers.
> > > > > > > > Having it
> > > > > > > > > as a byte[] with length
> value
> > makes
> > > > this a lot
> > > > > > easier to skip.
> > > > > > > >
> > > > > > > >
> > > > > > > > So the broker will parse the
> > headers
> > > > and validate
> > > > > > them. Good. The
> > > > > > > only
> > > > > > > > reason remaining that I can
> see to
> > > > leave the headers
> > > > > > as a byte array
> > > > > > > > is to
> > > > > > > > make it easier for the
> client to
> > skip
> > > > past them. Are
> > > > > > we sure this is
> > > > > > > > not
> > > > > > > > premature optimization? Are
> there
> > any
> > > > performance
> > > > > > results which show
> > > > > > > > that
> > > > > > > > this is worthwhile?
> > > > > > > >
> > > > > > > > What’s the issue with
> exposing a
> > > > method getHeaders on
> > > > > > the
> > > > > > > > producer/consumer
> > > > > > > > > record? It doesn’t break
> > anything.
> > > > We don’t need
> > > > > any
> > > > > > special
> > > > > > > version.
> > > > > > > >
> > > > > > > >
> > > > > > > > See my previous explanation.
> What I
> > > am
> > > > trying to
> > > > > > resist is the
> > > > > > > headers
> > > > > > > > becoming a general
> > application-level
> > > > facility. The
> > > > > > primary use case
> > > > > > > as
> > > > > > > > far
> > > > > > > > as I can tell is middleware,
> which
> > is
> > > > the use case
> > > > > > that the
> > > > > > > > interceptors
> > > > > > > > are providing.
> > > > > > > >
> > > > > > > > Current batch consumer model
> and
> > > > consumer
> > > > > interceptors
> > > > > > don’t work
> > > > > > > where
> > > > > > > > > headers need to be acted
> on at
> > per
> > > > message level at
> > > > > > time of
> > > > > > > > processing,
> > > > > > > > > very case is APM (the core
> one),
> > > > where the header
> > > > > > value is used to
> > > > > > > > continue
> > > > > > > > > tracing.
> > > > > > > >
> > > > > > > >
> > > > > > > > I still don't understand the
> point
> > > > about batching.
> > > > > The
> > > > > > consumer
> > > > > > > > records are
> > > > > > > > exposed as a batch in the
> consumer
> > > > interceptor, but
> > > > > > you can still
> > > > > > > > iterate
> > > > > > > > them individually. It is no
> > different
> > > > for the
> > > > > consumer
> > > > > > API itself.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at
> 12:48 PM,
> > > > Michael Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Re:
> > > > > > > > >
> > > > > > > > > “ The point about
> creation of
> > > > maps seems
> > > > > > orthogonal. We can
> > > > > > > still
> > > > > > > > > represent
> > > > > > > > > the headers as a slice
> of
> > bytes
> > > > until the time
> > > > > > it is accessed.”
> > > > > > > > >
> > > > > > > > > That’s exactly what we’re
> doing
> > the
> > > > headers are a
> > > > > > slice of bytes,
> > > > > > > > which
> > > > > > > > > then gets parsed later if
> needed,
> > > or
> > > > can be parsed
> > > > > > right away, the
> > > > > > > > headers
> > > > > > > > > is part of the protocol,
> so can
> > > > still be validated
> > > > > > if wanted.
> > > > > > > > >
> > > > > > > > > If you had a header count
> then
> > you
> > > > would have to go
> > > > > > through each
> > > > > > > > header
> > > > > > > > > key and value length value
> to
> > work
> > > > out how much to
> > > > > > skip to get to
> > > > > > > > say the
> > > > > > > > > value or any future
> component in
> > > the
> > > > message after
> > > > > > the headers.
> > > > > > > > Having it
> > > > > > > > > as a byte[] with length
> value
> > makes
> > > > this a lot
> > > > > > easier to skip.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 17/02/2017, 20:37,
> "Michael
> > > > Pearce" <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > What’s the issue with
> > exposing
> > > a
> > > > method
> > > > > > getHeaders on the
> > > > > > > > > producer/consumer record?
> It
> > > doesn’t
> > > > break
> > > > > anything.
> > > > > > We don’t need
> > > > > > > > any
> > > > > > > > > special version.
> > > > > > > > >
> > > > > > > > > Current batch consumer
> model
> > > and
> > > > consumer
> > > > > > interceptors don’t
> > > > > > > work
> > > > > > > > > where headers need to be
> acted on
> > > at
> > > > per message
> > > > > > level at time of
> > > > > > > > > processing, very case is
> APM (the
> > > > core one), where
> > > > > > the header value
> > > > > > > > is used
> > > > > > > > > to continue tracing.
> JMS/HTTP etc
> > > > all expose these,
> > > > > > without issues.
> > > > > > > > I would
> > > > > > > > > NOT want to lock this down
> to
> > only
> > > > be usable
> > > > > > accessible via
> > > > > > > > interceptors,
> > > > > > > > > as we’d fail on one of the
> main
> > > > goals.
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 17/02/2017, 20:21,
> "Jason
> > > > Gustafson" <
> > > > > > jason@confluent.io>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > The point about
> creation
> > of
> > > > maps seems
> > > > > > orthogonal. We can
> > > > > > > > still
> > > > > > > > > represent
> > > > > > > > > the headers as a
> slice of
> > > > bytes until the
> > > > > > time it is
> > > > > > > > accessed.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Yes exactly we
> have
> > > access
> > > > to the records
> > > > > > thus why the
> > > > > > > > header
> > > > > > > > > should be
> > > > > > > > > > accessible via
> it and
> > not
> > > > hidden for only
> > > > > > interceptors to
> > > > > > > > access.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > As explained
> above, the
> > > > point is to make
> > > > > the
> > > > > > intended usage
> > > > > > > > clear.
> > > > > > > > > Applications should
> > > continue
> > > > to rely on the
> > > > > > key/value
> > > > > > > fields
> > > > > > > > to
> > > > > > > > > serialize
> > > > > > > > > their own headers,
> and it
> > > > would be more
> > > > > > ideal if we can
> > > > > > > avoid
> > > > > > > > > leaking
> > > > > > > > > third-party
> headers into
> > > > applications. This
> > > > > > is difficult to
> > > > > > > > do
> > > > > > > > > with the
> > > > > > > > > current
> interceptors
> > > because
> > > > they share the
> > > > > > record objects
> > > > > > > > with
> > > > > > > > > the common
> > > > > > > > > API. What I had in
> mind
> > is
> > > > something like
> > > > > an
> > > > > > extension of
> > > > > > > the
> > > > > > > > > current
> > > > > > > > > interceptors which
> > exposed
> > > a
> > > > different
> > > > > > object (e.g.
> > > > > > > > > `RecordAndHeaders`).
> > > > > > > > > The challenge is
> for
> > > MM-like
> > > > use cases. Let
> > > > > > me see if I can
> > > > > > > > come
> > > > > > > > > up with a
> > > > > > > > > concrete proposal
> for
> > that
> > > > problem.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 17,
> 2017 at
> > > > 11:55 AM, Michael
> > > > > > Pearce <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I am happy to
> move the
> > > > definition of the
> > > > > > header into the
> > > > > > > > message
> > > > > > > > > body, but
> > > > > > > > > > would cause us
> not to
> > > lazy
> > > > > > initialise/parse the headers,
> > > > > > > as
> > > > > > > > > obviously, we
> > > > > > > > > > would have to
> traverse
> > > > these reading the
> > > > > > message.
> > > > > > > > > >
> > > > > > > > > > This was
> actually one
> > of
> > > > Jay’s requests:
> > > > > > > > > >
> > > > > > > > > > “ 2. I think
> we
> > should
> > > > think about
> > > > > > creating the Map
> > > > > > > > lazily to
> > > > > > > > > avoid
> > > > > > > > > > parsing out
> all the
> > > > headers into
> > > > > > little objects.
> > > > > > > > HashMaps
> > > > > > > > > themselves
> > > > > > > > > > are
> > > > > > > > > > kind of
> expensive
> > and
> > > > the consumer is
> > > > > > very perf
> > > > > > > > sensitive so
> > > > > > > > > and making
> > > > > > > > > > gazillions of
> > > hashmaps
> > > > that may or
> > > > > may
> > > > > > not get used
> > > > > > > is
> > > > > > > > > probably a bad
> > > > > > > > > > idea.”
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 17/02/2017,
> 19:44,
> > > > "Michael Pearce" <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Yes exactly
> we have
> > > > access to the
> > > > > > records thus why
> > > > > > > the
> > > > > > > > > header should
> > > > > > > > > > be accessible
> via it
> > and
> > > > not hidden for
> > > > > > only interceptors
> > > > > > > > to
> > > > > > > > > access.
> > > > > > > > > >
> > > > > > > > > > Sent using
> OWA for
> > > > iPhone
> > > > > > > > > >
> > > > ______________________________
> > > > > > __________
> > > > > > > > > > From: Magnus
> > > Edenhill <
> > > > > > magnus@edenhill.se>
> > > > > > > > > > Sent: Friday,
> > > February
> > > > 17, 2017
> > > > > > 7:34:49 PM
> > > > > > > > > > To:
> > > > dev@kafka.apache.org
> > > > > > > > > > Subject: Re:
> > > [DISCUSS]
> > > > KIP-82 - Add
> > > > > > Record Headers
> > > > > > > > > >
> > > > > > > > > > Big +1 on
> VarInts.
> > > > > > > > > > CPUs are
> fast,
> > memory
> > > > is slow.
> > > > > > > > > >
> > > > > > > > > > I agree with
> Jason
> > > > that we'll want to
> > > > > > continue
> > > > > > > > verifying
> > > > > > > > > messages,
> > > > > > > > > > including
> their
> > > > headers, so while I
> > > > > > appreciate the
> > > > > > > > idea of
> > > > > > > > > the opaque
> > > > > > > > > > header blob
> it
> > won't
> > > > be useful in
> > > > > > practice.
> > > > > > > > > >
> > > > > > > > > > /Magnus
> > > > > > > > > >
> > > > > > > > > > 2017-02-17
> 10:41
> > > > GMT-08:00 Jason
> > > > > > Gustafson <
> > > > > > > > > jason@confluent.io>:
> > > > > > > > > >
> > > > > > > > > > > Sorry, my
> > mistake.
> > > > The consumer
> > > > > > interceptor is per
> > > > > > > > batch,
> > > > > > > > > though I'm
> > > > > > > > > > not
> > > > > > > > > > > sure
> that's an
> > > > actual limitation
> > > > > > since you still
> > > > > > > have
> > > > > > > > > access to the
> > > > > > > > > > > individual
> > records.
> > > > > > > > > > >
> > > > > > > > > > > -Jason
> > > > > > > > > > >
> > > > > > > > > > > On Fri,
> Feb 17,
> > > 2017
> > > > at 10:39 AM,
> > > > > > Jason Gustafson <
> > > > > > > > > >
> jason@confluent.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Re
> headers as
> > > byte
> > > > array and
> > > > > > future use by
> > > > > > > broker.
> > > > > > > > This
> > > > > > > > > doesn't
> > > > > > > > > > take away
> > > > > > > > > > > >> from
> that at
> > > all.
> > > > Nor makes it
> > > > > > difficult at all
> > > > > > > > in my
> > > > > > > > > opinion.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Yeah, I
> didn't
> > > say
> > > > it was
> > > > > > difficult, only
> > > > > > > awkward.
> > > > > > > > You
> > > > > > > > > wouldn't
> > > > > > > > > > write the
> > > > > > > > > > > > schema
> that way
> > > if
> > > > you were
> > > > > > planning to use it on
> > > > > > > > the
> > > > > > > > > brokers from
> > > > > > > > > > the
> > > > > > > > > > > >
> beginning. Note
> > > > also that one of
> > > > > > the benefits of
> > > > > > > > letting
> > > > > > > > > the broker
> > > > > > > > > > > >
> understand
> > > headers
> > > > is that it can
> > > > > > validate that
> > > > > > > > they are
> > > > > > > > > properly
> > > > > > > > > > > >
> formatted. If
> > > cost
> > > > is the only
> > > > > > concern, we should
> > > > > > > > > confirm its
> > > > > > > > > > impact
> > > > > > > > > > > > through
> > > > performance testing.
> > > > > > > > > > > >
> > > > > > > > > > > > One of
> the key
> > > use
> > > > cases requires
> > > > > > access on
> > > > > > > > consume at
> > > > > > > > > per
> > > > > > > > > > event/message
> > > > > > > > > > > >> level
> at the
> > > > point that message
> > > > > > is being
> > > > > > > > processed, as
> > > > > > > > > such the
> > > > > > > > > > batch
> > > > > > > > > > > >>
> interceptors
> > and
> > > > batch consume
> > > > > > api isn't
> > > > > > > > suitable. It
> > > > > > > > > needs to be
> > > > > > > > > > at the
> > > > > > > > > > > >> record
> level.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I'm not
> sure I
> > > > understand the
> > > > > > point about
> > > > > > > batching.
> > > > > > > > > Interceptors
> > > > > > > > > > are
> > > > > > > > > > > > applied
> > > > per-message, right?
> > > > > > > > > > > >
> > > > > > > > > > > > My
> intent on
> > > > interceptors is to
> > > > > > keep the usage of
> > > > > > > > headers
> > > > > > > > > > well-defined so
> > > > > > > > > > > > that
> they don't
> > > > start leaking
> > > > > > unnecessarily into
> > > > > > > > > applications. My
> > > > > > > > > > guess
> > > > > > > > > > > is
> > > > > > > > > > > > that it's
> > > probably
> > > > inevitable,
> > > > > but
> > > > > > isolating it
> > > > > > > in
> > > > > > > > the
> > > > > > > > > > interceptors
> would
> > > > > > > > > > > > at least
> give
> > > > people a second
> > > > > > thought before
> > > > > > > > deciding to
> > > > > > > > > use it.
> > > > > > > > > > The main
> > > > > > > > > > > >
> challenge in my
> > > > mind is figuring
> > > > > > out how an MM
> > > > > > > use
> > > > > > > > case
> > > > > > > > > would
> > > > > > > > > > work. It
> > > > > > > > > > > > would be
> more
> > > > cumbersome to
> > > > > > replicate headers
> > > > > > > > through an
> > > > > > > > > > interceptor,
> > > > > > > > > > > > though
> arguably
> > > MM
> > > > should be
> > > > > > working at a lower
> > > > > > > > level
> > > > > > > > > anyway.
> > > > > > > > > > > >
> > > > > > > > > > > > -Jason
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri,
> Feb 17,
> > > > 2017 at 10:16 AM,
> > > > > > Michael Pearce
> > > > > > > <
> > > > > > > > > >
> Michael.Pearce@ig.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Re
> headers
> > > > available on the
> > > > > > record va
> > > > > > > > interceptors only
> > > > > > > > > > > >>
> > > > > > > > > > > >> One of
> the key
> > > > use cases
> > > > > requires
> > > > > > access on
> > > > > > > > consume at
> > > > > > > > > per
> > > > > > > > > > event/message
> > > > > > > > > > > >> level
> at the
> > > > point that message
> > > > > > is being
> > > > > > > > processed, as
> > > > > > > > > such the
> > > > > > > > > > batch
> > > > > > > > > > > >>
> interceptors
> > and
> > > > batch consume
> > > > > > api isn't
> > > > > > > > suitable. It
> > > > > > > > > needs to be
> > > > > > > > > > at the
> > > > > > > > > > > >> record
> level.
> > > > > > > > > > > >>
> > > > > > > > > > > >> This
> anyhow is
> > > > similar to
> > > > > > jms/http/amqp where
> > > > > > > > headers
> > > > > > > > > are
> > > > > > > > > > available to
> > > > > > > > > > > >>
> consuming
> > > > applications.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Re
> headers as
> > > > byte array and
> > > > > > future use by
> > > > > > > > broker. This
> > > > > > > > > doesn't
> > > > > > > > > > take
> > > > > > > > > > > away
> > > > > > > > > > > >> from
> that at
> > > all.
> > > > Nor makes it
> > > > > > difficult at all
> > > > > > > > in my
> > > > > > > > > opinion.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> Sent
> using OWA
> > > > for iPhone
> > > > > > > > > > > >>
> > > > ______________________________
> > > > > > __________
> > > > > > > > > > > >> From:
> Jason
> > > > Gustafson <
> > > > > > jason@confluent.io>
> > > > > > > > > > > >> Sent:
> Friday,
> > > > February 17, 2017
> > > > > > 5:55:42 PM
> > > > > > > > > > > >> To:
> > > > dev@kafka.apache.org
> > > > > > > > > > > >>
> Subject: Re:
> > > > [DISCUSS] KIP-82 -
> > > > > > Add Record
> > > > > > > Headers
> > > > > > > > > > > >>
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Would
> you be
> > > > proposing in
> > > > > > KIP-98 to convert
> > > > > > > the
> > > > > > > > other
> > > > > > > > > message
> > > > > > > > > > int’s
> > > > > > > > > > > (key
> > > > > > > > > > > >> >
> length,
> > value
> > > > length) also to
> > > > > > varint to keep
> > > > > > > it
> > > > > > > > > uniform.
> > > > > > > > > > > >> > Also I
> > assume
> > > > there will be a
> > > > > > static or helper
> > > > > > > > method
> > > > > > > > > made to
> > > > > > > > > > > write/read
> > > > > > > > > > > >> > these
> in the
> > > > client and
> > > > > server.
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> Yes,
> that is
> > > what
> > > > we are
> > > > > > proposing, so using
> > > > > > > > varints
> > > > > > > > > for headers
> > > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > >>
> consistent
> > with
> > > > the rest of the
> > > > > > message. We have
> > > > > > > > used
> > > > > > > > > static
> > > > > > > > > > helper
> > > > > > > > > > > >> methods
> > > > > > > > > > > >> in our
> > prototype
> > > > implementation.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The
> cost of
> > > > parsing, we want to
> > > > > > parse/interpret
> > > > > > > > the
> > > > > > > > > headers
> > > > > > > > > > lazily (this
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > a key
> point
> > > > brought up earlier
> > > > > > in discussions)
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> I'm a
> bit
> > > > skeptical of this. Has
> > > > > > anyone done the
> > > > > > > > > performance
> > > > > > > > > > testing? I
> > > > > > > > > > > >> can
> > > > > > > > > > > >> probably
> > > > implement it and test
> > > > > it
> > > > > > if no one else
> > > > > > > > has. I
> > > > > > > > > was also
> > > > > > > > > > under
> > > > > > > > > > > the
> > > > > > > > > > > >>
> impression
> > that
> > > > there may be use
> > > > > > cases down the
> > > > > > > > road
> > > > > > > > > where the
> > > > > > > > > > broker
> > > > > > > > > > > >> would
> > > > > > > > > > > >> need to
> > > interpret
> > > > headers. That
> > > > > > wouldn't be off
> > > > > > > > the
> > > > > > > > > table in the
> > > > > > > > > > future
> > > > > > > > > > > if
> > > > > > > > > > > >> it's
> > represented
> > > > as bytes, but
> > > > > it
> > > > > > would be quite
> > > > > > > > a bit
> > > > > > > > > more
> > > > > > > > > > awkward,
> > > > > > > > > > > >> right?
> > > > > > > > > > > >>
> > > > > > > > > > > >> By the
> way,
> > one
> > > > question I have
> > > > > > been wondering
> > > > > > > > about. My
> > > > > > > > > > understanding
> > > > > > > > > > > is
> > > > > > > > > > > >> that
> headers
> > are
> > > > primarily for
> > > > > > use cases where a
> > > > > > > > > third-party
> > > > > > > > > > components
> > > > > > > > > > > >> wants to
> > enrich
> > > > messages without
> > > > > > needing to
> > > > > > > > understand
> > > > > > > > > or modify
> > > > > > > > > > the
> > > > > > > > > > > >> schema
> > > > > > > > > > > >> of the
> message
> > > > key and value.
> > > > > For
> > > > > > the
> > > > > > > applications
> > > > > > > > > which directly
> > > > > > > > > > > produce
> > > > > > > > > > > >> and
> consume
> > the
> > > > messages and
> > > > > > control the
> > > > > > > key/value
> > > > > > > > > schema
> > > > > > > > > > directly, it
> > > > > > > > > > > >> seems
> we would
> > > > rather have them
> > > > > > implement
> > > > > > > headers
> > > > > > > > > directly in
> > > > > > > > > > their own
> > > > > > > > > > > >> schema.
> > > Supposing
> > > > for the sake
> > > > > of
> > > > > > argument that
> > > > > > > > it was
> > > > > > > > > possible,
> > > > > > > > > > my
> > > > > > > > > > > >>
> question is
> > > > whether it be
> > > > > > sufficient to expose
> > > > > > > the
> > > > > > > > > headers in the
> > > > > > > > > > > >>
> interceptor
> > API
> > > > and not in the
> > > > > > common API?
> > > > > > > > > > > >>
> > > > > > > > > > > >> -Jason
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Fri,
> Feb
> > 17,
> > > > 2017 at 3:26 AM,
> > > > > > Michael Pearce
> > > > > > > <
> > > > > > > > > >
> Michael.Pearce@ig.com>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > On
> the point
> > > of
> > > > varInts
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Would
> you be
> > > > proposing in
> > > > > > KIP-98 to convert
> > > > > > > the
> > > > > > > > other
> > > > > > > > > message
> > > > > > > > > > int’s
> > > > > > > > > > > (key
> > > > > > > > > > > >> >
> length,
> > value
> > > > length) also to
> > > > > > varint to keep
> > > > > > > it
> > > > > > > > > uniform.
> > > > > > > > > > > >> > Also I
> > assume
> > > > there will be a
> > > > > > static or helper
> > > > > > > > method
> > > > > > > > > made to
> > > > > > > > > > > write/read
> > > > > > > > > > > >> > these
> in the
> > > > client and
> > > > > server.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Cheers
> > > > > > > > > > > >> > Mike
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On
> > 17/02/2017,
> > > > 11:22, "Michael
> > > > > > Pearce" <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> On the
> > > > point re: headers
> > > > > in
> > > > > > the message
> > > > > > > > protocol
> > > > > > > > > being a
> > > > > > > > > > byte
> > > > > > > > > > > array
> > > > > > > > > > > >> > and
> not a
> > > count
> > > > of elements
> > > > > > followed by the
> > > > > > > > elements.
> > > > > > > > > Again
> > > > > > > > > > this was
> > > > > > > > > > > >> >
> > > > discussed/argued previously.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> It was
> > > > agreed on for a few
> > > > > > reasons some of
> > > > > > > > which
> > > > > > > > > you have
> > > > > > > > > > > obviously
> > > > > > > > > > > >> >
> picked up
> > on:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> Broker
> > is
> > > > able to pass it
> > > > > > through opaquely
> > > > > > > > > > > >> >
> The cost
> > > of
> > > > parsing, we
> > > > > > want to
> > > > > > > > parse/interpret
> > > > > > > > > the headers
> > > > > > > > > > lazily
> > > > > > > > > > > >> > (this
> is a
> > key
> > > > point brought
> > > > > up
> > > > > > earlier in
> > > > > > > > > discussions)
> > > > > > > > > > > >> >
> Headers
> > > can
> > > > be copied from
> > > > > > consumer record
> > > > > > > > to
> > > > > > > > > producer
> > > > > > > > > > record (aka
> > > > > > > > > > > >> > mirror
> > makers
> > > > etc) without
> > > > > > parsing if no
> > > > > > > > changes are
> > > > > > > > > being made
> > > > > > > > > > or
> > > > > > > > > > > being
> > > > > > > > > > > >> >
> looked at.
> > > > > > > > > > > >> >
> Keeps
> > the
> > > > broker agnostic
> > > > > > to the format
> > > > > > > > > > > >> >
> You need
> > > an
> > > > int32 either
> > > > > > for the byte size
> > > > > > > > of the
> > > > > > > > > headers,
> > > > > > > > > > or for
> > > > > > > > > > > >> the
> > > > > > > > > > > >> > count
> of
> > > > elements, so
> > > > > overheads
> > > > > > are the same,
> > > > > > > > but
> > > > > > > > > with going
> > > > > > > > > > with an
> > > > > > > > > > > >> opaque
> > > > > > > > > > > >> > byte
> array
> > has
> > > > the above
> > > > > > advantages.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> Cheers
> > > > > > > > > > > >> >
> Mike
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On
> > > > 17/02/2017, 02:50,
> > > > > > "Jason Gustafson" <
> > > > > > > > > jason@confluent.io
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > Sorry,
> > > > should have
> > > > > > noted that the
> > > > > > > > performance
> > > > > > > > > testing
> > > > > > > > > > was done
> > > > > > > > > > > >> > using
> the
> > > > > > > > > > > >> >
> > > > producer performance
> > > > > > tool shipped with
> > > > > > > > Kafka.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > -Jason
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> On
> > > Thu,
> > > > Feb 16, 2017
> > > > > at
> > > > > > 6:44 PM, Jason
> > > > > > > > > Gustafson <
> > > > > > > > > > > >> >
> > > > jason@confluent.io> wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> >
> > Hey
> > > > Nacho,
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > I've
> > > > compared
> > > > > > performance of our
> > > > > > > > KIP-98
> > > > > > > > > > implementation
> with
> > > > > > > > > > > >> and
> > > > > > > > > > > >> >
> without
> > > > > > > > > > > >> >
> >
> > > > varints. For
> > > > > messages
> > > > > > around 128
> > > > > > > > bytes, we
> > > > > > > > > see an
> > > > > > > > > > increase
> > > > > > > > > > > in
> > > > > > > > > > > >> >
> throughput
> > of
> > > > > > > > > > > >> >
> >
> > > about
> > > > 30% using the
> > > > > > default
> > > > > > > > configuration
> > > > > > > > > settings.
> > > > > > > > > > At 256
> > > > > > > > > > > >> >
> bytes, the
> > > > > > > > > > > >> >
> >
> > > > increase is around
> > > > > > 16%. Obviously
> > > > > > > the
> > > > > > > > > performance
> > > > > > > > > > converge
> > > > > > > > > > > as
> > > > > > > > > > > >> >
> messages get
> > > > > > > > > > > >> >
> >
> > > > larger, but it seems
> > > > > > well worth the
> > > > > > > > cost.
> > > > > > > > > Note that
> > > > > > > > > > we are
> > > > > > > > > > > >> also
> > > > > > > > > > > >> >
> seeing a
> > > > > > > > > > > >> >
> >
> > > > substantial
> > > > > > performance increase
> > > > > > > > against
> > > > > > > > > trunk
> > > > > > > > > > primarily
> > > > > > > > > > > >> because
> > > > > > > > > > > >> > of the
> > > > > > > > > > > >> >
> >
> > much
> > > > more efficient
> > > > > > packing that
> > > > > > > > varints
> > > > > > > > > provide us.
> > > > > > > > > > > Anything
> > > > > > > > > > > >> >
> adding to
> > > > > > > > > > > >> >
> >
> > > > message overhead,
> > > > > > such as record
> > > > > > > > headers,
> > > > > > > > > would only
> > > > > > > > > > > increase
> > > > > > > > > > > >> > the
> relative
> > > > > > > > > > > >> >
> >
> > > > difference. (Of
> > > > > > course take these
> > > > > > > > numbers
> > > > > > > > > with a
> > > > > > > > > > grain of
> > > > > > > > > > > salt
> > > > > > > > > > > >> > since
> I have
> > > > > > > > > > > >> >
> >
> > only
> > > > used the
> > > > > default
> > > > > > settings with
> > > > > > > > both
> > > > > > > > > the producer
> > > > > > > > > > and
> > > > > > > > > > > >> broker
> > > > > > > > > > > >> > on my
> > > > > > > > > > > >> >
> >
> > > local
> > > > machine. We
> > > > > > intend to provide
> > > > > > > > more
> > > > > > > > > extensive
> > > > > > > > > > > performance
> > > > > > > > > > > >> >
> details as
> > > > > > > > > > > >> >
> >
> > part
> > > > of the work for
> > > > > > KIP-98.)
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > The
> > > > implementation
> > > > > we
> > > > > > are using is
> > > > > > > > from
> > > > > > > > > protobuf (
> > > > > > > > > > > >> >
> >
> > > > > > https://developers.google.com/
> > > > > > > > > > >
> > > > protocol-buffers/docs/encoding
> > > > > > > > > > > >> ),
> > > > > > > > > > > >> > which
> is
> > > > > > > > > > > >> >
> >
> > also
> > > > used in HBase.
> > > > > > It is trivial to
> > > > > > > > > implement and as
> > > > > > > > > > far
> > > > > > > > > > > as I
> > > > > > > > > > > >> > know
> doesn't
> > > > > > > > > > > >> >
> >
> > > > suffer from the
> > > > > > aliasing problem you
> > > > > > > > are
> > > > > > > > > describing. I
> > > > > > > > > > > checked
> > > > > > > > > > > >> > with
> Magnus
> > > > > > > > > > > >> >
> >
> > (the
> > > > author of
> > > > > > librdkafka) and he
> > > > > > > > agreed
> > > > > > > > > that the
> > > > > > > > > > savings
> > > > > > > > > > > >> seemed
> > > > > > > > > > > >> > worth
> the
> > > > > > > > > > > >> >
> >
> > cost
> > > > of
> > > > > > implementation.
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > > -Jason
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> > On
> > > > Thu, Feb 16, 2017
> > > > > > at 4:32 PM,
> > > > > > > > Ignacio
> > > > > > > > > Solis <
> > > > > > > > > > > >>
> > isolis@igso.net
> > > >
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >>
> > > > -VarInts
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > I'm
> > > > one of the
> > > > > > people (if not the
> > > > > > > > most)
> > > > > > > > > opposed to
> > > > > > > > > > VarInts.
> > > > > > > > > > > >> >
> VarInts
> > > > > > > > > > > >> >
> >>
> > > have
> > > > a place, but
> > > > > > this is not it.
> > > > > > > > (We
> > > > > > > > > had a large
> > > > > > > > > > > >>
> discussion
> > > > > > > > > > > >> > about
> > > > > > > > > > > >> >
> >>
> > > them
> > > > at the
> > > > > > beginning of KIP-82
> > > > > > > time)
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > If
> > > > anybody has real
> > > > > > life
> > > > > > > performance
> > > > > > > > > numbers of
> > > > > > > > > > VarInts
> > > > > > > > > > > >> >
> improving
> > > > > > > > > > > >> >
> >>
> > > > things or
> > > > > > significantly reducing
> > > > > > > > resources
> > > > > > > > > I would
> > > > > > > > > > like to
> > > > > > > > > > > >> know
> > > > > > > > > > > >> > what
> > > > > > > > > > > >> >
> >>
> > > that
> > > > case may be.
> > > > > > Yes, you can save
> > > > > > > > some
> > > > > > > > > bytes here
> > > > > > > > > > and
> > > > > > > > > > > >> there,
> > > > > > > > > > > >> > but
> > > > > > > > > > > >> >
> >>
> > > this
> > > > is probably
> > > > > > insignificant to
> > > > > > > the
> > > > > > > > > overall system
> > > > > > > > > > > behavior
> > > > > > > > > > > >> > and
> > > > > > > > > > > >> >
> >>
> > > > storage
> > > > > > requirements. -- I say
> > > > > > > this
> > > > > > > > with
> > > > > > > > > respect to
> > > > > > > > > > using
> > > > > > > > > > > >> >
> VarInts in
> > > > > > > > > > > >> >
> >>
> > the
> > > > protocol
> > > > > itself,
> > > > > > not as part of
> > > > > > > > the
> > > > > > > > > data.
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > VarInts require you
> > > > > > to parse the
> > > > > > > Int
> > > > > > > > > before using it
> > > > > > > > > > and
> > > > > > > > > > > >> >
> depending on
> > > > > > > > > > > >> >
> >>
> > the
> > > > encoding they
> > > > > > can suffer from
> > > > > > > > aliasing
> > > > > > > > > (multiple
> > > > > > > > > > > >> >
> > > representations
> > > > > > > > > > > >> >
> >>
> > for
> > > > the same
> > > > > value).
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > Why
> > > > add complexity?
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > Nacho
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > On
> > > > Thu, Feb 16,
> > > > > 2017
> > > > > > at 10:29 AM,
> > > > > > > > Colin
> > > > > > > > > McCabe <
> > > > > > > > > > > >> >
> > > > cmccabe@apache.org>
> > > > > > > > > > > >> >
> >>
> > > > wrote:
> > > > > > > > > > > >> >
> >> >
> > > +1
> > > > for varints
> > > > > > here-- it would
> > > > > > > save
> > > > > > > > > quite a bit of
> > > > > > > > > > space.
> > > > > > > > > > > >> > They
> are
> > > > > > > > > > > >> >
> >> >
> > > > pretty quick to
> > > > > > implement as
> > > > > > > well.
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >>
> > > I
> > > > think it makes
> > > > > > sense for values
> > > > > > > > to be
> > > > > > > > > byte
> > > > > > > > > > arrays.
> > > > > > > > > > > Users
> > > > > > > > > > > >> > might
> want
> > > > > > > > > > > >> >
> >> >
> > > to
> > > > attach
> > > > > > arbitrary payloads;
> > > > > > > they
> > > > > > > > > shouldn't be
> > > > > > > > > > forced to
> > > > > > > > > > > >> >
> serialize
> > > > > > > > > > > >> >
> >> >
> > > > everything to
> > > > > Java
> > > > > > strings.
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >> >
> > > > best,
> > > > > > > > > > > >> >
> >> >
> > > > Colin
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >> >
> > > On
> > > > Thu, Feb 16,
> > > > > > 2017, at 09:52,
> > > > > > > > Jason
> > > > > > > > > Gustafson
> > > > > > > > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > Hey Michael,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > Hmm, I guess the
> > > > > > point of
> > > > > > > > representing
> > > > > > > > > it as
> > > > > > > > > > bytes is to
> > > > > > > > > > > >> > allow
> the
> > > > > > > > > > > >> >
> >>
> > > > broker
> > > > > > > > > > > >> >
> >>
> > >>
> > > > to pass it
> > > > > > through opaquely? Is
> > > > > > > > the
> > > > > > > > > cost of
> > > > > > > > > > parsing
> > > > > > > > > > > them a
> > > > > > > > > > > >> >
> concern, or
> > > > > > > > > > > >> >
> >>
> > >>
> > > > are
> > > > > > > > > > > >> >
> >>
> > >>
> > > > we simply trying
> > > > > > to ensure that
> > > > > > > > the
> > > > > > > > > broker stays
> > > > > > > > > > > agnostic
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> >
> >>
> > > > format?
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > On varints, I
> > > > > > think adding
> > > > > > > > support for
> > > > > > > > > them makes
> > > > > > > > > > less
> > > > > > > > > > > >> sense
> > > > > > > > > > > >> > for an
> > > > > > > > > > > >> >
> >>
> > >>
> > > > isolated use
> > > > > > case, but as part
> > > > > > > of
> > > > > > > > a
> > > > > > > > > more holistic
> > > > > > > > > > change
> > > > > > > > > > > >> > (such
> as
> > what
> > > > > > > > > > > >> >
> >>
> > we
> > > > > > > > > > > >> >
> >>
> > >>
> > > > have proposed in
> > > > > > KIP-98), I
> > > > > > > think
> > > > > > > > they
> > > > > > > > > are
> > > > > > > > > > justifiable.
> > > > > > > > > > > If
> > > > > > > > > > > >> > we
> add them,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > then the need to
> > > > > > use attributes
> > > > > > > > becomes
> > > > > > > > > quite a
> > > > > > > > > > bit
> > > > > > > > > > > >> weaker,
> > > > > > > > > > > >> >
> right? The
> > > > > > > > > > > >> >
> >>
> > >>
> > > > other thing I
> > > > > > find slightly odd
> > > > > > > > is the
> > > > > > > > > fact that
> > > > > > > > > > null
> > > > > > > > > > > >> >
> headers has
> > no
> > > > > > > > > > > >> >
> >>
> > >>
> > > > actual
> > > > > > > > > > > >> >
> >>
> > >>
> > > > semantic meaning
> > > > > > for the message
> > > > > > > > > (unlike null
> > > > > > > > > > keys and
> > > > > > > > > > > >> >
> values). It
> > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > just
> > > > > > > > > > > >> >
> >>
> > >>
> > > a
> > > > space
> > > > > > optimization. It seems a
> > > > > > > > bit
> > > > > > > > > better to
> > > > > > > > > > always
> > > > > > > > > > > use
> > > > > > > > > > > >> > size
> 0 to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > indicate having
> > > > > > no headers.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > Overall, the
> > > > > main
> > > > > > point is
> > > > > > > > ensuring
> > > > > > > > > that the
> > > > > > > > > > message
> > > > > > > > > > > >> schema
> > > > > > > > > > > >> >
> remains
> > > > > > > > > > > >> >
> >>
> > >>
> > > > consistent,
> > > > > > either within the
> > > > > > > > larger
> > > > > > > > > protocol, or
> > > > > > > > > > at a
> > > > > > > > > > > >> >
> minimum
> > within
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > message itself.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > -Jason
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > On Thu, Feb 16,
> > > > > > 2017 at 6:39 AM,
> > > > > > > > > Michael Pearce <
> > > > > > > > > > > >> >
> > > > Michael.Pearce@ig.com
> > > > > > > > > > > >> >
> >> >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Hi Jason,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On point 1) in
> > > > > > the message
> > > > > > > > protocol
> > > > > > > > > the headers
> > > > > > > > > > are
> > > > > > > > > > > >> simply
> > > > > > > > > > > >> > a byte
> > > > > > > > > > > >> >
> >>
> > > > array,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > as like the
> > > > > key
> > > > > > or value, this
> > > > > > > > is to
> > > > > > > > > clearly
> > > > > > > > > > demarcate
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >
> header in
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > core message.
> > > > > > Then the header
> > > > > > > > byte
> > > > > > > > > array in the
> > > > > > > > > > core
> > > > > > > > > > > >> >
> message is
> > an
> > > > > > > > > > > >> >
> >>
> > > > array of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > key, value
> > > > > > pairs. This is what
> > > > > > > > it is
> > > > > > > > > denoting.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Then this
> > > > > would
> > > > > > be I guess in
> > > > > > > > the
> > > > > > > > > given
> > > > > > > > > > notation:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Headers =>
> > > > > > [KeyLength, Key,
> > > > > > > > > ValueLength, Value]
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > KeyLength
> > > > > > => int32
> > > > > > > > > <-----------------NEW
> > > > > > > > > > size of
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> byte[] of
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > serialised key
> > > > > > value
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Key =>
> > > > > bytes
> > > > > > > > > <---------------------- NEW
> > > > > > > > > > > serialised
> > > > > > > > > > > >> > string
> > (UTF8)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > bytes of the
> > > > > > header key
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > ValueLength
> > > > > > => int32
> > > > > > > > > <-------------- NEW
> > > > > > > > > > size of
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> byte[] of
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > serialised
> > > > > > header value
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Value =>
> > > > > > bytes
> > > > > > > > > <-------------------- NEW
> > > > > > > > > > > serialised
> > > > > > > > > > > >> > form
> of the
> > > > > > > > > > > >> >
> >>
> > > > header
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > value
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > The key length
> > > > > > and value
> > > > > > > length
> > > > > > > > is
> > > > > > > > > matching the
> > > > > > > > > > way
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> protocol is
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > defined in the
> > > > > > core message
> > > > > > > > currently.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On point 2)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Var sized
> > > > > ints,
> > > > > > this was
> > > > > > > > discussed
> > > > > > > > > much earlier
> > > > > > > > > > on, in
> > > > > > > > > > > >> > fact
> I had
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > suggested it
> > > > > > myself (with
> > > > > > > Hadoop
> > > > > > > > > references),
> > > > > > > > > > the
> > > > > > > > > > > >> >
> complexity
> > of
> > > > this
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > compared to
> > > > > > having a simpler
> > > > > > > > protocol
> > > > > > > > > was
> > > > > > > > > > argued and
> > > > > > > > > > > >> >
> agreed it
> > > > > > > > > > > >> >
> >>
> > > > wasn’t worth
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the complexity
> > > > > > as all other
> > > > > > > > clients
> > > > > > > > > in other
> > > > > > > > > > languages
> > > > > > > > > > > >> > would
> need
> > to
> > > > > > > > > > > >> >
> >>
> > > > ensure
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > theyre using
> > > > > > the right var
> > > > > > > size
> > > > > > > > > algorithm, as
> > > > > > > > > > there
> > > > > > > > > > > is a
> > > > > > > > > > > >> > few.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On point 3)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > We did the
> > > > > > attributes,
> > > > > > > optional
> > > > > > > > > approach as
> > > > > > > > > > originally
> > > > > > > > > > > >> > there
> was
> > > > > > > > > > > >> >
> >>
> > > > marked
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > concern that
> > > > > > headers would
> > > > > > > > cause a
> > > > > > > > > message size
> > > > > > > > > > > overhead
> > > > > > > > > > > >> > for
> others,
> > > > > > > > > > > >> >
> >>
> > who
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > don’t want
> > > > > > them. As such this
> > > > > > > > is the
> > > > > > > > > clean
> > > > > > > > > > solution to
> > > > > > > > > > > >> >
> achieve
> > that.
> > > > > > > > > > > >> >
> >>
> > If
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > that no longer
> > > > > > holds, and we
> > > > > > > > don’t
> > > > > > > > > care that we
> > > > > > > > > > add
> > > > > > > > > > > >> 4bytes
> > > > > > > > > > > >> >
> overhead,
> > > > > > > > > > > >> >
> >>
> > > then
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > im happy to
> > > > > > remove.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > I’m personally
> > > > > > in favour of
> > > > > > > > keeping
> > > > > > > > > the message
> > > > > > > > > > as
> > > > > > > > > > > small
> > > > > > > > > > > >> > as
> possible
> > > > > > > > > > > >> >
> >>
> > so
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > people don’t
> > > > > > get shocks in
> > > > > > > perf
> > > > > > > > and
> > > > > > > > > throughputs
> > > > > > > > > > dues
> > > > > > > > > > > to
> > > > > > > > > > > >> >
> message
> > size,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > unless they
> > > > > > actively use the
> > > > > > > > feature,
> > > > > > > > > as such I
> > > > > > > > > > do
> > > > > > > > > > > >> prefer
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> >
> >>
> > > > attribute bit
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > wise feature
> > > > > > flag approach
> > > > > > > > myself.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > On 16/02/2017,
> > > > > > 05:40, "Jason
> > > > > > > > > Gustafson" <
> > > > > > > > > > > >> >
> > > > jason@confluent.io> wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > We have
> > > > > > proposed a few
> > > > > > > > > significant changes
> > > > > > > > > > to the
> > > > > > > > > > > >> >
> message
> > format
> > > > > > > > > > > >> >
> >>
> > in
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > KIP-98
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > which now
> > > > > > seems likely to
> > > > > > > > pass
> > > > > > > > > (perhaps
> > > > > > > > > > with some
> > > > > > > > > > > >> >
> iterations
> > on
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > implementation details).
> > > > > > > It
> > > > > > > > would
> > > > > > > > > be good
> > > > > > > > > > to try
> > > > > > > > > > > and
> > > > > > > > > > > >> >
> coordinate
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > changes
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > in both of
> > > > > > the proposals
> > > > > > > to
> > > > > > > > make
> > > > > > > > > sure they
> > > > > > > > > > are
> > > > > > > > > > > >> >
> consistent
> > and
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > compatible.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > I think
> > > > > > using the
> > > > > > > > attributes to
> > > > > > > > > indicate
> > > > > > > > > > null
> > > > > > > > > > > >> headers
> > > > > > > > > > > >> > is a
> > > > > > > > > > > >> >
> >>
> > > > reasonable
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > approach.
> > > > > > We have proposed
> > > > > > > > to do
> > > > > > > > > the same
> > > > > > > > > > thing
> > > > > > > > > > > for
> > > > > > > > > > > >> > the
> message
> > > > > > > > > > > >> >
> >>
> > key
> > > > and
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > value.
> > > > > That
> > > > > > said, I
> > > > > > > > sympathize
> > > > > > > > > with Jay's
> > > > > > > > > > > argument.
> > > > > > > > > > > >> > Having
> > > > > > > > > > > >> >
> >>
> > > > multiple
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > ways to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > specify a
> > > > > > null value
> > > > > > > > increases
> > > > > > > > > the overall
> > > > > > > > > > > >>
> complexity
> > > > > > > > > > > >> > of the
> > > > > > > > > > > >> >
> >>
> > > > protocol.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > You
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > can see
> > > > > > this just from the
> > > > > > > > fact
> > > > > > > > > that you
> > > > > > > > > > need the
> > > > > > > > > > > >> > extra
> > verbiage
> > > > > > > > > > > >> >
> >>
> > in
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > protocol
> > > > > > specification in
> > > > > > > > this
> > > > > > > > > KIP and in
> > > > > > > > > > KIP-98
> > > > > > > > > > > to
> > > > > > > > > > > >> >
> describe the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > dependence
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > between
> > > > > the
> > > > > > fields and the
> > > > > > > > > attributes. It
> > > > > > > > > > seems
> > > > > > > > > > > >> like a
> > > > > > > > > > > >> >
> slippery
> > > > > > > > > > > >> >
> >>
> > > slope
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > if
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > you start
> > > > > > allowing
> > > > > > > different
> > > > > > > > > request types
> > > > > > > > > > to
> > > > > > > > > > > >> >
> implement
> > the
> > > > > > > > > > > >> >
> >>
> > > > protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > specification differently.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > You can
> > > > > > also argue that
> > > > > > > the
> > > > > > > > > messages
> > > > > > > > > > already are
> > > > > > > > > > > and
> > > > > > > > > > > >> > are
> likely
> > > > > > > > > > > >> >
> >>
> > to
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > remain a
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > special
> > > > > > case. For example,
> > > > > > > > there
> > > > > > > > > is
> > > > > > > > > > currently no
> > > > > > > > > > > >> >
> generality
> > in
> > > > > > > > > > > >> >
> >>
> > how
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > compressed
> > > > > > message sets
> > > > > > > are
> > > > > > > > > represented
> > > > > > > > > > that would
> > > > > > > > > > > >> be
> > > > > > > > > > > >> >
> applicable
> > > > > > > > > > > >> >
> >>
> > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > other
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > request
> > > > > > types. Some might
> > > > > > > > see this
> > > > > > > > > > divergence as
> > > > > > > > > > > an
> > > > > > > > > > > >> >
> unfortunate
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > deficiency
> > > > > > which should be
> > > > > > > > fixed;
> > > > > > > > > others
> > > > > > > > > > might see
> > > > > > > > > > > >> it
> > > > > > > > > > > >> > as
> sort of
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > inevitability of needing
> > > > > > > to
> > > > > > > > > optimize where
> > > > > > > > > > it
> > > > > > > > > > > counts
> > > > > > > > > > > >> > most.
> I'm
> > > > > > > > > > > >> >
> >>
> > > > probably
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > somewhere
> > > > > > in between, but
> > > > > > > I
> > > > > > > > think
> > > > > > > > > we
> > > > > > > > > > probably all
> > > > > > > > > > > >> > share
> the
> > > > > > > > > > > >> >
> >>
> > > > intuition
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > > protocol should be
> > > > > > > kept
> > > > > > > > as
> > > > > > > > > consistent as
> > > > > > > > > > > >>
> possible.
> > > > > > > > > > > >> > With
> that
> > > > > > > > > > > >> >
> >>
> > in
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > mind,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > here are a
> > > > > > few comments:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > 1. One
> > > > > > thing I found a
> > > > > > > > little odd
> > > > > > > > > when
> > > > > > > > > > reading the
> > > > > > > > > > > >> >
> current
> > > > > > > > > > > >> >
> >>
> > > > proposal is
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > headers
> > > > > > are both
> > > > > > > > represented
> > > > > > > > > as an
> > > > > > > > > > array of
> > > > > > > > > > > >> bytes
> > > > > > > > > > > >> > and
> as an
> > > > > > > > > > > >> >
> >>
> > > array
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > of
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > key/value
> > > > > > pairs. I'd
> > > > > > > > probably
> > > > > > > > > suggest
> > > > > > > > > > something
> > > > > > > > > > > like
> > > > > > > > > > > >> > this:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Headers =>
> > > > > > [HeaderKey
> > > > > > > > HeaderValue]
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > HeaderKey
> > > > > > => String
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > HeaderValue => Bytes
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > An array
> > > > > in
> > > > > > the Kafka
> > > > > > > > protocol is
> > > > > > > > > > represented as a
> > > > > > > > > > > >> > 4-byte
> > integer
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > indicating
> > > > > > the number of
> > > > > > > > elements
> > > > > > > > > in the
> > > > > > > > > > array
> > > > > > > > > > > >> >
> followed by
> > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > serialization of the
> > > > > > > > elements.
> > > > > > > > > Unless I'm
> > > > > > > > > > > >> >
> > > > misunderstanding, what
> > > > > > > > > > > >> >
> >>
> > you
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > have
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > instead is
> > > > > > the total size
> > > > > > > > of the
> > > > > > > > > headers in
> > > > > > > > > > bytes
> > > > > > > > > > > >> >
> followed by
> > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > elements.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > I'm not
> > > > > > sure I see any
> > > > > > > > reason for
> > > > > > > > > this
> > > > > > > > > > > >>
> inconsistency.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > 2. In
> > > > > > KIP-98, we've
> > > > > > > > introduced
> > > > > > > > > > variable-length
> > > > > > > > > > > >> integer
> > > > > > > > > > > >> >
> fields.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Effectively,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > we've
> > > > > > enriched (or
> > > > > > > > "complicated"
> > > > > > > > > as Jay
> > > > > > > > > > might say
> > > > > > > > > > > ;)
> > > > > > > > > > > >> > the
> protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > specification to include
> > > > > > > the
> > > > > > > > > following
> > > > > > > > > > types:
> > > > > > > > > > > >> VarInt,
> > > > > > > > > > > >> >
> VarLong,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > UnsignedVarInt and
> > > > > > > > > UnsignedVarLong.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Along with
> > > > > > these
> > > > > > > > primitives, we
> > > > > > > > > could
> > > > > > > > > > introduce
> > > > > > > > > > > the
> > > > > > > > > > > >> >
> following
> > > > > > > > > > > >> >
> >>
> > > > types:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > VarSizeArray =>
> > > > > > > > NumberOfItems
> > > > > > > > > Item1 Item2
> > > > > > > > > > .. ItemN
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > NumberOfItems =>
> > > > > > > > UnsignedVarInt
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > VarSizeNullableArray =>
> > > > > > > > > NumberOfItemsOrNull
> > > > > > > > > > Item1
> > > > > > > > > > > >> > Item2
> ..
> > ItemN
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > NumberOfItemsOrNull =>
> > > > > > > > VarInt
> > > > > > > > > (-1 means
> > > > > > > > > > null)
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > And
> > > > > > similarly for the
> > > > > > > > `String`
> > > > > > > > > and `Bytes`
> > > > > > > > > > types.
> > > > > > > > > > > >> > These
> types
> > > > > > > > > > > >> >
> >>
> > can
> > > > save
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > a
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > considerable amount of
> > > > > > > > space in
> > > > > > > > > this
> > > > > > > > > > proposal
> > > > > > > > > > > >> because
> > > > > > > > > > > >> > they
> can
> > > > > > > > > > > >> >
> >>
> > be
> > > > used
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > both the
> > > > > > number of headers
> > > > > > > > > included in the
> > > > > > > > > > message
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > the
> > > > > > > > > > > >> >
> >>
> > > > lengths of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > header
> > > > > keys
> > > > > > and values. We
> > > > > > > > could
> > > > > > > > > do this
> > > > > > > > > > instead:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Headers =>
> > > > > > > > VarSizeArray[HeaderKey
> > > > > > > > > > HeaderValue]
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > HeaderKey
> > > > > > =>
> > > > > > > VarSizeString
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > HeaderValue =>
> > > > > > > > VarSizeBytes
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > Combining
> > > > > > the savings from
> > > > > > > > the
> > > > > > > > > use of
> > > > > > > > > > variable
> > > > > > > > > > > >> length
> > > > > > > > > > > >> >
> fields, the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > benefit
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > of using
> > > > > > the attributes to
> > > > > > > > > represent null
> > > > > > > > > > seems
> > > > > > > > > > > >> pretty
> > > > > > > > > > > >> > small.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > 3.
> > > > > > Whichever way we go
> > > > > > > > (whether
> > > > > > > > > we use the
> > > > > > > > > > > >>
> attributes
> > > > > > > > > > > >> > or
> not), we
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > should at
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > least be
> > > > > > consistent
> > > > > > > between
> > > > > > > > this
> > > > > > > > > KIP and
> > > > > > > > > > KIP-98.
> > > > > > > > > > > It
> > > > > > > > > > > >> > would
> be
> > very
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > strange
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > to have
> > > > > two
> > > > > > ways to
> > > > > > > > represent
> > > > > > > > > null values
> > > > > > > > > > in the
> > > > > > > > > > > >> same
> > > > > > > > > > > >> >
> schema.
> > > > > > > > > > > >> >
> >>
> > > > Either
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > way is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > OK with
> > > > > me.
> > > > > > I think some
> > > > > > > > > message-level
> > > > > > > > > > >
> optimizations
> > > > > > > > > > > >> > are
> > > > > > > > > > > >> >
> >>
> > > > justifiable,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > but
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > savings
> > > > > > here seem
> > > > > > > > minimal (a
> > > > > > > > > few bytes
> > > > > > > > > > per
> > > > > > > > > > > >> >
> message), so
> > > > > > > > > > > >> >
> >>
> > > > maybe it's
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > not
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > worth the
> > > > > > cost of letting
> > > > > > > > the
> > > > > > > > > message
> > > > > > > > > > diverge even
> > > > > > > > > > > >> >
> further from
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > rest of
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > the
> > > > > > protocol.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > -Jason
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > On Wed,
> > > > > Feb
> > > > > > 15, 2017 at
> > > > > > > > 8:52 AM,
> > > > > > > > > radai <
> > > > > > > > > > > >> >
> >>
> > > > > > radai.rosenblatt@gmail.com>
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > I've
> > > > > > trimmed the inline
> > > > > > > > > contents as this
> > > > > > > > > > mail is
> > > > > > > > > > > >> >
> getting too
> > > > > > > > > > > >> >
> >>
> > big
> > > > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > apache
> > > > > > mailing list
> > > > > > > > software to
> > > > > > > > > deliver
> > > > > > > > > > :-(
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 1. the
> > > > > > important thing
> > > > > > > for
> > > > > > > > > > interoperability
> is
> > > > > > > > > > > for
> > > > > > > > > > > >> >
> different
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > "interested
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > parties"
> > > > > > (plugins, infra
> > > > > > > > > layers/wrappers,
> > > > > > > > > > > >>
> user-code)
> > > > > > > > > > > >> > to be
> > > > > > > > > > > >> >
> >>
> > > able
> > > > to
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > stick
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > pieces
> > > > > of
> > > > > > metadata onto
> > > > > > > > msgs
> > > > > > > > > without
> > > > > > > > > > getting in
> > > > > > > > > > > >> each
> > > > > > > > > > > >> >
> other's
> > > > > > > > > > > >> >
> >>
> > > way.
> > > > a
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > common
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > key
> > > > > > scheme (Strings, as
> > > > > > > > of the
> > > > > > > > > time of
> > > > > > > > > > this
> > > > > > > > > > > >> >
> writing?) is
> > > all
> > > > > > > > > > > >> >
> >>
> > > thats
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > required
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > for
> > > > > that.
> > > > > > it is assumed
> > > > > > > > that
> > > > > > > > > the other end
> > > > > > > > > > > >> >
> interested
> > in
> > > > any
> > > > > > > > > > > >> >
> >>
> > > such
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > piece of
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > metadata
> > > > > > knows the
> > > > > > > > encoding,
> > > > > > > > > and byte[]
> > > > > > > > > > provides
> > > > > > > > > > > >> for
> > > > > > > > > > > >> > the
> most
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > flexibility.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > i
> > > > > believe
> > > > > > this is the
> > > > > > > same
> > > > > > > > > logic behind
> > > > > > > > > > core
> > > > > > > > > > > kafka
> > > > > > > > > > > >> > being
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > byte[]/byte[]
> > > > > -
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > Strings
> > > > > > are more
> > > > > > > "usable"
> > > > > > > > but
> > > > > > > > > bytes are
> > > > > > > > > > flexible
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > so
> were
> > > > > > > > > > > >> >
> >>
> > > > chosen.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > Also -
> > > > > > core kafka doesnt
> > > > > > > > even
> > > > > > > > > do that
> > > > > > > > > > good of a
> > > > > > > > > > > >> job
> > > > > > > > > > > >> > on
> > > > > > > > > > > >> >
> >>
> > > > usability of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > payload
> > > > > > (example - i
> > > > > > > have
> > > > > > > > to
> > > > > > > > > specify the
> > > > > > > > > > nop
> > > > > > > > > > > >> byte[]
> > > > > > > > > > > >> >
> "decoders"
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > explicitly
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > in
> > > > > conf),
> > > > > > and again
> > > > > > > > sacrificies
> > > > > > > > > usability
> > > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > >> > sake
> of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > performance
> > > > > (no
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > convenient single-record
> > > > > > > > > processing as
> > > > > > > > > > poll is a
> > > > > > > > > > > >> >
> batch, lots
> > of
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > obscure
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > little
> > > > > > config details
> > > > > > > > exposing
> > > > > > > > > internals
> > > > > > > > > > of the
> > > > > > > > > > > >> >
> batching
> > > > > > > > > > > >> >
> >>
> > > > mechanism,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > etc)
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > this is
> > > > > > also why i
> > > > > > > really
> > > > > > > > > dislike the
> > > > > > > > > > idea of a
> > > > > > > > > > > >> > "type
> > system"
> > > > > > > > > > > >> >
> >>
> > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > header
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > values,
> > > > > > it further
> > > > > > > > degrades the
> > > > > > > > > > usability, adds
> > > > > > > > > > > >> >
> complexity
> > and
> > > > > > > > > > > >> >
> >>
> > > will
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > eventually get in
> > > > > > > > people's way,
> > > > > > > > > also, it
> > > > > > > > > > would
> > > > > > > > > > > be
> > > > > > > > > > > >> > the
> 2nd/3rd
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > home-group
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > serialization mechanism
> > > > > > > > in core
> > > > > > > > > kafka
> > > > > > > > > > (counting
> > > > > > > > > > > 2
> > > > > > > > > > > >> >
> iterations
> > > > > > > > > > > >> >
> >>
> > of
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > "type
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > definition DSL")
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 2. this
> > > > > > is an
> > > > > > > > implementation
> > > > > > > > > detail, and
> > > > > > > > > > not
> > > > > > > > > > > even
> > > > > > > > > > > >> a
> > > > > > > > > > > >> > very
> "user
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > facing" one?
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > to the
> > > > > > best of my
> > > > > > > > understanding
> > > > > > > > > the vote
> > > > > > > > > > process
> > > > > > > > > > > >> is
> > > > > > > > > > > >> > on
> proposed
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > API/behaviour. also -
> > > > > > > > since
> > > > > > > > > we're willing
> > > > > > > > > > to go
> > > > > > > > > > > >> with
> > > > > > > > > > > >> >
> strings
> > > > > > > > > > > >> >
> >>
> > > just
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > serialize
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > a
> > > > > 0-sized
> > > > > > header blob
> > > > > > > and
> > > > > > > > IIUC
> > > > > > > > > you dont
> > > > > > > > > > need any
> > > > > > > > > > > >> >
> optionals
> > > > > > > > > > > >> >
> >>
> > > > anymore.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 3. yes,
> > > > > > we can :-)
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > On Tue,
> > > > > > Feb 14, 2017 at
> > > > > > > > 11:56
> > > > > > > > > PM, Michael
> > > > > > > > > > > Pearce <
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > wrote:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Hi
> > > > > Jay,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 1)
> > > > > > There was some
> > > > > > > > initial
> > > > > > > > > debate on the
> > > > > > > > > > value
> > > > > > > > > > > >> > part,
> as
> > youll
> > > > > > > > > > > >> >
> >>
> > > note
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > String,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > String
> > > > > > headers were
> > > > > > > > > discounted early
> > > > > > > > > > on. The
> > > > > > > > > > > >> >
> reason for
> > > this
> > > > > > > > > > > >> >
> >>
> > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > flexibility
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > and
> > > > > > keeping in line
> > > > > > > > with the
> > > > > > > > > > flexibility of
> > > > > > > > > > > key,
> > > > > > > > > > > >> > value
> of the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > message
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > object
> > > > > > itself. I don’t
> > > > > > > > think
> > > > > > > > > it takes
> > > > > > > > > > away
> > > > > > > > > > > from
> > > > > > > > > > > >> an
> > > > > > > > > > > >> >
> ecosystem
> > > > > > > > > > > >> >
> >>
> > as
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > each
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > plugin
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > will
> > > > > > care for their
> > > > > > > own
> > > > > > > > key,
> > > > > > > > > this way
> > > > > > > > > > ints,
> > > > > > > > > > > >> >
> booleans ,
> > > > exotic
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > custom
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > binary
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > can
> > > > > all
> > > > > > be catered
> > > > > > > for=.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a. If
> > > > > > you really
> > > > > > > wanted
> > > > > > > > to
> > > > > > > > > push for a
> > > > > > > > > > typed
> > > > > > > > > > > >> value
> > > > > > > > > > > >> >
> interface,
> > > > > > > > > > > >> >
> >> I
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > wouldn’t
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > want
> > > > > > just String
> > > > > > > values
> > > > > > > > > supported, but
> > > > > > > > > > the the
> > > > > > > > > > > >> >
> primatives
> > > > > > > > > > > >> >
> >>
> > > plus
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > string and
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > also
> > > > > > still keeping the
> > > > > > > > > ability to have a
> > > > > > > > > > > binary
> > > > > > > > > > > >> > for
> custom
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > binaries that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > some
> > > > > > organisations may
> > > > > > > > have.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > i. I
> > > > > > have written this
> > > > > > > > slight
> > > > > > > > > > alternative
> > > > > > > > > > > here,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > https://cwiki.apache.org/
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > > >> >
> > > > 82+-+Add+Record+Headers+-+
> > > > > Typed
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > ii.
> > > > > > Essentially the
> > > > > > > > value
> > > > > > > > > bytes, has a
> > > > > > > > > > leading
> > > > > > > > > > > >> > byte
> > overhead.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 1.
> > > > > > This tells you
> > > > > > > what
> > > > > > > > type
> > > > > > > > > the value
> > > > > > > > > > is,
> > > > > > > > > > > >> before
> > > > > > > > > > > >> >
> reading
> > > > > > > > > > > >> >
> >>
> > the
> > > > rest
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > of the
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > bytes,
> > > > > > allowing
> > > > > > > > > >
> > > > serialisation/deserialization
> > > > > > > > > > > to
> > > > > > > > > > > >> > and
> from the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > primitives,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > string
> > > > > > and byte[].
> > > > > > > This
> > > > > > > > is
> > > > > > > > > akin to some
> > > > > > > > > > other
> > > > > > > > > > > >> >
> messaging
> > > > > > > > > > > >> >
> >>
> > > > systems.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 2) We
> > > > > > are making it
> > > > > > > > optional,
> > > > > > > > > so that
> > > > > > > > > > for
> > > > > > > > > > > those
> > > > > > > > > > > >> > not
> wanting
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > headers have
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > 0
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > bytes
> > > > > > overhead (think
> > > > > > > > of it
> > > > > > > > > as a feature
> > > > > > > > > > > flag),
> > > > > > > > > > > >> I
> > > > > > > > > > > >> > don’t
> > > > > > > > > > > >> >
> >>
> > > > think this
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > complex, especially if
> > > > > > > > > comparing to
> > > > > > > > > > changes
> > > > > > > > > > > >> >
> proposed in
> > > > > > > > > > > >> >
> >>
> > > > other kips
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > like
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > kip-98.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a. If
> > > > > > you really
> > > > > > > really
> > > > > > > > don’t
> > > > > > > > > like
> > > > > > > > > > this, we
> > > > > > > > > > > can
> > > > > > > > > > > >> > drop
> it, but
> > > > > > > > > > > >> >
> >>
> > it
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > would
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > mean
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > buying
> > > > > > into 4 bytes
> > > > > > > > extra
> > > > > > > > > overhead for
> > > > > > > > > > users
> > > > > > > > > > > who
> > > > > > > > > > > >> > do
> not want
> > > > > > > > > > > >> >
> >>
> > to
> > > > use
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > headers.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 3) In
> > > > > > the summary yes,
> > > > > > > > it is
> > > > > > > > > at a higher
> > > > > > > > > > > level,
> > > > > > > > > > > >> > but I
> think
> > > > > > > > > > > >> >
> >>
> > > this
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > is well
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > documented in the
> > > > > > > > proposed
> > > > > > > > > changes
> > > > > > > > > > section.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a.
> > > > > > Added getHeaders
> > > > > > > > method to
> > > > > > > > > > >
> Producer/Consumer
> > > > > > > > > > > >> >
> record (that
> > > > > > > > > > > >> >
> >>
> > is
> > > > it)
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > b.
> > > > > > We’ve also detailed
> > > > > > > > the
> > > > > > > > > new Headers
> > > > > > > > > > class
> > > > > > > > > > > >> that
> > > > > > > > > > > >> > this
> method
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > returns
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > that
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > encapsulates the
> > > > > > > headers
> > > > > > > > > protocol and
> > > > > > > > > > logic.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Best,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Mike
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > ==Original questions
> > > > > > > > from the
> > > > > > > > > vote
> > > > > > > > > > thread from
> > > > > > > > > > > >> > Jay.==
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > Couple
> > > > > > of things I
> > > > > > > > think we
> > > > > > > > > still need
> > > > > > > > > > to work
> > > > > > > > > > > >> out:
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 1.
> > > > > I
> > > > > > think we agree
> > > > > > > > about
> > > > > > > > > the key,
> > > > > > > > > > but I
> > > > > > > > > > > >> think
> > > > > > > > > > > >> > we
> haven't
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > talked about
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > the
> > > > > > value yet. I
> > > > > > > > think if
> > > > > > > > > our goal
> > > > > > > > > > is an
> > > > > > > > > > > open
> > > > > > > > > > > >> >
> ecosystem
> > > > > > > > > > > >> >
> >>
> > of
> > > > these
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > header
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > spread across many
> > > > > > > > plugins
> > > > > > > > > from many
> > > > > > > > > > > systems
> > > > > > > > > > > >> we
> > > > > > > > > > > >> > should
> > > > > > > > > > > >> >
> >>
> > > > consider
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > making
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > this
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a
> > > > > > string as well so
> > > > > > > > it can
> > > > > > > > > be
> > > > > > > > > > printed, set
> > > > > > > > > > > >> via
> > > > > > > > > > > >> > a UI,
> set
> > > > > > > > > > > >> >
> >>
> > in
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > config,
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > etc.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > Basically
> > > > > > > encouraging
> > > > > > > > > pluggable
> > > > > > > > > > >
> serialization
> > > > > > > > > > > >> >
> formats
> > > > > > > > > > > >> >
> >>
> > > here
> > > > will
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > lead
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > a
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > bit
> > > > > > of a tower of
> > > > > > > > babel.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 2.
> > > > > > This proposal
> > > > > > > > still
> > > > > > > > > includes a
> > > > > > > > > > pretty
> > > > > > > > > > > big
> > > > > > > > > > > >> >
> change to
> > our
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > serialization
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > and
> > > > > > protocol
> > > > > > > > definition
> > > > > > > > > layer.
> > > > > > > > > > Essentially
> > > > > > > > > > > >> it is
> > > > > > > > > > > >> >
> >>
> > > > introducing an
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > optional
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > type, where the
> > > > > > > > format is
> > > > > > > > > data
> > > > > > > > > > dependent. I
> > > > > > > > > > > >> > think
> this
> > is
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > actually a
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > big
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > change though it
> > > > > > > > doesn't
> > > > > > > > > seem like
> > > > > > > > > > it. It
> > > > > > > > > > > >> means
> > > > > > > > > > > >> > you
> can no
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > longer
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > specify
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > this
> > > > > > type with our
> > > > > > > > type
> > > > > > > > > definition
> > > > > > > > > > DSL, and
> > > > > > > > > > > >> >
> likewise it
> > > > > > > > > > > >> >
> >>
> > > > requires
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > custom
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > handling in client
> > > > > > > > libs.
> > > > > > > > > This isn't
> > > > > > > > > > a huge
> > > > > > > > > > > >> >
> thing, since
> > > > > > > > > > > >> >
> >>
> > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Record
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > definition is
> > > > > > > custom
> > > > > > > > > anyway, but I
> > > > > > > > > > think
> > > > > > > > > > > this
> > > > > > > > > > > >> > kind
> of
> > > > > > > > > > > >> >
> >>
> > > > protocol
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > inconsistency is
> > > > > > > very
> > > > > > > > > non-desirable
> > > > > > > > > > and
> > > > > > > > > > > ties
> > > > > > > > > > > >> > you to
> > > > > > > > > > > >> >
> >>
> > > > hand-coding
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > things.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > I
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > think the type
> > > > > > > should
> > > > > > > > > instead by [Key
> > > > > > > > > > > Value]
> > > > > > > > > > > >> in
> > > > > > > > > > > >> > our
> BNF,
> > > > > > > > > > > >> >
> >>
> > > where
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > key and
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > value are both
> > > > > > > short
> > > > > > > > > strings as used
> > > > > > > > > > > >>
> elsewhere.
> > > > > > > > > > > >> > This
> > > > > > > > > > > >> >
> >>
> > > > brings it
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > in line
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > with
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > the
> > > > > > rest of the
> > > > > > > > protocol.
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > 3.
> > > > > > Could we get
> > > > > > > more
> > > > > > > > > specific about
> > > > > > > > > > the
> > > > > > > > > > > exact
> > > > > > > > > > > >> > Java
> API
> > > > > > > > > > > >> >
> >>
> > > > change to
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > ProducerRecord,
> > > > > > > > > ConsumerRecord,
> > > > > > > > > > Record,
> > > > > > > > > > > etc?
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > > -Jay
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > The
> > > > > information
> > > > > > contained in
> > > > > > > > this
> > > > > > > > > email is
> > > > > > > > > > strictly
> > > > > > > > > > > >> >
> confidential
> > > and
> > > > > > > > > > > >> >
> >>
> > for
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the use of the
> > > > > > addressee only,
> > > > > > > > unless
> > > > > > > > > otherwise
> > > > > > > > > > > >>
> indicated.
> > > > > > > > > > > >> > If you
> > > > > > > > > > > >> >
> >>
> > are
> > > > not
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > the intended
> > > > > > recipient, please
> > > > > > > > do not
> > > > > > > > > read,
> > > > > > > > > > copy, use
> > > > > > > > > > > or
> > > > > > > > > > > >> >
> disclose to
> > > > > > > > > > > >> >
> >>
> > > > others
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > this message
> > > > > or
> > > > > > any
> > > > > > > attachment.
> > > > > > > > > Please also
> > > > > > > > > > notify the
> > > > > > > > > > > >> >
> sender by
> > > > > > > > > > > >> >
> >>
> > > > replying
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > to this email
> > > > > > or by telephone
> > > > > > > > > (+44(020 7896
> > > > > > > > > > 0011) and
> > > > > > > > > > > >> then
> > > > > > > > > > > >> > delete
> > > > > > > > > > > >> >
> >>
> > the
> > > > email
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > and any copies
> > > > > > of it.
> > > > > > > Opinions,
> > > > > > > > > conclusion
> > > > > > > > > > (etc) that
> > > > > > > > > > > do
> > > > > > > > > > > >> > not
> relate
> > > > > > > > > > > >> >
> >>
> > to
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > official
> > > > > > business of this
> > > > > > > > company
> > > > > > > > > shall be
> > > > > > > > > > understood
> > > > > > > > > > > as
> > > > > > > > > > > >> >
> neither
> > > > > > > > > > > >> >
> >>
> > > > given nor
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > endorsed by
> > > > > it.
> > > > > > IG is a
> > > > > > > trading
> > > > > > > > name
> > > > > > > > > of IG
> > > > > > > > > > Markets
> > > > > > > > > > > >> Limited
> > > > > > > > > > > >> > (a
> company
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > registered in
> > > > > > England and
> > > > > > > Wales,
> > > > > > > > > company number
> > > > > > > > > > > >>
> 04008957)
> > > > > > > > > > > >> > and IG
> > > > > > > > > > > >> >
> >>
> > > Index
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Limited (a
> > > > > > company registered
> > > > > > > in
> > > > > > > > > England and
> > > > > > > > > > Wales,
> > > > > > > > > > > >> >
> company
> > number
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > 01190902).
> > > > > > Registered address
> > > > > > > at
> > > > > > > > > Cannon Bridge
> > > > > > > > > > House,
> > > > > > > > > > > 25
> > > > > > > > > > > >> >
> Dowgate
> > > > > > > > > > > >> >
> >>
> > > Hill,
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > London EC4R
> > > > > > 2YA. Both IG
> > > > > > > Markets
> > > > > > > > > Limited
> > > > > > > > > > (register
> > > > > > > > > > > >> number
> > > > > > > > > > > >> >
> 195355)
> > > > > > > > > > > >> >
> >>
> > and
> > > > IG
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Index Limited
> > > > > > (register number
> > > > > > > > > 114059) are
> > > > > > > > > > authorised
> > > > > > > > > > > >> and
> > > > > > > > > > > >> >
> regulated
> > > > > > > > > > > >> >
> >>
> > by
> > > > the
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > Financial
> > > > > > Conduct Authority.
> > > > > > > > > > > >> >
> >>
> > >>
> > > >
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >>
> > --
> > > > > > > > > > > >> >
> >>
> > > > Nacho - Ignacio
> > > > > > Solis -
> > > > > > > > isolis@igso.net
> > > > > > > > > > > >> >
> >>
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > The
> > > information
> > > > contained in
> > > > > > this email is
> > > > > > > > strictly
> > > > > > > > > > confidential and
> > > > > > > > > > > for
> > > > > > > > > > > >> > the
> use of
> > the
> > > > addressee only,
> > > > > > unless
> > > > > > > otherwise
> > > > > > > > > indicated. If
> > > > > > > > > > you are
> > > > > > > > > > > >> not
> > > > > > > > > > > >> > the
> intended
> > > > recipient, please
> > > > > > do not read,
> > > > > > > > copy, use
> > > > > > > > > or
> > > > > > > > > > disclose to
> > > > > > > > > > > >> others
> > > > > > > > > > > >> > this
> message
> > > or
> > > > any
> > > > > attachment.
> > > > > > Please also
> > > > > > > > notify
> > > > > > > > > the sender by
> > > > > > > > > > > >> replying
> > > > > > > > > > > >> > to
> this
> > email
> > > > or by telephone
> > > > > > (+44(020 7896
> > > > > > > > 0011) and
> > > > > > > > > then
> > > > > > > > > > delete the
> > > > > > > > > > > >> email
> > > > > > > > > > > >> > and
> any
> > copies
> > > > of it.
> > > > > Opinions,
> > > > > > conclusion
> > > > > > > > (etc) that
> > > > > > > > > do not
> > > > > > > > > > relate to
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >
> official
> > > > business of this
> > > > > > company shall be
> > > > > > > > understood
> > > > > > > > > as
> > > > > > > > > > neither given
> > > > > > > > > > > >> nor
> > > > > > > > > > > >> >
> endorsed by
> > > it.
> > > > IG is a
> > > > > trading
> > > > > > name of IG
> > > > > > > > Markets
> > > > > > > > > Limited (a
> > > > > > > > > > company
> > > > > > > > > > > >> >
> registered
> > in
> > > > England and
> > > > > > Wales, company
> > > > > > > number
> > > > > > > > > 04008957) and
> > > > > > > > > > IG Index
> > > > > > > > > > > >> >
> Limited (a
> > > > company registered
> > > > > > in England and
> > > > > > > > Wales,
> > > > > > > > > company
> > > > > > > > > > number
> > > > > > > > > > > >> >
> 01190902).
> > > > Registered address
> > > > > > at Cannon Bridge
> > > > > > > > House,
> > > > > > > > > 25
> > > > > > > > > > Dowgate Hill,
> > > > > > > > > > > >> >
> London EC4R
> > > > 2YA. Both IG
> > > > > > Markets Limited
> > > > > > > > (register
> > > > > > > > > number
> > > > > > > > > > 195355) and
> > > > > > > > > > > IG
> > > > > > > > > > > >> > Index
> > Limited
> > > > (register number
> > > > > > 114059) are
> > > > > > > > authorised
> > > > > > > > > and
> > > > > > > > > > regulated by
> > > > > > > > > > > >> the
> > > > > > > > > > > >> >
> Financial
> > > > Conduct Authority.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> The
> > information
> > > > contained in
> > > > > this
> > > > > > email is
> > > > > > > > strictly
> > > > > > > > > confidential
> > > > > > > > > > and for
> > > > > > > > > > > >> the use
> of the
> > > > addressee only,
> > > > > > unless otherwise
> > > > > > > > > indicated. If you
> > > > > > > > > > are
> > > > > > > > > > > not
> > > > > > > > > > > >> the
> intended
> > > > recipient, please
> > > > > do
> > > > > > not read,
> > > > > > > copy,
> > > > > > > > use
> > > > > > > > > or disclose
> > > > > > > > > > to
> > > > > > > > > > > others
> > > > > > > > > > > >> this
> message
> > or
> > > > any attachment.
> > > > > > Please also
> > > > > > > > notify the
> > > > > > > > > sender by
> > > > > > > > > > > replying
> > > > > > > > > > > >> to this
> email
> > or
> > > > by telephone
> > > > > > (+44(020 7896
> > > > > > > 0011)
> > > > > > > > and
> > > > > > > > > then delete
> > > > > > > > > > the
> > > > > > > > > > > email
> > > > > > > > > > > >> and any
> copies
> > > of
> > > > it. Opinions,
> > > > > > conclusion (etc)
> > > > > > > > that
> > > > > > > > > do not
> > > > > > > > > > relate to
> > > > > > > > > > > the
> > > > > > > > > > > >> official
> > > business
> > > > of this
> > > > > company
> > > > > > shall be
> > > > > > > > understood
> > > > > > > > > as neither
> > > > > > > > > > given
> > > > > > > > > > > nor
> > > > > > > > > > > >>
> endorsed by
> > it.
> > > > IG is a trading
> > > > > > name of IG
> > > > > > > Markets
> > > > > > > > > Limited (a
> > > > > > > > > > company
> > > > > > > > > > > >>
> registered in
> > > > England and Wales,
> > > > > > company number
> > > > > > > > > 04008957) and IG
> > > > > > > > > > Index
> > > > > > > > > > > >> Limited
> (a
> > > > company registered in
> > > > > > England and
> > > > > > > > Wales,
> > > > > > > > > company number
> > > > > > > > > > > >>
> 01190902).
> > > > Registered address at
> > > > > > Cannon Bridge
> > > > > > > > House,
> > > > > > > > > 25 Dowgate
> > > > > > > > > > Hill,
> > > > > > > > > > > >> London
> EC4R
> > 2YA.
> > > > Both IG Markets
> > > > > > Limited
> > > > > > > (register
> > > > > > > > > number 195355)
> > > > > > > > > > and IG
> > > > > > > > > > > >> Index
> Limited
> > > > (register number
> > > > > > 114059) are
> > > > > > > > authorised
> > > > > > > > > and
> > > > > > > > > > regulated by
> > > > > > > > > > > the
> > > > > > > > > > > >>
> Financial
> > > Conduct
> > > > Authority.
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > The
> information
> > > > contained in this
> > > > > > email is strictly
> > > > > > > > > confidential and
> > > > > > > > > > for the use of
> the
> > > > addressee only, unless
> > > > > > otherwise
> > > > > > > > indicated.
> > > > > > > > > If you are
> > > > > > > > > > not the intended
> > > > recipient, please do not
> > > > > > read, copy, use
> > > > > > > > or
> > > > > > > > > disclose to
> > > > > > > > > > others this
> message or
> > > any
> > > > attachment.
> > > > > > Please also notify
> > > > > > > > the
> > > > > > > > > sender by
> > > > > > > > > > replying to this
> email
> > or
> > > > by telephone
> > > > > > (+44(020 7896
> > > > > > > 0011)
> > > > > > > > and
> > > > > > > > > then delete
> > > > > > > > > > the email and any
> > copies
> > > > of it. Opinions,
> > > > > > conclusion
> > > > > > > (etc)
> > > > > > > > that
> > > > > > > > > do not
> > > > > > > > > > relate to the
> official
> > > > business of this
> > > > > > company shall be
> > > > > > > > > understood as
> > > > > > > > > > neither given nor
> > > endorsed
> > > > by it. IG is a
> > > > > > trading name of
> > > > > > > > IG
> > > > > > > > > Markets
> > > > > > > > > > Limited (a
> company
> > > > registered in England
> > > > > > and Wales,
> > > > > > > company
> > > > > > > > > number
> > > > > > > > > > 04008957) and IG
> Index
> > > > Limited (a company
> > > > > > registered in
> > > > > > > > England
> > > > > > > > > and Wales,
> > > > > > > > > > company number
> > 01190902).
> > > > Registered
> > > > > > address at Cannon
> > > > > > > > Bridge
> > > > > > > > > House, 25
> > > > > > > > > > Dowgate Hill,
> London
> > EC4R
> > > > 2YA. Both IG
> > > > > > Markets Limited
> > > > > > > > (register
> > > > > > > > > number
> > > > > > > > > > 195355) and IG
> Index
> > > > Limited (register
> > > > > > number 114059) are
> > > > > > > > > authorised and
> > > > > > > > > > regulated by the
> > > Financial
> > > > Conduct
> > > > > > Authority.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > The information
> contained in
> > > > this email is
> > > > > > strictly
> > > > > > > confidential
> > > > > > > > and
> > > > > > > > > for the use of the
> addressee
> > only,
> > > > unless otherwise
> > > > > > indicated. If
> > > > > > > > you are
> > > > > > > > > not the intended recipient,
> > please
> > > > do not read,
> > > > > > copy, use or
> > > > > > > > disclose to
> > > > > > > > > others this message or any
> > > > attachment. Please also
> > > > > > notify the
> > > > > > > sender
> > > > > > > > by
> > > > > > > > > replying to this email or
> by
> > > > telephone (+44(020
> > > > > 7896
> > > > > > 0011) and then
> > > > > > > > delete
> > > > > > > > > the email and any copies
> of it.
> > > > Opinions,
> > > > > conclusion
> > > > > > (etc) that do
> > > > > > > > not
> > > > > > > > > relate to the official
> business
> > of
> > > > this company
> > > > > > shall be understood
> > > > > > > > as
> > > > > > > > > neither given nor endorsed
> by it.
> > > IG
> > > > is a trading
> > > > > > name of IG
> > > > > > > Markets
> > > > > > > > > Limited (a company
> registered in
> > > > England and Wales,
> > > > > > company number
> > > > > > > > > 04008957) and IG Index
> Limited (a
> > > > company
> > > > > registered
> > > > > > in England and
> > > > > > > > Wales,
> > > > > > > > > company number 01190902).
> > > Registered
> > > > address at
> > > > > > Cannon Bridge
> > > > > > > House,
> > > > > > > > 25
> > > > > > > > > Dowgate Hill, London EC4R
> 2YA.
> > Both
> > > > IG Markets
> > > > > > Limited (register
> > > > > > > > number
> > > > > > > > > 195355) and IG Index
> Limited
> > > > (register number
> > > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated by the Financial
> > Conduct
> > > > Authority.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in
> this email
> > > is
> > > > strictly
> > > > > > confidential and for
> > > > > > > > the use of the addressee only,
> unless
> > > > otherwise
> > > > > indicated.
> > > > > > If you are not
> > > > > > > > the intended recipient, please
> do not
> > > > read, copy, use or
> > > > > > disclose to
> > > > > > > others
> > > > > > > > this message or any attachment.
> Please
> > > > also notify the
> > > > > > sender by replying
> > > > > > > > to this email or by telephone
> (+44(020
> > > > 7896 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > email
> > > > > > > > and any copies of it. Opinions,
> > > conclusion
> > > > (etc) that do
> > > > > > not relate to
> > > > > > > the
> > > > > > > > official business of this
> company shall
> > > be
> > > > understood as
> > > > > > neither given
> > > > > > > nor
> > > > > > > > endorsed by it. IG is a trading
> name of
> > > IG
> > > > Markets
> > > > > Limited
> > > > > > (a company
> > > > > > > > registered in England and Wales,
> > company
> > > > number 04008957)
> > > > > > and IG Index
> > > > > > > > Limited (a company registered in
> > England
> > > > and Wales,
> > > > > > company number
> > > > > > > > 01190902). Registered address at
> Cannon
> > > > Bridge House, 25
> > > > > > Dowgate Hill,
> > > > > > > > London EC4R 2YA. Both IG Markets
> > Limited
> > > > (register number
> > > > > > 195355) and IG
> > > > > > > > Index Limited (register number
> 114059)
> > > are
> > > > authorised and
> > > > > > regulated by
> > > > > > > the
> > > > > > > > Financial Conduct Authority.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email is
> strictly
> > > > confidential and
> > > > > > for the use of the addressee only, unless
> otherwise
> > > > indicated. If you are
> > > > > > not the intended recipient, please do not read,
> copy,
> > use
> > > > or disclose to
> > > > > > others this message or any attachment. Please
> also
> > notify
> > > > the sender by
> > > > > > replying to this email or by telephone (+44(020
> 7896
> > > 0011)
> > > > and then
> > > > > delete
> > > > > > the email and any copies of it. Opinions,
> conclusion
> > > (etc)
> > > > that do not
> > > > > > relate to the official business of this company
> shall
> > be
> > > > understood as
> > > > > > neither given nor endorsed by it. IG is a
> trading name
> > of
> > > > IG Markets
> > > > > > Limited (a company registered in England and
> Wales,
> > > > company number
> > > > > > 04008957) and IG Index Limited (a company
> registered in
> > > > England and
> > > > > Wales,
> > > > > > company number 01190902). Registered address at
> Cannon
> > > > Bridge House, 25
> > > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> Limited
> > > > (register number
> > > > > > 195355) and IG Index Limited (register number
> 114059)
> > are
> > > > authorised and
> > > > > > regulated by the Financial Conduct Authority.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > The information contained in this email is strictly confidential
> and
> > for
> > > > the use of the addressee only, unless otherwise indicated. If
> you are
> > not
> > > > the intended recipient, please do not read, copy, use or
> disclose to
> > > others
> > > > this message or any attachment. Please also notify the sender by
> > replying
> > > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > > the
> > > > official business of this company shall be understood as neither
> given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > > > registered in England and Wales, company number 04008957) and IG
> Index
> > > > Limited (a company registered in England and Wales, company
> number
> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
> > IG
> > > > Index Limited (register number 114059) are authorised and
> regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
This works whilst no other feature needs similar. I prefer to keep it contained/encapsulated it then allows the bytes of the headers to also be copied transferred as one without any parsing what so ever, but if you really feel strongly on the below ill go with it, just want to flag this concern.
Btw, the below notation seems to be different to the notation used in previous kips like kip-32 which I took the notation from, where the length of the following bytes for byte arrays is clearly defined. So im 100% on what you propose could you define it in this original styled notation?
On 20/02/2017, 19:58, "Jason Gustafson" <ja...@confluent.io> wrote:
Hey Jun, that's a good thought. Maybe then the full schema could look like
this:
Record =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
Key => bytes
Value => bytes
Headers => [HeaderKey HeaderValue]
HeaderKey => string
HeaderValue => bytes
(where the bytes and string type use a variable length size field)
Does that work for you, Michael?
-Jason
On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
> Regarding the overhead of skipping the headers in the message. Another way
> could be to add the headers at the end of the message after the value part.
> This way, there is no need to skip the headers unless you really need it.
> This will allow us to represent the headers explicitly as an array of
> key/value pairs without the skipping overhead when headers are not needed.
>
> Thanks,
>
> Jun
>
> On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Michael,
> >
> > Here's a link to the branch with the message format changes:
> > https://github.com/confluentinc/kafka/tree/exactly-once-message-format.
> > The
> > class to modify to add headers is EosLogRecord (name will definitely
> change
> > prior to merging in trunk). This branch uses attributes to indicate null
> > key and value. I really don't have a strong preference for or against
> this
> > as long as we're consistent. I do feel it's kind of a pity to need the
> > signed version of the varint just to be able to indicate null. For
> headers,
> > it's an easier call because we don't actually need to represent null
> > headers (using size zero will work fine).
> >
> > -Jason
> >
> > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > We can make it, implement Iterable<Header>, no concerns on this.
> > >
> > > Ive noted this on the KIP.
> > >
> > >
> > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com> wrote:
> > >
> > > Btw Jason, can you share the branch with the VarInts helper/util
> > class
> > > that KIP-98 will be introducing and also will have the current modified
> > > (key and value lengths) message, so we can look at, thanks.
> > >
> > > On 20/02/2017, 01:11, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > RE “ProducerRecord and ConsumerRecord are immutable
> currently”. I
> > > beg to differ on this, if I set a byte[] as my value of my record, I
> can
> > > later call value() and then alter the bytes in the array. As such it is
> > > very much mutable.
> > >
> > > Headers also will anyhow by nature be mutable on produce, as
> you
> > > add to them.
> > >
> > >
> > > On 20/02/2017, 00:22, "Jason Gustafson" <ja...@confluent.io>
> > > wrote:
> > >
> > > Re; having a dedicated Headers object. I think it makes
> > sense.
> > > Maybe it
> > > could implement Iterable<Header>?
> > >
> > > One thing I'm not too sure about is mutability. Both the
> > > ProducerRecord and
> > > ConsumerRecord are immutable currently. It would be nice to
> > > keep it that
> > > way. Perhaps Headers could itself be immutable and passed
> > > through the
> > > constructors?
> > >
> > > Just going through how this is, I don’t believe this would
> > > provide much
> > > > saving. If issue trying to resolve is around
> object/garbage
> > > creation of
> > > > Map.Entry objects.
> > >
> > >
> > > It's definitely better because we only allocate the space
> we
> > > will use, but
> > > I assume most of the actual memory cost is in the header
> data
> > > itself which
> > > we can't do much about. Maybe we can start with the simple
> > > array
> > > representation and optimize from there after we find it's
> > > worthwhile to do
> > > so? If we use the Headers object, we can change this later
> > > without breaking
> > > the API.
> > >
> > > -Jason
> > >
> > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> > > radai.rosenblatt@gmail.com> wrote:
> > >
> > > > "(By the way, doesn't it feel a bit odd that we seem to
> be
> > > designing a
> > > > feature which is optimized for people not using?)"
> > > >
> > > > very. this (i claim :-) ) is the result of very intense
> > > opposition to the
> > > > usefulness of the feature early on, and not a real design
> > > goal
> > > >
> > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > Whilst if having an array of a Header[] would mean
> > getting
> > > this through,
> > > > > im happy changing this, as already done.
> > > > >
> > > > > Just going through how this is, I don’t believe this
> > > would provide much
> > > > > saving. If issue trying to resolve is around
> > > object/garbage creation of
> > > > > Map.Entry objects.
> > > > >
> > > > > All you’re doing here is replacing the equiv of a
> > > HashMap.Node
> > > > (Map.Entry)
> > > > > which simply holds reference to key and value objects
> > with
> > > a custom
> > > > variant.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 19/02/2017, 20:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com> wrote:
> > > > >
> > > > > On point 1 & 2 Ive updated KIP to show varints (and
> > > removed the bit
> > > > > flag). (on the assumption KIP 98 is getting the
> agreement
> > > the protocol is
> > > > > moving from int32 to varInts as standard)
> > > > >
> > > > > On point 3 ive updated to use an array of Header
> > > class, instead of a
> > > > > MultiMap in the Headers class object
> > > > >
> > > > >
> > > > >
> > > > > On 19/02/2017, 20:06, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > On points 1 and 2 I agree.
> > > > >
> > > > > This also affects kip-98, I should expect this
> > > resolved before
> > > > > that vote also passes. If it is accepted there (I’m
> > > assuming this is
> > > > > getting discussed on that KIP? As you’re driving the
> move
> > > to VarInts), I
> > > > am
> > > > > happy to make this KIP will simply follow suit to
> > whatever
> > > is agreed in
> > > > > KIP-98.
> > > > >
> > > > > On 3) Agreed this is a simpler form, as long as
> > no
> > > one is
> > > > > expecting hashmap lookup performance ( O(1) ), I still
> > > would prefer an
> > > > > encapsulated class, so if we find that holding it as an
> > > array in future
> > > > is
> > > > > causing some perf issues, the internals are not exposed
> > to
> > > end users,
> > > > > allowing the internal structure to move to a map.
> > > > >
> > > > > Can we compromise on?
> > > > >
> > > > > class ConsumerRecord<K, V> {
> > > > > K key;
> > > > > V value;
> > > > > Headers headers;
> > > > > }
> > > > >
> > > > > class Headers {
> > > > > Header[] headers;
> > > > >
> > > > > add(String key, byte[] value)
> > > > > Collection<byte[]> get(String key)
> > > > > }
> > > > >
> > > > > class Header {
> > > > > String key;
> > > > > byte[] value;
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > headers dont "leak" into application
> code.
> > > they are useful
> > > > > to application
> > > > > > code as well.
> > > > >
> > > > >
> > > > > This is exactly what I have been trying to
> > get
> > > at. The use
> > > > > cases documented
> > > > > here are middleware:
> > > > > https://cwiki.apache.org/
> > > confluence/display/KAFKA/A+
> > > > > Case+for+Kafka+Headers.
> > > > > If headers are intended for the application
> > > space as well,
> > > > the
> > > > > document
> > > > > should be updated accordingly so that it is
> > an
> > > explicit
> > > > design
> > > > > goal and not
> > > > > an unfortunate design byproduct. There may
> be
> > > some
> > > > > disagreement on whether
> > > > > it _should_ be a design goal, but it may
> not
> > > matter much
> > > > since
> > > > > the current
> > > > > interceptor API is probably insufficient
> for
> > > middleware
> > > > > applications (I
> > > > > haven't thought of a way to do this that
> > isn't
> > > cumbersome).
> > > > >
> > > > > That aside, the three unresolved points for
> > me
> > > are the
> > > > > following:
> > > > >
> > > > > 1. The use of varints. As a one-off for use
> > > only with record
> > > > > headers, the
> > > > > case was weak, but if we are using them
> > > throughout the
> > > > message
> > > > > format, then
> > > > > we should do so here as well. The
> additional
> > > complexity is
> > > > > minimal and
> > > > > early performance testing fully justifies
> > > their use.
> > > > >
> > > > > 2. If varints are used, the case for using
> > > attributes to
> > > > > indicate null
> > > > > headers also becomes weak. We only add one
> > > additional byte in
> > > > > each message
> > > > > if there are no headers. Whatever the case,
> > > let us be
> > > > > consistent with how
> > > > > we treat null keys and values.
> > > > >
> > > > > 3. We have apparently agreed that the
> broker
> > > will validate
> > > > > headers (please
> > > > > add this to the KIP). That being the case,
> I
> > > would prefer to
> > > > > use Kafka's
> > > > > conventional format for arrays. The
> downside
> > > is that it takes
> > > > > more work to
> > > > > skip over the headers on the consumer,
> though
> > > it's unclear if
> > > > > the cost
> > > > > would matter in practice. Some concern was
> > > previously
> > > > > expressed about the
> > > > > allocation of maps. An alternative would be
> > to
> > > use arrays,
> > > > i.e.
> > > > >
> > > > > class ConsumerRecord<K, V> {
> > > > > K key;
> > > > > V value;
> > > > > Header[] headers;
> > > > > }
> > > > >
> > > > > class Header {
> > > > > String key;
> > > > > byte[] value;
> > > > > }
> > > > >
> > > > > This would work nicely with the
> conventional
> > > array format and
> > > > > my guess is
> > > > > it would obviate the need do any lazy
> > > initialization. If we
> > > > > use the map as
> > > > > is currently documented, then it is
> possible
> > > with either
> > > > > representation to
> > > > > slice the headers and initialize them
> lazily.
> > > Either way, it
> > > > > might be a
> > > > > good idea to use a separate object to
> > > represent the headers
> > > > in
> > > > > case we need
> > > > > to modify it in the future in some way.
> > > > >
> > > > > (By the way, doesn't it feel a bit odd that
> > we
> > > seem to be
> > > > > designing a
> > > > > feature which is optimized for people not
> > > using?)
> > > > >
> > > > >
> > > > > If we can resolve these points, then at
> least
> > > you will get my
> > > > > vote.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > > > > radai.rosenblatt@gmail.com> wrote:
> > > > >
> > > > > > headers dont "leak" into application
> code.
> > > they are useful
> > > > > to application
> > > > > > code as well.
> > > > > > IIUC samze currently has headers "in-V"
> and
> > > would just
> > > > > switch over to kafka
> > > > > > headers if they exist.
> > > > > > im sure plenty of other users of kafka
> > would
> > > have a use for
> > > > > headers.
> > > > > > im pretty sure use cases exist around
> > > shuffling data
> > > > > into/out-of kafka
> > > > > > (kafka connect or equivalent) where
> > metadata
> > > from one end
> > > > > could copied over
> > > > > > to the other (S3, for example uses http
> > > headers for
> > > > > user-accessible
> > > > > > metadata). it will be kafka client code
> > > getting/setting
> > > > > those headers. not
> > > > > > an interceptor.
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 1:41 PM, Michael
> > > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > For APM single event tracing, need
> access
> > > to the header
> > > > at
> > > > > the point of
> > > > > > > processing on the processing thread.
> > > > > > >
> > > > > > > As such interceptors will not work/be
> > > suitable for these,
> > > > > due to the fact
> > > > > > > they act on the ConsumerRecords as a
> > > batch, before the
> > > > > handling thread
> > > > > > can
> > > > > > > split out and process per message which
> > is
> > > the point
> > > > these
> > > > > tools will
> > > > > > need
> > > > > > > to continue to transaction tracing.
> > > > > > >
> > > > > > > Like wise tools and infra pieces will
> > need
> > > access to the
> > > > > message outside
> > > > > > > the interceptor realm.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 17/02/2017, 21:26, "Jason
> Gustafson" <
> > > > > jason@confluent.io> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > That’s exactly what we’re doing
> the
> > > headers are a
> > > > > slice of bytes,
> > > > > > > which
> > > > > > > > then gets parsed later if needed,
> > or
> > > can be parsed
> > > > > right away, the
> > > > > > > headers
> > > > > > > > is part of the protocol, so can
> > > still be validated
> > > > > if wanted.
> > > > > > > > If you had a header count then
> you
> > > would have to go
> > > > > through each
> > > > > > > header
> > > > > > > > key and value length value to
> work
> > > out how much to
> > > > > skip to get to
> > > > > > > say the
> > > > > > > > value or any future component in
> > the
> > > message after
> > > > > the headers.
> > > > > > > Having it
> > > > > > > > as a byte[] with length value
> makes
> > > this a lot
> > > > > easier to skip.
> > > > > > >
> > > > > > >
> > > > > > > So the broker will parse the
> headers
> > > and validate
> > > > > them. Good. The
> > > > > > only
> > > > > > > reason remaining that I can see to
> > > leave the headers
> > > > > as a byte array
> > > > > > > is to
> > > > > > > make it easier for the client to
> skip
> > > past them. Are
> > > > > we sure this is
> > > > > > > not
> > > > > > > premature optimization? Are there
> any
> > > performance
> > > > > results which show
> > > > > > > that
> > > > > > > this is worthwhile?
> > > > > > >
> > > > > > > What’s the issue with exposing a
> > > method getHeaders on
> > > > > the
> > > > > > > producer/consumer
> > > > > > > > record? It doesn’t break
> anything.
> > > We don’t need
> > > > any
> > > > > special
> > > > > > version.
> > > > > > >
> > > > > > >
> > > > > > > See my previous explanation. What I
> > am
> > > trying to
> > > > > resist is the
> > > > > > headers
> > > > > > > becoming a general
> application-level
> > > facility. The
> > > > > primary use case
> > > > > > as
> > > > > > > far
> > > > > > > as I can tell is middleware, which
> is
> > > the use case
> > > > > that the
> > > > > > > interceptors
> > > > > > > are providing.
> > > > > > >
> > > > > > > Current batch consumer model and
> > > consumer
> > > > interceptors
> > > > > don’t work
> > > > > > where
> > > > > > > > headers need to be acted on at
> per
> > > message level at
> > > > > time of
> > > > > > > processing,
> > > > > > > > very case is APM (the core one),
> > > where the header
> > > > > value is used to
> > > > > > > continue
> > > > > > > > tracing.
> > > > > > >
> > > > > > >
> > > > > > > I still don't understand the point
> > > about batching.
> > > > The
> > > > > consumer
> > > > > > > records are
> > > > > > > exposed as a batch in the consumer
> > > interceptor, but
> > > > > you can still
> > > > > > > iterate
> > > > > > > them individually. It is no
> different
> > > for the
> > > > consumer
> > > > > API itself.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 12:48 PM,
> > > Michael Pearce <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Re:
> > > > > > > >
> > > > > > > > “ The point about creation of
> > > maps seems
> > > > > orthogonal. We can
> > > > > > still
> > > > > > > > represent
> > > > > > > > the headers as a slice of
> bytes
> > > until the time
> > > > > it is accessed.”
> > > > > > > >
> > > > > > > > That’s exactly what we’re doing
> the
> > > headers are a
> > > > > slice of bytes,
> > > > > > > which
> > > > > > > > then gets parsed later if needed,
> > or
> > > can be parsed
> > > > > right away, the
> > > > > > > headers
> > > > > > > > is part of the protocol, so can
> > > still be validated
> > > > > if wanted.
> > > > > > > >
> > > > > > > > If you had a header count then
> you
> > > would have to go
> > > > > through each
> > > > > > > header
> > > > > > > > key and value length value to
> work
> > > out how much to
> > > > > skip to get to
> > > > > > > say the
> > > > > > > > value or any future component in
> > the
> > > message after
> > > > > the headers.
> > > > > > > Having it
> > > > > > > > as a byte[] with length value
> makes
> > > this a lot
> > > > > easier to skip.
> > > > > > > >
> > > > > > > >
> > > > > > > > On 17/02/2017, 20:37, "Michael
> > > Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > What’s the issue with
> exposing
> > a
> > > method
> > > > > getHeaders on the
> > > > > > > > producer/consumer record? It
> > doesn’t
> > > break
> > > > anything.
> > > > > We don’t need
> > > > > > > any
> > > > > > > > special version.
> > > > > > > >
> > > > > > > > Current batch consumer model
> > and
> > > consumer
> > > > > interceptors don’t
> > > > > > work
> > > > > > > > where headers need to be acted on
> > at
> > > per message
> > > > > level at time of
> > > > > > > > processing, very case is APM (the
> > > core one), where
> > > > > the header value
> > > > > > > is used
> > > > > > > > to continue tracing. JMS/HTTP etc
> > > all expose these,
> > > > > without issues.
> > > > > > > I would
> > > > > > > > NOT want to lock this down to
> only
> > > be usable
> > > > > accessible via
> > > > > > > interceptors,
> > > > > > > > as we’d fail on one of the main
> > > goals.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Mike
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 17/02/2017, 20:21, "Jason
> > > Gustafson" <
> > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > The point about creation
> of
> > > maps seems
> > > > > orthogonal. We can
> > > > > > > still
> > > > > > > > represent
> > > > > > > > the headers as a slice of
> > > bytes until the
> > > > > time it is
> > > > > > > accessed.
> > > > > > > >
> > > > > > > >
> > > > > > > > > Yes exactly we have
> > access
> > > to the records
> > > > > thus why the
> > > > > > > header
> > > > > > > > should be
> > > > > > > > > accessible via it and
> not
> > > hidden for only
> > > > > interceptors to
> > > > > > > access.
> > > > > > > >
> > > > > > > >
> > > > > > > > As explained above, the
> > > point is to make
> > > > the
> > > > > intended usage
> > > > > > > clear.
> > > > > > > > Applications should
> > continue
> > > to rely on the
> > > > > key/value
> > > > > > fields
> > > > > > > to
> > > > > > > > serialize
> > > > > > > > their own headers, and it
> > > would be more
> > > > > ideal if we can
> > > > > > avoid
> > > > > > > > leaking
> > > > > > > > third-party headers into
> > > applications. This
> > > > > is difficult to
> > > > > > > do
> > > > > > > > with the
> > > > > > > > current interceptors
> > because
> > > they share the
> > > > > record objects
> > > > > > > with
> > > > > > > > the common
> > > > > > > > API. What I had in mind
> is
> > > something like
> > > > an
> > > > > extension of
> > > > > > the
> > > > > > > > current
> > > > > > > > interceptors which
> exposed
> > a
> > > different
> > > > > object (e.g.
> > > > > > > > `RecordAndHeaders`).
> > > > > > > > The challenge is for
> > MM-like
> > > use cases. Let
> > > > > me see if I can
> > > > > > > come
> > > > > > > > up with a
> > > > > > > > concrete proposal for
> that
> > > problem.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at
> > > 11:55 AM, Michael
> > > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I am happy to move the
> > > definition of the
> > > > > header into the
> > > > > > > message
> > > > > > > > body, but
> > > > > > > > > would cause us not to
> > lazy
> > > > > initialise/parse the headers,
> > > > > > as
> > > > > > > > obviously, we
> > > > > > > > > would have to traverse
> > > these reading the
> > > > > message.
> > > > > > > > >
> > > > > > > > > This was actually one
> of
> > > Jay’s requests:
> > > > > > > > >
> > > > > > > > > “ 2. I think we
> should
> > > think about
> > > > > creating the Map
> > > > > > > lazily to
> > > > > > > > avoid
> > > > > > > > > parsing out all the
> > > headers into
> > > > > little objects.
> > > > > > > HashMaps
> > > > > > > > themselves
> > > > > > > > > are
> > > > > > > > > kind of expensive
> and
> > > the consumer is
> > > > > very perf
> > > > > > > sensitive so
> > > > > > > > and making
> > > > > > > > > gazillions of
> > hashmaps
> > > that may or
> > > > may
> > > > > not get used
> > > > > > is
> > > > > > > > probably a bad
> > > > > > > > > idea.”
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 17/02/2017, 19:44,
> > > "Michael Pearce" <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Yes exactly we have
> > > access to the
> > > > > records thus why
> > > > > > the
> > > > > > > > header should
> > > > > > > > > be accessible via it
> and
> > > not hidden for
> > > > > only interceptors
> > > > > > > to
> > > > > > > > access.
> > > > > > > > >
> > > > > > > > > Sent using OWA for
> > > iPhone
> > > > > > > > >
> > > ______________________________
> > > > > __________
> > > > > > > > > From: Magnus
> > Edenhill <
> > > > > magnus@edenhill.se>
> > > > > > > > > Sent: Friday,
> > February
> > > 17, 2017
> > > > > 7:34:49 PM
> > > > > > > > > To:
> > > dev@kafka.apache.org
> > > > > > > > > Subject: Re:
> > [DISCUSS]
> > > KIP-82 - Add
> > > > > Record Headers
> > > > > > > > >
> > > > > > > > > Big +1 on VarInts.
> > > > > > > > > CPUs are fast,
> memory
> > > is slow.
> > > > > > > > >
> > > > > > > > > I agree with Jason
> > > that we'll want to
> > > > > continue
> > > > > > > verifying
> > > > > > > > messages,
> > > > > > > > > including their
> > > headers, so while I
> > > > > appreciate the
> > > > > > > idea of
> > > > > > > > the opaque
> > > > > > > > > header blob it
> won't
> > > be useful in
> > > > > practice.
> > > > > > > > >
> > > > > > > > > /Magnus
> > > > > > > > >
> > > > > > > > > 2017-02-17 10:41
> > > GMT-08:00 Jason
> > > > > Gustafson <
> > > > > > > > jason@confluent.io>:
> > > > > > > > >
> > > > > > > > > > Sorry, my
> mistake.
> > > The consumer
> > > > > interceptor is per
> > > > > > > batch,
> > > > > > > > though I'm
> > > > > > > > > not
> > > > > > > > > > sure that's an
> > > actual limitation
> > > > > since you still
> > > > > > have
> > > > > > > > access to the
> > > > > > > > > > individual
> records.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 17,
> > 2017
> > > at 10:39 AM,
> > > > > Jason Gustafson <
> > > > > > > > > jason@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Re headers as
> > byte
> > > array and
> > > > > future use by
> > > > > > broker.
> > > > > > > This
> > > > > > > > doesn't
> > > > > > > > > take away
> > > > > > > > > > >> from that at
> > all.
> > > Nor makes it
> > > > > difficult at all
> > > > > > > in my
> > > > > > > > opinion.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yeah, I didn't
> > say
> > > it was
> > > > > difficult, only
> > > > > > awkward.
> > > > > > > You
> > > > > > > > wouldn't
> > > > > > > > > write the
> > > > > > > > > > > schema that way
> > if
> > > you were
> > > > > planning to use it on
> > > > > > > the
> > > > > > > > brokers from
> > > > > > > > > the
> > > > > > > > > > > beginning. Note
> > > also that one of
> > > > > the benefits of
> > > > > > > letting
> > > > > > > > the broker
> > > > > > > > > > > understand
> > headers
> > > is that it can
> > > > > validate that
> > > > > > > they are
> > > > > > > > properly
> > > > > > > > > > > formatted. If
> > cost
> > > is the only
> > > > > concern, we should
> > > > > > > > confirm its
> > > > > > > > > impact
> > > > > > > > > > > through
> > > performance testing.
> > > > > > > > > > >
> > > > > > > > > > > One of the key
> > use
> > > cases requires
> > > > > access on
> > > > > > > consume at
> > > > > > > > per
> > > > > > > > > event/message
> > > > > > > > > > >> level at the
> > > point that message
> > > > > is being
> > > > > > > processed, as
> > > > > > > > such the
> > > > > > > > > batch
> > > > > > > > > > >> interceptors
> and
> > > batch consume
> > > > > api isn't
> > > > > > > suitable. It
> > > > > > > > needs to be
> > > > > > > > > at the
> > > > > > > > > > >> record level.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I'm not sure I
> > > understand the
> > > > > point about
> > > > > > batching.
> > > > > > > > Interceptors
> > > > > > > > > are
> > > > > > > > > > > applied
> > > per-message, right?
> > > > > > > > > > >
> > > > > > > > > > > My intent on
> > > interceptors is to
> > > > > keep the usage of
> > > > > > > headers
> > > > > > > > > well-defined so
> > > > > > > > > > > that they don't
> > > start leaking
> > > > > unnecessarily into
> > > > > > > > applications. My
> > > > > > > > > guess
> > > > > > > > > > is
> > > > > > > > > > > that it's
> > probably
> > > inevitable,
> > > > but
> > > > > isolating it
> > > > > > in
> > > > > > > the
> > > > > > > > > interceptors would
> > > > > > > > > > > at least give
> > > people a second
> > > > > thought before
> > > > > > > deciding to
> > > > > > > > use it.
> > > > > > > > > The main
> > > > > > > > > > > challenge in my
> > > mind is figuring
> > > > > out how an MM
> > > > > > use
> > > > > > > case
> > > > > > > > would
> > > > > > > > > work. It
> > > > > > > > > > > would be more
> > > cumbersome to
> > > > > replicate headers
> > > > > > > through an
> > > > > > > > > interceptor,
> > > > > > > > > > > though arguably
> > MM
> > > should be
> > > > > working at a lower
> > > > > > > level
> > > > > > > > anyway.
> > > > > > > > > > >
> > > > > > > > > > > -Jason
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 17,
> > > 2017 at 10:16 AM,
> > > > > Michael Pearce
> > > > > > <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Re headers
> > > available on the
> > > > > record va
> > > > > > > interceptors only
> > > > > > > > > > >>
> > > > > > > > > > >> One of the key
> > > use cases
> > > > requires
> > > > > access on
> > > > > > > consume at
> > > > > > > > per
> > > > > > > > > event/message
> > > > > > > > > > >> level at the
> > > point that message
> > > > > is being
> > > > > > > processed, as
> > > > > > > > such the
> > > > > > > > > batch
> > > > > > > > > > >> interceptors
> and
> > > batch consume
> > > > > api isn't
> > > > > > > suitable. It
> > > > > > > > needs to be
> > > > > > > > > at the
> > > > > > > > > > >> record level.
> > > > > > > > > > >>
> > > > > > > > > > >> This anyhow is
> > > similar to
> > > > > jms/http/amqp where
> > > > > > > headers
> > > > > > > > are
> > > > > > > > > available to
> > > > > > > > > > >> consuming
> > > applications.
> > > > > > > > > > >>
> > > > > > > > > > >> Re headers as
> > > byte array and
> > > > > future use by
> > > > > > > broker. This
> > > > > > > > doesn't
> > > > > > > > > take
> > > > > > > > > > away
> > > > > > > > > > >> from that at
> > all.
> > > Nor makes it
> > > > > difficult at all
> > > > > > > in my
> > > > > > > > opinion.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Sent using OWA
> > > for iPhone
> > > > > > > > > > >>
> > > ______________________________
> > > > > __________
> > > > > > > > > > >> From: Jason
> > > Gustafson <
> > > > > jason@confluent.io>
> > > > > > > > > > >> Sent: Friday,
> > > February 17, 2017
> > > > > 5:55:42 PM
> > > > > > > > > > >> To:
> > > dev@kafka.apache.org
> > > > > > > > > > >> Subject: Re:
> > > [DISCUSS] KIP-82 -
> > > > > Add Record
> > > > > > Headers
> > > > > > > > > > >>
> > > > > > > > > > >> >
> > > > > > > > > > >> > Would you be
> > > proposing in
> > > > > KIP-98 to convert
> > > > > > the
> > > > > > > other
> > > > > > > > message
> > > > > > > > > int’s
> > > > > > > > > > (key
> > > > > > > > > > >> > length,
> value
> > > length) also to
> > > > > varint to keep
> > > > > > it
> > > > > > > > uniform.
> > > > > > > > > > >> > Also I
> assume
> > > there will be a
> > > > > static or helper
> > > > > > > method
> > > > > > > > made to
> > > > > > > > > > write/read
> > > > > > > > > > >> > these in the
> > > client and
> > > > server.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Yes, that is
> > what
> > > we are
> > > > > proposing, so using
> > > > > > > varints
> > > > > > > > for headers
> > > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > >> consistent
> with
> > > the rest of the
> > > > > message. We have
> > > > > > > used
> > > > > > > > static
> > > > > > > > > helper
> > > > > > > > > > >> methods
> > > > > > > > > > >> in our
> prototype
> > > implementation.
> > > > > > > > > > >>
> > > > > > > > > > >> The cost of
> > > parsing, we want to
> > > > > parse/interpret
> > > > > > > the
> > > > > > > > headers
> > > > > > > > > lazily (this
> > > > > > > > > > >> is
> > > > > > > > > > >> > a key point
> > > brought up earlier
> > > > > in discussions)
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> I'm a bit
> > > skeptical of this. Has
> > > > > anyone done the
> > > > > > > > performance
> > > > > > > > > testing? I
> > > > > > > > > > >> can
> > > > > > > > > > >> probably
> > > implement it and test
> > > > it
> > > > > if no one else
> > > > > > > has. I
> > > > > > > > was also
> > > > > > > > > under
> > > > > > > > > > the
> > > > > > > > > > >> impression
> that
> > > there may be use
> > > > > cases down the
> > > > > > > road
> > > > > > > > where the
> > > > > > > > > broker
> > > > > > > > > > >> would
> > > > > > > > > > >> need to
> > interpret
> > > headers. That
> > > > > wouldn't be off
> > > > > > > the
> > > > > > > > table in the
> > > > > > > > > future
> > > > > > > > > > if
> > > > > > > > > > >> it's
> represented
> > > as bytes, but
> > > > it
> > > > > would be quite
> > > > > > > a bit
> > > > > > > > more
> > > > > > > > > awkward,
> > > > > > > > > > >> right?
> > > > > > > > > > >>
> > > > > > > > > > >> By the way,
> one
> > > question I have
> > > > > been wondering
> > > > > > > about. My
> > > > > > > > > understanding
> > > > > > > > > > is
> > > > > > > > > > >> that headers
> are
> > > primarily for
> > > > > use cases where a
> > > > > > > > third-party
> > > > > > > > > components
> > > > > > > > > > >> wants to
> enrich
> > > messages without
> > > > > needing to
> > > > > > > understand
> > > > > > > > or modify
> > > > > > > > > the
> > > > > > > > > > >> schema
> > > > > > > > > > >> of the message
> > > key and value.
> > > > For
> > > > > the
> > > > > > applications
> > > > > > > > which directly
> > > > > > > > > > produce
> > > > > > > > > > >> and consume
> the
> > > messages and
> > > > > control the
> > > > > > key/value
> > > > > > > > schema
> > > > > > > > > directly, it
> > > > > > > > > > >> seems we would
> > > rather have them
> > > > > implement
> > > > > > headers
> > > > > > > > directly in
> > > > > > > > > their own
> > > > > > > > > > >> schema.
> > Supposing
> > > for the sake
> > > > of
> > > > > argument that
> > > > > > > it was
> > > > > > > > possible,
> > > > > > > > > my
> > > > > > > > > > >> question is
> > > whether it be
> > > > > sufficient to expose
> > > > > > the
> > > > > > > > headers in the
> > > > > > > > > > >> interceptor
> API
> > > and not in the
> > > > > common API?
> > > > > > > > > > >>
> > > > > > > > > > >> -Jason
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, Feb
> 17,
> > > 2017 at 3:26 AM,
> > > > > Michael Pearce
> > > > > > <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > On the point
> > of
> > > varInts
> > > > > > > > > > >> >
> > > > > > > > > > >> > Would you be
> > > proposing in
> > > > > KIP-98 to convert
> > > > > > the
> > > > > > > other
> > > > > > > > message
> > > > > > > > > int’s
> > > > > > > > > > (key
> > > > > > > > > > >> > length,
> value
> > > length) also to
> > > > > varint to keep
> > > > > > it
> > > > > > > > uniform.
> > > > > > > > > > >> > Also I
> assume
> > > there will be a
> > > > > static or helper
> > > > > > > method
> > > > > > > > made to
> > > > > > > > > > write/read
> > > > > > > > > > >> > these in the
> > > client and
> > > > server.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Cheers
> > > > > > > > > > >> > Mike
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On
> 17/02/2017,
> > > 11:22, "Michael
> > > > > Pearce" <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > On the
> > > point re: headers
> > > > in
> > > > > the message
> > > > > > > protocol
> > > > > > > > being a
> > > > > > > > > byte
> > > > > > > > > > array
> > > > > > > > > > >> > and not a
> > count
> > > of elements
> > > > > followed by the
> > > > > > > elements.
> > > > > > > > Again
> > > > > > > > > this was
> > > > > > > > > > >> >
> > > discussed/argued previously.
> > > > > > > > > > >> >
> > > > > > > > > > >> > It was
> > > agreed on for a few
> > > > > reasons some of
> > > > > > > which
> > > > > > > > you have
> > > > > > > > > > obviously
> > > > > > > > > > >> > picked up
> on:
> > > > > > > > > > >> >
> > > > > > > > > > >> > Broker
> is
> > > able to pass it
> > > > > through opaquely
> > > > > > > > > > >> > The cost
> > of
> > > parsing, we
> > > > > want to
> > > > > > > parse/interpret
> > > > > > > > the headers
> > > > > > > > > lazily
> > > > > > > > > > >> > (this is a
> key
> > > point brought
> > > > up
> > > > > earlier in
> > > > > > > > discussions)
> > > > > > > > > > >> > Headers
> > can
> > > be copied from
> > > > > consumer record
> > > > > > > to
> > > > > > > > producer
> > > > > > > > > record (aka
> > > > > > > > > > >> > mirror
> makers
> > > etc) without
> > > > > parsing if no
> > > > > > > changes are
> > > > > > > > being made
> > > > > > > > > or
> > > > > > > > > > being
> > > > > > > > > > >> > looked at.
> > > > > > > > > > >> > Keeps
> the
> > > broker agnostic
> > > > > to the format
> > > > > > > > > > >> > You need
> > an
> > > int32 either
> > > > > for the byte size
> > > > > > > of the
> > > > > > > > headers,
> > > > > > > > > or for
> > > > > > > > > > >> the
> > > > > > > > > > >> > count of
> > > elements, so
> > > > overheads
> > > > > are the same,
> > > > > > > but
> > > > > > > > with going
> > > > > > > > > with an
> > > > > > > > > > >> opaque
> > > > > > > > > > >> > byte array
> has
> > > the above
> > > > > advantages.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Cheers
> > > > > > > > > > >> > Mike
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On
> > > 17/02/2017, 02:50,
> > > > > "Jason Gustafson" <
> > > > > > > > jason@confluent.io
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> Sorry,
> > > should have
> > > > > noted that the
> > > > > > > performance
> > > > > > > > testing
> > > > > > > > > was done
> > > > > > > > > > >> > using the
> > > > > > > > > > >> >
> > > producer performance
> > > > > tool shipped with
> > > > > > > Kafka.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> -Jason
> > > > > > > > > > >> >
> > > > > > > > > > >> > On
> > Thu,
> > > Feb 16, 2017
> > > > at
> > > > > 6:44 PM, Jason
> > > > > > > > Gustafson <
> > > > > > > > > > >> >
> > > jason@confluent.io> wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > >
> Hey
> > > Nacho,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> I've
> > > compared
> > > > > performance of our
> > > > > > > KIP-98
> > > > > > > > > implementation with
> > > > > > > > > > >> and
> > > > > > > > > > >> > without
> > > > > > > > > > >> > >
> > > varints. For
> > > > messages
> > > > > around 128
> > > > > > > bytes, we
> > > > > > > > see an
> > > > > > > > > increase
> > > > > > > > > > in
> > > > > > > > > > >> > throughput
> of
> > > > > > > > > > >> > >
> > about
> > > 30% using the
> > > > > default
> > > > > > > configuration
> > > > > > > > settings.
> > > > > > > > > At 256
> > > > > > > > > > >> > bytes, the
> > > > > > > > > > >> > >
> > > increase is around
> > > > > 16%. Obviously
> > > > > > the
> > > > > > > > performance
> > > > > > > > > converge
> > > > > > > > > > as
> > > > > > > > > > >> > messages get
> > > > > > > > > > >> > >
> > > larger, but it seems
> > > > > well worth the
> > > > > > > cost.
> > > > > > > > Note that
> > > > > > > > > we are
> > > > > > > > > > >> also
> > > > > > > > > > >> > seeing a
> > > > > > > > > > >> > >
> > > substantial
> > > > > performance increase
> > > > > > > against
> > > > > > > > trunk
> > > > > > > > > primarily
> > > > > > > > > > >> because
> > > > > > > > > > >> > of the
> > > > > > > > > > >> > >
> much
> > > more efficient
> > > > > packing that
> > > > > > > varints
> > > > > > > > provide us.
> > > > > > > > > > Anything
> > > > > > > > > > >> > adding to
> > > > > > > > > > >> > >
> > > message overhead,
> > > > > such as record
> > > > > > > headers,
> > > > > > > > would only
> > > > > > > > > > increase
> > > > > > > > > > >> > the relative
> > > > > > > > > > >> > >
> > > difference. (Of
> > > > > course take these
> > > > > > > numbers
> > > > > > > > with a
> > > > > > > > > grain of
> > > > > > > > > > salt
> > > > > > > > > > >> > since I have
> > > > > > > > > > >> > >
> only
> > > used the
> > > > default
> > > > > settings with
> > > > > > > both
> > > > > > > > the producer
> > > > > > > > > and
> > > > > > > > > > >> broker
> > > > > > > > > > >> > on my
> > > > > > > > > > >> > >
> > local
> > > machine. We
> > > > > intend to provide
> > > > > > > more
> > > > > > > > extensive
> > > > > > > > > > performance
> > > > > > > > > > >> > details as
> > > > > > > > > > >> > >
> part
> > > of the work for
> > > > > KIP-98.)
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> The
> > > implementation
> > > > we
> > > > > are using is
> > > > > > > from
> > > > > > > > protobuf (
> > > > > > > > > > >> > >
> > > > > https://developers.google.com/
> > > > > > > > > >
> > > protocol-buffers/docs/encoding
> > > > > > > > > > >> ),
> > > > > > > > > > >> > which is
> > > > > > > > > > >> > >
> also
> > > used in HBase.
> > > > > It is trivial to
> > > > > > > > implement and as
> > > > > > > > > far
> > > > > > > > > > as I
> > > > > > > > > > >> > know doesn't
> > > > > > > > > > >> > >
> > > suffer from the
> > > > > aliasing problem you
> > > > > > > are
> > > > > > > > describing. I
> > > > > > > > > > checked
> > > > > > > > > > >> > with Magnus
> > > > > > > > > > >> > >
> (the
> > > author of
> > > > > librdkafka) and he
> > > > > > > agreed
> > > > > > > > that the
> > > > > > > > > savings
> > > > > > > > > > >> seemed
> > > > > > > > > > >> > worth the
> > > > > > > > > > >> > >
> cost
> > > of
> > > > > implementation.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > -Jason
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On
> > > Thu, Feb 16, 2017
> > > > > at 4:32 PM,
> > > > > > > Ignacio
> > > > > > > > Solis <
> > > > > > > > > > >>
> isolis@igso.net
> > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >>
> > > -VarInts
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> I'm
> > > one of the
> > > > > people (if not the
> > > > > > > most)
> > > > > > > > opposed to
> > > > > > > > > VarInts.
> > > > > > > > > > >> > VarInts
> > > > > > > > > > >> > >>
> > have
> > > a place, but
> > > > > this is not it.
> > > > > > > (We
> > > > > > > > had a large
> > > > > > > > > > >> discussion
> > > > > > > > > > >> > about
> > > > > > > > > > >> > >>
> > them
> > > at the
> > > > > beginning of KIP-82
> > > > > > time)
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> If
> > > anybody has real
> > > > > life
> > > > > > performance
> > > > > > > > numbers of
> > > > > > > > > VarInts
> > > > > > > > > > >> > improving
> > > > > > > > > > >> > >>
> > > things or
> > > > > significantly reducing
> > > > > > > resources
> > > > > > > > I would
> > > > > > > > > like to
> > > > > > > > > > >> know
> > > > > > > > > > >> > what
> > > > > > > > > > >> > >>
> > that
> > > case may be.
> > > > > Yes, you can save
> > > > > > > some
> > > > > > > > bytes here
> > > > > > > > > and
> > > > > > > > > > >> there,
> > > > > > > > > > >> > but
> > > > > > > > > > >> > >>
> > this
> > > is probably
> > > > > insignificant to
> > > > > > the
> > > > > > > > overall system
> > > > > > > > > > behavior
> > > > > > > > > > >> > and
> > > > > > > > > > >> > >>
> > > storage
> > > > > requirements. -- I say
> > > > > > this
> > > > > > > with
> > > > > > > > respect to
> > > > > > > > > using
> > > > > > > > > > >> > VarInts in
> > > > > > > > > > >> > >>
> the
> > > protocol
> > > > itself,
> > > > > not as part of
> > > > > > > the
> > > > > > > > data.
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > VarInts require you
> > > > > to parse the
> > > > > > Int
> > > > > > > > before using it
> > > > > > > > > and
> > > > > > > > > > >> > depending on
> > > > > > > > > > >> > >>
> the
> > > encoding they
> > > > > can suffer from
> > > > > > > aliasing
> > > > > > > > (multiple
> > > > > > > > > > >> >
> > representations
> > > > > > > > > > >> > >>
> for
> > > the same
> > > > value).
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> Why
> > > add complexity?
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > Nacho
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> On
> > > Thu, Feb 16,
> > > > 2017
> > > > > at 10:29 AM,
> > > > > > > Colin
> > > > > > > > McCabe <
> > > > > > > > > > >> >
> > > cmccabe@apache.org>
> > > > > > > > > > >> > >>
> > > wrote:
> > > > > > > > > > >> > >> >
> > +1
> > > for varints
> > > > > here-- it would
> > > > > > save
> > > > > > > > quite a bit of
> > > > > > > > > space.
> > > > > > > > > > >> > They are
> > > > > > > > > > >> > >> >
> > > pretty quick to
> > > > > implement as
> > > > > > well.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >>
> > I
> > > think it makes
> > > > > sense for values
> > > > > > > to be
> > > > > > > > byte
> > > > > > > > > arrays.
> > > > > > > > > > Users
> > > > > > > > > > >> > might want
> > > > > > > > > > >> > >> >
> > to
> > > attach
> > > > > arbitrary payloads;
> > > > > > they
> > > > > > > > shouldn't be
> > > > > > > > > forced to
> > > > > > > > > > >> > serialize
> > > > > > > > > > >> > >> >
> > > everything to
> > > > Java
> > > > > strings.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > best,
> > > > > > > > > > >> > >> >
> > > Colin
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > On
> > > Thu, Feb 16,
> > > > > 2017, at 09:52,
> > > > > > > Jason
> > > > > > > > Gustafson
> > > > > > > > > wrote:
> > > > > > > > > > >> > >>
> >>
> > > Hey Michael,
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > Hmm, I guess the
> > > > > point of
> > > > > > > representing
> > > > > > > > it as
> > > > > > > > > bytes is to
> > > > > > > > > > >> > allow the
> > > > > > > > > > >> > >>
> > > broker
> > > > > > > > > > >> > >>
> >>
> > > to pass it
> > > > > through opaquely? Is
> > > > > > > the
> > > > > > > > cost of
> > > > > > > > > parsing
> > > > > > > > > > them a
> > > > > > > > > > >> > concern, or
> > > > > > > > > > >> > >>
> >>
> > > are
> > > > > > > > > > >> > >>
> >>
> > > we simply trying
> > > > > to ensure that
> > > > > > > the
> > > > > > > > broker stays
> > > > > > > > > > agnostic
> > > > > > > > > > >> to
> > > > > > > > > > >> > the
> > > > > > > > > > >> > >>
> > > format?
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > On varints, I
> > > > > think adding
> > > > > > > support for
> > > > > > > > them makes
> > > > > > > > > less
> > > > > > > > > > >> sense
> > > > > > > > > > >> > for an
> > > > > > > > > > >> > >>
> >>
> > > isolated use
> > > > > case, but as part
> > > > > > of
> > > > > > > a
> > > > > > > > more holistic
> > > > > > > > > change
> > > > > > > > > > >> > (such as
> what
> > > > > > > > > > >> > >>
> we
> > > > > > > > > > >> > >>
> >>
> > > have proposed in
> > > > > KIP-98), I
> > > > > > think
> > > > > > > they
> > > > > > > > are
> > > > > > > > > justifiable.
> > > > > > > > > > If
> > > > > > > > > > >> > we add them,
> > > > > > > > > > >> > >>
> >>
> > > then the need to
> > > > > use attributes
> > > > > > > becomes
> > > > > > > > quite a
> > > > > > > > > bit
> > > > > > > > > > >> weaker,
> > > > > > > > > > >> > right? The
> > > > > > > > > > >> > >>
> >>
> > > other thing I
> > > > > find slightly odd
> > > > > > > is the
> > > > > > > > fact that
> > > > > > > > > null
> > > > > > > > > > >> > headers has
> no
> > > > > > > > > > >> > >>
> >>
> > > actual
> > > > > > > > > > >> > >>
> >>
> > > semantic meaning
> > > > > for the message
> > > > > > > > (unlike null
> > > > > > > > > keys and
> > > > > > > > > > >> > values). It
> is
> > > > > > > > > > >> > >>
> >>
> > > just
> > > > > > > > > > >> > >>
> >>
> > a
> > > space
> > > > > optimization. It seems a
> > > > > > > bit
> > > > > > > > better to
> > > > > > > > > always
> > > > > > > > > > use
> > > > > > > > > > >> > size 0 to
> > > > > > > > > > >> > >>
> >>
> > > indicate having
> > > > > no headers.
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > Overall, the
> > > > main
> > > > > point is
> > > > > > > ensuring
> > > > > > > > that the
> > > > > > > > > message
> > > > > > > > > > >> schema
> > > > > > > > > > >> > remains
> > > > > > > > > > >> > >>
> >>
> > > consistent,
> > > > > either within the
> > > > > > > larger
> > > > > > > > protocol, or
> > > > > > > > > at a
> > > > > > > > > > >> > minimum
> within
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > > message itself.
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > -Jason
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > On Thu, Feb 16,
> > > > > 2017 at 6:39 AM,
> > > > > > > > Michael Pearce <
> > > > > > > > > > >> >
> > > Michael.Pearce@ig.com
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >>
> >>
> > > wrote:
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > >
> > > Hi Jason,
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On point 1) in
> > > > > the message
> > > > > > > protocol
> > > > > > > > the headers
> > > > > > > > > are
> > > > > > > > > > >> simply
> > > > > > > > > > >> > a byte
> > > > > > > > > > >> > >>
> > > array,
> > > > > > > > > > >> > >>
> >>
> > >
> > > as like the
> > > > key
> > > > > or value, this
> > > > > > > is to
> > > > > > > > clearly
> > > > > > > > > demarcate
> > > > > > > > > > >> the
> > > > > > > > > > >> > header in
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > core message.
> > > > > Then the header
> > > > > > > byte
> > > > > > > > array in the
> > > > > > > > > core
> > > > > > > > > > >> > message is
> an
> > > > > > > > > > >> > >>
> > > array of
> > > > > > > > > > >> > >>
> >>
> > >
> > > key, value
> > > > > pairs. This is what
> > > > > > > it is
> > > > > > > > denoting.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > Then this
> > > > would
> > > > > be I guess in
> > > > > > > the
> > > > > > > > given
> > > > > > > > > notation:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > Headers =>
> > > > > [KeyLength, Key,
> > > > > > > > ValueLength, Value]
> > > > > > > > > > >> > >>
> >>
> > > > KeyLength
> > > > > => int32
> > > > > > > > <-----------------NEW
> > > > > > > > > size of
> > > > > > > > > > the
> > > > > > > > > > >> > byte[] of
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > serialised key
> > > > > value
> > > > > > > > > > >> > >>
> >>
> > > > Key =>
> > > > bytes
> > > > > > > > <---------------------- NEW
> > > > > > > > > > serialised
> > > > > > > > > > >> > string
> (UTF8)
> > > > > > > > > > >> > >>
> >>
> > >
> > > bytes of the
> > > > > header key
> > > > > > > > > > >> > >>
> >>
> > >
> > > > ValueLength
> > > > > => int32
> > > > > > > > <-------------- NEW
> > > > > > > > > size of
> > > > > > > > > > the
> > > > > > > > > > >> > byte[] of
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > serialised
> > > > > header value
> > > > > > > > > > >> > >>
> >>
> > > > Value =>
> > > > > bytes
> > > > > > > > <-------------------- NEW
> > > > > > > > > > serialised
> > > > > > > > > > >> > form of the
> > > > > > > > > > >> > >>
> > > header
> > > > > > > > > > >> > >>
> >>
> > >
> > > value
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > The key length
> > > > > and value
> > > > > > length
> > > > > > > is
> > > > > > > > matching the
> > > > > > > > > way
> > > > > > > > > > the
> > > > > > > > > > >> > protocol is
> > > > > > > > > > >> > >>
> >>
> > >
> > > defined in the
> > > > > core message
> > > > > > > currently.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On point 2)
> > > > > > > > > > >> > >>
> >>
> > >
> > > Var sized
> > > > ints,
> > > > > this was
> > > > > > > discussed
> > > > > > > > much earlier
> > > > > > > > > on, in
> > > > > > > > > > >> > fact I had
> > > > > > > > > > >> > >>
> >>
> > >
> > > suggested it
> > > > > myself (with
> > > > > > Hadoop
> > > > > > > > references),
> > > > > > > > > the
> > > > > > > > > > >> > complexity
> of
> > > this
> > > > > > > > > > >> > >>
> >>
> > >
> > > compared to
> > > > > having a simpler
> > > > > > > protocol
> > > > > > > > was
> > > > > > > > > argued and
> > > > > > > > > > >> > agreed it
> > > > > > > > > > >> > >>
> > > wasn’t worth
> > > > > > > > > > >> > >>
> >>
> > >
> > > the complexity
> > > > > as all other
> > > > > > > clients
> > > > > > > > in other
> > > > > > > > > languages
> > > > > > > > > > >> > would need
> to
> > > > > > > > > > >> > >>
> > > ensure
> > > > > > > > > > >> > >>
> >>
> > >
> > > theyre using
> > > > > the right var
> > > > > > size
> > > > > > > > algorithm, as
> > > > > > > > > there
> > > > > > > > > > is a
> > > > > > > > > > >> > few.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On point 3)
> > > > > > > > > > >> > >>
> >>
> > >
> > > We did the
> > > > > attributes,
> > > > > > optional
> > > > > > > > approach as
> > > > > > > > > originally
> > > > > > > > > > >> > there was
> > > > > > > > > > >> > >>
> > > marked
> > > > > > > > > > >> > >>
> >>
> > >
> > > concern that
> > > > > headers would
> > > > > > > cause a
> > > > > > > > message size
> > > > > > > > > > overhead
> > > > > > > > > > >> > for others,
> > > > > > > > > > >> > >>
> who
> > > > > > > > > > >> > >>
> >>
> > >
> > > don’t want
> > > > > them. As such this
> > > > > > > is the
> > > > > > > > clean
> > > > > > > > > solution to
> > > > > > > > > > >> > achieve
> that.
> > > > > > > > > > >> > >>
> If
> > > > > > > > > > >> > >>
> >>
> > >
> > > that no longer
> > > > > holds, and we
> > > > > > > don’t
> > > > > > > > care that we
> > > > > > > > > add
> > > > > > > > > > >> 4bytes
> > > > > > > > > > >> > overhead,
> > > > > > > > > > >> > >>
> > then
> > > > > > > > > > >> > >>
> >>
> > >
> > > im happy to
> > > > > remove.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > I’m personally
> > > > > in favour of
> > > > > > > keeping
> > > > > > > > the message
> > > > > > > > > as
> > > > > > > > > > small
> > > > > > > > > > >> > as possible
> > > > > > > > > > >> > >>
> so
> > > > > > > > > > >> > >>
> >>
> > >
> > > people don’t
> > > > > get shocks in
> > > > > > perf
> > > > > > > and
> > > > > > > > throughputs
> > > > > > > > > dues
> > > > > > > > > > to
> > > > > > > > > > >> > message
> size,
> > > > > > > > > > >> > >>
> >>
> > >
> > > unless they
> > > > > actively use the
> > > > > > > feature,
> > > > > > > > as such I
> > > > > > > > > do
> > > > > > > > > > >> prefer
> > > > > > > > > > >> > the
> > > > > > > > > > >> > >>
> > > attribute bit
> > > > > > > > > > >> > >>
> >>
> > >
> > > wise feature
> > > > > flag approach
> > > > > > > myself.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On 16/02/2017,
> > > > > 05:40, "Jason
> > > > > > > > Gustafson" <
> > > > > > > > > > >> >
> > > jason@confluent.io> wrote:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > We have
> > > > > proposed a few
> > > > > > > > significant changes
> > > > > > > > > to the
> > > > > > > > > > >> > message
> format
> > > > > > > > > > >> > >>
> in
> > > > > > > > > > >> > >>
> >>
> > >
> > > KIP-98
> > > > > > > > > > >> > >>
> >>
> > > > which now
> > > > > seems likely to
> > > > > > > pass
> > > > > > > > (perhaps
> > > > > > > > > with some
> > > > > > > > > > >> > iterations
> on
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > implementation details).
> > > > > > It
> > > > > > > would
> > > > > > > > be good
> > > > > > > > > to try
> > > > > > > > > > and
> > > > > > > > > > >> > coordinate
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > changes
> > > > > > > > > > >> > >>
> >>
> > > > in both of
> > > > > the proposals
> > > > > > to
> > > > > > > make
> > > > > > > > sure they
> > > > > > > > > are
> > > > > > > > > > >> > consistent
> and
> > > > > > > > > > >> > >>
> >>
> > >
> > > compatible.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > I think
> > > > > using the
> > > > > > > attributes to
> > > > > > > > indicate
> > > > > > > > > null
> > > > > > > > > > >> headers
> > > > > > > > > > >> > is a
> > > > > > > > > > >> > >>
> > > reasonable
> > > > > > > > > > >> > >>
> >>
> > > > approach.
> > > > > We have proposed
> > > > > > > to do
> > > > > > > > the same
> > > > > > > > > thing
> > > > > > > > > > for
> > > > > > > > > > >> > the message
> > > > > > > > > > >> > >>
> key
> > > and
> > > > > > > > > > >> > >>
> >>
> > > > value.
> > > > That
> > > > > said, I
> > > > > > > sympathize
> > > > > > > > with Jay's
> > > > > > > > > > argument.
> > > > > > > > > > >> > Having
> > > > > > > > > > >> > >>
> > > multiple
> > > > > > > > > > >> > >>
> >>
> > >
> > > ways to
> > > > > > > > > > >> > >>
> >>
> > > > specify a
> > > > > null value
> > > > > > > increases
> > > > > > > > the overall
> > > > > > > > > > >> complexity
> > > > > > > > > > >> > of the
> > > > > > > > > > >> > >>
> > > protocol.
> > > > > > > > > > >> > >>
> >>
> > >
> > > You
> > > > > > > > > > >> > >>
> >>
> > > > can see
> > > > > this just from the
> > > > > > > fact
> > > > > > > > that you
> > > > > > > > > need the
> > > > > > > > > > >> > extra
> verbiage
> > > > > > > > > > >> > >>
> in
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > protocol
> > > > > specification in
> > > > > > > this
> > > > > > > > KIP and in
> > > > > > > > > KIP-98
> > > > > > > > > > to
> > > > > > > > > > >> > describe the
> > > > > > > > > > >> > >>
> >>
> > >
> > > dependence
> > > > > > > > > > >> > >>
> >>
> > > > between
> > > > the
> > > > > fields and the
> > > > > > > > attributes. It
> > > > > > > > > seems
> > > > > > > > > > >> like a
> > > > > > > > > > >> > slippery
> > > > > > > > > > >> > >>
> > slope
> > > > > > > > > > >> > >>
> >>
> > >
> > > if
> > > > > > > > > > >> > >>
> >>
> > > > you start
> > > > > allowing
> > > > > > different
> > > > > > > > request types
> > > > > > > > > to
> > > > > > > > > > >> > implement
> the
> > > > > > > > > > >> > >>
> > > protocol
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > specification differently.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > You can
> > > > > also argue that
> > > > > > the
> > > > > > > > messages
> > > > > > > > > already are
> > > > > > > > > > and
> > > > > > > > > > >> > are likely
> > > > > > > > > > >> > >>
> to
> > > > > > > > > > >> > >>
> >>
> > >
> > > remain a
> > > > > > > > > > >> > >>
> >>
> > > > special
> > > > > case. For example,
> > > > > > > there
> > > > > > > > is
> > > > > > > > > currently no
> > > > > > > > > > >> > generality
> in
> > > > > > > > > > >> > >>
> how
> > > > > > > > > > >> > >>
> >>
> > > > compressed
> > > > > message sets
> > > > > > are
> > > > > > > > represented
> > > > > > > > > that would
> > > > > > > > > > >> be
> > > > > > > > > > >> > applicable
> > > > > > > > > > >> > >>
> for
> > > > > > > > > > >> > >>
> >>
> > >
> > > other
> > > > > > > > > > >> > >>
> >>
> > > > request
> > > > > types. Some might
> > > > > > > see this
> > > > > > > > > divergence as
> > > > > > > > > > an
> > > > > > > > > > >> > unfortunate
> > > > > > > > > > >> > >>
> >>
> > >
> > > protocol
> > > > > > > > > > >> > >>
> >>
> > > > deficiency
> > > > > which should be
> > > > > > > fixed;
> > > > > > > > others
> > > > > > > > > might see
> > > > > > > > > > >> it
> > > > > > > > > > >> > as sort of
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > inevitability of needing
> > > > > > to
> > > > > > > > optimize where
> > > > > > > > > it
> > > > > > > > > > counts
> > > > > > > > > > >> > most. I'm
> > > > > > > > > > >> > >>
> > > probably
> > > > > > > > > > >> > >>
> >>
> > > > somewhere
> > > > > in between, but
> > > > > > I
> > > > > > > think
> > > > > > > > we
> > > > > > > > > probably all
> > > > > > > > > > >> > share the
> > > > > > > > > > >> > >>
> > > intuition
> > > > > > > > > > >> > >>
> >>
> > >
> > > that
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > > protocol should be
> > > > > > kept
> > > > > > > as
> > > > > > > > consistent as
> > > > > > > > > > >> possible.
> > > > > > > > > > >> > With that
> > > > > > > > > > >> > >>
> in
> > > > > > > > > > >> > >>
> >>
> > >
> > > mind,
> > > > > > > > > > >> > >>
> >>
> > > > here are a
> > > > > few comments:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > 1. One
> > > > > thing I found a
> > > > > > > little odd
> > > > > > > > when
> > > > > > > > > reading the
> > > > > > > > > > >> > current
> > > > > > > > > > >> > >>
> > > proposal is
> > > > > > > > > > >> > >>
> >>
> > >
> > > that
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > headers
> > > > > are both
> > > > > > > represented
> > > > > > > > as an
> > > > > > > > > array of
> > > > > > > > > > >> bytes
> > > > > > > > > > >> > and as an
> > > > > > > > > > >> > >>
> > array
> > > > > > > > > > >> > >>
> >>
> > >
> > > of
> > > > > > > > > > >> > >>
> >>
> > > > key/value
> > > > > pairs. I'd
> > > > > > > probably
> > > > > > > > suggest
> > > > > > > > > something
> > > > > > > > > > like
> > > > > > > > > > >> > this:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Headers =>
> > > > > [HeaderKey
> > > > > > > HeaderValue]
> > > > > > > > > > >> > >>
> >>
> > > > HeaderKey
> > > > > => String
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > HeaderValue => Bytes
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > An array
> > > > in
> > > > > the Kafka
> > > > > > > protocol is
> > > > > > > > > represented as a
> > > > > > > > > > >> > 4-byte
> integer
> > > > > > > > > > >> > >>
> >>
> > > > indicating
> > > > > the number of
> > > > > > > elements
> > > > > > > > in the
> > > > > > > > > array
> > > > > > > > > > >> > followed by
> > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > serialization of the
> > > > > > > elements.
> > > > > > > > Unless I'm
> > > > > > > > > > >> >
> > > misunderstanding, what
> > > > > > > > > > >> > >>
> you
> > > > > > > > > > >> > >>
> >>
> > >
> > > have
> > > > > > > > > > >> > >>
> >>
> > > > instead is
> > > > > the total size
> > > > > > > of the
> > > > > > > > headers in
> > > > > > > > > bytes
> > > > > > > > > > >> > followed by
> > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > elements.
> > > > > > > > > > >> > >>
> >>
> > > > I'm not
> > > > > sure I see any
> > > > > > > reason for
> > > > > > > > this
> > > > > > > > > > >> inconsistency.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > 2. In
> > > > > KIP-98, we've
> > > > > > > introduced
> > > > > > > > > variable-length
> > > > > > > > > > >> integer
> > > > > > > > > > >> > fields.
> > > > > > > > > > >> > >>
> >>
> > >
> > > Effectively,
> > > > > > > > > > >> > >>
> >>
> > > > we've
> > > > > enriched (or
> > > > > > > "complicated"
> > > > > > > > as Jay
> > > > > > > > > might say
> > > > > > > > > > ;)
> > > > > > > > > > >> > the protocol
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > specification to include
> > > > > > the
> > > > > > > > following
> > > > > > > > > types:
> > > > > > > > > > >> VarInt,
> > > > > > > > > > >> > VarLong,
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > UnsignedVarInt and
> > > > > > > > UnsignedVarLong.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Along with
> > > > > these
> > > > > > > primitives, we
> > > > > > > > could
> > > > > > > > > introduce
> > > > > > > > > > the
> > > > > > > > > > >> > following
> > > > > > > > > > >> > >>
> > > types:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > VarSizeArray =>
> > > > > > > NumberOfItems
> > > > > > > > Item1 Item2
> > > > > > > > > .. ItemN
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > NumberOfItems =>
> > > > > > > UnsignedVarInt
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > VarSizeNullableArray =>
> > > > > > > > NumberOfItemsOrNull
> > > > > > > > > Item1
> > > > > > > > > > >> > Item2 ..
> ItemN
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > NumberOfItemsOrNull =>
> > > > > > > VarInt
> > > > > > > > (-1 means
> > > > > > > > > null)
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > And
> > > > > similarly for the
> > > > > > > `String`
> > > > > > > > and `Bytes`
> > > > > > > > > types.
> > > > > > > > > > >> > These types
> > > > > > > > > > >> > >>
> can
> > > save
> > > > > > > > > > >> > >>
> >>
> > >
> > > a
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > considerable amount of
> > > > > > > space in
> > > > > > > > this
> > > > > > > > > proposal
> > > > > > > > > > >> because
> > > > > > > > > > >> > they can
> > > > > > > > > > >> > >>
> be
> > > used
> > > > > > > > > > >> > >>
> >>
> > >
> > > for
> > > > > > > > > > >> > >>
> >>
> > > > both the
> > > > > number of headers
> > > > > > > > included in the
> > > > > > > > > message
> > > > > > > > > > >> and
> > > > > > > > > > >> > the
> > > > > > > > > > >> > >>
> > > lengths of
> > > > > > > > > > >> > >>
> >>
> > >
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > header
> > > > keys
> > > > > and values. We
> > > > > > > could
> > > > > > > > do this
> > > > > > > > > instead:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Headers =>
> > > > > > > VarSizeArray[HeaderKey
> > > > > > > > > HeaderValue]
> > > > > > > > > > >> > >>
> >>
> > >
> > > > HeaderKey
> > > > > =>
> > > > > > VarSizeString
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > HeaderValue =>
> > > > > > > VarSizeBytes
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Combining
> > > > > the savings from
> > > > > > > the
> > > > > > > > use of
> > > > > > > > > variable
> > > > > > > > > > >> length
> > > > > > > > > > >> > fields, the
> > > > > > > > > > >> > >>
> >>
> > >
> > > benefit
> > > > > > > > > > >> > >>
> >>
> > > > of using
> > > > > the attributes to
> > > > > > > > represent null
> > > > > > > > > seems
> > > > > > > > > > >> pretty
> > > > > > > > > > >> > small.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > 3.
> > > > > Whichever way we go
> > > > > > > (whether
> > > > > > > > we use the
> > > > > > > > > > >> attributes
> > > > > > > > > > >> > or not), we
> > > > > > > > > > >> > >>
> >>
> > >
> > > should at
> > > > > > > > > > >> > >>
> >>
> > > > least be
> > > > > consistent
> > > > > > between
> > > > > > > this
> > > > > > > > KIP and
> > > > > > > > > KIP-98.
> > > > > > > > > > It
> > > > > > > > > > >> > would be
> very
> > > > > > > > > > >> > >>
> >>
> > >
> > > strange
> > > > > > > > > > >> > >>
> >>
> > > > to have
> > > > two
> > > > > ways to
> > > > > > > represent
> > > > > > > > null values
> > > > > > > > > in the
> > > > > > > > > > >> same
> > > > > > > > > > >> > schema.
> > > > > > > > > > >> > >>
> > > Either
> > > > > > > > > > >> > >>
> >>
> > >
> > > way is
> > > > > > > > > > >> > >>
> >>
> > > > OK with
> > > > me.
> > > > > I think some
> > > > > > > > message-level
> > > > > > > > > > optimizations
> > > > > > > > > > >> > are
> > > > > > > > > > >> > >>
> > > justifiable,
> > > > > > > > > > >> > >>
> >>
> > >
> > > but
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > savings
> > > > > here seem
> > > > > > > minimal (a
> > > > > > > > few bytes
> > > > > > > > > per
> > > > > > > > > > >> > message), so
> > > > > > > > > > >> > >>
> > > maybe it's
> > > > > > > > > > >> > >>
> >>
> > >
> > > not
> > > > > > > > > > >> > >>
> >>
> > > > worth the
> > > > > cost of letting
> > > > > > > the
> > > > > > > > message
> > > > > > > > > diverge even
> > > > > > > > > > >> > further from
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > rest of
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > > protocol.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > -Jason
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > On Wed,
> > > > Feb
> > > > > 15, 2017 at
> > > > > > > 8:52 AM,
> > > > > > > > radai <
> > > > > > > > > > >> > >>
> > > > > radai.rosenblatt@gmail.com>
> > > > > > > > > > >> > >>
> >>
> > >
> > > wrote:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > > I've
> > > > > trimmed the inline
> > > > > > > > contents as this
> > > > > > > > > mail is
> > > > > > > > > > >> > getting too
> > > > > > > > > > >> > >>
> big
> > > for
> > > > > > > > > > >> > >>
> >>
> > >
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > > apache
> > > > > mailing list
> > > > > > > software to
> > > > > > > > deliver
> > > > > > > > > :-(
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > 1. the
> > > > > important thing
> > > > > > for
> > > > > > > > > interoperability is
> > > > > > > > > > for
> > > > > > > > > > >> > different
> > > > > > > > > > >> > >>
> >>
> > >
> > > "interested
> > > > > > > > > > >> > >>
> >>
> > > > > parties"
> > > > > (plugins, infra
> > > > > > > > layers/wrappers,
> > > > > > > > > > >> user-code)
> > > > > > > > > > >> > to be
> > > > > > > > > > >> > >>
> > able
> > > to
> > > > > > > > > > >> > >>
> >>
> > >
> > > stick
> > > > > > > > > > >> > >>
> >>
> > > > > pieces
> > > > of
> > > > > metadata onto
> > > > > > > msgs
> > > > > > > > without
> > > > > > > > > getting in
> > > > > > > > > > >> each
> > > > > > > > > > >> > other's
> > > > > > > > > > >> > >>
> > way.
> > > a
> > > > > > > > > > >> > >>
> >>
> > >
> > > common
> > > > > > > > > > >> > >>
> >>
> > > > > key
> > > > > scheme (Strings, as
> > > > > > > of the
> > > > > > > > time of
> > > > > > > > > this
> > > > > > > > > > >> > writing?) is
> > all
> > > > > > > > > > >> > >>
> > thats
> > > > > > > > > > >> > >>
> >>
> > >
> > > required
> > > > > > > > > > >> > >>
> >>
> > > > > for
> > > > that.
> > > > > it is assumed
> > > > > > > that
> > > > > > > > the other end
> > > > > > > > > > >> > interested
> in
> > > any
> > > > > > > > > > >> > >>
> > such
> > > > > > > > > > >> > >>
> >>
> > >
> > > piece of
> > > > > > > > > > >> > >>
> >>
> > > > > metadata
> > > > > knows the
> > > > > > > encoding,
> > > > > > > > and byte[]
> > > > > > > > > provides
> > > > > > > > > > >> for
> > > > > > > > > > >> > the most
> > > > > > > > > > >> > >>
> >>
> > >
> > > flexibility.
> > > > > > > > > > >> > >>
> >>
> > > > > i
> > > > believe
> > > > > this is the
> > > > > > same
> > > > > > > > logic behind
> > > > > > > > > core
> > > > > > > > > > kafka
> > > > > > > > > > >> > being
> > > > > > > > > > >> > >>
> >>
> > >
> > > byte[]/byte[]
> > > > -
> > > > > > > > > > >> > >>
> >>
> > > > > Strings
> > > > > are more
> > > > > > "usable"
> > > > > > > but
> > > > > > > > bytes are
> > > > > > > > > flexible
> > > > > > > > > > >> and
> > > > > > > > > > >> > so were
> > > > > > > > > > >> > >>
> > > chosen.
> > > > > > > > > > >> > >>
> >>
> > > > > Also -
> > > > > core kafka doesnt
> > > > > > > even
> > > > > > > > do that
> > > > > > > > > good of a
> > > > > > > > > > >> job
> > > > > > > > > > >> > on
> > > > > > > > > > >> > >>
> > > usability of
> > > > > > > > > > >> > >>
> >>
> > >
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > > payload
> > > > > (example - i
> > > > > > have
> > > > > > > to
> > > > > > > > specify the
> > > > > > > > > nop
> > > > > > > > > > >> byte[]
> > > > > > > > > > >> > "decoders"
> > > > > > > > > > >> > >>
> >>
> > >
> > > explicitly
> > > > > > > > > > >> > >>
> >>
> > > > > in
> > > > conf),
> > > > > and again
> > > > > > > sacrificies
> > > > > > > > usability
> > > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > >> > sake of
> > > > > > > > > > >> > >>
> >>
> > >
> > > performance
> > > > (no
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > convenient single-record
> > > > > > > > processing as
> > > > > > > > > poll is a
> > > > > > > > > > >> > batch, lots
> of
> > > > > > > > > > >> > >>
> >>
> > >
> > > obscure
> > > > > > > > > > >> > >>
> >>
> > > > > little
> > > > > config details
> > > > > > > exposing
> > > > > > > > internals
> > > > > > > > > of the
> > > > > > > > > > >> > batching
> > > > > > > > > > >> > >>
> > > mechanism,
> > > > > > > > > > >> > >>
> >>
> > >
> > > etc)
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > this is
> > > > > also why i
> > > > > > really
> > > > > > > > dislike the
> > > > > > > > > idea of a
> > > > > > > > > > >> > "type
> system"
> > > > > > > > > > >> > >>
> for
> > > > > > > > > > >> > >>
> >>
> > >
> > > header
> > > > > > > > > > >> > >>
> >>
> > > > > values,
> > > > > it further
> > > > > > > degrades the
> > > > > > > > > usability, adds
> > > > > > > > > > >> > complexity
> and
> > > > > > > > > > >> > >>
> > will
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > eventually get in
> > > > > > > people's way,
> > > > > > > > also, it
> > > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > >> > the 2nd/3rd
> > > > > > > > > > >> > >>
> >>
> > >
> > > home-group
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > serialization mechanism
> > > > > > > in core
> > > > > > > > kafka
> > > > > > > > > (counting
> > > > > > > > > > 2
> > > > > > > > > > >> > iterations
> > > > > > > > > > >> > >>
> of
> > > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > "type
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > definition DSL")
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > 2. this
> > > > > is an
> > > > > > > implementation
> > > > > > > > detail, and
> > > > > > > > > not
> > > > > > > > > > even
> > > > > > > > > > >> a
> > > > > > > > > > >> > very "user
> > > > > > > > > > >> > >>
> >>
> > >
> > > facing" one?
> > > > > > > > > > >> > >>
> >>
> > > > > to the
> > > > > best of my
> > > > > > > understanding
> > > > > > > > the vote
> > > > > > > > > process
> > > > > > > > > > >> is
> > > > > > > > > > >> > on proposed
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > API/behaviour. also -
> > > > > > > since
> > > > > > > > we're willing
> > > > > > > > > to go
> > > > > > > > > > >> with
> > > > > > > > > > >> > strings
> > > > > > > > > > >> > >>
> > just
> > > > > > > > > > >> > >>
> >>
> > >
> > > serialize
> > > > > > > > > > >> > >>
> >>
> > > > > a
> > > > 0-sized
> > > > > header blob
> > > > > > and
> > > > > > > IIUC
> > > > > > > > you dont
> > > > > > > > > need any
> > > > > > > > > > >> > optionals
> > > > > > > > > > >> > >>
> > > anymore.
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > 3. yes,
> > > > > we can :-)
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > On Tue,
> > > > > Feb 14, 2017 at
> > > > > > > 11:56
> > > > > > > > PM, Michael
> > > > > > > > > > Pearce <
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > Michael.Pearce@ig.com>
> > > > > > > > > > >> > >>
> >>
> > > > > wrote:
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > Hi
> > > > Jay,
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > 1)
> > > > > There was some
> > > > > > > initial
> > > > > > > > debate on the
> > > > > > > > > value
> > > > > > > > > > >> > part, as
> youll
> > > > > > > > > > >> > >>
> > note
> > > > > > > > > > >> > >>
> >>
> > >
> > > String,
> > > > > > > > > > >> > >>
> >>
> > > > > > String
> > > > > headers were
> > > > > > > > discounted early
> > > > > > > > > on. The
> > > > > > > > > > >> > reason for
> > this
> > > > > > > > > > >> > >>
> is
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > flexibility
> > > > > > > > > > >> > >>
> >>
> > > > > > and
> > > > > keeping in line
> > > > > > > with the
> > > > > > > > > flexibility of
> > > > > > > > > > key,
> > > > > > > > > > >> > value of the
> > > > > > > > > > >> > >>
> >>
> > >
> > > message
> > > > > > > > > > >> > >>
> >>
> > > > > > object
> > > > > itself. I don’t
> > > > > > > think
> > > > > > > > it takes
> > > > > > > > > away
> > > > > > > > > > from
> > > > > > > > > > >> an
> > > > > > > > > > >> > ecosystem
> > > > > > > > > > >> > >>
> as
> > > > > > > > > > >> > >>
> >>
> > >
> > > each
> > > > > > > > > > >> > >>
> >>
> > > > > plugin
> > > > > > > > > > >> > >>
> >>
> > > > > > will
> > > > > care for their
> > > > > > own
> > > > > > > key,
> > > > > > > > this way
> > > > > > > > > ints,
> > > > > > > > > > >> > booleans ,
> > > exotic
> > > > > > > > > > >> > >>
> >>
> > >
> > > custom
> > > > > > > > > > >> > >>
> >>
> > > > > binary
> > > > > > > > > > >> > >>
> >>
> > > > > > can
> > > > all
> > > > > be catered
> > > > > > for=.
> > > > > > > > > > >> > >>
> >>
> > > > > > a. If
> > > > > you really
> > > > > > wanted
> > > > > > > to
> > > > > > > > push for a
> > > > > > > > > typed
> > > > > > > > > > >> value
> > > > > > > > > > >> > interface,
> > > > > > > > > > >> > >> I
> > > > > > > > > > >> > >>
> >>
> > >
> > > wouldn’t
> > > > > > > > > > >> > >>
> >>
> > > > > > want
> > > > > just String
> > > > > > values
> > > > > > > > supported, but
> > > > > > > > > the the
> > > > > > > > > > >> > primatives
> > > > > > > > > > >> > >>
> > plus
> > > > > > > > > > >> > >>
> >>
> > >
> > > string and
> > > > > > > > > > >> > >>
> >>
> > > > > > also
> > > > > still keeping the
> > > > > > > > ability to have a
> > > > > > > > > > binary
> > > > > > > > > > >> > for custom
> > > > > > > > > > >> > >>
> >>
> > >
> > > binaries that
> > > > > > > > > > >> > >>
> >>
> > > > > > some
> > > > > organisations may
> > > > > > > have.
> > > > > > > > > > >> > >>
> >>
> > > > > > i. I
> > > > > have written this
> > > > > > > slight
> > > > > > > > > alternative
> > > > > > > > > > here,
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > https://cwiki.apache.org/
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > >> >
> > > 82+-+Add+Record+Headers+-+
> > > > Typed
> > > > > > > > > > >> > >>
> >>
> > > > > > ii.
> > > > > Essentially the
> > > > > > > value
> > > > > > > > bytes, has a
> > > > > > > > > leading
> > > > > > > > > > >> > byte
> overhead.
> > > > > > > > > > >> > >>
> >>
> > > > > > 1.
> > > > > This tells you
> > > > > > what
> > > > > > > type
> > > > > > > > the value
> > > > > > > > > is,
> > > > > > > > > > >> before
> > > > > > > > > > >> > reading
> > > > > > > > > > >> > >>
> the
> > > rest
> > > > > > > > > > >> > >>
> >>
> > >
> > > of the
> > > > > > > > > > >> > >>
> >>
> > > > > > bytes,
> > > > > allowing
> > > > > > > > >
> > > serialisation/deserialization
> > > > > > > > > > to
> > > > > > > > > > >> > and from the
> > > > > > > > > > >> > >>
> >>
> > >
> > > primitives,
> > > > > > > > > > >> > >>
> >>
> > > > > > string
> > > > > and byte[].
> > > > > > This
> > > > > > > is
> > > > > > > > akin to some
> > > > > > > > > other
> > > > > > > > > > >> > messaging
> > > > > > > > > > >> > >>
> > > systems.
> > > > > > > > > > >> > >>
> >>
> > > > > > 2) We
> > > > > are making it
> > > > > > > optional,
> > > > > > > > so that
> > > > > > > > > for
> > > > > > > > > > those
> > > > > > > > > > >> > not wanting
> > > > > > > > > > >> > >>
> >>
> > >
> > > headers have
> > > > > > > > > > >> > >>
> >>
> > > > > 0
> > > > > > > > > > >> > >>
> >>
> > > > > > bytes
> > > > > overhead (think
> > > > > > > of it
> > > > > > > > as a feature
> > > > > > > > > > flag),
> > > > > > > > > > >> I
> > > > > > > > > > >> > don’t
> > > > > > > > > > >> > >>
> > > think this
> > > > > > > > > > >> > >>
> >>
> > >
> > > is
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > complex, especially if
> > > > > > > > comparing to
> > > > > > > > > changes
> > > > > > > > > > >> > proposed in
> > > > > > > > > > >> > >>
> > > other kips
> > > > > > > > > > >> > >>
> >>
> > >
> > > like
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > kip-98.
> > > > > > > > > > >> > >>
> >>
> > > > > > a. If
> > > > > you really
> > > > > > really
> > > > > > > don’t
> > > > > > > > like
> > > > > > > > > this, we
> > > > > > > > > > can
> > > > > > > > > > >> > drop it, but
> > > > > > > > > > >> > >>
> it
> > > > > > > > > > >> > >>
> >>
> > >
> > > would
> > > > > > > > > > >> > >>
> >>
> > > > > mean
> > > > > > > > > > >> > >>
> >>
> > > > > > buying
> > > > > into 4 bytes
> > > > > > > extra
> > > > > > > > overhead for
> > > > > > > > > users
> > > > > > > > > > who
> > > > > > > > > > >> > do not want
> > > > > > > > > > >> > >>
> to
> > > use
> > > > > > > > > > >> > >>
> >>
> > > > > headers.
> > > > > > > > > > >> > >>
> >>
> > > > > > 3) In
> > > > > the summary yes,
> > > > > > > it is
> > > > > > > > at a higher
> > > > > > > > > > level,
> > > > > > > > > > >> > but I think
> > > > > > > > > > >> > >>
> > this
> > > > > > > > > > >> > >>
> >>
> > >
> > > is well
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > documented in the
> > > > > > > proposed
> > > > > > > > changes
> > > > > > > > > section.
> > > > > > > > > > >> > >>
> >>
> > > > > > a.
> > > > > Added getHeaders
> > > > > > > method to
> > > > > > > > > > Producer/Consumer
> > > > > > > > > > >> > record (that
> > > > > > > > > > >> > >>
> is
> > > it)
> > > > > > > > > > >> > >>
> >>
> > > > > > b.
> > > > > We’ve also detailed
> > > > > > > the
> > > > > > > > new Headers
> > > > > > > > > class
> > > > > > > > > > >> that
> > > > > > > > > > >> > this method
> > > > > > > > > > >> > >>
> >>
> > >
> > > returns
> > > > > > > > > > >> > >>
> >>
> > > > > that
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > encapsulates the
> > > > > > headers
> > > > > > > > protocol and
> > > > > > > > > logic.
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > Best,
> > > > > > > > > > >> > >>
> >>
> > > > > > Mike
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > ==Original questions
> > > > > > > from the
> > > > > > > > vote
> > > > > > > > > thread from
> > > > > > > > > > >> > Jay.==
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > Couple
> > > > > of things I
> > > > > > > think we
> > > > > > > > still need
> > > > > > > > > to work
> > > > > > > > > > >> out:
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > 1.
> > > > I
> > > > > think we agree
> > > > > > > about
> > > > > > > > the key,
> > > > > > > > > but I
> > > > > > > > > > >> think
> > > > > > > > > > >> > we haven't
> > > > > > > > > > >> > >>
> >>
> > >
> > > talked about
> > > > > > > > > > >> > >>
> >>
> > > > > > the
> > > > > value yet. I
> > > > > > > think if
> > > > > > > > our goal
> > > > > > > > > is an
> > > > > > > > > > open
> > > > > > > > > > >> > ecosystem
> > > > > > > > > > >> > >>
> of
> > > these
> > > > > > > > > > >> > >>
> >>
> > > > > header
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > spread across many
> > > > > > > plugins
> > > > > > > > from many
> > > > > > > > > > systems
> > > > > > > > > > >> we
> > > > > > > > > > >> > should
> > > > > > > > > > >> > >>
> > > consider
> > > > > > > > > > >> > >>
> >>
> > >
> > > making
> > > > > > > > > > >> > >>
> >>
> > > > > > this
> > > > > > > > > > >> > >>
> >>
> > > > > > a
> > > > > string as well so
> > > > > > > it can
> > > > > > > > be
> > > > > > > > > printed, set
> > > > > > > > > > >> via
> > > > > > > > > > >> > a UI, set
> > > > > > > > > > >> > >>
> in
> > > > > > > > > > >> > >>
> >>
> > >
> > > config,
> > > > > > > > > > >> > >>
> >>
> > > > > etc.
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > Basically
> > > > > > encouraging
> > > > > > > > pluggable
> > > > > > > > > > serialization
> > > > > > > > > > >> > formats
> > > > > > > > > > >> > >>
> > here
> > > will
> > > > > > > > > > >> > >>
> >>
> > >
> > > lead
> > > > > > > > > > >> > >>
> >>
> > > > > to
> > > > > > > > > > >> > >>
> >>
> > > > > > a
> > > > > > > > > > >> > >>
> >>
> > > > > > bit
> > > > > of a tower of
> > > > > > > babel.
> > > > > > > > > > >> > >>
> >>
> > > > > > 2.
> > > > > This proposal
> > > > > > > still
> > > > > > > > includes a
> > > > > > > > > pretty
> > > > > > > > > > big
> > > > > > > > > > >> > change to
> our
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > serialization
> > > > > > > > > > >> > >>
> >>
> > > > > > and
> > > > > protocol
> > > > > > > definition
> > > > > > > > layer.
> > > > > > > > > Essentially
> > > > > > > > > > >> it is
> > > > > > > > > > >> > >>
> > > introducing an
> > > > > > > > > > >> > >>
> >>
> > > > > optional
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > type, where the
> > > > > > > format is
> > > > > > > > data
> > > > > > > > > dependent. I
> > > > > > > > > > >> > think this
> is
> > > > > > > > > > >> > >>
> >>
> > >
> > > actually a
> > > > > > > > > > >> > >>
> >>
> > > > > big
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > change though it
> > > > > > > doesn't
> > > > > > > > seem like
> > > > > > > > > it. It
> > > > > > > > > > >> means
> > > > > > > > > > >> > you can no
> > > > > > > > > > >> > >>
> >>
> > >
> > > longer
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > specify
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > this
> > > > > type with our
> > > > > > > type
> > > > > > > > definition
> > > > > > > > > DSL, and
> > > > > > > > > > >> > likewise it
> > > > > > > > > > >> > >>
> > > requires
> > > > > > > > > > >> > >>
> >>
> > > > > custom
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > handling in client
> > > > > > > libs.
> > > > > > > > This isn't
> > > > > > > > > a huge
> > > > > > > > > > >> > thing, since
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > Record
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > definition is
> > > > > > custom
> > > > > > > > anyway, but I
> > > > > > > > > think
> > > > > > > > > > this
> > > > > > > > > > >> > kind of
> > > > > > > > > > >> > >>
> > > protocol
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > inconsistency is
> > > > > > very
> > > > > > > > non-desirable
> > > > > > > > > and
> > > > > > > > > > ties
> > > > > > > > > > >> > you to
> > > > > > > > > > >> > >>
> > > hand-coding
> > > > > > > > > > >> > >>
> >>
> > > > > things.
> > > > > > > > > > >> > >>
> >>
> > > > > > I
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > think the type
> > > > > > should
> > > > > > > > instead by [Key
> > > > > > > > > > Value]
> > > > > > > > > > >> in
> > > > > > > > > > >> > our BNF,
> > > > > > > > > > >> > >>
> > where
> > > > > > > > > > >> > >>
> >>
> > >
> > > key and
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > value are both
> > > > > > short
> > > > > > > > strings as used
> > > > > > > > > > >> elsewhere.
> > > > > > > > > > >> > This
> > > > > > > > > > >> > >>
> > > brings it
> > > > > > > > > > >> > >>
> >>
> > >
> > > in line
> > > > > > > > > > >> > >>
> >>
> > > > > > with
> > > > > > > > > > >> > >>
> >>
> > > > > > the
> > > > > rest of the
> > > > > > > protocol.
> > > > > > > > > > >> > >>
> >>
> > > > > > 3.
> > > > > Could we get
> > > > > > more
> > > > > > > > specific about
> > > > > > > > > the
> > > > > > > > > > exact
> > > > > > > > > > >> > Java API
> > > > > > > > > > >> > >>
> > > change to
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > ProducerRecord,
> > > > > > > > ConsumerRecord,
> > > > > > > > > Record,
> > > > > > > > > > etc?
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > -Jay
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > The
> > > > information
> > > > > contained in
> > > > > > > this
> > > > > > > > email is
> > > > > > > > > strictly
> > > > > > > > > > >> > confidential
> > and
> > > > > > > > > > >> > >>
> for
> > > > > > > > > > >> > >>
> >>
> > >
> > > the use of the
> > > > > addressee only,
> > > > > > > unless
> > > > > > > > otherwise
> > > > > > > > > > >> indicated.
> > > > > > > > > > >> > If you
> > > > > > > > > > >> > >>
> are
> > > not
> > > > > > > > > > >> > >>
> >>
> > >
> > > the intended
> > > > > recipient, please
> > > > > > > do not
> > > > > > > > read,
> > > > > > > > > copy, use
> > > > > > > > > > or
> > > > > > > > > > >> > disclose to
> > > > > > > > > > >> > >>
> > > others
> > > > > > > > > > >> > >>
> >>
> > >
> > > this message
> > > > or
> > > > > any
> > > > > > attachment.
> > > > > > > > Please also
> > > > > > > > > notify the
> > > > > > > > > > >> > sender by
> > > > > > > > > > >> > >>
> > > replying
> > > > > > > > > > >> > >>
> >>
> > >
> > > to this email
> > > > > or by telephone
> > > > > > > > (+44(020 7896
> > > > > > > > > 0011) and
> > > > > > > > > > >> then
> > > > > > > > > > >> > delete
> > > > > > > > > > >> > >>
> the
> > > email
> > > > > > > > > > >> > >>
> >>
> > >
> > > and any copies
> > > > > of it.
> > > > > > Opinions,
> > > > > > > > conclusion
> > > > > > > > > (etc) that
> > > > > > > > > > do
> > > > > > > > > > >> > not relate
> > > > > > > > > > >> > >>
> to
> > > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > official
> > > > > business of this
> > > > > > > company
> > > > > > > > shall be
> > > > > > > > > understood
> > > > > > > > > > as
> > > > > > > > > > >> > neither
> > > > > > > > > > >> > >>
> > > given nor
> > > > > > > > > > >> > >>
> >>
> > >
> > > endorsed by
> > > > it.
> > > > > IG is a
> > > > > > trading
> > > > > > > name
> > > > > > > > of IG
> > > > > > > > > Markets
> > > > > > > > > > >> Limited
> > > > > > > > > > >> > (a company
> > > > > > > > > > >> > >>
> >>
> > >
> > > registered in
> > > > > England and
> > > > > > Wales,
> > > > > > > > company number
> > > > > > > > > > >> 04008957)
> > > > > > > > > > >> > and IG
> > > > > > > > > > >> > >>
> > Index
> > > > > > > > > > >> > >>
> >>
> > >
> > > Limited (a
> > > > > company registered
> > > > > > in
> > > > > > > > England and
> > > > > > > > > Wales,
> > > > > > > > > > >> > company
> number
> > > > > > > > > > >> > >>
> >>
> > >
> > > 01190902).
> > > > > Registered address
> > > > > > at
> > > > > > > > Cannon Bridge
> > > > > > > > > House,
> > > > > > > > > > 25
> > > > > > > > > > >> > Dowgate
> > > > > > > > > > >> > >>
> > Hill,
> > > > > > > > > > >> > >>
> >>
> > >
> > > London EC4R
> > > > > 2YA. Both IG
> > > > > > Markets
> > > > > > > > Limited
> > > > > > > > > (register
> > > > > > > > > > >> number
> > > > > > > > > > >> > 195355)
> > > > > > > > > > >> > >>
> and
> > > IG
> > > > > > > > > > >> > >>
> >>
> > >
> > > Index Limited
> > > > > (register number
> > > > > > > > 114059) are
> > > > > > > > > authorised
> > > > > > > > > > >> and
> > > > > > > > > > >> > regulated
> > > > > > > > > > >> > >>
> by
> > > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > Financial
> > > > > Conduct Authority.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> --
> > > > > > > > > > >> > >>
> > > Nacho - Ignacio
> > > > > Solis -
> > > > > > > isolis@igso.net
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > The
> > information
> > > contained in
> > > > > this email is
> > > > > > > strictly
> > > > > > > > > confidential and
> > > > > > > > > > for
> > > > > > > > > > >> > the use of
> the
> > > addressee only,
> > > > > unless
> > > > > > otherwise
> > > > > > > > indicated. If
> > > > > > > > > you are
> > > > > > > > > > >> not
> > > > > > > > > > >> > the intended
> > > recipient, please
> > > > > do not read,
> > > > > > > copy, use
> > > > > > > > or
> > > > > > > > > disclose to
> > > > > > > > > > >> others
> > > > > > > > > > >> > this message
> > or
> > > any
> > > > attachment.
> > > > > Please also
> > > > > > > notify
> > > > > > > > the sender by
> > > > > > > > > > >> replying
> > > > > > > > > > >> > to this
> email
> > > or by telephone
> > > > > (+44(020 7896
> > > > > > > 0011) and
> > > > > > > > then
> > > > > > > > > delete the
> > > > > > > > > > >> email
> > > > > > > > > > >> > and any
> copies
> > > of it.
> > > > Opinions,
> > > > > conclusion
> > > > > > > (etc) that
> > > > > > > > do not
> > > > > > > > > relate to
> > > > > > > > > > >> the
> > > > > > > > > > >> > official
> > > business of this
> > > > > company shall be
> > > > > > > understood
> > > > > > > > as
> > > > > > > > > neither given
> > > > > > > > > > >> nor
> > > > > > > > > > >> > endorsed by
> > it.
> > > IG is a
> > > > trading
> > > > > name of IG
> > > > > > > Markets
> > > > > > > > Limited (a
> > > > > > > > > company
> > > > > > > > > > >> > registered
> in
> > > England and
> > > > > Wales, company
> > > > > > number
> > > > > > > > 04008957) and
> > > > > > > > > IG Index
> > > > > > > > > > >> > Limited (a
> > > company registered
> > > > > in England and
> > > > > > > Wales,
> > > > > > > > company
> > > > > > > > > number
> > > > > > > > > > >> > 01190902).
> > > Registered address
> > > > > at Cannon Bridge
> > > > > > > House,
> > > > > > > > 25
> > > > > > > > > Dowgate Hill,
> > > > > > > > > > >> > London EC4R
> > > 2YA. Both IG
> > > > > Markets Limited
> > > > > > > (register
> > > > > > > > number
> > > > > > > > > 195355) and
> > > > > > > > > > IG
> > > > > > > > > > >> > Index
> Limited
> > > (register number
> > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated by
> > > > > > > > > > >> the
> > > > > > > > > > >> > Financial
> > > Conduct Authority.
> > > > > > > > > > >> >
> > > > > > > > > > >> The
> information
> > > contained in
> > > > this
> > > > > email is
> > > > > > > strictly
> > > > > > > > confidential
> > > > > > > > > and for
> > > > > > > > > > >> the use of the
> > > addressee only,
> > > > > unless otherwise
> > > > > > > > indicated. If you
> > > > > > > > > are
> > > > > > > > > > not
> > > > > > > > > > >> the intended
> > > recipient, please
> > > > do
> > > > > not read,
> > > > > > copy,
> > > > > > > use
> > > > > > > > or disclose
> > > > > > > > > to
> > > > > > > > > > others
> > > > > > > > > > >> this message
> or
> > > any attachment.
> > > > > Please also
> > > > > > > notify the
> > > > > > > > sender by
> > > > > > > > > > replying
> > > > > > > > > > >> to this email
> or
> > > by telephone
> > > > > (+44(020 7896
> > > > > > 0011)
> > > > > > > and
> > > > > > > > then delete
> > > > > > > > > the
> > > > > > > > > > email
> > > > > > > > > > >> and any copies
> > of
> > > it. Opinions,
> > > > > conclusion (etc)
> > > > > > > that
> > > > > > > > do not
> > > > > > > > > relate to
> > > > > > > > > > the
> > > > > > > > > > >> official
> > business
> > > of this
> > > > company
> > > > > shall be
> > > > > > > understood
> > > > > > > > as neither
> > > > > > > > > given
> > > > > > > > > > nor
> > > > > > > > > > >> endorsed by
> it.
> > > IG is a trading
> > > > > name of IG
> > > > > > Markets
> > > > > > > > Limited (a
> > > > > > > > > company
> > > > > > > > > > >> registered in
> > > England and Wales,
> > > > > company number
> > > > > > > > 04008957) and IG
> > > > > > > > > Index
> > > > > > > > > > >> Limited (a
> > > company registered in
> > > > > England and
> > > > > > > Wales,
> > > > > > > > company number
> > > > > > > > > > >> 01190902).
> > > Registered address at
> > > > > Cannon Bridge
> > > > > > > House,
> > > > > > > > 25 Dowgate
> > > > > > > > > Hill,
> > > > > > > > > > >> London EC4R
> 2YA.
> > > Both IG Markets
> > > > > Limited
> > > > > > (register
> > > > > > > > number 195355)
> > > > > > > > > and IG
> > > > > > > > > > >> Index Limited
> > > (register number
> > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated by
> > > > > > > > > > the
> > > > > > > > > > >> Financial
> > Conduct
> > > Authority.
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > The information
> > > contained in this
> > > > > email is strictly
> > > > > > > > confidential and
> > > > > > > > > for the use of the
> > > addressee only, unless
> > > > > otherwise
> > > > > > > indicated.
> > > > > > > > If you are
> > > > > > > > > not the intended
> > > recipient, please do not
> > > > > read, copy, use
> > > > > > > or
> > > > > > > > disclose to
> > > > > > > > > others this message or
> > any
> > > attachment.
> > > > > Please also notify
> > > > > > > the
> > > > > > > > sender by
> > > > > > > > > replying to this email
> or
> > > by telephone
> > > > > (+44(020 7896
> > > > > > 0011)
> > > > > > > and
> > > > > > > > then delete
> > > > > > > > > the email and any
> copies
> > > of it. Opinions,
> > > > > conclusion
> > > > > > (etc)
> > > > > > > that
> > > > > > > > do not
> > > > > > > > > relate to the official
> > > business of this
> > > > > company shall be
> > > > > > > > understood as
> > > > > > > > > neither given nor
> > endorsed
> > > by it. IG is a
> > > > > trading name of
> > > > > > > IG
> > > > > > > > Markets
> > > > > > > > > Limited (a company
> > > registered in England
> > > > > and Wales,
> > > > > > company
> > > > > > > > number
> > > > > > > > > 04008957) and IG Index
> > > Limited (a company
> > > > > registered in
> > > > > > > England
> > > > > > > > and Wales,
> > > > > > > > > company number
> 01190902).
> > > Registered
> > > > > address at Cannon
> > > > > > > Bridge
> > > > > > > > House, 25
> > > > > > > > > Dowgate Hill, London
> EC4R
> > > 2YA. Both IG
> > > > > Markets Limited
> > > > > > > (register
> > > > > > > > number
> > > > > > > > > 195355) and IG Index
> > > Limited (register
> > > > > number 114059) are
> > > > > > > > authorised and
> > > > > > > > > regulated by the
> > Financial
> > > Conduct
> > > > > Authority.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in
> > > this email is
> > > > > strictly
> > > > > > confidential
> > > > > > > and
> > > > > > > > for the use of the addressee
> only,
> > > unless otherwise
> > > > > indicated. If
> > > > > > > you are
> > > > > > > > not the intended recipient,
> please
> > > do not read,
> > > > > copy, use or
> > > > > > > disclose to
> > > > > > > > others this message or any
> > > attachment. Please also
> > > > > notify the
> > > > > > sender
> > > > > > > by
> > > > > > > > replying to this email or by
> > > telephone (+44(020
> > > > 7896
> > > > > 0011) and then
> > > > > > > delete
> > > > > > > > the email and any copies of it.
> > > Opinions,
> > > > conclusion
> > > > > (etc) that do
> > > > > > > not
> > > > > > > > relate to the official business
> of
> > > this company
> > > > > shall be understood
> > > > > > > as
> > > > > > > > neither given nor endorsed by it.
> > IG
> > > is a trading
> > > > > name of IG
> > > > > > Markets
> > > > > > > > Limited (a company registered in
> > > England and Wales,
> > > > > company number
> > > > > > > > 04008957) and IG Index Limited (a
> > > company
> > > > registered
> > > > > in England and
> > > > > > > Wales,
> > > > > > > > company number 01190902).
> > Registered
> > > address at
> > > > > Cannon Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > > Dowgate Hill, London EC4R 2YA.
> Both
> > > IG Markets
> > > > > Limited (register
> > > > > > > number
> > > > > > > > 195355) and IG Index Limited
> > > (register number
> > > > > 114059) are
> > > > > > authorised
> > > > > > > and
> > > > > > > > regulated by the Financial
> Conduct
> > > Authority.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email
> > is
> > > strictly
> > > > > confidential and for
> > > > > > > the use of the addressee only, unless
> > > otherwise
> > > > indicated.
> > > > > If you are not
> > > > > > > the intended recipient, please do not
> > > read, copy, use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please
> > > also notify the
> > > > > sender by replying
> > > > > > > to this email or by telephone (+44(020
> > > 7896 0011) and
> > > > then
> > > > > delete the
> > > > > > email
> > > > > > > and any copies of it. Opinions,
> > conclusion
> > > (etc) that do
> > > > > not relate to
> > > > > > the
> > > > > > > official business of this company shall
> > be
> > > understood as
> > > > > neither given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name of
> > IG
> > > Markets
> > > > Limited
> > > > > (a company
> > > > > > > registered in England and Wales,
> company
> > > number 04008957)
> > > > > and IG Index
> > > > > > > Limited (a company registered in
> England
> > > and Wales,
> > > > > company number
> > > > > > > 01190902). Registered address at Cannon
> > > Bridge House, 25
> > > > > Dowgate Hill,
> > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > > (register number
> > > > > 195355) and IG
> > > > > > > Index Limited (register number 114059)
> > are
> > > authorised and
> > > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > > confidential and
> > > > > for the use of the addressee only, unless otherwise
> > > indicated. If you are
> > > > > not the intended recipient, please do not read, copy,
> use
> > > or disclose to
> > > > > others this message or any attachment. Please also
> notify
> > > the sender by
> > > > > replying to this email or by telephone (+44(020 7896
> > 0011)
> > > and then
> > > > delete
> > > > > the email and any copies of it. Opinions, conclusion
> > (etc)
> > > that do not
> > > > > relate to the official business of this company shall
> be
> > > understood as
> > > > > neither given nor endorsed by it. IG is a trading name
> of
> > > IG Markets
> > > > > Limited (a company registered in England and Wales,
> > > company number
> > > > > 04008957) and IG Index Limited (a company registered in
> > > England and
> > > > Wales,
> > > > > company number 01190902). Registered address at Cannon
> > > Bridge House, 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > > (register number
> > > > > 195355) and IG Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Jun, that's a good thought. Maybe then the full schema could look like
this:
Record =>
Length => uintVar
Attributes => int8
TimestampDelta => intVar
OffsetDelta => uintVar
Key => bytes
Value => bytes
Headers => [HeaderKey HeaderValue]
HeaderKey => string
HeaderValue => bytes
(where the bytes and string type use a variable length size field)
Does that work for you, Michael?
-Jason
On Mon, Feb 20, 2017 at 9:28 AM, Jun Rao <ju...@confluent.io> wrote:
> Regarding the overhead of skipping the headers in the message. Another way
> could be to add the headers at the end of the message after the value part.
> This way, there is no need to skip the headers unless you really need it.
> This will allow us to represent the headers explicitly as an array of
> key/value pairs without the skipping overhead when headers are not needed.
>
> Thanks,
>
> Jun
>
> On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Michael,
> >
> > Here's a link to the branch with the message format changes:
> > https://github.com/confluentinc/kafka/tree/exactly-once-message-format.
> > The
> > class to modify to add headers is EosLogRecord (name will definitely
> change
> > prior to merging in trunk). This branch uses attributes to indicate null
> > key and value. I really don't have a strong preference for or against
> this
> > as long as we're consistent. I do feel it's kind of a pity to need the
> > signed version of the varint just to be able to indicate null. For
> headers,
> > it's an easier call because we don't actually need to represent null
> > headers (using size zero will work fine).
> >
> > -Jason
> >
> > On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > We can make it, implement Iterable<Header>, no concerns on this.
> > >
> > > Ive noted this on the KIP.
> > >
> > >
> > > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com> wrote:
> > >
> > > Btw Jason, can you share the branch with the VarInts helper/util
> > class
> > > that KIP-98 will be introducing and also will have the current modified
> > > (key and value lengths) message, so we can look at, thanks.
> > >
> > > On 20/02/2017, 01:11, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > RE “ProducerRecord and ConsumerRecord are immutable
> currently”. I
> > > beg to differ on this, if I set a byte[] as my value of my record, I
> can
> > > later call value() and then alter the bytes in the array. As such it is
> > > very much mutable.
> > >
> > > Headers also will anyhow by nature be mutable on produce, as
> you
> > > add to them.
> > >
> > >
> > > On 20/02/2017, 00:22, "Jason Gustafson" <ja...@confluent.io>
> > > wrote:
> > >
> > > Re; having a dedicated Headers object. I think it makes
> > sense.
> > > Maybe it
> > > could implement Iterable<Header>?
> > >
> > > One thing I'm not too sure about is mutability. Both the
> > > ProducerRecord and
> > > ConsumerRecord are immutable currently. It would be nice to
> > > keep it that
> > > way. Perhaps Headers could itself be immutable and passed
> > > through the
> > > constructors?
> > >
> > > Just going through how this is, I don’t believe this would
> > > provide much
> > > > saving. If issue trying to resolve is around
> object/garbage
> > > creation of
> > > > Map.Entry objects.
> > >
> > >
> > > It's definitely better because we only allocate the space
> we
> > > will use, but
> > > I assume most of the actual memory cost is in the header
> data
> > > itself which
> > > we can't do much about. Maybe we can start with the simple
> > > array
> > > representation and optimize from there after we find it's
> > > worthwhile to do
> > > so? If we use the Headers object, we can change this later
> > > without breaking
> > > the API.
> > >
> > > -Jason
> > >
> > > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> > > radai.rosenblatt@gmail.com> wrote:
> > >
> > > > "(By the way, doesn't it feel a bit odd that we seem to
> be
> > > designing a
> > > > feature which is optimized for people not using?)"
> > > >
> > > > very. this (i claim :-) ) is the result of very intense
> > > opposition to the
> > > > usefulness of the feature early on, and not a real design
> > > goal
> > > >
> > > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > Whilst if having an array of a Header[] would mean
> > getting
> > > this through,
> > > > > im happy changing this, as already done.
> > > > >
> > > > > Just going through how this is, I don’t believe this
> > > would provide much
> > > > > saving. If issue trying to resolve is around
> > > object/garbage creation of
> > > > > Map.Entry objects.
> > > > >
> > > > > All you’re doing here is replacing the equiv of a
> > > HashMap.Node
> > > > (Map.Entry)
> > > > > which simply holds reference to key and value objects
> > with
> > > a custom
> > > > variant.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 19/02/2017, 20:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com> wrote:
> > > > >
> > > > > On point 1 & 2 Ive updated KIP to show varints (and
> > > removed the bit
> > > > > flag). (on the assumption KIP 98 is getting the
> agreement
> > > the protocol is
> > > > > moving from int32 to varInts as standard)
> > > > >
> > > > > On point 3 ive updated to use an array of Header
> > > class, instead of a
> > > > > MultiMap in the Headers class object
> > > > >
> > > > >
> > > > >
> > > > > On 19/02/2017, 20:06, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > On points 1 and 2 I agree.
> > > > >
> > > > > This also affects kip-98, I should expect this
> > > resolved before
> > > > > that vote also passes. If it is accepted there (I’m
> > > assuming this is
> > > > > getting discussed on that KIP? As you’re driving the
> move
> > > to VarInts), I
> > > > am
> > > > > happy to make this KIP will simply follow suit to
> > whatever
> > > is agreed in
> > > > > KIP-98.
> > > > >
> > > > > On 3) Agreed this is a simpler form, as long as
> > no
> > > one is
> > > > > expecting hashmap lookup performance ( O(1) ), I still
> > > would prefer an
> > > > > encapsulated class, so if we find that holding it as an
> > > array in future
> > > > is
> > > > > causing some perf issues, the internals are not exposed
> > to
> > > end users,
> > > > > allowing the internal structure to move to a map.
> > > > >
> > > > > Can we compromise on?
> > > > >
> > > > > class ConsumerRecord<K, V> {
> > > > > K key;
> > > > > V value;
> > > > > Headers headers;
> > > > > }
> > > > >
> > > > > class Headers {
> > > > > Header[] headers;
> > > > >
> > > > > add(String key, byte[] value)
> > > > > Collection<byte[]> get(String key)
> > > > > }
> > > > >
> > > > > class Header {
> > > > > String key;
> > > > > byte[] value;
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > > On 19/02/2017, 18:54, "Jason Gustafson" <
> > > jason@confluent.io>
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > headers dont "leak" into application
> code.
> > > they are useful
> > > > > to application
> > > > > > code as well.
> > > > >
> > > > >
> > > > > This is exactly what I have been trying to
> > get
> > > at. The use
> > > > > cases documented
> > > > > here are middleware:
> > > > > https://cwiki.apache.org/
> > > confluence/display/KAFKA/A+
> > > > > Case+for+Kafka+Headers.
> > > > > If headers are intended for the application
> > > space as well,
> > > > the
> > > > > document
> > > > > should be updated accordingly so that it is
> > an
> > > explicit
> > > > design
> > > > > goal and not
> > > > > an unfortunate design byproduct. There may
> be
> > > some
> > > > > disagreement on whether
> > > > > it _should_ be a design goal, but it may
> not
> > > matter much
> > > > since
> > > > > the current
> > > > > interceptor API is probably insufficient
> for
> > > middleware
> > > > > applications (I
> > > > > haven't thought of a way to do this that
> > isn't
> > > cumbersome).
> > > > >
> > > > > That aside, the three unresolved points for
> > me
> > > are the
> > > > > following:
> > > > >
> > > > > 1. The use of varints. As a one-off for use
> > > only with record
> > > > > headers, the
> > > > > case was weak, but if we are using them
> > > throughout the
> > > > message
> > > > > format, then
> > > > > we should do so here as well. The
> additional
> > > complexity is
> > > > > minimal and
> > > > > early performance testing fully justifies
> > > their use.
> > > > >
> > > > > 2. If varints are used, the case for using
> > > attributes to
> > > > > indicate null
> > > > > headers also becomes weak. We only add one
> > > additional byte in
> > > > > each message
> > > > > if there are no headers. Whatever the case,
> > > let us be
> > > > > consistent with how
> > > > > we treat null keys and values.
> > > > >
> > > > > 3. We have apparently agreed that the
> broker
> > > will validate
> > > > > headers (please
> > > > > add this to the KIP). That being the case,
> I
> > > would prefer to
> > > > > use Kafka's
> > > > > conventional format for arrays. The
> downside
> > > is that it takes
> > > > > more work to
> > > > > skip over the headers on the consumer,
> though
> > > it's unclear if
> > > > > the cost
> > > > > would matter in practice. Some concern was
> > > previously
> > > > > expressed about the
> > > > > allocation of maps. An alternative would be
> > to
> > > use arrays,
> > > > i.e.
> > > > >
> > > > > class ConsumerRecord<K, V> {
> > > > > K key;
> > > > > V value;
> > > > > Header[] headers;
> > > > > }
> > > > >
> > > > > class Header {
> > > > > String key;
> > > > > byte[] value;
> > > > > }
> > > > >
> > > > > This would work nicely with the
> conventional
> > > array format and
> > > > > my guess is
> > > > > it would obviate the need do any lazy
> > > initialization. If we
> > > > > use the map as
> > > > > is currently documented, then it is
> possible
> > > with either
> > > > > representation to
> > > > > slice the headers and initialize them
> lazily.
> > > Either way, it
> > > > > might be a
> > > > > good idea to use a separate object to
> > > represent the headers
> > > > in
> > > > > case we need
> > > > > to modify it in the future in some way.
> > > > >
> > > > > (By the way, doesn't it feel a bit odd that
> > we
> > > seem to be
> > > > > designing a
> > > > > feature which is optimized for people not
> > > using?)
> > > > >
> > > > >
> > > > > If we can resolve these points, then at
> least
> > > you will get my
> > > > > vote.
> > > > >
> > > > > Thanks,
> > > > > Jason
> > > > >
> > > > > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > > > > radai.rosenblatt@gmail.com> wrote:
> > > > >
> > > > > > headers dont "leak" into application
> code.
> > > they are useful
> > > > > to application
> > > > > > code as well.
> > > > > > IIUC samze currently has headers "in-V"
> and
> > > would just
> > > > > switch over to kafka
> > > > > > headers if they exist.
> > > > > > im sure plenty of other users of kafka
> > would
> > > have a use for
> > > > > headers.
> > > > > > im pretty sure use cases exist around
> > > shuffling data
> > > > > into/out-of kafka
> > > > > > (kafka connect or equivalent) where
> > metadata
> > > from one end
> > > > > could copied over
> > > > > > to the other (S3, for example uses http
> > > headers for
> > > > > user-accessible
> > > > > > metadata). it will be kafka client code
> > > getting/setting
> > > > > those headers. not
> > > > > > an interceptor.
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 1:41 PM, Michael
> > > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > For APM single event tracing, need
> access
> > > to the header
> > > > at
> > > > > the point of
> > > > > > > processing on the processing thread.
> > > > > > >
> > > > > > > As such interceptors will not work/be
> > > suitable for these,
> > > > > due to the fact
> > > > > > > they act on the ConsumerRecords as a
> > > batch, before the
> > > > > handling thread
> > > > > > can
> > > > > > > split out and process per message which
> > is
> > > the point
> > > > these
> > > > > tools will
> > > > > > need
> > > > > > > to continue to transaction tracing.
> > > > > > >
> > > > > > > Like wise tools and infra pieces will
> > need
> > > access to the
> > > > > message outside
> > > > > > > the interceptor realm.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 17/02/2017, 21:26, "Jason
> Gustafson" <
> > > > > jason@confluent.io> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > That’s exactly what we’re doing
> the
> > > headers are a
> > > > > slice of bytes,
> > > > > > > which
> > > > > > > > then gets parsed later if needed,
> > or
> > > can be parsed
> > > > > right away, the
> > > > > > > headers
> > > > > > > > is part of the protocol, so can
> > > still be validated
> > > > > if wanted.
> > > > > > > > If you had a header count then
> you
> > > would have to go
> > > > > through each
> > > > > > > header
> > > > > > > > key and value length value to
> work
> > > out how much to
> > > > > skip to get to
> > > > > > > say the
> > > > > > > > value or any future component in
> > the
> > > message after
> > > > > the headers.
> > > > > > > Having it
> > > > > > > > as a byte[] with length value
> makes
> > > this a lot
> > > > > easier to skip.
> > > > > > >
> > > > > > >
> > > > > > > So the broker will parse the
> headers
> > > and validate
> > > > > them. Good. The
> > > > > > only
> > > > > > > reason remaining that I can see to
> > > leave the headers
> > > > > as a byte array
> > > > > > > is to
> > > > > > > make it easier for the client to
> skip
> > > past them. Are
> > > > > we sure this is
> > > > > > > not
> > > > > > > premature optimization? Are there
> any
> > > performance
> > > > > results which show
> > > > > > > that
> > > > > > > this is worthwhile?
> > > > > > >
> > > > > > > What’s the issue with exposing a
> > > method getHeaders on
> > > > > the
> > > > > > > producer/consumer
> > > > > > > > record? It doesn’t break
> anything.
> > > We don’t need
> > > > any
> > > > > special
> > > > > > version.
> > > > > > >
> > > > > > >
> > > > > > > See my previous explanation. What I
> > am
> > > trying to
> > > > > resist is the
> > > > > > headers
> > > > > > > becoming a general
> application-level
> > > facility. The
> > > > > primary use case
> > > > > > as
> > > > > > > far
> > > > > > > as I can tell is middleware, which
> is
> > > the use case
> > > > > that the
> > > > > > > interceptors
> > > > > > > are providing.
> > > > > > >
> > > > > > > Current batch consumer model and
> > > consumer
> > > > interceptors
> > > > > don’t work
> > > > > > where
> > > > > > > > headers need to be acted on at
> per
> > > message level at
> > > > > time of
> > > > > > > processing,
> > > > > > > > very case is APM (the core one),
> > > where the header
> > > > > value is used to
> > > > > > > continue
> > > > > > > > tracing.
> > > > > > >
> > > > > > >
> > > > > > > I still don't understand the point
> > > about batching.
> > > > The
> > > > > consumer
> > > > > > > records are
> > > > > > > exposed as a batch in the consumer
> > > interceptor, but
> > > > > you can still
> > > > > > > iterate
> > > > > > > them individually. It is no
> different
> > > for the
> > > > consumer
> > > > > API itself.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 12:48 PM,
> > > Michael Pearce <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Re:
> > > > > > > >
> > > > > > > > “ The point about creation of
> > > maps seems
> > > > > orthogonal. We can
> > > > > > still
> > > > > > > > represent
> > > > > > > > the headers as a slice of
> bytes
> > > until the time
> > > > > it is accessed.”
> > > > > > > >
> > > > > > > > That’s exactly what we’re doing
> the
> > > headers are a
> > > > > slice of bytes,
> > > > > > > which
> > > > > > > > then gets parsed later if needed,
> > or
> > > can be parsed
> > > > > right away, the
> > > > > > > headers
> > > > > > > > is part of the protocol, so can
> > > still be validated
> > > > > if wanted.
> > > > > > > >
> > > > > > > > If you had a header count then
> you
> > > would have to go
> > > > > through each
> > > > > > > header
> > > > > > > > key and value length value to
> work
> > > out how much to
> > > > > skip to get to
> > > > > > > say the
> > > > > > > > value or any future component in
> > the
> > > message after
> > > > > the headers.
> > > > > > > Having it
> > > > > > > > as a byte[] with length value
> makes
> > > this a lot
> > > > > easier to skip.
> > > > > > > >
> > > > > > > >
> > > > > > > > On 17/02/2017, 20:37, "Michael
> > > Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > What’s the issue with
> exposing
> > a
> > > method
> > > > > getHeaders on the
> > > > > > > > producer/consumer record? It
> > doesn’t
> > > break
> > > > anything.
> > > > > We don’t need
> > > > > > > any
> > > > > > > > special version.
> > > > > > > >
> > > > > > > > Current batch consumer model
> > and
> > > consumer
> > > > > interceptors don’t
> > > > > > work
> > > > > > > > where headers need to be acted on
> > at
> > > per message
> > > > > level at time of
> > > > > > > > processing, very case is APM (the
> > > core one), where
> > > > > the header value
> > > > > > > is used
> > > > > > > > to continue tracing. JMS/HTTP etc
> > > all expose these,
> > > > > without issues.
> > > > > > > I would
> > > > > > > > NOT want to lock this down to
> only
> > > be usable
> > > > > accessible via
> > > > > > > interceptors,
> > > > > > > > as we’d fail on one of the main
> > > goals.
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Mike
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 17/02/2017, 20:21, "Jason
> > > Gustafson" <
> > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > The point about creation
> of
> > > maps seems
> > > > > orthogonal. We can
> > > > > > > still
> > > > > > > > represent
> > > > > > > > the headers as a slice of
> > > bytes until the
> > > > > time it is
> > > > > > > accessed.
> > > > > > > >
> > > > > > > >
> > > > > > > > > Yes exactly we have
> > access
> > > to the records
> > > > > thus why the
> > > > > > > header
> > > > > > > > should be
> > > > > > > > > accessible via it and
> not
> > > hidden for only
> > > > > interceptors to
> > > > > > > access.
> > > > > > > >
> > > > > > > >
> > > > > > > > As explained above, the
> > > point is to make
> > > > the
> > > > > intended usage
> > > > > > > clear.
> > > > > > > > Applications should
> > continue
> > > to rely on the
> > > > > key/value
> > > > > > fields
> > > > > > > to
> > > > > > > > serialize
> > > > > > > > their own headers, and it
> > > would be more
> > > > > ideal if we can
> > > > > > avoid
> > > > > > > > leaking
> > > > > > > > third-party headers into
> > > applications. This
> > > > > is difficult to
> > > > > > > do
> > > > > > > > with the
> > > > > > > > current interceptors
> > because
> > > they share the
> > > > > record objects
> > > > > > > with
> > > > > > > > the common
> > > > > > > > API. What I had in mind
> is
> > > something like
> > > > an
> > > > > extension of
> > > > > > the
> > > > > > > > current
> > > > > > > > interceptors which
> exposed
> > a
> > > different
> > > > > object (e.g.
> > > > > > > > `RecordAndHeaders`).
> > > > > > > > The challenge is for
> > MM-like
> > > use cases. Let
> > > > > me see if I can
> > > > > > > come
> > > > > > > > up with a
> > > > > > > > concrete proposal for
> that
> > > problem.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at
> > > 11:55 AM, Michael
> > > > > Pearce <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I am happy to move the
> > > definition of the
> > > > > header into the
> > > > > > > message
> > > > > > > > body, but
> > > > > > > > > would cause us not to
> > lazy
> > > > > initialise/parse the headers,
> > > > > > as
> > > > > > > > obviously, we
> > > > > > > > > would have to traverse
> > > these reading the
> > > > > message.
> > > > > > > > >
> > > > > > > > > This was actually one
> of
> > > Jay’s requests:
> > > > > > > > >
> > > > > > > > > “ 2. I think we
> should
> > > think about
> > > > > creating the Map
> > > > > > > lazily to
> > > > > > > > avoid
> > > > > > > > > parsing out all the
> > > headers into
> > > > > little objects.
> > > > > > > HashMaps
> > > > > > > > themselves
> > > > > > > > > are
> > > > > > > > > kind of expensive
> and
> > > the consumer is
> > > > > very perf
> > > > > > > sensitive so
> > > > > > > > and making
> > > > > > > > > gazillions of
> > hashmaps
> > > that may or
> > > > may
> > > > > not get used
> > > > > > is
> > > > > > > > probably a bad
> > > > > > > > > idea.”
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 17/02/2017, 19:44,
> > > "Michael Pearce" <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Yes exactly we have
> > > access to the
> > > > > records thus why
> > > > > > the
> > > > > > > > header should
> > > > > > > > > be accessible via it
> and
> > > not hidden for
> > > > > only interceptors
> > > > > > > to
> > > > > > > > access.
> > > > > > > > >
> > > > > > > > > Sent using OWA for
> > > iPhone
> > > > > > > > >
> > > ______________________________
> > > > > __________
> > > > > > > > > From: Magnus
> > Edenhill <
> > > > > magnus@edenhill.se>
> > > > > > > > > Sent: Friday,
> > February
> > > 17, 2017
> > > > > 7:34:49 PM
> > > > > > > > > To:
> > > dev@kafka.apache.org
> > > > > > > > > Subject: Re:
> > [DISCUSS]
> > > KIP-82 - Add
> > > > > Record Headers
> > > > > > > > >
> > > > > > > > > Big +1 on VarInts.
> > > > > > > > > CPUs are fast,
> memory
> > > is slow.
> > > > > > > > >
> > > > > > > > > I agree with Jason
> > > that we'll want to
> > > > > continue
> > > > > > > verifying
> > > > > > > > messages,
> > > > > > > > > including their
> > > headers, so while I
> > > > > appreciate the
> > > > > > > idea of
> > > > > > > > the opaque
> > > > > > > > > header blob it
> won't
> > > be useful in
> > > > > practice.
> > > > > > > > >
> > > > > > > > > /Magnus
> > > > > > > > >
> > > > > > > > > 2017-02-17 10:41
> > > GMT-08:00 Jason
> > > > > Gustafson <
> > > > > > > > jason@confluent.io>:
> > > > > > > > >
> > > > > > > > > > Sorry, my
> mistake.
> > > The consumer
> > > > > interceptor is per
> > > > > > > batch,
> > > > > > > > though I'm
> > > > > > > > > not
> > > > > > > > > > sure that's an
> > > actual limitation
> > > > > since you still
> > > > > > have
> > > > > > > > access to the
> > > > > > > > > > individual
> records.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 17,
> > 2017
> > > at 10:39 AM,
> > > > > Jason Gustafson <
> > > > > > > > > jason@confluent.io>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Re headers as
> > byte
> > > array and
> > > > > future use by
> > > > > > broker.
> > > > > > > This
> > > > > > > > doesn't
> > > > > > > > > take away
> > > > > > > > > > >> from that at
> > all.
> > > Nor makes it
> > > > > difficult at all
> > > > > > > in my
> > > > > > > > opinion.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Yeah, I didn't
> > say
> > > it was
> > > > > difficult, only
> > > > > > awkward.
> > > > > > > You
> > > > > > > > wouldn't
> > > > > > > > > write the
> > > > > > > > > > > schema that way
> > if
> > > you were
> > > > > planning to use it on
> > > > > > > the
> > > > > > > > brokers from
> > > > > > > > > the
> > > > > > > > > > > beginning. Note
> > > also that one of
> > > > > the benefits of
> > > > > > > letting
> > > > > > > > the broker
> > > > > > > > > > > understand
> > headers
> > > is that it can
> > > > > validate that
> > > > > > > they are
> > > > > > > > properly
> > > > > > > > > > > formatted. If
> > cost
> > > is the only
> > > > > concern, we should
> > > > > > > > confirm its
> > > > > > > > > impact
> > > > > > > > > > > through
> > > performance testing.
> > > > > > > > > > >
> > > > > > > > > > > One of the key
> > use
> > > cases requires
> > > > > access on
> > > > > > > consume at
> > > > > > > > per
> > > > > > > > > event/message
> > > > > > > > > > >> level at the
> > > point that message
> > > > > is being
> > > > > > > processed, as
> > > > > > > > such the
> > > > > > > > > batch
> > > > > > > > > > >> interceptors
> and
> > > batch consume
> > > > > api isn't
> > > > > > > suitable. It
> > > > > > > > needs to be
> > > > > > > > > at the
> > > > > > > > > > >> record level.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I'm not sure I
> > > understand the
> > > > > point about
> > > > > > batching.
> > > > > > > > Interceptors
> > > > > > > > > are
> > > > > > > > > > > applied
> > > per-message, right?
> > > > > > > > > > >
> > > > > > > > > > > My intent on
> > > interceptors is to
> > > > > keep the usage of
> > > > > > > headers
> > > > > > > > > well-defined so
> > > > > > > > > > > that they don't
> > > start leaking
> > > > > unnecessarily into
> > > > > > > > applications. My
> > > > > > > > > guess
> > > > > > > > > > is
> > > > > > > > > > > that it's
> > probably
> > > inevitable,
> > > > but
> > > > > isolating it
> > > > > > in
> > > > > > > the
> > > > > > > > > interceptors would
> > > > > > > > > > > at least give
> > > people a second
> > > > > thought before
> > > > > > > deciding to
> > > > > > > > use it.
> > > > > > > > > The main
> > > > > > > > > > > challenge in my
> > > mind is figuring
> > > > > out how an MM
> > > > > > use
> > > > > > > case
> > > > > > > > would
> > > > > > > > > work. It
> > > > > > > > > > > would be more
> > > cumbersome to
> > > > > replicate headers
> > > > > > > through an
> > > > > > > > > interceptor,
> > > > > > > > > > > though arguably
> > MM
> > > should be
> > > > > working at a lower
> > > > > > > level
> > > > > > > > anyway.
> > > > > > > > > > >
> > > > > > > > > > > -Jason
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Feb 17,
> > > 2017 at 10:16 AM,
> > > > > Michael Pearce
> > > > > > <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Re headers
> > > available on the
> > > > > record va
> > > > > > > interceptors only
> > > > > > > > > > >>
> > > > > > > > > > >> One of the key
> > > use cases
> > > > requires
> > > > > access on
> > > > > > > consume at
> > > > > > > > per
> > > > > > > > > event/message
> > > > > > > > > > >> level at the
> > > point that message
> > > > > is being
> > > > > > > processed, as
> > > > > > > > such the
> > > > > > > > > batch
> > > > > > > > > > >> interceptors
> and
> > > batch consume
> > > > > api isn't
> > > > > > > suitable. It
> > > > > > > > needs to be
> > > > > > > > > at the
> > > > > > > > > > >> record level.
> > > > > > > > > > >>
> > > > > > > > > > >> This anyhow is
> > > similar to
> > > > > jms/http/amqp where
> > > > > > > headers
> > > > > > > > are
> > > > > > > > > available to
> > > > > > > > > > >> consuming
> > > applications.
> > > > > > > > > > >>
> > > > > > > > > > >> Re headers as
> > > byte array and
> > > > > future use by
> > > > > > > broker. This
> > > > > > > > doesn't
> > > > > > > > > take
> > > > > > > > > > away
> > > > > > > > > > >> from that at
> > all.
> > > Nor makes it
> > > > > difficult at all
> > > > > > > in my
> > > > > > > > opinion.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Sent using OWA
> > > for iPhone
> > > > > > > > > > >>
> > > ______________________________
> > > > > __________
> > > > > > > > > > >> From: Jason
> > > Gustafson <
> > > > > jason@confluent.io>
> > > > > > > > > > >> Sent: Friday,
> > > February 17, 2017
> > > > > 5:55:42 PM
> > > > > > > > > > >> To:
> > > dev@kafka.apache.org
> > > > > > > > > > >> Subject: Re:
> > > [DISCUSS] KIP-82 -
> > > > > Add Record
> > > > > > Headers
> > > > > > > > > > >>
> > > > > > > > > > >> >
> > > > > > > > > > >> > Would you be
> > > proposing in
> > > > > KIP-98 to convert
> > > > > > the
> > > > > > > other
> > > > > > > > message
> > > > > > > > > int’s
> > > > > > > > > > (key
> > > > > > > > > > >> > length,
> value
> > > length) also to
> > > > > varint to keep
> > > > > > it
> > > > > > > > uniform.
> > > > > > > > > > >> > Also I
> assume
> > > there will be a
> > > > > static or helper
> > > > > > > method
> > > > > > > > made to
> > > > > > > > > > write/read
> > > > > > > > > > >> > these in the
> > > client and
> > > > server.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Yes, that is
> > what
> > > we are
> > > > > proposing, so using
> > > > > > > varints
> > > > > > > > for headers
> > > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > >> consistent
> with
> > > the rest of the
> > > > > message. We have
> > > > > > > used
> > > > > > > > static
> > > > > > > > > helper
> > > > > > > > > > >> methods
> > > > > > > > > > >> in our
> prototype
> > > implementation.
> > > > > > > > > > >>
> > > > > > > > > > >> The cost of
> > > parsing, we want to
> > > > > parse/interpret
> > > > > > > the
> > > > > > > > headers
> > > > > > > > > lazily (this
> > > > > > > > > > >> is
> > > > > > > > > > >> > a key point
> > > brought up earlier
> > > > > in discussions)
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> I'm a bit
> > > skeptical of this. Has
> > > > > anyone done the
> > > > > > > > performance
> > > > > > > > > testing? I
> > > > > > > > > > >> can
> > > > > > > > > > >> probably
> > > implement it and test
> > > > it
> > > > > if no one else
> > > > > > > has. I
> > > > > > > > was also
> > > > > > > > > under
> > > > > > > > > > the
> > > > > > > > > > >> impression
> that
> > > there may be use
> > > > > cases down the
> > > > > > > road
> > > > > > > > where the
> > > > > > > > > broker
> > > > > > > > > > >> would
> > > > > > > > > > >> need to
> > interpret
> > > headers. That
> > > > > wouldn't be off
> > > > > > > the
> > > > > > > > table in the
> > > > > > > > > future
> > > > > > > > > > if
> > > > > > > > > > >> it's
> represented
> > > as bytes, but
> > > > it
> > > > > would be quite
> > > > > > > a bit
> > > > > > > > more
> > > > > > > > > awkward,
> > > > > > > > > > >> right?
> > > > > > > > > > >>
> > > > > > > > > > >> By the way,
> one
> > > question I have
> > > > > been wondering
> > > > > > > about. My
> > > > > > > > > understanding
> > > > > > > > > > is
> > > > > > > > > > >> that headers
> are
> > > primarily for
> > > > > use cases where a
> > > > > > > > third-party
> > > > > > > > > components
> > > > > > > > > > >> wants to
> enrich
> > > messages without
> > > > > needing to
> > > > > > > understand
> > > > > > > > or modify
> > > > > > > > > the
> > > > > > > > > > >> schema
> > > > > > > > > > >> of the message
> > > key and value.
> > > > For
> > > > > the
> > > > > > applications
> > > > > > > > which directly
> > > > > > > > > > produce
> > > > > > > > > > >> and consume
> the
> > > messages and
> > > > > control the
> > > > > > key/value
> > > > > > > > schema
> > > > > > > > > directly, it
> > > > > > > > > > >> seems we would
> > > rather have them
> > > > > implement
> > > > > > headers
> > > > > > > > directly in
> > > > > > > > > their own
> > > > > > > > > > >> schema.
> > Supposing
> > > for the sake
> > > > of
> > > > > argument that
> > > > > > > it was
> > > > > > > > possible,
> > > > > > > > > my
> > > > > > > > > > >> question is
> > > whether it be
> > > > > sufficient to expose
> > > > > > the
> > > > > > > > headers in the
> > > > > > > > > > >> interceptor
> API
> > > and not in the
> > > > > common API?
> > > > > > > > > > >>
> > > > > > > > > > >> -Jason
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, Feb
> 17,
> > > 2017 at 3:26 AM,
> > > > > Michael Pearce
> > > > > > <
> > > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > On the point
> > of
> > > varInts
> > > > > > > > > > >> >
> > > > > > > > > > >> > Would you be
> > > proposing in
> > > > > KIP-98 to convert
> > > > > > the
> > > > > > > other
> > > > > > > > message
> > > > > > > > > int’s
> > > > > > > > > > (key
> > > > > > > > > > >> > length,
> value
> > > length) also to
> > > > > varint to keep
> > > > > > it
> > > > > > > > uniform.
> > > > > > > > > > >> > Also I
> assume
> > > there will be a
> > > > > static or helper
> > > > > > > method
> > > > > > > > made to
> > > > > > > > > > write/read
> > > > > > > > > > >> > these in the
> > > client and
> > > > server.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Cheers
> > > > > > > > > > >> > Mike
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On
> 17/02/2017,
> > > 11:22, "Michael
> > > > > Pearce" <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > On the
> > > point re: headers
> > > > in
> > > > > the message
> > > > > > > protocol
> > > > > > > > being a
> > > > > > > > > byte
> > > > > > > > > > array
> > > > > > > > > > >> > and not a
> > count
> > > of elements
> > > > > followed by the
> > > > > > > elements.
> > > > > > > > Again
> > > > > > > > > this was
> > > > > > > > > > >> >
> > > discussed/argued previously.
> > > > > > > > > > >> >
> > > > > > > > > > >> > It was
> > > agreed on for a few
> > > > > reasons some of
> > > > > > > which
> > > > > > > > you have
> > > > > > > > > > obviously
> > > > > > > > > > >> > picked up
> on:
> > > > > > > > > > >> >
> > > > > > > > > > >> > Broker
> is
> > > able to pass it
> > > > > through opaquely
> > > > > > > > > > >> > The cost
> > of
> > > parsing, we
> > > > > want to
> > > > > > > parse/interpret
> > > > > > > > the headers
> > > > > > > > > lazily
> > > > > > > > > > >> > (this is a
> key
> > > point brought
> > > > up
> > > > > earlier in
> > > > > > > > discussions)
> > > > > > > > > > >> > Headers
> > can
> > > be copied from
> > > > > consumer record
> > > > > > > to
> > > > > > > > producer
> > > > > > > > > record (aka
> > > > > > > > > > >> > mirror
> makers
> > > etc) without
> > > > > parsing if no
> > > > > > > changes are
> > > > > > > > being made
> > > > > > > > > or
> > > > > > > > > > being
> > > > > > > > > > >> > looked at.
> > > > > > > > > > >> > Keeps
> the
> > > broker agnostic
> > > > > to the format
> > > > > > > > > > >> > You need
> > an
> > > int32 either
> > > > > for the byte size
> > > > > > > of the
> > > > > > > > headers,
> > > > > > > > > or for
> > > > > > > > > > >> the
> > > > > > > > > > >> > count of
> > > elements, so
> > > > overheads
> > > > > are the same,
> > > > > > > but
> > > > > > > > with going
> > > > > > > > > with an
> > > > > > > > > > >> opaque
> > > > > > > > > > >> > byte array
> has
> > > the above
> > > > > advantages.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Cheers
> > > > > > > > > > >> > Mike
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On
> > > 17/02/2017, 02:50,
> > > > > "Jason Gustafson" <
> > > > > > > > jason@confluent.io
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> Sorry,
> > > should have
> > > > > noted that the
> > > > > > > performance
> > > > > > > > testing
> > > > > > > > > was done
> > > > > > > > > > >> > using the
> > > > > > > > > > >> >
> > > producer performance
> > > > > tool shipped with
> > > > > > > Kafka.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> -Jason
> > > > > > > > > > >> >
> > > > > > > > > > >> > On
> > Thu,
> > > Feb 16, 2017
> > > > at
> > > > > 6:44 PM, Jason
> > > > > > > > Gustafson <
> > > > > > > > > > >> >
> > > jason@confluent.io> wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > >
> Hey
> > > Nacho,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> I've
> > > compared
> > > > > performance of our
> > > > > > > KIP-98
> > > > > > > > > implementation with
> > > > > > > > > > >> and
> > > > > > > > > > >> > without
> > > > > > > > > > >> > >
> > > varints. For
> > > > messages
> > > > > around 128
> > > > > > > bytes, we
> > > > > > > > see an
> > > > > > > > > increase
> > > > > > > > > > in
> > > > > > > > > > >> > throughput
> of
> > > > > > > > > > >> > >
> > about
> > > 30% using the
> > > > > default
> > > > > > > configuration
> > > > > > > > settings.
> > > > > > > > > At 256
> > > > > > > > > > >> > bytes, the
> > > > > > > > > > >> > >
> > > increase is around
> > > > > 16%. Obviously
> > > > > > the
> > > > > > > > performance
> > > > > > > > > converge
> > > > > > > > > > as
> > > > > > > > > > >> > messages get
> > > > > > > > > > >> > >
> > > larger, but it seems
> > > > > well worth the
> > > > > > > cost.
> > > > > > > > Note that
> > > > > > > > > we are
> > > > > > > > > > >> also
> > > > > > > > > > >> > seeing a
> > > > > > > > > > >> > >
> > > substantial
> > > > > performance increase
> > > > > > > against
> > > > > > > > trunk
> > > > > > > > > primarily
> > > > > > > > > > >> because
> > > > > > > > > > >> > of the
> > > > > > > > > > >> > >
> much
> > > more efficient
> > > > > packing that
> > > > > > > varints
> > > > > > > > provide us.
> > > > > > > > > > Anything
> > > > > > > > > > >> > adding to
> > > > > > > > > > >> > >
> > > message overhead,
> > > > > such as record
> > > > > > > headers,
> > > > > > > > would only
> > > > > > > > > > increase
> > > > > > > > > > >> > the relative
> > > > > > > > > > >> > >
> > > difference. (Of
> > > > > course take these
> > > > > > > numbers
> > > > > > > > with a
> > > > > > > > > grain of
> > > > > > > > > > salt
> > > > > > > > > > >> > since I have
> > > > > > > > > > >> > >
> only
> > > used the
> > > > default
> > > > > settings with
> > > > > > > both
> > > > > > > > the producer
> > > > > > > > > and
> > > > > > > > > > >> broker
> > > > > > > > > > >> > on my
> > > > > > > > > > >> > >
> > local
> > > machine. We
> > > > > intend to provide
> > > > > > > more
> > > > > > > > extensive
> > > > > > > > > > performance
> > > > > > > > > > >> > details as
> > > > > > > > > > >> > >
> part
> > > of the work for
> > > > > KIP-98.)
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> The
> > > implementation
> > > > we
> > > > > are using is
> > > > > > > from
> > > > > > > > protobuf (
> > > > > > > > > > >> > >
> > > > > https://developers.google.com/
> > > > > > > > > >
> > > protocol-buffers/docs/encoding
> > > > > > > > > > >> ),
> > > > > > > > > > >> > which is
> > > > > > > > > > >> > >
> also
> > > used in HBase.
> > > > > It is trivial to
> > > > > > > > implement and as
> > > > > > > > > far
> > > > > > > > > > as I
> > > > > > > > > > >> > know doesn't
> > > > > > > > > > >> > >
> > > suffer from the
> > > > > aliasing problem you
> > > > > > > are
> > > > > > > > describing. I
> > > > > > > > > > checked
> > > > > > > > > > >> > with Magnus
> > > > > > > > > > >> > >
> (the
> > > author of
> > > > > librdkafka) and he
> > > > > > > agreed
> > > > > > > > that the
> > > > > > > > > savings
> > > > > > > > > > >> seemed
> > > > > > > > > > >> > worth the
> > > > > > > > > > >> > >
> cost
> > > of
> > > > > implementation.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > -Jason
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On
> > > Thu, Feb 16, 2017
> > > > > at 4:32 PM,
> > > > > > > Ignacio
> > > > > > > > Solis <
> > > > > > > > > > >>
> isolis@igso.net
> > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >>
> > > -VarInts
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> I'm
> > > one of the
> > > > > people (if not the
> > > > > > > most)
> > > > > > > > opposed to
> > > > > > > > > VarInts.
> > > > > > > > > > >> > VarInts
> > > > > > > > > > >> > >>
> > have
> > > a place, but
> > > > > this is not it.
> > > > > > > (We
> > > > > > > > had a large
> > > > > > > > > > >> discussion
> > > > > > > > > > >> > about
> > > > > > > > > > >> > >>
> > them
> > > at the
> > > > > beginning of KIP-82
> > > > > > time)
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> If
> > > anybody has real
> > > > > life
> > > > > > performance
> > > > > > > > numbers of
> > > > > > > > > VarInts
> > > > > > > > > > >> > improving
> > > > > > > > > > >> > >>
> > > things or
> > > > > significantly reducing
> > > > > > > resources
> > > > > > > > I would
> > > > > > > > > like to
> > > > > > > > > > >> know
> > > > > > > > > > >> > what
> > > > > > > > > > >> > >>
> > that
> > > case may be.
> > > > > Yes, you can save
> > > > > > > some
> > > > > > > > bytes here
> > > > > > > > > and
> > > > > > > > > > >> there,
> > > > > > > > > > >> > but
> > > > > > > > > > >> > >>
> > this
> > > is probably
> > > > > insignificant to
> > > > > > the
> > > > > > > > overall system
> > > > > > > > > > behavior
> > > > > > > > > > >> > and
> > > > > > > > > > >> > >>
> > > storage
> > > > > requirements. -- I say
> > > > > > this
> > > > > > > with
> > > > > > > > respect to
> > > > > > > > > using
> > > > > > > > > > >> > VarInts in
> > > > > > > > > > >> > >>
> the
> > > protocol
> > > > itself,
> > > > > not as part of
> > > > > > > the
> > > > > > > > data.
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > VarInts require you
> > > > > to parse the
> > > > > > Int
> > > > > > > > before using it
> > > > > > > > > and
> > > > > > > > > > >> > depending on
> > > > > > > > > > >> > >>
> the
> > > encoding they
> > > > > can suffer from
> > > > > > > aliasing
> > > > > > > > (multiple
> > > > > > > > > > >> >
> > representations
> > > > > > > > > > >> > >>
> for
> > > the same
> > > > value).
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> Why
> > > add complexity?
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > Nacho
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> On
> > > Thu, Feb 16,
> > > > 2017
> > > > > at 10:29 AM,
> > > > > > > Colin
> > > > > > > > McCabe <
> > > > > > > > > > >> >
> > > cmccabe@apache.org>
> > > > > > > > > > >> > >>
> > > wrote:
> > > > > > > > > > >> > >> >
> > +1
> > > for varints
> > > > > here-- it would
> > > > > > save
> > > > > > > > quite a bit of
> > > > > > > > > space.
> > > > > > > > > > >> > They are
> > > > > > > > > > >> > >> >
> > > pretty quick to
> > > > > implement as
> > > > > > well.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >>
> > I
> > > think it makes
> > > > > sense for values
> > > > > > > to be
> > > > > > > > byte
> > > > > > > > > arrays.
> > > > > > > > > > Users
> > > > > > > > > > >> > might want
> > > > > > > > > > >> > >> >
> > to
> > > attach
> > > > > arbitrary payloads;
> > > > > > they
> > > > > > > > shouldn't be
> > > > > > > > > forced to
> > > > > > > > > > >> > serialize
> > > > > > > > > > >> > >> >
> > > everything to
> > > > Java
> > > > > strings.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > best,
> > > > > > > > > > >> > >> >
> > > Colin
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > On
> > > Thu, Feb 16,
> > > > > 2017, at 09:52,
> > > > > > > Jason
> > > > > > > > Gustafson
> > > > > > > > > wrote:
> > > > > > > > > > >> > >>
> >>
> > > Hey Michael,
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > Hmm, I guess the
> > > > > point of
> > > > > > > representing
> > > > > > > > it as
> > > > > > > > > bytes is to
> > > > > > > > > > >> > allow the
> > > > > > > > > > >> > >>
> > > broker
> > > > > > > > > > >> > >>
> >>
> > > to pass it
> > > > > through opaquely? Is
> > > > > > > the
> > > > > > > > cost of
> > > > > > > > > parsing
> > > > > > > > > > them a
> > > > > > > > > > >> > concern, or
> > > > > > > > > > >> > >>
> >>
> > > are
> > > > > > > > > > >> > >>
> >>
> > > we simply trying
> > > > > to ensure that
> > > > > > > the
> > > > > > > > broker stays
> > > > > > > > > > agnostic
> > > > > > > > > > >> to
> > > > > > > > > > >> > the
> > > > > > > > > > >> > >>
> > > format?
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > On varints, I
> > > > > think adding
> > > > > > > support for
> > > > > > > > them makes
> > > > > > > > > less
> > > > > > > > > > >> sense
> > > > > > > > > > >> > for an
> > > > > > > > > > >> > >>
> >>
> > > isolated use
> > > > > case, but as part
> > > > > > of
> > > > > > > a
> > > > > > > > more holistic
> > > > > > > > > change
> > > > > > > > > > >> > (such as
> what
> > > > > > > > > > >> > >>
> we
> > > > > > > > > > >> > >>
> >>
> > > have proposed in
> > > > > KIP-98), I
> > > > > > think
> > > > > > > they
> > > > > > > > are
> > > > > > > > > justifiable.
> > > > > > > > > > If
> > > > > > > > > > >> > we add them,
> > > > > > > > > > >> > >>
> >>
> > > then the need to
> > > > > use attributes
> > > > > > > becomes
> > > > > > > > quite a
> > > > > > > > > bit
> > > > > > > > > > >> weaker,
> > > > > > > > > > >> > right? The
> > > > > > > > > > >> > >>
> >>
> > > other thing I
> > > > > find slightly odd
> > > > > > > is the
> > > > > > > > fact that
> > > > > > > > > null
> > > > > > > > > > >> > headers has
> no
> > > > > > > > > > >> > >>
> >>
> > > actual
> > > > > > > > > > >> > >>
> >>
> > > semantic meaning
> > > > > for the message
> > > > > > > > (unlike null
> > > > > > > > > keys and
> > > > > > > > > > >> > values). It
> is
> > > > > > > > > > >> > >>
> >>
> > > just
> > > > > > > > > > >> > >>
> >>
> > a
> > > space
> > > > > optimization. It seems a
> > > > > > > bit
> > > > > > > > better to
> > > > > > > > > always
> > > > > > > > > > use
> > > > > > > > > > >> > size 0 to
> > > > > > > > > > >> > >>
> >>
> > > indicate having
> > > > > no headers.
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > Overall, the
> > > > main
> > > > > point is
> > > > > > > ensuring
> > > > > > > > that the
> > > > > > > > > message
> > > > > > > > > > >> schema
> > > > > > > > > > >> > remains
> > > > > > > > > > >> > >>
> >>
> > > consistent,
> > > > > either within the
> > > > > > > larger
> > > > > > > > protocol, or
> > > > > > > > > at a
> > > > > > > > > > >> > minimum
> within
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > > message itself.
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > -Jason
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > > On Thu, Feb 16,
> > > > > 2017 at 6:39 AM,
> > > > > > > > Michael Pearce <
> > > > > > > > > > >> >
> > > Michael.Pearce@ig.com
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >>
> >>
> > > wrote:
> > > > > > > > > > >> > >>
> >>
> > > > > > > > > > >> > >>
> >>
> > >
> > > Hi Jason,
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On point 1) in
> > > > > the message
> > > > > > > protocol
> > > > > > > > the headers
> > > > > > > > > are
> > > > > > > > > > >> simply
> > > > > > > > > > >> > a byte
> > > > > > > > > > >> > >>
> > > array,
> > > > > > > > > > >> > >>
> >>
> > >
> > > as like the
> > > > key
> > > > > or value, this
> > > > > > > is to
> > > > > > > > clearly
> > > > > > > > > demarcate
> > > > > > > > > > >> the
> > > > > > > > > > >> > header in
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > core message.
> > > > > Then the header
> > > > > > > byte
> > > > > > > > array in the
> > > > > > > > > core
> > > > > > > > > > >> > message is
> an
> > > > > > > > > > >> > >>
> > > array of
> > > > > > > > > > >> > >>
> >>
> > >
> > > key, value
> > > > > pairs. This is what
> > > > > > > it is
> > > > > > > > denoting.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > Then this
> > > > would
> > > > > be I guess in
> > > > > > > the
> > > > > > > > given
> > > > > > > > > notation:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > Headers =>
> > > > > [KeyLength, Key,
> > > > > > > > ValueLength, Value]
> > > > > > > > > > >> > >>
> >>
> > > > KeyLength
> > > > > => int32
> > > > > > > > <-----------------NEW
> > > > > > > > > size of
> > > > > > > > > > the
> > > > > > > > > > >> > byte[] of
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > serialised key
> > > > > value
> > > > > > > > > > >> > >>
> >>
> > > > Key =>
> > > > bytes
> > > > > > > > <---------------------- NEW
> > > > > > > > > > serialised
> > > > > > > > > > >> > string
> (UTF8)
> > > > > > > > > > >> > >>
> >>
> > >
> > > bytes of the
> > > > > header key
> > > > > > > > > > >> > >>
> >>
> > >
> > > > ValueLength
> > > > > => int32
> > > > > > > > <-------------- NEW
> > > > > > > > > size of
> > > > > > > > > > the
> > > > > > > > > > >> > byte[] of
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > serialised
> > > > > header value
> > > > > > > > > > >> > >>
> >>
> > > > Value =>
> > > > > bytes
> > > > > > > > <-------------------- NEW
> > > > > > > > > > serialised
> > > > > > > > > > >> > form of the
> > > > > > > > > > >> > >>
> > > header
> > > > > > > > > > >> > >>
> >>
> > >
> > > value
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > The key length
> > > > > and value
> > > > > > length
> > > > > > > is
> > > > > > > > matching the
> > > > > > > > > way
> > > > > > > > > > the
> > > > > > > > > > >> > protocol is
> > > > > > > > > > >> > >>
> >>
> > >
> > > defined in the
> > > > > core message
> > > > > > > currently.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On point 2)
> > > > > > > > > > >> > >>
> >>
> > >
> > > Var sized
> > > > ints,
> > > > > this was
> > > > > > > discussed
> > > > > > > > much earlier
> > > > > > > > > on, in
> > > > > > > > > > >> > fact I had
> > > > > > > > > > >> > >>
> >>
> > >
> > > suggested it
> > > > > myself (with
> > > > > > Hadoop
> > > > > > > > references),
> > > > > > > > > the
> > > > > > > > > > >> > complexity
> of
> > > this
> > > > > > > > > > >> > >>
> >>
> > >
> > > compared to
> > > > > having a simpler
> > > > > > > protocol
> > > > > > > > was
> > > > > > > > > argued and
> > > > > > > > > > >> > agreed it
> > > > > > > > > > >> > >>
> > > wasn’t worth
> > > > > > > > > > >> > >>
> >>
> > >
> > > the complexity
> > > > > as all other
> > > > > > > clients
> > > > > > > > in other
> > > > > > > > > languages
> > > > > > > > > > >> > would need
> to
> > > > > > > > > > >> > >>
> > > ensure
> > > > > > > > > > >> > >>
> >>
> > >
> > > theyre using
> > > > > the right var
> > > > > > size
> > > > > > > > algorithm, as
> > > > > > > > > there
> > > > > > > > > > is a
> > > > > > > > > > >> > few.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On point 3)
> > > > > > > > > > >> > >>
> >>
> > >
> > > We did the
> > > > > attributes,
> > > > > > optional
> > > > > > > > approach as
> > > > > > > > > originally
> > > > > > > > > > >> > there was
> > > > > > > > > > >> > >>
> > > marked
> > > > > > > > > > >> > >>
> >>
> > >
> > > concern that
> > > > > headers would
> > > > > > > cause a
> > > > > > > > message size
> > > > > > > > > > overhead
> > > > > > > > > > >> > for others,
> > > > > > > > > > >> > >>
> who
> > > > > > > > > > >> > >>
> >>
> > >
> > > don’t want
> > > > > them. As such this
> > > > > > > is the
> > > > > > > > clean
> > > > > > > > > solution to
> > > > > > > > > > >> > achieve
> that.
> > > > > > > > > > >> > >>
> If
> > > > > > > > > > >> > >>
> >>
> > >
> > > that no longer
> > > > > holds, and we
> > > > > > > don’t
> > > > > > > > care that we
> > > > > > > > > add
> > > > > > > > > > >> 4bytes
> > > > > > > > > > >> > overhead,
> > > > > > > > > > >> > >>
> > then
> > > > > > > > > > >> > >>
> >>
> > >
> > > im happy to
> > > > > remove.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > I’m personally
> > > > > in favour of
> > > > > > > keeping
> > > > > > > > the message
> > > > > > > > > as
> > > > > > > > > > small
> > > > > > > > > > >> > as possible
> > > > > > > > > > >> > >>
> so
> > > > > > > > > > >> > >>
> >>
> > >
> > > people don’t
> > > > > get shocks in
> > > > > > perf
> > > > > > > and
> > > > > > > > throughputs
> > > > > > > > > dues
> > > > > > > > > > to
> > > > > > > > > > >> > message
> size,
> > > > > > > > > > >> > >>
> >>
> > >
> > > unless they
> > > > > actively use the
> > > > > > > feature,
> > > > > > > > as such I
> > > > > > > > > do
> > > > > > > > > > >> prefer
> > > > > > > > > > >> > the
> > > > > > > > > > >> > >>
> > > attribute bit
> > > > > > > > > > >> > >>
> >>
> > >
> > > wise feature
> > > > > flag approach
> > > > > > > myself.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > On 16/02/2017,
> > > > > 05:40, "Jason
> > > > > > > > Gustafson" <
> > > > > > > > > > >> >
> > > jason@confluent.io> wrote:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > We have
> > > > > proposed a few
> > > > > > > > significant changes
> > > > > > > > > to the
> > > > > > > > > > >> > message
> format
> > > > > > > > > > >> > >>
> in
> > > > > > > > > > >> > >>
> >>
> > >
> > > KIP-98
> > > > > > > > > > >> > >>
> >>
> > > > which now
> > > > > seems likely to
> > > > > > > pass
> > > > > > > > (perhaps
> > > > > > > > > with some
> > > > > > > > > > >> > iterations
> on
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > implementation details).
> > > > > > It
> > > > > > > would
> > > > > > > > be good
> > > > > > > > > to try
> > > > > > > > > > and
> > > > > > > > > > >> > coordinate
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > changes
> > > > > > > > > > >> > >>
> >>
> > > > in both of
> > > > > the proposals
> > > > > > to
> > > > > > > make
> > > > > > > > sure they
> > > > > > > > > are
> > > > > > > > > > >> > consistent
> and
> > > > > > > > > > >> > >>
> >>
> > >
> > > compatible.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > I think
> > > > > using the
> > > > > > > attributes to
> > > > > > > > indicate
> > > > > > > > > null
> > > > > > > > > > >> headers
> > > > > > > > > > >> > is a
> > > > > > > > > > >> > >>
> > > reasonable
> > > > > > > > > > >> > >>
> >>
> > > > approach.
> > > > > We have proposed
> > > > > > > to do
> > > > > > > > the same
> > > > > > > > > thing
> > > > > > > > > > for
> > > > > > > > > > >> > the message
> > > > > > > > > > >> > >>
> key
> > > and
> > > > > > > > > > >> > >>
> >>
> > > > value.
> > > > That
> > > > > said, I
> > > > > > > sympathize
> > > > > > > > with Jay's
> > > > > > > > > > argument.
> > > > > > > > > > >> > Having
> > > > > > > > > > >> > >>
> > > multiple
> > > > > > > > > > >> > >>
> >>
> > >
> > > ways to
> > > > > > > > > > >> > >>
> >>
> > > > specify a
> > > > > null value
> > > > > > > increases
> > > > > > > > the overall
> > > > > > > > > > >> complexity
> > > > > > > > > > >> > of the
> > > > > > > > > > >> > >>
> > > protocol.
> > > > > > > > > > >> > >>
> >>
> > >
> > > You
> > > > > > > > > > >> > >>
> >>
> > > > can see
> > > > > this just from the
> > > > > > > fact
> > > > > > > > that you
> > > > > > > > > need the
> > > > > > > > > > >> > extra
> verbiage
> > > > > > > > > > >> > >>
> in
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > protocol
> > > > > specification in
> > > > > > > this
> > > > > > > > KIP and in
> > > > > > > > > KIP-98
> > > > > > > > > > to
> > > > > > > > > > >> > describe the
> > > > > > > > > > >> > >>
> >>
> > >
> > > dependence
> > > > > > > > > > >> > >>
> >>
> > > > between
> > > > the
> > > > > fields and the
> > > > > > > > attributes. It
> > > > > > > > > seems
> > > > > > > > > > >> like a
> > > > > > > > > > >> > slippery
> > > > > > > > > > >> > >>
> > slope
> > > > > > > > > > >> > >>
> >>
> > >
> > > if
> > > > > > > > > > >> > >>
> >>
> > > > you start
> > > > > allowing
> > > > > > different
> > > > > > > > request types
> > > > > > > > > to
> > > > > > > > > > >> > implement
> the
> > > > > > > > > > >> > >>
> > > protocol
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > specification differently.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > You can
> > > > > also argue that
> > > > > > the
> > > > > > > > messages
> > > > > > > > > already are
> > > > > > > > > > and
> > > > > > > > > > >> > are likely
> > > > > > > > > > >> > >>
> to
> > > > > > > > > > >> > >>
> >>
> > >
> > > remain a
> > > > > > > > > > >> > >>
> >>
> > > > special
> > > > > case. For example,
> > > > > > > there
> > > > > > > > is
> > > > > > > > > currently no
> > > > > > > > > > >> > generality
> in
> > > > > > > > > > >> > >>
> how
> > > > > > > > > > >> > >>
> >>
> > > > compressed
> > > > > message sets
> > > > > > are
> > > > > > > > represented
> > > > > > > > > that would
> > > > > > > > > > >> be
> > > > > > > > > > >> > applicable
> > > > > > > > > > >> > >>
> for
> > > > > > > > > > >> > >>
> >>
> > >
> > > other
> > > > > > > > > > >> > >>
> >>
> > > > request
> > > > > types. Some might
> > > > > > > see this
> > > > > > > > > divergence as
> > > > > > > > > > an
> > > > > > > > > > >> > unfortunate
> > > > > > > > > > >> > >>
> >>
> > >
> > > protocol
> > > > > > > > > > >> > >>
> >>
> > > > deficiency
> > > > > which should be
> > > > > > > fixed;
> > > > > > > > others
> > > > > > > > > might see
> > > > > > > > > > >> it
> > > > > > > > > > >> > as sort of
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > inevitability of needing
> > > > > > to
> > > > > > > > optimize where
> > > > > > > > > it
> > > > > > > > > > counts
> > > > > > > > > > >> > most. I'm
> > > > > > > > > > >> > >>
> > > probably
> > > > > > > > > > >> > >>
> >>
> > > > somewhere
> > > > > in between, but
> > > > > > I
> > > > > > > think
> > > > > > > > we
> > > > > > > > > probably all
> > > > > > > > > > >> > share the
> > > > > > > > > > >> > >>
> > > intuition
> > > > > > > > > > >> > >>
> >>
> > >
> > > that
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > > protocol should be
> > > > > > kept
> > > > > > > as
> > > > > > > > consistent as
> > > > > > > > > > >> possible.
> > > > > > > > > > >> > With that
> > > > > > > > > > >> > >>
> in
> > > > > > > > > > >> > >>
> >>
> > >
> > > mind,
> > > > > > > > > > >> > >>
> >>
> > > > here are a
> > > > > few comments:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > 1. One
> > > > > thing I found a
> > > > > > > little odd
> > > > > > > > when
> > > > > > > > > reading the
> > > > > > > > > > >> > current
> > > > > > > > > > >> > >>
> > > proposal is
> > > > > > > > > > >> > >>
> >>
> > >
> > > that
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > headers
> > > > > are both
> > > > > > > represented
> > > > > > > > as an
> > > > > > > > > array of
> > > > > > > > > > >> bytes
> > > > > > > > > > >> > and as an
> > > > > > > > > > >> > >>
> > array
> > > > > > > > > > >> > >>
> >>
> > >
> > > of
> > > > > > > > > > >> > >>
> >>
> > > > key/value
> > > > > pairs. I'd
> > > > > > > probably
> > > > > > > > suggest
> > > > > > > > > something
> > > > > > > > > > like
> > > > > > > > > > >> > this:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Headers =>
> > > > > [HeaderKey
> > > > > > > HeaderValue]
> > > > > > > > > > >> > >>
> >>
> > > > HeaderKey
> > > > > => String
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > HeaderValue => Bytes
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > An array
> > > > in
> > > > > the Kafka
> > > > > > > protocol is
> > > > > > > > > represented as a
> > > > > > > > > > >> > 4-byte
> integer
> > > > > > > > > > >> > >>
> >>
> > > > indicating
> > > > > the number of
> > > > > > > elements
> > > > > > > > in the
> > > > > > > > > array
> > > > > > > > > > >> > followed by
> > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > serialization of the
> > > > > > > elements.
> > > > > > > > Unless I'm
> > > > > > > > > > >> >
> > > misunderstanding, what
> > > > > > > > > > >> > >>
> you
> > > > > > > > > > >> > >>
> >>
> > >
> > > have
> > > > > > > > > > >> > >>
> >>
> > > > instead is
> > > > > the total size
> > > > > > > of the
> > > > > > > > headers in
> > > > > > > > > bytes
> > > > > > > > > > >> > followed by
> > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > elements.
> > > > > > > > > > >> > >>
> >>
> > > > I'm not
> > > > > sure I see any
> > > > > > > reason for
> > > > > > > > this
> > > > > > > > > > >> inconsistency.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > 2. In
> > > > > KIP-98, we've
> > > > > > > introduced
> > > > > > > > > variable-length
> > > > > > > > > > >> integer
> > > > > > > > > > >> > fields.
> > > > > > > > > > >> > >>
> >>
> > >
> > > Effectively,
> > > > > > > > > > >> > >>
> >>
> > > > we've
> > > > > enriched (or
> > > > > > > "complicated"
> > > > > > > > as Jay
> > > > > > > > > might say
> > > > > > > > > > ;)
> > > > > > > > > > >> > the protocol
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > specification to include
> > > > > > the
> > > > > > > > following
> > > > > > > > > types:
> > > > > > > > > > >> VarInt,
> > > > > > > > > > >> > VarLong,
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > UnsignedVarInt and
> > > > > > > > UnsignedVarLong.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Along with
> > > > > these
> > > > > > > primitives, we
> > > > > > > > could
> > > > > > > > > introduce
> > > > > > > > > > the
> > > > > > > > > > >> > following
> > > > > > > > > > >> > >>
> > > types:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > VarSizeArray =>
> > > > > > > NumberOfItems
> > > > > > > > Item1 Item2
> > > > > > > > > .. ItemN
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > NumberOfItems =>
> > > > > > > UnsignedVarInt
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > VarSizeNullableArray =>
> > > > > > > > NumberOfItemsOrNull
> > > > > > > > > Item1
> > > > > > > > > > >> > Item2 ..
> ItemN
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > NumberOfItemsOrNull =>
> > > > > > > VarInt
> > > > > > > > (-1 means
> > > > > > > > > null)
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > And
> > > > > similarly for the
> > > > > > > `String`
> > > > > > > > and `Bytes`
> > > > > > > > > types.
> > > > > > > > > > >> > These types
> > > > > > > > > > >> > >>
> can
> > > save
> > > > > > > > > > >> > >>
> >>
> > >
> > > a
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > considerable amount of
> > > > > > > space in
> > > > > > > > this
> > > > > > > > > proposal
> > > > > > > > > > >> because
> > > > > > > > > > >> > they can
> > > > > > > > > > >> > >>
> be
> > > used
> > > > > > > > > > >> > >>
> >>
> > >
> > > for
> > > > > > > > > > >> > >>
> >>
> > > > both the
> > > > > number of headers
> > > > > > > > included in the
> > > > > > > > > message
> > > > > > > > > > >> and
> > > > > > > > > > >> > the
> > > > > > > > > > >> > >>
> > > lengths of
> > > > > > > > > > >> > >>
> >>
> > >
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > header
> > > > keys
> > > > > and values. We
> > > > > > > could
> > > > > > > > do this
> > > > > > > > > instead:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Headers =>
> > > > > > > VarSizeArray[HeaderKey
> > > > > > > > > HeaderValue]
> > > > > > > > > > >> > >>
> >>
> > >
> > > > HeaderKey
> > > > > =>
> > > > > > VarSizeString
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > HeaderValue =>
> > > > > > > VarSizeBytes
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > Combining
> > > > > the savings from
> > > > > > > the
> > > > > > > > use of
> > > > > > > > > variable
> > > > > > > > > > >> length
> > > > > > > > > > >> > fields, the
> > > > > > > > > > >> > >>
> >>
> > >
> > > benefit
> > > > > > > > > > >> > >>
> >>
> > > > of using
> > > > > the attributes to
> > > > > > > > represent null
> > > > > > > > > seems
> > > > > > > > > > >> pretty
> > > > > > > > > > >> > small.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > 3.
> > > > > Whichever way we go
> > > > > > > (whether
> > > > > > > > we use the
> > > > > > > > > > >> attributes
> > > > > > > > > > >> > or not), we
> > > > > > > > > > >> > >>
> >>
> > >
> > > should at
> > > > > > > > > > >> > >>
> >>
> > > > least be
> > > > > consistent
> > > > > > between
> > > > > > > this
> > > > > > > > KIP and
> > > > > > > > > KIP-98.
> > > > > > > > > > It
> > > > > > > > > > >> > would be
> very
> > > > > > > > > > >> > >>
> >>
> > >
> > > strange
> > > > > > > > > > >> > >>
> >>
> > > > to have
> > > > two
> > > > > ways to
> > > > > > > represent
> > > > > > > > null values
> > > > > > > > > in the
> > > > > > > > > > >> same
> > > > > > > > > > >> > schema.
> > > > > > > > > > >> > >>
> > > Either
> > > > > > > > > > >> > >>
> >>
> > >
> > > way is
> > > > > > > > > > >> > >>
> >>
> > > > OK with
> > > > me.
> > > > > I think some
> > > > > > > > message-level
> > > > > > > > > > optimizations
> > > > > > > > > > >> > are
> > > > > > > > > > >> > >>
> > > justifiable,
> > > > > > > > > > >> > >>
> >>
> > >
> > > but
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > savings
> > > > > here seem
> > > > > > > minimal (a
> > > > > > > > few bytes
> > > > > > > > > per
> > > > > > > > > > >> > message), so
> > > > > > > > > > >> > >>
> > > maybe it's
> > > > > > > > > > >> > >>
> >>
> > >
> > > not
> > > > > > > > > > >> > >>
> >>
> > > > worth the
> > > > > cost of letting
> > > > > > > the
> > > > > > > > message
> > > > > > > > > diverge even
> > > > > > > > > > >> > further from
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > rest of
> > > > > > > > > > >> > >>
> >>
> > > > the
> > > > > protocol.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > -Jason
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > On Wed,
> > > > Feb
> > > > > 15, 2017 at
> > > > > > > 8:52 AM,
> > > > > > > > radai <
> > > > > > > > > > >> > >>
> > > > > radai.rosenblatt@gmail.com>
> > > > > > > > > > >> > >>
> >>
> > >
> > > wrote:
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > > > > I've
> > > > > trimmed the inline
> > > > > > > > contents as this
> > > > > > > > > mail is
> > > > > > > > > > >> > getting too
> > > > > > > > > > >> > >>
> big
> > > for
> > > > > > > > > > >> > >>
> >>
> > >
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > > apache
> > > > > mailing list
> > > > > > > software to
> > > > > > > > deliver
> > > > > > > > > :-(
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > 1. the
> > > > > important thing
> > > > > > for
> > > > > > > > > interoperability is
> > > > > > > > > > for
> > > > > > > > > > >> > different
> > > > > > > > > > >> > >>
> >>
> > >
> > > "interested
> > > > > > > > > > >> > >>
> >>
> > > > > parties"
> > > > > (plugins, infra
> > > > > > > > layers/wrappers,
> > > > > > > > > > >> user-code)
> > > > > > > > > > >> > to be
> > > > > > > > > > >> > >>
> > able
> > > to
> > > > > > > > > > >> > >>
> >>
> > >
> > > stick
> > > > > > > > > > >> > >>
> >>
> > > > > pieces
> > > > of
> > > > > metadata onto
> > > > > > > msgs
> > > > > > > > without
> > > > > > > > > getting in
> > > > > > > > > > >> each
> > > > > > > > > > >> > other's
> > > > > > > > > > >> > >>
> > way.
> > > a
> > > > > > > > > > >> > >>
> >>
> > >
> > > common
> > > > > > > > > > >> > >>
> >>
> > > > > key
> > > > > scheme (Strings, as
> > > > > > > of the
> > > > > > > > time of
> > > > > > > > > this
> > > > > > > > > > >> > writing?) is
> > all
> > > > > > > > > > >> > >>
> > thats
> > > > > > > > > > >> > >>
> >>
> > >
> > > required
> > > > > > > > > > >> > >>
> >>
> > > > > for
> > > > that.
> > > > > it is assumed
> > > > > > > that
> > > > > > > > the other end
> > > > > > > > > > >> > interested
> in
> > > any
> > > > > > > > > > >> > >>
> > such
> > > > > > > > > > >> > >>
> >>
> > >
> > > piece of
> > > > > > > > > > >> > >>
> >>
> > > > > metadata
> > > > > knows the
> > > > > > > encoding,
> > > > > > > > and byte[]
> > > > > > > > > provides
> > > > > > > > > > >> for
> > > > > > > > > > >> > the most
> > > > > > > > > > >> > >>
> >>
> > >
> > > flexibility.
> > > > > > > > > > >> > >>
> >>
> > > > > i
> > > > believe
> > > > > this is the
> > > > > > same
> > > > > > > > logic behind
> > > > > > > > > core
> > > > > > > > > > kafka
> > > > > > > > > > >> > being
> > > > > > > > > > >> > >>
> >>
> > >
> > > byte[]/byte[]
> > > > -
> > > > > > > > > > >> > >>
> >>
> > > > > Strings
> > > > > are more
> > > > > > "usable"
> > > > > > > but
> > > > > > > > bytes are
> > > > > > > > > flexible
> > > > > > > > > > >> and
> > > > > > > > > > >> > so were
> > > > > > > > > > >> > >>
> > > chosen.
> > > > > > > > > > >> > >>
> >>
> > > > > Also -
> > > > > core kafka doesnt
> > > > > > > even
> > > > > > > > do that
> > > > > > > > > good of a
> > > > > > > > > > >> job
> > > > > > > > > > >> > on
> > > > > > > > > > >> > >>
> > > usability of
> > > > > > > > > > >> > >>
> >>
> > >
> > > the
> > > > > > > > > > >> > >>
> >>
> > > > > payload
> > > > > (example - i
> > > > > > have
> > > > > > > to
> > > > > > > > specify the
> > > > > > > > > nop
> > > > > > > > > > >> byte[]
> > > > > > > > > > >> > "decoders"
> > > > > > > > > > >> > >>
> >>
> > >
> > > explicitly
> > > > > > > > > > >> > >>
> >>
> > > > > in
> > > > conf),
> > > > > and again
> > > > > > > sacrificies
> > > > > > > > usability
> > > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > >> > sake of
> > > > > > > > > > >> > >>
> >>
> > >
> > > performance
> > > > (no
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > convenient single-record
> > > > > > > > processing as
> > > > > > > > > poll is a
> > > > > > > > > > >> > batch, lots
> of
> > > > > > > > > > >> > >>
> >>
> > >
> > > obscure
> > > > > > > > > > >> > >>
> >>
> > > > > little
> > > > > config details
> > > > > > > exposing
> > > > > > > > internals
> > > > > > > > > of the
> > > > > > > > > > >> > batching
> > > > > > > > > > >> > >>
> > > mechanism,
> > > > > > > > > > >> > >>
> >>
> > >
> > > etc)
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > this is
> > > > > also why i
> > > > > > really
> > > > > > > > dislike the
> > > > > > > > > idea of a
> > > > > > > > > > >> > "type
> system"
> > > > > > > > > > >> > >>
> for
> > > > > > > > > > >> > >>
> >>
> > >
> > > header
> > > > > > > > > > >> > >>
> >>
> > > > > values,
> > > > > it further
> > > > > > > degrades the
> > > > > > > > > usability, adds
> > > > > > > > > > >> > complexity
> and
> > > > > > > > > > >> > >>
> > will
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > eventually get in
> > > > > > > people's way,
> > > > > > > > also, it
> > > > > > > > > would
> > > > > > > > > > be
> > > > > > > > > > >> > the 2nd/3rd
> > > > > > > > > > >> > >>
> >>
> > >
> > > home-group
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > serialization mechanism
> > > > > > > in core
> > > > > > > > kafka
> > > > > > > > > (counting
> > > > > > > > > > 2
> > > > > > > > > > >> > iterations
> > > > > > > > > > >> > >>
> of
> > > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > "type
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > definition DSL")
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > 2. this
> > > > > is an
> > > > > > > implementation
> > > > > > > > detail, and
> > > > > > > > > not
> > > > > > > > > > even
> > > > > > > > > > >> a
> > > > > > > > > > >> > very "user
> > > > > > > > > > >> > >>
> >>
> > >
> > > facing" one?
> > > > > > > > > > >> > >>
> >>
> > > > > to the
> > > > > best of my
> > > > > > > understanding
> > > > > > > > the vote
> > > > > > > > > process
> > > > > > > > > > >> is
> > > > > > > > > > >> > on proposed
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > API/behaviour. also -
> > > > > > > since
> > > > > > > > we're willing
> > > > > > > > > to go
> > > > > > > > > > >> with
> > > > > > > > > > >> > strings
> > > > > > > > > > >> > >>
> > just
> > > > > > > > > > >> > >>
> >>
> > >
> > > serialize
> > > > > > > > > > >> > >>
> >>
> > > > > a
> > > > 0-sized
> > > > > header blob
> > > > > > and
> > > > > > > IIUC
> > > > > > > > you dont
> > > > > > > > > need any
> > > > > > > > > > >> > optionals
> > > > > > > > > > >> > >>
> > > anymore.
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > 3. yes,
> > > > > we can :-)
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > On Tue,
> > > > > Feb 14, 2017 at
> > > > > > > 11:56
> > > > > > > > PM, Michael
> > > > > > > > > > Pearce <
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > Michael.Pearce@ig.com>
> > > > > > > > > > >> > >>
> >>
> > > > > wrote:
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > Hi
> > > > Jay,
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > 1)
> > > > > There was some
> > > > > > > initial
> > > > > > > > debate on the
> > > > > > > > > value
> > > > > > > > > > >> > part, as
> youll
> > > > > > > > > > >> > >>
> > note
> > > > > > > > > > >> > >>
> >>
> > >
> > > String,
> > > > > > > > > > >> > >>
> >>
> > > > > > String
> > > > > headers were
> > > > > > > > discounted early
> > > > > > > > > on. The
> > > > > > > > > > >> > reason for
> > this
> > > > > > > > > > >> > >>
> is
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > flexibility
> > > > > > > > > > >> > >>
> >>
> > > > > > and
> > > > > keeping in line
> > > > > > > with the
> > > > > > > > > flexibility of
> > > > > > > > > > key,
> > > > > > > > > > >> > value of the
> > > > > > > > > > >> > >>
> >>
> > >
> > > message
> > > > > > > > > > >> > >>
> >>
> > > > > > object
> > > > > itself. I don’t
> > > > > > > think
> > > > > > > > it takes
> > > > > > > > > away
> > > > > > > > > > from
> > > > > > > > > > >> an
> > > > > > > > > > >> > ecosystem
> > > > > > > > > > >> > >>
> as
> > > > > > > > > > >> > >>
> >>
> > >
> > > each
> > > > > > > > > > >> > >>
> >>
> > > > > plugin
> > > > > > > > > > >> > >>
> >>
> > > > > > will
> > > > > care for their
> > > > > > own
> > > > > > > key,
> > > > > > > > this way
> > > > > > > > > ints,
> > > > > > > > > > >> > booleans ,
> > > exotic
> > > > > > > > > > >> > >>
> >>
> > >
> > > custom
> > > > > > > > > > >> > >>
> >>
> > > > > binary
> > > > > > > > > > >> > >>
> >>
> > > > > > can
> > > > all
> > > > > be catered
> > > > > > for=.
> > > > > > > > > > >> > >>
> >>
> > > > > > a. If
> > > > > you really
> > > > > > wanted
> > > > > > > to
> > > > > > > > push for a
> > > > > > > > > typed
> > > > > > > > > > >> value
> > > > > > > > > > >> > interface,
> > > > > > > > > > >> > >> I
> > > > > > > > > > >> > >>
> >>
> > >
> > > wouldn’t
> > > > > > > > > > >> > >>
> >>
> > > > > > want
> > > > > just String
> > > > > > values
> > > > > > > > supported, but
> > > > > > > > > the the
> > > > > > > > > > >> > primatives
> > > > > > > > > > >> > >>
> > plus
> > > > > > > > > > >> > >>
> >>
> > >
> > > string and
> > > > > > > > > > >> > >>
> >>
> > > > > > also
> > > > > still keeping the
> > > > > > > > ability to have a
> > > > > > > > > > binary
> > > > > > > > > > >> > for custom
> > > > > > > > > > >> > >>
> >>
> > >
> > > binaries that
> > > > > > > > > > >> > >>
> >>
> > > > > > some
> > > > > organisations may
> > > > > > > have.
> > > > > > > > > > >> > >>
> >>
> > > > > > i. I
> > > > > have written this
> > > > > > > slight
> > > > > > > > > alternative
> > > > > > > > > > here,
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > https://cwiki.apache.org/
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > >> >
> > > 82+-+Add+Record+Headers+-+
> > > > Typed
> > > > > > > > > > >> > >>
> >>
> > > > > > ii.
> > > > > Essentially the
> > > > > > > value
> > > > > > > > bytes, has a
> > > > > > > > > leading
> > > > > > > > > > >> > byte
> overhead.
> > > > > > > > > > >> > >>
> >>
> > > > > > 1.
> > > > > This tells you
> > > > > > what
> > > > > > > type
> > > > > > > > the value
> > > > > > > > > is,
> > > > > > > > > > >> before
> > > > > > > > > > >> > reading
> > > > > > > > > > >> > >>
> the
> > > rest
> > > > > > > > > > >> > >>
> >>
> > >
> > > of the
> > > > > > > > > > >> > >>
> >>
> > > > > > bytes,
> > > > > allowing
> > > > > > > > >
> > > serialisation/deserialization
> > > > > > > > > > to
> > > > > > > > > > >> > and from the
> > > > > > > > > > >> > >>
> >>
> > >
> > > primitives,
> > > > > > > > > > >> > >>
> >>
> > > > > > string
> > > > > and byte[].
> > > > > > This
> > > > > > > is
> > > > > > > > akin to some
> > > > > > > > > other
> > > > > > > > > > >> > messaging
> > > > > > > > > > >> > >>
> > > systems.
> > > > > > > > > > >> > >>
> >>
> > > > > > 2) We
> > > > > are making it
> > > > > > > optional,
> > > > > > > > so that
> > > > > > > > > for
> > > > > > > > > > those
> > > > > > > > > > >> > not wanting
> > > > > > > > > > >> > >>
> >>
> > >
> > > headers have
> > > > > > > > > > >> > >>
> >>
> > > > > 0
> > > > > > > > > > >> > >>
> >>
> > > > > > bytes
> > > > > overhead (think
> > > > > > > of it
> > > > > > > > as a feature
> > > > > > > > > > flag),
> > > > > > > > > > >> I
> > > > > > > > > > >> > don’t
> > > > > > > > > > >> > >>
> > > think this
> > > > > > > > > > >> > >>
> >>
> > >
> > > is
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > complex, especially if
> > > > > > > > comparing to
> > > > > > > > > changes
> > > > > > > > > > >> > proposed in
> > > > > > > > > > >> > >>
> > > other kips
> > > > > > > > > > >> > >>
> >>
> > >
> > > like
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > kip-98.
> > > > > > > > > > >> > >>
> >>
> > > > > > a. If
> > > > > you really
> > > > > > really
> > > > > > > don’t
> > > > > > > > like
> > > > > > > > > this, we
> > > > > > > > > > can
> > > > > > > > > > >> > drop it, but
> > > > > > > > > > >> > >>
> it
> > > > > > > > > > >> > >>
> >>
> > >
> > > would
> > > > > > > > > > >> > >>
> >>
> > > > > mean
> > > > > > > > > > >> > >>
> >>
> > > > > > buying
> > > > > into 4 bytes
> > > > > > > extra
> > > > > > > > overhead for
> > > > > > > > > users
> > > > > > > > > > who
> > > > > > > > > > >> > do not want
> > > > > > > > > > >> > >>
> to
> > > use
> > > > > > > > > > >> > >>
> >>
> > > > > headers.
> > > > > > > > > > >> > >>
> >>
> > > > > > 3) In
> > > > > the summary yes,
> > > > > > > it is
> > > > > > > > at a higher
> > > > > > > > > > level,
> > > > > > > > > > >> > but I think
> > > > > > > > > > >> > >>
> > this
> > > > > > > > > > >> > >>
> >>
> > >
> > > is well
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > documented in the
> > > > > > > proposed
> > > > > > > > changes
> > > > > > > > > section.
> > > > > > > > > > >> > >>
> >>
> > > > > > a.
> > > > > Added getHeaders
> > > > > > > method to
> > > > > > > > > > Producer/Consumer
> > > > > > > > > > >> > record (that
> > > > > > > > > > >> > >>
> is
> > > it)
> > > > > > > > > > >> > >>
> >>
> > > > > > b.
> > > > > We’ve also detailed
> > > > > > > the
> > > > > > > > new Headers
> > > > > > > > > class
> > > > > > > > > > >> that
> > > > > > > > > > >> > this method
> > > > > > > > > > >> > >>
> >>
> > >
> > > returns
> > > > > > > > > > >> > >>
> >>
> > > > > that
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > encapsulates the
> > > > > > headers
> > > > > > > > protocol and
> > > > > > > > > logic.
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > Best,
> > > > > > > > > > >> > >>
> >>
> > > > > > Mike
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > ==Original questions
> > > > > > > from the
> > > > > > > > vote
> > > > > > > > > thread from
> > > > > > > > > > >> > Jay.==
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > Couple
> > > > > of things I
> > > > > > > think we
> > > > > > > > still need
> > > > > > > > > to work
> > > > > > > > > > >> out:
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > 1.
> > > > I
> > > > > think we agree
> > > > > > > about
> > > > > > > > the key,
> > > > > > > > > but I
> > > > > > > > > > >> think
> > > > > > > > > > >> > we haven't
> > > > > > > > > > >> > >>
> >>
> > >
> > > talked about
> > > > > > > > > > >> > >>
> >>
> > > > > > the
> > > > > value yet. I
> > > > > > > think if
> > > > > > > > our goal
> > > > > > > > > is an
> > > > > > > > > > open
> > > > > > > > > > >> > ecosystem
> > > > > > > > > > >> > >>
> of
> > > these
> > > > > > > > > > >> > >>
> >>
> > > > > header
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > spread across many
> > > > > > > plugins
> > > > > > > > from many
> > > > > > > > > > systems
> > > > > > > > > > >> we
> > > > > > > > > > >> > should
> > > > > > > > > > >> > >>
> > > consider
> > > > > > > > > > >> > >>
> >>
> > >
> > > making
> > > > > > > > > > >> > >>
> >>
> > > > > > this
> > > > > > > > > > >> > >>
> >>
> > > > > > a
> > > > > string as well so
> > > > > > > it can
> > > > > > > > be
> > > > > > > > > printed, set
> > > > > > > > > > >> via
> > > > > > > > > > >> > a UI, set
> > > > > > > > > > >> > >>
> in
> > > > > > > > > > >> > >>
> >>
> > >
> > > config,
> > > > > > > > > > >> > >>
> >>
> > > > > etc.
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > Basically
> > > > > > encouraging
> > > > > > > > pluggable
> > > > > > > > > > serialization
> > > > > > > > > > >> > formats
> > > > > > > > > > >> > >>
> > here
> > > will
> > > > > > > > > > >> > >>
> >>
> > >
> > > lead
> > > > > > > > > > >> > >>
> >>
> > > > > to
> > > > > > > > > > >> > >>
> >>
> > > > > > a
> > > > > > > > > > >> > >>
> >>
> > > > > > bit
> > > > > of a tower of
> > > > > > > babel.
> > > > > > > > > > >> > >>
> >>
> > > > > > 2.
> > > > > This proposal
> > > > > > > still
> > > > > > > > includes a
> > > > > > > > > pretty
> > > > > > > > > > big
> > > > > > > > > > >> > change to
> our
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > serialization
> > > > > > > > > > >> > >>
> >>
> > > > > > and
> > > > > protocol
> > > > > > > definition
> > > > > > > > layer.
> > > > > > > > > Essentially
> > > > > > > > > > >> it is
> > > > > > > > > > >> > >>
> > > introducing an
> > > > > > > > > > >> > >>
> >>
> > > > > optional
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > type, where the
> > > > > > > format is
> > > > > > > > data
> > > > > > > > > dependent. I
> > > > > > > > > > >> > think this
> is
> > > > > > > > > > >> > >>
> >>
> > >
> > > actually a
> > > > > > > > > > >> > >>
> >>
> > > > > big
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > change though it
> > > > > > > doesn't
> > > > > > > > seem like
> > > > > > > > > it. It
> > > > > > > > > > >> means
> > > > > > > > > > >> > you can no
> > > > > > > > > > >> > >>
> >>
> > >
> > > longer
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > specify
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > this
> > > > > type with our
> > > > > > > type
> > > > > > > > definition
> > > > > > > > > DSL, and
> > > > > > > > > > >> > likewise it
> > > > > > > > > > >> > >>
> > > requires
> > > > > > > > > > >> > >>
> >>
> > > > > custom
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > handling in client
> > > > > > > libs.
> > > > > > > > This isn't
> > > > > > > > > a huge
> > > > > > > > > > >> > thing, since
> > > > > > > > > > >> > >>
> the
> > > > > > > > > > >> > >>
> >>
> > >
> > > Record
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > definition is
> > > > > > custom
> > > > > > > > anyway, but I
> > > > > > > > > think
> > > > > > > > > > this
> > > > > > > > > > >> > kind of
> > > > > > > > > > >> > >>
> > > protocol
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > inconsistency is
> > > > > > very
> > > > > > > > non-desirable
> > > > > > > > > and
> > > > > > > > > > ties
> > > > > > > > > > >> > you to
> > > > > > > > > > >> > >>
> > > hand-coding
> > > > > > > > > > >> > >>
> >>
> > > > > things.
> > > > > > > > > > >> > >>
> >>
> > > > > > I
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > think the type
> > > > > > should
> > > > > > > > instead by [Key
> > > > > > > > > > Value]
> > > > > > > > > > >> in
> > > > > > > > > > >> > our BNF,
> > > > > > > > > > >> > >>
> > where
> > > > > > > > > > >> > >>
> >>
> > >
> > > key and
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > value are both
> > > > > > short
> > > > > > > > strings as used
> > > > > > > > > > >> elsewhere.
> > > > > > > > > > >> > This
> > > > > > > > > > >> > >>
> > > brings it
> > > > > > > > > > >> > >>
> >>
> > >
> > > in line
> > > > > > > > > > >> > >>
> >>
> > > > > > with
> > > > > > > > > > >> > >>
> >>
> > > > > > the
> > > > > rest of the
> > > > > > > protocol.
> > > > > > > > > > >> > >>
> >>
> > > > > > 3.
> > > > > Could we get
> > > > > > more
> > > > > > > > specific about
> > > > > > > > > the
> > > > > > > > > > exact
> > > > > > > > > > >> > Java API
> > > > > > > > > > >> > >>
> > > change to
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > ProducerRecord,
> > > > > > > > ConsumerRecord,
> > > > > > > > > Record,
> > > > > > > > > > etc?
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > > > -Jay
> > > > > > > > > > >> > >>
> >>
> > > > > >
> > > > > > > > > > >> > >>
> >>
> > > > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> >>
> > >
> > > The
> > > > information
> > > > > contained in
> > > > > > > this
> > > > > > > > email is
> > > > > > > > > strictly
> > > > > > > > > > >> > confidential
> > and
> > > > > > > > > > >> > >>
> for
> > > > > > > > > > >> > >>
> >>
> > >
> > > the use of the
> > > > > addressee only,
> > > > > > > unless
> > > > > > > > otherwise
> > > > > > > > > > >> indicated.
> > > > > > > > > > >> > If you
> > > > > > > > > > >> > >>
> are
> > > not
> > > > > > > > > > >> > >>
> >>
> > >
> > > the intended
> > > > > recipient, please
> > > > > > > do not
> > > > > > > > read,
> > > > > > > > > copy, use
> > > > > > > > > > or
> > > > > > > > > > >> > disclose to
> > > > > > > > > > >> > >>
> > > others
> > > > > > > > > > >> > >>
> >>
> > >
> > > this message
> > > > or
> > > > > any
> > > > > > attachment.
> > > > > > > > Please also
> > > > > > > > > notify the
> > > > > > > > > > >> > sender by
> > > > > > > > > > >> > >>
> > > replying
> > > > > > > > > > >> > >>
> >>
> > >
> > > to this email
> > > > > or by telephone
> > > > > > > > (+44(020 7896
> > > > > > > > > 0011) and
> > > > > > > > > > >> then
> > > > > > > > > > >> > delete
> > > > > > > > > > >> > >>
> the
> > > email
> > > > > > > > > > >> > >>
> >>
> > >
> > > and any copies
> > > > > of it.
> > > > > > Opinions,
> > > > > > > > conclusion
> > > > > > > > > (etc) that
> > > > > > > > > > do
> > > > > > > > > > >> > not relate
> > > > > > > > > > >> > >>
> to
> > > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > official
> > > > > business of this
> > > > > > > company
> > > > > > > > shall be
> > > > > > > > > understood
> > > > > > > > > > as
> > > > > > > > > > >> > neither
> > > > > > > > > > >> > >>
> > > given nor
> > > > > > > > > > >> > >>
> >>
> > >
> > > endorsed by
> > > > it.
> > > > > IG is a
> > > > > > trading
> > > > > > > name
> > > > > > > > of IG
> > > > > > > > > Markets
> > > > > > > > > > >> Limited
> > > > > > > > > > >> > (a company
> > > > > > > > > > >> > >>
> >>
> > >
> > > registered in
> > > > > England and
> > > > > > Wales,
> > > > > > > > company number
> > > > > > > > > > >> 04008957)
> > > > > > > > > > >> > and IG
> > > > > > > > > > >> > >>
> > Index
> > > > > > > > > > >> > >>
> >>
> > >
> > > Limited (a
> > > > > company registered
> > > > > > in
> > > > > > > > England and
> > > > > > > > > Wales,
> > > > > > > > > > >> > company
> number
> > > > > > > > > > >> > >>
> >>
> > >
> > > 01190902).
> > > > > Registered address
> > > > > > at
> > > > > > > > Cannon Bridge
> > > > > > > > > House,
> > > > > > > > > > 25
> > > > > > > > > > >> > Dowgate
> > > > > > > > > > >> > >>
> > Hill,
> > > > > > > > > > >> > >>
> >>
> > >
> > > London EC4R
> > > > > 2YA. Both IG
> > > > > > Markets
> > > > > > > > Limited
> > > > > > > > > (register
> > > > > > > > > > >> number
> > > > > > > > > > >> > 195355)
> > > > > > > > > > >> > >>
> and
> > > IG
> > > > > > > > > > >> > >>
> >>
> > >
> > > Index Limited
> > > > > (register number
> > > > > > > > 114059) are
> > > > > > > > > authorised
> > > > > > > > > > >> and
> > > > > > > > > > >> > regulated
> > > > > > > > > > >> > >>
> by
> > > the
> > > > > > > > > > >> > >>
> >>
> > >
> > > Financial
> > > > > Conduct Authority.
> > > > > > > > > > >> > >>
> >>
> > >
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>
> --
> > > > > > > > > > >> > >>
> > > Nacho - Ignacio
> > > > > Solis -
> > > > > > > isolis@igso.net
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > The
> > information
> > > contained in
> > > > > this email is
> > > > > > > strictly
> > > > > > > > > confidential and
> > > > > > > > > > for
> > > > > > > > > > >> > the use of
> the
> > > addressee only,
> > > > > unless
> > > > > > otherwise
> > > > > > > > indicated. If
> > > > > > > > > you are
> > > > > > > > > > >> not
> > > > > > > > > > >> > the intended
> > > recipient, please
> > > > > do not read,
> > > > > > > copy, use
> > > > > > > > or
> > > > > > > > > disclose to
> > > > > > > > > > >> others
> > > > > > > > > > >> > this message
> > or
> > > any
> > > > attachment.
> > > > > Please also
> > > > > > > notify
> > > > > > > > the sender by
> > > > > > > > > > >> replying
> > > > > > > > > > >> > to this
> email
> > > or by telephone
> > > > > (+44(020 7896
> > > > > > > 0011) and
> > > > > > > > then
> > > > > > > > > delete the
> > > > > > > > > > >> email
> > > > > > > > > > >> > and any
> copies
> > > of it.
> > > > Opinions,
> > > > > conclusion
> > > > > > > (etc) that
> > > > > > > > do not
> > > > > > > > > relate to
> > > > > > > > > > >> the
> > > > > > > > > > >> > official
> > > business of this
> > > > > company shall be
> > > > > > > understood
> > > > > > > > as
> > > > > > > > > neither given
> > > > > > > > > > >> nor
> > > > > > > > > > >> > endorsed by
> > it.
> > > IG is a
> > > > trading
> > > > > name of IG
> > > > > > > Markets
> > > > > > > > Limited (a
> > > > > > > > > company
> > > > > > > > > > >> > registered
> in
> > > England and
> > > > > Wales, company
> > > > > > number
> > > > > > > > 04008957) and
> > > > > > > > > IG Index
> > > > > > > > > > >> > Limited (a
> > > company registered
> > > > > in England and
> > > > > > > Wales,
> > > > > > > > company
> > > > > > > > > number
> > > > > > > > > > >> > 01190902).
> > > Registered address
> > > > > at Cannon Bridge
> > > > > > > House,
> > > > > > > > 25
> > > > > > > > > Dowgate Hill,
> > > > > > > > > > >> > London EC4R
> > > 2YA. Both IG
> > > > > Markets Limited
> > > > > > > (register
> > > > > > > > number
> > > > > > > > > 195355) and
> > > > > > > > > > IG
> > > > > > > > > > >> > Index
> Limited
> > > (register number
> > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated by
> > > > > > > > > > >> the
> > > > > > > > > > >> > Financial
> > > Conduct Authority.
> > > > > > > > > > >> >
> > > > > > > > > > >> The
> information
> > > contained in
> > > > this
> > > > > email is
> > > > > > > strictly
> > > > > > > > confidential
> > > > > > > > > and for
> > > > > > > > > > >> the use of the
> > > addressee only,
> > > > > unless otherwise
> > > > > > > > indicated. If you
> > > > > > > > > are
> > > > > > > > > > not
> > > > > > > > > > >> the intended
> > > recipient, please
> > > > do
> > > > > not read,
> > > > > > copy,
> > > > > > > use
> > > > > > > > or disclose
> > > > > > > > > to
> > > > > > > > > > others
> > > > > > > > > > >> this message
> or
> > > any attachment.
> > > > > Please also
> > > > > > > notify the
> > > > > > > > sender by
> > > > > > > > > > replying
> > > > > > > > > > >> to this email
> or
> > > by telephone
> > > > > (+44(020 7896
> > > > > > 0011)
> > > > > > > and
> > > > > > > > then delete
> > > > > > > > > the
> > > > > > > > > > email
> > > > > > > > > > >> and any copies
> > of
> > > it. Opinions,
> > > > > conclusion (etc)
> > > > > > > that
> > > > > > > > do not
> > > > > > > > > relate to
> > > > > > > > > > the
> > > > > > > > > > >> official
> > business
> > > of this
> > > > company
> > > > > shall be
> > > > > > > understood
> > > > > > > > as neither
> > > > > > > > > given
> > > > > > > > > > nor
> > > > > > > > > > >> endorsed by
> it.
> > > IG is a trading
> > > > > name of IG
> > > > > > Markets
> > > > > > > > Limited (a
> > > > > > > > > company
> > > > > > > > > > >> registered in
> > > England and Wales,
> > > > > company number
> > > > > > > > 04008957) and IG
> > > > > > > > > Index
> > > > > > > > > > >> Limited (a
> > > company registered in
> > > > > England and
> > > > > > > Wales,
> > > > > > > > company number
> > > > > > > > > > >> 01190902).
> > > Registered address at
> > > > > Cannon Bridge
> > > > > > > House,
> > > > > > > > 25 Dowgate
> > > > > > > > > Hill,
> > > > > > > > > > >> London EC4R
> 2YA.
> > > Both IG Markets
> > > > > Limited
> > > > > > (register
> > > > > > > > number 195355)
> > > > > > > > > and IG
> > > > > > > > > > >> Index Limited
> > > (register number
> > > > > 114059) are
> > > > > > > authorised
> > > > > > > > and
> > > > > > > > > regulated by
> > > > > > > > > > the
> > > > > > > > > > >> Financial
> > Conduct
> > > Authority.
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > The information
> > > contained in this
> > > > > email is strictly
> > > > > > > > confidential and
> > > > > > > > > for the use of the
> > > addressee only, unless
> > > > > otherwise
> > > > > > > indicated.
> > > > > > > > If you are
> > > > > > > > > not the intended
> > > recipient, please do not
> > > > > read, copy, use
> > > > > > > or
> > > > > > > > disclose to
> > > > > > > > > others this message or
> > any
> > > attachment.
> > > > > Please also notify
> > > > > > > the
> > > > > > > > sender by
> > > > > > > > > replying to this email
> or
> > > by telephone
> > > > > (+44(020 7896
> > > > > > 0011)
> > > > > > > and
> > > > > > > > then delete
> > > > > > > > > the email and any
> copies
> > > of it. Opinions,
> > > > > conclusion
> > > > > > (etc)
> > > > > > > that
> > > > > > > > do not
> > > > > > > > > relate to the official
> > > business of this
> > > > > company shall be
> > > > > > > > understood as
> > > > > > > > > neither given nor
> > endorsed
> > > by it. IG is a
> > > > > trading name of
> > > > > > > IG
> > > > > > > > Markets
> > > > > > > > > Limited (a company
> > > registered in England
> > > > > and Wales,
> > > > > > company
> > > > > > > > number
> > > > > > > > > 04008957) and IG Index
> > > Limited (a company
> > > > > registered in
> > > > > > > England
> > > > > > > > and Wales,
> > > > > > > > > company number
> 01190902).
> > > Registered
> > > > > address at Cannon
> > > > > > > Bridge
> > > > > > > > House, 25
> > > > > > > > > Dowgate Hill, London
> EC4R
> > > 2YA. Both IG
> > > > > Markets Limited
> > > > > > > (register
> > > > > > > > number
> > > > > > > > > 195355) and IG Index
> > > Limited (register
> > > > > number 114059) are
> > > > > > > > authorised and
> > > > > > > > > regulated by the
> > Financial
> > > Conduct
> > > > > Authority.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > The information contained in
> > > this email is
> > > > > strictly
> > > > > > confidential
> > > > > > > and
> > > > > > > > for the use of the addressee
> only,
> > > unless otherwise
> > > > > indicated. If
> > > > > > > you are
> > > > > > > > not the intended recipient,
> please
> > > do not read,
> > > > > copy, use or
> > > > > > > disclose to
> > > > > > > > others this message or any
> > > attachment. Please also
> > > > > notify the
> > > > > > sender
> > > > > > > by
> > > > > > > > replying to this email or by
> > > telephone (+44(020
> > > > 7896
> > > > > 0011) and then
> > > > > > > delete
> > > > > > > > the email and any copies of it.
> > > Opinions,
> > > > conclusion
> > > > > (etc) that do
> > > > > > > not
> > > > > > > > relate to the official business
> of
> > > this company
> > > > > shall be understood
> > > > > > > as
> > > > > > > > neither given nor endorsed by it.
> > IG
> > > is a trading
> > > > > name of IG
> > > > > > Markets
> > > > > > > > Limited (a company registered in
> > > England and Wales,
> > > > > company number
> > > > > > > > 04008957) and IG Index Limited (a
> > > company
> > > > registered
> > > > > in England and
> > > > > > > Wales,
> > > > > > > > company number 01190902).
> > Registered
> > > address at
> > > > > Cannon Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > > Dowgate Hill, London EC4R 2YA.
> Both
> > > IG Markets
> > > > > Limited (register
> > > > > > > number
> > > > > > > > 195355) and IG Index Limited
> > > (register number
> > > > > 114059) are
> > > > > > authorised
> > > > > > > and
> > > > > > > > regulated by the Financial
> Conduct
> > > Authority.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in this email
> > is
> > > strictly
> > > > > confidential and for
> > > > > > > the use of the addressee only, unless
> > > otherwise
> > > > indicated.
> > > > > If you are not
> > > > > > > the intended recipient, please do not
> > > read, copy, use or
> > > > > disclose to
> > > > > > others
> > > > > > > this message or any attachment. Please
> > > also notify the
> > > > > sender by replying
> > > > > > > to this email or by telephone (+44(020
> > > 7896 0011) and
> > > > then
> > > > > delete the
> > > > > > email
> > > > > > > and any copies of it. Opinions,
> > conclusion
> > > (etc) that do
> > > > > not relate to
> > > > > > the
> > > > > > > official business of this company shall
> > be
> > > understood as
> > > > > neither given
> > > > > > nor
> > > > > > > endorsed by it. IG is a trading name of
> > IG
> > > Markets
> > > > Limited
> > > > > (a company
> > > > > > > registered in England and Wales,
> company
> > > number 04008957)
> > > > > and IG Index
> > > > > > > Limited (a company registered in
> England
> > > and Wales,
> > > > > company number
> > > > > > > 01190902). Registered address at Cannon
> > > Bridge House, 25
> > > > > Dowgate Hill,
> > > > > > > London EC4R 2YA. Both IG Markets
> Limited
> > > (register number
> > > > > 195355) and IG
> > > > > > > Index Limited (register number 114059)
> > are
> > > authorised and
> > > > > regulated by
> > > > > > the
> > > > > > > Financial Conduct Authority.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is strictly
> > > confidential and
> > > > > for the use of the addressee only, unless otherwise
> > > indicated. If you are
> > > > > not the intended recipient, please do not read, copy,
> use
> > > or disclose to
> > > > > others this message or any attachment. Please also
> notify
> > > the sender by
> > > > > replying to this email or by telephone (+44(020 7896
> > 0011)
> > > and then
> > > > delete
> > > > > the email and any copies of it. Opinions, conclusion
> > (etc)
> > > that do not
> > > > > relate to the official business of this company shall
> be
> > > understood as
> > > > > neither given nor endorsed by it. IG is a trading name
> of
> > > IG Markets
> > > > > Limited (a company registered in England and Wales,
> > > company number
> > > > > 04008957) and IG Index Limited (a company registered in
> > > England and
> > > > Wales,
> > > > > company number 01190902). Registered address at Cannon
> > > Bridge House, 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > > (register number
> > > > > 195355) and IG Index Limited (register number 114059)
> are
> > > authorised and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jun Rao <ju...@confluent.io>.
Regarding the overhead of skipping the headers in the message. Another way
could be to add the headers at the end of the message after the value part.
This way, there is no need to skip the headers unless you really need it.
This will allow us to represent the headers explicitly as an array of
key/value pairs without the skipping overhead when headers are not needed.
Thanks,
Jun
On Sun, Feb 19, 2017 at 7:13 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Michael,
>
> Here's a link to the branch with the message format changes:
> https://github.com/confluentinc/kafka/tree/exactly-once-message-format.
> The
> class to modify to add headers is EosLogRecord (name will definitely change
> prior to merging in trunk). This branch uses attributes to indicate null
> key and value. I really don't have a strong preference for or against this
> as long as we're consistent. I do feel it's kind of a pity to need the
> signed version of the varint just to be able to indicate null. For headers,
> it's an easier call because we don't actually need to represent null
> headers (using size zero will work fine).
>
> -Jason
>
> On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > We can make it, implement Iterable<Header>, no concerns on this.
> >
> > Ive noted this on the KIP.
> >
> >
> > On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> > Btw Jason, can you share the branch with the VarInts helper/util
> class
> > that KIP-98 will be introducing and also will have the current modified
> > (key and value lengths) message, so we can look at, thanks.
> >
> > On 20/02/2017, 01:11, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > RE “ProducerRecord and ConsumerRecord are immutable currently”. I
> > beg to differ on this, if I set a byte[] as my value of my record, I can
> > later call value() and then alter the bytes in the array. As such it is
> > very much mutable.
> >
> > Headers also will anyhow by nature be mutable on produce, as you
> > add to them.
> >
> >
> > On 20/02/2017, 00:22, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> >
> > Re; having a dedicated Headers object. I think it makes
> sense.
> > Maybe it
> > could implement Iterable<Header>?
> >
> > One thing I'm not too sure about is mutability. Both the
> > ProducerRecord and
> > ConsumerRecord are immutable currently. It would be nice to
> > keep it that
> > way. Perhaps Headers could itself be immutable and passed
> > through the
> > constructors?
> >
> > Just going through how this is, I don’t believe this would
> > provide much
> > > saving. If issue trying to resolve is around object/garbage
> > creation of
> > > Map.Entry objects.
> >
> >
> > It's definitely better because we only allocate the space we
> > will use, but
> > I assume most of the actual memory cost is in the header data
> > itself which
> > we can't do much about. Maybe we can start with the simple
> > array
> > representation and optimize from there after we find it's
> > worthwhile to do
> > so? If we use the Headers object, we can change this later
> > without breaking
> > the API.
> >
> > -Jason
> >
> > On Sun, Feb 19, 2017 at 2:39 PM, radai <
> > radai.rosenblatt@gmail.com> wrote:
> >
> > > "(By the way, doesn't it feel a bit odd that we seem to be
> > designing a
> > > feature which is optimized for people not using?)"
> > >
> > > very. this (i claim :-) ) is the result of very intense
> > opposition to the
> > > usefulness of the feature early on, and not a real design
> > goal
> > >
> > > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > Whilst if having an array of a Header[] would mean
> getting
> > this through,
> > > > im happy changing this, as already done.
> > > >
> > > > Just going through how this is, I don’t believe this
> > would provide much
> > > > saving. If issue trying to resolve is around
> > object/garbage creation of
> > > > Map.Entry objects.
> > > >
> > > > All you’re doing here is replacing the equiv of a
> > HashMap.Node
> > > (Map.Entry)
> > > > which simply holds reference to key and value objects
> with
> > a custom
> > > variant.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 19/02/2017, 20:22, "Michael Pearce" <
> > Michael.Pearce@ig.com> wrote:
> > > >
> > > > On point 1 & 2 Ive updated KIP to show varints (and
> > removed the bit
> > > > flag). (on the assumption KIP 98 is getting the agreement
> > the protocol is
> > > > moving from int32 to varInts as standard)
> > > >
> > > > On point 3 ive updated to use an array of Header
> > class, instead of a
> > > > MultiMap in the Headers class object
> > > >
> > > >
> > > >
> > > > On 19/02/2017, 20:06, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > On points 1 and 2 I agree.
> > > >
> > > > This also affects kip-98, I should expect this
> > resolved before
> > > > that vote also passes. If it is accepted there (I’m
> > assuming this is
> > > > getting discussed on that KIP? As you’re driving the move
> > to VarInts), I
> > > am
> > > > happy to make this KIP will simply follow suit to
> whatever
> > is agreed in
> > > > KIP-98.
> > > >
> > > > On 3) Agreed this is a simpler form, as long as
> no
> > one is
> > > > expecting hashmap lookup performance ( O(1) ), I still
> > would prefer an
> > > > encapsulated class, so if we find that holding it as an
> > array in future
> > > is
> > > > causing some perf issues, the internals are not exposed
> to
> > end users,
> > > > allowing the internal structure to move to a map.
> > > >
> > > > Can we compromise on?
> > > >
> > > > class ConsumerRecord<K, V> {
> > > > K key;
> > > > V value;
> > > > Headers headers;
> > > > }
> > > >
> > > > class Headers {
> > > > Header[] headers;
> > > >
> > > > add(String key, byte[] value)
> > > > Collection<byte[]> get(String key)
> > > > }
> > > >
> > > > class Header {
> > > > String key;
> > > > byte[] value;
> > > > }
> > > >
> > > >
> > > >
> > > > On 19/02/2017, 18:54, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > >
> > > > >
> > > > > headers dont "leak" into application code.
> > they are useful
> > > > to application
> > > > > code as well.
> > > >
> > > >
> > > > This is exactly what I have been trying to
> get
> > at. The use
> > > > cases documented
> > > > here are middleware:
> > > > https://cwiki.apache.org/
> > confluence/display/KAFKA/A+
> > > > Case+for+Kafka+Headers.
> > > > If headers are intended for the application
> > space as well,
> > > the
> > > > document
> > > > should be updated accordingly so that it is
> an
> > explicit
> > > design
> > > > goal and not
> > > > an unfortunate design byproduct. There may be
> > some
> > > > disagreement on whether
> > > > it _should_ be a design goal, but it may not
> > matter much
> > > since
> > > > the current
> > > > interceptor API is probably insufficient for
> > middleware
> > > > applications (I
> > > > haven't thought of a way to do this that
> isn't
> > cumbersome).
> > > >
> > > > That aside, the three unresolved points for
> me
> > are the
> > > > following:
> > > >
> > > > 1. The use of varints. As a one-off for use
> > only with record
> > > > headers, the
> > > > case was weak, but if we are using them
> > throughout the
> > > message
> > > > format, then
> > > > we should do so here as well. The additional
> > complexity is
> > > > minimal and
> > > > early performance testing fully justifies
> > their use.
> > > >
> > > > 2. If varints are used, the case for using
> > attributes to
> > > > indicate null
> > > > headers also becomes weak. We only add one
> > additional byte in
> > > > each message
> > > > if there are no headers. Whatever the case,
> > let us be
> > > > consistent with how
> > > > we treat null keys and values.
> > > >
> > > > 3. We have apparently agreed that the broker
> > will validate
> > > > headers (please
> > > > add this to the KIP). That being the case, I
> > would prefer to
> > > > use Kafka's
> > > > conventional format for arrays. The downside
> > is that it takes
> > > > more work to
> > > > skip over the headers on the consumer, though
> > it's unclear if
> > > > the cost
> > > > would matter in practice. Some concern was
> > previously
> > > > expressed about the
> > > > allocation of maps. An alternative would be
> to
> > use arrays,
> > > i.e.
> > > >
> > > > class ConsumerRecord<K, V> {
> > > > K key;
> > > > V value;
> > > > Header[] headers;
> > > > }
> > > >
> > > > class Header {
> > > > String key;
> > > > byte[] value;
> > > > }
> > > >
> > > > This would work nicely with the conventional
> > array format and
> > > > my guess is
> > > > it would obviate the need do any lazy
> > initialization. If we
> > > > use the map as
> > > > is currently documented, then it is possible
> > with either
> > > > representation to
> > > > slice the headers and initialize them lazily.
> > Either way, it
> > > > might be a
> > > > good idea to use a separate object to
> > represent the headers
> > > in
> > > > case we need
> > > > to modify it in the future in some way.
> > > >
> > > > (By the way, doesn't it feel a bit odd that
> we
> > seem to be
> > > > designing a
> > > > feature which is optimized for people not
> > using?)
> > > >
> > > >
> > > > If we can resolve these points, then at least
> > you will get my
> > > > vote.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > > > radai.rosenblatt@gmail.com> wrote:
> > > >
> > > > > headers dont "leak" into application code.
> > they are useful
> > > > to application
> > > > > code as well.
> > > > > IIUC samze currently has headers "in-V" and
> > would just
> > > > switch over to kafka
> > > > > headers if they exist.
> > > > > im sure plenty of other users of kafka
> would
> > have a use for
> > > > headers.
> > > > > im pretty sure use cases exist around
> > shuffling data
> > > > into/out-of kafka
> > > > > (kafka connect or equivalent) where
> metadata
> > from one end
> > > > could copied over
> > > > > to the other (S3, for example uses http
> > headers for
> > > > user-accessible
> > > > > metadata). it will be kafka client code
> > getting/setting
> > > > those headers. not
> > > > > an interceptor.
> > > > >
> > > > > On Fri, Feb 17, 2017 at 1:41 PM, Michael
> > Pearce <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > For APM single event tracing, need access
> > to the header
> > > at
> > > > the point of
> > > > > > processing on the processing thread.
> > > > > >
> > > > > > As such interceptors will not work/be
> > suitable for these,
> > > > due to the fact
> > > > > > they act on the ConsumerRecords as a
> > batch, before the
> > > > handling thread
> > > > > can
> > > > > > split out and process per message which
> is
> > the point
> > > these
> > > > tools will
> > > > > need
> > > > > > to continue to transaction tracing.
> > > > > >
> > > > > > Like wise tools and infra pieces will
> need
> > access to the
> > > > message outside
> > > > > > the interceptor realm.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 17/02/2017, 21:26, "Jason Gustafson" <
> > > > jason@confluent.io> wrote:
> > > > > >
> > > > > > >
> > > > > > > That’s exactly what we’re doing the
> > headers are a
> > > > slice of bytes,
> > > > > > which
> > > > > > > then gets parsed later if needed,
> or
> > can be parsed
> > > > right away, the
> > > > > > headers
> > > > > > > is part of the protocol, so can
> > still be validated
> > > > if wanted.
> > > > > > > If you had a header count then you
> > would have to go
> > > > through each
> > > > > > header
> > > > > > > key and value length value to work
> > out how much to
> > > > skip to get to
> > > > > > say the
> > > > > > > value or any future component in
> the
> > message after
> > > > the headers.
> > > > > > Having it
> > > > > > > as a byte[] with length value makes
> > this a lot
> > > > easier to skip.
> > > > > >
> > > > > >
> > > > > > So the broker will parse the headers
> > and validate
> > > > them. Good. The
> > > > > only
> > > > > > reason remaining that I can see to
> > leave the headers
> > > > as a byte array
> > > > > > is to
> > > > > > make it easier for the client to skip
> > past them. Are
> > > > we sure this is
> > > > > > not
> > > > > > premature optimization? Are there any
> > performance
> > > > results which show
> > > > > > that
> > > > > > this is worthwhile?
> > > > > >
> > > > > > What’s the issue with exposing a
> > method getHeaders on
> > > > the
> > > > > > producer/consumer
> > > > > > > record? It doesn’t break anything.
> > We don’t need
> > > any
> > > > special
> > > > > version.
> > > > > >
> > > > > >
> > > > > > See my previous explanation. What I
> am
> > trying to
> > > > resist is the
> > > > > headers
> > > > > > becoming a general application-level
> > facility. The
> > > > primary use case
> > > > > as
> > > > > > far
> > > > > > as I can tell is middleware, which is
> > the use case
> > > > that the
> > > > > > interceptors
> > > > > > are providing.
> > > > > >
> > > > > > Current batch consumer model and
> > consumer
> > > interceptors
> > > > don’t work
> > > > > where
> > > > > > > headers need to be acted on at per
> > message level at
> > > > time of
> > > > > > processing,
> > > > > > > very case is APM (the core one),
> > where the header
> > > > value is used to
> > > > > > continue
> > > > > > > tracing.
> > > > > >
> > > > > >
> > > > > > I still don't understand the point
> > about batching.
> > > The
> > > > consumer
> > > > > > records are
> > > > > > exposed as a batch in the consumer
> > interceptor, but
> > > > you can still
> > > > > > iterate
> > > > > > them individually. It is no different
> > for the
> > > consumer
> > > > API itself.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 12:48 PM,
> > Michael Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Re:
> > > > > > >
> > > > > > > “ The point about creation of
> > maps seems
> > > > orthogonal. We can
> > > > > still
> > > > > > > represent
> > > > > > > the headers as a slice of bytes
> > until the time
> > > > it is accessed.”
> > > > > > >
> > > > > > > That’s exactly what we’re doing the
> > headers are a
> > > > slice of bytes,
> > > > > > which
> > > > > > > then gets parsed later if needed,
> or
> > can be parsed
> > > > right away, the
> > > > > > headers
> > > > > > > is part of the protocol, so can
> > still be validated
> > > > if wanted.
> > > > > > >
> > > > > > > If you had a header count then you
> > would have to go
> > > > through each
> > > > > > header
> > > > > > > key and value length value to work
> > out how much to
> > > > skip to get to
> > > > > > say the
> > > > > > > value or any future component in
> the
> > message after
> > > > the headers.
> > > > > > Having it
> > > > > > > as a byte[] with length value makes
> > this a lot
> > > > easier to skip.
> > > > > > >
> > > > > > >
> > > > > > > On 17/02/2017, 20:37, "Michael
> > Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > What’s the issue with exposing
> a
> > method
> > > > getHeaders on the
> > > > > > > producer/consumer record? It
> doesn’t
> > break
> > > anything.
> > > > We don’t need
> > > > > > any
> > > > > > > special version.
> > > > > > >
> > > > > > > Current batch consumer model
> and
> > consumer
> > > > interceptors don’t
> > > > > work
> > > > > > > where headers need to be acted on
> at
> > per message
> > > > level at time of
> > > > > > > processing, very case is APM (the
> > core one), where
> > > > the header value
> > > > > > is used
> > > > > > > to continue tracing. JMS/HTTP etc
> > all expose these,
> > > > without issues.
> > > > > > I would
> > > > > > > NOT want to lock this down to only
> > be usable
> > > > accessible via
> > > > > > interceptors,
> > > > > > > as we’d fail on one of the main
> > goals.
> > > > > > >
> > > > > > > Regards
> > > > > > > Mike
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 17/02/2017, 20:21, "Jason
> > Gustafson" <
> > > > jason@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > The point about creation of
> > maps seems
> > > > orthogonal. We can
> > > > > > still
> > > > > > > represent
> > > > > > > the headers as a slice of
> > bytes until the
> > > > time it is
> > > > > > accessed.
> > > > > > >
> > > > > > >
> > > > > > > > Yes exactly we have
> access
> > to the records
> > > > thus why the
> > > > > > header
> > > > > > > should be
> > > > > > > > accessible via it and not
> > hidden for only
> > > > interceptors to
> > > > > > access.
> > > > > > >
> > > > > > >
> > > > > > > As explained above, the
> > point is to make
> > > the
> > > > intended usage
> > > > > > clear.
> > > > > > > Applications should
> continue
> > to rely on the
> > > > key/value
> > > > > fields
> > > > > > to
> > > > > > > serialize
> > > > > > > their own headers, and it
> > would be more
> > > > ideal if we can
> > > > > avoid
> > > > > > > leaking
> > > > > > > third-party headers into
> > applications. This
> > > > is difficult to
> > > > > > do
> > > > > > > with the
> > > > > > > current interceptors
> because
> > they share the
> > > > record objects
> > > > > > with
> > > > > > > the common
> > > > > > > API. What I had in mind is
> > something like
> > > an
> > > > extension of
> > > > > the
> > > > > > > current
> > > > > > > interceptors which exposed
> a
> > different
> > > > object (e.g.
> > > > > > > `RecordAndHeaders`).
> > > > > > > The challenge is for
> MM-like
> > use cases. Let
> > > > me see if I can
> > > > > > come
> > > > > > > up with a
> > > > > > > concrete proposal for that
> > problem.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at
> > 11:55 AM, Michael
> > > > Pearce <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I am happy to move the
> > definition of the
> > > > header into the
> > > > > > message
> > > > > > > body, but
> > > > > > > > would cause us not to
> lazy
> > > > initialise/parse the headers,
> > > > > as
> > > > > > > obviously, we
> > > > > > > > would have to traverse
> > these reading the
> > > > message.
> > > > > > > >
> > > > > > > > This was actually one of
> > Jay’s requests:
> > > > > > > >
> > > > > > > > “ 2. I think we should
> > think about
> > > > creating the Map
> > > > > > lazily to
> > > > > > > avoid
> > > > > > > > parsing out all the
> > headers into
> > > > little objects.
> > > > > > HashMaps
> > > > > > > themselves
> > > > > > > > are
> > > > > > > > kind of expensive and
> > the consumer is
> > > > very perf
> > > > > > sensitive so
> > > > > > > and making
> > > > > > > > gazillions of
> hashmaps
> > that may or
> > > may
> > > > not get used
> > > > > is
> > > > > > > probably a bad
> > > > > > > > idea.”
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 17/02/2017, 19:44,
> > "Michael Pearce" <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Yes exactly we have
> > access to the
> > > > records thus why
> > > > > the
> > > > > > > header should
> > > > > > > > be accessible via it and
> > not hidden for
> > > > only interceptors
> > > > > > to
> > > > > > > access.
> > > > > > > >
> > > > > > > > Sent using OWA for
> > iPhone
> > > > > > > >
> > ______________________________
> > > > __________
> > > > > > > > From: Magnus
> Edenhill <
> > > > magnus@edenhill.se>
> > > > > > > > Sent: Friday,
> February
> > 17, 2017
> > > > 7:34:49 PM
> > > > > > > > To:
> > dev@kafka.apache.org
> > > > > > > > Subject: Re:
> [DISCUSS]
> > KIP-82 - Add
> > > > Record Headers
> > > > > > > >
> > > > > > > > Big +1 on VarInts.
> > > > > > > > CPUs are fast, memory
> > is slow.
> > > > > > > >
> > > > > > > > I agree with Jason
> > that we'll want to
> > > > continue
> > > > > > verifying
> > > > > > > messages,
> > > > > > > > including their
> > headers, so while I
> > > > appreciate the
> > > > > > idea of
> > > > > > > the opaque
> > > > > > > > header blob it won't
> > be useful in
> > > > practice.
> > > > > > > >
> > > > > > > > /Magnus
> > > > > > > >
> > > > > > > > 2017-02-17 10:41
> > GMT-08:00 Jason
> > > > Gustafson <
> > > > > > > jason@confluent.io>:
> > > > > > > >
> > > > > > > > > Sorry, my mistake.
> > The consumer
> > > > interceptor is per
> > > > > > batch,
> > > > > > > though I'm
> > > > > > > > not
> > > > > > > > > sure that's an
> > actual limitation
> > > > since you still
> > > > > have
> > > > > > > access to the
> > > > > > > > > individual records.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > > On Fri, Feb 17,
> 2017
> > at 10:39 AM,
> > > > Jason Gustafson <
> > > > > > > > jason@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Re headers as
> byte
> > array and
> > > > future use by
> > > > > broker.
> > > > > > This
> > > > > > > doesn't
> > > > > > > > take away
> > > > > > > > > >> from that at
> all.
> > Nor makes it
> > > > difficult at all
> > > > > > in my
> > > > > > > opinion.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Yeah, I didn't
> say
> > it was
> > > > difficult, only
> > > > > awkward.
> > > > > > You
> > > > > > > wouldn't
> > > > > > > > write the
> > > > > > > > > > schema that way
> if
> > you were
> > > > planning to use it on
> > > > > > the
> > > > > > > brokers from
> > > > > > > > the
> > > > > > > > > > beginning. Note
> > also that one of
> > > > the benefits of
> > > > > > letting
> > > > > > > the broker
> > > > > > > > > > understand
> headers
> > is that it can
> > > > validate that
> > > > > > they are
> > > > > > > properly
> > > > > > > > > > formatted. If
> cost
> > is the only
> > > > concern, we should
> > > > > > > confirm its
> > > > > > > > impact
> > > > > > > > > > through
> > performance testing.
> > > > > > > > > >
> > > > > > > > > > One of the key
> use
> > cases requires
> > > > access on
> > > > > > consume at
> > > > > > > per
> > > > > > > > event/message
> > > > > > > > > >> level at the
> > point that message
> > > > is being
> > > > > > processed, as
> > > > > > > such the
> > > > > > > > batch
> > > > > > > > > >> interceptors and
> > batch consume
> > > > api isn't
> > > > > > suitable. It
> > > > > > > needs to be
> > > > > > > > at the
> > > > > > > > > >> record level.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I'm not sure I
> > understand the
> > > > point about
> > > > > batching.
> > > > > > > Interceptors
> > > > > > > > are
> > > > > > > > > > applied
> > per-message, right?
> > > > > > > > > >
> > > > > > > > > > My intent on
> > interceptors is to
> > > > keep the usage of
> > > > > > headers
> > > > > > > > well-defined so
> > > > > > > > > > that they don't
> > start leaking
> > > > unnecessarily into
> > > > > > > applications. My
> > > > > > > > guess
> > > > > > > > > is
> > > > > > > > > > that it's
> probably
> > inevitable,
> > > but
> > > > isolating it
> > > > > in
> > > > > > the
> > > > > > > > interceptors would
> > > > > > > > > > at least give
> > people a second
> > > > thought before
> > > > > > deciding to
> > > > > > > use it.
> > > > > > > > The main
> > > > > > > > > > challenge in my
> > mind is figuring
> > > > out how an MM
> > > > > use
> > > > > > case
> > > > > > > would
> > > > > > > > work. It
> > > > > > > > > > would be more
> > cumbersome to
> > > > replicate headers
> > > > > > through an
> > > > > > > > interceptor,
> > > > > > > > > > though arguably
> MM
> > should be
> > > > working at a lower
> > > > > > level
> > > > > > > anyway.
> > > > > > > > > >
> > > > > > > > > > -Jason
> > > > > > > > > >
> > > > > > > > > > On Fri, Feb 17,
> > 2017 at 10:16 AM,
> > > > Michael Pearce
> > > > > <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Re headers
> > available on the
> > > > record va
> > > > > > interceptors only
> > > > > > > > > >>
> > > > > > > > > >> One of the key
> > use cases
> > > requires
> > > > access on
> > > > > > consume at
> > > > > > > per
> > > > > > > > event/message
> > > > > > > > > >> level at the
> > point that message
> > > > is being
> > > > > > processed, as
> > > > > > > such the
> > > > > > > > batch
> > > > > > > > > >> interceptors and
> > batch consume
> > > > api isn't
> > > > > > suitable. It
> > > > > > > needs to be
> > > > > > > > at the
> > > > > > > > > >> record level.
> > > > > > > > > >>
> > > > > > > > > >> This anyhow is
> > similar to
> > > > jms/http/amqp where
> > > > > > headers
> > > > > > > are
> > > > > > > > available to
> > > > > > > > > >> consuming
> > applications.
> > > > > > > > > >>
> > > > > > > > > >> Re headers as
> > byte array and
> > > > future use by
> > > > > > broker. This
> > > > > > > doesn't
> > > > > > > > take
> > > > > > > > > away
> > > > > > > > > >> from that at
> all.
> > Nor makes it
> > > > difficult at all
> > > > > > in my
> > > > > > > opinion.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> Sent using OWA
> > for iPhone
> > > > > > > > > >>
> > ______________________________
> > > > __________
> > > > > > > > > >> From: Jason
> > Gustafson <
> > > > jason@confluent.io>
> > > > > > > > > >> Sent: Friday,
> > February 17, 2017
> > > > 5:55:42 PM
> > > > > > > > > >> To:
> > dev@kafka.apache.org
> > > > > > > > > >> Subject: Re:
> > [DISCUSS] KIP-82 -
> > > > Add Record
> > > > > Headers
> > > > > > > > > >>
> > > > > > > > > >> >
> > > > > > > > > >> > Would you be
> > proposing in
> > > > KIP-98 to convert
> > > > > the
> > > > > > other
> > > > > > > message
> > > > > > > > int’s
> > > > > > > > > (key
> > > > > > > > > >> > length, value
> > length) also to
> > > > varint to keep
> > > > > it
> > > > > > > uniform.
> > > > > > > > > >> > Also I assume
> > there will be a
> > > > static or helper
> > > > > > method
> > > > > > > made to
> > > > > > > > > write/read
> > > > > > > > > >> > these in the
> > client and
> > > server.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> Yes, that is
> what
> > we are
> > > > proposing, so using
> > > > > > varints
> > > > > > > for headers
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > >> consistent with
> > the rest of the
> > > > message. We have
> > > > > > used
> > > > > > > static
> > > > > > > > helper
> > > > > > > > > >> methods
> > > > > > > > > >> in our prototype
> > implementation.
> > > > > > > > > >>
> > > > > > > > > >> The cost of
> > parsing, we want to
> > > > parse/interpret
> > > > > > the
> > > > > > > headers
> > > > > > > > lazily (this
> > > > > > > > > >> is
> > > > > > > > > >> > a key point
> > brought up earlier
> > > > in discussions)
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> I'm a bit
> > skeptical of this. Has
> > > > anyone done the
> > > > > > > performance
> > > > > > > > testing? I
> > > > > > > > > >> can
> > > > > > > > > >> probably
> > implement it and test
> > > it
> > > > if no one else
> > > > > > has. I
> > > > > > > was also
> > > > > > > > under
> > > > > > > > > the
> > > > > > > > > >> impression that
> > there may be use
> > > > cases down the
> > > > > > road
> > > > > > > where the
> > > > > > > > broker
> > > > > > > > > >> would
> > > > > > > > > >> need to
> interpret
> > headers. That
> > > > wouldn't be off
> > > > > > the
> > > > > > > table in the
> > > > > > > > future
> > > > > > > > > if
> > > > > > > > > >> it's represented
> > as bytes, but
> > > it
> > > > would be quite
> > > > > > a bit
> > > > > > > more
> > > > > > > > awkward,
> > > > > > > > > >> right?
> > > > > > > > > >>
> > > > > > > > > >> By the way, one
> > question I have
> > > > been wondering
> > > > > > about. My
> > > > > > > > understanding
> > > > > > > > > is
> > > > > > > > > >> that headers are
> > primarily for
> > > > use cases where a
> > > > > > > third-party
> > > > > > > > components
> > > > > > > > > >> wants to enrich
> > messages without
> > > > needing to
> > > > > > understand
> > > > > > > or modify
> > > > > > > > the
> > > > > > > > > >> schema
> > > > > > > > > >> of the message
> > key and value.
> > > For
> > > > the
> > > > > applications
> > > > > > > which directly
> > > > > > > > > produce
> > > > > > > > > >> and consume the
> > messages and
> > > > control the
> > > > > key/value
> > > > > > > schema
> > > > > > > > directly, it
> > > > > > > > > >> seems we would
> > rather have them
> > > > implement
> > > > > headers
> > > > > > > directly in
> > > > > > > > their own
> > > > > > > > > >> schema.
> Supposing
> > for the sake
> > > of
> > > > argument that
> > > > > > it was
> > > > > > > possible,
> > > > > > > > my
> > > > > > > > > >> question is
> > whether it be
> > > > sufficient to expose
> > > > > the
> > > > > > > headers in the
> > > > > > > > > >> interceptor API
> > and not in the
> > > > common API?
> > > > > > > > > >>
> > > > > > > > > >> -Jason
> > > > > > > > > >>
> > > > > > > > > >> On Fri, Feb 17,
> > 2017 at 3:26 AM,
> > > > Michael Pearce
> > > > > <
> > > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > On the point
> of
> > varInts
> > > > > > > > > >> >
> > > > > > > > > >> > Would you be
> > proposing in
> > > > KIP-98 to convert
> > > > > the
> > > > > > other
> > > > > > > message
> > > > > > > > int’s
> > > > > > > > > (key
> > > > > > > > > >> > length, value
> > length) also to
> > > > varint to keep
> > > > > it
> > > > > > > uniform.
> > > > > > > > > >> > Also I assume
> > there will be a
> > > > static or helper
> > > > > > method
> > > > > > > made to
> > > > > > > > > write/read
> > > > > > > > > >> > these in the
> > client and
> > > server.
> > > > > > > > > >> >
> > > > > > > > > >> > Cheers
> > > > > > > > > >> > Mike
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On 17/02/2017,
> > 11:22, "Michael
> > > > Pearce" <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > On the
> > point re: headers
> > > in
> > > > the message
> > > > > > protocol
> > > > > > > being a
> > > > > > > > byte
> > > > > > > > > array
> > > > > > > > > >> > and not a
> count
> > of elements
> > > > followed by the
> > > > > > elements.
> > > > > > > Again
> > > > > > > > this was
> > > > > > > > > >> >
> > discussed/argued previously.
> > > > > > > > > >> >
> > > > > > > > > >> > It was
> > agreed on for a few
> > > > reasons some of
> > > > > > which
> > > > > > > you have
> > > > > > > > > obviously
> > > > > > > > > >> > picked up on:
> > > > > > > > > >> >
> > > > > > > > > >> > Broker is
> > able to pass it
> > > > through opaquely
> > > > > > > > > >> > The cost
> of
> > parsing, we
> > > > want to
> > > > > > parse/interpret
> > > > > > > the headers
> > > > > > > > lazily
> > > > > > > > > >> > (this is a key
> > point brought
> > > up
> > > > earlier in
> > > > > > > discussions)
> > > > > > > > > >> > Headers
> can
> > be copied from
> > > > consumer record
> > > > > > to
> > > > > > > producer
> > > > > > > > record (aka
> > > > > > > > > >> > mirror makers
> > etc) without
> > > > parsing if no
> > > > > > changes are
> > > > > > > being made
> > > > > > > > or
> > > > > > > > > being
> > > > > > > > > >> > looked at.
> > > > > > > > > >> > Keeps the
> > broker agnostic
> > > > to the format
> > > > > > > > > >> > You need
> an
> > int32 either
> > > > for the byte size
> > > > > > of the
> > > > > > > headers,
> > > > > > > > or for
> > > > > > > > > >> the
> > > > > > > > > >> > count of
> > elements, so
> > > overheads
> > > > are the same,
> > > > > > but
> > > > > > > with going
> > > > > > > > with an
> > > > > > > > > >> opaque
> > > > > > > > > >> > byte array has
> > the above
> > > > advantages.
> > > > > > > > > >> >
> > > > > > > > > >> > Cheers
> > > > > > > > > >> > Mike
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On
> > 17/02/2017, 02:50,
> > > > "Jason Gustafson" <
> > > > > > > jason@confluent.io
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > Sorry,
> > should have
> > > > noted that the
> > > > > > performance
> > > > > > > testing
> > > > > > > > was done
> > > > > > > > > >> > using the
> > > > > > > > > >> >
> > producer performance
> > > > tool shipped with
> > > > > > Kafka.
> > > > > > > > > >> >
> > > > > > > > > >> > -Jason
> > > > > > > > > >> >
> > > > > > > > > >> > On
> Thu,
> > Feb 16, 2017
> > > at
> > > > 6:44 PM, Jason
> > > > > > > Gustafson <
> > > > > > > > > >> >
> > jason@confluent.io> wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Hey
> > Nacho,
> > > > > > > > > >> > >
> > > > > > > > > >> > > I've
> > compared
> > > > performance of our
> > > > > > KIP-98
> > > > > > > > implementation with
> > > > > > > > > >> and
> > > > > > > > > >> > without
> > > > > > > > > >> > >
> > varints. For
> > > messages
> > > > around 128
> > > > > > bytes, we
> > > > > > > see an
> > > > > > > > increase
> > > > > > > > > in
> > > > > > > > > >> > throughput of
> > > > > > > > > >> > >
> about
> > 30% using the
> > > > default
> > > > > > configuration
> > > > > > > settings.
> > > > > > > > At 256
> > > > > > > > > >> > bytes, the
> > > > > > > > > >> > >
> > increase is around
> > > > 16%. Obviously
> > > > > the
> > > > > > > performance
> > > > > > > > converge
> > > > > > > > > as
> > > > > > > > > >> > messages get
> > > > > > > > > >> > >
> > larger, but it seems
> > > > well worth the
> > > > > > cost.
> > > > > > > Note that
> > > > > > > > we are
> > > > > > > > > >> also
> > > > > > > > > >> > seeing a
> > > > > > > > > >> > >
> > substantial
> > > > performance increase
> > > > > > against
> > > > > > > trunk
> > > > > > > > primarily
> > > > > > > > > >> because
> > > > > > > > > >> > of the
> > > > > > > > > >> > > much
> > more efficient
> > > > packing that
> > > > > > varints
> > > > > > > provide us.
> > > > > > > > > Anything
> > > > > > > > > >> > adding to
> > > > > > > > > >> > >
> > message overhead,
> > > > such as record
> > > > > > headers,
> > > > > > > would only
> > > > > > > > > increase
> > > > > > > > > >> > the relative
> > > > > > > > > >> > >
> > difference. (Of
> > > > course take these
> > > > > > numbers
> > > > > > > with a
> > > > > > > > grain of
> > > > > > > > > salt
> > > > > > > > > >> > since I have
> > > > > > > > > >> > > only
> > used the
> > > default
> > > > settings with
> > > > > > both
> > > > > > > the producer
> > > > > > > > and
> > > > > > > > > >> broker
> > > > > > > > > >> > on my
> > > > > > > > > >> > >
> local
> > machine. We
> > > > intend to provide
> > > > > > more
> > > > > > > extensive
> > > > > > > > > performance
> > > > > > > > > >> > details as
> > > > > > > > > >> > > part
> > of the work for
> > > > KIP-98.)
> > > > > > > > > >> > >
> > > > > > > > > >> > > The
> > implementation
> > > we
> > > > are using is
> > > > > > from
> > > > > > > protobuf (
> > > > > > > > > >> > >
> > > > https://developers.google.com/
> > > > > > > > >
> > protocol-buffers/docs/encoding
> > > > > > > > > >> ),
> > > > > > > > > >> > which is
> > > > > > > > > >> > > also
> > used in HBase.
> > > > It is trivial to
> > > > > > > implement and as
> > > > > > > > far
> > > > > > > > > as I
> > > > > > > > > >> > know doesn't
> > > > > > > > > >> > >
> > suffer from the
> > > > aliasing problem you
> > > > > > are
> > > > > > > describing. I
> > > > > > > > > checked
> > > > > > > > > >> > with Magnus
> > > > > > > > > >> > > (the
> > author of
> > > > librdkafka) and he
> > > > > > agreed
> > > > > > > that the
> > > > > > > > savings
> > > > > > > > > >> seemed
> > > > > > > > > >> > worth the
> > > > > > > > > >> > > cost
> > of
> > > > implementation.
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> -Jason
> > > > > > > > > >> > >
> > > > > > > > > >> > > On
> > Thu, Feb 16, 2017
> > > > at 4:32 PM,
> > > > > > Ignacio
> > > > > > > Solis <
> > > > > > > > > >> isolis@igso.net
> >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > >>
> > -VarInts
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> I'm
> > one of the
> > > > people (if not the
> > > > > > most)
> > > > > > > opposed to
> > > > > > > > VarInts.
> > > > > > > > > >> > VarInts
> > > > > > > > > >> > >>
> have
> > a place, but
> > > > this is not it.
> > > > > > (We
> > > > > > > had a large
> > > > > > > > > >> discussion
> > > > > > > > > >> > about
> > > > > > > > > >> > >>
> them
> > at the
> > > > beginning of KIP-82
> > > > > time)
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> If
> > anybody has real
> > > > life
> > > > > performance
> > > > > > > numbers of
> > > > > > > > VarInts
> > > > > > > > > >> > improving
> > > > > > > > > >> > >>
> > things or
> > > > significantly reducing
> > > > > > resources
> > > > > > > I would
> > > > > > > > like to
> > > > > > > > > >> know
> > > > > > > > > >> > what
> > > > > > > > > >> > >>
> that
> > case may be.
> > > > Yes, you can save
> > > > > > some
> > > > > > > bytes here
> > > > > > > > and
> > > > > > > > > >> there,
> > > > > > > > > >> > but
> > > > > > > > > >> > >>
> this
> > is probably
> > > > insignificant to
> > > > > the
> > > > > > > overall system
> > > > > > > > > behavior
> > > > > > > > > >> > and
> > > > > > > > > >> > >>
> > storage
> > > > requirements. -- I say
> > > > > this
> > > > > > with
> > > > > > > respect to
> > > > > > > > using
> > > > > > > > > >> > VarInts in
> > > > > > > > > >> > >> the
> > protocol
> > > itself,
> > > > not as part of
> > > > > > the
> > > > > > > data.
> > > > > > > > > >> > >>
> > > > > > > > > >> > >>
> > VarInts require you
> > > > to parse the
> > > > > Int
> > > > > > > before using it
> > > > > > > > and
> > > > > > > > > >> > depending on
> > > > > > > > > >> > >> the
> > encoding they
> > > > can suffer from
> > > > > > aliasing
> > > > > > > (multiple
> > > > > > > > > >> >
> representations
> > > > > > > > > >> > >> for
> > the same
> > > value).
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> Why
> > add complexity?
> > > > > > > > > >> > >>
> > > > > > > > > >> > >>
> Nacho
> > > > > > > > > >> > >>
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> On
> > Thu, Feb 16,
> > > 2017
> > > > at 10:29 AM,
> > > > > > Colin
> > > > > > > McCabe <
> > > > > > > > > >> >
> > cmccabe@apache.org>
> > > > > > > > > >> > >>
> > wrote:
> > > > > > > > > >> > >> >
> +1
> > for varints
> > > > here-- it would
> > > > > save
> > > > > > > quite a bit of
> > > > > > > > space.
> > > > > > > > > >> > They are
> > > > > > > > > >> > >> >
> > pretty quick to
> > > > implement as
> > > > > well.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > I
> > think it makes
> > > > sense for values
> > > > > > to be
> > > > > > > byte
> > > > > > > > arrays.
> > > > > > > > > Users
> > > > > > > > > >> > might want
> > > > > > > > > >> > >> >
> to
> > attach
> > > > arbitrary payloads;
> > > > > they
> > > > > > > shouldn't be
> > > > > > > > forced to
> > > > > > > > > >> > serialize
> > > > > > > > > >> > >> >
> > everything to
> > > Java
> > > > strings.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > best,
> > > > > > > > > >> > >> >
> > Colin
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> On
> > Thu, Feb 16,
> > > > 2017, at 09:52,
> > > > > > Jason
> > > > > > > Gustafson
> > > > > > > > wrote:
> > > > > > > > > >> > >> >>
> > Hey Michael,
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > Hmm, I guess the
> > > > point of
> > > > > > representing
> > > > > > > it as
> > > > > > > > bytes is to
> > > > > > > > > >> > allow the
> > > > > > > > > >> > >>
> > broker
> > > > > > > > > >> > >> >>
> > to pass it
> > > > through opaquely? Is
> > > > > > the
> > > > > > > cost of
> > > > > > > > parsing
> > > > > > > > > them a
> > > > > > > > > >> > concern, or
> > > > > > > > > >> > >> >>
> > are
> > > > > > > > > >> > >> >>
> > we simply trying
> > > > to ensure that
> > > > > > the
> > > > > > > broker stays
> > > > > > > > > agnostic
> > > > > > > > > >> to
> > > > > > > > > >> > the
> > > > > > > > > >> > >>
> > format?
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > On varints, I
> > > > think adding
> > > > > > support for
> > > > > > > them makes
> > > > > > > > less
> > > > > > > > > >> sense
> > > > > > > > > >> > for an
> > > > > > > > > >> > >> >>
> > isolated use
> > > > case, but as part
> > > > > of
> > > > > > a
> > > > > > > more holistic
> > > > > > > > change
> > > > > > > > > >> > (such as what
> > > > > > > > > >> > >> we
> > > > > > > > > >> > >> >>
> > have proposed in
> > > > KIP-98), I
> > > > > think
> > > > > > they
> > > > > > > are
> > > > > > > > justifiable.
> > > > > > > > > If
> > > > > > > > > >> > we add them,
> > > > > > > > > >> > >> >>
> > then the need to
> > > > use attributes
> > > > > > becomes
> > > > > > > quite a
> > > > > > > > bit
> > > > > > > > > >> weaker,
> > > > > > > > > >> > right? The
> > > > > > > > > >> > >> >>
> > other thing I
> > > > find slightly odd
> > > > > > is the
> > > > > > > fact that
> > > > > > > > null
> > > > > > > > > >> > headers has no
> > > > > > > > > >> > >> >>
> > actual
> > > > > > > > > >> > >> >>
> > semantic meaning
> > > > for the message
> > > > > > > (unlike null
> > > > > > > > keys and
> > > > > > > > > >> > values). It is
> > > > > > > > > >> > >> >>
> > just
> > > > > > > > > >> > >> >>
> a
> > space
> > > > optimization. It seems a
> > > > > > bit
> > > > > > > better to
> > > > > > > > always
> > > > > > > > > use
> > > > > > > > > >> > size 0 to
> > > > > > > > > >> > >> >>
> > indicate having
> > > > no headers.
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > Overall, the
> > > main
> > > > point is
> > > > > > ensuring
> > > > > > > that the
> > > > > > > > message
> > > > > > > > > >> schema
> > > > > > > > > >> > remains
> > > > > > > > > >> > >> >>
> > consistent,
> > > > either within the
> > > > > > larger
> > > > > > > protocol, or
> > > > > > > > at a
> > > > > > > > > >> > minimum within
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> > message itself.
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > -Jason
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> > On Thu, Feb 16,
> > > > 2017 at 6:39 AM,
> > > > > > > Michael Pearce <
> > > > > > > > > >> >
> > Michael.Pearce@ig.com
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >>
> > wrote:
> > > > > > > > > >> > >> >>
> > > > > > > > > >> > >> >>
> >
> > Hi Jason,
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > On point 1) in
> > > > the message
> > > > > > protocol
> > > > > > > the headers
> > > > > > > > are
> > > > > > > > > >> simply
> > > > > > > > > >> > a byte
> > > > > > > > > >> > >>
> > array,
> > > > > > > > > >> > >> >>
> >
> > as like the
> > > key
> > > > or value, this
> > > > > > is to
> > > > > > > clearly
> > > > > > > > demarcate
> > > > > > > > > >> the
> > > > > > > > > >> > header in
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> >
> > core message.
> > > > Then the header
> > > > > > byte
> > > > > > > array in the
> > > > > > > > core
> > > > > > > > > >> > message is an
> > > > > > > > > >> > >>
> > array of
> > > > > > > > > >> > >> >>
> >
> > key, value
> > > > pairs. This is what
> > > > > > it is
> > > > > > > denoting.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > Then this
> > > would
> > > > be I guess in
> > > > > > the
> > > > > > > given
> > > > > > > > notation:
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > Headers =>
> > > > [KeyLength, Key,
> > > > > > > ValueLength, Value]
> > > > > > > > > >> > >> >>
> > > KeyLength
> > > > => int32
> > > > > > > <-----------------NEW
> > > > > > > > size of
> > > > > > > > > the
> > > > > > > > > >> > byte[] of
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> >
> > serialised key
> > > > value
> > > > > > > > > >> > >> >>
> > > Key =>
> > > bytes
> > > > > > > <---------------------- NEW
> > > > > > > > > serialised
> > > > > > > > > >> > string (UTF8)
> > > > > > > > > >> > >> >>
> >
> > bytes of the
> > > > header key
> > > > > > > > > >> > >> >>
> >
> > > ValueLength
> > > > => int32
> > > > > > > <-------------- NEW
> > > > > > > > size of
> > > > > > > > > the
> > > > > > > > > >> > byte[] of
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> >
> > serialised
> > > > header value
> > > > > > > > > >> > >> >>
> > > Value =>
> > > > bytes
> > > > > > > <-------------------- NEW
> > > > > > > > > serialised
> > > > > > > > > >> > form of the
> > > > > > > > > >> > >>
> > header
> > > > > > > > > >> > >> >>
> >
> > value
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > The key length
> > > > and value
> > > > > length
> > > > > > is
> > > > > > > matching the
> > > > > > > > way
> > > > > > > > > the
> > > > > > > > > >> > protocol is
> > > > > > > > > >> > >> >>
> >
> > defined in the
> > > > core message
> > > > > > currently.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > On point 2)
> > > > > > > > > >> > >> >>
> >
> > Var sized
> > > ints,
> > > > this was
> > > > > > discussed
> > > > > > > much earlier
> > > > > > > > on, in
> > > > > > > > > >> > fact I had
> > > > > > > > > >> > >> >>
> >
> > suggested it
> > > > myself (with
> > > > > Hadoop
> > > > > > > references),
> > > > > > > > the
> > > > > > > > > >> > complexity of
> > this
> > > > > > > > > >> > >> >>
> >
> > compared to
> > > > having a simpler
> > > > > > protocol
> > > > > > > was
> > > > > > > > argued and
> > > > > > > > > >> > agreed it
> > > > > > > > > >> > >>
> > wasn’t worth
> > > > > > > > > >> > >> >>
> >
> > the complexity
> > > > as all other
> > > > > > clients
> > > > > > > in other
> > > > > > > > languages
> > > > > > > > > >> > would need to
> > > > > > > > > >> > >>
> > ensure
> > > > > > > > > >> > >> >>
> >
> > theyre using
> > > > the right var
> > > > > size
> > > > > > > algorithm, as
> > > > > > > > there
> > > > > > > > > is a
> > > > > > > > > >> > few.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > On point 3)
> > > > > > > > > >> > >> >>
> >
> > We did the
> > > > attributes,
> > > > > optional
> > > > > > > approach as
> > > > > > > > originally
> > > > > > > > > >> > there was
> > > > > > > > > >> > >>
> > marked
> > > > > > > > > >> > >> >>
> >
> > concern that
> > > > headers would
> > > > > > cause a
> > > > > > > message size
> > > > > > > > > overhead
> > > > > > > > > >> > for others,
> > > > > > > > > >> > >> who
> > > > > > > > > >> > >> >>
> >
> > don’t want
> > > > them. As such this
> > > > > > is the
> > > > > > > clean
> > > > > > > > solution to
> > > > > > > > > >> > achieve that.
> > > > > > > > > >> > >> If
> > > > > > > > > >> > >> >>
> >
> > that no longer
> > > > holds, and we
> > > > > > don’t
> > > > > > > care that we
> > > > > > > > add
> > > > > > > > > >> 4bytes
> > > > > > > > > >> > overhead,
> > > > > > > > > >> > >>
> then
> > > > > > > > > >> > >> >>
> >
> > im happy to
> > > > remove.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > I’m personally
> > > > in favour of
> > > > > > keeping
> > > > > > > the message
> > > > > > > > as
> > > > > > > > > small
> > > > > > > > > >> > as possible
> > > > > > > > > >> > >> so
> > > > > > > > > >> > >> >>
> >
> > people don’t
> > > > get shocks in
> > > > > perf
> > > > > > and
> > > > > > > throughputs
> > > > > > > > dues
> > > > > > > > > to
> > > > > > > > > >> > message size,
> > > > > > > > > >> > >> >>
> >
> > unless they
> > > > actively use the
> > > > > > feature,
> > > > > > > as such I
> > > > > > > > do
> > > > > > > > > >> prefer
> > > > > > > > > >> > the
> > > > > > > > > >> > >>
> > attribute bit
> > > > > > > > > >> > >> >>
> >
> > wise feature
> > > > flag approach
> > > > > > myself.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > On 16/02/2017,
> > > > 05:40, "Jason
> > > > > > > Gustafson" <
> > > > > > > > > >> >
> > jason@confluent.io> wrote:
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > We have
> > > > proposed a few
> > > > > > > significant changes
> > > > > > > > to the
> > > > > > > > > >> > message format
> > > > > > > > > >> > >> in
> > > > > > > > > >> > >> >>
> >
> > KIP-98
> > > > > > > > > >> > >> >>
> > > which now
> > > > seems likely to
> > > > > > pass
> > > > > > > (perhaps
> > > > > > > > with some
> > > > > > > > > >> > iterations on
> > > > > > > > > >> > >> >>
> >
> > > > implementation details).
> > > > > It
> > > > > > would
> > > > > > > be good
> > > > > > > > to try
> > > > > > > > > and
> > > > > > > > > >> > coordinate
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> >
> > changes
> > > > > > > > > >> > >> >>
> > > in both of
> > > > the proposals
> > > > > to
> > > > > > make
> > > > > > > sure they
> > > > > > > > are
> > > > > > > > > >> > consistent and
> > > > > > > > > >> > >> >>
> >
> > compatible.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > I think
> > > > using the
> > > > > > attributes to
> > > > > > > indicate
> > > > > > > > null
> > > > > > > > > >> headers
> > > > > > > > > >> > is a
> > > > > > > > > >> > >>
> > reasonable
> > > > > > > > > >> > >> >>
> > > approach.
> > > > We have proposed
> > > > > > to do
> > > > > > > the same
> > > > > > > > thing
> > > > > > > > > for
> > > > > > > > > >> > the message
> > > > > > > > > >> > >> key
> > and
> > > > > > > > > >> > >> >>
> > > value.
> > > That
> > > > said, I
> > > > > > sympathize
> > > > > > > with Jay's
> > > > > > > > > argument.
> > > > > > > > > >> > Having
> > > > > > > > > >> > >>
> > multiple
> > > > > > > > > >> > >> >>
> >
> > ways to
> > > > > > > > > >> > >> >>
> > > specify a
> > > > null value
> > > > > > increases
> > > > > > > the overall
> > > > > > > > > >> complexity
> > > > > > > > > >> > of the
> > > > > > > > > >> > >>
> > protocol.
> > > > > > > > > >> > >> >>
> >
> > You
> > > > > > > > > >> > >> >>
> > > can see
> > > > this just from the
> > > > > > fact
> > > > > > > that you
> > > > > > > > need the
> > > > > > > > > >> > extra verbiage
> > > > > > > > > >> > >> in
> > the
> > > > > > > > > >> > >> >>
> > > protocol
> > > > specification in
> > > > > > this
> > > > > > > KIP and in
> > > > > > > > KIP-98
> > > > > > > > > to
> > > > > > > > > >> > describe the
> > > > > > > > > >> > >> >>
> >
> > dependence
> > > > > > > > > >> > >> >>
> > > between
> > > the
> > > > fields and the
> > > > > > > attributes. It
> > > > > > > > seems
> > > > > > > > > >> like a
> > > > > > > > > >> > slippery
> > > > > > > > > >> > >>
> slope
> > > > > > > > > >> > >> >>
> >
> > if
> > > > > > > > > >> > >> >>
> > > you start
> > > > allowing
> > > > > different
> > > > > > > request types
> > > > > > > > to
> > > > > > > > > >> > implement the
> > > > > > > > > >> > >>
> > protocol
> > > > > > > > > >> > >> >>
> >
> > > > specification differently.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > You can
> > > > also argue that
> > > > > the
> > > > > > > messages
> > > > > > > > already are
> > > > > > > > > and
> > > > > > > > > >> > are likely
> > > > > > > > > >> > >> to
> > > > > > > > > >> > >> >>
> >
> > remain a
> > > > > > > > > >> > >> >>
> > > special
> > > > case. For example,
> > > > > > there
> > > > > > > is
> > > > > > > > currently no
> > > > > > > > > >> > generality in
> > > > > > > > > >> > >> how
> > > > > > > > > >> > >> >>
> > > compressed
> > > > message sets
> > > > > are
> > > > > > > represented
> > > > > > > > that would
> > > > > > > > > >> be
> > > > > > > > > >> > applicable
> > > > > > > > > >> > >> for
> > > > > > > > > >> > >> >>
> >
> > other
> > > > > > > > > >> > >> >>
> > > request
> > > > types. Some might
> > > > > > see this
> > > > > > > > divergence as
> > > > > > > > > an
> > > > > > > > > >> > unfortunate
> > > > > > > > > >> > >> >>
> >
> > protocol
> > > > > > > > > >> > >> >>
> > > deficiency
> > > > which should be
> > > > > > fixed;
> > > > > > > others
> > > > > > > > might see
> > > > > > > > > >> it
> > > > > > > > > >> > as sort of
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> >
> > > > inevitability of needing
> > > > > to
> > > > > > > optimize where
> > > > > > > > it
> > > > > > > > > counts
> > > > > > > > > >> > most. I'm
> > > > > > > > > >> > >>
> > probably
> > > > > > > > > >> > >> >>
> > > somewhere
> > > > in between, but
> > > > > I
> > > > > > think
> > > > > > > we
> > > > > > > > probably all
> > > > > > > > > >> > share the
> > > > > > > > > >> > >>
> > intuition
> > > > > > > > > >> > >> >>
> >
> > that
> > > > > > > > > >> > >> >>
> > > the
> > > > protocol should be
> > > > > kept
> > > > > > as
> > > > > > > consistent as
> > > > > > > > > >> possible.
> > > > > > > > > >> > With that
> > > > > > > > > >> > >> in
> > > > > > > > > >> > >> >>
> >
> > mind,
> > > > > > > > > >> > >> >>
> > > here are a
> > > > few comments:
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > 1. One
> > > > thing I found a
> > > > > > little odd
> > > > > > > when
> > > > > > > > reading the
> > > > > > > > > >> > current
> > > > > > > > > >> > >>
> > proposal is
> > > > > > > > > >> > >> >>
> >
> > that
> > > > > > > > > >> > >> >>
> > > the
> > > headers
> > > > are both
> > > > > > represented
> > > > > > > as an
> > > > > > > > array of
> > > > > > > > > >> bytes
> > > > > > > > > >> > and as an
> > > > > > > > > >> > >>
> array
> > > > > > > > > >> > >> >>
> >
> > of
> > > > > > > > > >> > >> >>
> > > key/value
> > > > pairs. I'd
> > > > > > probably
> > > > > > > suggest
> > > > > > > > something
> > > > > > > > > like
> > > > > > > > > >> > this:
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > Headers =>
> > > > [HeaderKey
> > > > > > HeaderValue]
> > > > > > > > > >> > >> >>
> > > HeaderKey
> > > > => String
> > > > > > > > > >> > >> >>
> >
> > > > HeaderValue => Bytes
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > An array
> > > in
> > > > the Kafka
> > > > > > protocol is
> > > > > > > > represented as a
> > > > > > > > > >> > 4-byte integer
> > > > > > > > > >> > >> >>
> > > indicating
> > > > the number of
> > > > > > elements
> > > > > > > in the
> > > > > > > > array
> > > > > > > > > >> > followed by
> the
> > > > > > > > > >> > >> >>
> >
> > > > serialization of the
> > > > > > elements.
> > > > > > > Unless I'm
> > > > > > > > > >> >
> > misunderstanding, what
> > > > > > > > > >> > >> you
> > > > > > > > > >> > >> >>
> >
> > have
> > > > > > > > > >> > >> >>
> > > instead is
> > > > the total size
> > > > > > of the
> > > > > > > headers in
> > > > > > > > bytes
> > > > > > > > > >> > followed by
> the
> > > > > > > > > >> > >> >>
> >
> > elements.
> > > > > > > > > >> > >> >>
> > > I'm not
> > > > sure I see any
> > > > > > reason for
> > > > > > > this
> > > > > > > > > >> inconsistency.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > 2. In
> > > > KIP-98, we've
> > > > > > introduced
> > > > > > > > variable-length
> > > > > > > > > >> integer
> > > > > > > > > >> > fields.
> > > > > > > > > >> > >> >>
> >
> > Effectively,
> > > > > > > > > >> > >> >>
> > > we've
> > > > enriched (or
> > > > > > "complicated"
> > > > > > > as Jay
> > > > > > > > might say
> > > > > > > > > ;)
> > > > > > > > > >> > the protocol
> > > > > > > > > >> > >> >>
> >
> > > > specification to include
> > > > > the
> > > > > > > following
> > > > > > > > types:
> > > > > > > > > >> VarInt,
> > > > > > > > > >> > VarLong,
> > > > > > > > > >> > >> >>
> >
> > > > UnsignedVarInt and
> > > > > > > UnsignedVarLong.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > Along with
> > > > these
> > > > > > primitives, we
> > > > > > > could
> > > > > > > > introduce
> > > > > > > > > the
> > > > > > > > > >> > following
> > > > > > > > > >> > >>
> > types:
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > VarSizeArray =>
> > > > > > NumberOfItems
> > > > > > > Item1 Item2
> > > > > > > > .. ItemN
> > > > > > > > > >> > >> >>
> >
> > > > NumberOfItems =>
> > > > > > UnsignedVarInt
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > VarSizeNullableArray =>
> > > > > > > NumberOfItemsOrNull
> > > > > > > > Item1
> > > > > > > > > >> > Item2 .. ItemN
> > > > > > > > > >> > >> >>
> >
> > > > NumberOfItemsOrNull =>
> > > > > > VarInt
> > > > > > > (-1 means
> > > > > > > > null)
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > And
> > > > similarly for the
> > > > > > `String`
> > > > > > > and `Bytes`
> > > > > > > > types.
> > > > > > > > > >> > These types
> > > > > > > > > >> > >> can
> > save
> > > > > > > > > >> > >> >>
> >
> > a
> > > > > > > > > >> > >> >>
> >
> > > > considerable amount of
> > > > > > space in
> > > > > > > this
> > > > > > > > proposal
> > > > > > > > > >> because
> > > > > > > > > >> > they can
> > > > > > > > > >> > >> be
> > used
> > > > > > > > > >> > >> >>
> >
> > for
> > > > > > > > > >> > >> >>
> > > both the
> > > > number of headers
> > > > > > > included in the
> > > > > > > > message
> > > > > > > > > >> and
> > > > > > > > > >> > the
> > > > > > > > > >> > >>
> > lengths of
> > > > > > > > > >> > >> >>
> >
> > the
> > > > > > > > > >> > >> >>
> > > header
> > > keys
> > > > and values. We
> > > > > > could
> > > > > > > do this
> > > > > > > > instead:
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > Headers =>
> > > > > > VarSizeArray[HeaderKey
> > > > > > > > HeaderValue]
> > > > > > > > > >> > >> >>
> >
> > > HeaderKey
> > > > =>
> > > > > VarSizeString
> > > > > > > > > >> > >> >>
> >
> > > > HeaderValue =>
> > > > > > VarSizeBytes
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > Combining
> > > > the savings from
> > > > > > the
> > > > > > > use of
> > > > > > > > variable
> > > > > > > > > >> length
> > > > > > > > > >> > fields, the
> > > > > > > > > >> > >> >>
> >
> > benefit
> > > > > > > > > >> > >> >>
> > > of using
> > > > the attributes to
> > > > > > > represent null
> > > > > > > > seems
> > > > > > > > > >> pretty
> > > > > > > > > >> > small.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > 3.
> > > > Whichever way we go
> > > > > > (whether
> > > > > > > we use the
> > > > > > > > > >> attributes
> > > > > > > > > >> > or not), we
> > > > > > > > > >> > >> >>
> >
> > should at
> > > > > > > > > >> > >> >>
> > > least be
> > > > consistent
> > > > > between
> > > > > > this
> > > > > > > KIP and
> > > > > > > > KIP-98.
> > > > > > > > > It
> > > > > > > > > >> > would be very
> > > > > > > > > >> > >> >>
> >
> > strange
> > > > > > > > > >> > >> >>
> > > to have
> > > two
> > > > ways to
> > > > > > represent
> > > > > > > null values
> > > > > > > > in the
> > > > > > > > > >> same
> > > > > > > > > >> > schema.
> > > > > > > > > >> > >>
> > Either
> > > > > > > > > >> > >> >>
> >
> > way is
> > > > > > > > > >> > >> >>
> > > OK with
> > > me.
> > > > I think some
> > > > > > > message-level
> > > > > > > > > optimizations
> > > > > > > > > >> > are
> > > > > > > > > >> > >>
> > justifiable,
> > > > > > > > > >> > >> >>
> >
> > but
> > > > > > > > > >> > >> >>
> > > the
> > > savings
> > > > here seem
> > > > > > minimal (a
> > > > > > > few bytes
> > > > > > > > per
> > > > > > > > > >> > message), so
> > > > > > > > > >> > >>
> > maybe it's
> > > > > > > > > >> > >> >>
> >
> > not
> > > > > > > > > >> > >> >>
> > > worth the
> > > > cost of letting
> > > > > > the
> > > > > > > message
> > > > > > > > diverge even
> > > > > > > > > >> > further from
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> >
> > rest of
> > > > > > > > > >> > >> >>
> > > the
> > > > protocol.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > -Jason
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > On Wed,
> > > Feb
> > > > 15, 2017 at
> > > > > > 8:52 AM,
> > > > > > > radai <
> > > > > > > > > >> > >>
> > > > radai.rosenblatt@gmail.com>
> > > > > > > > > >> > >> >>
> >
> > wrote:
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> > > > I've
> > > > trimmed the inline
> > > > > > > contents as this
> > > > > > > > mail is
> > > > > > > > > >> > getting too
> > > > > > > > > >> > >> big
> > for
> > > > > > > > > >> > >> >>
> >
> > the
> > > > > > > > > >> > >> >>
> > > > apache
> > > > mailing list
> > > > > > software to
> > > > > > > deliver
> > > > > > > > :-(
> > > > > > > > > >> > >> >>
> > > >
> > > > > > > > > >> > >> >>
> > > > 1. the
> > > > important thing
> > > > > for
> > > > > > > > interoperability is
> > > > > > > > > for
> > > > > > > > > >> > different
> > > > > > > > > >> > >> >>
> >
> > "interested
> > > > > > > > > >> > >> >>
> > > > parties"
> > > > (plugins, infra
> > > > > > > layers/wrappers,
> > > > > > > > > >> user-code)
> > > > > > > > > >> > to be
> > > > > > > > > >> > >>
> able
> > to
> > > > > > > > > >> > >> >>
> >
> > stick
> > > > > > > > > >> > >> >>
> > > > pieces
> > > of
> > > > metadata onto
> > > > > > msgs
> > > > > > > without
> > > > > > > > getting in
> > > > > > > > > >> each
> > > > > > > > > >> > other's
> > > > > > > > > >> > >>
> way.
> > a
> > > > > > > > > >> > >> >>
> >
> > common
> > > > > > > > > >> > >> >>
> > > > key
> > > > scheme (Strings, as
> > > > > > of the
> > > > > > > time of
> > > > > > > > this
> > > > > > > > > >> > writing?) is
> all
> > > > > > > > > >> > >>
> thats
> > > > > > > > > >> > >> >>
> >
> > required
> > > > > > > > > >> > >> >>
> > > > for
> > > that.
> > > > it is assumed
> > > > > > that
> > > > > > > the other end
> > > > > > > > > >> > interested in
> > any
> > > > > > > > > >> > >>
> such
> > > > > > > > > >> > >> >>
> >
> > piece of
> > > > > > > > > >> > >> >>
> > > > metadata
> > > > knows the
> > > > > > encoding,
> > > > > > > and byte[]
> > > > > > > > provides
> > > > > > > > > >> for
> > > > > > > > > >> > the most
> > > > > > > > > >> > >> >>
> >
> > flexibility.
> > > > > > > > > >> > >> >>
> > > > i
> > > believe
> > > > this is the
> > > > > same
> > > > > > > logic behind
> > > > > > > > core
> > > > > > > > > kafka
> > > > > > > > > >> > being
> > > > > > > > > >> > >> >>
> >
> > byte[]/byte[]
> > > -
> > > > > > > > > >> > >> >>
> > > > Strings
> > > > are more
> > > > > "usable"
> > > > > > but
> > > > > > > bytes are
> > > > > > > > flexible
> > > > > > > > > >> and
> > > > > > > > > >> > so were
> > > > > > > > > >> > >>
> > chosen.
> > > > > > > > > >> > >> >>
> > > > Also -
> > > > core kafka doesnt
> > > > > > even
> > > > > > > do that
> > > > > > > > good of a
> > > > > > > > > >> job
> > > > > > > > > >> > on
> > > > > > > > > >> > >>
> > usability of
> > > > > > > > > >> > >> >>
> >
> > the
> > > > > > > > > >> > >> >>
> > > > payload
> > > > (example - i
> > > > > have
> > > > > > to
> > > > > > > specify the
> > > > > > > > nop
> > > > > > > > > >> byte[]
> > > > > > > > > >> > "decoders"
> > > > > > > > > >> > >> >>
> >
> > explicitly
> > > > > > > > > >> > >> >>
> > > > in
> > > conf),
> > > > and again
> > > > > > sacrificies
> > > > > > > usability
> > > > > > > > for
> > > > > > > > > the
> > > > > > > > > >> > sake of
> > > > > > > > > >> > >> >>
> >
> > performance
> > > (no
> > > > > > > > > >> > >> >>
> > > >
> > > > convenient single-record
> > > > > > > processing as
> > > > > > > > poll is a
> > > > > > > > > >> > batch, lots of
> > > > > > > > > >> > >> >>
> >
> > obscure
> > > > > > > > > >> > >> >>
> > > > little
> > > > config details
> > > > > > exposing
> > > > > > > internals
> > > > > > > > of the
> > > > > > > > > >> > batching
> > > > > > > > > >> > >>
> > mechanism,
> > > > > > > > > >> > >> >>
> >
> > etc)
> > > > > > > > > >> > >> >>
> > > >
> > > > > > > > > >> > >> >>
> > > > this is
> > > > also why i
> > > > > really
> > > > > > > dislike the
> > > > > > > > idea of a
> > > > > > > > > >> > "type system"
> > > > > > > > > >> > >> for
> > > > > > > > > >> > >> >>
> >
> > header
> > > > > > > > > >> > >> >>
> > > > values,
> > > > it further
> > > > > > degrades the
> > > > > > > > usability, adds
> > > > > > > > > >> > complexity and
> > > > > > > > > >> > >>
> will
> > > > > > > > > >> > >> >>
> > > >
> > > > eventually get in
> > > > > > people's way,
> > > > > > > also, it
> > > > > > > > would
> > > > > > > > > be
> > > > > > > > > >> > the 2nd/3rd
> > > > > > > > > >> > >> >>
> >
> > home-group
> > > > > > > > > >> > >> >>
> > > >
> > > > serialization mechanism
> > > > > > in core
> > > > > > > kafka
> > > > > > > > (counting
> > > > > > > > > 2
> > > > > > > > > >> > iterations
> > > > > > > > > >> > >> of
> > the
> > > > > > > > > >> > >> >>
> >
> > "type
> > > > > > > > > >> > >> >>
> > > >
> > > > definition DSL")
> > > > > > > > > >> > >> >>
> > > >
> > > > > > > > > >> > >> >>
> > > > 2. this
> > > > is an
> > > > > > implementation
> > > > > > > detail, and
> > > > > > > > not
> > > > > > > > > even
> > > > > > > > > >> a
> > > > > > > > > >> > very "user
> > > > > > > > > >> > >> >>
> >
> > facing" one?
> > > > > > > > > >> > >> >>
> > > > to the
> > > > best of my
> > > > > > understanding
> > > > > > > the vote
> > > > > > > > process
> > > > > > > > > >> is
> > > > > > > > > >> > on proposed
> > > > > > > > > >> > >> >>
> > > >
> > > > API/behaviour. also -
> > > > > > since
> > > > > > > we're willing
> > > > > > > > to go
> > > > > > > > > >> with
> > > > > > > > > >> > strings
> > > > > > > > > >> > >>
> just
> > > > > > > > > >> > >> >>
> >
> > serialize
> > > > > > > > > >> > >> >>
> > > > a
> > > 0-sized
> > > > header blob
> > > > > and
> > > > > > IIUC
> > > > > > > you dont
> > > > > > > > need any
> > > > > > > > > >> > optionals
> > > > > > > > > >> > >>
> > anymore.
> > > > > > > > > >> > >> >>
> > > >
> > > > > > > > > >> > >> >>
> > > > 3. yes,
> > > > we can :-)
> > > > > > > > > >> > >> >>
> > > >
> > > > > > > > > >> > >> >>
> > > > On Tue,
> > > > Feb 14, 2017 at
> > > > > > 11:56
> > > > > > > PM, Michael
> > > > > > > > > Pearce <
> > > > > > > > > >> > >> >>
> >
> > > > Michael.Pearce@ig.com>
> > > > > > > > > >> > >> >>
> > > > wrote:
> > > > > > > > > >> > >> >>
> > > >
> > > > > > > > > >> > >> >>
> > > > > Hi
> > > Jay,
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > > > > >> > >> >>
> > > > > 1)
> > > > There was some
> > > > > > initial
> > > > > > > debate on the
> > > > > > > > value
> > > > > > > > > >> > part, as youll
> > > > > > > > > >> > >>
> note
> > > > > > > > > >> > >> >>
> >
> > String,
> > > > > > > > > >> > >> >>
> > > > > String
> > > > headers were
> > > > > > > discounted early
> > > > > > > > on. The
> > > > > > > > > >> > reason for
> this
> > > > > > > > > >> > >> is
> > > > > > > > > >> > >> >>
> > > >
> > > > flexibility
> > > > > > > > > >> > >> >>
> > > > > and
> > > > keeping in line
> > > > > > with the
> > > > > > > > flexibility of
> > > > > > > > > key,
> > > > > > > > > >> > value of the
> > > > > > > > > >> > >> >>
> >
> > message
> > > > > > > > > >> > >> >>
> > > > > object
> > > > itself. I don’t
> > > > > > think
> > > > > > > it takes
> > > > > > > > away
> > > > > > > > > from
> > > > > > > > > >> an
> > > > > > > > > >> > ecosystem
> > > > > > > > > >> > >> as
> > > > > > > > > >> > >> >>
> >
> > each
> > > > > > > > > >> > >> >>
> > > > plugin
> > > > > > > > > >> > >> >>
> > > > > will
> > > > care for their
> > > > > own
> > > > > > key,
> > > > > > > this way
> > > > > > > > ints,
> > > > > > > > > >> > booleans ,
> > exotic
> > > > > > > > > >> > >> >>
> >
> > custom
> > > > > > > > > >> > >> >>
> > > > binary
> > > > > > > > > >> > >> >>
> > > > > can
> > > all
> > > > be catered
> > > > > for=.
> > > > > > > > > >> > >> >>
> > > > > a. If
> > > > you really
> > > > > wanted
> > > > > > to
> > > > > > > push for a
> > > > > > > > typed
> > > > > > > > > >> value
> > > > > > > > > >> > interface,
> > > > > > > > > >> > >> I
> > > > > > > > > >> > >> >>
> >
> > wouldn’t
> > > > > > > > > >> > >> >>
> > > > > want
> > > > just String
> > > > > values
> > > > > > > supported, but
> > > > > > > > the the
> > > > > > > > > >> > primatives
> > > > > > > > > >> > >>
> plus
> > > > > > > > > >> > >> >>
> >
> > string and
> > > > > > > > > >> > >> >>
> > > > > also
> > > > still keeping the
> > > > > > > ability to have a
> > > > > > > > > binary
> > > > > > > > > >> > for custom
> > > > > > > > > >> > >> >>
> >
> > binaries that
> > > > > > > > > >> > >> >>
> > > > > some
> > > > organisations may
> > > > > > have.
> > > > > > > > > >> > >> >>
> > > > > i. I
> > > > have written this
> > > > > > slight
> > > > > > > > alternative
> > > > > > > > > here,
> > > > > > > > > >> > >> >>
> > > >
> > > > > https://cwiki.apache.org/
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > >> >
> > 82+-+Add+Record+Headers+-+
> > > Typed
> > > > > > > > > >> > >> >>
> > > > > ii.
> > > > Essentially the
> > > > > > value
> > > > > > > bytes, has a
> > > > > > > > leading
> > > > > > > > > >> > byte overhead.
> > > > > > > > > >> > >> >>
> > > > > 1.
> > > > This tells you
> > > > > what
> > > > > > type
> > > > > > > the value
> > > > > > > > is,
> > > > > > > > > >> before
> > > > > > > > > >> > reading
> > > > > > > > > >> > >> the
> > rest
> > > > > > > > > >> > >> >>
> >
> > of the
> > > > > > > > > >> > >> >>
> > > > > bytes,
> > > > allowing
> > > > > > > >
> > serialisation/deserialization
> > > > > > > > > to
> > > > > > > > > >> > and from the
> > > > > > > > > >> > >> >>
> >
> > primitives,
> > > > > > > > > >> > >> >>
> > > > > string
> > > > and byte[].
> > > > > This
> > > > > > is
> > > > > > > akin to some
> > > > > > > > other
> > > > > > > > > >> > messaging
> > > > > > > > > >> > >>
> > systems.
> > > > > > > > > >> > >> >>
> > > > > 2) We
> > > > are making it
> > > > > > optional,
> > > > > > > so that
> > > > > > > > for
> > > > > > > > > those
> > > > > > > > > >> > not wanting
> > > > > > > > > >> > >> >>
> >
> > headers have
> > > > > > > > > >> > >> >>
> > > > 0
> > > > > > > > > >> > >> >>
> > > > > bytes
> > > > overhead (think
> > > > > > of it
> > > > > > > as a feature
> > > > > > > > > flag),
> > > > > > > > > >> I
> > > > > > > > > >> > don’t
> > > > > > > > > >> > >>
> > think this
> > > > > > > > > >> > >> >>
> >
> > is
> > > > > > > > > >> > >> >>
> > > > >
> > > > complex, especially if
> > > > > > > comparing to
> > > > > > > > changes
> > > > > > > > > >> > proposed in
> > > > > > > > > >> > >>
> > other kips
> > > > > > > > > >> > >> >>
> >
> > like
> > > > > > > > > >> > >> >>
> > > > >
> > > kip-98.
> > > > > > > > > >> > >> >>
> > > > > a. If
> > > > you really
> > > > > really
> > > > > > don’t
> > > > > > > like
> > > > > > > > this, we
> > > > > > > > > can
> > > > > > > > > >> > drop it, but
> > > > > > > > > >> > >> it
> > > > > > > > > >> > >> >>
> >
> > would
> > > > > > > > > >> > >> >>
> > > > mean
> > > > > > > > > >> > >> >>
> > > > > buying
> > > > into 4 bytes
> > > > > > extra
> > > > > > > overhead for
> > > > > > > > users
> > > > > > > > > who
> > > > > > > > > >> > do not want
> > > > > > > > > >> > >> to
> > use
> > > > > > > > > >> > >> >>
> > > > headers.
> > > > > > > > > >> > >> >>
> > > > > 3) In
> > > > the summary yes,
> > > > > > it is
> > > > > > > at a higher
> > > > > > > > > level,
> > > > > > > > > >> > but I think
> > > > > > > > > >> > >>
> this
> > > > > > > > > >> > >> >>
> >
> > is well
> > > > > > > > > >> > >> >>
> > > > >
> > > > documented in the
> > > > > > proposed
> > > > > > > changes
> > > > > > > > section.
> > > > > > > > > >> > >> >>
> > > > > a.
> > > > Added getHeaders
> > > > > > method to
> > > > > > > > > Producer/Consumer
> > > > > > > > > >> > record (that
> > > > > > > > > >> > >> is
> > it)
> > > > > > > > > >> > >> >>
> > > > > b.
> > > > We’ve also detailed
> > > > > > the
> > > > > > > new Headers
> > > > > > > > class
> > > > > > > > > >> that
> > > > > > > > > >> > this method
> > > > > > > > > >> > >> >>
> >
> > returns
> > > > > > > > > >> > >> >>
> > > > that
> > > > > > > > > >> > >> >>
> > > > >
> > > > encapsulates the
> > > > > headers
> > > > > > > protocol and
> > > > > > > > logic.
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > > > > >> > >> >>
> > > > > Best,
> > > > > > > > > >> > >> >>
> > > > > Mike
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > > > > >> > >> >>
> > > > >
> > > > ==Original questions
> > > > > > from the
> > > > > > > vote
> > > > > > > > thread from
> > > > > > > > > >> > Jay.==
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > > > > >> > >> >>
> > > > > Couple
> > > > of things I
> > > > > > think we
> > > > > > > still need
> > > > > > > > to work
> > > > > > > > > >> out:
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > > > > >> > >> >>
> > > > > 1.
> > > I
> > > > think we agree
> > > > > > about
> > > > > > > the key,
> > > > > > > > but I
> > > > > > > > > >> think
> > > > > > > > > >> > we haven't
> > > > > > > > > >> > >> >>
> >
> > talked about
> > > > > > > > > >> > >> >>
> > > > > the
> > > > value yet. I
> > > > > > think if
> > > > > > > our goal
> > > > > > > > is an
> > > > > > > > > open
> > > > > > > > > >> > ecosystem
> > > > > > > > > >> > >> of
> > these
> > > > > > > > > >> > >> >>
> > > > header
> > > > > > > > > >> > >> >>
> > > > >
> > > > spread across many
> > > > > > plugins
> > > > > > > from many
> > > > > > > > > systems
> > > > > > > > > >> we
> > > > > > > > > >> > should
> > > > > > > > > >> > >>
> > consider
> > > > > > > > > >> > >> >>
> >
> > making
> > > > > > > > > >> > >> >>
> > > > > this
> > > > > > > > > >> > >> >>
> > > > > a
> > > > string as well so
> > > > > > it can
> > > > > > > be
> > > > > > > > printed, set
> > > > > > > > > >> via
> > > > > > > > > >> > a UI, set
> > > > > > > > > >> > >> in
> > > > > > > > > >> > >> >>
> >
> > config,
> > > > > > > > > >> > >> >>
> > > > etc.
> > > > > > > > > >> > >> >>
> > > > >
> > > > Basically
> > > > > encouraging
> > > > > > > pluggable
> > > > > > > > > serialization
> > > > > > > > > >> > formats
> > > > > > > > > >> > >>
> here
> > will
> > > > > > > > > >> > >> >>
> >
> > lead
> > > > > > > > > >> > >> >>
> > > > to
> > > > > > > > > >> > >> >>
> > > > > a
> > > > > > > > > >> > >> >>
> > > > > bit
> > > > of a tower of
> > > > > > babel.
> > > > > > > > > >> > >> >>
> > > > > 2.
> > > > This proposal
> > > > > > still
> > > > > > > includes a
> > > > > > > > pretty
> > > > > > > > > big
> > > > > > > > > >> > change to our
> > > > > > > > > >> > >> >>
> > > >
> > > > serialization
> > > > > > > > > >> > >> >>
> > > > > and
> > > > protocol
> > > > > > definition
> > > > > > > layer.
> > > > > > > > Essentially
> > > > > > > > > >> it is
> > > > > > > > > >> > >>
> > introducing an
> > > > > > > > > >> > >> >>
> > > > optional
> > > > > > > > > >> > >> >>
> > > > >
> > > > type, where the
> > > > > > format is
> > > > > > > data
> > > > > > > > dependent. I
> > > > > > > > > >> > think this is
> > > > > > > > > >> > >> >>
> >
> > actually a
> > > > > > > > > >> > >> >>
> > > > big
> > > > > > > > > >> > >> >>
> > > > >
> > > > change though it
> > > > > > doesn't
> > > > > > > seem like
> > > > > > > > it. It
> > > > > > > > > >> means
> > > > > > > > > >> > you can no
> > > > > > > > > >> > >> >>
> >
> > longer
> > > > > > > > > >> > >> >>
> > > > >
> > > specify
> > > > > > > > > >> > >> >>
> > > > >
> > > this
> > > > type with our
> > > > > > type
> > > > > > > definition
> > > > > > > > DSL, and
> > > > > > > > > >> > likewise it
> > > > > > > > > >> > >>
> > requires
> > > > > > > > > >> > >> >>
> > > > custom
> > > > > > > > > >> > >> >>
> > > > >
> > > > handling in client
> > > > > > libs.
> > > > > > > This isn't
> > > > > > > > a huge
> > > > > > > > > >> > thing, since
> > > > > > > > > >> > >> the
> > > > > > > > > >> > >> >>
> >
> > Record
> > > > > > > > > >> > >> >>
> > > > >
> > > > definition is
> > > > > custom
> > > > > > > anyway, but I
> > > > > > > > think
> > > > > > > > > this
> > > > > > > > > >> > kind of
> > > > > > > > > >> > >>
> > protocol
> > > > > > > > > >> > >> >>
> > > > >
> > > > inconsistency is
> > > > > very
> > > > > > > non-desirable
> > > > > > > > and
> > > > > > > > > ties
> > > > > > > > > >> > you to
> > > > > > > > > >> > >>
> > hand-coding
> > > > > > > > > >> > >> >>
> > > > things.
> > > > > > > > > >> > >> >>
> > > > > I
> > > > > > > > > >> > >> >>
> > > > >
> > > > think the type
> > > > > should
> > > > > > > instead by [Key
> > > > > > > > > Value]
> > > > > > > > > >> in
> > > > > > > > > >> > our BNF,
> > > > > > > > > >> > >>
> where
> > > > > > > > > >> > >> >>
> >
> > key and
> > > > > > > > > >> > >> >>
> > > > >
> > > > value are both
> > > > > short
> > > > > > > strings as used
> > > > > > > > > >> elsewhere.
> > > > > > > > > >> > This
> > > > > > > > > >> > >>
> > brings it
> > > > > > > > > >> > >> >>
> >
> > in line
> > > > > > > > > >> > >> >>
> > > > > with
> > > > > > > > > >> > >> >>
> > > > > the
> > > > rest of the
> > > > > > protocol.
> > > > > > > > > >> > >> >>
> > > > > 3.
> > > > Could we get
> > > > > more
> > > > > > > specific about
> > > > > > > > the
> > > > > > > > > exact
> > > > > > > > > >> > Java API
> > > > > > > > > >> > >>
> > change to
> > > > > > > > > >> > >> >>
> > > > >
> > > > ProducerRecord,
> > > > > > > ConsumerRecord,
> > > > > > > > Record,
> > > > > > > > > etc?
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > > > > >> > >> >>
> > > > > -Jay
> > > > > > > > > >> > >> >>
> > > > >
> > > > > > > > > >> > >> >>
> > > >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >> >>
> >
> > The
> > > information
> > > > contained in
> > > > > > this
> > > > > > > email is
> > > > > > > > strictly
> > > > > > > > > >> > confidential
> and
> > > > > > > > > >> > >> for
> > > > > > > > > >> > >> >>
> >
> > the use of the
> > > > addressee only,
> > > > > > unless
> > > > > > > otherwise
> > > > > > > > > >> indicated.
> > > > > > > > > >> > If you
> > > > > > > > > >> > >> are
> > not
> > > > > > > > > >> > >> >>
> >
> > the intended
> > > > recipient, please
> > > > > > do not
> > > > > > > read,
> > > > > > > > copy, use
> > > > > > > > > or
> > > > > > > > > >> > disclose to
> > > > > > > > > >> > >>
> > others
> > > > > > > > > >> > >> >>
> >
> > this message
> > > or
> > > > any
> > > > > attachment.
> > > > > > > Please also
> > > > > > > > notify the
> > > > > > > > > >> > sender by
> > > > > > > > > >> > >>
> > replying
> > > > > > > > > >> > >> >>
> >
> > to this email
> > > > or by telephone
> > > > > > > (+44(020 7896
> > > > > > > > 0011) and
> > > > > > > > > >> then
> > > > > > > > > >> > delete
> > > > > > > > > >> > >> the
> > email
> > > > > > > > > >> > >> >>
> >
> > and any copies
> > > > of it.
> > > > > Opinions,
> > > > > > > conclusion
> > > > > > > > (etc) that
> > > > > > > > > do
> > > > > > > > > >> > not relate
> > > > > > > > > >> > >> to
> > the
> > > > > > > > > >> > >> >>
> >
> > official
> > > > business of this
> > > > > > company
> > > > > > > shall be
> > > > > > > > understood
> > > > > > > > > as
> > > > > > > > > >> > neither
> > > > > > > > > >> > >>
> > given nor
> > > > > > > > > >> > >> >>
> >
> > endorsed by
> > > it.
> > > > IG is a
> > > > > trading
> > > > > > name
> > > > > > > of IG
> > > > > > > > Markets
> > > > > > > > > >> Limited
> > > > > > > > > >> > (a company
> > > > > > > > > >> > >> >>
> >
> > registered in
> > > > England and
> > > > > Wales,
> > > > > > > company number
> > > > > > > > > >> 04008957)
> > > > > > > > > >> > and IG
> > > > > > > > > >> > >>
> Index
> > > > > > > > > >> > >> >>
> >
> > Limited (a
> > > > company registered
> > > > > in
> > > > > > > England and
> > > > > > > > Wales,
> > > > > > > > > >> > company number
> > > > > > > > > >> > >> >>
> >
> > 01190902).
> > > > Registered address
> > > > > at
> > > > > > > Cannon Bridge
> > > > > > > > House,
> > > > > > > > > 25
> > > > > > > > > >> > Dowgate
> > > > > > > > > >> > >>
> Hill,
> > > > > > > > > >> > >> >>
> >
> > London EC4R
> > > > 2YA. Both IG
> > > > > Markets
> > > > > > > Limited
> > > > > > > > (register
> > > > > > > > > >> number
> > > > > > > > > >> > 195355)
> > > > > > > > > >> > >> and
> > IG
> > > > > > > > > >> > >> >>
> >
> > Index Limited
> > > > (register number
> > > > > > > 114059) are
> > > > > > > > authorised
> > > > > > > > > >> and
> > > > > > > > > >> > regulated
> > > > > > > > > >> > >> by
> > the
> > > > > > > > > >> > >> >>
> >
> > Financial
> > > > Conduct Authority.
> > > > > > > > > >> > >> >>
> >
> > > > > > > > > >> > >>
> > > > > > > > > >> > >>
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> --
> > > > > > > > > >> > >>
> > Nacho - Ignacio
> > > > Solis -
> > > > > > isolis@igso.net
> > > > > > > > > >> > >>
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > The
> information
> > contained in
> > > > this email is
> > > > > > strictly
> > > > > > > > confidential and
> > > > > > > > > for
> > > > > > > > > >> > the use of the
> > addressee only,
> > > > unless
> > > > > otherwise
> > > > > > > indicated. If
> > > > > > > > you are
> > > > > > > > > >> not
> > > > > > > > > >> > the intended
> > recipient, please
> > > > do not read,
> > > > > > copy, use
> > > > > > > or
> > > > > > > > disclose to
> > > > > > > > > >> others
> > > > > > > > > >> > this message
> or
> > any
> > > attachment.
> > > > Please also
> > > > > > notify
> > > > > > > the sender by
> > > > > > > > > >> replying
> > > > > > > > > >> > to this email
> > or by telephone
> > > > (+44(020 7896
> > > > > > 0011) and
> > > > > > > then
> > > > > > > > delete the
> > > > > > > > > >> email
> > > > > > > > > >> > and any copies
> > of it.
> > > Opinions,
> > > > conclusion
> > > > > > (etc) that
> > > > > > > do not
> > > > > > > > relate to
> > > > > > > > > >> the
> > > > > > > > > >> > official
> > business of this
> > > > company shall be
> > > > > > understood
> > > > > > > as
> > > > > > > > neither given
> > > > > > > > > >> nor
> > > > > > > > > >> > endorsed by
> it.
> > IG is a
> > > trading
> > > > name of IG
> > > > > > Markets
> > > > > > > Limited (a
> > > > > > > > company
> > > > > > > > > >> > registered in
> > England and
> > > > Wales, company
> > > > > number
> > > > > > > 04008957) and
> > > > > > > > IG Index
> > > > > > > > > >> > Limited (a
> > company registered
> > > > in England and
> > > > > > Wales,
> > > > > > > company
> > > > > > > > number
> > > > > > > > > >> > 01190902).
> > Registered address
> > > > at Cannon Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > > Dowgate Hill,
> > > > > > > > > >> > London EC4R
> > 2YA. Both IG
> > > > Markets Limited
> > > > > > (register
> > > > > > > number
> > > > > > > > 195355) and
> > > > > > > > > IG
> > > > > > > > > >> > Index Limited
> > (register number
> > > > 114059) are
> > > > > > authorised
> > > > > > > and
> > > > > > > > regulated by
> > > > > > > > > >> the
> > > > > > > > > >> > Financial
> > Conduct Authority.
> > > > > > > > > >> >
> > > > > > > > > >> The information
> > contained in
> > > this
> > > > email is
> > > > > > strictly
> > > > > > > confidential
> > > > > > > > and for
> > > > > > > > > >> the use of the
> > addressee only,
> > > > unless otherwise
> > > > > > > indicated. If you
> > > > > > > > are
> > > > > > > > > not
> > > > > > > > > >> the intended
> > recipient, please
> > > do
> > > > not read,
> > > > > copy,
> > > > > > use
> > > > > > > or disclose
> > > > > > > > to
> > > > > > > > > others
> > > > > > > > > >> this message or
> > any attachment.
> > > > Please also
> > > > > > notify the
> > > > > > > sender by
> > > > > > > > > replying
> > > > > > > > > >> to this email or
> > by telephone
> > > > (+44(020 7896
> > > > > 0011)
> > > > > > and
> > > > > > > then delete
> > > > > > > > the
> > > > > > > > > email
> > > > > > > > > >> and any copies
> of
> > it. Opinions,
> > > > conclusion (etc)
> > > > > > that
> > > > > > > do not
> > > > > > > > relate to
> > > > > > > > > the
> > > > > > > > > >> official
> business
> > of this
> > > company
> > > > shall be
> > > > > > understood
> > > > > > > as neither
> > > > > > > > given
> > > > > > > > > nor
> > > > > > > > > >> endorsed by it.
> > IG is a trading
> > > > name of IG
> > > > > Markets
> > > > > > > Limited (a
> > > > > > > > company
> > > > > > > > > >> registered in
> > England and Wales,
> > > > company number
> > > > > > > 04008957) and IG
> > > > > > > > Index
> > > > > > > > > >> Limited (a
> > company registered in
> > > > England and
> > > > > > Wales,
> > > > > > > company number
> > > > > > > > > >> 01190902).
> > Registered address at
> > > > Cannon Bridge
> > > > > > House,
> > > > > > > 25 Dowgate
> > > > > > > > Hill,
> > > > > > > > > >> London EC4R 2YA.
> > Both IG Markets
> > > > Limited
> > > > > (register
> > > > > > > number 195355)
> > > > > > > > and IG
> > > > > > > > > >> Index Limited
> > (register number
> > > > 114059) are
> > > > > > authorised
> > > > > > > and
> > > > > > > > regulated by
> > > > > > > > > the
> > > > > > > > > >> Financial
> Conduct
> > Authority.
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > The information
> > contained in this
> > > > email is strictly
> > > > > > > confidential and
> > > > > > > > for the use of the
> > addressee only, unless
> > > > otherwise
> > > > > > indicated.
> > > > > > > If you are
> > > > > > > > not the intended
> > recipient, please do not
> > > > read, copy, use
> > > > > > or
> > > > > > > disclose to
> > > > > > > > others this message or
> any
> > attachment.
> > > > Please also notify
> > > > > > the
> > > > > > > sender by
> > > > > > > > replying to this email or
> > by telephone
> > > > (+44(020 7896
> > > > > 0011)
> > > > > > and
> > > > > > > then delete
> > > > > > > > the email and any copies
> > of it. Opinions,
> > > > conclusion
> > > > > (etc)
> > > > > > that
> > > > > > > do not
> > > > > > > > relate to the official
> > business of this
> > > > company shall be
> > > > > > > understood as
> > > > > > > > neither given nor
> endorsed
> > by it. IG is a
> > > > trading name of
> > > > > > IG
> > > > > > > Markets
> > > > > > > > Limited (a company
> > registered in England
> > > > and Wales,
> > > > > company
> > > > > > > number
> > > > > > > > 04008957) and IG Index
> > Limited (a company
> > > > registered in
> > > > > > England
> > > > > > > and Wales,
> > > > > > > > company number 01190902).
> > Registered
> > > > address at Cannon
> > > > > > Bridge
> > > > > > > House, 25
> > > > > > > > Dowgate Hill, London EC4R
> > 2YA. Both IG
> > > > Markets Limited
> > > > > > (register
> > > > > > > number
> > > > > > > > 195355) and IG Index
> > Limited (register
> > > > number 114059) are
> > > > > > > authorised and
> > > > > > > > regulated by the
> Financial
> > Conduct
> > > > Authority.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > The information contained in
> > this email is
> > > > strictly
> > > > > confidential
> > > > > > and
> > > > > > > for the use of the addressee only,
> > unless otherwise
> > > > indicated. If
> > > > > > you are
> > > > > > > not the intended recipient, please
> > do not read,
> > > > copy, use or
> > > > > > disclose to
> > > > > > > others this message or any
> > attachment. Please also
> > > > notify the
> > > > > sender
> > > > > > by
> > > > > > > replying to this email or by
> > telephone (+44(020
> > > 7896
> > > > 0011) and then
> > > > > > delete
> > > > > > > the email and any copies of it.
> > Opinions,
> > > conclusion
> > > > (etc) that do
> > > > > > not
> > > > > > > relate to the official business of
> > this company
> > > > shall be understood
> > > > > > as
> > > > > > > neither given nor endorsed by it.
> IG
> > is a trading
> > > > name of IG
> > > > > Markets
> > > > > > > Limited (a company registered in
> > England and Wales,
> > > > company number
> > > > > > > 04008957) and IG Index Limited (a
> > company
> > > registered
> > > > in England and
> > > > > > Wales,
> > > > > > > company number 01190902).
> Registered
> > address at
> > > > Cannon Bridge
> > > > > House,
> > > > > > 25
> > > > > > > Dowgate Hill, London EC4R 2YA. Both
> > IG Markets
> > > > Limited (register
> > > > > > number
> > > > > > > 195355) and IG Index Limited
> > (register number
> > > > 114059) are
> > > > > authorised
> > > > > > and
> > > > > > > regulated by the Financial Conduct
> > Authority.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in this email
> is
> > strictly
> > > > confidential and for
> > > > > > the use of the addressee only, unless
> > otherwise
> > > indicated.
> > > > If you are not
> > > > > > the intended recipient, please do not
> > read, copy, use or
> > > > disclose to
> > > > > others
> > > > > > this message or any attachment. Please
> > also notify the
> > > > sender by replying
> > > > > > to this email or by telephone (+44(020
> > 7896 0011) and
> > > then
> > > > delete the
> > > > > email
> > > > > > and any copies of it. Opinions,
> conclusion
> > (etc) that do
> > > > not relate to
> > > > > the
> > > > > > official business of this company shall
> be
> > understood as
> > > > neither given
> > > > > nor
> > > > > > endorsed by it. IG is a trading name of
> IG
> > Markets
> > > Limited
> > > > (a company
> > > > > > registered in England and Wales, company
> > number 04008957)
> > > > and IG Index
> > > > > > Limited (a company registered in England
> > and Wales,
> > > > company number
> > > > > > 01190902). Registered address at Cannon
> > Bridge House, 25
> > > > Dowgate Hill,
> > > > > > London EC4R 2YA. Both IG Markets Limited
> > (register number
> > > > 195355) and IG
> > > > > > Index Limited (register number 114059)
> are
> > authorised and
> > > > regulated by
> > > > > the
> > > > > > Financial Conduct Authority.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and
> > > > for the use of the addressee only, unless otherwise
> > indicated. If you are
> > > > not the intended recipient, please do not read, copy, use
> > or disclose to
> > > > others this message or any attachment. Please also notify
> > the sender by
> > > > replying to this email or by telephone (+44(020 7896
> 0011)
> > and then
> > > delete
> > > > the email and any copies of it. Opinions, conclusion
> (etc)
> > that do not
> > > > relate to the official business of this company shall be
> > understood as
> > > > neither given nor endorsed by it. IG is a trading name of
> > IG Markets
> > > > Limited (a company registered in England and Wales,
> > company number
> > > > 04008957) and IG Index Limited (a company registered in
> > England and
> > > Wales,
> > > > company number 01190902). Registered address at Cannon
> > Bridge House, 25
> > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register number
> > > > 195355) and IG Index Limited (register number 114059) are
> > authorised and
> > > > regulated by the Financial Conduct Authority.
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Michael,
Here's a link to the branch with the message format changes:
https://github.com/confluentinc/kafka/tree/exactly-once-message-format. The
class to modify to add headers is EosLogRecord (name will definitely change
prior to merging in trunk). This branch uses attributes to indicate null
key and value. I really don't have a strong preference for or against this
as long as we're consistent. I do feel it's kind of a pity to need the
signed version of the varint just to be able to indicate null. For headers,
it's an easier call because we don't actually need to represent null
headers (using size zero will work fine).
-Jason
On Sun, Feb 19, 2017 at 5:38 PM, Michael Pearce <Mi...@ig.com>
wrote:
> We can make it, implement Iterable<Header>, no concerns on this.
>
> Ive noted this on the KIP.
>
>
> On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Btw Jason, can you share the branch with the VarInts helper/util class
> that KIP-98 will be introducing and also will have the current modified
> (key and value lengths) message, so we can look at, thanks.
>
> On 20/02/2017, 01:11, "Michael Pearce" <Mi...@ig.com> wrote:
>
> RE “ProducerRecord and ConsumerRecord are immutable currently”. I
> beg to differ on this, if I set a byte[] as my value of my record, I can
> later call value() and then alter the bytes in the array. As such it is
> very much mutable.
>
> Headers also will anyhow by nature be mutable on produce, as you
> add to them.
>
>
> On 20/02/2017, 00:22, "Jason Gustafson" <ja...@confluent.io>
> wrote:
>
> Re; having a dedicated Headers object. I think it makes sense.
> Maybe it
> could implement Iterable<Header>?
>
> One thing I'm not too sure about is mutability. Both the
> ProducerRecord and
> ConsumerRecord are immutable currently. It would be nice to
> keep it that
> way. Perhaps Headers could itself be immutable and passed
> through the
> constructors?
>
> Just going through how this is, I don’t believe this would
> provide much
> > saving. If issue trying to resolve is around object/garbage
> creation of
> > Map.Entry objects.
>
>
> It's definitely better because we only allocate the space we
> will use, but
> I assume most of the actual memory cost is in the header data
> itself which
> we can't do much about. Maybe we can start with the simple
> array
> representation and optimize from there after we find it's
> worthwhile to do
> so? If we use the Headers object, we can change this later
> without breaking
> the API.
>
> -Jason
>
> On Sun, Feb 19, 2017 at 2:39 PM, radai <
> radai.rosenblatt@gmail.com> wrote:
>
> > "(By the way, doesn't it feel a bit odd that we seem to be
> designing a
> > feature which is optimized for people not using?)"
> >
> > very. this (i claim :-) ) is the result of very intense
> opposition to the
> > usefulness of the feature early on, and not a real design
> goal
> >
> > On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > wrote:
> >
> > > Whilst if having an array of a Header[] would mean getting
> this through,
> > > im happy changing this, as already done.
> > >
> > > Just going through how this is, I don’t believe this
> would provide much
> > > saving. If issue trying to resolve is around
> object/garbage creation of
> > > Map.Entry objects.
> > >
> > > All you’re doing here is replacing the equiv of a
> HashMap.Node
> > (Map.Entry)
> > > which simply holds reference to key and value objects with
> a custom
> > variant.
> > >
> > >
> > >
> > >
> > >
> > > On 19/02/2017, 20:22, "Michael Pearce" <
> Michael.Pearce@ig.com> wrote:
> > >
> > > On point 1 & 2 Ive updated KIP to show varints (and
> removed the bit
> > > flag). (on the assumption KIP 98 is getting the agreement
> the protocol is
> > > moving from int32 to varInts as standard)
> > >
> > > On point 3 ive updated to use an array of Header
> class, instead of a
> > > MultiMap in the Headers class object
> > >
> > >
> > >
> > > On 19/02/2017, 20:06, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > wrote:
> > >
> > > On points 1 and 2 I agree.
> > >
> > > This also affects kip-98, I should expect this
> resolved before
> > > that vote also passes. If it is accepted there (I’m
> assuming this is
> > > getting discussed on that KIP? As you’re driving the move
> to VarInts), I
> > am
> > > happy to make this KIP will simply follow suit to whatever
> is agreed in
> > > KIP-98.
> > >
> > > On 3) Agreed this is a simpler form, as long as no
> one is
> > > expecting hashmap lookup performance ( O(1) ), I still
> would prefer an
> > > encapsulated class, so if we find that holding it as an
> array in future
> > is
> > > causing some perf issues, the internals are not exposed to
> end users,
> > > allowing the internal structure to move to a map.
> > >
> > > Can we compromise on?
> > >
> > > class ConsumerRecord<K, V> {
> > > K key;
> > > V value;
> > > Headers headers;
> > > }
> > >
> > > class Headers {
> > > Header[] headers;
> > >
> > > add(String key, byte[] value)
> > > Collection<byte[]> get(String key)
> > > }
> > >
> > > class Header {
> > > String key;
> > > byte[] value;
> > > }
> > >
> > >
> > >
> > > On 19/02/2017, 18:54, "Jason Gustafson" <
> jason@confluent.io>
> > > wrote:
> > >
> > > >
> > > > headers dont "leak" into application code.
> they are useful
> > > to application
> > > > code as well.
> > >
> > >
> > > This is exactly what I have been trying to get
> at. The use
> > > cases documented
> > > here are middleware:
> > > https://cwiki.apache.org/
> confluence/display/KAFKA/A+
> > > Case+for+Kafka+Headers.
> > > If headers are intended for the application
> space as well,
> > the
> > > document
> > > should be updated accordingly so that it is an
> explicit
> > design
> > > goal and not
> > > an unfortunate design byproduct. There may be
> some
> > > disagreement on whether
> > > it _should_ be a design goal, but it may not
> matter much
> > since
> > > the current
> > > interceptor API is probably insufficient for
> middleware
> > > applications (I
> > > haven't thought of a way to do this that isn't
> cumbersome).
> > >
> > > That aside, the three unresolved points for me
> are the
> > > following:
> > >
> > > 1. The use of varints. As a one-off for use
> only with record
> > > headers, the
> > > case was weak, but if we are using them
> throughout the
> > message
> > > format, then
> > > we should do so here as well. The additional
> complexity is
> > > minimal and
> > > early performance testing fully justifies
> their use.
> > >
> > > 2. If varints are used, the case for using
> attributes to
> > > indicate null
> > > headers also becomes weak. We only add one
> additional byte in
> > > each message
> > > if there are no headers. Whatever the case,
> let us be
> > > consistent with how
> > > we treat null keys and values.
> > >
> > > 3. We have apparently agreed that the broker
> will validate
> > > headers (please
> > > add this to the KIP). That being the case, I
> would prefer to
> > > use Kafka's
> > > conventional format for arrays. The downside
> is that it takes
> > > more work to
> > > skip over the headers on the consumer, though
> it's unclear if
> > > the cost
> > > would matter in practice. Some concern was
> previously
> > > expressed about the
> > > allocation of maps. An alternative would be to
> use arrays,
> > i.e.
> > >
> > > class ConsumerRecord<K, V> {
> > > K key;
> > > V value;
> > > Header[] headers;
> > > }
> > >
> > > class Header {
> > > String key;
> > > byte[] value;
> > > }
> > >
> > > This would work nicely with the conventional
> array format and
> > > my guess is
> > > it would obviate the need do any lazy
> initialization. If we
> > > use the map as
> > > is currently documented, then it is possible
> with either
> > > representation to
> > > slice the headers and initialize them lazily.
> Either way, it
> > > might be a
> > > good idea to use a separate object to
> represent the headers
> > in
> > > case we need
> > > to modify it in the future in some way.
> > >
> > > (By the way, doesn't it feel a bit odd that we
> seem to be
> > > designing a
> > > feature which is optimized for people not
> using?)
> > >
> > >
> > > If we can resolve these points, then at least
> you will get my
> > > vote.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > > radai.rosenblatt@gmail.com> wrote:
> > >
> > > > headers dont "leak" into application code.
> they are useful
> > > to application
> > > > code as well.
> > > > IIUC samze currently has headers "in-V" and
> would just
> > > switch over to kafka
> > > > headers if they exist.
> > > > im sure plenty of other users of kafka would
> have a use for
> > > headers.
> > > > im pretty sure use cases exist around
> shuffling data
> > > into/out-of kafka
> > > > (kafka connect or equivalent) where metadata
> from one end
> > > could copied over
> > > > to the other (S3, for example uses http
> headers for
> > > user-accessible
> > > > metadata). it will be kafka client code
> getting/setting
> > > those headers. not
> > > > an interceptor.
> > > >
> > > > On Fri, Feb 17, 2017 at 1:41 PM, Michael
> Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > For APM single event tracing, need access
> to the header
> > at
> > > the point of
> > > > > processing on the processing thread.
> > > > >
> > > > > As such interceptors will not work/be
> suitable for these,
> > > due to the fact
> > > > > they act on the ConsumerRecords as a
> batch, before the
> > > handling thread
> > > > can
> > > > > split out and process per message which is
> the point
> > these
> > > tools will
> > > > need
> > > > > to continue to transaction tracing.
> > > > >
> > > > > Like wise tools and infra pieces will need
> access to the
> > > message outside
> > > > > the interceptor realm.
> > > > >
> > > > >
> > > > >
> > > > > On 17/02/2017, 21:26, "Jason Gustafson" <
> > > jason@confluent.io> wrote:
> > > > >
> > > > > >
> > > > > > That’s exactly what we’re doing the
> headers are a
> > > slice of bytes,
> > > > > which
> > > > > > then gets parsed later if needed, or
> can be parsed
> > > right away, the
> > > > > headers
> > > > > > is part of the protocol, so can
> still be validated
> > > if wanted.
> > > > > > If you had a header count then you
> would have to go
> > > through each
> > > > > header
> > > > > > key and value length value to work
> out how much to
> > > skip to get to
> > > > > say the
> > > > > > value or any future component in the
> message after
> > > the headers.
> > > > > Having it
> > > > > > as a byte[] with length value makes
> this a lot
> > > easier to skip.
> > > > >
> > > > >
> > > > > So the broker will parse the headers
> and validate
> > > them. Good. The
> > > > only
> > > > > reason remaining that I can see to
> leave the headers
> > > as a byte array
> > > > > is to
> > > > > make it easier for the client to skip
> past them. Are
> > > we sure this is
> > > > > not
> > > > > premature optimization? Are there any
> performance
> > > results which show
> > > > > that
> > > > > this is worthwhile?
> > > > >
> > > > > What’s the issue with exposing a
> method getHeaders on
> > > the
> > > > > producer/consumer
> > > > > > record? It doesn’t break anything.
> We don’t need
> > any
> > > special
> > > > version.
> > > > >
> > > > >
> > > > > See my previous explanation. What I am
> trying to
> > > resist is the
> > > > headers
> > > > > becoming a general application-level
> facility. The
> > > primary use case
> > > > as
> > > > > far
> > > > > as I can tell is middleware, which is
> the use case
> > > that the
> > > > > interceptors
> > > > > are providing.
> > > > >
> > > > > Current batch consumer model and
> consumer
> > interceptors
> > > don’t work
> > > > where
> > > > > > headers need to be acted on at per
> message level at
> > > time of
> > > > > processing,
> > > > > > very case is APM (the core one),
> where the header
> > > value is used to
> > > > > continue
> > > > > > tracing.
> > > > >
> > > > >
> > > > > I still don't understand the point
> about batching.
> > The
> > > consumer
> > > > > records are
> > > > > exposed as a batch in the consumer
> interceptor, but
> > > you can still
> > > > > iterate
> > > > > them individually. It is no different
> for the
> > consumer
> > > API itself.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 17, 2017 at 12:48 PM,
> Michael Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > Re:
> > > > > >
> > > > > > “ The point about creation of
> maps seems
> > > orthogonal. We can
> > > > still
> > > > > > represent
> > > > > > the headers as a slice of bytes
> until the time
> > > it is accessed.”
> > > > > >
> > > > > > That’s exactly what we’re doing the
> headers are a
> > > slice of bytes,
> > > > > which
> > > > > > then gets parsed later if needed, or
> can be parsed
> > > right away, the
> > > > > headers
> > > > > > is part of the protocol, so can
> still be validated
> > > if wanted.
> > > > > >
> > > > > > If you had a header count then you
> would have to go
> > > through each
> > > > > header
> > > > > > key and value length value to work
> out how much to
> > > skip to get to
> > > > > say the
> > > > > > value or any future component in the
> message after
> > > the headers.
> > > > > Having it
> > > > > > as a byte[] with length value makes
> this a lot
> > > easier to skip.
> > > > > >
> > > > > >
> > > > > > On 17/02/2017, 20:37, "Michael
> Pearce" <
> > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > What’s the issue with exposing a
> method
> > > getHeaders on the
> > > > > > producer/consumer record? It doesn’t
> break
> > anything.
> > > We don’t need
> > > > > any
> > > > > > special version.
> > > > > >
> > > > > > Current batch consumer model and
> consumer
> > > interceptors don’t
> > > > work
> > > > > > where headers need to be acted on at
> per message
> > > level at time of
> > > > > > processing, very case is APM (the
> core one), where
> > > the header value
> > > > > is used
> > > > > > to continue tracing. JMS/HTTP etc
> all expose these,
> > > without issues.
> > > > > I would
> > > > > > NOT want to lock this down to only
> be usable
> > > accessible via
> > > > > interceptors,
> > > > > > as we’d fail on one of the main
> goals.
> > > > > >
> > > > > > Regards
> > > > > > Mike
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 17/02/2017, 20:21, "Jason
> Gustafson" <
> > > jason@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > The point about creation of
> maps seems
> > > orthogonal. We can
> > > > > still
> > > > > > represent
> > > > > > the headers as a slice of
> bytes until the
> > > time it is
> > > > > accessed.
> > > > > >
> > > > > >
> > > > > > > Yes exactly we have access
> to the records
> > > thus why the
> > > > > header
> > > > > > should be
> > > > > > > accessible via it and not
> hidden for only
> > > interceptors to
> > > > > access.
> > > > > >
> > > > > >
> > > > > > As explained above, the
> point is to make
> > the
> > > intended usage
> > > > > clear.
> > > > > > Applications should continue
> to rely on the
> > > key/value
> > > > fields
> > > > > to
> > > > > > serialize
> > > > > > their own headers, and it
> would be more
> > > ideal if we can
> > > > avoid
> > > > > > leaking
> > > > > > third-party headers into
> applications. This
> > > is difficult to
> > > > > do
> > > > > > with the
> > > > > > current interceptors because
> they share the
> > > record objects
> > > > > with
> > > > > > the common
> > > > > > API. What I had in mind is
> something like
> > an
> > > extension of
> > > > the
> > > > > > current
> > > > > > interceptors which exposed a
> different
> > > object (e.g.
> > > > > > `RecordAndHeaders`).
> > > > > > The challenge is for MM-like
> use cases. Let
> > > me see if I can
> > > > > come
> > > > > > up with a
> > > > > > concrete proposal for that
> problem.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at
> 11:55 AM, Michael
> > > Pearce <
> > > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I am happy to move the
> definition of the
> > > header into the
> > > > > message
> > > > > > body, but
> > > > > > > would cause us not to lazy
> > > initialise/parse the headers,
> > > > as
> > > > > > obviously, we
> > > > > > > would have to traverse
> these reading the
> > > message.
> > > > > > >
> > > > > > > This was actually one of
> Jay’s requests:
> > > > > > >
> > > > > > > “ 2. I think we should
> think about
> > > creating the Map
> > > > > lazily to
> > > > > > avoid
> > > > > > > parsing out all the
> headers into
> > > little objects.
> > > > > HashMaps
> > > > > > themselves
> > > > > > > are
> > > > > > > kind of expensive and
> the consumer is
> > > very perf
> > > > > sensitive so
> > > > > > and making
> > > > > > > gazillions of hashmaps
> that may or
> > may
> > > not get used
> > > > is
> > > > > > probably a bad
> > > > > > > idea.”
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 17/02/2017, 19:44,
> "Michael Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Yes exactly we have
> access to the
> > > records thus why
> > > > the
> > > > > > header should
> > > > > > > be accessible via it and
> not hidden for
> > > only interceptors
> > > > > to
> > > > > > access.
> > > > > > >
> > > > > > > Sent using OWA for
> iPhone
> > > > > > >
> ______________________________
> > > __________
> > > > > > > From: Magnus Edenhill <
> > > magnus@edenhill.se>
> > > > > > > Sent: Friday, February
> 17, 2017
> > > 7:34:49 PM
> > > > > > > To:
> dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS]
> KIP-82 - Add
> > > Record Headers
> > > > > > >
> > > > > > > Big +1 on VarInts.
> > > > > > > CPUs are fast, memory
> is slow.
> > > > > > >
> > > > > > > I agree with Jason
> that we'll want to
> > > continue
> > > > > verifying
> > > > > > messages,
> > > > > > > including their
> headers, so while I
> > > appreciate the
> > > > > idea of
> > > > > > the opaque
> > > > > > > header blob it won't
> be useful in
> > > practice.
> > > > > > >
> > > > > > > /Magnus
> > > > > > >
> > > > > > > 2017-02-17 10:41
> GMT-08:00 Jason
> > > Gustafson <
> > > > > > jason@confluent.io>:
> > > > > > >
> > > > > > > > Sorry, my mistake.
> The consumer
> > > interceptor is per
> > > > > batch,
> > > > > > though I'm
> > > > > > > not
> > > > > > > > sure that's an
> actual limitation
> > > since you still
> > > > have
> > > > > > access to the
> > > > > > > > individual records.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017
> at 10:39 AM,
> > > Jason Gustafson <
> > > > > > > jason@confluent.io>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Re headers as byte
> array and
> > > future use by
> > > > broker.
> > > > > This
> > > > > > doesn't
> > > > > > > take away
> > > > > > > > >> from that at all.
> Nor makes it
> > > difficult at all
> > > > > in my
> > > > > > opinion.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Yeah, I didn't say
> it was
> > > difficult, only
> > > > awkward.
> > > > > You
> > > > > > wouldn't
> > > > > > > write the
> > > > > > > > > schema that way if
> you were
> > > planning to use it on
> > > > > the
> > > > > > brokers from
> > > > > > > the
> > > > > > > > > beginning. Note
> also that one of
> > > the benefits of
> > > > > letting
> > > > > > the broker
> > > > > > > > > understand headers
> is that it can
> > > validate that
> > > > > they are
> > > > > > properly
> > > > > > > > > formatted. If cost
> is the only
> > > concern, we should
> > > > > > confirm its
> > > > > > > impact
> > > > > > > > > through
> performance testing.
> > > > > > > > >
> > > > > > > > > One of the key use
> cases requires
> > > access on
> > > > > consume at
> > > > > > per
> > > > > > > event/message
> > > > > > > > >> level at the
> point that message
> > > is being
> > > > > processed, as
> > > > > > such the
> > > > > > > batch
> > > > > > > > >> interceptors and
> batch consume
> > > api isn't
> > > > > suitable. It
> > > > > > needs to be
> > > > > > > at the
> > > > > > > > >> record level.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I'm not sure I
> understand the
> > > point about
> > > > batching.
> > > > > > Interceptors
> > > > > > > are
> > > > > > > > > applied
> per-message, right?
> > > > > > > > >
> > > > > > > > > My intent on
> interceptors is to
> > > keep the usage of
> > > > > headers
> > > > > > > well-defined so
> > > > > > > > > that they don't
> start leaking
> > > unnecessarily into
> > > > > > applications. My
> > > > > > > guess
> > > > > > > > is
> > > > > > > > > that it's probably
> inevitable,
> > but
> > > isolating it
> > > > in
> > > > > the
> > > > > > > interceptors would
> > > > > > > > > at least give
> people a second
> > > thought before
> > > > > deciding to
> > > > > > use it.
> > > > > > > The main
> > > > > > > > > challenge in my
> mind is figuring
> > > out how an MM
> > > > use
> > > > > case
> > > > > > would
> > > > > > > work. It
> > > > > > > > > would be more
> cumbersome to
> > > replicate headers
> > > > > through an
> > > > > > > interceptor,
> > > > > > > > > though arguably MM
> should be
> > > working at a lower
> > > > > level
> > > > > > anyway.
> > > > > > > > >
> > > > > > > > > -Jason
> > > > > > > > >
> > > > > > > > > On Fri, Feb 17,
> 2017 at 10:16 AM,
> > > Michael Pearce
> > > > <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Re headers
> available on the
> > > record va
> > > > > interceptors only
> > > > > > > > >>
> > > > > > > > >> One of the key
> use cases
> > requires
> > > access on
> > > > > consume at
> > > > > > per
> > > > > > > event/message
> > > > > > > > >> level at the
> point that message
> > > is being
> > > > > processed, as
> > > > > > such the
> > > > > > > batch
> > > > > > > > >> interceptors and
> batch consume
> > > api isn't
> > > > > suitable. It
> > > > > > needs to be
> > > > > > > at the
> > > > > > > > >> record level.
> > > > > > > > >>
> > > > > > > > >> This anyhow is
> similar to
> > > jms/http/amqp where
> > > > > headers
> > > > > > are
> > > > > > > available to
> > > > > > > > >> consuming
> applications.
> > > > > > > > >>
> > > > > > > > >> Re headers as
> byte array and
> > > future use by
> > > > > broker. This
> > > > > > doesn't
> > > > > > > take
> > > > > > > > away
> > > > > > > > >> from that at all.
> Nor makes it
> > > difficult at all
> > > > > in my
> > > > > > opinion.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Sent using OWA
> for iPhone
> > > > > > > > >>
> ______________________________
> > > __________
> > > > > > > > >> From: Jason
> Gustafson <
> > > jason@confluent.io>
> > > > > > > > >> Sent: Friday,
> February 17, 2017
> > > 5:55:42 PM
> > > > > > > > >> To:
> dev@kafka.apache.org
> > > > > > > > >> Subject: Re:
> [DISCUSS] KIP-82 -
> > > Add Record
> > > > Headers
> > > > > > > > >>
> > > > > > > > >> >
> > > > > > > > >> > Would you be
> proposing in
> > > KIP-98 to convert
> > > > the
> > > > > other
> > > > > > message
> > > > > > > int’s
> > > > > > > > (key
> > > > > > > > >> > length, value
> length) also to
> > > varint to keep
> > > > it
> > > > > > uniform.
> > > > > > > > >> > Also I assume
> there will be a
> > > static or helper
> > > > > method
> > > > > > made to
> > > > > > > > write/read
> > > > > > > > >> > these in the
> client and
> > server.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Yes, that is what
> we are
> > > proposing, so using
> > > > > varints
> > > > > > for headers
> > > > > > > would
> > > > > > > > be
> > > > > > > > >> consistent with
> the rest of the
> > > message. We have
> > > > > used
> > > > > > static
> > > > > > > helper
> > > > > > > > >> methods
> > > > > > > > >> in our prototype
> implementation.
> > > > > > > > >>
> > > > > > > > >> The cost of
> parsing, we want to
> > > parse/interpret
> > > > > the
> > > > > > headers
> > > > > > > lazily (this
> > > > > > > > >> is
> > > > > > > > >> > a key point
> brought up earlier
> > > in discussions)
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> I'm a bit
> skeptical of this. Has
> > > anyone done the
> > > > > > performance
> > > > > > > testing? I
> > > > > > > > >> can
> > > > > > > > >> probably
> implement it and test
> > it
> > > if no one else
> > > > > has. I
> > > > > > was also
> > > > > > > under
> > > > > > > > the
> > > > > > > > >> impression that
> there may be use
> > > cases down the
> > > > > road
> > > > > > where the
> > > > > > > broker
> > > > > > > > >> would
> > > > > > > > >> need to interpret
> headers. That
> > > wouldn't be off
> > > > > the
> > > > > > table in the
> > > > > > > future
> > > > > > > > if
> > > > > > > > >> it's represented
> as bytes, but
> > it
> > > would be quite
> > > > > a bit
> > > > > > more
> > > > > > > awkward,
> > > > > > > > >> right?
> > > > > > > > >>
> > > > > > > > >> By the way, one
> question I have
> > > been wondering
> > > > > about. My
> > > > > > > understanding
> > > > > > > > is
> > > > > > > > >> that headers are
> primarily for
> > > use cases where a
> > > > > > third-party
> > > > > > > components
> > > > > > > > >> wants to enrich
> messages without
> > > needing to
> > > > > understand
> > > > > > or modify
> > > > > > > the
> > > > > > > > >> schema
> > > > > > > > >> of the message
> key and value.
> > For
> > > the
> > > > applications
> > > > > > which directly
> > > > > > > > produce
> > > > > > > > >> and consume the
> messages and
> > > control the
> > > > key/value
> > > > > > schema
> > > > > > > directly, it
> > > > > > > > >> seems we would
> rather have them
> > > implement
> > > > headers
> > > > > > directly in
> > > > > > > their own
> > > > > > > > >> schema. Supposing
> for the sake
> > of
> > > argument that
> > > > > it was
> > > > > > possible,
> > > > > > > my
> > > > > > > > >> question is
> whether it be
> > > sufficient to expose
> > > > the
> > > > > > headers in the
> > > > > > > > >> interceptor API
> and not in the
> > > common API?
> > > > > > > > >>
> > > > > > > > >> -Jason
> > > > > > > > >>
> > > > > > > > >> On Fri, Feb 17,
> 2017 at 3:26 AM,
> > > Michael Pearce
> > > > <
> > > > > > > Michael.Pearce@ig.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > On the point of
> varInts
> > > > > > > > >> >
> > > > > > > > >> > Would you be
> proposing in
> > > KIP-98 to convert
> > > > the
> > > > > other
> > > > > > message
> > > > > > > int’s
> > > > > > > > (key
> > > > > > > > >> > length, value
> length) also to
> > > varint to keep
> > > > it
> > > > > > uniform.
> > > > > > > > >> > Also I assume
> there will be a
> > > static or helper
> > > > > method
> > > > > > made to
> > > > > > > > write/read
> > > > > > > > >> > these in the
> client and
> > server.
> > > > > > > > >> >
> > > > > > > > >> > Cheers
> > > > > > > > >> > Mike
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On 17/02/2017,
> 11:22, "Michael
> > > Pearce" <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > On the
> point re: headers
> > in
> > > the message
> > > > > protocol
> > > > > > being a
> > > > > > > byte
> > > > > > > > array
> > > > > > > > >> > and not a count
> of elements
> > > followed by the
> > > > > elements.
> > > > > > Again
> > > > > > > this was
> > > > > > > > >> >
> discussed/argued previously.
> > > > > > > > >> >
> > > > > > > > >> > It was
> agreed on for a few
> > > reasons some of
> > > > > which
> > > > > > you have
> > > > > > > > obviously
> > > > > > > > >> > picked up on:
> > > > > > > > >> >
> > > > > > > > >> > Broker is
> able to pass it
> > > through opaquely
> > > > > > > > >> > The cost of
> parsing, we
> > > want to
> > > > > parse/interpret
> > > > > > the headers
> > > > > > > lazily
> > > > > > > > >> > (this is a key
> point brought
> > up
> > > earlier in
> > > > > > discussions)
> > > > > > > > >> > Headers can
> be copied from
> > > consumer record
> > > > > to
> > > > > > producer
> > > > > > > record (aka
> > > > > > > > >> > mirror makers
> etc) without
> > > parsing if no
> > > > > changes are
> > > > > > being made
> > > > > > > or
> > > > > > > > being
> > > > > > > > >> > looked at.
> > > > > > > > >> > Keeps the
> broker agnostic
> > > to the format
> > > > > > > > >> > You need an
> int32 either
> > > for the byte size
> > > > > of the
> > > > > > headers,
> > > > > > > or for
> > > > > > > > >> the
> > > > > > > > >> > count of
> elements, so
> > overheads
> > > are the same,
> > > > > but
> > > > > > with going
> > > > > > > with an
> > > > > > > > >> opaque
> > > > > > > > >> > byte array has
> the above
> > > advantages.
> > > > > > > > >> >
> > > > > > > > >> > Cheers
> > > > > > > > >> > Mike
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On
> 17/02/2017, 02:50,
> > > "Jason Gustafson" <
> > > > > > jason@confluent.io
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >> >
> > > > > > > > >> > Sorry,
> should have
> > > noted that the
> > > > > performance
> > > > > > testing
> > > > > > > was done
> > > > > > > > >> > using the
> > > > > > > > >> >
> producer performance
> > > tool shipped with
> > > > > Kafka.
> > > > > > > > >> >
> > > > > > > > >> > -Jason
> > > > > > > > >> >
> > > > > > > > >> > On Thu,
> Feb 16, 2017
> > at
> > > 6:44 PM, Jason
> > > > > > Gustafson <
> > > > > > > > >> >
> jason@confluent.io> wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hey
> Nacho,
> > > > > > > > >> > >
> > > > > > > > >> > > I've
> compared
> > > performance of our
> > > > > KIP-98
> > > > > > > implementation with
> > > > > > > > >> and
> > > > > > > > >> > without
> > > > > > > > >> > >
> varints. For
> > messages
> > > around 128
> > > > > bytes, we
> > > > > > see an
> > > > > > > increase
> > > > > > > > in
> > > > > > > > >> > throughput of
> > > > > > > > >> > > about
> 30% using the
> > > default
> > > > > configuration
> > > > > > settings.
> > > > > > > At 256
> > > > > > > > >> > bytes, the
> > > > > > > > >> > >
> increase is around
> > > 16%. Obviously
> > > > the
> > > > > > performance
> > > > > > > converge
> > > > > > > > as
> > > > > > > > >> > messages get
> > > > > > > > >> > >
> larger, but it seems
> > > well worth the
> > > > > cost.
> > > > > > Note that
> > > > > > > we are
> > > > > > > > >> also
> > > > > > > > >> > seeing a
> > > > > > > > >> > >
> substantial
> > > performance increase
> > > > > against
> > > > > > trunk
> > > > > > > primarily
> > > > > > > > >> because
> > > > > > > > >> > of the
> > > > > > > > >> > > much
> more efficient
> > > packing that
> > > > > varints
> > > > > > provide us.
> > > > > > > > Anything
> > > > > > > > >> > adding to
> > > > > > > > >> > >
> message overhead,
> > > such as record
> > > > > headers,
> > > > > > would only
> > > > > > > > increase
> > > > > > > > >> > the relative
> > > > > > > > >> > >
> difference. (Of
> > > course take these
> > > > > numbers
> > > > > > with a
> > > > > > > grain of
> > > > > > > > salt
> > > > > > > > >> > since I have
> > > > > > > > >> > > only
> used the
> > default
> > > settings with
> > > > > both
> > > > > > the producer
> > > > > > > and
> > > > > > > > >> broker
> > > > > > > > >> > on my
> > > > > > > > >> > > local
> machine. We
> > > intend to provide
> > > > > more
> > > > > > extensive
> > > > > > > > performance
> > > > > > > > >> > details as
> > > > > > > > >> > > part
> of the work for
> > > KIP-98.)
> > > > > > > > >> > >
> > > > > > > > >> > > The
> implementation
> > we
> > > are using is
> > > > > from
> > > > > > protobuf (
> > > > > > > > >> > >
> > > https://developers.google.com/
> > > > > > > >
> protocol-buffers/docs/encoding
> > > > > > > > >> ),
> > > > > > > > >> > which is
> > > > > > > > >> > > also
> used in HBase.
> > > It is trivial to
> > > > > > implement and as
> > > > > > > far
> > > > > > > > as I
> > > > > > > > >> > know doesn't
> > > > > > > > >> > >
> suffer from the
> > > aliasing problem you
> > > > > are
> > > > > > describing. I
> > > > > > > > checked
> > > > > > > > >> > with Magnus
> > > > > > > > >> > > (the
> author of
> > > librdkafka) and he
> > > > > agreed
> > > > > > that the
> > > > > > > savings
> > > > > > > > >> seemed
> > > > > > > > >> > worth the
> > > > > > > > >> > > cost
> of
> > > implementation.
> > > > > > > > >> > >
> > > > > > > > >> > > -Jason
> > > > > > > > >> > >
> > > > > > > > >> > > On
> Thu, Feb 16, 2017
> > > at 4:32 PM,
> > > > > Ignacio
> > > > > > Solis <
> > > > > > > > >> isolis@igso.net>
> > > > > > > > >> > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > >>
> -VarInts
> > > > > > > > >> > >>
> > > > > > > > >> > >> I'm
> one of the
> > > people (if not the
> > > > > most)
> > > > > > opposed to
> > > > > > > VarInts.
> > > > > > > > >> > VarInts
> > > > > > > > >> > >> have
> a place, but
> > > this is not it.
> > > > > (We
> > > > > > had a large
> > > > > > > > >> discussion
> > > > > > > > >> > about
> > > > > > > > >> > >> them
> at the
> > > beginning of KIP-82
> > > > time)
> > > > > > > > >> > >>
> > > > > > > > >> > >> If
> anybody has real
> > > life
> > > > performance
> > > > > > numbers of
> > > > > > > VarInts
> > > > > > > > >> > improving
> > > > > > > > >> > >>
> things or
> > > significantly reducing
> > > > > resources
> > > > > > I would
> > > > > > > like to
> > > > > > > > >> know
> > > > > > > > >> > what
> > > > > > > > >> > >> that
> case may be.
> > > Yes, you can save
> > > > > some
> > > > > > bytes here
> > > > > > > and
> > > > > > > > >> there,
> > > > > > > > >> > but
> > > > > > > > >> > >> this
> is probably
> > > insignificant to
> > > > the
> > > > > > overall system
> > > > > > > > behavior
> > > > > > > > >> > and
> > > > > > > > >> > >>
> storage
> > > requirements. -- I say
> > > > this
> > > > > with
> > > > > > respect to
> > > > > > > using
> > > > > > > > >> > VarInts in
> > > > > > > > >> > >> the
> protocol
> > itself,
> > > not as part of
> > > > > the
> > > > > > data.
> > > > > > > > >> > >>
> > > > > > > > >> > >>
> VarInts require you
> > > to parse the
> > > > Int
> > > > > > before using it
> > > > > > > and
> > > > > > > > >> > depending on
> > > > > > > > >> > >> the
> encoding they
> > > can suffer from
> > > > > aliasing
> > > > > > (multiple
> > > > > > > > >> > representations
> > > > > > > > >> > >> for
> the same
> > value).
> > > > > > > > >> > >>
> > > > > > > > >> > >> Why
> add complexity?
> > > > > > > > >> > >>
> > > > > > > > >> > >> Nacho
> > > > > > > > >> > >>
> > > > > > > > >> > >>
> > > > > > > > >> > >> On
> Thu, Feb 16,
> > 2017
> > > at 10:29 AM,
> > > > > Colin
> > > > > > McCabe <
> > > > > > > > >> >
> cmccabe@apache.org>
> > > > > > > > >> > >>
> wrote:
> > > > > > > > >> > >> > +1
> for varints
> > > here-- it would
> > > > save
> > > > > > quite a bit of
> > > > > > > space.
> > > > > > > > >> > They are
> > > > > > > > >> > >> >
> pretty quick to
> > > implement as
> > > > well.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > I
> think it makes
> > > sense for values
> > > > > to be
> > > > > > byte
> > > > > > > arrays.
> > > > > > > > Users
> > > > > > > > >> > might want
> > > > > > > > >> > >> > to
> attach
> > > arbitrary payloads;
> > > > they
> > > > > > shouldn't be
> > > > > > > forced to
> > > > > > > > >> > serialize
> > > > > > > > >> > >> >
> everything to
> > Java
> > > strings.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> best,
> > > > > > > > >> > >> >
> Colin
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > On
> Thu, Feb 16,
> > > 2017, at 09:52,
> > > > > Jason
> > > > > > Gustafson
> > > > > > > wrote:
> > > > > > > > >> > >> >>
> Hey Michael,
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> Hmm, I guess the
> > > point of
> > > > > representing
> > > > > > it as
> > > > > > > bytes is to
> > > > > > > > >> > allow the
> > > > > > > > >> > >>
> broker
> > > > > > > > >> > >> >>
> to pass it
> > > through opaquely? Is
> > > > > the
> > > > > > cost of
> > > > > > > parsing
> > > > > > > > them a
> > > > > > > > >> > concern, or
> > > > > > > > >> > >> >>
> are
> > > > > > > > >> > >> >>
> we simply trying
> > > to ensure that
> > > > > the
> > > > > > broker stays
> > > > > > > > agnostic
> > > > > > > > >> to
> > > > > > > > >> > the
> > > > > > > > >> > >>
> format?
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> On varints, I
> > > think adding
> > > > > support for
> > > > > > them makes
> > > > > > > less
> > > > > > > > >> sense
> > > > > > > > >> > for an
> > > > > > > > >> > >> >>
> isolated use
> > > case, but as part
> > > > of
> > > > > a
> > > > > > more holistic
> > > > > > > change
> > > > > > > > >> > (such as what
> > > > > > > > >> > >> we
> > > > > > > > >> > >> >>
> have proposed in
> > > KIP-98), I
> > > > think
> > > > > they
> > > > > > are
> > > > > > > justifiable.
> > > > > > > > If
> > > > > > > > >> > we add them,
> > > > > > > > >> > >> >>
> then the need to
> > > use attributes
> > > > > becomes
> > > > > > quite a
> > > > > > > bit
> > > > > > > > >> weaker,
> > > > > > > > >> > right? The
> > > > > > > > >> > >> >>
> other thing I
> > > find slightly odd
> > > > > is the
> > > > > > fact that
> > > > > > > null
> > > > > > > > >> > headers has no
> > > > > > > > >> > >> >>
> actual
> > > > > > > > >> > >> >>
> semantic meaning
> > > for the message
> > > > > > (unlike null
> > > > > > > keys and
> > > > > > > > >> > values). It is
> > > > > > > > >> > >> >>
> just
> > > > > > > > >> > >> >> a
> space
> > > optimization. It seems a
> > > > > bit
> > > > > > better to
> > > > > > > always
> > > > > > > > use
> > > > > > > > >> > size 0 to
> > > > > > > > >> > >> >>
> indicate having
> > > no headers.
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> Overall, the
> > main
> > > point is
> > > > > ensuring
> > > > > > that the
> > > > > > > message
> > > > > > > > >> schema
> > > > > > > > >> > remains
> > > > > > > > >> > >> >>
> consistent,
> > > either within the
> > > > > larger
> > > > > > protocol, or
> > > > > > > at a
> > > > > > > > >> > minimum within
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >>
> message itself.
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> -Jason
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >>
> On Thu, Feb 16,
> > > 2017 at 6:39 AM,
> > > > > > Michael Pearce <
> > > > > > > > >> >
> Michael.Pearce@ig.com
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >>
> wrote:
> > > > > > > > >> > >> >>
> > > > > > > > >> > >> >> >
> Hi Jason,
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> On point 1) in
> > > the message
> > > > > protocol
> > > > > > the headers
> > > > > > > are
> > > > > > > > >> simply
> > > > > > > > >> > a byte
> > > > > > > > >> > >>
> array,
> > > > > > > > >> > >> >> >
> as like the
> > key
> > > or value, this
> > > > > is to
> > > > > > clearly
> > > > > > > demarcate
> > > > > > > > >> the
> > > > > > > > >> > header in
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> >
> core message.
> > > Then the header
> > > > > byte
> > > > > > array in the
> > > > > > > core
> > > > > > > > >> > message is an
> > > > > > > > >> > >>
> array of
> > > > > > > > >> > >> >> >
> key, value
> > > pairs. This is what
> > > > > it is
> > > > > > denoting.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> Then this
> > would
> > > be I guess in
> > > > > the
> > > > > > given
> > > > > > > notation:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> Headers =>
> > > [KeyLength, Key,
> > > > > > ValueLength, Value]
> > > > > > > > >> > >> >>
> > KeyLength
> > > => int32
> > > > > > <-----------------NEW
> > > > > > > size of
> > > > > > > > the
> > > > > > > > >> > byte[] of
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> >
> serialised key
> > > value
> > > > > > > > >> > >> >>
> > Key =>
> > bytes
> > > > > > <---------------------- NEW
> > > > > > > > serialised
> > > > > > > > >> > string (UTF8)
> > > > > > > > >> > >> >> >
> bytes of the
> > > header key
> > > > > > > > >> > >> >> >
> > ValueLength
> > > => int32
> > > > > > <-------------- NEW
> > > > > > > size of
> > > > > > > > the
> > > > > > > > >> > byte[] of
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> >
> serialised
> > > header value
> > > > > > > > >> > >> >>
> > Value =>
> > > bytes
> > > > > > <-------------------- NEW
> > > > > > > > serialised
> > > > > > > > >> > form of the
> > > > > > > > >> > >>
> header
> > > > > > > > >> > >> >> >
> value
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> The key length
> > > and value
> > > > length
> > > > > is
> > > > > > matching the
> > > > > > > way
> > > > > > > > the
> > > > > > > > >> > protocol is
> > > > > > > > >> > >> >> >
> defined in the
> > > core message
> > > > > currently.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> On point 2)
> > > > > > > > >> > >> >> >
> Var sized
> > ints,
> > > this was
> > > > > discussed
> > > > > > much earlier
> > > > > > > on, in
> > > > > > > > >> > fact I had
> > > > > > > > >> > >> >> >
> suggested it
> > > myself (with
> > > > Hadoop
> > > > > > references),
> > > > > > > the
> > > > > > > > >> > complexity of
> this
> > > > > > > > >> > >> >> >
> compared to
> > > having a simpler
> > > > > protocol
> > > > > > was
> > > > > > > argued and
> > > > > > > > >> > agreed it
> > > > > > > > >> > >>
> wasn’t worth
> > > > > > > > >> > >> >> >
> the complexity
> > > as all other
> > > > > clients
> > > > > > in other
> > > > > > > languages
> > > > > > > > >> > would need to
> > > > > > > > >> > >>
> ensure
> > > > > > > > >> > >> >> >
> theyre using
> > > the right var
> > > > size
> > > > > > algorithm, as
> > > > > > > there
> > > > > > > > is a
> > > > > > > > >> > few.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> On point 3)
> > > > > > > > >> > >> >> >
> We did the
> > > attributes,
> > > > optional
> > > > > > approach as
> > > > > > > originally
> > > > > > > > >> > there was
> > > > > > > > >> > >>
> marked
> > > > > > > > >> > >> >> >
> concern that
> > > headers would
> > > > > cause a
> > > > > > message size
> > > > > > > > overhead
> > > > > > > > >> > for others,
> > > > > > > > >> > >> who
> > > > > > > > >> > >> >> >
> don’t want
> > > them. As such this
> > > > > is the
> > > > > > clean
> > > > > > > solution to
> > > > > > > > >> > achieve that.
> > > > > > > > >> > >> If
> > > > > > > > >> > >> >> >
> that no longer
> > > holds, and we
> > > > > don’t
> > > > > > care that we
> > > > > > > add
> > > > > > > > >> 4bytes
> > > > > > > > >> > overhead,
> > > > > > > > >> > >> then
> > > > > > > > >> > >> >> >
> im happy to
> > > remove.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> I’m personally
> > > in favour of
> > > > > keeping
> > > > > > the message
> > > > > > > as
> > > > > > > > small
> > > > > > > > >> > as possible
> > > > > > > > >> > >> so
> > > > > > > > >> > >> >> >
> people don’t
> > > get shocks in
> > > > perf
> > > > > and
> > > > > > throughputs
> > > > > > > dues
> > > > > > > > to
> > > > > > > > >> > message size,
> > > > > > > > >> > >> >> >
> unless they
> > > actively use the
> > > > > feature,
> > > > > > as such I
> > > > > > > do
> > > > > > > > >> prefer
> > > > > > > > >> > the
> > > > > > > > >> > >>
> attribute bit
> > > > > > > > >> > >> >> >
> wise feature
> > > flag approach
> > > > > myself.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> On 16/02/2017,
> > > 05:40, "Jason
> > > > > > Gustafson" <
> > > > > > > > >> >
> jason@confluent.io> wrote:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > We have
> > > proposed a few
> > > > > > significant changes
> > > > > > > to the
> > > > > > > > >> > message format
> > > > > > > > >> > >> in
> > > > > > > > >> > >> >> >
> KIP-98
> > > > > > > > >> > >> >>
> > which now
> > > seems likely to
> > > > > pass
> > > > > > (perhaps
> > > > > > > with some
> > > > > > > > >> > iterations on
> > > > > > > > >> > >> >> >
> > > implementation details).
> > > > It
> > > > > would
> > > > > > be good
> > > > > > > to try
> > > > > > > > and
> > > > > > > > >> > coordinate
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> >
> changes
> > > > > > > > >> > >> >>
> > in both of
> > > the proposals
> > > > to
> > > > > make
> > > > > > sure they
> > > > > > > are
> > > > > > > > >> > consistent and
> > > > > > > > >> > >> >> >
> compatible.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > I think
> > > using the
> > > > > attributes to
> > > > > > indicate
> > > > > > > null
> > > > > > > > >> headers
> > > > > > > > >> > is a
> > > > > > > > >> > >>
> reasonable
> > > > > > > > >> > >> >>
> > approach.
> > > We have proposed
> > > > > to do
> > > > > > the same
> > > > > > > thing
> > > > > > > > for
> > > > > > > > >> > the message
> > > > > > > > >> > >> key
> and
> > > > > > > > >> > >> >>
> > value.
> > That
> > > said, I
> > > > > sympathize
> > > > > > with Jay's
> > > > > > > > argument.
> > > > > > > > >> > Having
> > > > > > > > >> > >>
> multiple
> > > > > > > > >> > >> >> >
> ways to
> > > > > > > > >> > >> >>
> > specify a
> > > null value
> > > > > increases
> > > > > > the overall
> > > > > > > > >> complexity
> > > > > > > > >> > of the
> > > > > > > > >> > >>
> protocol.
> > > > > > > > >> > >> >> >
> You
> > > > > > > > >> > >> >>
> > can see
> > > this just from the
> > > > > fact
> > > > > > that you
> > > > > > > need the
> > > > > > > > >> > extra verbiage
> > > > > > > > >> > >> in
> the
> > > > > > > > >> > >> >>
> > protocol
> > > specification in
> > > > > this
> > > > > > KIP and in
> > > > > > > KIP-98
> > > > > > > > to
> > > > > > > > >> > describe the
> > > > > > > > >> > >> >> >
> dependence
> > > > > > > > >> > >> >>
> > between
> > the
> > > fields and the
> > > > > > attributes. It
> > > > > > > seems
> > > > > > > > >> like a
> > > > > > > > >> > slippery
> > > > > > > > >> > >> slope
> > > > > > > > >> > >> >> >
> if
> > > > > > > > >> > >> >>
> > you start
> > > allowing
> > > > different
> > > > > > request types
> > > > > > > to
> > > > > > > > >> > implement the
> > > > > > > > >> > >>
> protocol
> > > > > > > > >> > >> >> >
> > > specification differently.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > You can
> > > also argue that
> > > > the
> > > > > > messages
> > > > > > > already are
> > > > > > > > and
> > > > > > > > >> > are likely
> > > > > > > > >> > >> to
> > > > > > > > >> > >> >> >
> remain a
> > > > > > > > >> > >> >>
> > special
> > > case. For example,
> > > > > there
> > > > > > is
> > > > > > > currently no
> > > > > > > > >> > generality in
> > > > > > > > >> > >> how
> > > > > > > > >> > >> >>
> > compressed
> > > message sets
> > > > are
> > > > > > represented
> > > > > > > that would
> > > > > > > > >> be
> > > > > > > > >> > applicable
> > > > > > > > >> > >> for
> > > > > > > > >> > >> >> >
> other
> > > > > > > > >> > >> >>
> > request
> > > types. Some might
> > > > > see this
> > > > > > > divergence as
> > > > > > > > an
> > > > > > > > >> > unfortunate
> > > > > > > > >> > >> >> >
> protocol
> > > > > > > > >> > >> >>
> > deficiency
> > > which should be
> > > > > fixed;
> > > > > > others
> > > > > > > might see
> > > > > > > > >> it
> > > > > > > > >> > as sort of
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> >
> > > inevitability of needing
> > > > to
> > > > > > optimize where
> > > > > > > it
> > > > > > > > counts
> > > > > > > > >> > most. I'm
> > > > > > > > >> > >>
> probably
> > > > > > > > >> > >> >>
> > somewhere
> > > in between, but
> > > > I
> > > > > think
> > > > > > we
> > > > > > > probably all
> > > > > > > > >> > share the
> > > > > > > > >> > >>
> intuition
> > > > > > > > >> > >> >> >
> that
> > > > > > > > >> > >> >>
> > the
> > > protocol should be
> > > > kept
> > > > > as
> > > > > > consistent as
> > > > > > > > >> possible.
> > > > > > > > >> > With that
> > > > > > > > >> > >> in
> > > > > > > > >> > >> >> >
> mind,
> > > > > > > > >> > >> >>
> > here are a
> > > few comments:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > 1. One
> > > thing I found a
> > > > > little odd
> > > > > > when
> > > > > > > reading the
> > > > > > > > >> > current
> > > > > > > > >> > >>
> proposal is
> > > > > > > > >> > >> >> >
> that
> > > > > > > > >> > >> >>
> > the
> > headers
> > > are both
> > > > > represented
> > > > > > as an
> > > > > > > array of
> > > > > > > > >> bytes
> > > > > > > > >> > and as an
> > > > > > > > >> > >> array
> > > > > > > > >> > >> >> >
> of
> > > > > > > > >> > >> >>
> > key/value
> > > pairs. I'd
> > > > > probably
> > > > > > suggest
> > > > > > > something
> > > > > > > > like
> > > > > > > > >> > this:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > Headers =>
> > > [HeaderKey
> > > > > HeaderValue]
> > > > > > > > >> > >> >>
> > HeaderKey
> > > => String
> > > > > > > > >> > >> >> >
> > > HeaderValue => Bytes
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > An array
> > in
> > > the Kafka
> > > > > protocol is
> > > > > > > represented as a
> > > > > > > > >> > 4-byte integer
> > > > > > > > >> > >> >>
> > indicating
> > > the number of
> > > > > elements
> > > > > > in the
> > > > > > > array
> > > > > > > > >> > followed by the
> > > > > > > > >> > >> >> >
> > > serialization of the
> > > > > elements.
> > > > > > Unless I'm
> > > > > > > > >> >
> misunderstanding, what
> > > > > > > > >> > >> you
> > > > > > > > >> > >> >> >
> have
> > > > > > > > >> > >> >>
> > instead is
> > > the total size
> > > > > of the
> > > > > > headers in
> > > > > > > bytes
> > > > > > > > >> > followed by the
> > > > > > > > >> > >> >> >
> elements.
> > > > > > > > >> > >> >>
> > I'm not
> > > sure I see any
> > > > > reason for
> > > > > > this
> > > > > > > > >> inconsistency.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > 2. In
> > > KIP-98, we've
> > > > > introduced
> > > > > > > variable-length
> > > > > > > > >> integer
> > > > > > > > >> > fields.
> > > > > > > > >> > >> >> >
> Effectively,
> > > > > > > > >> > >> >>
> > we've
> > > enriched (or
> > > > > "complicated"
> > > > > > as Jay
> > > > > > > might say
> > > > > > > > ;)
> > > > > > > > >> > the protocol
> > > > > > > > >> > >> >> >
> > > specification to include
> > > > the
> > > > > > following
> > > > > > > types:
> > > > > > > > >> VarInt,
> > > > > > > > >> > VarLong,
> > > > > > > > >> > >> >> >
> > > UnsignedVarInt and
> > > > > > UnsignedVarLong.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > Along with
> > > these
> > > > > primitives, we
> > > > > > could
> > > > > > > introduce
> > > > > > > > the
> > > > > > > > >> > following
> > > > > > > > >> > >>
> types:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > VarSizeArray =>
> > > > > NumberOfItems
> > > > > > Item1 Item2
> > > > > > > .. ItemN
> > > > > > > > >> > >> >> >
> > > NumberOfItems =>
> > > > > UnsignedVarInt
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > VarSizeNullableArray =>
> > > > > > NumberOfItemsOrNull
> > > > > > > Item1
> > > > > > > > >> > Item2 .. ItemN
> > > > > > > > >> > >> >> >
> > > NumberOfItemsOrNull =>
> > > > > VarInt
> > > > > > (-1 means
> > > > > > > null)
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > And
> > > similarly for the
> > > > > `String`
> > > > > > and `Bytes`
> > > > > > > types.
> > > > > > > > >> > These types
> > > > > > > > >> > >> can
> save
> > > > > > > > >> > >> >> >
> a
> > > > > > > > >> > >> >> >
> > > considerable amount of
> > > > > space in
> > > > > > this
> > > > > > > proposal
> > > > > > > > >> because
> > > > > > > > >> > they can
> > > > > > > > >> > >> be
> used
> > > > > > > > >> > >> >> >
> for
> > > > > > > > >> > >> >>
> > both the
> > > number of headers
> > > > > > included in the
> > > > > > > message
> > > > > > > > >> and
> > > > > > > > >> > the
> > > > > > > > >> > >>
> lengths of
> > > > > > > > >> > >> >> >
> the
> > > > > > > > >> > >> >>
> > header
> > keys
> > > and values. We
> > > > > could
> > > > > > do this
> > > > > > > instead:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > Headers =>
> > > > > VarSizeArray[HeaderKey
> > > > > > > HeaderValue]
> > > > > > > > >> > >> >> >
> > HeaderKey
> > > =>
> > > > VarSizeString
> > > > > > > > >> > >> >> >
> > > HeaderValue =>
> > > > > VarSizeBytes
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > Combining
> > > the savings from
> > > > > the
> > > > > > use of
> > > > > > > variable
> > > > > > > > >> length
> > > > > > > > >> > fields, the
> > > > > > > > >> > >> >> >
> benefit
> > > > > > > > >> > >> >>
> > of using
> > > the attributes to
> > > > > > represent null
> > > > > > > seems
> > > > > > > > >> pretty
> > > > > > > > >> > small.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > 3.
> > > Whichever way we go
> > > > > (whether
> > > > > > we use the
> > > > > > > > >> attributes
> > > > > > > > >> > or not), we
> > > > > > > > >> > >> >> >
> should at
> > > > > > > > >> > >> >>
> > least be
> > > consistent
> > > > between
> > > > > this
> > > > > > KIP and
> > > > > > > KIP-98.
> > > > > > > > It
> > > > > > > > >> > would be very
> > > > > > > > >> > >> >> >
> strange
> > > > > > > > >> > >> >>
> > to have
> > two
> > > ways to
> > > > > represent
> > > > > > null values
> > > > > > > in the
> > > > > > > > >> same
> > > > > > > > >> > schema.
> > > > > > > > >> > >>
> Either
> > > > > > > > >> > >> >> >
> way is
> > > > > > > > >> > >> >>
> > OK with
> > me.
> > > I think some
> > > > > > message-level
> > > > > > > > optimizations
> > > > > > > > >> > are
> > > > > > > > >> > >>
> justifiable,
> > > > > > > > >> > >> >> >
> but
> > > > > > > > >> > >> >>
> > the
> > savings
> > > here seem
> > > > > minimal (a
> > > > > > few bytes
> > > > > > > per
> > > > > > > > >> > message), so
> > > > > > > > >> > >>
> maybe it's
> > > > > > > > >> > >> >> >
> not
> > > > > > > > >> > >> >>
> > worth the
> > > cost of letting
> > > > > the
> > > > > > message
> > > > > > > diverge even
> > > > > > > > >> > further from
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> >
> rest of
> > > > > > > > >> > >> >>
> > the
> > > protocol.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > -Jason
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > On Wed,
> > Feb
> > > 15, 2017 at
> > > > > 8:52 AM,
> > > > > > radai <
> > > > > > > > >> > >>
> > > radai.rosenblatt@gmail.com>
> > > > > > > > >> > >> >> >
> wrote:
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >>
> > > I've
> > > trimmed the inline
> > > > > > contents as this
> > > > > > > mail is
> > > > > > > > >> > getting too
> > > > > > > > >> > >> big
> for
> > > > > > > > >> > >> >> >
> the
> > > > > > > > >> > >> >>
> > > apache
> > > mailing list
> > > > > software to
> > > > > > deliver
> > > > > > > :-(
> > > > > > > > >> > >> >>
> > >
> > > > > > > > >> > >> >>
> > > 1. the
> > > important thing
> > > > for
> > > > > > > interoperability is
> > > > > > > > for
> > > > > > > > >> > different
> > > > > > > > >> > >> >> >
> "interested
> > > > > > > > >> > >> >>
> > > parties"
> > > (plugins, infra
> > > > > > layers/wrappers,
> > > > > > > > >> user-code)
> > > > > > > > >> > to be
> > > > > > > > >> > >> able
> to
> > > > > > > > >> > >> >> >
> stick
> > > > > > > > >> > >> >>
> > > pieces
> > of
> > > metadata onto
> > > > > msgs
> > > > > > without
> > > > > > > getting in
> > > > > > > > >> each
> > > > > > > > >> > other's
> > > > > > > > >> > >> way.
> a
> > > > > > > > >> > >> >> >
> common
> > > > > > > > >> > >> >>
> > > key
> > > scheme (Strings, as
> > > > > of the
> > > > > > time of
> > > > > > > this
> > > > > > > > >> > writing?) is all
> > > > > > > > >> > >> thats
> > > > > > > > >> > >> >> >
> required
> > > > > > > > >> > >> >>
> > > for
> > that.
> > > it is assumed
> > > > > that
> > > > > > the other end
> > > > > > > > >> > interested in
> any
> > > > > > > > >> > >> such
> > > > > > > > >> > >> >> >
> piece of
> > > > > > > > >> > >> >>
> > > metadata
> > > knows the
> > > > > encoding,
> > > > > > and byte[]
> > > > > > > provides
> > > > > > > > >> for
> > > > > > > > >> > the most
> > > > > > > > >> > >> >> >
> flexibility.
> > > > > > > > >> > >> >>
> > > i
> > believe
> > > this is the
> > > > same
> > > > > > logic behind
> > > > > > > core
> > > > > > > > kafka
> > > > > > > > >> > being
> > > > > > > > >> > >> >> >
> byte[]/byte[]
> > -
> > > > > > > > >> > >> >>
> > > Strings
> > > are more
> > > > "usable"
> > > > > but
> > > > > > bytes are
> > > > > > > flexible
> > > > > > > > >> and
> > > > > > > > >> > so were
> > > > > > > > >> > >>
> chosen.
> > > > > > > > >> > >> >>
> > > Also -
> > > core kafka doesnt
> > > > > even
> > > > > > do that
> > > > > > > good of a
> > > > > > > > >> job
> > > > > > > > >> > on
> > > > > > > > >> > >>
> usability of
> > > > > > > > >> > >> >> >
> the
> > > > > > > > >> > >> >>
> > > payload
> > > (example - i
> > > > have
> > > > > to
> > > > > > specify the
> > > > > > > nop
> > > > > > > > >> byte[]
> > > > > > > > >> > "decoders"
> > > > > > > > >> > >> >> >
> explicitly
> > > > > > > > >> > >> >>
> > > in
> > conf),
> > > and again
> > > > > sacrificies
> > > > > > usability
> > > > > > > for
> > > > > > > > the
> > > > > > > > >> > sake of
> > > > > > > > >> > >> >> >
> performance
> > (no
> > > > > > > > >> > >> >>
> > >
> > > convenient single-record
> > > > > > processing as
> > > > > > > poll is a
> > > > > > > > >> > batch, lots of
> > > > > > > > >> > >> >> >
> obscure
> > > > > > > > >> > >> >>
> > > little
> > > config details
> > > > > exposing
> > > > > > internals
> > > > > > > of the
> > > > > > > > >> > batching
> > > > > > > > >> > >>
> mechanism,
> > > > > > > > >> > >> >> >
> etc)
> > > > > > > > >> > >> >>
> > >
> > > > > > > > >> > >> >>
> > > this is
> > > also why i
> > > > really
> > > > > > dislike the
> > > > > > > idea of a
> > > > > > > > >> > "type system"
> > > > > > > > >> > >> for
> > > > > > > > >> > >> >> >
> header
> > > > > > > > >> > >> >>
> > > values,
> > > it further
> > > > > degrades the
> > > > > > > usability, adds
> > > > > > > > >> > complexity and
> > > > > > > > >> > >> will
> > > > > > > > >> > >> >>
> > >
> > > eventually get in
> > > > > people's way,
> > > > > > also, it
> > > > > > > would
> > > > > > > > be
> > > > > > > > >> > the 2nd/3rd
> > > > > > > > >> > >> >> >
> home-group
> > > > > > > > >> > >> >>
> > >
> > > serialization mechanism
> > > > > in core
> > > > > > kafka
> > > > > > > (counting
> > > > > > > > 2
> > > > > > > > >> > iterations
> > > > > > > > >> > >> of
> the
> > > > > > > > >> > >> >> >
> "type
> > > > > > > > >> > >> >>
> > >
> > > definition DSL")
> > > > > > > > >> > >> >>
> > >
> > > > > > > > >> > >> >>
> > > 2. this
> > > is an
> > > > > implementation
> > > > > > detail, and
> > > > > > > not
> > > > > > > > even
> > > > > > > > >> a
> > > > > > > > >> > very "user
> > > > > > > > >> > >> >> >
> facing" one?
> > > > > > > > >> > >> >>
> > > to the
> > > best of my
> > > > > understanding
> > > > > > the vote
> > > > > > > process
> > > > > > > > >> is
> > > > > > > > >> > on proposed
> > > > > > > > >> > >> >>
> > >
> > > API/behaviour. also -
> > > > > since
> > > > > > we're willing
> > > > > > > to go
> > > > > > > > >> with
> > > > > > > > >> > strings
> > > > > > > > >> > >> just
> > > > > > > > >> > >> >> >
> serialize
> > > > > > > > >> > >> >>
> > > a
> > 0-sized
> > > header blob
> > > > and
> > > > > IIUC
> > > > > > you dont
> > > > > > > need any
> > > > > > > > >> > optionals
> > > > > > > > >> > >>
> anymore.
> > > > > > > > >> > >> >>
> > >
> > > > > > > > >> > >> >>
> > > 3. yes,
> > > we can :-)
> > > > > > > > >> > >> >>
> > >
> > > > > > > > >> > >> >>
> > > On Tue,
> > > Feb 14, 2017 at
> > > > > 11:56
> > > > > > PM, Michael
> > > > > > > > Pearce <
> > > > > > > > >> > >> >> >
> > > Michael.Pearce@ig.com>
> > > > > > > > >> > >> >>
> > > wrote:
> > > > > > > > >> > >> >>
> > >
> > > > > > > > >> > >> >>
> > > > Hi
> > Jay,
> > > > > > > > >> > >> >>
> > > >
> > > > > > > > >> > >> >>
> > > > 1)
> > > There was some
> > > > > initial
> > > > > > debate on the
> > > > > > > value
> > > > > > > > >> > part, as youll
> > > > > > > > >> > >> note
> > > > > > > > >> > >> >> >
> String,
> > > > > > > > >> > >> >>
> > > > String
> > > headers were
> > > > > > discounted early
> > > > > > > on. The
> > > > > > > > >> > reason for this
> > > > > > > > >> > >> is
> > > > > > > > >> > >> >>
> > >
> > > flexibility
> > > > > > > > >> > >> >>
> > > > and
> > > keeping in line
> > > > > with the
> > > > > > > flexibility of
> > > > > > > > key,
> > > > > > > > >> > value of the
> > > > > > > > >> > >> >> >
> message
> > > > > > > > >> > >> >>
> > > > object
> > > itself. I don’t
> > > > > think
> > > > > > it takes
> > > > > > > away
> > > > > > > > from
> > > > > > > > >> an
> > > > > > > > >> > ecosystem
> > > > > > > > >> > >> as
> > > > > > > > >> > >> >> >
> each
> > > > > > > > >> > >> >>
> > > plugin
> > > > > > > > >> > >> >>
> > > > will
> > > care for their
> > > > own
> > > > > key,
> > > > > > this way
> > > > > > > ints,
> > > > > > > > >> > booleans ,
> exotic
> > > > > > > > >> > >> >> >
> custom
> > > > > > > > >> > >> >>
> > > binary
> > > > > > > > >> > >> >>
> > > > can
> > all
> > > be catered
> > > > for=.
> > > > > > > > >> > >> >>
> > > > a. If
> > > you really
> > > > wanted
> > > > > to
> > > > > > push for a
> > > > > > > typed
> > > > > > > > >> value
> > > > > > > > >> > interface,
> > > > > > > > >> > >> I
> > > > > > > > >> > >> >> >
> wouldn’t
> > > > > > > > >> > >> >>
> > > > want
> > > just String
> > > > values
> > > > > > supported, but
> > > > > > > the the
> > > > > > > > >> > primatives
> > > > > > > > >> > >> plus
> > > > > > > > >> > >> >> >
> string and
> > > > > > > > >> > >> >>
> > > > also
> > > still keeping the
> > > > > > ability to have a
> > > > > > > > binary
> > > > > > > > >> > for custom
> > > > > > > > >> > >> >> >
> binaries that
> > > > > > > > >> > >> >>
> > > > some
> > > organisations may
> > > > > have.
> > > > > > > > >> > >> >>
> > > > i. I
> > > have written this
> > > > > slight
> > > > > > > alternative
> > > > > > > > here,
> > > > > > > > >> > >> >>
> > >
> > > > https://cwiki.apache.org/
> > > > > > > > >> > >> >>
> > > >
> > > > > confluence/display/KAFKA/KIP-
> > > > > > > > >> >
> 82+-+Add+Record+Headers+-+
> > Typed
> > > > > > > > >> > >> >>
> > > > ii.
> > > Essentially the
> > > > > value
> > > > > > bytes, has a
> > > > > > > leading
> > > > > > > > >> > byte overhead.
> > > > > > > > >> > >> >>
> > > > 1.
> > > This tells you
> > > > what
> > > > > type
> > > > > > the value
> > > > > > > is,
> > > > > > > > >> before
> > > > > > > > >> > reading
> > > > > > > > >> > >> the
> rest
> > > > > > > > >> > >> >> >
> of the
> > > > > > > > >> > >> >>
> > > > bytes,
> > > allowing
> > > > > > >
> serialisation/deserialization
> > > > > > > > to
> > > > > > > > >> > and from the
> > > > > > > > >> > >> >> >
> primitives,
> > > > > > > > >> > >> >>
> > > > string
> > > and byte[].
> > > > This
> > > > > is
> > > > > > akin to some
> > > > > > > other
> > > > > > > > >> > messaging
> > > > > > > > >> > >>
> systems.
> > > > > > > > >> > >> >>
> > > > 2) We
> > > are making it
> > > > > optional,
> > > > > > so that
> > > > > > > for
> > > > > > > > those
> > > > > > > > >> > not wanting
> > > > > > > > >> > >> >> >
> headers have
> > > > > > > > >> > >> >>
> > > 0
> > > > > > > > >> > >> >>
> > > > bytes
> > > overhead (think
> > > > > of it
> > > > > > as a feature
> > > > > > > > flag),
> > > > > > > > >> I
> > > > > > > > >> > don’t
> > > > > > > > >> > >>
> think this
> > > > > > > > >> > >> >> >
> is
> > > > > > > > >> > >> >>
> > > >
> > > complex, especially if
> > > > > > comparing to
> > > > > > > changes
> > > > > > > > >> > proposed in
> > > > > > > > >> > >>
> other kips
> > > > > > > > >> > >> >> >
> like
> > > > > > > > >> > >> >>
> > > >
> > kip-98.
> > > > > > > > >> > >> >>
> > > > a. If
> > > you really
> > > > really
> > > > > don’t
> > > > > > like
> > > > > > > this, we
> > > > > > > > can
> > > > > > > > >> > drop it, but
> > > > > > > > >> > >> it
> > > > > > > > >> > >> >> >
> would
> > > > > > > > >> > >> >>
> > > mean
> > > > > > > > >> > >> >>
> > > > buying
> > > into 4 bytes
> > > > > extra
> > > > > > overhead for
> > > > > > > users
> > > > > > > > who
> > > > > > > > >> > do not want
> > > > > > > > >> > >> to
> use
> > > > > > > > >> > >> >>
> > > headers.
> > > > > > > > >> > >> >>
> > > > 3) In
> > > the summary yes,
> > > > > it is
> > > > > > at a higher
> > > > > > > > level,
> > > > > > > > >> > but I think
> > > > > > > > >> > >> this
> > > > > > > > >> > >> >> >
> is well
> > > > > > > > >> > >> >>
> > > >
> > > documented in the
> > > > > proposed
> > > > > > changes
> > > > > > > section.
> > > > > > > > >> > >> >>
> > > > a.
> > > Added getHeaders
> > > > > method to
> > > > > > > > Producer/Consumer
> > > > > > > > >> > record (that
> > > > > > > > >> > >> is
> it)
> > > > > > > > >> > >> >>
> > > > b.
> > > We’ve also detailed
> > > > > the
> > > > > > new Headers
> > > > > > > class
> > > > > > > > >> that
> > > > > > > > >> > this method
> > > > > > > > >> > >> >> >
> returns
> > > > > > > > >> > >> >>
> > > that
> > > > > > > > >> > >> >>
> > > >
> > > encapsulates the
> > > > headers
> > > > > > protocol and
> > > > > > > logic.
> > > > > > > > >> > >> >>
> > > >
> > > > > > > > >> > >> >>
> > > > Best,
> > > > > > > > >> > >> >>
> > > > Mike
> > > > > > > > >> > >> >>
> > > >
> > > > > > > > >> > >> >>
> > > >
> > > ==Original questions
> > > > > from the
> > > > > > vote
> > > > > > > thread from
> > > > > > > > >> > Jay.==
> > > > > > > > >> > >> >>
> > > >
> > > > > > > > >> > >> >>
> > > > Couple
> > > of things I
> > > > > think we
> > > > > > still need
> > > > > > > to work
> > > > > > > > >> out:
> > > > > > > > >> > >> >>
> > > >
> > > > > > > > >> > >> >>
> > > > 1.
> > I
> > > think we agree
> > > > > about
> > > > > > the key,
> > > > > > > but I
> > > > > > > > >> think
> > > > > > > > >> > we haven't
> > > > > > > > >> > >> >> >
> talked about
> > > > > > > > >> > >> >>
> > > > the
> > > value yet. I
> > > > > think if
> > > > > > our goal
> > > > > > > is an
> > > > > > > > open
> > > > > > > > >> > ecosystem
> > > > > > > > >> > >> of
> these
> > > > > > > > >> > >> >>
> > > header
> > > > > > > > >> > >> >>
> > > >
> > > spread across many
> > > > > plugins
> > > > > > from many
> > > > > > > > systems
> > > > > > > > >> we
> > > > > > > > >> > should
> > > > > > > > >> > >>
> consider
> > > > > > > > >> > >> >> >
> making
> > > > > > > > >> > >> >>
> > > > this
> > > > > > > > >> > >> >>
> > > > a
> > > string as well so
> > > > > it can
> > > > > > be
> > > > > > > printed, set
> > > > > > > > >> via
> > > > > > > > >> > a UI, set
> > > > > > > > >> > >> in
> > > > > > > > >> > >> >> >
> config,
> > > > > > > > >> > >> >>
> > > etc.
> > > > > > > > >> > >> >>
> > > >
> > > Basically
> > > > encouraging
> > > > > > pluggable
> > > > > > > > serialization
> > > > > > > > >> > formats
> > > > > > > > >> > >> here
> will
> > > > > > > > >> > >> >> >
> lead
> > > > > > > > >> > >> >>
> > > to
> > > > > > > > >> > >> >>
> > > > a
> > > > > > > > >> > >> >>
> > > > bit
> > > of a tower of
> > > > > babel.
> > > > > > > > >> > >> >>
> > > > 2.
> > > This proposal
> > > > > still
> > > > > > includes a
> > > > > > > pretty
> > > > > > > > big
> > > > > > > > >> > change to our
> > > > > > > > >> > >> >>
> > >
> > > serialization
> > > > > > > > >> > >> >>
> > > > and
> > > protocol
> > > > > definition
> > > > > > layer.
> > > > > > > Essentially
> > > > > > > > >> it is
> > > > > > > > >> > >>
> introducing an
> > > > > > > > >> > >> >>
> > > optional
> > > > > > > > >> > >> >>
> > > >
> > > type, where the
> > > > > format is
> > > > > > data
> > > > > > > dependent. I
> > > > > > > > >> > think this is
> > > > > > > > >> > >> >> >
> actually a
> > > > > > > > >> > >> >>
> > > big
> > > > > > > > >> > >> >>
> > > >
> > > change though it
> > > > > doesn't
> > > > > > seem like
> > > > > > > it. It
> > > > > > > > >> means
> > > > > > > > >> > you can no
> > > > > > > > >> > >> >> >
> longer
> > > > > > > > >> > >> >>
> > > >
> > specify
> > > > > > > > >> > >> >>
> > > >
> > this
> > > type with our
> > > > > type
> > > > > > definition
> > > > > > > DSL, and
> > > > > > > > >> > likewise it
> > > > > > > > >> > >>
> requires
> > > > > > > > >> > >> >>
> > > custom
> > > > > > > > >> > >> >>
> > > >
> > > handling in client
> > > > > libs.
> > > > > > This isn't
> > > > > > > a huge
> > > > > > > > >> > thing, since
> > > > > > > > >> > >> the
> > > > > > > > >> > >> >> >
> Record
> > > > > > > > >> > >> >>
> > > >
> > > definition is
> > > > custom
> > > > > > anyway, but I
> > > > > > > think
> > > > > > > > this
> > > > > > > > >> > kind of
> > > > > > > > >> > >>
> protocol
> > > > > > > > >> > >> >>
> > > >
> > > inconsistency is
> > > > very
> > > > > > non-desirable
> > > > > > > and
> > > > > > > > ties
> > > > > > > > >> > you to
> > > > > > > > >> > >>
> hand-coding
> > > > > > > > >> > >> >>
> > > things.
> > > > > > > > >> > >> >>
> > > > I
> > > > > > > > >> > >> >>
> > > >
> > > think the type
> > > > should
> > > > > > instead by [Key
> > > > > > > > Value]
> > > > > > > > >> in
> > > > > > > > >> > our BNF,
> > > > > > > > >> > >> where
> > > > > > > > >> > >> >> >
> key and
> > > > > > > > >> > >> >>
> > > >
> > > value are both
> > > > short
> > > > > > strings as used
> > > > > > > > >> elsewhere.
> > > > > > > > >> > This
> > > > > > > > >> > >>
> brings it
> > > > > > > > >> > >> >> >
> in line
> > > > > > > > >> > >> >>
> > > > with
> > > > > > > > >> > >> >>
> > > > the
> > > rest of the
> > > > > protocol.
> > > > > > > > >> > >> >>
> > > > 3.
> > > Could we get
> > > > more
> > > > > > specific about
> > > > > > > the
> > > > > > > > exact
> > > > > > > > >> > Java API
> > > > > > > > >> > >>
> change to
> > > > > > > > >> > >> >>
> > > >
> > > ProducerRecord,
> > > > > > ConsumerRecord,
> > > > > > > Record,
> > > > > > > > etc?
> > > > > > > > >> > >> >>
> > > >
> > > > > > > > >> > >> >>
> > > > -Jay
> > > > > > > > >> > >> >>
> > > >
> > > > > > > > >> > >> >>
> > >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >> >> >
> The
> > information
> > > contained in
> > > > > this
> > > > > > email is
> > > > > > > strictly
> > > > > > > > >> > confidential and
> > > > > > > > >> > >> for
> > > > > > > > >> > >> >> >
> the use of the
> > > addressee only,
> > > > > unless
> > > > > > otherwise
> > > > > > > > >> indicated.
> > > > > > > > >> > If you
> > > > > > > > >> > >> are
> not
> > > > > > > > >> > >> >> >
> the intended
> > > recipient, please
> > > > > do not
> > > > > > read,
> > > > > > > copy, use
> > > > > > > > or
> > > > > > > > >> > disclose to
> > > > > > > > >> > >>
> others
> > > > > > > > >> > >> >> >
> this message
> > or
> > > any
> > > > attachment.
> > > > > > Please also
> > > > > > > notify the
> > > > > > > > >> > sender by
> > > > > > > > >> > >>
> replying
> > > > > > > > >> > >> >> >
> to this email
> > > or by telephone
> > > > > > (+44(020 7896
> > > > > > > 0011) and
> > > > > > > > >> then
> > > > > > > > >> > delete
> > > > > > > > >> > >> the
> email
> > > > > > > > >> > >> >> >
> and any copies
> > > of it.
> > > > Opinions,
> > > > > > conclusion
> > > > > > > (etc) that
> > > > > > > > do
> > > > > > > > >> > not relate
> > > > > > > > >> > >> to
> the
> > > > > > > > >> > >> >> >
> official
> > > business of this
> > > > > company
> > > > > > shall be
> > > > > > > understood
> > > > > > > > as
> > > > > > > > >> > neither
> > > > > > > > >> > >>
> given nor
> > > > > > > > >> > >> >> >
> endorsed by
> > it.
> > > IG is a
> > > > trading
> > > > > name
> > > > > > of IG
> > > > > > > Markets
> > > > > > > > >> Limited
> > > > > > > > >> > (a company
> > > > > > > > >> > >> >> >
> registered in
> > > England and
> > > > Wales,
> > > > > > company number
> > > > > > > > >> 04008957)
> > > > > > > > >> > and IG
> > > > > > > > >> > >> Index
> > > > > > > > >> > >> >> >
> Limited (a
> > > company registered
> > > > in
> > > > > > England and
> > > > > > > Wales,
> > > > > > > > >> > company number
> > > > > > > > >> > >> >> >
> 01190902).
> > > Registered address
> > > > at
> > > > > > Cannon Bridge
> > > > > > > House,
> > > > > > > > 25
> > > > > > > > >> > Dowgate
> > > > > > > > >> > >> Hill,
> > > > > > > > >> > >> >> >
> London EC4R
> > > 2YA. Both IG
> > > > Markets
> > > > > > Limited
> > > > > > > (register
> > > > > > > > >> number
> > > > > > > > >> > 195355)
> > > > > > > > >> > >> and
> IG
> > > > > > > > >> > >> >> >
> Index Limited
> > > (register number
> > > > > > 114059) are
> > > > > > > authorised
> > > > > > > > >> and
> > > > > > > > >> > regulated
> > > > > > > > >> > >> by
> the
> > > > > > > > >> > >> >> >
> Financial
> > > Conduct Authority.
> > > > > > > > >> > >> >> >
> > > > > > > > >> > >>
> > > > > > > > >> > >>
> > > > > > > > >> > >>
> > > > > > > > >> > >> --
> > > > > > > > >> > >>
> Nacho - Ignacio
> > > Solis -
> > > > > isolis@igso.net
> > > > > > > > >> > >>
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > The information
> contained in
> > > this email is
> > > > > strictly
> > > > > > > confidential and
> > > > > > > > for
> > > > > > > > >> > the use of the
> addressee only,
> > > unless
> > > > otherwise
> > > > > > indicated. If
> > > > > > > you are
> > > > > > > > >> not
> > > > > > > > >> > the intended
> recipient, please
> > > do not read,
> > > > > copy, use
> > > > > > or
> > > > > > > disclose to
> > > > > > > > >> others
> > > > > > > > >> > this message or
> any
> > attachment.
> > > Please also
> > > > > notify
> > > > > > the sender by
> > > > > > > > >> replying
> > > > > > > > >> > to this email
> or by telephone
> > > (+44(020 7896
> > > > > 0011) and
> > > > > > then
> > > > > > > delete the
> > > > > > > > >> email
> > > > > > > > >> > and any copies
> of it.
> > Opinions,
> > > conclusion
> > > > > (etc) that
> > > > > > do not
> > > > > > > relate to
> > > > > > > > >> the
> > > > > > > > >> > official
> business of this
> > > company shall be
> > > > > understood
> > > > > > as
> > > > > > > neither given
> > > > > > > > >> nor
> > > > > > > > >> > endorsed by it.
> IG is a
> > trading
> > > name of IG
> > > > > Markets
> > > > > > Limited (a
> > > > > > > company
> > > > > > > > >> > registered in
> England and
> > > Wales, company
> > > > number
> > > > > > 04008957) and
> > > > > > > IG Index
> > > > > > > > >> > Limited (a
> company registered
> > > in England and
> > > > > Wales,
> > > > > > company
> > > > > > > number
> > > > > > > > >> > 01190902).
> Registered address
> > > at Cannon Bridge
> > > > > House,
> > > > > > 25
> > > > > > > Dowgate Hill,
> > > > > > > > >> > London EC4R
> 2YA. Both IG
> > > Markets Limited
> > > > > (register
> > > > > > number
> > > > > > > 195355) and
> > > > > > > > IG
> > > > > > > > >> > Index Limited
> (register number
> > > 114059) are
> > > > > authorised
> > > > > > and
> > > > > > > regulated by
> > > > > > > > >> the
> > > > > > > > >> > Financial
> Conduct Authority.
> > > > > > > > >> >
> > > > > > > > >> The information
> contained in
> > this
> > > email is
> > > > > strictly
> > > > > > confidential
> > > > > > > and for
> > > > > > > > >> the use of the
> addressee only,
> > > unless otherwise
> > > > > > indicated. If you
> > > > > > > are
> > > > > > > > not
> > > > > > > > >> the intended
> recipient, please
> > do
> > > not read,
> > > > copy,
> > > > > use
> > > > > > or disclose
> > > > > > > to
> > > > > > > > others
> > > > > > > > >> this message or
> any attachment.
> > > Please also
> > > > > notify the
> > > > > > sender by
> > > > > > > > replying
> > > > > > > > >> to this email or
> by telephone
> > > (+44(020 7896
> > > > 0011)
> > > > > and
> > > > > > then delete
> > > > > > > the
> > > > > > > > email
> > > > > > > > >> and any copies of
> it. Opinions,
> > > conclusion (etc)
> > > > > that
> > > > > > do not
> > > > > > > relate to
> > > > > > > > the
> > > > > > > > >> official business
> of this
> > company
> > > shall be
> > > > > understood
> > > > > > as neither
> > > > > > > given
> > > > > > > > nor
> > > > > > > > >> endorsed by it.
> IG is a trading
> > > name of IG
> > > > Markets
> > > > > > Limited (a
> > > > > > > company
> > > > > > > > >> registered in
> England and Wales,
> > > company number
> > > > > > 04008957) and IG
> > > > > > > Index
> > > > > > > > >> Limited (a
> company registered in
> > > England and
> > > > > Wales,
> > > > > > company number
> > > > > > > > >> 01190902).
> Registered address at
> > > Cannon Bridge
> > > > > House,
> > > > > > 25 Dowgate
> > > > > > > Hill,
> > > > > > > > >> London EC4R 2YA.
> Both IG Markets
> > > Limited
> > > > (register
> > > > > > number 195355)
> > > > > > > and IG
> > > > > > > > >> Index Limited
> (register number
> > > 114059) are
> > > > > authorised
> > > > > > and
> > > > > > > regulated by
> > > > > > > > the
> > > > > > > > >> Financial Conduct
> Authority.
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > The information
> contained in this
> > > email is strictly
> > > > > > confidential and
> > > > > > > for the use of the
> addressee only, unless
> > > otherwise
> > > > > indicated.
> > > > > > If you are
> > > > > > > not the intended
> recipient, please do not
> > > read, copy, use
> > > > > or
> > > > > > disclose to
> > > > > > > others this message or any
> attachment.
> > > Please also notify
> > > > > the
> > > > > > sender by
> > > > > > > replying to this email or
> by telephone
> > > (+44(020 7896
> > > > 0011)
> > > > > and
> > > > > > then delete
> > > > > > > the email and any copies
> of it. Opinions,
> > > conclusion
> > > > (etc)
> > > > > that
> > > > > > do not
> > > > > > > relate to the official
> business of this
> > > company shall be
> > > > > > understood as
> > > > > > > neither given nor endorsed
> by it. IG is a
> > > trading name of
> > > > > IG
> > > > > > Markets
> > > > > > > Limited (a company
> registered in England
> > > and Wales,
> > > > company
> > > > > > number
> > > > > > > 04008957) and IG Index
> Limited (a company
> > > registered in
> > > > > England
> > > > > > and Wales,
> > > > > > > company number 01190902).
> Registered
> > > address at Cannon
> > > > > Bridge
> > > > > > House, 25
> > > > > > > Dowgate Hill, London EC4R
> 2YA. Both IG
> > > Markets Limited
> > > > > (register
> > > > > > number
> > > > > > > 195355) and IG Index
> Limited (register
> > > number 114059) are
> > > > > > authorised and
> > > > > > > regulated by the Financial
> Conduct
> > > Authority.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > The information contained in
> this email is
> > > strictly
> > > > confidential
> > > > > and
> > > > > > for the use of the addressee only,
> unless otherwise
> > > indicated. If
> > > > > you are
> > > > > > not the intended recipient, please
> do not read,
> > > copy, use or
> > > > > disclose to
> > > > > > others this message or any
> attachment. Please also
> > > notify the
> > > > sender
> > > > > by
> > > > > > replying to this email or by
> telephone (+44(020
> > 7896
> > > 0011) and then
> > > > > delete
> > > > > > the email and any copies of it.
> Opinions,
> > conclusion
> > > (etc) that do
> > > > > not
> > > > > > relate to the official business of
> this company
> > > shall be understood
> > > > > as
> > > > > > neither given nor endorsed by it. IG
> is a trading
> > > name of IG
> > > > Markets
> > > > > > Limited (a company registered in
> England and Wales,
> > > company number
> > > > > > 04008957) and IG Index Limited (a
> company
> > registered
> > > in England and
> > > > > Wales,
> > > > > > company number 01190902). Registered
> address at
> > > Cannon Bridge
> > > > House,
> > > > > 25
> > > > > > Dowgate Hill, London EC4R 2YA. Both
> IG Markets
> > > Limited (register
> > > > > number
> > > > > > 195355) and IG Index Limited
> (register number
> > > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by the Financial Conduct
> Authority.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is
> strictly
> > > confidential and for
> > > > > the use of the addressee only, unless
> otherwise
> > indicated.
> > > If you are not
> > > > > the intended recipient, please do not
> read, copy, use or
> > > disclose to
> > > > others
> > > > > this message or any attachment. Please
> also notify the
> > > sender by replying
> > > > > to this email or by telephone (+44(020
> 7896 0011) and
> > then
> > > delete the
> > > > email
> > > > > and any copies of it. Opinions, conclusion
> (etc) that do
> > > not relate to
> > > > the
> > > > > official business of this company shall be
> understood as
> > > neither given
> > > > nor
> > > > > endorsed by it. IG is a trading name of IG
> Markets
> > Limited
> > > (a company
> > > > > registered in England and Wales, company
> number 04008957)
> > > and IG Index
> > > > > Limited (a company registered in England
> and Wales,
> > > company number
> > > > > 01190902). Registered address at Cannon
> Bridge House, 25
> > > Dowgate Hill,
> > > > > London EC4R 2YA. Both IG Markets Limited
> (register number
> > > 195355) and IG
> > > > > Index Limited (register number 114059) are
> authorised and
> > > regulated by
> > > > the
> > > > > Financial Conduct Authority.
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and
> > > for the use of the addressee only, unless otherwise
> indicated. If you are
> > > not the intended recipient, please do not read, copy, use
> or disclose to
> > > others this message or any attachment. Please also notify
> the sender by
> > > replying to this email or by telephone (+44(020 7896 0011)
> and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc)
> that do not
> > > relate to the official business of this company shall be
> understood as
> > > neither given nor endorsed by it. IG is a trading name of
> IG Markets
> > > Limited (a company registered in England and Wales,
> company number
> > > 04008957) and IG Index Limited (a company registered in
> England and
> > Wales,
> > > company number 01190902). Registered address at Cannon
> Bridge House, 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> (register number
> > > 195355) and IG Index Limited (register number 114059) are
> authorised and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> >
>
>
>
>
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
We can make it, implement Iterable<Header>, no concerns on this.
Ive noted this on the KIP.
On 20/02/2017, 01:24, "Michael Pearce" <Mi...@ig.com> wrote:
Btw Jason, can you share the branch with the VarInts helper/util class that KIP-98 will be introducing and also will have the current modified (key and value lengths) message, so we can look at, thanks.
On 20/02/2017, 01:11, "Michael Pearce" <Mi...@ig.com> wrote:
RE “ProducerRecord and ConsumerRecord are immutable currently”. I beg to differ on this, if I set a byte[] as my value of my record, I can later call value() and then alter the bytes in the array. As such it is very much mutable.
Headers also will anyhow by nature be mutable on produce, as you add to them.
On 20/02/2017, 00:22, "Jason Gustafson" <ja...@confluent.io> wrote:
Re; having a dedicated Headers object. I think it makes sense. Maybe it
could implement Iterable<Header>?
One thing I'm not too sure about is mutability. Both the ProducerRecord and
ConsumerRecord are immutable currently. It would be nice to keep it that
way. Perhaps Headers could itself be immutable and passed through the
constructors?
Just going through how this is, I don’t believe this would provide much
> saving. If issue trying to resolve is around object/garbage creation of
> Map.Entry objects.
It's definitely better because we only allocate the space we will use, but
I assume most of the actual memory cost is in the header data itself which
we can't do much about. Maybe we can start with the simple array
representation and optimize from there after we find it's worthwhile to do
so? If we use the Headers object, we can change this later without breaking
the API.
-Jason
On Sun, Feb 19, 2017 at 2:39 PM, radai <ra...@gmail.com> wrote:
> "(By the way, doesn't it feel a bit odd that we seem to be designing a
> feature which is optimized for people not using?)"
>
> very. this (i claim :-) ) is the result of very intense opposition to the
> usefulness of the feature early on, and not a real design goal
>
> On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Whilst if having an array of a Header[] would mean getting this through,
> > im happy changing this, as already done.
> >
> > Just going through how this is, I don’t believe this would provide much
> > saving. If issue trying to resolve is around object/garbage creation of
> > Map.Entry objects.
> >
> > All you’re doing here is replacing the equiv of a HashMap.Node
> (Map.Entry)
> > which simply holds reference to key and value objects with a custom
> variant.
> >
> >
> >
> >
> >
> > On 19/02/2017, 20:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> > On point 1 & 2 Ive updated KIP to show varints (and removed the bit
> > flag). (on the assumption KIP 98 is getting the agreement the protocol is
> > moving from int32 to varInts as standard)
> >
> > On point 3 ive updated to use an array of Header class, instead of a
> > MultiMap in the Headers class object
> >
> >
> >
> > On 19/02/2017, 20:06, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > On points 1 and 2 I agree.
> >
> > This also affects kip-98, I should expect this resolved before
> > that vote also passes. If it is accepted there (I’m assuming this is
> > getting discussed on that KIP? As you’re driving the move to VarInts), I
> am
> > happy to make this KIP will simply follow suit to whatever is agreed in
> > KIP-98.
> >
> > On 3) Agreed this is a simpler form, as long as no one is
> > expecting hashmap lookup performance ( O(1) ), I still would prefer an
> > encapsulated class, so if we find that holding it as an array in future
> is
> > causing some perf issues, the internals are not exposed to end users,
> > allowing the internal structure to move to a map.
> >
> > Can we compromise on?
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Headers headers;
> > }
> >
> > class Headers {
> > Header[] headers;
> >
> > add(String key, byte[] value)
> > Collection<byte[]> get(String key)
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> >
> >
> > On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> >
> > >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> >
> >
> > This is exactly what I have been trying to get at. The use
> > cases documented
> > here are middleware:
> > https://cwiki.apache.org/confluence/display/KAFKA/A+
> > Case+for+Kafka+Headers.
> > If headers are intended for the application space as well,
> the
> > document
> > should be updated accordingly so that it is an explicit
> design
> > goal and not
> > an unfortunate design byproduct. There may be some
> > disagreement on whether
> > it _should_ be a design goal, but it may not matter much
> since
> > the current
> > interceptor API is probably insufficient for middleware
> > applications (I
> > haven't thought of a way to do this that isn't cumbersome).
> >
> > That aside, the three unresolved points for me are the
> > following:
> >
> > 1. The use of varints. As a one-off for use only with record
> > headers, the
> > case was weak, but if we are using them throughout the
> message
> > format, then
> > we should do so here as well. The additional complexity is
> > minimal and
> > early performance testing fully justifies their use.
> >
> > 2. If varints are used, the case for using attributes to
> > indicate null
> > headers also becomes weak. We only add one additional byte in
> > each message
> > if there are no headers. Whatever the case, let us be
> > consistent with how
> > we treat null keys and values.
> >
> > 3. We have apparently agreed that the broker will validate
> > headers (please
> > add this to the KIP). That being the case, I would prefer to
> > use Kafka's
> > conventional format for arrays. The downside is that it takes
> > more work to
> > skip over the headers on the consumer, though it's unclear if
> > the cost
> > would matter in practice. Some concern was previously
> > expressed about the
> > allocation of maps. An alternative would be to use arrays,
> i.e.
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Header[] headers;
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> > This would work nicely with the conventional array format and
> > my guess is
> > it would obviate the need do any lazy initialization. If we
> > use the map as
> > is currently documented, then it is possible with either
> > representation to
> > slice the headers and initialize them lazily. Either way, it
> > might be a
> > good idea to use a separate object to represent the headers
> in
> > case we need
> > to modify it in the future in some way.
> >
> > (By the way, doesn't it feel a bit odd that we seem to be
> > designing a
> > feature which is optimized for people not using?)
> >
> >
> > If we can resolve these points, then at least you will get my
> > vote.
> >
> > Thanks,
> > Jason
> >
> > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > radai.rosenblatt@gmail.com> wrote:
> >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> > > IIUC samze currently has headers "in-V" and would just
> > switch over to kafka
> > > headers if they exist.
> > > im sure plenty of other users of kafka would have a use for
> > headers.
> > > im pretty sure use cases exist around shuffling data
> > into/out-of kafka
> > > (kafka connect or equivalent) where metadata from one end
> > could copied over
> > > to the other (S3, for example uses http headers for
> > user-accessible
> > > metadata). it will be kafka client code getting/setting
> > those headers. not
> > > an interceptor.
> > >
> > > On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > For APM single event tracing, need access to the header
> at
> > the point of
> > > > processing on the processing thread.
> > > >
> > > > As such interceptors will not work/be suitable for these,
> > due to the fact
> > > > they act on the ConsumerRecords as a batch, before the
> > handling thread
> > > can
> > > > split out and process per message which is the point
> these
> > tools will
> > > need
> > > > to continue to transaction tracing.
> > > >
> > > > Like wise tools and infra pieces will need access to the
> > message outside
> > > > the interceptor realm.
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 21:26, "Jason Gustafson" <
> > jason@confluent.io> wrote:
> > > >
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > >
> > > >
> > > > So the broker will parse the headers and validate
> > them. Good. The
> > > only
> > > > reason remaining that I can see to leave the headers
> > as a byte array
> > > > is to
> > > > make it easier for the client to skip past them. Are
> > we sure this is
> > > > not
> > > > premature optimization? Are there any performance
> > results which show
> > > > that
> > > > this is worthwhile?
> > > >
> > > > What’s the issue with exposing a method getHeaders on
> > the
> > > > producer/consumer
> > > > > record? It doesn’t break anything. We don’t need
> any
> > special
> > > version.
> > > >
> > > >
> > > > See my previous explanation. What I am trying to
> > resist is the
> > > headers
> > > > becoming a general application-level facility. The
> > primary use case
> > > as
> > > > far
> > > > as I can tell is middleware, which is the use case
> > that the
> > > > interceptors
> > > > are providing.
> > > >
> > > > Current batch consumer model and consumer
> interceptors
> > don’t work
> > > where
> > > > > headers need to be acted on at per message level at
> > time of
> > > > processing,
> > > > > very case is APM (the core one), where the header
> > value is used to
> > > > continue
> > > > > tracing.
> > > >
> > > >
> > > > I still don't understand the point about batching.
> The
> > consumer
> > > > records are
> > > > exposed as a batch in the consumer interceptor, but
> > you can still
> > > > iterate
> > > > them individually. It is no different for the
> consumer
> > API itself.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > Re:
> > > > >
> > > > > “ The point about creation of maps seems
> > orthogonal. We can
> > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the time
> > it is accessed.”
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > >
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:37, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > What’s the issue with exposing a method
> > getHeaders on the
> > > > > producer/consumer record? It doesn’t break
> anything.
> > We don’t need
> > > > any
> > > > > special version.
> > > > >
> > > > > Current batch consumer model and consumer
> > interceptors don’t
> > > work
> > > > > where headers need to be acted on at per message
> > level at time of
> > > > > processing, very case is APM (the core one), where
> > the header value
> > > > is used
> > > > > to continue tracing. JMS/HTTP etc all expose these,
> > without issues.
> > > > I would
> > > > > NOT want to lock this down to only be usable
> > accessible via
> > > > interceptors,
> > > > > as we’d fail on one of the main goals.
> > > > >
> > > > > Regards
> > > > > Mike
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:21, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > >
> > > > > The point about creation of maps seems
> > orthogonal. We can
> > > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the
> > time it is
> > > > accessed.
> > > > >
> > > > >
> > > > > > Yes exactly we have access to the records
> > thus why the
> > > > header
> > > > > should be
> > > > > > accessible via it and not hidden for only
> > interceptors to
> > > > access.
> > > > >
> > > > >
> > > > > As explained above, the point is to make
> the
> > intended usage
> > > > clear.
> > > > > Applications should continue to rely on the
> > key/value
> > > fields
> > > > to
> > > > > serialize
> > > > > their own headers, and it would be more
> > ideal if we can
> > > avoid
> > > > > leaking
> > > > > third-party headers into applications. This
> > is difficult to
> > > > do
> > > > > with the
> > > > > current interceptors because they share the
> > record objects
> > > > with
> > > > > the common
> > > > > API. What I had in mind is something like
> an
> > extension of
> > > the
> > > > > current
> > > > > interceptors which exposed a different
> > object (e.g.
> > > > > `RecordAndHeaders`).
> > > > > The challenge is for MM-like use cases. Let
> > me see if I can
> > > > come
> > > > > up with a
> > > > > concrete proposal for that problem.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 17, 2017 at 11:55 AM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > I am happy to move the definition of the
> > header into the
> > > > message
> > > > > body, but
> > > > > > would cause us not to lazy
> > initialise/parse the headers,
> > > as
> > > > > obviously, we
> > > > > > would have to traverse these reading the
> > message.
> > > > > >
> > > > > > This was actually one of Jay’s requests:
> > > > > >
> > > > > > “ 2. I think we should think about
> > creating the Map
> > > > lazily to
> > > > > avoid
> > > > > > parsing out all the headers into
> > little objects.
> > > > HashMaps
> > > > > themselves
> > > > > > are
> > > > > > kind of expensive and the consumer is
> > very perf
> > > > sensitive so
> > > > > and making
> > > > > > gazillions of hashmaps that may or
> may
> > not get used
> > > is
> > > > > probably a bad
> > > > > > idea.”
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > Yes exactly we have access to the
> > records thus why
> > > the
> > > > > header should
> > > > > > be accessible via it and not hidden for
> > only interceptors
> > > > to
> > > > > access.
> > > > > >
> > > > > > Sent using OWA for iPhone
> > > > > > ______________________________
> > __________
> > > > > > From: Magnus Edenhill <
> > magnus@edenhill.se>
> > > > > > Sent: Friday, February 17, 2017
> > 7:34:49 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> > Record Headers
> > > > > >
> > > > > > Big +1 on VarInts.
> > > > > > CPUs are fast, memory is slow.
> > > > > >
> > > > > > I agree with Jason that we'll want to
> > continue
> > > > verifying
> > > > > messages,
> > > > > > including their headers, so while I
> > appreciate the
> > > > idea of
> > > > > the opaque
> > > > > > header blob it won't be useful in
> > practice.
> > > > > >
> > > > > > /Magnus
> > > > > >
> > > > > > 2017-02-17 10:41 GMT-08:00 Jason
> > Gustafson <
> > > > > jason@confluent.io>:
> > > > > >
> > > > > > > Sorry, my mistake. The consumer
> > interceptor is per
> > > > batch,
> > > > > though I'm
> > > > > > not
> > > > > > > sure that's an actual limitation
> > since you still
> > > have
> > > > > access to the
> > > > > > > individual records.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 10:39 AM,
> > Jason Gustafson <
> > > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Re headers as byte array and
> > future use by
> > > broker.
> > > > This
> > > > > doesn't
> > > > > > take away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >
> > > > > > > >
> > > > > > > > Yeah, I didn't say it was
> > difficult, only
> > > awkward.
> > > > You
> > > > > wouldn't
> > > > > > write the
> > > > > > > > schema that way if you were
> > planning to use it on
> > > > the
> > > > > brokers from
> > > > > > the
> > > > > > > > beginning. Note also that one of
> > the benefits of
> > > > letting
> > > > > the broker
> > > > > > > > understand headers is that it can
> > validate that
> > > > they are
> > > > > properly
> > > > > > > > formatted. If cost is the only
> > concern, we should
> > > > > confirm its
> > > > > > impact
> > > > > > > > through performance testing.
> > > > > > > >
> > > > > > > > One of the key use cases requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >
> > > > > > > >
> > > > > > > > I'm not sure I understand the
> > point about
> > > batching.
> > > > > Interceptors
> > > > > > are
> > > > > > > > applied per-message, right?
> > > > > > > >
> > > > > > > > My intent on interceptors is to
> > keep the usage of
> > > > headers
> > > > > > well-defined so
> > > > > > > > that they don't start leaking
> > unnecessarily into
> > > > > applications. My
> > > > > > guess
> > > > > > > is
> > > > > > > > that it's probably inevitable,
> but
> > isolating it
> > > in
> > > > the
> > > > > > interceptors would
> > > > > > > > at least give people a second
> > thought before
> > > > deciding to
> > > > > use it.
> > > > > > The main
> > > > > > > > challenge in my mind is figuring
> > out how an MM
> > > use
> > > > case
> > > > > would
> > > > > > work. It
> > > > > > > > would be more cumbersome to
> > replicate headers
> > > > through an
> > > > > > interceptor,
> > > > > > > > though arguably MM should be
> > working at a lower
> > > > level
> > > > > anyway.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at 10:16 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Re headers available on the
> > record va
> > > > interceptors only
> > > > > > > >>
> > > > > > > >> One of the key use cases
> requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >>
> > > > > > > >> This anyhow is similar to
> > jms/http/amqp where
> > > > headers
> > > > > are
> > > > > > available to
> > > > > > > >> consuming applications.
> > > > > > > >>
> > > > > > > >> Re headers as byte array and
> > future use by
> > > > broker. This
> > > > > doesn't
> > > > > > take
> > > > > > > away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Sent using OWA for iPhone
> > > > > > > >> ______________________________
> > __________
> > > > > > > >> From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > >> Sent: Friday, February 17, 2017
> > 5:55:42 PM
> > > > > > > >> To: dev@kafka.apache.org
> > > > > > > >> Subject: Re: [DISCUSS] KIP-82 -
> > Add Record
> > > Headers
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Yes, that is what we are
> > proposing, so using
> > > > varints
> > > > > for headers
> > > > > > would
> > > > > > > be
> > > > > > > >> consistent with the rest of the
> > message. We have
> > > > used
> > > > > static
> > > > > > helper
> > > > > > > >> methods
> > > > > > > >> in our prototype implementation.
> > > > > > > >>
> > > > > > > >> The cost of parsing, we want to
> > parse/interpret
> > > > the
> > > > > headers
> > > > > > lazily (this
> > > > > > > >> is
> > > > > > > >> > a key point brought up earlier
> > in discussions)
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> I'm a bit skeptical of this. Has
> > anyone done the
> > > > > performance
> > > > > > testing? I
> > > > > > > >> can
> > > > > > > >> probably implement it and test
> it
> > if no one else
> > > > has. I
> > > > > was also
> > > > > > under
> > > > > > > the
> > > > > > > >> impression that there may be use
> > cases down the
> > > > road
> > > > > where the
> > > > > > broker
> > > > > > > >> would
> > > > > > > >> need to interpret headers. That
> > wouldn't be off
> > > > the
> > > > > table in the
> > > > > > future
> > > > > > > if
> > > > > > > >> it's represented as bytes, but
> it
> > would be quite
> > > > a bit
> > > > > more
> > > > > > awkward,
> > > > > > > >> right?
> > > > > > > >>
> > > > > > > >> By the way, one question I have
> > been wondering
> > > > about. My
> > > > > > understanding
> > > > > > > is
> > > > > > > >> that headers are primarily for
> > use cases where a
> > > > > third-party
> > > > > > components
> > > > > > > >> wants to enrich messages without
> > needing to
> > > > understand
> > > > > or modify
> > > > > > the
> > > > > > > >> schema
> > > > > > > >> of the message key and value.
> For
> > the
> > > applications
> > > > > which directly
> > > > > > > produce
> > > > > > > >> and consume the messages and
> > control the
> > > key/value
> > > > > schema
> > > > > > directly, it
> > > > > > > >> seems we would rather have them
> > implement
> > > headers
> > > > > directly in
> > > > > > their own
> > > > > > > >> schema. Supposing for the sake
> of
> > argument that
> > > > it was
> > > > > possible,
> > > > > > my
> > > > > > > >> question is whether it be
> > sufficient to expose
> > > the
> > > > > headers in the
> > > > > > > >> interceptor API and not in the
> > common API?
> > > > > > > >>
> > > > > > > >> -Jason
> > > > > > > >>
> > > > > > > >> On Fri, Feb 17, 2017 at 3:26 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > On the point of varInts
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 11:22, "Michael
> > Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > On the point re: headers
> in
> > the message
> > > > protocol
> > > > > being a
> > > > > > byte
> > > > > > > array
> > > > > > > >> > and not a count of elements
> > followed by the
> > > > elements.
> > > > > Again
> > > > > > this was
> > > > > > > >> > discussed/argued previously.
> > > > > > > >> >
> > > > > > > >> > It was agreed on for a few
> > reasons some of
> > > > which
> > > > > you have
> > > > > > > obviously
> > > > > > > >> > picked up on:
> > > > > > > >> >
> > > > > > > >> > Broker is able to pass it
> > through opaquely
> > > > > > > >> > The cost of parsing, we
> > want to
> > > > parse/interpret
> > > > > the headers
> > > > > > lazily
> > > > > > > >> > (this is a key point brought
> up
> > earlier in
> > > > > discussions)
> > > > > > > >> > Headers can be copied from
> > consumer record
> > > > to
> > > > > producer
> > > > > > record (aka
> > > > > > > >> > mirror makers etc) without
> > parsing if no
> > > > changes are
> > > > > being made
> > > > > > or
> > > > > > > being
> > > > > > > >> > looked at.
> > > > > > > >> > Keeps the broker agnostic
> > to the format
> > > > > > > >> > You need an int32 either
> > for the byte size
> > > > of the
> > > > > headers,
> > > > > > or for
> > > > > > > >> the
> > > > > > > >> > count of elements, so
> overheads
> > are the same,
> > > > but
> > > > > with going
> > > > > > with an
> > > > > > > >> opaque
> > > > > > > >> > byte array has the above
> > advantages.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 02:50,
> > "Jason Gustafson" <
> > > > > jason@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > Sorry, should have
> > noted that the
> > > > performance
> > > > > testing
> > > > > > was done
> > > > > > > >> > using the
> > > > > > > >> > producer performance
> > tool shipped with
> > > > Kafka.
> > > > > > > >> >
> > > > > > > >> > -Jason
> > > > > > > >> >
> > > > > > > >> > On Thu, Feb 16, 2017
> at
> > 6:44 PM, Jason
> > > > > Gustafson <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hey Nacho,
> > > > > > > >> > >
> > > > > > > >> > > I've compared
> > performance of our
> > > > KIP-98
> > > > > > implementation with
> > > > > > > >> and
> > > > > > > >> > without
> > > > > > > >> > > varints. For
> messages
> > around 128
> > > > bytes, we
> > > > > see an
> > > > > > increase
> > > > > > > in
> > > > > > > >> > throughput of
> > > > > > > >> > > about 30% using the
> > default
> > > > configuration
> > > > > settings.
> > > > > > At 256
> > > > > > > >> > bytes, the
> > > > > > > >> > > increase is around
> > 16%. Obviously
> > > the
> > > > > performance
> > > > > > converge
> > > > > > > as
> > > > > > > >> > messages get
> > > > > > > >> > > larger, but it seems
> > well worth the
> > > > cost.
> > > > > Note that
> > > > > > we are
> > > > > > > >> also
> > > > > > > >> > seeing a
> > > > > > > >> > > substantial
> > performance increase
> > > > against
> > > > > trunk
> > > > > > primarily
> > > > > > > >> because
> > > > > > > >> > of the
> > > > > > > >> > > much more efficient
> > packing that
> > > > varints
> > > > > provide us.
> > > > > > > Anything
> > > > > > > >> > adding to
> > > > > > > >> > > message overhead,
> > such as record
> > > > headers,
> > > > > would only
> > > > > > > increase
> > > > > > > >> > the relative
> > > > > > > >> > > difference. (Of
> > course take these
> > > > numbers
> > > > > with a
> > > > > > grain of
> > > > > > > salt
> > > > > > > >> > since I have
> > > > > > > >> > > only used the
> default
> > settings with
> > > > both
> > > > > the producer
> > > > > > and
> > > > > > > >> broker
> > > > > > > >> > on my
> > > > > > > >> > > local machine. We
> > intend to provide
> > > > more
> > > > > extensive
> > > > > > > performance
> > > > > > > >> > details as
> > > > > > > >> > > part of the work for
> > KIP-98.)
> > > > > > > >> > >
> > > > > > > >> > > The implementation
> we
> > are using is
> > > > from
> > > > > protobuf (
> > > > > > > >> > >
> > https://developers.google.com/
> > > > > > > protocol-buffers/docs/encoding
> > > > > > > >> ),
> > > > > > > >> > which is
> > > > > > > >> > > also used in HBase.
> > It is trivial to
> > > > > implement and as
> > > > > > far
> > > > > > > as I
> > > > > > > >> > know doesn't
> > > > > > > >> > > suffer from the
> > aliasing problem you
> > > > are
> > > > > describing. I
> > > > > > > checked
> > > > > > > >> > with Magnus
> > > > > > > >> > > (the author of
> > librdkafka) and he
> > > > agreed
> > > > > that the
> > > > > > savings
> > > > > > > >> seemed
> > > > > > > >> > worth the
> > > > > > > >> > > cost of
> > implementation.
> > > > > > > >> > >
> > > > > > > >> > > -Jason
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Feb 16, 2017
> > at 4:32 PM,
> > > > Ignacio
> > > > > Solis <
> > > > > > > >> isolis@igso.net>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> -VarInts
> > > > > > > >> > >>
> > > > > > > >> > >> I'm one of the
> > people (if not the
> > > > most)
> > > > > opposed to
> > > > > > VarInts.
> > > > > > > >> > VarInts
> > > > > > > >> > >> have a place, but
> > this is not it.
> > > > (We
> > > > > had a large
> > > > > > > >> discussion
> > > > > > > >> > about
> > > > > > > >> > >> them at the
> > beginning of KIP-82
> > > time)
> > > > > > > >> > >>
> > > > > > > >> > >> If anybody has real
> > life
> > > performance
> > > > > numbers of
> > > > > > VarInts
> > > > > > > >> > improving
> > > > > > > >> > >> things or
> > significantly reducing
> > > > resources
> > > > > I would
> > > > > > like to
> > > > > > > >> know
> > > > > > > >> > what
> > > > > > > >> > >> that case may be.
> > Yes, you can save
> > > > some
> > > > > bytes here
> > > > > > and
> > > > > > > >> there,
> > > > > > > >> > but
> > > > > > > >> > >> this is probably
> > insignificant to
> > > the
> > > > > overall system
> > > > > > > behavior
> > > > > > > >> > and
> > > > > > > >> > >> storage
> > requirements. -- I say
> > > this
> > > > with
> > > > > respect to
> > > > > > using
> > > > > > > >> > VarInts in
> > > > > > > >> > >> the protocol
> itself,
> > not as part of
> > > > the
> > > > > data.
> > > > > > > >> > >>
> > > > > > > >> > >> VarInts require you
> > to parse the
> > > Int
> > > > > before using it
> > > > > > and
> > > > > > > >> > depending on
> > > > > > > >> > >> the encoding they
> > can suffer from
> > > > aliasing
> > > > > (multiple
> > > > > > > >> > representations
> > > > > > > >> > >> for the same
> value).
> > > > > > > >> > >>
> > > > > > > >> > >> Why add complexity?
> > > > > > > >> > >>
> > > > > > > >> > >> Nacho
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> On Thu, Feb 16,
> 2017
> > at 10:29 AM,
> > > > Colin
> > > > > McCabe <
> > > > > > > >> > cmccabe@apache.org>
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> > +1 for varints
> > here-- it would
> > > save
> > > > > quite a bit of
> > > > > > space.
> > > > > > > >> > They are
> > > > > > > >> > >> > pretty quick to
> > implement as
> > > well.
> > > > > > > >> > >> >
> > > > > > > >> > >> > I think it makes
> > sense for values
> > > > to be
> > > > > byte
> > > > > > arrays.
> > > > > > > Users
> > > > > > > >> > might want
> > > > > > > >> > >> > to attach
> > arbitrary payloads;
> > > they
> > > > > shouldn't be
> > > > > > forced to
> > > > > > > >> > serialize
> > > > > > > >> > >> > everything to
> Java
> > strings.
> > > > > > > >> > >> >
> > > > > > > >> > >> > best,
> > > > > > > >> > >> > Colin
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Thu, Feb 16,
> > 2017, at 09:52,
> > > > Jason
> > > > > Gustafson
> > > > > > wrote:
> > > > > > > >> > >> >> Hey Michael,
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Hmm, I guess the
> > point of
> > > > representing
> > > > > it as
> > > > > > bytes is to
> > > > > > > >> > allow the
> > > > > > > >> > >> broker
> > > > > > > >> > >> >> to pass it
> > through opaquely? Is
> > > > the
> > > > > cost of
> > > > > > parsing
> > > > > > > them a
> > > > > > > >> > concern, or
> > > > > > > >> > >> >> are
> > > > > > > >> > >> >> we simply trying
> > to ensure that
> > > > the
> > > > > broker stays
> > > > > > > agnostic
> > > > > > > >> to
> > > > > > > >> > the
> > > > > > > >> > >> format?
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On varints, I
> > think adding
> > > > support for
> > > > > them makes
> > > > > > less
> > > > > > > >> sense
> > > > > > > >> > for an
> > > > > > > >> > >> >> isolated use
> > case, but as part
> > > of
> > > > a
> > > > > more holistic
> > > > > > change
> > > > > > > >> > (such as what
> > > > > > > >> > >> we
> > > > > > > >> > >> >> have proposed in
> > KIP-98), I
> > > think
> > > > they
> > > > > are
> > > > > > justifiable.
> > > > > > > If
> > > > > > > >> > we add them,
> > > > > > > >> > >> >> then the need to
> > use attributes
> > > > becomes
> > > > > quite a
> > > > > > bit
> > > > > > > >> weaker,
> > > > > > > >> > right? The
> > > > > > > >> > >> >> other thing I
> > find slightly odd
> > > > is the
> > > > > fact that
> > > > > > null
> > > > > > > >> > headers has no
> > > > > > > >> > >> >> actual
> > > > > > > >> > >> >> semantic meaning
> > for the message
> > > > > (unlike null
> > > > > > keys and
> > > > > > > >> > values). It is
> > > > > > > >> > >> >> just
> > > > > > > >> > >> >> a space
> > optimization. It seems a
> > > > bit
> > > > > better to
> > > > > > always
> > > > > > > use
> > > > > > > >> > size 0 to
> > > > > > > >> > >> >> indicate having
> > no headers.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Overall, the
> main
> > point is
> > > > ensuring
> > > > > that the
> > > > > > message
> > > > > > > >> schema
> > > > > > > >> > remains
> > > > > > > >> > >> >> consistent,
> > either within the
> > > > larger
> > > > > protocol, or
> > > > > > at a
> > > > > > > >> > minimum within
> > > > > > > >> > >> the
> > > > > > > >> > >> >> message itself.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> -Jason
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Thu, Feb 16,
> > 2017 at 6:39 AM,
> > > > > Michael Pearce <
> > > > > > > >> > Michael.Pearce@ig.com
> > > > > > > >> > >> >
> > > > > > > >> > >> >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> > Hi Jason,
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 1) in
> > the message
> > > > protocol
> > > > > the headers
> > > > > > are
> > > > > > > >> simply
> > > > > > > >> > a byte
> > > > > > > >> > >> array,
> > > > > > > >> > >> >> > as like the
> key
> > or value, this
> > > > is to
> > > > > clearly
> > > > > > demarcate
> > > > > > > >> the
> > > > > > > >> > header in
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > core message.
> > Then the header
> > > > byte
> > > > > array in the
> > > > > > core
> > > > > > > >> > message is an
> > > > > > > >> > >> array of
> > > > > > > >> > >> >> > key, value
> > pairs. This is what
> > > > it is
> > > > > denoting.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Then this
> would
> > be I guess in
> > > > the
> > > > > given
> > > > > > notation:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [KeyLength, Key,
> > > > > ValueLength, Value]
> > > > > > > >> > >> >> > KeyLength
> > => int32
> > > > > <-----------------NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised key
> > value
> > > > > > > >> > >> >> > Key =>
> bytes
> > > > > <---------------------- NEW
> > > > > > > serialised
> > > > > > > >> > string (UTF8)
> > > > > > > >> > >> >> > bytes of the
> > header key
> > > > > > > >> > >> >> >
> ValueLength
> > => int32
> > > > > <-------------- NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised
> > header value
> > > > > > > >> > >> >> > Value =>
> > bytes
> > > > > <-------------------- NEW
> > > > > > > serialised
> > > > > > > >> > form of the
> > > > > > > >> > >> header
> > > > > > > >> > >> >> > value
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The key length
> > and value
> > > length
> > > > is
> > > > > matching the
> > > > > > way
> > > > > > > the
> > > > > > > >> > protocol is
> > > > > > > >> > >> >> > defined in the
> > core message
> > > > currently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 2)
> > > > > > > >> > >> >> > Var sized
> ints,
> > this was
> > > > discussed
> > > > > much earlier
> > > > > > on, in
> > > > > > > >> > fact I had
> > > > > > > >> > >> >> > suggested it
> > myself (with
> > > Hadoop
> > > > > references),
> > > > > > the
> > > > > > > >> > complexity of this
> > > > > > > >> > >> >> > compared to
> > having a simpler
> > > > protocol
> > > > > was
> > > > > > argued and
> > > > > > > >> > agreed it
> > > > > > > >> > >> wasn’t worth
> > > > > > > >> > >> >> > the complexity
> > as all other
> > > > clients
> > > > > in other
> > > > > > languages
> > > > > > > >> > would need to
> > > > > > > >> > >> ensure
> > > > > > > >> > >> >> > theyre using
> > the right var
> > > size
> > > > > algorithm, as
> > > > > > there
> > > > > > > is a
> > > > > > > >> > few.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 3)
> > > > > > > >> > >> >> > We did the
> > attributes,
> > > optional
> > > > > approach as
> > > > > > originally
> > > > > > > >> > there was
> > > > > > > >> > >> marked
> > > > > > > >> > >> >> > concern that
> > headers would
> > > > cause a
> > > > > message size
> > > > > > > overhead
> > > > > > > >> > for others,
> > > > > > > >> > >> who
> > > > > > > >> > >> >> > don’t want
> > them. As such this
> > > > is the
> > > > > clean
> > > > > > solution to
> > > > > > > >> > achieve that.
> > > > > > > >> > >> If
> > > > > > > >> > >> >> > that no longer
> > holds, and we
> > > > don’t
> > > > > care that we
> > > > > > add
> > > > > > > >> 4bytes
> > > > > > > >> > overhead,
> > > > > > > >> > >> then
> > > > > > > >> > >> >> > im happy to
> > remove.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I’m personally
> > in favour of
> > > > keeping
> > > > > the message
> > > > > > as
> > > > > > > small
> > > > > > > >> > as possible
> > > > > > > >> > >> so
> > > > > > > >> > >> >> > people don’t
> > get shocks in
> > > perf
> > > > and
> > > > > throughputs
> > > > > > dues
> > > > > > > to
> > > > > > > >> > message size,
> > > > > > > >> > >> >> > unless they
> > actively use the
> > > > feature,
> > > > > as such I
> > > > > > do
> > > > > > > >> prefer
> > > > > > > >> > the
> > > > > > > >> > >> attribute bit
> > > > > > > >> > >> >> > wise feature
> > flag approach
> > > > myself.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On 16/02/2017,
> > 05:40, "Jason
> > > > > Gustafson" <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > We have
> > proposed a few
> > > > > significant changes
> > > > > > to the
> > > > > > > >> > message format
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > KIP-98
> > > > > > > >> > >> >> > which now
> > seems likely to
> > > > pass
> > > > > (perhaps
> > > > > > with some
> > > > > > > >> > iterations on
> > > > > > > >> > >> >> >
> > implementation details).
> > > It
> > > > would
> > > > > be good
> > > > > > to try
> > > > > > > and
> > > > > > > >> > coordinate
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > changes
> > > > > > > >> > >> >> > in both of
> > the proposals
> > > to
> > > > make
> > > > > sure they
> > > > > > are
> > > > > > > >> > consistent and
> > > > > > > >> > >> >> > compatible.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I think
> > using the
> > > > attributes to
> > > > > indicate
> > > > > > null
> > > > > > > >> headers
> > > > > > > >> > is a
> > > > > > > >> > >> reasonable
> > > > > > > >> > >> >> > approach.
> > We have proposed
> > > > to do
> > > > > the same
> > > > > > thing
> > > > > > > for
> > > > > > > >> > the message
> > > > > > > >> > >> key and
> > > > > > > >> > >> >> > value.
> That
> > said, I
> > > > sympathize
> > > > > with Jay's
> > > > > > > argument.
> > > > > > > >> > Having
> > > > > > > >> > >> multiple
> > > > > > > >> > >> >> > ways to
> > > > > > > >> > >> >> > specify a
> > null value
> > > > increases
> > > > > the overall
> > > > > > > >> complexity
> > > > > > > >> > of the
> > > > > > > >> > >> protocol.
> > > > > > > >> > >> >> > You
> > > > > > > >> > >> >> > can see
> > this just from the
> > > > fact
> > > > > that you
> > > > > > need the
> > > > > > > >> > extra verbiage
> > > > > > > >> > >> in the
> > > > > > > >> > >> >> > protocol
> > specification in
> > > > this
> > > > > KIP and in
> > > > > > KIP-98
> > > > > > > to
> > > > > > > >> > describe the
> > > > > > > >> > >> >> > dependence
> > > > > > > >> > >> >> > between
> the
> > fields and the
> > > > > attributes. It
> > > > > > seems
> > > > > > > >> like a
> > > > > > > >> > slippery
> > > > > > > >> > >> slope
> > > > > > > >> > >> >> > if
> > > > > > > >> > >> >> > you start
> > allowing
> > > different
> > > > > request types
> > > > > > to
> > > > > > > >> > implement the
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> >
> > specification differently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > You can
> > also argue that
> > > the
> > > > > messages
> > > > > > already are
> > > > > > > and
> > > > > > > >> > are likely
> > > > > > > >> > >> to
> > > > > > > >> > >> >> > remain a
> > > > > > > >> > >> >> > special
> > case. For example,
> > > > there
> > > > > is
> > > > > > currently no
> > > > > > > >> > generality in
> > > > > > > >> > >> how
> > > > > > > >> > >> >> > compressed
> > message sets
> > > are
> > > > > represented
> > > > > > that would
> > > > > > > >> be
> > > > > > > >> > applicable
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > other
> > > > > > > >> > >> >> > request
> > types. Some might
> > > > see this
> > > > > > divergence as
> > > > > > > an
> > > > > > > >> > unfortunate
> > > > > > > >> > >> >> > protocol
> > > > > > > >> > >> >> > deficiency
> > which should be
> > > > fixed;
> > > > > others
> > > > > > might see
> > > > > > > >> it
> > > > > > > >> > as sort of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> >
> > inevitability of needing
> > > to
> > > > > optimize where
> > > > > > it
> > > > > > > counts
> > > > > > > >> > most. I'm
> > > > > > > >> > >> probably
> > > > > > > >> > >> >> > somewhere
> > in between, but
> > > I
> > > > think
> > > > > we
> > > > > > probably all
> > > > > > > >> > share the
> > > > > > > >> > >> intuition
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> > protocol should be
> > > kept
> > > > as
> > > > > consistent as
> > > > > > > >> possible.
> > > > > > > >> > With that
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > mind,
> > > > > > > >> > >> >> > here are a
> > few comments:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 1. One
> > thing I found a
> > > > little odd
> > > > > when
> > > > > > reading the
> > > > > > > >> > current
> > > > > > > >> > >> proposal is
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> headers
> > are both
> > > > represented
> > > > > as an
> > > > > > array of
> > > > > > > >> bytes
> > > > > > > >> > and as an
> > > > > > > >> > >> array
> > > > > > > >> > >> >> > of
> > > > > > > >> > >> >> > key/value
> > pairs. I'd
> > > > probably
> > > > > suggest
> > > > > > something
> > > > > > > like
> > > > > > > >> > this:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [HeaderKey
> > > > HeaderValue]
> > > > > > > >> > >> >> > HeaderKey
> > => String
> > > > > > > >> > >> >> >
> > HeaderValue => Bytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > An array
> in
> > the Kafka
> > > > protocol is
> > > > > > represented as a
> > > > > > > >> > 4-byte integer
> > > > > > > >> > >> >> > indicating
> > the number of
> > > > elements
> > > > > in the
> > > > > > array
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> >
> > serialization of the
> > > > elements.
> > > > > Unless I'm
> > > > > > > >> > misunderstanding, what
> > > > > > > >> > >> you
> > > > > > > >> > >> >> > have
> > > > > > > >> > >> >> > instead is
> > the total size
> > > > of the
> > > > > headers in
> > > > > > bytes
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> > elements.
> > > > > > > >> > >> >> > I'm not
> > sure I see any
> > > > reason for
> > > > > this
> > > > > > > >> inconsistency.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 2. In
> > KIP-98, we've
> > > > introduced
> > > > > > variable-length
> > > > > > > >> integer
> > > > > > > >> > fields.
> > > > > > > >> > >> >> > Effectively,
> > > > > > > >> > >> >> > we've
> > enriched (or
> > > > "complicated"
> > > > > as Jay
> > > > > > might say
> > > > > > > ;)
> > > > > > > >> > the protocol
> > > > > > > >> > >> >> >
> > specification to include
> > > the
> > > > > following
> > > > > > types:
> > > > > > > >> VarInt,
> > > > > > > >> > VarLong,
> > > > > > > >> > >> >> >
> > UnsignedVarInt and
> > > > > UnsignedVarLong.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Along with
> > these
> > > > primitives, we
> > > > > could
> > > > > > introduce
> > > > > > > the
> > > > > > > >> > following
> > > > > > > >> > >> types:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeArray =>
> > > > NumberOfItems
> > > > > Item1 Item2
> > > > > > .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItems =>
> > > > UnsignedVarInt
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeNullableArray =>
> > > > > NumberOfItemsOrNull
> > > > > > Item1
> > > > > > > >> > Item2 .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItemsOrNull =>
> > > > VarInt
> > > > > (-1 means
> > > > > > null)
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > And
> > similarly for the
> > > > `String`
> > > > > and `Bytes`
> > > > > > types.
> > > > > > > >> > These types
> > > > > > > >> > >> can save
> > > > > > > >> > >> >> > a
> > > > > > > >> > >> >> >
> > considerable amount of
> > > > space in
> > > > > this
> > > > > > proposal
> > > > > > > >> because
> > > > > > > >> > they can
> > > > > > > >> > >> be used
> > > > > > > >> > >> >> > for
> > > > > > > >> > >> >> > both the
> > number of headers
> > > > > included in the
> > > > > > message
> > > > > > > >> and
> > > > > > > >> > the
> > > > > > > >> > >> lengths of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > header
> keys
> > and values. We
> > > > could
> > > > > do this
> > > > > > instead:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > > > VarSizeArray[HeaderKey
> > > > > > HeaderValue]
> > > > > > > >> > >> >> >
> HeaderKey
> > =>
> > > VarSizeString
> > > > > > > >> > >> >> >
> > HeaderValue =>
> > > > VarSizeBytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Combining
> > the savings from
> > > > the
> > > > > use of
> > > > > > variable
> > > > > > > >> length
> > > > > > > >> > fields, the
> > > > > > > >> > >> >> > benefit
> > > > > > > >> > >> >> > of using
> > the attributes to
> > > > > represent null
> > > > > > seems
> > > > > > > >> pretty
> > > > > > > >> > small.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 3.
> > Whichever way we go
> > > > (whether
> > > > > we use the
> > > > > > > >> attributes
> > > > > > > >> > or not), we
> > > > > > > >> > >> >> > should at
> > > > > > > >> > >> >> > least be
> > consistent
> > > between
> > > > this
> > > > > KIP and
> > > > > > KIP-98.
> > > > > > > It
> > > > > > > >> > would be very
> > > > > > > >> > >> >> > strange
> > > > > > > >> > >> >> > to have
> two
> > ways to
> > > > represent
> > > > > null values
> > > > > > in the
> > > > > > > >> same
> > > > > > > >> > schema.
> > > > > > > >> > >> Either
> > > > > > > >> > >> >> > way is
> > > > > > > >> > >> >> > OK with
> me.
> > I think some
> > > > > message-level
> > > > > > > optimizations
> > > > > > > >> > are
> > > > > > > >> > >> justifiable,
> > > > > > > >> > >> >> > but
> > > > > > > >> > >> >> > the
> savings
> > here seem
> > > > minimal (a
> > > > > few bytes
> > > > > > per
> > > > > > > >> > message), so
> > > > > > > >> > >> maybe it's
> > > > > > > >> > >> >> > not
> > > > > > > >> > >> >> > worth the
> > cost of letting
> > > > the
> > > > > message
> > > > > > diverge even
> > > > > > > >> > further from
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > rest of
> > > > > > > >> > >> >> > the
> > protocol.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > -Jason
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On Wed,
> Feb
> > 15, 2017 at
> > > > 8:52 AM,
> > > > > radai <
> > > > > > > >> > >>
> > radai.rosenblatt@gmail.com>
> > > > > > > >> > >> >> > wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > > I've
> > trimmed the inline
> > > > > contents as this
> > > > > > mail is
> > > > > > > >> > getting too
> > > > > > > >> > >> big for
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > apache
> > mailing list
> > > > software to
> > > > > deliver
> > > > > > :-(
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 1. the
> > important thing
> > > for
> > > > > > interoperability is
> > > > > > > for
> > > > > > > >> > different
> > > > > > > >> > >> >> > "interested
> > > > > > > >> > >> >> > > parties"
> > (plugins, infra
> > > > > layers/wrappers,
> > > > > > > >> user-code)
> > > > > > > >> > to be
> > > > > > > >> > >> able to
> > > > > > > >> > >> >> > stick
> > > > > > > >> > >> >> > > pieces
> of
> > metadata onto
> > > > msgs
> > > > > without
> > > > > > getting in
> > > > > > > >> each
> > > > > > > >> > other's
> > > > > > > >> > >> way. a
> > > > > > > >> > >> >> > common
> > > > > > > >> > >> >> > > key
> > scheme (Strings, as
> > > > of the
> > > > > time of
> > > > > > this
> > > > > > > >> > writing?) is all
> > > > > > > >> > >> thats
> > > > > > > >> > >> >> > required
> > > > > > > >> > >> >> > > for
> that.
> > it is assumed
> > > > that
> > > > > the other end
> > > > > > > >> > interested in any
> > > > > > > >> > >> such
> > > > > > > >> > >> >> > piece of
> > > > > > > >> > >> >> > > metadata
> > knows the
> > > > encoding,
> > > > > and byte[]
> > > > > > provides
> > > > > > > >> for
> > > > > > > >> > the most
> > > > > > > >> > >> >> > flexibility.
> > > > > > > >> > >> >> > > i
> believe
> > this is the
> > > same
> > > > > logic behind
> > > > > > core
> > > > > > > kafka
> > > > > > > >> > being
> > > > > > > >> > >> >> > byte[]/byte[]
> -
> > > > > > > >> > >> >> > > Strings
> > are more
> > > "usable"
> > > > but
> > > > > bytes are
> > > > > > flexible
> > > > > > > >> and
> > > > > > > >> > so were
> > > > > > > >> > >> chosen.
> > > > > > > >> > >> >> > > Also -
> > core kafka doesnt
> > > > even
> > > > > do that
> > > > > > good of a
> > > > > > > >> job
> > > > > > > >> > on
> > > > > > > >> > >> usability of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > payload
> > (example - i
> > > have
> > > > to
> > > > > specify the
> > > > > > nop
> > > > > > > >> byte[]
> > > > > > > >> > "decoders"
> > > > > > > >> > >> >> > explicitly
> > > > > > > >> > >> >> > > in
> conf),
> > and again
> > > > sacrificies
> > > > > usability
> > > > > > for
> > > > > > > the
> > > > > > > >> > sake of
> > > > > > > >> > >> >> > performance
> (no
> > > > > > > >> > >> >> > >
> > convenient single-record
> > > > > processing as
> > > > > > poll is a
> > > > > > > >> > batch, lots of
> > > > > > > >> > >> >> > obscure
> > > > > > > >> > >> >> > > little
> > config details
> > > > exposing
> > > > > internals
> > > > > > of the
> > > > > > > >> > batching
> > > > > > > >> > >> mechanism,
> > > > > > > >> > >> >> > etc)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > this is
> > also why i
> > > really
> > > > > dislike the
> > > > > > idea of a
> > > > > > > >> > "type system"
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > header
> > > > > > > >> > >> >> > > values,
> > it further
> > > > degrades the
> > > > > > usability, adds
> > > > > > > >> > complexity and
> > > > > > > >> > >> will
> > > > > > > >> > >> >> > >
> > eventually get in
> > > > people's way,
> > > > > also, it
> > > > > > would
> > > > > > > be
> > > > > > > >> > the 2nd/3rd
> > > > > > > >> > >> >> > home-group
> > > > > > > >> > >> >> > >
> > serialization mechanism
> > > > in core
> > > > > kafka
> > > > > > (counting
> > > > > > > 2
> > > > > > > >> > iterations
> > > > > > > >> > >> of the
> > > > > > > >> > >> >> > "type
> > > > > > > >> > >> >> > >
> > definition DSL")
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 2. this
> > is an
> > > > implementation
> > > > > detail, and
> > > > > > not
> > > > > > > even
> > > > > > > >> a
> > > > > > > >> > very "user
> > > > > > > >> > >> >> > facing" one?
> > > > > > > >> > >> >> > > to the
> > best of my
> > > > understanding
> > > > > the vote
> > > > > > process
> > > > > > > >> is
> > > > > > > >> > on proposed
> > > > > > > >> > >> >> > >
> > API/behaviour. also -
> > > > since
> > > > > we're willing
> > > > > > to go
> > > > > > > >> with
> > > > > > > >> > strings
> > > > > > > >> > >> just
> > > > > > > >> > >> >> > serialize
> > > > > > > >> > >> >> > > a
> 0-sized
> > header blob
> > > and
> > > > IIUC
> > > > > you dont
> > > > > > need any
> > > > > > > >> > optionals
> > > > > > > >> > >> anymore.
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 3. yes,
> > we can :-)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > On Tue,
> > Feb 14, 2017 at
> > > > 11:56
> > > > > PM, Michael
> > > > > > > Pearce <
> > > > > > > >> > >> >> >
> > Michael.Pearce@ig.com>
> > > > > > > >> > >> >> > > wrote:
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > > Hi
> Jay,
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1)
> > There was some
> > > > initial
> > > > > debate on the
> > > > > > value
> > > > > > > >> > part, as youll
> > > > > > > >> > >> note
> > > > > > > >> > >> >> > String,
> > > > > > > >> > >> >> > > > String
> > headers were
> > > > > discounted early
> > > > > > on. The
> > > > > > > >> > reason for this
> > > > > > > >> > >> is
> > > > > > > >> > >> >> > >
> > flexibility
> > > > > > > >> > >> >> > > > and
> > keeping in line
> > > > with the
> > > > > > flexibility of
> > > > > > > key,
> > > > > > > >> > value of the
> > > > > > > >> > >> >> > message
> > > > > > > >> > >> >> > > > object
> > itself. I don’t
> > > > think
> > > > > it takes
> > > > > > away
> > > > > > > from
> > > > > > > >> an
> > > > > > > >> > ecosystem
> > > > > > > >> > >> as
> > > > > > > >> > >> >> > each
> > > > > > > >> > >> >> > > plugin
> > > > > > > >> > >> >> > > > will
> > care for their
> > > own
> > > > key,
> > > > > this way
> > > > > > ints,
> > > > > > > >> > booleans , exotic
> > > > > > > >> > >> >> > custom
> > > > > > > >> > >> >> > > binary
> > > > > > > >> > >> >> > > > can
> all
> > be catered
> > > for=.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > wanted
> > > > to
> > > > > push for a
> > > > > > typed
> > > > > > > >> value
> > > > > > > >> > interface,
> > > > > > > >> > >> I
> > > > > > > >> > >> >> > wouldn’t
> > > > > > > >> > >> >> > > > want
> > just String
> > > values
> > > > > supported, but
> > > > > > the the
> > > > > > > >> > primatives
> > > > > > > >> > >> plus
> > > > > > > >> > >> >> > string and
> > > > > > > >> > >> >> > > > also
> > still keeping the
> > > > > ability to have a
> > > > > > > binary
> > > > > > > >> > for custom
> > > > > > > >> > >> >> > binaries that
> > > > > > > >> > >> >> > > > some
> > organisations may
> > > > have.
> > > > > > > >> > >> >> > > > i. I
> > have written this
> > > > slight
> > > > > > alternative
> > > > > > > here,
> > > > > > > >> > >> >> > >
> > > https://cwiki.apache.org/
> > > > > > > >> > >> >> > > >
> > > > confluence/display/KAFKA/KIP-
> > > > > > > >> > 82+-+Add+Record+Headers+-+
> Typed
> > > > > > > >> > >> >> > > > ii.
> > Essentially the
> > > > value
> > > > > bytes, has a
> > > > > > leading
> > > > > > > >> > byte overhead.
> > > > > > > >> > >> >> > > > 1.
> > This tells you
> > > what
> > > > type
> > > > > the value
> > > > > > is,
> > > > > > > >> before
> > > > > > > >> > reading
> > > > > > > >> > >> the rest
> > > > > > > >> > >> >> > of the
> > > > > > > >> > >> >> > > > bytes,
> > allowing
> > > > > > serialisation/deserialization
> > > > > > > to
> > > > > > > >> > and from the
> > > > > > > >> > >> >> > primitives,
> > > > > > > >> > >> >> > > > string
> > and byte[].
> > > This
> > > > is
> > > > > akin to some
> > > > > > other
> > > > > > > >> > messaging
> > > > > > > >> > >> systems.
> > > > > > > >> > >> >> > > > 2) We
> > are making it
> > > > optional,
> > > > > so that
> > > > > > for
> > > > > > > those
> > > > > > > >> > not wanting
> > > > > > > >> > >> >> > headers have
> > > > > > > >> > >> >> > > 0
> > > > > > > >> > >> >> > > > bytes
> > overhead (think
> > > > of it
> > > > > as a feature
> > > > > > > flag),
> > > > > > > >> I
> > > > > > > >> > don’t
> > > > > > > >> > >> think this
> > > > > > > >> > >> >> > is
> > > > > > > >> > >> >> > > >
> > complex, especially if
> > > > > comparing to
> > > > > > changes
> > > > > > > >> > proposed in
> > > > > > > >> > >> other kips
> > > > > > > >> > >> >> > like
> > > > > > > >> > >> >> > > >
> kip-98.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > really
> > > > don’t
> > > > > like
> > > > > > this, we
> > > > > > > can
> > > > > > > >> > drop it, but
> > > > > > > >> > >> it
> > > > > > > >> > >> >> > would
> > > > > > > >> > >> >> > > mean
> > > > > > > >> > >> >> > > > buying
> > into 4 bytes
> > > > extra
> > > > > overhead for
> > > > > > users
> > > > > > > who
> > > > > > > >> > do not want
> > > > > > > >> > >> to use
> > > > > > > >> > >> >> > > headers.
> > > > > > > >> > >> >> > > > 3) In
> > the summary yes,
> > > > it is
> > > > > at a higher
> > > > > > > level,
> > > > > > > >> > but I think
> > > > > > > >> > >> this
> > > > > > > >> > >> >> > is well
> > > > > > > >> > >> >> > > >
> > documented in the
> > > > proposed
> > > > > changes
> > > > > > section.
> > > > > > > >> > >> >> > > > a.
> > Added getHeaders
> > > > method to
> > > > > > > Producer/Consumer
> > > > > > > >> > record (that
> > > > > > > >> > >> is it)
> > > > > > > >> > >> >> > > > b.
> > We’ve also detailed
> > > > the
> > > > > new Headers
> > > > > > class
> > > > > > > >> that
> > > > > > > >> > this method
> > > > > > > >> > >> >> > returns
> > > > > > > >> > >> >> > > that
> > > > > > > >> > >> >> > > >
> > encapsulates the
> > > headers
> > > > > protocol and
> > > > > > logic.
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Best,
> > > > > > > >> > >> >> > > > Mike
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > >
> > ==Original questions
> > > > from the
> > > > > vote
> > > > > > thread from
> > > > > > > >> > Jay.==
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Couple
> > of things I
> > > > think we
> > > > > still need
> > > > > > to work
> > > > > > > >> out:
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1.
> I
> > think we agree
> > > > about
> > > > > the key,
> > > > > > but I
> > > > > > > >> think
> > > > > > > >> > we haven't
> > > > > > > >> > >> >> > talked about
> > > > > > > >> > >> >> > > > the
> > value yet. I
> > > > think if
> > > > > our goal
> > > > > > is an
> > > > > > > open
> > > > > > > >> > ecosystem
> > > > > > > >> > >> of these
> > > > > > > >> > >> >> > > header
> > > > > > > >> > >> >> > > >
> > spread across many
> > > > plugins
> > > > > from many
> > > > > > > systems
> > > > > > > >> we
> > > > > > > >> > should
> > > > > > > >> > >> consider
> > > > > > > >> > >> >> > making
> > > > > > > >> > >> >> > > > this
> > > > > > > >> > >> >> > > > a
> > string as well so
> > > > it can
> > > > > be
> > > > > > printed, set
> > > > > > > >> via
> > > > > > > >> > a UI, set
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > config,
> > > > > > > >> > >> >> > > etc.
> > > > > > > >> > >> >> > > >
> > Basically
> > > encouraging
> > > > > pluggable
> > > > > > > serialization
> > > > > > > >> > formats
> > > > > > > >> > >> here will
> > > > > > > >> > >> >> > lead
> > > > > > > >> > >> >> > > to
> > > > > > > >> > >> >> > > > a
> > > > > > > >> > >> >> > > > bit
> > of a tower of
> > > > babel.
> > > > > > > >> > >> >> > > > 2.
> > This proposal
> > > > still
> > > > > includes a
> > > > > > pretty
> > > > > > > big
> > > > > > > >> > change to our
> > > > > > > >> > >> >> > >
> > serialization
> > > > > > > >> > >> >> > > > and
> > protocol
> > > > definition
> > > > > layer.
> > > > > > Essentially
> > > > > > > >> it is
> > > > > > > >> > >> introducing an
> > > > > > > >> > >> >> > > optional
> > > > > > > >> > >> >> > > >
> > type, where the
> > > > format is
> > > > > data
> > > > > > dependent. I
> > > > > > > >> > think this is
> > > > > > > >> > >> >> > actually a
> > > > > > > >> > >> >> > > big
> > > > > > > >> > >> >> > > >
> > change though it
> > > > doesn't
> > > > > seem like
> > > > > > it. It
> > > > > > > >> means
> > > > > > > >> > you can no
> > > > > > > >> > >> >> > longer
> > > > > > > >> > >> >> > > >
> specify
> > > > > > > >> > >> >> > > >
> this
> > type with our
> > > > type
> > > > > definition
> > > > > > DSL, and
> > > > > > > >> > likewise it
> > > > > > > >> > >> requires
> > > > > > > >> > >> >> > > custom
> > > > > > > >> > >> >> > > >
> > handling in client
> > > > libs.
> > > > > This isn't
> > > > > > a huge
> > > > > > > >> > thing, since
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > Record
> > > > > > > >> > >> >> > > >
> > definition is
> > > custom
> > > > > anyway, but I
> > > > > > think
> > > > > > > this
> > > > > > > >> > kind of
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> > > >
> > inconsistency is
> > > very
> > > > > non-desirable
> > > > > > and
> > > > > > > ties
> > > > > > > >> > you to
> > > > > > > >> > >> hand-coding
> > > > > > > >> > >> >> > > things.
> > > > > > > >> > >> >> > > > I
> > > > > > > >> > >> >> > > >
> > think the type
> > > should
> > > > > instead by [Key
> > > > > > > Value]
> > > > > > > >> in
> > > > > > > >> > our BNF,
> > > > > > > >> > >> where
> > > > > > > >> > >> >> > key and
> > > > > > > >> > >> >> > > >
> > value are both
> > > short
> > > > > strings as used
> > > > > > > >> elsewhere.
> > > > > > > >> > This
> > > > > > > >> > >> brings it
> > > > > > > >> > >> >> > in line
> > > > > > > >> > >> >> > > > with
> > > > > > > >> > >> >> > > > the
> > rest of the
> > > > protocol.
> > > > > > > >> > >> >> > > > 3.
> > Could we get
> > > more
> > > > > specific about
> > > > > > the
> > > > > > > exact
> > > > > > > >> > Java API
> > > > > > > >> > >> change to
> > > > > > > >> > >> >> > > >
> > ProducerRecord,
> > > > > ConsumerRecord,
> > > > > > Record,
> > > > > > > etc?
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > -Jay
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The
> information
> > contained in
> > > > this
> > > > > email is
> > > > > > strictly
> > > > > > > >> > confidential and
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > the use of the
> > addressee only,
> > > > unless
> > > > > otherwise
> > > > > > > >> indicated.
> > > > > > > >> > If you
> > > > > > > >> > >> are not
> > > > > > > >> > >> >> > the intended
> > recipient, please
> > > > do not
> > > > > read,
> > > > > > copy, use
> > > > > > > or
> > > > > > > >> > disclose to
> > > > > > > >> > >> others
> > > > > > > >> > >> >> > this message
> or
> > any
> > > attachment.
> > > > > Please also
> > > > > > notify the
> > > > > > > >> > sender by
> > > > > > > >> > >> replying
> > > > > > > >> > >> >> > to this email
> > or by telephone
> > > > > (+44(020 7896
> > > > > > 0011) and
> > > > > > > >> then
> > > > > > > >> > delete
> > > > > > > >> > >> the email
> > > > > > > >> > >> >> > and any copies
> > of it.
> > > Opinions,
> > > > > conclusion
> > > > > > (etc) that
> > > > > > > do
> > > > > > > >> > not relate
> > > > > > > >> > >> to the
> > > > > > > >> > >> >> > official
> > business of this
> > > > company
> > > > > shall be
> > > > > > understood
> > > > > > > as
> > > > > > > >> > neither
> > > > > > > >> > >> given nor
> > > > > > > >> > >> >> > endorsed by
> it.
> > IG is a
> > > trading
> > > > name
> > > > > of IG
> > > > > > Markets
> > > > > > > >> Limited
> > > > > > > >> > (a company
> > > > > > > >> > >> >> > registered in
> > England and
> > > Wales,
> > > > > company number
> > > > > > > >> 04008957)
> > > > > > > >> > and IG
> > > > > > > >> > >> Index
> > > > > > > >> > >> >> > Limited (a
> > company registered
> > > in
> > > > > England and
> > > > > > Wales,
> > > > > > > >> > company number
> > > > > > > >> > >> >> > 01190902).
> > Registered address
> > > at
> > > > > Cannon Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > >> > Dowgate
> > > > > > > >> > >> Hill,
> > > > > > > >> > >> >> > London EC4R
> > 2YA. Both IG
> > > Markets
> > > > > Limited
> > > > > > (register
> > > > > > > >> number
> > > > > > > >> > 195355)
> > > > > > > >> > >> and IG
> > > > > > > >> > >> >> > Index Limited
> > (register number
> > > > > 114059) are
> > > > > > authorised
> > > > > > > >> and
> > > > > > > >> > regulated
> > > > > > > >> > >> by the
> > > > > > > >> > >> >> > Financial
> > Conduct Authority.
> > > > > > > >> > >> >> >
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> --
> > > > > > > >> > >> Nacho - Ignacio
> > Solis -
> > > > isolis@igso.net
> > > > > > > >> > >>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > The information contained in
> > this email is
> > > > strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > indicated. If
> > > > > > you are
> > > > > > > >> not
> > > > > > > >> > the intended recipient, please
> > do not read,
> > > > copy, use
> > > > > or
> > > > > > disclose to
> > > > > > > >> others
> > > > > > > >> > this message or any
> attachment.
> > Please also
> > > > notify
> > > > > the sender by
> > > > > > > >> replying
> > > > > > > >> > to this email or by telephone
> > (+44(020 7896
> > > > 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > >> email
> > > > > > > >> > and any copies of it.
> Opinions,
> > conclusion
> > > > (etc) that
> > > > > do not
> > > > > > relate to
> > > > > > > >> the
> > > > > > > >> > official business of this
> > company shall be
> > > > understood
> > > > > as
> > > > > > neither given
> > > > > > > >> nor
> > > > > > > >> > endorsed by it. IG is a
> trading
> > name of IG
> > > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> > registered in England and
> > Wales, company
> > > number
> > > > > 04008957) and
> > > > > > IG Index
> > > > > > > >> > Limited (a company registered
> > in England and
> > > > Wales,
> > > > > company
> > > > > > number
> > > > > > > >> > 01190902). Registered address
> > at Cannon Bridge
> > > > House,
> > > > > 25
> > > > > > Dowgate Hill,
> > > > > > > >> > London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and
> > > > > > > IG
> > > > > > > >> > Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > >> the
> > > > > > > >> > Financial Conduct Authority.
> > > > > > > >> >
> > > > > > > >> The information contained in
> this
> > email is
> > > > strictly
> > > > > confidential
> > > > > > and for
> > > > > > > >> the use of the addressee only,
> > unless otherwise
> > > > > indicated. If you
> > > > > > are
> > > > > > > not
> > > > > > > >> the intended recipient, please
> do
> > not read,
> > > copy,
> > > > use
> > > > > or disclose
> > > > > > to
> > > > > > > others
> > > > > > > >> this message or any attachment.
> > Please also
> > > > notify the
> > > > > sender by
> > > > > > > replying
> > > > > > > >> to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the
> > > > > > > email
> > > > > > > >> and any copies of it. Opinions,
> > conclusion (etc)
> > > > that
> > > > > do not
> > > > > > relate to
> > > > > > > the
> > > > > > > >> official business of this
> company
> > shall be
> > > > understood
> > > > > as neither
> > > > > > given
> > > > > > > nor
> > > > > > > >> endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> registered in England and Wales,
> > company number
> > > > > 04008957) and IG
> > > > > > Index
> > > > > > > >> Limited (a company registered in
> > England and
> > > > Wales,
> > > > > company number
> > > > > > > >> 01190902). Registered address at
> > Cannon Bridge
> > > > House,
> > > > > 25 Dowgate
> > > > > > Hill,
> > > > > > > >> London EC4R 2YA. Both IG Markets
> > Limited
> > > (register
> > > > > number 195355)
> > > > > > and IG
> > > > > > > >> Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > the
> > > > > > > >> Financial Conduct Authority.
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > The information contained in this
> > email is strictly
> > > > > confidential and
> > > > > > for the use of the addressee only, unless
> > otherwise
> > > > indicated.
> > > > > If you are
> > > > > > not the intended recipient, please do not
> > read, copy, use
> > > > or
> > > > > disclose to
> > > > > > others this message or any attachment.
> > Please also notify
> > > > the
> > > > > sender by
> > > > > > replying to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that
> > > > > do not
> > > > > > relate to the official business of this
> > company shall be
> > > > > understood as
> > > > > > neither given nor endorsed by it. IG is a
> > trading name of
> > > > IG
> > > > > Markets
> > > > > > Limited (a company registered in England
> > and Wales,
> > > company
> > > > > number
> > > > > > 04008957) and IG Index Limited (a company
> > registered in
> > > > England
> > > > > and Wales,
> > > > > > company number 01190902). Registered
> > address at Cannon
> > > > Bridge
> > > > > House, 25
> > > > > > Dowgate Hill, London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and IG Index Limited (register
> > number 114059) are
> > > > > authorised and
> > > > > > regulated by the Financial Conduct
> > Authority.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is
> > strictly
> > > confidential
> > > > and
> > > > > for the use of the addressee only, unless otherwise
> > indicated. If
> > > > you are
> > > > > not the intended recipient, please do not read,
> > copy, use or
> > > > disclose to
> > > > > others this message or any attachment. Please also
> > notify the
> > > sender
> > > > by
> > > > > replying to this email or by telephone (+44(020
> 7896
> > 0011) and then
> > > > delete
> > > > > the email and any copies of it. Opinions,
> conclusion
> > (etc) that do
> > > > not
> > > > > relate to the official business of this company
> > shall be understood
> > > > as
> > > > > neither given nor endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a company registered in England and Wales,
> > company number
> > > > > 04008957) and IG Index Limited (a company
> registered
> > in England and
> > > > Wales,
> > > > > company number 01190902). Registered address at
> > Cannon Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > Limited (register
> > > > number
> > > > > 195355) and IG Index Limited (register number
> > 114059) are
> > > authorised
> > > > and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> > sender by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> > not relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > registered in England and Wales, company number 04008957)
> > and IG Index
> > > > Limited (a company registered in England and Wales,
> > company number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> > for the use of the addressee only, unless otherwise indicated. If you are
> > not the intended recipient, please do not read, copy, use or disclose to
> > others this message or any attachment. Please also notify the sender by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do not
> > relate to the official business of this company shall be understood as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG Index Limited (register number 114059) are authorised and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Btw Jason, can you share the branch with the VarInts helper/util class that KIP-98 will be introducing and also will have the current modified (key and value lengths) message, so we can look at, thanks.
On 20/02/2017, 01:11, "Michael Pearce" <Mi...@ig.com> wrote:
RE “ProducerRecord and ConsumerRecord are immutable currently”. I beg to differ on this, if I set a byte[] as my value of my record, I can later call value() and then alter the bytes in the array. As such it is very much mutable.
Headers also will anyhow by nature be mutable on produce, as you add to them.
On 20/02/2017, 00:22, "Jason Gustafson" <ja...@confluent.io> wrote:
Re; having a dedicated Headers object. I think it makes sense. Maybe it
could implement Iterable<Header>?
One thing I'm not too sure about is mutability. Both the ProducerRecord and
ConsumerRecord are immutable currently. It would be nice to keep it that
way. Perhaps Headers could itself be immutable and passed through the
constructors?
Just going through how this is, I don’t believe this would provide much
> saving. If issue trying to resolve is around object/garbage creation of
> Map.Entry objects.
It's definitely better because we only allocate the space we will use, but
I assume most of the actual memory cost is in the header data itself which
we can't do much about. Maybe we can start with the simple array
representation and optimize from there after we find it's worthwhile to do
so? If we use the Headers object, we can change this later without breaking
the API.
-Jason
On Sun, Feb 19, 2017 at 2:39 PM, radai <ra...@gmail.com> wrote:
> "(By the way, doesn't it feel a bit odd that we seem to be designing a
> feature which is optimized for people not using?)"
>
> very. this (i claim :-) ) is the result of very intense opposition to the
> usefulness of the feature early on, and not a real design goal
>
> On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Whilst if having an array of a Header[] would mean getting this through,
> > im happy changing this, as already done.
> >
> > Just going through how this is, I don’t believe this would provide much
> > saving. If issue trying to resolve is around object/garbage creation of
> > Map.Entry objects.
> >
> > All you’re doing here is replacing the equiv of a HashMap.Node
> (Map.Entry)
> > which simply holds reference to key and value objects with a custom
> variant.
> >
> >
> >
> >
> >
> > On 19/02/2017, 20:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> > On point 1 & 2 Ive updated KIP to show varints (and removed the bit
> > flag). (on the assumption KIP 98 is getting the agreement the protocol is
> > moving from int32 to varInts as standard)
> >
> > On point 3 ive updated to use an array of Header class, instead of a
> > MultiMap in the Headers class object
> >
> >
> >
> > On 19/02/2017, 20:06, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > On points 1 and 2 I agree.
> >
> > This also affects kip-98, I should expect this resolved before
> > that vote also passes. If it is accepted there (I’m assuming this is
> > getting discussed on that KIP? As you’re driving the move to VarInts), I
> am
> > happy to make this KIP will simply follow suit to whatever is agreed in
> > KIP-98.
> >
> > On 3) Agreed this is a simpler form, as long as no one is
> > expecting hashmap lookup performance ( O(1) ), I still would prefer an
> > encapsulated class, so if we find that holding it as an array in future
> is
> > causing some perf issues, the internals are not exposed to end users,
> > allowing the internal structure to move to a map.
> >
> > Can we compromise on?
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Headers headers;
> > }
> >
> > class Headers {
> > Header[] headers;
> >
> > add(String key, byte[] value)
> > Collection<byte[]> get(String key)
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> >
> >
> > On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> >
> > >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> >
> >
> > This is exactly what I have been trying to get at. The use
> > cases documented
> > here are middleware:
> > https://cwiki.apache.org/confluence/display/KAFKA/A+
> > Case+for+Kafka+Headers.
> > If headers are intended for the application space as well,
> the
> > document
> > should be updated accordingly so that it is an explicit
> design
> > goal and not
> > an unfortunate design byproduct. There may be some
> > disagreement on whether
> > it _should_ be a design goal, but it may not matter much
> since
> > the current
> > interceptor API is probably insufficient for middleware
> > applications (I
> > haven't thought of a way to do this that isn't cumbersome).
> >
> > That aside, the three unresolved points for me are the
> > following:
> >
> > 1. The use of varints. As a one-off for use only with record
> > headers, the
> > case was weak, but if we are using them throughout the
> message
> > format, then
> > we should do so here as well. The additional complexity is
> > minimal and
> > early performance testing fully justifies their use.
> >
> > 2. If varints are used, the case for using attributes to
> > indicate null
> > headers also becomes weak. We only add one additional byte in
> > each message
> > if there are no headers. Whatever the case, let us be
> > consistent with how
> > we treat null keys and values.
> >
> > 3. We have apparently agreed that the broker will validate
> > headers (please
> > add this to the KIP). That being the case, I would prefer to
> > use Kafka's
> > conventional format for arrays. The downside is that it takes
> > more work to
> > skip over the headers on the consumer, though it's unclear if
> > the cost
> > would matter in practice. Some concern was previously
> > expressed about the
> > allocation of maps. An alternative would be to use arrays,
> i.e.
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Header[] headers;
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> > This would work nicely with the conventional array format and
> > my guess is
> > it would obviate the need do any lazy initialization. If we
> > use the map as
> > is currently documented, then it is possible with either
> > representation to
> > slice the headers and initialize them lazily. Either way, it
> > might be a
> > good idea to use a separate object to represent the headers
> in
> > case we need
> > to modify it in the future in some way.
> >
> > (By the way, doesn't it feel a bit odd that we seem to be
> > designing a
> > feature which is optimized for people not using?)
> >
> >
> > If we can resolve these points, then at least you will get my
> > vote.
> >
> > Thanks,
> > Jason
> >
> > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > radai.rosenblatt@gmail.com> wrote:
> >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> > > IIUC samze currently has headers "in-V" and would just
> > switch over to kafka
> > > headers if they exist.
> > > im sure plenty of other users of kafka would have a use for
> > headers.
> > > im pretty sure use cases exist around shuffling data
> > into/out-of kafka
> > > (kafka connect or equivalent) where metadata from one end
> > could copied over
> > > to the other (S3, for example uses http headers for
> > user-accessible
> > > metadata). it will be kafka client code getting/setting
> > those headers. not
> > > an interceptor.
> > >
> > > On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > For APM single event tracing, need access to the header
> at
> > the point of
> > > > processing on the processing thread.
> > > >
> > > > As such interceptors will not work/be suitable for these,
> > due to the fact
> > > > they act on the ConsumerRecords as a batch, before the
> > handling thread
> > > can
> > > > split out and process per message which is the point
> these
> > tools will
> > > need
> > > > to continue to transaction tracing.
> > > >
> > > > Like wise tools and infra pieces will need access to the
> > message outside
> > > > the interceptor realm.
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 21:26, "Jason Gustafson" <
> > jason@confluent.io> wrote:
> > > >
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > >
> > > >
> > > > So the broker will parse the headers and validate
> > them. Good. The
> > > only
> > > > reason remaining that I can see to leave the headers
> > as a byte array
> > > > is to
> > > > make it easier for the client to skip past them. Are
> > we sure this is
> > > > not
> > > > premature optimization? Are there any performance
> > results which show
> > > > that
> > > > this is worthwhile?
> > > >
> > > > What’s the issue with exposing a method getHeaders on
> > the
> > > > producer/consumer
> > > > > record? It doesn’t break anything. We don’t need
> any
> > special
> > > version.
> > > >
> > > >
> > > > See my previous explanation. What I am trying to
> > resist is the
> > > headers
> > > > becoming a general application-level facility. The
> > primary use case
> > > as
> > > > far
> > > > as I can tell is middleware, which is the use case
> > that the
> > > > interceptors
> > > > are providing.
> > > >
> > > > Current batch consumer model and consumer
> interceptors
> > don’t work
> > > where
> > > > > headers need to be acted on at per message level at
> > time of
> > > > processing,
> > > > > very case is APM (the core one), where the header
> > value is used to
> > > > continue
> > > > > tracing.
> > > >
> > > >
> > > > I still don't understand the point about batching.
> The
> > consumer
> > > > records are
> > > > exposed as a batch in the consumer interceptor, but
> > you can still
> > > > iterate
> > > > them individually. It is no different for the
> consumer
> > API itself.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > Re:
> > > > >
> > > > > “ The point about creation of maps seems
> > orthogonal. We can
> > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the time
> > it is accessed.”
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > >
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:37, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > What’s the issue with exposing a method
> > getHeaders on the
> > > > > producer/consumer record? It doesn’t break
> anything.
> > We don’t need
> > > > any
> > > > > special version.
> > > > >
> > > > > Current batch consumer model and consumer
> > interceptors don’t
> > > work
> > > > > where headers need to be acted on at per message
> > level at time of
> > > > > processing, very case is APM (the core one), where
> > the header value
> > > > is used
> > > > > to continue tracing. JMS/HTTP etc all expose these,
> > without issues.
> > > > I would
> > > > > NOT want to lock this down to only be usable
> > accessible via
> > > > interceptors,
> > > > > as we’d fail on one of the main goals.
> > > > >
> > > > > Regards
> > > > > Mike
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:21, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > >
> > > > > The point about creation of maps seems
> > orthogonal. We can
> > > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the
> > time it is
> > > > accessed.
> > > > >
> > > > >
> > > > > > Yes exactly we have access to the records
> > thus why the
> > > > header
> > > > > should be
> > > > > > accessible via it and not hidden for only
> > interceptors to
> > > > access.
> > > > >
> > > > >
> > > > > As explained above, the point is to make
> the
> > intended usage
> > > > clear.
> > > > > Applications should continue to rely on the
> > key/value
> > > fields
> > > > to
> > > > > serialize
> > > > > their own headers, and it would be more
> > ideal if we can
> > > avoid
> > > > > leaking
> > > > > third-party headers into applications. This
> > is difficult to
> > > > do
> > > > > with the
> > > > > current interceptors because they share the
> > record objects
> > > > with
> > > > > the common
> > > > > API. What I had in mind is something like
> an
> > extension of
> > > the
> > > > > current
> > > > > interceptors which exposed a different
> > object (e.g.
> > > > > `RecordAndHeaders`).
> > > > > The challenge is for MM-like use cases. Let
> > me see if I can
> > > > come
> > > > > up with a
> > > > > concrete proposal for that problem.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 17, 2017 at 11:55 AM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > I am happy to move the definition of the
> > header into the
> > > > message
> > > > > body, but
> > > > > > would cause us not to lazy
> > initialise/parse the headers,
> > > as
> > > > > obviously, we
> > > > > > would have to traverse these reading the
> > message.
> > > > > >
> > > > > > This was actually one of Jay’s requests:
> > > > > >
> > > > > > “ 2. I think we should think about
> > creating the Map
> > > > lazily to
> > > > > avoid
> > > > > > parsing out all the headers into
> > little objects.
> > > > HashMaps
> > > > > themselves
> > > > > > are
> > > > > > kind of expensive and the consumer is
> > very perf
> > > > sensitive so
> > > > > and making
> > > > > > gazillions of hashmaps that may or
> may
> > not get used
> > > is
> > > > > probably a bad
> > > > > > idea.”
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > Yes exactly we have access to the
> > records thus why
> > > the
> > > > > header should
> > > > > > be accessible via it and not hidden for
> > only interceptors
> > > > to
> > > > > access.
> > > > > >
> > > > > > Sent using OWA for iPhone
> > > > > > ______________________________
> > __________
> > > > > > From: Magnus Edenhill <
> > magnus@edenhill.se>
> > > > > > Sent: Friday, February 17, 2017
> > 7:34:49 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> > Record Headers
> > > > > >
> > > > > > Big +1 on VarInts.
> > > > > > CPUs are fast, memory is slow.
> > > > > >
> > > > > > I agree with Jason that we'll want to
> > continue
> > > > verifying
> > > > > messages,
> > > > > > including their headers, so while I
> > appreciate the
> > > > idea of
> > > > > the opaque
> > > > > > header blob it won't be useful in
> > practice.
> > > > > >
> > > > > > /Magnus
> > > > > >
> > > > > > 2017-02-17 10:41 GMT-08:00 Jason
> > Gustafson <
> > > > > jason@confluent.io>:
> > > > > >
> > > > > > > Sorry, my mistake. The consumer
> > interceptor is per
> > > > batch,
> > > > > though I'm
> > > > > > not
> > > > > > > sure that's an actual limitation
> > since you still
> > > have
> > > > > access to the
> > > > > > > individual records.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 10:39 AM,
> > Jason Gustafson <
> > > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Re headers as byte array and
> > future use by
> > > broker.
> > > > This
> > > > > doesn't
> > > > > > take away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >
> > > > > > > >
> > > > > > > > Yeah, I didn't say it was
> > difficult, only
> > > awkward.
> > > > You
> > > > > wouldn't
> > > > > > write the
> > > > > > > > schema that way if you were
> > planning to use it on
> > > > the
> > > > > brokers from
> > > > > > the
> > > > > > > > beginning. Note also that one of
> > the benefits of
> > > > letting
> > > > > the broker
> > > > > > > > understand headers is that it can
> > validate that
> > > > they are
> > > > > properly
> > > > > > > > formatted. If cost is the only
> > concern, we should
> > > > > confirm its
> > > > > > impact
> > > > > > > > through performance testing.
> > > > > > > >
> > > > > > > > One of the key use cases requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >
> > > > > > > >
> > > > > > > > I'm not sure I understand the
> > point about
> > > batching.
> > > > > Interceptors
> > > > > > are
> > > > > > > > applied per-message, right?
> > > > > > > >
> > > > > > > > My intent on interceptors is to
> > keep the usage of
> > > > headers
> > > > > > well-defined so
> > > > > > > > that they don't start leaking
> > unnecessarily into
> > > > > applications. My
> > > > > > guess
> > > > > > > is
> > > > > > > > that it's probably inevitable,
> but
> > isolating it
> > > in
> > > > the
> > > > > > interceptors would
> > > > > > > > at least give people a second
> > thought before
> > > > deciding to
> > > > > use it.
> > > > > > The main
> > > > > > > > challenge in my mind is figuring
> > out how an MM
> > > use
> > > > case
> > > > > would
> > > > > > work. It
> > > > > > > > would be more cumbersome to
> > replicate headers
> > > > through an
> > > > > > interceptor,
> > > > > > > > though arguably MM should be
> > working at a lower
> > > > level
> > > > > anyway.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at 10:16 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Re headers available on the
> > record va
> > > > interceptors only
> > > > > > > >>
> > > > > > > >> One of the key use cases
> requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >>
> > > > > > > >> This anyhow is similar to
> > jms/http/amqp where
> > > > headers
> > > > > are
> > > > > > available to
> > > > > > > >> consuming applications.
> > > > > > > >>
> > > > > > > >> Re headers as byte array and
> > future use by
> > > > broker. This
> > > > > doesn't
> > > > > > take
> > > > > > > away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Sent using OWA for iPhone
> > > > > > > >> ______________________________
> > __________
> > > > > > > >> From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > >> Sent: Friday, February 17, 2017
> > 5:55:42 PM
> > > > > > > >> To: dev@kafka.apache.org
> > > > > > > >> Subject: Re: [DISCUSS] KIP-82 -
> > Add Record
> > > Headers
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Yes, that is what we are
> > proposing, so using
> > > > varints
> > > > > for headers
> > > > > > would
> > > > > > > be
> > > > > > > >> consistent with the rest of the
> > message. We have
> > > > used
> > > > > static
> > > > > > helper
> > > > > > > >> methods
> > > > > > > >> in our prototype implementation.
> > > > > > > >>
> > > > > > > >> The cost of parsing, we want to
> > parse/interpret
> > > > the
> > > > > headers
> > > > > > lazily (this
> > > > > > > >> is
> > > > > > > >> > a key point brought up earlier
> > in discussions)
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> I'm a bit skeptical of this. Has
> > anyone done the
> > > > > performance
> > > > > > testing? I
> > > > > > > >> can
> > > > > > > >> probably implement it and test
> it
> > if no one else
> > > > has. I
> > > > > was also
> > > > > > under
> > > > > > > the
> > > > > > > >> impression that there may be use
> > cases down the
> > > > road
> > > > > where the
> > > > > > broker
> > > > > > > >> would
> > > > > > > >> need to interpret headers. That
> > wouldn't be off
> > > > the
> > > > > table in the
> > > > > > future
> > > > > > > if
> > > > > > > >> it's represented as bytes, but
> it
> > would be quite
> > > > a bit
> > > > > more
> > > > > > awkward,
> > > > > > > >> right?
> > > > > > > >>
> > > > > > > >> By the way, one question I have
> > been wondering
> > > > about. My
> > > > > > understanding
> > > > > > > is
> > > > > > > >> that headers are primarily for
> > use cases where a
> > > > > third-party
> > > > > > components
> > > > > > > >> wants to enrich messages without
> > needing to
> > > > understand
> > > > > or modify
> > > > > > the
> > > > > > > >> schema
> > > > > > > >> of the message key and value.
> For
> > the
> > > applications
> > > > > which directly
> > > > > > > produce
> > > > > > > >> and consume the messages and
> > control the
> > > key/value
> > > > > schema
> > > > > > directly, it
> > > > > > > >> seems we would rather have them
> > implement
> > > headers
> > > > > directly in
> > > > > > their own
> > > > > > > >> schema. Supposing for the sake
> of
> > argument that
> > > > it was
> > > > > possible,
> > > > > > my
> > > > > > > >> question is whether it be
> > sufficient to expose
> > > the
> > > > > headers in the
> > > > > > > >> interceptor API and not in the
> > common API?
> > > > > > > >>
> > > > > > > >> -Jason
> > > > > > > >>
> > > > > > > >> On Fri, Feb 17, 2017 at 3:26 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > On the point of varInts
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 11:22, "Michael
> > Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > On the point re: headers
> in
> > the message
> > > > protocol
> > > > > being a
> > > > > > byte
> > > > > > > array
> > > > > > > >> > and not a count of elements
> > followed by the
> > > > elements.
> > > > > Again
> > > > > > this was
> > > > > > > >> > discussed/argued previously.
> > > > > > > >> >
> > > > > > > >> > It was agreed on for a few
> > reasons some of
> > > > which
> > > > > you have
> > > > > > > obviously
> > > > > > > >> > picked up on:
> > > > > > > >> >
> > > > > > > >> > Broker is able to pass it
> > through opaquely
> > > > > > > >> > The cost of parsing, we
> > want to
> > > > parse/interpret
> > > > > the headers
> > > > > > lazily
> > > > > > > >> > (this is a key point brought
> up
> > earlier in
> > > > > discussions)
> > > > > > > >> > Headers can be copied from
> > consumer record
> > > > to
> > > > > producer
> > > > > > record (aka
> > > > > > > >> > mirror makers etc) without
> > parsing if no
> > > > changes are
> > > > > being made
> > > > > > or
> > > > > > > being
> > > > > > > >> > looked at.
> > > > > > > >> > Keeps the broker agnostic
> > to the format
> > > > > > > >> > You need an int32 either
> > for the byte size
> > > > of the
> > > > > headers,
> > > > > > or for
> > > > > > > >> the
> > > > > > > >> > count of elements, so
> overheads
> > are the same,
> > > > but
> > > > > with going
> > > > > > with an
> > > > > > > >> opaque
> > > > > > > >> > byte array has the above
> > advantages.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 02:50,
> > "Jason Gustafson" <
> > > > > jason@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > Sorry, should have
> > noted that the
> > > > performance
> > > > > testing
> > > > > > was done
> > > > > > > >> > using the
> > > > > > > >> > producer performance
> > tool shipped with
> > > > Kafka.
> > > > > > > >> >
> > > > > > > >> > -Jason
> > > > > > > >> >
> > > > > > > >> > On Thu, Feb 16, 2017
> at
> > 6:44 PM, Jason
> > > > > Gustafson <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hey Nacho,
> > > > > > > >> > >
> > > > > > > >> > > I've compared
> > performance of our
> > > > KIP-98
> > > > > > implementation with
> > > > > > > >> and
> > > > > > > >> > without
> > > > > > > >> > > varints. For
> messages
> > around 128
> > > > bytes, we
> > > > > see an
> > > > > > increase
> > > > > > > in
> > > > > > > >> > throughput of
> > > > > > > >> > > about 30% using the
> > default
> > > > configuration
> > > > > settings.
> > > > > > At 256
> > > > > > > >> > bytes, the
> > > > > > > >> > > increase is around
> > 16%. Obviously
> > > the
> > > > > performance
> > > > > > converge
> > > > > > > as
> > > > > > > >> > messages get
> > > > > > > >> > > larger, but it seems
> > well worth the
> > > > cost.
> > > > > Note that
> > > > > > we are
> > > > > > > >> also
> > > > > > > >> > seeing a
> > > > > > > >> > > substantial
> > performance increase
> > > > against
> > > > > trunk
> > > > > > primarily
> > > > > > > >> because
> > > > > > > >> > of the
> > > > > > > >> > > much more efficient
> > packing that
> > > > varints
> > > > > provide us.
> > > > > > > Anything
> > > > > > > >> > adding to
> > > > > > > >> > > message overhead,
> > such as record
> > > > headers,
> > > > > would only
> > > > > > > increase
> > > > > > > >> > the relative
> > > > > > > >> > > difference. (Of
> > course take these
> > > > numbers
> > > > > with a
> > > > > > grain of
> > > > > > > salt
> > > > > > > >> > since I have
> > > > > > > >> > > only used the
> default
> > settings with
> > > > both
> > > > > the producer
> > > > > > and
> > > > > > > >> broker
> > > > > > > >> > on my
> > > > > > > >> > > local machine. We
> > intend to provide
> > > > more
> > > > > extensive
> > > > > > > performance
> > > > > > > >> > details as
> > > > > > > >> > > part of the work for
> > KIP-98.)
> > > > > > > >> > >
> > > > > > > >> > > The implementation
> we
> > are using is
> > > > from
> > > > > protobuf (
> > > > > > > >> > >
> > https://developers.google.com/
> > > > > > > protocol-buffers/docs/encoding
> > > > > > > >> ),
> > > > > > > >> > which is
> > > > > > > >> > > also used in HBase.
> > It is trivial to
> > > > > implement and as
> > > > > > far
> > > > > > > as I
> > > > > > > >> > know doesn't
> > > > > > > >> > > suffer from the
> > aliasing problem you
> > > > are
> > > > > describing. I
> > > > > > > checked
> > > > > > > >> > with Magnus
> > > > > > > >> > > (the author of
> > librdkafka) and he
> > > > agreed
> > > > > that the
> > > > > > savings
> > > > > > > >> seemed
> > > > > > > >> > worth the
> > > > > > > >> > > cost of
> > implementation.
> > > > > > > >> > >
> > > > > > > >> > > -Jason
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Feb 16, 2017
> > at 4:32 PM,
> > > > Ignacio
> > > > > Solis <
> > > > > > > >> isolis@igso.net>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> -VarInts
> > > > > > > >> > >>
> > > > > > > >> > >> I'm one of the
> > people (if not the
> > > > most)
> > > > > opposed to
> > > > > > VarInts.
> > > > > > > >> > VarInts
> > > > > > > >> > >> have a place, but
> > this is not it.
> > > > (We
> > > > > had a large
> > > > > > > >> discussion
> > > > > > > >> > about
> > > > > > > >> > >> them at the
> > beginning of KIP-82
> > > time)
> > > > > > > >> > >>
> > > > > > > >> > >> If anybody has real
> > life
> > > performance
> > > > > numbers of
> > > > > > VarInts
> > > > > > > >> > improving
> > > > > > > >> > >> things or
> > significantly reducing
> > > > resources
> > > > > I would
> > > > > > like to
> > > > > > > >> know
> > > > > > > >> > what
> > > > > > > >> > >> that case may be.
> > Yes, you can save
> > > > some
> > > > > bytes here
> > > > > > and
> > > > > > > >> there,
> > > > > > > >> > but
> > > > > > > >> > >> this is probably
> > insignificant to
> > > the
> > > > > overall system
> > > > > > > behavior
> > > > > > > >> > and
> > > > > > > >> > >> storage
> > requirements. -- I say
> > > this
> > > > with
> > > > > respect to
> > > > > > using
> > > > > > > >> > VarInts in
> > > > > > > >> > >> the protocol
> itself,
> > not as part of
> > > > the
> > > > > data.
> > > > > > > >> > >>
> > > > > > > >> > >> VarInts require you
> > to parse the
> > > Int
> > > > > before using it
> > > > > > and
> > > > > > > >> > depending on
> > > > > > > >> > >> the encoding they
> > can suffer from
> > > > aliasing
> > > > > (multiple
> > > > > > > >> > representations
> > > > > > > >> > >> for the same
> value).
> > > > > > > >> > >>
> > > > > > > >> > >> Why add complexity?
> > > > > > > >> > >>
> > > > > > > >> > >> Nacho
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> On Thu, Feb 16,
> 2017
> > at 10:29 AM,
> > > > Colin
> > > > > McCabe <
> > > > > > > >> > cmccabe@apache.org>
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> > +1 for varints
> > here-- it would
> > > save
> > > > > quite a bit of
> > > > > > space.
> > > > > > > >> > They are
> > > > > > > >> > >> > pretty quick to
> > implement as
> > > well.
> > > > > > > >> > >> >
> > > > > > > >> > >> > I think it makes
> > sense for values
> > > > to be
> > > > > byte
> > > > > > arrays.
> > > > > > > Users
> > > > > > > >> > might want
> > > > > > > >> > >> > to attach
> > arbitrary payloads;
> > > they
> > > > > shouldn't be
> > > > > > forced to
> > > > > > > >> > serialize
> > > > > > > >> > >> > everything to
> Java
> > strings.
> > > > > > > >> > >> >
> > > > > > > >> > >> > best,
> > > > > > > >> > >> > Colin
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Thu, Feb 16,
> > 2017, at 09:52,
> > > > Jason
> > > > > Gustafson
> > > > > > wrote:
> > > > > > > >> > >> >> Hey Michael,
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Hmm, I guess the
> > point of
> > > > representing
> > > > > it as
> > > > > > bytes is to
> > > > > > > >> > allow the
> > > > > > > >> > >> broker
> > > > > > > >> > >> >> to pass it
> > through opaquely? Is
> > > > the
> > > > > cost of
> > > > > > parsing
> > > > > > > them a
> > > > > > > >> > concern, or
> > > > > > > >> > >> >> are
> > > > > > > >> > >> >> we simply trying
> > to ensure that
> > > > the
> > > > > broker stays
> > > > > > > agnostic
> > > > > > > >> to
> > > > > > > >> > the
> > > > > > > >> > >> format?
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On varints, I
> > think adding
> > > > support for
> > > > > them makes
> > > > > > less
> > > > > > > >> sense
> > > > > > > >> > for an
> > > > > > > >> > >> >> isolated use
> > case, but as part
> > > of
> > > > a
> > > > > more holistic
> > > > > > change
> > > > > > > >> > (such as what
> > > > > > > >> > >> we
> > > > > > > >> > >> >> have proposed in
> > KIP-98), I
> > > think
> > > > they
> > > > > are
> > > > > > justifiable.
> > > > > > > If
> > > > > > > >> > we add them,
> > > > > > > >> > >> >> then the need to
> > use attributes
> > > > becomes
> > > > > quite a
> > > > > > bit
> > > > > > > >> weaker,
> > > > > > > >> > right? The
> > > > > > > >> > >> >> other thing I
> > find slightly odd
> > > > is the
> > > > > fact that
> > > > > > null
> > > > > > > >> > headers has no
> > > > > > > >> > >> >> actual
> > > > > > > >> > >> >> semantic meaning
> > for the message
> > > > > (unlike null
> > > > > > keys and
> > > > > > > >> > values). It is
> > > > > > > >> > >> >> just
> > > > > > > >> > >> >> a space
> > optimization. It seems a
> > > > bit
> > > > > better to
> > > > > > always
> > > > > > > use
> > > > > > > >> > size 0 to
> > > > > > > >> > >> >> indicate having
> > no headers.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Overall, the
> main
> > point is
> > > > ensuring
> > > > > that the
> > > > > > message
> > > > > > > >> schema
> > > > > > > >> > remains
> > > > > > > >> > >> >> consistent,
> > either within the
> > > > larger
> > > > > protocol, or
> > > > > > at a
> > > > > > > >> > minimum within
> > > > > > > >> > >> the
> > > > > > > >> > >> >> message itself.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> -Jason
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Thu, Feb 16,
> > 2017 at 6:39 AM,
> > > > > Michael Pearce <
> > > > > > > >> > Michael.Pearce@ig.com
> > > > > > > >> > >> >
> > > > > > > >> > >> >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> > Hi Jason,
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 1) in
> > the message
> > > > protocol
> > > > > the headers
> > > > > > are
> > > > > > > >> simply
> > > > > > > >> > a byte
> > > > > > > >> > >> array,
> > > > > > > >> > >> >> > as like the
> key
> > or value, this
> > > > is to
> > > > > clearly
> > > > > > demarcate
> > > > > > > >> the
> > > > > > > >> > header in
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > core message.
> > Then the header
> > > > byte
> > > > > array in the
> > > > > > core
> > > > > > > >> > message is an
> > > > > > > >> > >> array of
> > > > > > > >> > >> >> > key, value
> > pairs. This is what
> > > > it is
> > > > > denoting.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Then this
> would
> > be I guess in
> > > > the
> > > > > given
> > > > > > notation:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [KeyLength, Key,
> > > > > ValueLength, Value]
> > > > > > > >> > >> >> > KeyLength
> > => int32
> > > > > <-----------------NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised key
> > value
> > > > > > > >> > >> >> > Key =>
> bytes
> > > > > <---------------------- NEW
> > > > > > > serialised
> > > > > > > >> > string (UTF8)
> > > > > > > >> > >> >> > bytes of the
> > header key
> > > > > > > >> > >> >> >
> ValueLength
> > => int32
> > > > > <-------------- NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised
> > header value
> > > > > > > >> > >> >> > Value =>
> > bytes
> > > > > <-------------------- NEW
> > > > > > > serialised
> > > > > > > >> > form of the
> > > > > > > >> > >> header
> > > > > > > >> > >> >> > value
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The key length
> > and value
> > > length
> > > > is
> > > > > matching the
> > > > > > way
> > > > > > > the
> > > > > > > >> > protocol is
> > > > > > > >> > >> >> > defined in the
> > core message
> > > > currently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 2)
> > > > > > > >> > >> >> > Var sized
> ints,
> > this was
> > > > discussed
> > > > > much earlier
> > > > > > on, in
> > > > > > > >> > fact I had
> > > > > > > >> > >> >> > suggested it
> > myself (with
> > > Hadoop
> > > > > references),
> > > > > > the
> > > > > > > >> > complexity of this
> > > > > > > >> > >> >> > compared to
> > having a simpler
> > > > protocol
> > > > > was
> > > > > > argued and
> > > > > > > >> > agreed it
> > > > > > > >> > >> wasn’t worth
> > > > > > > >> > >> >> > the complexity
> > as all other
> > > > clients
> > > > > in other
> > > > > > languages
> > > > > > > >> > would need to
> > > > > > > >> > >> ensure
> > > > > > > >> > >> >> > theyre using
> > the right var
> > > size
> > > > > algorithm, as
> > > > > > there
> > > > > > > is a
> > > > > > > >> > few.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 3)
> > > > > > > >> > >> >> > We did the
> > attributes,
> > > optional
> > > > > approach as
> > > > > > originally
> > > > > > > >> > there was
> > > > > > > >> > >> marked
> > > > > > > >> > >> >> > concern that
> > headers would
> > > > cause a
> > > > > message size
> > > > > > > overhead
> > > > > > > >> > for others,
> > > > > > > >> > >> who
> > > > > > > >> > >> >> > don’t want
> > them. As such this
> > > > is the
> > > > > clean
> > > > > > solution to
> > > > > > > >> > achieve that.
> > > > > > > >> > >> If
> > > > > > > >> > >> >> > that no longer
> > holds, and we
> > > > don’t
> > > > > care that we
> > > > > > add
> > > > > > > >> 4bytes
> > > > > > > >> > overhead,
> > > > > > > >> > >> then
> > > > > > > >> > >> >> > im happy to
> > remove.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I’m personally
> > in favour of
> > > > keeping
> > > > > the message
> > > > > > as
> > > > > > > small
> > > > > > > >> > as possible
> > > > > > > >> > >> so
> > > > > > > >> > >> >> > people don’t
> > get shocks in
> > > perf
> > > > and
> > > > > throughputs
> > > > > > dues
> > > > > > > to
> > > > > > > >> > message size,
> > > > > > > >> > >> >> > unless they
> > actively use the
> > > > feature,
> > > > > as such I
> > > > > > do
> > > > > > > >> prefer
> > > > > > > >> > the
> > > > > > > >> > >> attribute bit
> > > > > > > >> > >> >> > wise feature
> > flag approach
> > > > myself.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On 16/02/2017,
> > 05:40, "Jason
> > > > > Gustafson" <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > We have
> > proposed a few
> > > > > significant changes
> > > > > > to the
> > > > > > > >> > message format
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > KIP-98
> > > > > > > >> > >> >> > which now
> > seems likely to
> > > > pass
> > > > > (perhaps
> > > > > > with some
> > > > > > > >> > iterations on
> > > > > > > >> > >> >> >
> > implementation details).
> > > It
> > > > would
> > > > > be good
> > > > > > to try
> > > > > > > and
> > > > > > > >> > coordinate
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > changes
> > > > > > > >> > >> >> > in both of
> > the proposals
> > > to
> > > > make
> > > > > sure they
> > > > > > are
> > > > > > > >> > consistent and
> > > > > > > >> > >> >> > compatible.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I think
> > using the
> > > > attributes to
> > > > > indicate
> > > > > > null
> > > > > > > >> headers
> > > > > > > >> > is a
> > > > > > > >> > >> reasonable
> > > > > > > >> > >> >> > approach.
> > We have proposed
> > > > to do
> > > > > the same
> > > > > > thing
> > > > > > > for
> > > > > > > >> > the message
> > > > > > > >> > >> key and
> > > > > > > >> > >> >> > value.
> That
> > said, I
> > > > sympathize
> > > > > with Jay's
> > > > > > > argument.
> > > > > > > >> > Having
> > > > > > > >> > >> multiple
> > > > > > > >> > >> >> > ways to
> > > > > > > >> > >> >> > specify a
> > null value
> > > > increases
> > > > > the overall
> > > > > > > >> complexity
> > > > > > > >> > of the
> > > > > > > >> > >> protocol.
> > > > > > > >> > >> >> > You
> > > > > > > >> > >> >> > can see
> > this just from the
> > > > fact
> > > > > that you
> > > > > > need the
> > > > > > > >> > extra verbiage
> > > > > > > >> > >> in the
> > > > > > > >> > >> >> > protocol
> > specification in
> > > > this
> > > > > KIP and in
> > > > > > KIP-98
> > > > > > > to
> > > > > > > >> > describe the
> > > > > > > >> > >> >> > dependence
> > > > > > > >> > >> >> > between
> the
> > fields and the
> > > > > attributes. It
> > > > > > seems
> > > > > > > >> like a
> > > > > > > >> > slippery
> > > > > > > >> > >> slope
> > > > > > > >> > >> >> > if
> > > > > > > >> > >> >> > you start
> > allowing
> > > different
> > > > > request types
> > > > > > to
> > > > > > > >> > implement the
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> >
> > specification differently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > You can
> > also argue that
> > > the
> > > > > messages
> > > > > > already are
> > > > > > > and
> > > > > > > >> > are likely
> > > > > > > >> > >> to
> > > > > > > >> > >> >> > remain a
> > > > > > > >> > >> >> > special
> > case. For example,
> > > > there
> > > > > is
> > > > > > currently no
> > > > > > > >> > generality in
> > > > > > > >> > >> how
> > > > > > > >> > >> >> > compressed
> > message sets
> > > are
> > > > > represented
> > > > > > that would
> > > > > > > >> be
> > > > > > > >> > applicable
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > other
> > > > > > > >> > >> >> > request
> > types. Some might
> > > > see this
> > > > > > divergence as
> > > > > > > an
> > > > > > > >> > unfortunate
> > > > > > > >> > >> >> > protocol
> > > > > > > >> > >> >> > deficiency
> > which should be
> > > > fixed;
> > > > > others
> > > > > > might see
> > > > > > > >> it
> > > > > > > >> > as sort of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> >
> > inevitability of needing
> > > to
> > > > > optimize where
> > > > > > it
> > > > > > > counts
> > > > > > > >> > most. I'm
> > > > > > > >> > >> probably
> > > > > > > >> > >> >> > somewhere
> > in between, but
> > > I
> > > > think
> > > > > we
> > > > > > probably all
> > > > > > > >> > share the
> > > > > > > >> > >> intuition
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> > protocol should be
> > > kept
> > > > as
> > > > > consistent as
> > > > > > > >> possible.
> > > > > > > >> > With that
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > mind,
> > > > > > > >> > >> >> > here are a
> > few comments:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 1. One
> > thing I found a
> > > > little odd
> > > > > when
> > > > > > reading the
> > > > > > > >> > current
> > > > > > > >> > >> proposal is
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> headers
> > are both
> > > > represented
> > > > > as an
> > > > > > array of
> > > > > > > >> bytes
> > > > > > > >> > and as an
> > > > > > > >> > >> array
> > > > > > > >> > >> >> > of
> > > > > > > >> > >> >> > key/value
> > pairs. I'd
> > > > probably
> > > > > suggest
> > > > > > something
> > > > > > > like
> > > > > > > >> > this:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [HeaderKey
> > > > HeaderValue]
> > > > > > > >> > >> >> > HeaderKey
> > => String
> > > > > > > >> > >> >> >
> > HeaderValue => Bytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > An array
> in
> > the Kafka
> > > > protocol is
> > > > > > represented as a
> > > > > > > >> > 4-byte integer
> > > > > > > >> > >> >> > indicating
> > the number of
> > > > elements
> > > > > in the
> > > > > > array
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> >
> > serialization of the
> > > > elements.
> > > > > Unless I'm
> > > > > > > >> > misunderstanding, what
> > > > > > > >> > >> you
> > > > > > > >> > >> >> > have
> > > > > > > >> > >> >> > instead is
> > the total size
> > > > of the
> > > > > headers in
> > > > > > bytes
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> > elements.
> > > > > > > >> > >> >> > I'm not
> > sure I see any
> > > > reason for
> > > > > this
> > > > > > > >> inconsistency.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 2. In
> > KIP-98, we've
> > > > introduced
> > > > > > variable-length
> > > > > > > >> integer
> > > > > > > >> > fields.
> > > > > > > >> > >> >> > Effectively,
> > > > > > > >> > >> >> > we've
> > enriched (or
> > > > "complicated"
> > > > > as Jay
> > > > > > might say
> > > > > > > ;)
> > > > > > > >> > the protocol
> > > > > > > >> > >> >> >
> > specification to include
> > > the
> > > > > following
> > > > > > types:
> > > > > > > >> VarInt,
> > > > > > > >> > VarLong,
> > > > > > > >> > >> >> >
> > UnsignedVarInt and
> > > > > UnsignedVarLong.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Along with
> > these
> > > > primitives, we
> > > > > could
> > > > > > introduce
> > > > > > > the
> > > > > > > >> > following
> > > > > > > >> > >> types:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeArray =>
> > > > NumberOfItems
> > > > > Item1 Item2
> > > > > > .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItems =>
> > > > UnsignedVarInt
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeNullableArray =>
> > > > > NumberOfItemsOrNull
> > > > > > Item1
> > > > > > > >> > Item2 .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItemsOrNull =>
> > > > VarInt
> > > > > (-1 means
> > > > > > null)
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > And
> > similarly for the
> > > > `String`
> > > > > and `Bytes`
> > > > > > types.
> > > > > > > >> > These types
> > > > > > > >> > >> can save
> > > > > > > >> > >> >> > a
> > > > > > > >> > >> >> >
> > considerable amount of
> > > > space in
> > > > > this
> > > > > > proposal
> > > > > > > >> because
> > > > > > > >> > they can
> > > > > > > >> > >> be used
> > > > > > > >> > >> >> > for
> > > > > > > >> > >> >> > both the
> > number of headers
> > > > > included in the
> > > > > > message
> > > > > > > >> and
> > > > > > > >> > the
> > > > > > > >> > >> lengths of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > header
> keys
> > and values. We
> > > > could
> > > > > do this
> > > > > > instead:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > > > VarSizeArray[HeaderKey
> > > > > > HeaderValue]
> > > > > > > >> > >> >> >
> HeaderKey
> > =>
> > > VarSizeString
> > > > > > > >> > >> >> >
> > HeaderValue =>
> > > > VarSizeBytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Combining
> > the savings from
> > > > the
> > > > > use of
> > > > > > variable
> > > > > > > >> length
> > > > > > > >> > fields, the
> > > > > > > >> > >> >> > benefit
> > > > > > > >> > >> >> > of using
> > the attributes to
> > > > > represent null
> > > > > > seems
> > > > > > > >> pretty
> > > > > > > >> > small.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 3.
> > Whichever way we go
> > > > (whether
> > > > > we use the
> > > > > > > >> attributes
> > > > > > > >> > or not), we
> > > > > > > >> > >> >> > should at
> > > > > > > >> > >> >> > least be
> > consistent
> > > between
> > > > this
> > > > > KIP and
> > > > > > KIP-98.
> > > > > > > It
> > > > > > > >> > would be very
> > > > > > > >> > >> >> > strange
> > > > > > > >> > >> >> > to have
> two
> > ways to
> > > > represent
> > > > > null values
> > > > > > in the
> > > > > > > >> same
> > > > > > > >> > schema.
> > > > > > > >> > >> Either
> > > > > > > >> > >> >> > way is
> > > > > > > >> > >> >> > OK with
> me.
> > I think some
> > > > > message-level
> > > > > > > optimizations
> > > > > > > >> > are
> > > > > > > >> > >> justifiable,
> > > > > > > >> > >> >> > but
> > > > > > > >> > >> >> > the
> savings
> > here seem
> > > > minimal (a
> > > > > few bytes
> > > > > > per
> > > > > > > >> > message), so
> > > > > > > >> > >> maybe it's
> > > > > > > >> > >> >> > not
> > > > > > > >> > >> >> > worth the
> > cost of letting
> > > > the
> > > > > message
> > > > > > diverge even
> > > > > > > >> > further from
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > rest of
> > > > > > > >> > >> >> > the
> > protocol.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > -Jason
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On Wed,
> Feb
> > 15, 2017 at
> > > > 8:52 AM,
> > > > > radai <
> > > > > > > >> > >>
> > radai.rosenblatt@gmail.com>
> > > > > > > >> > >> >> > wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > > I've
> > trimmed the inline
> > > > > contents as this
> > > > > > mail is
> > > > > > > >> > getting too
> > > > > > > >> > >> big for
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > apache
> > mailing list
> > > > software to
> > > > > deliver
> > > > > > :-(
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 1. the
> > important thing
> > > for
> > > > > > interoperability is
> > > > > > > for
> > > > > > > >> > different
> > > > > > > >> > >> >> > "interested
> > > > > > > >> > >> >> > > parties"
> > (plugins, infra
> > > > > layers/wrappers,
> > > > > > > >> user-code)
> > > > > > > >> > to be
> > > > > > > >> > >> able to
> > > > > > > >> > >> >> > stick
> > > > > > > >> > >> >> > > pieces
> of
> > metadata onto
> > > > msgs
> > > > > without
> > > > > > getting in
> > > > > > > >> each
> > > > > > > >> > other's
> > > > > > > >> > >> way. a
> > > > > > > >> > >> >> > common
> > > > > > > >> > >> >> > > key
> > scheme (Strings, as
> > > > of the
> > > > > time of
> > > > > > this
> > > > > > > >> > writing?) is all
> > > > > > > >> > >> thats
> > > > > > > >> > >> >> > required
> > > > > > > >> > >> >> > > for
> that.
> > it is assumed
> > > > that
> > > > > the other end
> > > > > > > >> > interested in any
> > > > > > > >> > >> such
> > > > > > > >> > >> >> > piece of
> > > > > > > >> > >> >> > > metadata
> > knows the
> > > > encoding,
> > > > > and byte[]
> > > > > > provides
> > > > > > > >> for
> > > > > > > >> > the most
> > > > > > > >> > >> >> > flexibility.
> > > > > > > >> > >> >> > > i
> believe
> > this is the
> > > same
> > > > > logic behind
> > > > > > core
> > > > > > > kafka
> > > > > > > >> > being
> > > > > > > >> > >> >> > byte[]/byte[]
> -
> > > > > > > >> > >> >> > > Strings
> > are more
> > > "usable"
> > > > but
> > > > > bytes are
> > > > > > flexible
> > > > > > > >> and
> > > > > > > >> > so were
> > > > > > > >> > >> chosen.
> > > > > > > >> > >> >> > > Also -
> > core kafka doesnt
> > > > even
> > > > > do that
> > > > > > good of a
> > > > > > > >> job
> > > > > > > >> > on
> > > > > > > >> > >> usability of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > payload
> > (example - i
> > > have
> > > > to
> > > > > specify the
> > > > > > nop
> > > > > > > >> byte[]
> > > > > > > >> > "decoders"
> > > > > > > >> > >> >> > explicitly
> > > > > > > >> > >> >> > > in
> conf),
> > and again
> > > > sacrificies
> > > > > usability
> > > > > > for
> > > > > > > the
> > > > > > > >> > sake of
> > > > > > > >> > >> >> > performance
> (no
> > > > > > > >> > >> >> > >
> > convenient single-record
> > > > > processing as
> > > > > > poll is a
> > > > > > > >> > batch, lots of
> > > > > > > >> > >> >> > obscure
> > > > > > > >> > >> >> > > little
> > config details
> > > > exposing
> > > > > internals
> > > > > > of the
> > > > > > > >> > batching
> > > > > > > >> > >> mechanism,
> > > > > > > >> > >> >> > etc)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > this is
> > also why i
> > > really
> > > > > dislike the
> > > > > > idea of a
> > > > > > > >> > "type system"
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > header
> > > > > > > >> > >> >> > > values,
> > it further
> > > > degrades the
> > > > > > usability, adds
> > > > > > > >> > complexity and
> > > > > > > >> > >> will
> > > > > > > >> > >> >> > >
> > eventually get in
> > > > people's way,
> > > > > also, it
> > > > > > would
> > > > > > > be
> > > > > > > >> > the 2nd/3rd
> > > > > > > >> > >> >> > home-group
> > > > > > > >> > >> >> > >
> > serialization mechanism
> > > > in core
> > > > > kafka
> > > > > > (counting
> > > > > > > 2
> > > > > > > >> > iterations
> > > > > > > >> > >> of the
> > > > > > > >> > >> >> > "type
> > > > > > > >> > >> >> > >
> > definition DSL")
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 2. this
> > is an
> > > > implementation
> > > > > detail, and
> > > > > > not
> > > > > > > even
> > > > > > > >> a
> > > > > > > >> > very "user
> > > > > > > >> > >> >> > facing" one?
> > > > > > > >> > >> >> > > to the
> > best of my
> > > > understanding
> > > > > the vote
> > > > > > process
> > > > > > > >> is
> > > > > > > >> > on proposed
> > > > > > > >> > >> >> > >
> > API/behaviour. also -
> > > > since
> > > > > we're willing
> > > > > > to go
> > > > > > > >> with
> > > > > > > >> > strings
> > > > > > > >> > >> just
> > > > > > > >> > >> >> > serialize
> > > > > > > >> > >> >> > > a
> 0-sized
> > header blob
> > > and
> > > > IIUC
> > > > > you dont
> > > > > > need any
> > > > > > > >> > optionals
> > > > > > > >> > >> anymore.
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 3. yes,
> > we can :-)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > On Tue,
> > Feb 14, 2017 at
> > > > 11:56
> > > > > PM, Michael
> > > > > > > Pearce <
> > > > > > > >> > >> >> >
> > Michael.Pearce@ig.com>
> > > > > > > >> > >> >> > > wrote:
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > > Hi
> Jay,
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1)
> > There was some
> > > > initial
> > > > > debate on the
> > > > > > value
> > > > > > > >> > part, as youll
> > > > > > > >> > >> note
> > > > > > > >> > >> >> > String,
> > > > > > > >> > >> >> > > > String
> > headers were
> > > > > discounted early
> > > > > > on. The
> > > > > > > >> > reason for this
> > > > > > > >> > >> is
> > > > > > > >> > >> >> > >
> > flexibility
> > > > > > > >> > >> >> > > > and
> > keeping in line
> > > > with the
> > > > > > flexibility of
> > > > > > > key,
> > > > > > > >> > value of the
> > > > > > > >> > >> >> > message
> > > > > > > >> > >> >> > > > object
> > itself. I don’t
> > > > think
> > > > > it takes
> > > > > > away
> > > > > > > from
> > > > > > > >> an
> > > > > > > >> > ecosystem
> > > > > > > >> > >> as
> > > > > > > >> > >> >> > each
> > > > > > > >> > >> >> > > plugin
> > > > > > > >> > >> >> > > > will
> > care for their
> > > own
> > > > key,
> > > > > this way
> > > > > > ints,
> > > > > > > >> > booleans , exotic
> > > > > > > >> > >> >> > custom
> > > > > > > >> > >> >> > > binary
> > > > > > > >> > >> >> > > > can
> all
> > be catered
> > > for=.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > wanted
> > > > to
> > > > > push for a
> > > > > > typed
> > > > > > > >> value
> > > > > > > >> > interface,
> > > > > > > >> > >> I
> > > > > > > >> > >> >> > wouldn’t
> > > > > > > >> > >> >> > > > want
> > just String
> > > values
> > > > > supported, but
> > > > > > the the
> > > > > > > >> > primatives
> > > > > > > >> > >> plus
> > > > > > > >> > >> >> > string and
> > > > > > > >> > >> >> > > > also
> > still keeping the
> > > > > ability to have a
> > > > > > > binary
> > > > > > > >> > for custom
> > > > > > > >> > >> >> > binaries that
> > > > > > > >> > >> >> > > > some
> > organisations may
> > > > have.
> > > > > > > >> > >> >> > > > i. I
> > have written this
> > > > slight
> > > > > > alternative
> > > > > > > here,
> > > > > > > >> > >> >> > >
> > > https://cwiki.apache.org/
> > > > > > > >> > >> >> > > >
> > > > confluence/display/KAFKA/KIP-
> > > > > > > >> > 82+-+Add+Record+Headers+-+
> Typed
> > > > > > > >> > >> >> > > > ii.
> > Essentially the
> > > > value
> > > > > bytes, has a
> > > > > > leading
> > > > > > > >> > byte overhead.
> > > > > > > >> > >> >> > > > 1.
> > This tells you
> > > what
> > > > type
> > > > > the value
> > > > > > is,
> > > > > > > >> before
> > > > > > > >> > reading
> > > > > > > >> > >> the rest
> > > > > > > >> > >> >> > of the
> > > > > > > >> > >> >> > > > bytes,
> > allowing
> > > > > > serialisation/deserialization
> > > > > > > to
> > > > > > > >> > and from the
> > > > > > > >> > >> >> > primitives,
> > > > > > > >> > >> >> > > > string
> > and byte[].
> > > This
> > > > is
> > > > > akin to some
> > > > > > other
> > > > > > > >> > messaging
> > > > > > > >> > >> systems.
> > > > > > > >> > >> >> > > > 2) We
> > are making it
> > > > optional,
> > > > > so that
> > > > > > for
> > > > > > > those
> > > > > > > >> > not wanting
> > > > > > > >> > >> >> > headers have
> > > > > > > >> > >> >> > > 0
> > > > > > > >> > >> >> > > > bytes
> > overhead (think
> > > > of it
> > > > > as a feature
> > > > > > > flag),
> > > > > > > >> I
> > > > > > > >> > don’t
> > > > > > > >> > >> think this
> > > > > > > >> > >> >> > is
> > > > > > > >> > >> >> > > >
> > complex, especially if
> > > > > comparing to
> > > > > > changes
> > > > > > > >> > proposed in
> > > > > > > >> > >> other kips
> > > > > > > >> > >> >> > like
> > > > > > > >> > >> >> > > >
> kip-98.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > really
> > > > don’t
> > > > > like
> > > > > > this, we
> > > > > > > can
> > > > > > > >> > drop it, but
> > > > > > > >> > >> it
> > > > > > > >> > >> >> > would
> > > > > > > >> > >> >> > > mean
> > > > > > > >> > >> >> > > > buying
> > into 4 bytes
> > > > extra
> > > > > overhead for
> > > > > > users
> > > > > > > who
> > > > > > > >> > do not want
> > > > > > > >> > >> to use
> > > > > > > >> > >> >> > > headers.
> > > > > > > >> > >> >> > > > 3) In
> > the summary yes,
> > > > it is
> > > > > at a higher
> > > > > > > level,
> > > > > > > >> > but I think
> > > > > > > >> > >> this
> > > > > > > >> > >> >> > is well
> > > > > > > >> > >> >> > > >
> > documented in the
> > > > proposed
> > > > > changes
> > > > > > section.
> > > > > > > >> > >> >> > > > a.
> > Added getHeaders
> > > > method to
> > > > > > > Producer/Consumer
> > > > > > > >> > record (that
> > > > > > > >> > >> is it)
> > > > > > > >> > >> >> > > > b.
> > We’ve also detailed
> > > > the
> > > > > new Headers
> > > > > > class
> > > > > > > >> that
> > > > > > > >> > this method
> > > > > > > >> > >> >> > returns
> > > > > > > >> > >> >> > > that
> > > > > > > >> > >> >> > > >
> > encapsulates the
> > > headers
> > > > > protocol and
> > > > > > logic.
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Best,
> > > > > > > >> > >> >> > > > Mike
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > >
> > ==Original questions
> > > > from the
> > > > > vote
> > > > > > thread from
> > > > > > > >> > Jay.==
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Couple
> > of things I
> > > > think we
> > > > > still need
> > > > > > to work
> > > > > > > >> out:
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1.
> I
> > think we agree
> > > > about
> > > > > the key,
> > > > > > but I
> > > > > > > >> think
> > > > > > > >> > we haven't
> > > > > > > >> > >> >> > talked about
> > > > > > > >> > >> >> > > > the
> > value yet. I
> > > > think if
> > > > > our goal
> > > > > > is an
> > > > > > > open
> > > > > > > >> > ecosystem
> > > > > > > >> > >> of these
> > > > > > > >> > >> >> > > header
> > > > > > > >> > >> >> > > >
> > spread across many
> > > > plugins
> > > > > from many
> > > > > > > systems
> > > > > > > >> we
> > > > > > > >> > should
> > > > > > > >> > >> consider
> > > > > > > >> > >> >> > making
> > > > > > > >> > >> >> > > > this
> > > > > > > >> > >> >> > > > a
> > string as well so
> > > > it can
> > > > > be
> > > > > > printed, set
> > > > > > > >> via
> > > > > > > >> > a UI, set
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > config,
> > > > > > > >> > >> >> > > etc.
> > > > > > > >> > >> >> > > >
> > Basically
> > > encouraging
> > > > > pluggable
> > > > > > > serialization
> > > > > > > >> > formats
> > > > > > > >> > >> here will
> > > > > > > >> > >> >> > lead
> > > > > > > >> > >> >> > > to
> > > > > > > >> > >> >> > > > a
> > > > > > > >> > >> >> > > > bit
> > of a tower of
> > > > babel.
> > > > > > > >> > >> >> > > > 2.
> > This proposal
> > > > still
> > > > > includes a
> > > > > > pretty
> > > > > > > big
> > > > > > > >> > change to our
> > > > > > > >> > >> >> > >
> > serialization
> > > > > > > >> > >> >> > > > and
> > protocol
> > > > definition
> > > > > layer.
> > > > > > Essentially
> > > > > > > >> it is
> > > > > > > >> > >> introducing an
> > > > > > > >> > >> >> > > optional
> > > > > > > >> > >> >> > > >
> > type, where the
> > > > format is
> > > > > data
> > > > > > dependent. I
> > > > > > > >> > think this is
> > > > > > > >> > >> >> > actually a
> > > > > > > >> > >> >> > > big
> > > > > > > >> > >> >> > > >
> > change though it
> > > > doesn't
> > > > > seem like
> > > > > > it. It
> > > > > > > >> means
> > > > > > > >> > you can no
> > > > > > > >> > >> >> > longer
> > > > > > > >> > >> >> > > >
> specify
> > > > > > > >> > >> >> > > >
> this
> > type with our
> > > > type
> > > > > definition
> > > > > > DSL, and
> > > > > > > >> > likewise it
> > > > > > > >> > >> requires
> > > > > > > >> > >> >> > > custom
> > > > > > > >> > >> >> > > >
> > handling in client
> > > > libs.
> > > > > This isn't
> > > > > > a huge
> > > > > > > >> > thing, since
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > Record
> > > > > > > >> > >> >> > > >
> > definition is
> > > custom
> > > > > anyway, but I
> > > > > > think
> > > > > > > this
> > > > > > > >> > kind of
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> > > >
> > inconsistency is
> > > very
> > > > > non-desirable
> > > > > > and
> > > > > > > ties
> > > > > > > >> > you to
> > > > > > > >> > >> hand-coding
> > > > > > > >> > >> >> > > things.
> > > > > > > >> > >> >> > > > I
> > > > > > > >> > >> >> > > >
> > think the type
> > > should
> > > > > instead by [Key
> > > > > > > Value]
> > > > > > > >> in
> > > > > > > >> > our BNF,
> > > > > > > >> > >> where
> > > > > > > >> > >> >> > key and
> > > > > > > >> > >> >> > > >
> > value are both
> > > short
> > > > > strings as used
> > > > > > > >> elsewhere.
> > > > > > > >> > This
> > > > > > > >> > >> brings it
> > > > > > > >> > >> >> > in line
> > > > > > > >> > >> >> > > > with
> > > > > > > >> > >> >> > > > the
> > rest of the
> > > > protocol.
> > > > > > > >> > >> >> > > > 3.
> > Could we get
> > > more
> > > > > specific about
> > > > > > the
> > > > > > > exact
> > > > > > > >> > Java API
> > > > > > > >> > >> change to
> > > > > > > >> > >> >> > > >
> > ProducerRecord,
> > > > > ConsumerRecord,
> > > > > > Record,
> > > > > > > etc?
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > -Jay
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The
> information
> > contained in
> > > > this
> > > > > email is
> > > > > > strictly
> > > > > > > >> > confidential and
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > the use of the
> > addressee only,
> > > > unless
> > > > > otherwise
> > > > > > > >> indicated.
> > > > > > > >> > If you
> > > > > > > >> > >> are not
> > > > > > > >> > >> >> > the intended
> > recipient, please
> > > > do not
> > > > > read,
> > > > > > copy, use
> > > > > > > or
> > > > > > > >> > disclose to
> > > > > > > >> > >> others
> > > > > > > >> > >> >> > this message
> or
> > any
> > > attachment.
> > > > > Please also
> > > > > > notify the
> > > > > > > >> > sender by
> > > > > > > >> > >> replying
> > > > > > > >> > >> >> > to this email
> > or by telephone
> > > > > (+44(020 7896
> > > > > > 0011) and
> > > > > > > >> then
> > > > > > > >> > delete
> > > > > > > >> > >> the email
> > > > > > > >> > >> >> > and any copies
> > of it.
> > > Opinions,
> > > > > conclusion
> > > > > > (etc) that
> > > > > > > do
> > > > > > > >> > not relate
> > > > > > > >> > >> to the
> > > > > > > >> > >> >> > official
> > business of this
> > > > company
> > > > > shall be
> > > > > > understood
> > > > > > > as
> > > > > > > >> > neither
> > > > > > > >> > >> given nor
> > > > > > > >> > >> >> > endorsed by
> it.
> > IG is a
> > > trading
> > > > name
> > > > > of IG
> > > > > > Markets
> > > > > > > >> Limited
> > > > > > > >> > (a company
> > > > > > > >> > >> >> > registered in
> > England and
> > > Wales,
> > > > > company number
> > > > > > > >> 04008957)
> > > > > > > >> > and IG
> > > > > > > >> > >> Index
> > > > > > > >> > >> >> > Limited (a
> > company registered
> > > in
> > > > > England and
> > > > > > Wales,
> > > > > > > >> > company number
> > > > > > > >> > >> >> > 01190902).
> > Registered address
> > > at
> > > > > Cannon Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > >> > Dowgate
> > > > > > > >> > >> Hill,
> > > > > > > >> > >> >> > London EC4R
> > 2YA. Both IG
> > > Markets
> > > > > Limited
> > > > > > (register
> > > > > > > >> number
> > > > > > > >> > 195355)
> > > > > > > >> > >> and IG
> > > > > > > >> > >> >> > Index Limited
> > (register number
> > > > > 114059) are
> > > > > > authorised
> > > > > > > >> and
> > > > > > > >> > regulated
> > > > > > > >> > >> by the
> > > > > > > >> > >> >> > Financial
> > Conduct Authority.
> > > > > > > >> > >> >> >
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> --
> > > > > > > >> > >> Nacho - Ignacio
> > Solis -
> > > > isolis@igso.net
> > > > > > > >> > >>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > The information contained in
> > this email is
> > > > strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > indicated. If
> > > > > > you are
> > > > > > > >> not
> > > > > > > >> > the intended recipient, please
> > do not read,
> > > > copy, use
> > > > > or
> > > > > > disclose to
> > > > > > > >> others
> > > > > > > >> > this message or any
> attachment.
> > Please also
> > > > notify
> > > > > the sender by
> > > > > > > >> replying
> > > > > > > >> > to this email or by telephone
> > (+44(020 7896
> > > > 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > >> email
> > > > > > > >> > and any copies of it.
> Opinions,
> > conclusion
> > > > (etc) that
> > > > > do not
> > > > > > relate to
> > > > > > > >> the
> > > > > > > >> > official business of this
> > company shall be
> > > > understood
> > > > > as
> > > > > > neither given
> > > > > > > >> nor
> > > > > > > >> > endorsed by it. IG is a
> trading
> > name of IG
> > > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> > registered in England and
> > Wales, company
> > > number
> > > > > 04008957) and
> > > > > > IG Index
> > > > > > > >> > Limited (a company registered
> > in England and
> > > > Wales,
> > > > > company
> > > > > > number
> > > > > > > >> > 01190902). Registered address
> > at Cannon Bridge
> > > > House,
> > > > > 25
> > > > > > Dowgate Hill,
> > > > > > > >> > London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and
> > > > > > > IG
> > > > > > > >> > Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > >> the
> > > > > > > >> > Financial Conduct Authority.
> > > > > > > >> >
> > > > > > > >> The information contained in
> this
> > email is
> > > > strictly
> > > > > confidential
> > > > > > and for
> > > > > > > >> the use of the addressee only,
> > unless otherwise
> > > > > indicated. If you
> > > > > > are
> > > > > > > not
> > > > > > > >> the intended recipient, please
> do
> > not read,
> > > copy,
> > > > use
> > > > > or disclose
> > > > > > to
> > > > > > > others
> > > > > > > >> this message or any attachment.
> > Please also
> > > > notify the
> > > > > sender by
> > > > > > > replying
> > > > > > > >> to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the
> > > > > > > email
> > > > > > > >> and any copies of it. Opinions,
> > conclusion (etc)
> > > > that
> > > > > do not
> > > > > > relate to
> > > > > > > the
> > > > > > > >> official business of this
> company
> > shall be
> > > > understood
> > > > > as neither
> > > > > > given
> > > > > > > nor
> > > > > > > >> endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> registered in England and Wales,
> > company number
> > > > > 04008957) and IG
> > > > > > Index
> > > > > > > >> Limited (a company registered in
> > England and
> > > > Wales,
> > > > > company number
> > > > > > > >> 01190902). Registered address at
> > Cannon Bridge
> > > > House,
> > > > > 25 Dowgate
> > > > > > Hill,
> > > > > > > >> London EC4R 2YA. Both IG Markets
> > Limited
> > > (register
> > > > > number 195355)
> > > > > > and IG
> > > > > > > >> Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > the
> > > > > > > >> Financial Conduct Authority.
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > The information contained in this
> > email is strictly
> > > > > confidential and
> > > > > > for the use of the addressee only, unless
> > otherwise
> > > > indicated.
> > > > > If you are
> > > > > > not the intended recipient, please do not
> > read, copy, use
> > > > or
> > > > > disclose to
> > > > > > others this message or any attachment.
> > Please also notify
> > > > the
> > > > > sender by
> > > > > > replying to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that
> > > > > do not
> > > > > > relate to the official business of this
> > company shall be
> > > > > understood as
> > > > > > neither given nor endorsed by it. IG is a
> > trading name of
> > > > IG
> > > > > Markets
> > > > > > Limited (a company registered in England
> > and Wales,
> > > company
> > > > > number
> > > > > > 04008957) and IG Index Limited (a company
> > registered in
> > > > England
> > > > > and Wales,
> > > > > > company number 01190902). Registered
> > address at Cannon
> > > > Bridge
> > > > > House, 25
> > > > > > Dowgate Hill, London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and IG Index Limited (register
> > number 114059) are
> > > > > authorised and
> > > > > > regulated by the Financial Conduct
> > Authority.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is
> > strictly
> > > confidential
> > > > and
> > > > > for the use of the addressee only, unless otherwise
> > indicated. If
> > > > you are
> > > > > not the intended recipient, please do not read,
> > copy, use or
> > > > disclose to
> > > > > others this message or any attachment. Please also
> > notify the
> > > sender
> > > > by
> > > > > replying to this email or by telephone (+44(020
> 7896
> > 0011) and then
> > > > delete
> > > > > the email and any copies of it. Opinions,
> conclusion
> > (etc) that do
> > > > not
> > > > > relate to the official business of this company
> > shall be understood
> > > > as
> > > > > neither given nor endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a company registered in England and Wales,
> > company number
> > > > > 04008957) and IG Index Limited (a company
> registered
> > in England and
> > > > Wales,
> > > > > company number 01190902). Registered address at
> > Cannon Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > Limited (register
> > > > number
> > > > > 195355) and IG Index Limited (register number
> > 114059) are
> > > authorised
> > > > and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> > sender by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> > not relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > registered in England and Wales, company number 04008957)
> > and IG Index
> > > > Limited (a company registered in England and Wales,
> > company number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> > for the use of the addressee only, unless otherwise indicated. If you are
> > not the intended recipient, please do not read, copy, use or disclose to
> > others this message or any attachment. Please also notify the sender by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do not
> > relate to the official business of this company shall be understood as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG Index Limited (register number 114059) are authorised and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
RE “ProducerRecord and ConsumerRecord are immutable currently”. I beg to differ on this, if I set a byte[] as my value of my record, I can later call value() and then alter the bytes in the array. As such it is very much mutable.
Headers also will anyhow by nature be mutable on produce, as you add to them.
On 20/02/2017, 00:22, "Jason Gustafson" <ja...@confluent.io> wrote:
Re; having a dedicated Headers object. I think it makes sense. Maybe it
could implement Iterable<Header>?
One thing I'm not too sure about is mutability. Both the ProducerRecord and
ConsumerRecord are immutable currently. It would be nice to keep it that
way. Perhaps Headers could itself be immutable and passed through the
constructors?
Just going through how this is, I don’t believe this would provide much
> saving. If issue trying to resolve is around object/garbage creation of
> Map.Entry objects.
It's definitely better because we only allocate the space we will use, but
I assume most of the actual memory cost is in the header data itself which
we can't do much about. Maybe we can start with the simple array
representation and optimize from there after we find it's worthwhile to do
so? If we use the Headers object, we can change this later without breaking
the API.
-Jason
On Sun, Feb 19, 2017 at 2:39 PM, radai <ra...@gmail.com> wrote:
> "(By the way, doesn't it feel a bit odd that we seem to be designing a
> feature which is optimized for people not using?)"
>
> very. this (i claim :-) ) is the result of very intense opposition to the
> usefulness of the feature early on, and not a real design goal
>
> On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Whilst if having an array of a Header[] would mean getting this through,
> > im happy changing this, as already done.
> >
> > Just going through how this is, I don’t believe this would provide much
> > saving. If issue trying to resolve is around object/garbage creation of
> > Map.Entry objects.
> >
> > All you’re doing here is replacing the equiv of a HashMap.Node
> (Map.Entry)
> > which simply holds reference to key and value objects with a custom
> variant.
> >
> >
> >
> >
> >
> > On 19/02/2017, 20:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> > On point 1 & 2 Ive updated KIP to show varints (and removed the bit
> > flag). (on the assumption KIP 98 is getting the agreement the protocol is
> > moving from int32 to varInts as standard)
> >
> > On point 3 ive updated to use an array of Header class, instead of a
> > MultiMap in the Headers class object
> >
> >
> >
> > On 19/02/2017, 20:06, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > On points 1 and 2 I agree.
> >
> > This also affects kip-98, I should expect this resolved before
> > that vote also passes. If it is accepted there (I’m assuming this is
> > getting discussed on that KIP? As you’re driving the move to VarInts), I
> am
> > happy to make this KIP will simply follow suit to whatever is agreed in
> > KIP-98.
> >
> > On 3) Agreed this is a simpler form, as long as no one is
> > expecting hashmap lookup performance ( O(1) ), I still would prefer an
> > encapsulated class, so if we find that holding it as an array in future
> is
> > causing some perf issues, the internals are not exposed to end users,
> > allowing the internal structure to move to a map.
> >
> > Can we compromise on?
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Headers headers;
> > }
> >
> > class Headers {
> > Header[] headers;
> >
> > add(String key, byte[] value)
> > Collection<byte[]> get(String key)
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> >
> >
> > On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> >
> > >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> >
> >
> > This is exactly what I have been trying to get at. The use
> > cases documented
> > here are middleware:
> > https://cwiki.apache.org/confluence/display/KAFKA/A+
> > Case+for+Kafka+Headers.
> > If headers are intended for the application space as well,
> the
> > document
> > should be updated accordingly so that it is an explicit
> design
> > goal and not
> > an unfortunate design byproduct. There may be some
> > disagreement on whether
> > it _should_ be a design goal, but it may not matter much
> since
> > the current
> > interceptor API is probably insufficient for middleware
> > applications (I
> > haven't thought of a way to do this that isn't cumbersome).
> >
> > That aside, the three unresolved points for me are the
> > following:
> >
> > 1. The use of varints. As a one-off for use only with record
> > headers, the
> > case was weak, but if we are using them throughout the
> message
> > format, then
> > we should do so here as well. The additional complexity is
> > minimal and
> > early performance testing fully justifies their use.
> >
> > 2. If varints are used, the case for using attributes to
> > indicate null
> > headers also becomes weak. We only add one additional byte in
> > each message
> > if there are no headers. Whatever the case, let us be
> > consistent with how
> > we treat null keys and values.
> >
> > 3. We have apparently agreed that the broker will validate
> > headers (please
> > add this to the KIP). That being the case, I would prefer to
> > use Kafka's
> > conventional format for arrays. The downside is that it takes
> > more work to
> > skip over the headers on the consumer, though it's unclear if
> > the cost
> > would matter in practice. Some concern was previously
> > expressed about the
> > allocation of maps. An alternative would be to use arrays,
> i.e.
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Header[] headers;
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> > This would work nicely with the conventional array format and
> > my guess is
> > it would obviate the need do any lazy initialization. If we
> > use the map as
> > is currently documented, then it is possible with either
> > representation to
> > slice the headers and initialize them lazily. Either way, it
> > might be a
> > good idea to use a separate object to represent the headers
> in
> > case we need
> > to modify it in the future in some way.
> >
> > (By the way, doesn't it feel a bit odd that we seem to be
> > designing a
> > feature which is optimized for people not using?)
> >
> >
> > If we can resolve these points, then at least you will get my
> > vote.
> >
> > Thanks,
> > Jason
> >
> > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > radai.rosenblatt@gmail.com> wrote:
> >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> > > IIUC samze currently has headers "in-V" and would just
> > switch over to kafka
> > > headers if they exist.
> > > im sure plenty of other users of kafka would have a use for
> > headers.
> > > im pretty sure use cases exist around shuffling data
> > into/out-of kafka
> > > (kafka connect or equivalent) where metadata from one end
> > could copied over
> > > to the other (S3, for example uses http headers for
> > user-accessible
> > > metadata). it will be kafka client code getting/setting
> > those headers. not
> > > an interceptor.
> > >
> > > On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > For APM single event tracing, need access to the header
> at
> > the point of
> > > > processing on the processing thread.
> > > >
> > > > As such interceptors will not work/be suitable for these,
> > due to the fact
> > > > they act on the ConsumerRecords as a batch, before the
> > handling thread
> > > can
> > > > split out and process per message which is the point
> these
> > tools will
> > > need
> > > > to continue to transaction tracing.
> > > >
> > > > Like wise tools and infra pieces will need access to the
> > message outside
> > > > the interceptor realm.
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 21:26, "Jason Gustafson" <
> > jason@confluent.io> wrote:
> > > >
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > >
> > > >
> > > > So the broker will parse the headers and validate
> > them. Good. The
> > > only
> > > > reason remaining that I can see to leave the headers
> > as a byte array
> > > > is to
> > > > make it easier for the client to skip past them. Are
> > we sure this is
> > > > not
> > > > premature optimization? Are there any performance
> > results which show
> > > > that
> > > > this is worthwhile?
> > > >
> > > > What’s the issue with exposing a method getHeaders on
> > the
> > > > producer/consumer
> > > > > record? It doesn’t break anything. We don’t need
> any
> > special
> > > version.
> > > >
> > > >
> > > > See my previous explanation. What I am trying to
> > resist is the
> > > headers
> > > > becoming a general application-level facility. The
> > primary use case
> > > as
> > > > far
> > > > as I can tell is middleware, which is the use case
> > that the
> > > > interceptors
> > > > are providing.
> > > >
> > > > Current batch consumer model and consumer
> interceptors
> > don’t work
> > > where
> > > > > headers need to be acted on at per message level at
> > time of
> > > > processing,
> > > > > very case is APM (the core one), where the header
> > value is used to
> > > > continue
> > > > > tracing.
> > > >
> > > >
> > > > I still don't understand the point about batching.
> The
> > consumer
> > > > records are
> > > > exposed as a batch in the consumer interceptor, but
> > you can still
> > > > iterate
> > > > them individually. It is no different for the
> consumer
> > API itself.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > Re:
> > > > >
> > > > > “ The point about creation of maps seems
> > orthogonal. We can
> > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the time
> > it is accessed.”
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > >
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:37, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > What’s the issue with exposing a method
> > getHeaders on the
> > > > > producer/consumer record? It doesn’t break
> anything.
> > We don’t need
> > > > any
> > > > > special version.
> > > > >
> > > > > Current batch consumer model and consumer
> > interceptors don’t
> > > work
> > > > > where headers need to be acted on at per message
> > level at time of
> > > > > processing, very case is APM (the core one), where
> > the header value
> > > > is used
> > > > > to continue tracing. JMS/HTTP etc all expose these,
> > without issues.
> > > > I would
> > > > > NOT want to lock this down to only be usable
> > accessible via
> > > > interceptors,
> > > > > as we’d fail on one of the main goals.
> > > > >
> > > > > Regards
> > > > > Mike
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:21, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > >
> > > > > The point about creation of maps seems
> > orthogonal. We can
> > > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the
> > time it is
> > > > accessed.
> > > > >
> > > > >
> > > > > > Yes exactly we have access to the records
> > thus why the
> > > > header
> > > > > should be
> > > > > > accessible via it and not hidden for only
> > interceptors to
> > > > access.
> > > > >
> > > > >
> > > > > As explained above, the point is to make
> the
> > intended usage
> > > > clear.
> > > > > Applications should continue to rely on the
> > key/value
> > > fields
> > > > to
> > > > > serialize
> > > > > their own headers, and it would be more
> > ideal if we can
> > > avoid
> > > > > leaking
> > > > > third-party headers into applications. This
> > is difficult to
> > > > do
> > > > > with the
> > > > > current interceptors because they share the
> > record objects
> > > > with
> > > > > the common
> > > > > API. What I had in mind is something like
> an
> > extension of
> > > the
> > > > > current
> > > > > interceptors which exposed a different
> > object (e.g.
> > > > > `RecordAndHeaders`).
> > > > > The challenge is for MM-like use cases. Let
> > me see if I can
> > > > come
> > > > > up with a
> > > > > concrete proposal for that problem.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 17, 2017 at 11:55 AM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > I am happy to move the definition of the
> > header into the
> > > > message
> > > > > body, but
> > > > > > would cause us not to lazy
> > initialise/parse the headers,
> > > as
> > > > > obviously, we
> > > > > > would have to traverse these reading the
> > message.
> > > > > >
> > > > > > This was actually one of Jay’s requests:
> > > > > >
> > > > > > “ 2. I think we should think about
> > creating the Map
> > > > lazily to
> > > > > avoid
> > > > > > parsing out all the headers into
> > little objects.
> > > > HashMaps
> > > > > themselves
> > > > > > are
> > > > > > kind of expensive and the consumer is
> > very perf
> > > > sensitive so
> > > > > and making
> > > > > > gazillions of hashmaps that may or
> may
> > not get used
> > > is
> > > > > probably a bad
> > > > > > idea.”
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > Yes exactly we have access to the
> > records thus why
> > > the
> > > > > header should
> > > > > > be accessible via it and not hidden for
> > only interceptors
> > > > to
> > > > > access.
> > > > > >
> > > > > > Sent using OWA for iPhone
> > > > > > ______________________________
> > __________
> > > > > > From: Magnus Edenhill <
> > magnus@edenhill.se>
> > > > > > Sent: Friday, February 17, 2017
> > 7:34:49 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> > Record Headers
> > > > > >
> > > > > > Big +1 on VarInts.
> > > > > > CPUs are fast, memory is slow.
> > > > > >
> > > > > > I agree with Jason that we'll want to
> > continue
> > > > verifying
> > > > > messages,
> > > > > > including their headers, so while I
> > appreciate the
> > > > idea of
> > > > > the opaque
> > > > > > header blob it won't be useful in
> > practice.
> > > > > >
> > > > > > /Magnus
> > > > > >
> > > > > > 2017-02-17 10:41 GMT-08:00 Jason
> > Gustafson <
> > > > > jason@confluent.io>:
> > > > > >
> > > > > > > Sorry, my mistake. The consumer
> > interceptor is per
> > > > batch,
> > > > > though I'm
> > > > > > not
> > > > > > > sure that's an actual limitation
> > since you still
> > > have
> > > > > access to the
> > > > > > > individual records.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 10:39 AM,
> > Jason Gustafson <
> > > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Re headers as byte array and
> > future use by
> > > broker.
> > > > This
> > > > > doesn't
> > > > > > take away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >
> > > > > > > >
> > > > > > > > Yeah, I didn't say it was
> > difficult, only
> > > awkward.
> > > > You
> > > > > wouldn't
> > > > > > write the
> > > > > > > > schema that way if you were
> > planning to use it on
> > > > the
> > > > > brokers from
> > > > > > the
> > > > > > > > beginning. Note also that one of
> > the benefits of
> > > > letting
> > > > > the broker
> > > > > > > > understand headers is that it can
> > validate that
> > > > they are
> > > > > properly
> > > > > > > > formatted. If cost is the only
> > concern, we should
> > > > > confirm its
> > > > > > impact
> > > > > > > > through performance testing.
> > > > > > > >
> > > > > > > > One of the key use cases requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >
> > > > > > > >
> > > > > > > > I'm not sure I understand the
> > point about
> > > batching.
> > > > > Interceptors
> > > > > > are
> > > > > > > > applied per-message, right?
> > > > > > > >
> > > > > > > > My intent on interceptors is to
> > keep the usage of
> > > > headers
> > > > > > well-defined so
> > > > > > > > that they don't start leaking
> > unnecessarily into
> > > > > applications. My
> > > > > > guess
> > > > > > > is
> > > > > > > > that it's probably inevitable,
> but
> > isolating it
> > > in
> > > > the
> > > > > > interceptors would
> > > > > > > > at least give people a second
> > thought before
> > > > deciding to
> > > > > use it.
> > > > > > The main
> > > > > > > > challenge in my mind is figuring
> > out how an MM
> > > use
> > > > case
> > > > > would
> > > > > > work. It
> > > > > > > > would be more cumbersome to
> > replicate headers
> > > > through an
> > > > > > interceptor,
> > > > > > > > though arguably MM should be
> > working at a lower
> > > > level
> > > > > anyway.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at 10:16 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Re headers available on the
> > record va
> > > > interceptors only
> > > > > > > >>
> > > > > > > >> One of the key use cases
> requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >>
> > > > > > > >> This anyhow is similar to
> > jms/http/amqp where
> > > > headers
> > > > > are
> > > > > > available to
> > > > > > > >> consuming applications.
> > > > > > > >>
> > > > > > > >> Re headers as byte array and
> > future use by
> > > > broker. This
> > > > > doesn't
> > > > > > take
> > > > > > > away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Sent using OWA for iPhone
> > > > > > > >> ______________________________
> > __________
> > > > > > > >> From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > >> Sent: Friday, February 17, 2017
> > 5:55:42 PM
> > > > > > > >> To: dev@kafka.apache.org
> > > > > > > >> Subject: Re: [DISCUSS] KIP-82 -
> > Add Record
> > > Headers
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Yes, that is what we are
> > proposing, so using
> > > > varints
> > > > > for headers
> > > > > > would
> > > > > > > be
> > > > > > > >> consistent with the rest of the
> > message. We have
> > > > used
> > > > > static
> > > > > > helper
> > > > > > > >> methods
> > > > > > > >> in our prototype implementation.
> > > > > > > >>
> > > > > > > >> The cost of parsing, we want to
> > parse/interpret
> > > > the
> > > > > headers
> > > > > > lazily (this
> > > > > > > >> is
> > > > > > > >> > a key point brought up earlier
> > in discussions)
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> I'm a bit skeptical of this. Has
> > anyone done the
> > > > > performance
> > > > > > testing? I
> > > > > > > >> can
> > > > > > > >> probably implement it and test
> it
> > if no one else
> > > > has. I
> > > > > was also
> > > > > > under
> > > > > > > the
> > > > > > > >> impression that there may be use
> > cases down the
> > > > road
> > > > > where the
> > > > > > broker
> > > > > > > >> would
> > > > > > > >> need to interpret headers. That
> > wouldn't be off
> > > > the
> > > > > table in the
> > > > > > future
> > > > > > > if
> > > > > > > >> it's represented as bytes, but
> it
> > would be quite
> > > > a bit
> > > > > more
> > > > > > awkward,
> > > > > > > >> right?
> > > > > > > >>
> > > > > > > >> By the way, one question I have
> > been wondering
> > > > about. My
> > > > > > understanding
> > > > > > > is
> > > > > > > >> that headers are primarily for
> > use cases where a
> > > > > third-party
> > > > > > components
> > > > > > > >> wants to enrich messages without
> > needing to
> > > > understand
> > > > > or modify
> > > > > > the
> > > > > > > >> schema
> > > > > > > >> of the message key and value.
> For
> > the
> > > applications
> > > > > which directly
> > > > > > > produce
> > > > > > > >> and consume the messages and
> > control the
> > > key/value
> > > > > schema
> > > > > > directly, it
> > > > > > > >> seems we would rather have them
> > implement
> > > headers
> > > > > directly in
> > > > > > their own
> > > > > > > >> schema. Supposing for the sake
> of
> > argument that
> > > > it was
> > > > > possible,
> > > > > > my
> > > > > > > >> question is whether it be
> > sufficient to expose
> > > the
> > > > > headers in the
> > > > > > > >> interceptor API and not in the
> > common API?
> > > > > > > >>
> > > > > > > >> -Jason
> > > > > > > >>
> > > > > > > >> On Fri, Feb 17, 2017 at 3:26 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > On the point of varInts
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 11:22, "Michael
> > Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > On the point re: headers
> in
> > the message
> > > > protocol
> > > > > being a
> > > > > > byte
> > > > > > > array
> > > > > > > >> > and not a count of elements
> > followed by the
> > > > elements.
> > > > > Again
> > > > > > this was
> > > > > > > >> > discussed/argued previously.
> > > > > > > >> >
> > > > > > > >> > It was agreed on for a few
> > reasons some of
> > > > which
> > > > > you have
> > > > > > > obviously
> > > > > > > >> > picked up on:
> > > > > > > >> >
> > > > > > > >> > Broker is able to pass it
> > through opaquely
> > > > > > > >> > The cost of parsing, we
> > want to
> > > > parse/interpret
> > > > > the headers
> > > > > > lazily
> > > > > > > >> > (this is a key point brought
> up
> > earlier in
> > > > > discussions)
> > > > > > > >> > Headers can be copied from
> > consumer record
> > > > to
> > > > > producer
> > > > > > record (aka
> > > > > > > >> > mirror makers etc) without
> > parsing if no
> > > > changes are
> > > > > being made
> > > > > > or
> > > > > > > being
> > > > > > > >> > looked at.
> > > > > > > >> > Keeps the broker agnostic
> > to the format
> > > > > > > >> > You need an int32 either
> > for the byte size
> > > > of the
> > > > > headers,
> > > > > > or for
> > > > > > > >> the
> > > > > > > >> > count of elements, so
> overheads
> > are the same,
> > > > but
> > > > > with going
> > > > > > with an
> > > > > > > >> opaque
> > > > > > > >> > byte array has the above
> > advantages.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 02:50,
> > "Jason Gustafson" <
> > > > > jason@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > Sorry, should have
> > noted that the
> > > > performance
> > > > > testing
> > > > > > was done
> > > > > > > >> > using the
> > > > > > > >> > producer performance
> > tool shipped with
> > > > Kafka.
> > > > > > > >> >
> > > > > > > >> > -Jason
> > > > > > > >> >
> > > > > > > >> > On Thu, Feb 16, 2017
> at
> > 6:44 PM, Jason
> > > > > Gustafson <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hey Nacho,
> > > > > > > >> > >
> > > > > > > >> > > I've compared
> > performance of our
> > > > KIP-98
> > > > > > implementation with
> > > > > > > >> and
> > > > > > > >> > without
> > > > > > > >> > > varints. For
> messages
> > around 128
> > > > bytes, we
> > > > > see an
> > > > > > increase
> > > > > > > in
> > > > > > > >> > throughput of
> > > > > > > >> > > about 30% using the
> > default
> > > > configuration
> > > > > settings.
> > > > > > At 256
> > > > > > > >> > bytes, the
> > > > > > > >> > > increase is around
> > 16%. Obviously
> > > the
> > > > > performance
> > > > > > converge
> > > > > > > as
> > > > > > > >> > messages get
> > > > > > > >> > > larger, but it seems
> > well worth the
> > > > cost.
> > > > > Note that
> > > > > > we are
> > > > > > > >> also
> > > > > > > >> > seeing a
> > > > > > > >> > > substantial
> > performance increase
> > > > against
> > > > > trunk
> > > > > > primarily
> > > > > > > >> because
> > > > > > > >> > of the
> > > > > > > >> > > much more efficient
> > packing that
> > > > varints
> > > > > provide us.
> > > > > > > Anything
> > > > > > > >> > adding to
> > > > > > > >> > > message overhead,
> > such as record
> > > > headers,
> > > > > would only
> > > > > > > increase
> > > > > > > >> > the relative
> > > > > > > >> > > difference. (Of
> > course take these
> > > > numbers
> > > > > with a
> > > > > > grain of
> > > > > > > salt
> > > > > > > >> > since I have
> > > > > > > >> > > only used the
> default
> > settings with
> > > > both
> > > > > the producer
> > > > > > and
> > > > > > > >> broker
> > > > > > > >> > on my
> > > > > > > >> > > local machine. We
> > intend to provide
> > > > more
> > > > > extensive
> > > > > > > performance
> > > > > > > >> > details as
> > > > > > > >> > > part of the work for
> > KIP-98.)
> > > > > > > >> > >
> > > > > > > >> > > The implementation
> we
> > are using is
> > > > from
> > > > > protobuf (
> > > > > > > >> > >
> > https://developers.google.com/
> > > > > > > protocol-buffers/docs/encoding
> > > > > > > >> ),
> > > > > > > >> > which is
> > > > > > > >> > > also used in HBase.
> > It is trivial to
> > > > > implement and as
> > > > > > far
> > > > > > > as I
> > > > > > > >> > know doesn't
> > > > > > > >> > > suffer from the
> > aliasing problem you
> > > > are
> > > > > describing. I
> > > > > > > checked
> > > > > > > >> > with Magnus
> > > > > > > >> > > (the author of
> > librdkafka) and he
> > > > agreed
> > > > > that the
> > > > > > savings
> > > > > > > >> seemed
> > > > > > > >> > worth the
> > > > > > > >> > > cost of
> > implementation.
> > > > > > > >> > >
> > > > > > > >> > > -Jason
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Feb 16, 2017
> > at 4:32 PM,
> > > > Ignacio
> > > > > Solis <
> > > > > > > >> isolis@igso.net>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> -VarInts
> > > > > > > >> > >>
> > > > > > > >> > >> I'm one of the
> > people (if not the
> > > > most)
> > > > > opposed to
> > > > > > VarInts.
> > > > > > > >> > VarInts
> > > > > > > >> > >> have a place, but
> > this is not it.
> > > > (We
> > > > > had a large
> > > > > > > >> discussion
> > > > > > > >> > about
> > > > > > > >> > >> them at the
> > beginning of KIP-82
> > > time)
> > > > > > > >> > >>
> > > > > > > >> > >> If anybody has real
> > life
> > > performance
> > > > > numbers of
> > > > > > VarInts
> > > > > > > >> > improving
> > > > > > > >> > >> things or
> > significantly reducing
> > > > resources
> > > > > I would
> > > > > > like to
> > > > > > > >> know
> > > > > > > >> > what
> > > > > > > >> > >> that case may be.
> > Yes, you can save
> > > > some
> > > > > bytes here
> > > > > > and
> > > > > > > >> there,
> > > > > > > >> > but
> > > > > > > >> > >> this is probably
> > insignificant to
> > > the
> > > > > overall system
> > > > > > > behavior
> > > > > > > >> > and
> > > > > > > >> > >> storage
> > requirements. -- I say
> > > this
> > > > with
> > > > > respect to
> > > > > > using
> > > > > > > >> > VarInts in
> > > > > > > >> > >> the protocol
> itself,
> > not as part of
> > > > the
> > > > > data.
> > > > > > > >> > >>
> > > > > > > >> > >> VarInts require you
> > to parse the
> > > Int
> > > > > before using it
> > > > > > and
> > > > > > > >> > depending on
> > > > > > > >> > >> the encoding they
> > can suffer from
> > > > aliasing
> > > > > (multiple
> > > > > > > >> > representations
> > > > > > > >> > >> for the same
> value).
> > > > > > > >> > >>
> > > > > > > >> > >> Why add complexity?
> > > > > > > >> > >>
> > > > > > > >> > >> Nacho
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> On Thu, Feb 16,
> 2017
> > at 10:29 AM,
> > > > Colin
> > > > > McCabe <
> > > > > > > >> > cmccabe@apache.org>
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> > +1 for varints
> > here-- it would
> > > save
> > > > > quite a bit of
> > > > > > space.
> > > > > > > >> > They are
> > > > > > > >> > >> > pretty quick to
> > implement as
> > > well.
> > > > > > > >> > >> >
> > > > > > > >> > >> > I think it makes
> > sense for values
> > > > to be
> > > > > byte
> > > > > > arrays.
> > > > > > > Users
> > > > > > > >> > might want
> > > > > > > >> > >> > to attach
> > arbitrary payloads;
> > > they
> > > > > shouldn't be
> > > > > > forced to
> > > > > > > >> > serialize
> > > > > > > >> > >> > everything to
> Java
> > strings.
> > > > > > > >> > >> >
> > > > > > > >> > >> > best,
> > > > > > > >> > >> > Colin
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Thu, Feb 16,
> > 2017, at 09:52,
> > > > Jason
> > > > > Gustafson
> > > > > > wrote:
> > > > > > > >> > >> >> Hey Michael,
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Hmm, I guess the
> > point of
> > > > representing
> > > > > it as
> > > > > > bytes is to
> > > > > > > >> > allow the
> > > > > > > >> > >> broker
> > > > > > > >> > >> >> to pass it
> > through opaquely? Is
> > > > the
> > > > > cost of
> > > > > > parsing
> > > > > > > them a
> > > > > > > >> > concern, or
> > > > > > > >> > >> >> are
> > > > > > > >> > >> >> we simply trying
> > to ensure that
> > > > the
> > > > > broker stays
> > > > > > > agnostic
> > > > > > > >> to
> > > > > > > >> > the
> > > > > > > >> > >> format?
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On varints, I
> > think adding
> > > > support for
> > > > > them makes
> > > > > > less
> > > > > > > >> sense
> > > > > > > >> > for an
> > > > > > > >> > >> >> isolated use
> > case, but as part
> > > of
> > > > a
> > > > > more holistic
> > > > > > change
> > > > > > > >> > (such as what
> > > > > > > >> > >> we
> > > > > > > >> > >> >> have proposed in
> > KIP-98), I
> > > think
> > > > they
> > > > > are
> > > > > > justifiable.
> > > > > > > If
> > > > > > > >> > we add them,
> > > > > > > >> > >> >> then the need to
> > use attributes
> > > > becomes
> > > > > quite a
> > > > > > bit
> > > > > > > >> weaker,
> > > > > > > >> > right? The
> > > > > > > >> > >> >> other thing I
> > find slightly odd
> > > > is the
> > > > > fact that
> > > > > > null
> > > > > > > >> > headers has no
> > > > > > > >> > >> >> actual
> > > > > > > >> > >> >> semantic meaning
> > for the message
> > > > > (unlike null
> > > > > > keys and
> > > > > > > >> > values). It is
> > > > > > > >> > >> >> just
> > > > > > > >> > >> >> a space
> > optimization. It seems a
> > > > bit
> > > > > better to
> > > > > > always
> > > > > > > use
> > > > > > > >> > size 0 to
> > > > > > > >> > >> >> indicate having
> > no headers.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Overall, the
> main
> > point is
> > > > ensuring
> > > > > that the
> > > > > > message
> > > > > > > >> schema
> > > > > > > >> > remains
> > > > > > > >> > >> >> consistent,
> > either within the
> > > > larger
> > > > > protocol, or
> > > > > > at a
> > > > > > > >> > minimum within
> > > > > > > >> > >> the
> > > > > > > >> > >> >> message itself.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> -Jason
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Thu, Feb 16,
> > 2017 at 6:39 AM,
> > > > > Michael Pearce <
> > > > > > > >> > Michael.Pearce@ig.com
> > > > > > > >> > >> >
> > > > > > > >> > >> >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> > Hi Jason,
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 1) in
> > the message
> > > > protocol
> > > > > the headers
> > > > > > are
> > > > > > > >> simply
> > > > > > > >> > a byte
> > > > > > > >> > >> array,
> > > > > > > >> > >> >> > as like the
> key
> > or value, this
> > > > is to
> > > > > clearly
> > > > > > demarcate
> > > > > > > >> the
> > > > > > > >> > header in
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > core message.
> > Then the header
> > > > byte
> > > > > array in the
> > > > > > core
> > > > > > > >> > message is an
> > > > > > > >> > >> array of
> > > > > > > >> > >> >> > key, value
> > pairs. This is what
> > > > it is
> > > > > denoting.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Then this
> would
> > be I guess in
> > > > the
> > > > > given
> > > > > > notation:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [KeyLength, Key,
> > > > > ValueLength, Value]
> > > > > > > >> > >> >> > KeyLength
> > => int32
> > > > > <-----------------NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised key
> > value
> > > > > > > >> > >> >> > Key =>
> bytes
> > > > > <---------------------- NEW
> > > > > > > serialised
> > > > > > > >> > string (UTF8)
> > > > > > > >> > >> >> > bytes of the
> > header key
> > > > > > > >> > >> >> >
> ValueLength
> > => int32
> > > > > <-------------- NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised
> > header value
> > > > > > > >> > >> >> > Value =>
> > bytes
> > > > > <-------------------- NEW
> > > > > > > serialised
> > > > > > > >> > form of the
> > > > > > > >> > >> header
> > > > > > > >> > >> >> > value
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The key length
> > and value
> > > length
> > > > is
> > > > > matching the
> > > > > > way
> > > > > > > the
> > > > > > > >> > protocol is
> > > > > > > >> > >> >> > defined in the
> > core message
> > > > currently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 2)
> > > > > > > >> > >> >> > Var sized
> ints,
> > this was
> > > > discussed
> > > > > much earlier
> > > > > > on, in
> > > > > > > >> > fact I had
> > > > > > > >> > >> >> > suggested it
> > myself (with
> > > Hadoop
> > > > > references),
> > > > > > the
> > > > > > > >> > complexity of this
> > > > > > > >> > >> >> > compared to
> > having a simpler
> > > > protocol
> > > > > was
> > > > > > argued and
> > > > > > > >> > agreed it
> > > > > > > >> > >> wasn’t worth
> > > > > > > >> > >> >> > the complexity
> > as all other
> > > > clients
> > > > > in other
> > > > > > languages
> > > > > > > >> > would need to
> > > > > > > >> > >> ensure
> > > > > > > >> > >> >> > theyre using
> > the right var
> > > size
> > > > > algorithm, as
> > > > > > there
> > > > > > > is a
> > > > > > > >> > few.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 3)
> > > > > > > >> > >> >> > We did the
> > attributes,
> > > optional
> > > > > approach as
> > > > > > originally
> > > > > > > >> > there was
> > > > > > > >> > >> marked
> > > > > > > >> > >> >> > concern that
> > headers would
> > > > cause a
> > > > > message size
> > > > > > > overhead
> > > > > > > >> > for others,
> > > > > > > >> > >> who
> > > > > > > >> > >> >> > don’t want
> > them. As such this
> > > > is the
> > > > > clean
> > > > > > solution to
> > > > > > > >> > achieve that.
> > > > > > > >> > >> If
> > > > > > > >> > >> >> > that no longer
> > holds, and we
> > > > don’t
> > > > > care that we
> > > > > > add
> > > > > > > >> 4bytes
> > > > > > > >> > overhead,
> > > > > > > >> > >> then
> > > > > > > >> > >> >> > im happy to
> > remove.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I’m personally
> > in favour of
> > > > keeping
> > > > > the message
> > > > > > as
> > > > > > > small
> > > > > > > >> > as possible
> > > > > > > >> > >> so
> > > > > > > >> > >> >> > people don’t
> > get shocks in
> > > perf
> > > > and
> > > > > throughputs
> > > > > > dues
> > > > > > > to
> > > > > > > >> > message size,
> > > > > > > >> > >> >> > unless they
> > actively use the
> > > > feature,
> > > > > as such I
> > > > > > do
> > > > > > > >> prefer
> > > > > > > >> > the
> > > > > > > >> > >> attribute bit
> > > > > > > >> > >> >> > wise feature
> > flag approach
> > > > myself.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On 16/02/2017,
> > 05:40, "Jason
> > > > > Gustafson" <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > We have
> > proposed a few
> > > > > significant changes
> > > > > > to the
> > > > > > > >> > message format
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > KIP-98
> > > > > > > >> > >> >> > which now
> > seems likely to
> > > > pass
> > > > > (perhaps
> > > > > > with some
> > > > > > > >> > iterations on
> > > > > > > >> > >> >> >
> > implementation details).
> > > It
> > > > would
> > > > > be good
> > > > > > to try
> > > > > > > and
> > > > > > > >> > coordinate
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > changes
> > > > > > > >> > >> >> > in both of
> > the proposals
> > > to
> > > > make
> > > > > sure they
> > > > > > are
> > > > > > > >> > consistent and
> > > > > > > >> > >> >> > compatible.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I think
> > using the
> > > > attributes to
> > > > > indicate
> > > > > > null
> > > > > > > >> headers
> > > > > > > >> > is a
> > > > > > > >> > >> reasonable
> > > > > > > >> > >> >> > approach.
> > We have proposed
> > > > to do
> > > > > the same
> > > > > > thing
> > > > > > > for
> > > > > > > >> > the message
> > > > > > > >> > >> key and
> > > > > > > >> > >> >> > value.
> That
> > said, I
> > > > sympathize
> > > > > with Jay's
> > > > > > > argument.
> > > > > > > >> > Having
> > > > > > > >> > >> multiple
> > > > > > > >> > >> >> > ways to
> > > > > > > >> > >> >> > specify a
> > null value
> > > > increases
> > > > > the overall
> > > > > > > >> complexity
> > > > > > > >> > of the
> > > > > > > >> > >> protocol.
> > > > > > > >> > >> >> > You
> > > > > > > >> > >> >> > can see
> > this just from the
> > > > fact
> > > > > that you
> > > > > > need the
> > > > > > > >> > extra verbiage
> > > > > > > >> > >> in the
> > > > > > > >> > >> >> > protocol
> > specification in
> > > > this
> > > > > KIP and in
> > > > > > KIP-98
> > > > > > > to
> > > > > > > >> > describe the
> > > > > > > >> > >> >> > dependence
> > > > > > > >> > >> >> > between
> the
> > fields and the
> > > > > attributes. It
> > > > > > seems
> > > > > > > >> like a
> > > > > > > >> > slippery
> > > > > > > >> > >> slope
> > > > > > > >> > >> >> > if
> > > > > > > >> > >> >> > you start
> > allowing
> > > different
> > > > > request types
> > > > > > to
> > > > > > > >> > implement the
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> >
> > specification differently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > You can
> > also argue that
> > > the
> > > > > messages
> > > > > > already are
> > > > > > > and
> > > > > > > >> > are likely
> > > > > > > >> > >> to
> > > > > > > >> > >> >> > remain a
> > > > > > > >> > >> >> > special
> > case. For example,
> > > > there
> > > > > is
> > > > > > currently no
> > > > > > > >> > generality in
> > > > > > > >> > >> how
> > > > > > > >> > >> >> > compressed
> > message sets
> > > are
> > > > > represented
> > > > > > that would
> > > > > > > >> be
> > > > > > > >> > applicable
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > other
> > > > > > > >> > >> >> > request
> > types. Some might
> > > > see this
> > > > > > divergence as
> > > > > > > an
> > > > > > > >> > unfortunate
> > > > > > > >> > >> >> > protocol
> > > > > > > >> > >> >> > deficiency
> > which should be
> > > > fixed;
> > > > > others
> > > > > > might see
> > > > > > > >> it
> > > > > > > >> > as sort of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> >
> > inevitability of needing
> > > to
> > > > > optimize where
> > > > > > it
> > > > > > > counts
> > > > > > > >> > most. I'm
> > > > > > > >> > >> probably
> > > > > > > >> > >> >> > somewhere
> > in between, but
> > > I
> > > > think
> > > > > we
> > > > > > probably all
> > > > > > > >> > share the
> > > > > > > >> > >> intuition
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> > protocol should be
> > > kept
> > > > as
> > > > > consistent as
> > > > > > > >> possible.
> > > > > > > >> > With that
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > mind,
> > > > > > > >> > >> >> > here are a
> > few comments:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 1. One
> > thing I found a
> > > > little odd
> > > > > when
> > > > > > reading the
> > > > > > > >> > current
> > > > > > > >> > >> proposal is
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> headers
> > are both
> > > > represented
> > > > > as an
> > > > > > array of
> > > > > > > >> bytes
> > > > > > > >> > and as an
> > > > > > > >> > >> array
> > > > > > > >> > >> >> > of
> > > > > > > >> > >> >> > key/value
> > pairs. I'd
> > > > probably
> > > > > suggest
> > > > > > something
> > > > > > > like
> > > > > > > >> > this:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [HeaderKey
> > > > HeaderValue]
> > > > > > > >> > >> >> > HeaderKey
> > => String
> > > > > > > >> > >> >> >
> > HeaderValue => Bytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > An array
> in
> > the Kafka
> > > > protocol is
> > > > > > represented as a
> > > > > > > >> > 4-byte integer
> > > > > > > >> > >> >> > indicating
> > the number of
> > > > elements
> > > > > in the
> > > > > > array
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> >
> > serialization of the
> > > > elements.
> > > > > Unless I'm
> > > > > > > >> > misunderstanding, what
> > > > > > > >> > >> you
> > > > > > > >> > >> >> > have
> > > > > > > >> > >> >> > instead is
> > the total size
> > > > of the
> > > > > headers in
> > > > > > bytes
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> > elements.
> > > > > > > >> > >> >> > I'm not
> > sure I see any
> > > > reason for
> > > > > this
> > > > > > > >> inconsistency.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 2. In
> > KIP-98, we've
> > > > introduced
> > > > > > variable-length
> > > > > > > >> integer
> > > > > > > >> > fields.
> > > > > > > >> > >> >> > Effectively,
> > > > > > > >> > >> >> > we've
> > enriched (or
> > > > "complicated"
> > > > > as Jay
> > > > > > might say
> > > > > > > ;)
> > > > > > > >> > the protocol
> > > > > > > >> > >> >> >
> > specification to include
> > > the
> > > > > following
> > > > > > types:
> > > > > > > >> VarInt,
> > > > > > > >> > VarLong,
> > > > > > > >> > >> >> >
> > UnsignedVarInt and
> > > > > UnsignedVarLong.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Along with
> > these
> > > > primitives, we
> > > > > could
> > > > > > introduce
> > > > > > > the
> > > > > > > >> > following
> > > > > > > >> > >> types:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeArray =>
> > > > NumberOfItems
> > > > > Item1 Item2
> > > > > > .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItems =>
> > > > UnsignedVarInt
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeNullableArray =>
> > > > > NumberOfItemsOrNull
> > > > > > Item1
> > > > > > > >> > Item2 .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItemsOrNull =>
> > > > VarInt
> > > > > (-1 means
> > > > > > null)
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > And
> > similarly for the
> > > > `String`
> > > > > and `Bytes`
> > > > > > types.
> > > > > > > >> > These types
> > > > > > > >> > >> can save
> > > > > > > >> > >> >> > a
> > > > > > > >> > >> >> >
> > considerable amount of
> > > > space in
> > > > > this
> > > > > > proposal
> > > > > > > >> because
> > > > > > > >> > they can
> > > > > > > >> > >> be used
> > > > > > > >> > >> >> > for
> > > > > > > >> > >> >> > both the
> > number of headers
> > > > > included in the
> > > > > > message
> > > > > > > >> and
> > > > > > > >> > the
> > > > > > > >> > >> lengths of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > header
> keys
> > and values. We
> > > > could
> > > > > do this
> > > > > > instead:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > > > VarSizeArray[HeaderKey
> > > > > > HeaderValue]
> > > > > > > >> > >> >> >
> HeaderKey
> > =>
> > > VarSizeString
> > > > > > > >> > >> >> >
> > HeaderValue =>
> > > > VarSizeBytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Combining
> > the savings from
> > > > the
> > > > > use of
> > > > > > variable
> > > > > > > >> length
> > > > > > > >> > fields, the
> > > > > > > >> > >> >> > benefit
> > > > > > > >> > >> >> > of using
> > the attributes to
> > > > > represent null
> > > > > > seems
> > > > > > > >> pretty
> > > > > > > >> > small.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 3.
> > Whichever way we go
> > > > (whether
> > > > > we use the
> > > > > > > >> attributes
> > > > > > > >> > or not), we
> > > > > > > >> > >> >> > should at
> > > > > > > >> > >> >> > least be
> > consistent
> > > between
> > > > this
> > > > > KIP and
> > > > > > KIP-98.
> > > > > > > It
> > > > > > > >> > would be very
> > > > > > > >> > >> >> > strange
> > > > > > > >> > >> >> > to have
> two
> > ways to
> > > > represent
> > > > > null values
> > > > > > in the
> > > > > > > >> same
> > > > > > > >> > schema.
> > > > > > > >> > >> Either
> > > > > > > >> > >> >> > way is
> > > > > > > >> > >> >> > OK with
> me.
> > I think some
> > > > > message-level
> > > > > > > optimizations
> > > > > > > >> > are
> > > > > > > >> > >> justifiable,
> > > > > > > >> > >> >> > but
> > > > > > > >> > >> >> > the
> savings
> > here seem
> > > > minimal (a
> > > > > few bytes
> > > > > > per
> > > > > > > >> > message), so
> > > > > > > >> > >> maybe it's
> > > > > > > >> > >> >> > not
> > > > > > > >> > >> >> > worth the
> > cost of letting
> > > > the
> > > > > message
> > > > > > diverge even
> > > > > > > >> > further from
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > rest of
> > > > > > > >> > >> >> > the
> > protocol.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > -Jason
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On Wed,
> Feb
> > 15, 2017 at
> > > > 8:52 AM,
> > > > > radai <
> > > > > > > >> > >>
> > radai.rosenblatt@gmail.com>
> > > > > > > >> > >> >> > wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > > I've
> > trimmed the inline
> > > > > contents as this
> > > > > > mail is
> > > > > > > >> > getting too
> > > > > > > >> > >> big for
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > apache
> > mailing list
> > > > software to
> > > > > deliver
> > > > > > :-(
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 1. the
> > important thing
> > > for
> > > > > > interoperability is
> > > > > > > for
> > > > > > > >> > different
> > > > > > > >> > >> >> > "interested
> > > > > > > >> > >> >> > > parties"
> > (plugins, infra
> > > > > layers/wrappers,
> > > > > > > >> user-code)
> > > > > > > >> > to be
> > > > > > > >> > >> able to
> > > > > > > >> > >> >> > stick
> > > > > > > >> > >> >> > > pieces
> of
> > metadata onto
> > > > msgs
> > > > > without
> > > > > > getting in
> > > > > > > >> each
> > > > > > > >> > other's
> > > > > > > >> > >> way. a
> > > > > > > >> > >> >> > common
> > > > > > > >> > >> >> > > key
> > scheme (Strings, as
> > > > of the
> > > > > time of
> > > > > > this
> > > > > > > >> > writing?) is all
> > > > > > > >> > >> thats
> > > > > > > >> > >> >> > required
> > > > > > > >> > >> >> > > for
> that.
> > it is assumed
> > > > that
> > > > > the other end
> > > > > > > >> > interested in any
> > > > > > > >> > >> such
> > > > > > > >> > >> >> > piece of
> > > > > > > >> > >> >> > > metadata
> > knows the
> > > > encoding,
> > > > > and byte[]
> > > > > > provides
> > > > > > > >> for
> > > > > > > >> > the most
> > > > > > > >> > >> >> > flexibility.
> > > > > > > >> > >> >> > > i
> believe
> > this is the
> > > same
> > > > > logic behind
> > > > > > core
> > > > > > > kafka
> > > > > > > >> > being
> > > > > > > >> > >> >> > byte[]/byte[]
> -
> > > > > > > >> > >> >> > > Strings
> > are more
> > > "usable"
> > > > but
> > > > > bytes are
> > > > > > flexible
> > > > > > > >> and
> > > > > > > >> > so were
> > > > > > > >> > >> chosen.
> > > > > > > >> > >> >> > > Also -
> > core kafka doesnt
> > > > even
> > > > > do that
> > > > > > good of a
> > > > > > > >> job
> > > > > > > >> > on
> > > > > > > >> > >> usability of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > payload
> > (example - i
> > > have
> > > > to
> > > > > specify the
> > > > > > nop
> > > > > > > >> byte[]
> > > > > > > >> > "decoders"
> > > > > > > >> > >> >> > explicitly
> > > > > > > >> > >> >> > > in
> conf),
> > and again
> > > > sacrificies
> > > > > usability
> > > > > > for
> > > > > > > the
> > > > > > > >> > sake of
> > > > > > > >> > >> >> > performance
> (no
> > > > > > > >> > >> >> > >
> > convenient single-record
> > > > > processing as
> > > > > > poll is a
> > > > > > > >> > batch, lots of
> > > > > > > >> > >> >> > obscure
> > > > > > > >> > >> >> > > little
> > config details
> > > > exposing
> > > > > internals
> > > > > > of the
> > > > > > > >> > batching
> > > > > > > >> > >> mechanism,
> > > > > > > >> > >> >> > etc)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > this is
> > also why i
> > > really
> > > > > dislike the
> > > > > > idea of a
> > > > > > > >> > "type system"
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > header
> > > > > > > >> > >> >> > > values,
> > it further
> > > > degrades the
> > > > > > usability, adds
> > > > > > > >> > complexity and
> > > > > > > >> > >> will
> > > > > > > >> > >> >> > >
> > eventually get in
> > > > people's way,
> > > > > also, it
> > > > > > would
> > > > > > > be
> > > > > > > >> > the 2nd/3rd
> > > > > > > >> > >> >> > home-group
> > > > > > > >> > >> >> > >
> > serialization mechanism
> > > > in core
> > > > > kafka
> > > > > > (counting
> > > > > > > 2
> > > > > > > >> > iterations
> > > > > > > >> > >> of the
> > > > > > > >> > >> >> > "type
> > > > > > > >> > >> >> > >
> > definition DSL")
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 2. this
> > is an
> > > > implementation
> > > > > detail, and
> > > > > > not
> > > > > > > even
> > > > > > > >> a
> > > > > > > >> > very "user
> > > > > > > >> > >> >> > facing" one?
> > > > > > > >> > >> >> > > to the
> > best of my
> > > > understanding
> > > > > the vote
> > > > > > process
> > > > > > > >> is
> > > > > > > >> > on proposed
> > > > > > > >> > >> >> > >
> > API/behaviour. also -
> > > > since
> > > > > we're willing
> > > > > > to go
> > > > > > > >> with
> > > > > > > >> > strings
> > > > > > > >> > >> just
> > > > > > > >> > >> >> > serialize
> > > > > > > >> > >> >> > > a
> 0-sized
> > header blob
> > > and
> > > > IIUC
> > > > > you dont
> > > > > > need any
> > > > > > > >> > optionals
> > > > > > > >> > >> anymore.
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 3. yes,
> > we can :-)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > On Tue,
> > Feb 14, 2017 at
> > > > 11:56
> > > > > PM, Michael
> > > > > > > Pearce <
> > > > > > > >> > >> >> >
> > Michael.Pearce@ig.com>
> > > > > > > >> > >> >> > > wrote:
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > > Hi
> Jay,
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1)
> > There was some
> > > > initial
> > > > > debate on the
> > > > > > value
> > > > > > > >> > part, as youll
> > > > > > > >> > >> note
> > > > > > > >> > >> >> > String,
> > > > > > > >> > >> >> > > > String
> > headers were
> > > > > discounted early
> > > > > > on. The
> > > > > > > >> > reason for this
> > > > > > > >> > >> is
> > > > > > > >> > >> >> > >
> > flexibility
> > > > > > > >> > >> >> > > > and
> > keeping in line
> > > > with the
> > > > > > flexibility of
> > > > > > > key,
> > > > > > > >> > value of the
> > > > > > > >> > >> >> > message
> > > > > > > >> > >> >> > > > object
> > itself. I don’t
> > > > think
> > > > > it takes
> > > > > > away
> > > > > > > from
> > > > > > > >> an
> > > > > > > >> > ecosystem
> > > > > > > >> > >> as
> > > > > > > >> > >> >> > each
> > > > > > > >> > >> >> > > plugin
> > > > > > > >> > >> >> > > > will
> > care for their
> > > own
> > > > key,
> > > > > this way
> > > > > > ints,
> > > > > > > >> > booleans , exotic
> > > > > > > >> > >> >> > custom
> > > > > > > >> > >> >> > > binary
> > > > > > > >> > >> >> > > > can
> all
> > be catered
> > > for=.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > wanted
> > > > to
> > > > > push for a
> > > > > > typed
> > > > > > > >> value
> > > > > > > >> > interface,
> > > > > > > >> > >> I
> > > > > > > >> > >> >> > wouldn’t
> > > > > > > >> > >> >> > > > want
> > just String
> > > values
> > > > > supported, but
> > > > > > the the
> > > > > > > >> > primatives
> > > > > > > >> > >> plus
> > > > > > > >> > >> >> > string and
> > > > > > > >> > >> >> > > > also
> > still keeping the
> > > > > ability to have a
> > > > > > > binary
> > > > > > > >> > for custom
> > > > > > > >> > >> >> > binaries that
> > > > > > > >> > >> >> > > > some
> > organisations may
> > > > have.
> > > > > > > >> > >> >> > > > i. I
> > have written this
> > > > slight
> > > > > > alternative
> > > > > > > here,
> > > > > > > >> > >> >> > >
> > > https://cwiki.apache.org/
> > > > > > > >> > >> >> > > >
> > > > confluence/display/KAFKA/KIP-
> > > > > > > >> > 82+-+Add+Record+Headers+-+
> Typed
> > > > > > > >> > >> >> > > > ii.
> > Essentially the
> > > > value
> > > > > bytes, has a
> > > > > > leading
> > > > > > > >> > byte overhead.
> > > > > > > >> > >> >> > > > 1.
> > This tells you
> > > what
> > > > type
> > > > > the value
> > > > > > is,
> > > > > > > >> before
> > > > > > > >> > reading
> > > > > > > >> > >> the rest
> > > > > > > >> > >> >> > of the
> > > > > > > >> > >> >> > > > bytes,
> > allowing
> > > > > > serialisation/deserialization
> > > > > > > to
> > > > > > > >> > and from the
> > > > > > > >> > >> >> > primitives,
> > > > > > > >> > >> >> > > > string
> > and byte[].
> > > This
> > > > is
> > > > > akin to some
> > > > > > other
> > > > > > > >> > messaging
> > > > > > > >> > >> systems.
> > > > > > > >> > >> >> > > > 2) We
> > are making it
> > > > optional,
> > > > > so that
> > > > > > for
> > > > > > > those
> > > > > > > >> > not wanting
> > > > > > > >> > >> >> > headers have
> > > > > > > >> > >> >> > > 0
> > > > > > > >> > >> >> > > > bytes
> > overhead (think
> > > > of it
> > > > > as a feature
> > > > > > > flag),
> > > > > > > >> I
> > > > > > > >> > don’t
> > > > > > > >> > >> think this
> > > > > > > >> > >> >> > is
> > > > > > > >> > >> >> > > >
> > complex, especially if
> > > > > comparing to
> > > > > > changes
> > > > > > > >> > proposed in
> > > > > > > >> > >> other kips
> > > > > > > >> > >> >> > like
> > > > > > > >> > >> >> > > >
> kip-98.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > really
> > > > don’t
> > > > > like
> > > > > > this, we
> > > > > > > can
> > > > > > > >> > drop it, but
> > > > > > > >> > >> it
> > > > > > > >> > >> >> > would
> > > > > > > >> > >> >> > > mean
> > > > > > > >> > >> >> > > > buying
> > into 4 bytes
> > > > extra
> > > > > overhead for
> > > > > > users
> > > > > > > who
> > > > > > > >> > do not want
> > > > > > > >> > >> to use
> > > > > > > >> > >> >> > > headers.
> > > > > > > >> > >> >> > > > 3) In
> > the summary yes,
> > > > it is
> > > > > at a higher
> > > > > > > level,
> > > > > > > >> > but I think
> > > > > > > >> > >> this
> > > > > > > >> > >> >> > is well
> > > > > > > >> > >> >> > > >
> > documented in the
> > > > proposed
> > > > > changes
> > > > > > section.
> > > > > > > >> > >> >> > > > a.
> > Added getHeaders
> > > > method to
> > > > > > > Producer/Consumer
> > > > > > > >> > record (that
> > > > > > > >> > >> is it)
> > > > > > > >> > >> >> > > > b.
> > We’ve also detailed
> > > > the
> > > > > new Headers
> > > > > > class
> > > > > > > >> that
> > > > > > > >> > this method
> > > > > > > >> > >> >> > returns
> > > > > > > >> > >> >> > > that
> > > > > > > >> > >> >> > > >
> > encapsulates the
> > > headers
> > > > > protocol and
> > > > > > logic.
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Best,
> > > > > > > >> > >> >> > > > Mike
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > >
> > ==Original questions
> > > > from the
> > > > > vote
> > > > > > thread from
> > > > > > > >> > Jay.==
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Couple
> > of things I
> > > > think we
> > > > > still need
> > > > > > to work
> > > > > > > >> out:
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1.
> I
> > think we agree
> > > > about
> > > > > the key,
> > > > > > but I
> > > > > > > >> think
> > > > > > > >> > we haven't
> > > > > > > >> > >> >> > talked about
> > > > > > > >> > >> >> > > > the
> > value yet. I
> > > > think if
> > > > > our goal
> > > > > > is an
> > > > > > > open
> > > > > > > >> > ecosystem
> > > > > > > >> > >> of these
> > > > > > > >> > >> >> > > header
> > > > > > > >> > >> >> > > >
> > spread across many
> > > > plugins
> > > > > from many
> > > > > > > systems
> > > > > > > >> we
> > > > > > > >> > should
> > > > > > > >> > >> consider
> > > > > > > >> > >> >> > making
> > > > > > > >> > >> >> > > > this
> > > > > > > >> > >> >> > > > a
> > string as well so
> > > > it can
> > > > > be
> > > > > > printed, set
> > > > > > > >> via
> > > > > > > >> > a UI, set
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > config,
> > > > > > > >> > >> >> > > etc.
> > > > > > > >> > >> >> > > >
> > Basically
> > > encouraging
> > > > > pluggable
> > > > > > > serialization
> > > > > > > >> > formats
> > > > > > > >> > >> here will
> > > > > > > >> > >> >> > lead
> > > > > > > >> > >> >> > > to
> > > > > > > >> > >> >> > > > a
> > > > > > > >> > >> >> > > > bit
> > of a tower of
> > > > babel.
> > > > > > > >> > >> >> > > > 2.
> > This proposal
> > > > still
> > > > > includes a
> > > > > > pretty
> > > > > > > big
> > > > > > > >> > change to our
> > > > > > > >> > >> >> > >
> > serialization
> > > > > > > >> > >> >> > > > and
> > protocol
> > > > definition
> > > > > layer.
> > > > > > Essentially
> > > > > > > >> it is
> > > > > > > >> > >> introducing an
> > > > > > > >> > >> >> > > optional
> > > > > > > >> > >> >> > > >
> > type, where the
> > > > format is
> > > > > data
> > > > > > dependent. I
> > > > > > > >> > think this is
> > > > > > > >> > >> >> > actually a
> > > > > > > >> > >> >> > > big
> > > > > > > >> > >> >> > > >
> > change though it
> > > > doesn't
> > > > > seem like
> > > > > > it. It
> > > > > > > >> means
> > > > > > > >> > you can no
> > > > > > > >> > >> >> > longer
> > > > > > > >> > >> >> > > >
> specify
> > > > > > > >> > >> >> > > >
> this
> > type with our
> > > > type
> > > > > definition
> > > > > > DSL, and
> > > > > > > >> > likewise it
> > > > > > > >> > >> requires
> > > > > > > >> > >> >> > > custom
> > > > > > > >> > >> >> > > >
> > handling in client
> > > > libs.
> > > > > This isn't
> > > > > > a huge
> > > > > > > >> > thing, since
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > Record
> > > > > > > >> > >> >> > > >
> > definition is
> > > custom
> > > > > anyway, but I
> > > > > > think
> > > > > > > this
> > > > > > > >> > kind of
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> > > >
> > inconsistency is
> > > very
> > > > > non-desirable
> > > > > > and
> > > > > > > ties
> > > > > > > >> > you to
> > > > > > > >> > >> hand-coding
> > > > > > > >> > >> >> > > things.
> > > > > > > >> > >> >> > > > I
> > > > > > > >> > >> >> > > >
> > think the type
> > > should
> > > > > instead by [Key
> > > > > > > Value]
> > > > > > > >> in
> > > > > > > >> > our BNF,
> > > > > > > >> > >> where
> > > > > > > >> > >> >> > key and
> > > > > > > >> > >> >> > > >
> > value are both
> > > short
> > > > > strings as used
> > > > > > > >> elsewhere.
> > > > > > > >> > This
> > > > > > > >> > >> brings it
> > > > > > > >> > >> >> > in line
> > > > > > > >> > >> >> > > > with
> > > > > > > >> > >> >> > > > the
> > rest of the
> > > > protocol.
> > > > > > > >> > >> >> > > > 3.
> > Could we get
> > > more
> > > > > specific about
> > > > > > the
> > > > > > > exact
> > > > > > > >> > Java API
> > > > > > > >> > >> change to
> > > > > > > >> > >> >> > > >
> > ProducerRecord,
> > > > > ConsumerRecord,
> > > > > > Record,
> > > > > > > etc?
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > -Jay
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The
> information
> > contained in
> > > > this
> > > > > email is
> > > > > > strictly
> > > > > > > >> > confidential and
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > the use of the
> > addressee only,
> > > > unless
> > > > > otherwise
> > > > > > > >> indicated.
> > > > > > > >> > If you
> > > > > > > >> > >> are not
> > > > > > > >> > >> >> > the intended
> > recipient, please
> > > > do not
> > > > > read,
> > > > > > copy, use
> > > > > > > or
> > > > > > > >> > disclose to
> > > > > > > >> > >> others
> > > > > > > >> > >> >> > this message
> or
> > any
> > > attachment.
> > > > > Please also
> > > > > > notify the
> > > > > > > >> > sender by
> > > > > > > >> > >> replying
> > > > > > > >> > >> >> > to this email
> > or by telephone
> > > > > (+44(020 7896
> > > > > > 0011) and
> > > > > > > >> then
> > > > > > > >> > delete
> > > > > > > >> > >> the email
> > > > > > > >> > >> >> > and any copies
> > of it.
> > > Opinions,
> > > > > conclusion
> > > > > > (etc) that
> > > > > > > do
> > > > > > > >> > not relate
> > > > > > > >> > >> to the
> > > > > > > >> > >> >> > official
> > business of this
> > > > company
> > > > > shall be
> > > > > > understood
> > > > > > > as
> > > > > > > >> > neither
> > > > > > > >> > >> given nor
> > > > > > > >> > >> >> > endorsed by
> it.
> > IG is a
> > > trading
> > > > name
> > > > > of IG
> > > > > > Markets
> > > > > > > >> Limited
> > > > > > > >> > (a company
> > > > > > > >> > >> >> > registered in
> > England and
> > > Wales,
> > > > > company number
> > > > > > > >> 04008957)
> > > > > > > >> > and IG
> > > > > > > >> > >> Index
> > > > > > > >> > >> >> > Limited (a
> > company registered
> > > in
> > > > > England and
> > > > > > Wales,
> > > > > > > >> > company number
> > > > > > > >> > >> >> > 01190902).
> > Registered address
> > > at
> > > > > Cannon Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > >> > Dowgate
> > > > > > > >> > >> Hill,
> > > > > > > >> > >> >> > London EC4R
> > 2YA. Both IG
> > > Markets
> > > > > Limited
> > > > > > (register
> > > > > > > >> number
> > > > > > > >> > 195355)
> > > > > > > >> > >> and IG
> > > > > > > >> > >> >> > Index Limited
> > (register number
> > > > > 114059) are
> > > > > > authorised
> > > > > > > >> and
> > > > > > > >> > regulated
> > > > > > > >> > >> by the
> > > > > > > >> > >> >> > Financial
> > Conduct Authority.
> > > > > > > >> > >> >> >
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> --
> > > > > > > >> > >> Nacho - Ignacio
> > Solis -
> > > > isolis@igso.net
> > > > > > > >> > >>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > The information contained in
> > this email is
> > > > strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > indicated. If
> > > > > > you are
> > > > > > > >> not
> > > > > > > >> > the intended recipient, please
> > do not read,
> > > > copy, use
> > > > > or
> > > > > > disclose to
> > > > > > > >> others
> > > > > > > >> > this message or any
> attachment.
> > Please also
> > > > notify
> > > > > the sender by
> > > > > > > >> replying
> > > > > > > >> > to this email or by telephone
> > (+44(020 7896
> > > > 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > >> email
> > > > > > > >> > and any copies of it.
> Opinions,
> > conclusion
> > > > (etc) that
> > > > > do not
> > > > > > relate to
> > > > > > > >> the
> > > > > > > >> > official business of this
> > company shall be
> > > > understood
> > > > > as
> > > > > > neither given
> > > > > > > >> nor
> > > > > > > >> > endorsed by it. IG is a
> trading
> > name of IG
> > > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> > registered in England and
> > Wales, company
> > > number
> > > > > 04008957) and
> > > > > > IG Index
> > > > > > > >> > Limited (a company registered
> > in England and
> > > > Wales,
> > > > > company
> > > > > > number
> > > > > > > >> > 01190902). Registered address
> > at Cannon Bridge
> > > > House,
> > > > > 25
> > > > > > Dowgate Hill,
> > > > > > > >> > London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and
> > > > > > > IG
> > > > > > > >> > Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > >> the
> > > > > > > >> > Financial Conduct Authority.
> > > > > > > >> >
> > > > > > > >> The information contained in
> this
> > email is
> > > > strictly
> > > > > confidential
> > > > > > and for
> > > > > > > >> the use of the addressee only,
> > unless otherwise
> > > > > indicated. If you
> > > > > > are
> > > > > > > not
> > > > > > > >> the intended recipient, please
> do
> > not read,
> > > copy,
> > > > use
> > > > > or disclose
> > > > > > to
> > > > > > > others
> > > > > > > >> this message or any attachment.
> > Please also
> > > > notify the
> > > > > sender by
> > > > > > > replying
> > > > > > > >> to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the
> > > > > > > email
> > > > > > > >> and any copies of it. Opinions,
> > conclusion (etc)
> > > > that
> > > > > do not
> > > > > > relate to
> > > > > > > the
> > > > > > > >> official business of this
> company
> > shall be
> > > > understood
> > > > > as neither
> > > > > > given
> > > > > > > nor
> > > > > > > >> endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> registered in England and Wales,
> > company number
> > > > > 04008957) and IG
> > > > > > Index
> > > > > > > >> Limited (a company registered in
> > England and
> > > > Wales,
> > > > > company number
> > > > > > > >> 01190902). Registered address at
> > Cannon Bridge
> > > > House,
> > > > > 25 Dowgate
> > > > > > Hill,
> > > > > > > >> London EC4R 2YA. Both IG Markets
> > Limited
> > > (register
> > > > > number 195355)
> > > > > > and IG
> > > > > > > >> Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > the
> > > > > > > >> Financial Conduct Authority.
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > The information contained in this
> > email is strictly
> > > > > confidential and
> > > > > > for the use of the addressee only, unless
> > otherwise
> > > > indicated.
> > > > > If you are
> > > > > > not the intended recipient, please do not
> > read, copy, use
> > > > or
> > > > > disclose to
> > > > > > others this message or any attachment.
> > Please also notify
> > > > the
> > > > > sender by
> > > > > > replying to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that
> > > > > do not
> > > > > > relate to the official business of this
> > company shall be
> > > > > understood as
> > > > > > neither given nor endorsed by it. IG is a
> > trading name of
> > > > IG
> > > > > Markets
> > > > > > Limited (a company registered in England
> > and Wales,
> > > company
> > > > > number
> > > > > > 04008957) and IG Index Limited (a company
> > registered in
> > > > England
> > > > > and Wales,
> > > > > > company number 01190902). Registered
> > address at Cannon
> > > > Bridge
> > > > > House, 25
> > > > > > Dowgate Hill, London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and IG Index Limited (register
> > number 114059) are
> > > > > authorised and
> > > > > > regulated by the Financial Conduct
> > Authority.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is
> > strictly
> > > confidential
> > > > and
> > > > > for the use of the addressee only, unless otherwise
> > indicated. If
> > > > you are
> > > > > not the intended recipient, please do not read,
> > copy, use or
> > > > disclose to
> > > > > others this message or any attachment. Please also
> > notify the
> > > sender
> > > > by
> > > > > replying to this email or by telephone (+44(020
> 7896
> > 0011) and then
> > > > delete
> > > > > the email and any copies of it. Opinions,
> conclusion
> > (etc) that do
> > > > not
> > > > > relate to the official business of this company
> > shall be understood
> > > > as
> > > > > neither given nor endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a company registered in England and Wales,
> > company number
> > > > > 04008957) and IG Index Limited (a company
> registered
> > in England and
> > > > Wales,
> > > > > company number 01190902). Registered address at
> > Cannon Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > Limited (register
> > > > number
> > > > > 195355) and IG Index Limited (register number
> > 114059) are
> > > authorised
> > > > and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> > sender by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> > not relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > registered in England and Wales, company number 04008957)
> > and IG Index
> > > > Limited (a company registered in England and Wales,
> > company number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> > for the use of the addressee only, unless otherwise indicated. If you are
> > not the intended recipient, please do not read, copy, use or disclose to
> > others this message or any attachment. Please also notify the sender by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do not
> > relate to the official business of this company shall be understood as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG Index Limited (register number 114059) are authorised and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Re; having a dedicated Headers object. I think it makes sense. Maybe it
could implement Iterable<Header>?
One thing I'm not too sure about is mutability. Both the ProducerRecord and
ConsumerRecord are immutable currently. It would be nice to keep it that
way. Perhaps Headers could itself be immutable and passed through the
constructors?
Just going through how this is, I don’t believe this would provide much
> saving. If issue trying to resolve is around object/garbage creation of
> Map.Entry objects.
It's definitely better because we only allocate the space we will use, but
I assume most of the actual memory cost is in the header data itself which
we can't do much about. Maybe we can start with the simple array
representation and optimize from there after we find it's worthwhile to do
so? If we use the Headers object, we can change this later without breaking
the API.
-Jason
On Sun, Feb 19, 2017 at 2:39 PM, radai <ra...@gmail.com> wrote:
> "(By the way, doesn't it feel a bit odd that we seem to be designing a
> feature which is optimized for people not using?)"
>
> very. this (i claim :-) ) is the result of very intense opposition to the
> usefulness of the feature early on, and not a real design goal
>
> On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Whilst if having an array of a Header[] would mean getting this through,
> > im happy changing this, as already done.
> >
> > Just going through how this is, I don’t believe this would provide much
> > saving. If issue trying to resolve is around object/garbage creation of
> > Map.Entry objects.
> >
> > All you’re doing here is replacing the equiv of a HashMap.Node
> (Map.Entry)
> > which simply holds reference to key and value objects with a custom
> variant.
> >
> >
> >
> >
> >
> > On 19/02/2017, 20:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> > On point 1 & 2 Ive updated KIP to show varints (and removed the bit
> > flag). (on the assumption KIP 98 is getting the agreement the protocol is
> > moving from int32 to varInts as standard)
> >
> > On point 3 ive updated to use an array of Header class, instead of a
> > MultiMap in the Headers class object
> >
> >
> >
> > On 19/02/2017, 20:06, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > On points 1 and 2 I agree.
> >
> > This also affects kip-98, I should expect this resolved before
> > that vote also passes. If it is accepted there (I’m assuming this is
> > getting discussed on that KIP? As you’re driving the move to VarInts), I
> am
> > happy to make this KIP will simply follow suit to whatever is agreed in
> > KIP-98.
> >
> > On 3) Agreed this is a simpler form, as long as no one is
> > expecting hashmap lookup performance ( O(1) ), I still would prefer an
> > encapsulated class, so if we find that holding it as an array in future
> is
> > causing some perf issues, the internals are not exposed to end users,
> > allowing the internal structure to move to a map.
> >
> > Can we compromise on?
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Headers headers;
> > }
> >
> > class Headers {
> > Header[] headers;
> >
> > add(String key, byte[] value)
> > Collection<byte[]> get(String key)
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> >
> >
> > On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> >
> > >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> >
> >
> > This is exactly what I have been trying to get at. The use
> > cases documented
> > here are middleware:
> > https://cwiki.apache.org/confluence/display/KAFKA/A+
> > Case+for+Kafka+Headers.
> > If headers are intended for the application space as well,
> the
> > document
> > should be updated accordingly so that it is an explicit
> design
> > goal and not
> > an unfortunate design byproduct. There may be some
> > disagreement on whether
> > it _should_ be a design goal, but it may not matter much
> since
> > the current
> > interceptor API is probably insufficient for middleware
> > applications (I
> > haven't thought of a way to do this that isn't cumbersome).
> >
> > That aside, the three unresolved points for me are the
> > following:
> >
> > 1. The use of varints. As a one-off for use only with record
> > headers, the
> > case was weak, but if we are using them throughout the
> message
> > format, then
> > we should do so here as well. The additional complexity is
> > minimal and
> > early performance testing fully justifies their use.
> >
> > 2. If varints are used, the case for using attributes to
> > indicate null
> > headers also becomes weak. We only add one additional byte in
> > each message
> > if there are no headers. Whatever the case, let us be
> > consistent with how
> > we treat null keys and values.
> >
> > 3. We have apparently agreed that the broker will validate
> > headers (please
> > add this to the KIP). That being the case, I would prefer to
> > use Kafka's
> > conventional format for arrays. The downside is that it takes
> > more work to
> > skip over the headers on the consumer, though it's unclear if
> > the cost
> > would matter in practice. Some concern was previously
> > expressed about the
> > allocation of maps. An alternative would be to use arrays,
> i.e.
> >
> > class ConsumerRecord<K, V> {
> > K key;
> > V value;
> > Header[] headers;
> > }
> >
> > class Header {
> > String key;
> > byte[] value;
> > }
> >
> > This would work nicely with the conventional array format and
> > my guess is
> > it would obviate the need do any lazy initialization. If we
> > use the map as
> > is currently documented, then it is possible with either
> > representation to
> > slice the headers and initialize them lazily. Either way, it
> > might be a
> > good idea to use a separate object to represent the headers
> in
> > case we need
> > to modify it in the future in some way.
> >
> > (By the way, doesn't it feel a bit odd that we seem to be
> > designing a
> > feature which is optimized for people not using?)
> >
> >
> > If we can resolve these points, then at least you will get my
> > vote.
> >
> > Thanks,
> > Jason
> >
> > On Sun, Feb 19, 2017 at 7:30 AM, radai <
> > radai.rosenblatt@gmail.com> wrote:
> >
> > > headers dont "leak" into application code. they are useful
> > to application
> > > code as well.
> > > IIUC samze currently has headers "in-V" and would just
> > switch over to kafka
> > > headers if they exist.
> > > im sure plenty of other users of kafka would have a use for
> > headers.
> > > im pretty sure use cases exist around shuffling data
> > into/out-of kafka
> > > (kafka connect or equivalent) where metadata from one end
> > could copied over
> > > to the other (S3, for example uses http headers for
> > user-accessible
> > > metadata). it will be kafka client code getting/setting
> > those headers. not
> > > an interceptor.
> > >
> > > On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > For APM single event tracing, need access to the header
> at
> > the point of
> > > > processing on the processing thread.
> > > >
> > > > As such interceptors will not work/be suitable for these,
> > due to the fact
> > > > they act on the ConsumerRecords as a batch, before the
> > handling thread
> > > can
> > > > split out and process per message which is the point
> these
> > tools will
> > > need
> > > > to continue to transaction tracing.
> > > >
> > > > Like wise tools and infra pieces will need access to the
> > message outside
> > > > the interceptor realm.
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 21:26, "Jason Gustafson" <
> > jason@confluent.io> wrote:
> > > >
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > >
> > > >
> > > > So the broker will parse the headers and validate
> > them. Good. The
> > > only
> > > > reason remaining that I can see to leave the headers
> > as a byte array
> > > > is to
> > > > make it easier for the client to skip past them. Are
> > we sure this is
> > > > not
> > > > premature optimization? Are there any performance
> > results which show
> > > > that
> > > > this is worthwhile?
> > > >
> > > > What’s the issue with exposing a method getHeaders on
> > the
> > > > producer/consumer
> > > > > record? It doesn’t break anything. We don’t need
> any
> > special
> > > version.
> > > >
> > > >
> > > > See my previous explanation. What I am trying to
> > resist is the
> > > headers
> > > > becoming a general application-level facility. The
> > primary use case
> > > as
> > > > far
> > > > as I can tell is middleware, which is the use case
> > that the
> > > > interceptors
> > > > are providing.
> > > >
> > > > Current batch consumer model and consumer
> interceptors
> > don’t work
> > > where
> > > > > headers need to be acted on at per message level at
> > time of
> > > > processing,
> > > > > very case is APM (the core one), where the header
> > value is used to
> > > > continue
> > > > > tracing.
> > > >
> > > >
> > > > I still don't understand the point about batching.
> The
> > consumer
> > > > records are
> > > > exposed as a batch in the consumer interceptor, but
> > you can still
> > > > iterate
> > > > them individually. It is no different for the
> consumer
> > API itself.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > Re:
> > > > >
> > > > > “ The point about creation of maps seems
> > orthogonal. We can
> > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the time
> > it is accessed.”
> > > > >
> > > > > That’s exactly what we’re doing the headers are a
> > slice of bytes,
> > > > which
> > > > > then gets parsed later if needed, or can be parsed
> > right away, the
> > > > headers
> > > > > is part of the protocol, so can still be validated
> > if wanted.
> > > > >
> > > > > If you had a header count then you would have to go
> > through each
> > > > header
> > > > > key and value length value to work out how much to
> > skip to get to
> > > > say the
> > > > > value or any future component in the message after
> > the headers.
> > > > Having it
> > > > > as a byte[] with length value makes this a lot
> > easier to skip.
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:37, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > What’s the issue with exposing a method
> > getHeaders on the
> > > > > producer/consumer record? It doesn’t break
> anything.
> > We don’t need
> > > > any
> > > > > special version.
> > > > >
> > > > > Current batch consumer model and consumer
> > interceptors don’t
> > > work
> > > > > where headers need to be acted on at per message
> > level at time of
> > > > > processing, very case is APM (the core one), where
> > the header value
> > > > is used
> > > > > to continue tracing. JMS/HTTP etc all expose these,
> > without issues.
> > > > I would
> > > > > NOT want to lock this down to only be usable
> > accessible via
> > > > interceptors,
> > > > > as we’d fail on one of the main goals.
> > > > >
> > > > > Regards
> > > > > Mike
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 17/02/2017, 20:21, "Jason Gustafson" <
> > jason@confluent.io>
> > > > wrote:
> > > > >
> > > > > The point about creation of maps seems
> > orthogonal. We can
> > > > still
> > > > > represent
> > > > > the headers as a slice of bytes until the
> > time it is
> > > > accessed.
> > > > >
> > > > >
> > > > > > Yes exactly we have access to the records
> > thus why the
> > > > header
> > > > > should be
> > > > > > accessible via it and not hidden for only
> > interceptors to
> > > > access.
> > > > >
> > > > >
> > > > > As explained above, the point is to make
> the
> > intended usage
> > > > clear.
> > > > > Applications should continue to rely on the
> > key/value
> > > fields
> > > > to
> > > > > serialize
> > > > > their own headers, and it would be more
> > ideal if we can
> > > avoid
> > > > > leaking
> > > > > third-party headers into applications. This
> > is difficult to
> > > > do
> > > > > with the
> > > > > current interceptors because they share the
> > record objects
> > > > with
> > > > > the common
> > > > > API. What I had in mind is something like
> an
> > extension of
> > > the
> > > > > current
> > > > > interceptors which exposed a different
> > object (e.g.
> > > > > `RecordAndHeaders`).
> > > > > The challenge is for MM-like use cases. Let
> > me see if I can
> > > > come
> > > > > up with a
> > > > > concrete proposal for that problem.
> > > > >
> > > > > -Jason
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Feb 17, 2017 at 11:55 AM, Michael
> > Pearce <
> > > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > > > I am happy to move the definition of the
> > header into the
> > > > message
> > > > > body, but
> > > > > > would cause us not to lazy
> > initialise/parse the headers,
> > > as
> > > > > obviously, we
> > > > > > would have to traverse these reading the
> > message.
> > > > > >
> > > > > > This was actually one of Jay’s requests:
> > > > > >
> > > > > > “ 2. I think we should think about
> > creating the Map
> > > > lazily to
> > > > > avoid
> > > > > > parsing out all the headers into
> > little objects.
> > > > HashMaps
> > > > > themselves
> > > > > > are
> > > > > > kind of expensive and the consumer is
> > very perf
> > > > sensitive so
> > > > > and making
> > > > > > gazillions of hashmaps that may or
> may
> > not get used
> > > is
> > > > > probably a bad
> > > > > > idea.”
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > >
> > > > > > Yes exactly we have access to the
> > records thus why
> > > the
> > > > > header should
> > > > > > be accessible via it and not hidden for
> > only interceptors
> > > > to
> > > > > access.
> > > > > >
> > > > > > Sent using OWA for iPhone
> > > > > > ______________________________
> > __________
> > > > > > From: Magnus Edenhill <
> > magnus@edenhill.se>
> > > > > > Sent: Friday, February 17, 2017
> > 7:34:49 PM
> > > > > > To: dev@kafka.apache.org
> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> > Record Headers
> > > > > >
> > > > > > Big +1 on VarInts.
> > > > > > CPUs are fast, memory is slow.
> > > > > >
> > > > > > I agree with Jason that we'll want to
> > continue
> > > > verifying
> > > > > messages,
> > > > > > including their headers, so while I
> > appreciate the
> > > > idea of
> > > > > the opaque
> > > > > > header blob it won't be useful in
> > practice.
> > > > > >
> > > > > > /Magnus
> > > > > >
> > > > > > 2017-02-17 10:41 GMT-08:00 Jason
> > Gustafson <
> > > > > jason@confluent.io>:
> > > > > >
> > > > > > > Sorry, my mistake. The consumer
> > interceptor is per
> > > > batch,
> > > > > though I'm
> > > > > > not
> > > > > > > sure that's an actual limitation
> > since you still
> > > have
> > > > > access to the
> > > > > > > individual records.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 10:39 AM,
> > Jason Gustafson <
> > > > > > jason@confluent.io>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Re headers as byte array and
> > future use by
> > > broker.
> > > > This
> > > > > doesn't
> > > > > > take away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >
> > > > > > > >
> > > > > > > > Yeah, I didn't say it was
> > difficult, only
> > > awkward.
> > > > You
> > > > > wouldn't
> > > > > > write the
> > > > > > > > schema that way if you were
> > planning to use it on
> > > > the
> > > > > brokers from
> > > > > > the
> > > > > > > > beginning. Note also that one of
> > the benefits of
> > > > letting
> > > > > the broker
> > > > > > > > understand headers is that it can
> > validate that
> > > > they are
> > > > > properly
> > > > > > > > formatted. If cost is the only
> > concern, we should
> > > > > confirm its
> > > > > > impact
> > > > > > > > through performance testing.
> > > > > > > >
> > > > > > > > One of the key use cases requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >
> > > > > > > >
> > > > > > > > I'm not sure I understand the
> > point about
> > > batching.
> > > > > Interceptors
> > > > > > are
> > > > > > > > applied per-message, right?
> > > > > > > >
> > > > > > > > My intent on interceptors is to
> > keep the usage of
> > > > headers
> > > > > > well-defined so
> > > > > > > > that they don't start leaking
> > unnecessarily into
> > > > > applications. My
> > > > > > guess
> > > > > > > is
> > > > > > > > that it's probably inevitable,
> but
> > isolating it
> > > in
> > > > the
> > > > > > interceptors would
> > > > > > > > at least give people a second
> > thought before
> > > > deciding to
> > > > > use it.
> > > > > > The main
> > > > > > > > challenge in my mind is figuring
> > out how an MM
> > > use
> > > > case
> > > > > would
> > > > > > work. It
> > > > > > > > would be more cumbersome to
> > replicate headers
> > > > through an
> > > > > > interceptor,
> > > > > > > > though arguably MM should be
> > working at a lower
> > > > level
> > > > > anyway.
> > > > > > > >
> > > > > > > > -Jason
> > > > > > > >
> > > > > > > > On Fri, Feb 17, 2017 at 10:16 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Re headers available on the
> > record va
> > > > interceptors only
> > > > > > > >>
> > > > > > > >> One of the key use cases
> requires
> > access on
> > > > consume at
> > > > > per
> > > > > > event/message
> > > > > > > >> level at the point that message
> > is being
> > > > processed, as
> > > > > such the
> > > > > > batch
> > > > > > > >> interceptors and batch consume
> > api isn't
> > > > suitable. It
> > > > > needs to be
> > > > > > at the
> > > > > > > >> record level.
> > > > > > > >>
> > > > > > > >> This anyhow is similar to
> > jms/http/amqp where
> > > > headers
> > > > > are
> > > > > > available to
> > > > > > > >> consuming applications.
> > > > > > > >>
> > > > > > > >> Re headers as byte array and
> > future use by
> > > > broker. This
> > > > > doesn't
> > > > > > take
> > > > > > > away
> > > > > > > >> from that at all. Nor makes it
> > difficult at all
> > > > in my
> > > > > opinion.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Sent using OWA for iPhone
> > > > > > > >> ______________________________
> > __________
> > > > > > > >> From: Jason Gustafson <
> > jason@confluent.io>
> > > > > > > >> Sent: Friday, February 17, 2017
> > 5:55:42 PM
> > > > > > > >> To: dev@kafka.apache.org
> > > > > > > >> Subject: Re: [DISCUSS] KIP-82 -
> > Add Record
> > > Headers
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Yes, that is what we are
> > proposing, so using
> > > > varints
> > > > > for headers
> > > > > > would
> > > > > > > be
> > > > > > > >> consistent with the rest of the
> > message. We have
> > > > used
> > > > > static
> > > > > > helper
> > > > > > > >> methods
> > > > > > > >> in our prototype implementation.
> > > > > > > >>
> > > > > > > >> The cost of parsing, we want to
> > parse/interpret
> > > > the
> > > > > headers
> > > > > > lazily (this
> > > > > > > >> is
> > > > > > > >> > a key point brought up earlier
> > in discussions)
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> I'm a bit skeptical of this. Has
> > anyone done the
> > > > > performance
> > > > > > testing? I
> > > > > > > >> can
> > > > > > > >> probably implement it and test
> it
> > if no one else
> > > > has. I
> > > > > was also
> > > > > > under
> > > > > > > the
> > > > > > > >> impression that there may be use
> > cases down the
> > > > road
> > > > > where the
> > > > > > broker
> > > > > > > >> would
> > > > > > > >> need to interpret headers. That
> > wouldn't be off
> > > > the
> > > > > table in the
> > > > > > future
> > > > > > > if
> > > > > > > >> it's represented as bytes, but
> it
> > would be quite
> > > > a bit
> > > > > more
> > > > > > awkward,
> > > > > > > >> right?
> > > > > > > >>
> > > > > > > >> By the way, one question I have
> > been wondering
> > > > about. My
> > > > > > understanding
> > > > > > > is
> > > > > > > >> that headers are primarily for
> > use cases where a
> > > > > third-party
> > > > > > components
> > > > > > > >> wants to enrich messages without
> > needing to
> > > > understand
> > > > > or modify
> > > > > > the
> > > > > > > >> schema
> > > > > > > >> of the message key and value.
> For
> > the
> > > applications
> > > > > which directly
> > > > > > > produce
> > > > > > > >> and consume the messages and
> > control the
> > > key/value
> > > > > schema
> > > > > > directly, it
> > > > > > > >> seems we would rather have them
> > implement
> > > headers
> > > > > directly in
> > > > > > their own
> > > > > > > >> schema. Supposing for the sake
> of
> > argument that
> > > > it was
> > > > > possible,
> > > > > > my
> > > > > > > >> question is whether it be
> > sufficient to expose
> > > the
> > > > > headers in the
> > > > > > > >> interceptor API and not in the
> > common API?
> > > > > > > >>
> > > > > > > >> -Jason
> > > > > > > >>
> > > > > > > >> On Fri, Feb 17, 2017 at 3:26 AM,
> > Michael Pearce
> > > <
> > > > > > Michael.Pearce@ig.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > On the point of varInts
> > > > > > > >> >
> > > > > > > >> > Would you be proposing in
> > KIP-98 to convert
> > > the
> > > > other
> > > > > message
> > > > > > int’s
> > > > > > > (key
> > > > > > > >> > length, value length) also to
> > varint to keep
> > > it
> > > > > uniform.
> > > > > > > >> > Also I assume there will be a
> > static or helper
> > > > method
> > > > > made to
> > > > > > > write/read
> > > > > > > >> > these in the client and
> server.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 11:22, "Michael
> > Pearce" <
> > > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > On the point re: headers
> in
> > the message
> > > > protocol
> > > > > being a
> > > > > > byte
> > > > > > > array
> > > > > > > >> > and not a count of elements
> > followed by the
> > > > elements.
> > > > > Again
> > > > > > this was
> > > > > > > >> > discussed/argued previously.
> > > > > > > >> >
> > > > > > > >> > It was agreed on for a few
> > reasons some of
> > > > which
> > > > > you have
> > > > > > > obviously
> > > > > > > >> > picked up on:
> > > > > > > >> >
> > > > > > > >> > Broker is able to pass it
> > through opaquely
> > > > > > > >> > The cost of parsing, we
> > want to
> > > > parse/interpret
> > > > > the headers
> > > > > > lazily
> > > > > > > >> > (this is a key point brought
> up
> > earlier in
> > > > > discussions)
> > > > > > > >> > Headers can be copied from
> > consumer record
> > > > to
> > > > > producer
> > > > > > record (aka
> > > > > > > >> > mirror makers etc) without
> > parsing if no
> > > > changes are
> > > > > being made
> > > > > > or
> > > > > > > being
> > > > > > > >> > looked at.
> > > > > > > >> > Keeps the broker agnostic
> > to the format
> > > > > > > >> > You need an int32 either
> > for the byte size
> > > > of the
> > > > > headers,
> > > > > > or for
> > > > > > > >> the
> > > > > > > >> > count of elements, so
> overheads
> > are the same,
> > > > but
> > > > > with going
> > > > > > with an
> > > > > > > >> opaque
> > > > > > > >> > byte array has the above
> > advantages.
> > > > > > > >> >
> > > > > > > >> > Cheers
> > > > > > > >> > Mike
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On 17/02/2017, 02:50,
> > "Jason Gustafson" <
> > > > > jason@confluent.io
> > > > > > >
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > Sorry, should have
> > noted that the
> > > > performance
> > > > > testing
> > > > > > was done
> > > > > > > >> > using the
> > > > > > > >> > producer performance
> > tool shipped with
> > > > Kafka.
> > > > > > > >> >
> > > > > > > >> > -Jason
> > > > > > > >> >
> > > > > > > >> > On Thu, Feb 16, 2017
> at
> > 6:44 PM, Jason
> > > > > Gustafson <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hey Nacho,
> > > > > > > >> > >
> > > > > > > >> > > I've compared
> > performance of our
> > > > KIP-98
> > > > > > implementation with
> > > > > > > >> and
> > > > > > > >> > without
> > > > > > > >> > > varints. For
> messages
> > around 128
> > > > bytes, we
> > > > > see an
> > > > > > increase
> > > > > > > in
> > > > > > > >> > throughput of
> > > > > > > >> > > about 30% using the
> > default
> > > > configuration
> > > > > settings.
> > > > > > At 256
> > > > > > > >> > bytes, the
> > > > > > > >> > > increase is around
> > 16%. Obviously
> > > the
> > > > > performance
> > > > > > converge
> > > > > > > as
> > > > > > > >> > messages get
> > > > > > > >> > > larger, but it seems
> > well worth the
> > > > cost.
> > > > > Note that
> > > > > > we are
> > > > > > > >> also
> > > > > > > >> > seeing a
> > > > > > > >> > > substantial
> > performance increase
> > > > against
> > > > > trunk
> > > > > > primarily
> > > > > > > >> because
> > > > > > > >> > of the
> > > > > > > >> > > much more efficient
> > packing that
> > > > varints
> > > > > provide us.
> > > > > > > Anything
> > > > > > > >> > adding to
> > > > > > > >> > > message overhead,
> > such as record
> > > > headers,
> > > > > would only
> > > > > > > increase
> > > > > > > >> > the relative
> > > > > > > >> > > difference. (Of
> > course take these
> > > > numbers
> > > > > with a
> > > > > > grain of
> > > > > > > salt
> > > > > > > >> > since I have
> > > > > > > >> > > only used the
> default
> > settings with
> > > > both
> > > > > the producer
> > > > > > and
> > > > > > > >> broker
> > > > > > > >> > on my
> > > > > > > >> > > local machine. We
> > intend to provide
> > > > more
> > > > > extensive
> > > > > > > performance
> > > > > > > >> > details as
> > > > > > > >> > > part of the work for
> > KIP-98.)
> > > > > > > >> > >
> > > > > > > >> > > The implementation
> we
> > are using is
> > > > from
> > > > > protobuf (
> > > > > > > >> > >
> > https://developers.google.com/
> > > > > > > protocol-buffers/docs/encoding
> > > > > > > >> ),
> > > > > > > >> > which is
> > > > > > > >> > > also used in HBase.
> > It is trivial to
> > > > > implement and as
> > > > > > far
> > > > > > > as I
> > > > > > > >> > know doesn't
> > > > > > > >> > > suffer from the
> > aliasing problem you
> > > > are
> > > > > describing. I
> > > > > > > checked
> > > > > > > >> > with Magnus
> > > > > > > >> > > (the author of
> > librdkafka) and he
> > > > agreed
> > > > > that the
> > > > > > savings
> > > > > > > >> seemed
> > > > > > > >> > worth the
> > > > > > > >> > > cost of
> > implementation.
> > > > > > > >> > >
> > > > > > > >> > > -Jason
> > > > > > > >> > >
> > > > > > > >> > > On Thu, Feb 16, 2017
> > at 4:32 PM,
> > > > Ignacio
> > > > > Solis <
> > > > > > > >> isolis@igso.net>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> -VarInts
> > > > > > > >> > >>
> > > > > > > >> > >> I'm one of the
> > people (if not the
> > > > most)
> > > > > opposed to
> > > > > > VarInts.
> > > > > > > >> > VarInts
> > > > > > > >> > >> have a place, but
> > this is not it.
> > > > (We
> > > > > had a large
> > > > > > > >> discussion
> > > > > > > >> > about
> > > > > > > >> > >> them at the
> > beginning of KIP-82
> > > time)
> > > > > > > >> > >>
> > > > > > > >> > >> If anybody has real
> > life
> > > performance
> > > > > numbers of
> > > > > > VarInts
> > > > > > > >> > improving
> > > > > > > >> > >> things or
> > significantly reducing
> > > > resources
> > > > > I would
> > > > > > like to
> > > > > > > >> know
> > > > > > > >> > what
> > > > > > > >> > >> that case may be.
> > Yes, you can save
> > > > some
> > > > > bytes here
> > > > > > and
> > > > > > > >> there,
> > > > > > > >> > but
> > > > > > > >> > >> this is probably
> > insignificant to
> > > the
> > > > > overall system
> > > > > > > behavior
> > > > > > > >> > and
> > > > > > > >> > >> storage
> > requirements. -- I say
> > > this
> > > > with
> > > > > respect to
> > > > > > using
> > > > > > > >> > VarInts in
> > > > > > > >> > >> the protocol
> itself,
> > not as part of
> > > > the
> > > > > data.
> > > > > > > >> > >>
> > > > > > > >> > >> VarInts require you
> > to parse the
> > > Int
> > > > > before using it
> > > > > > and
> > > > > > > >> > depending on
> > > > > > > >> > >> the encoding they
> > can suffer from
> > > > aliasing
> > > > > (multiple
> > > > > > > >> > representations
> > > > > > > >> > >> for the same
> value).
> > > > > > > >> > >>
> > > > > > > >> > >> Why add complexity?
> > > > > > > >> > >>
> > > > > > > >> > >> Nacho
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> On Thu, Feb 16,
> 2017
> > at 10:29 AM,
> > > > Colin
> > > > > McCabe <
> > > > > > > >> > cmccabe@apache.org>
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> > +1 for varints
> > here-- it would
> > > save
> > > > > quite a bit of
> > > > > > space.
> > > > > > > >> > They are
> > > > > > > >> > >> > pretty quick to
> > implement as
> > > well.
> > > > > > > >> > >> >
> > > > > > > >> > >> > I think it makes
> > sense for values
> > > > to be
> > > > > byte
> > > > > > arrays.
> > > > > > > Users
> > > > > > > >> > might want
> > > > > > > >> > >> > to attach
> > arbitrary payloads;
> > > they
> > > > > shouldn't be
> > > > > > forced to
> > > > > > > >> > serialize
> > > > > > > >> > >> > everything to
> Java
> > strings.
> > > > > > > >> > >> >
> > > > > > > >> > >> > best,
> > > > > > > >> > >> > Colin
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Thu, Feb 16,
> > 2017, at 09:52,
> > > > Jason
> > > > > Gustafson
> > > > > > wrote:
> > > > > > > >> > >> >> Hey Michael,
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Hmm, I guess the
> > point of
> > > > representing
> > > > > it as
> > > > > > bytes is to
> > > > > > > >> > allow the
> > > > > > > >> > >> broker
> > > > > > > >> > >> >> to pass it
> > through opaquely? Is
> > > > the
> > > > > cost of
> > > > > > parsing
> > > > > > > them a
> > > > > > > >> > concern, or
> > > > > > > >> > >> >> are
> > > > > > > >> > >> >> we simply trying
> > to ensure that
> > > > the
> > > > > broker stays
> > > > > > > agnostic
> > > > > > > >> to
> > > > > > > >> > the
> > > > > > > >> > >> format?
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On varints, I
> > think adding
> > > > support for
> > > > > them makes
> > > > > > less
> > > > > > > >> sense
> > > > > > > >> > for an
> > > > > > > >> > >> >> isolated use
> > case, but as part
> > > of
> > > > a
> > > > > more holistic
> > > > > > change
> > > > > > > >> > (such as what
> > > > > > > >> > >> we
> > > > > > > >> > >> >> have proposed in
> > KIP-98), I
> > > think
> > > > they
> > > > > are
> > > > > > justifiable.
> > > > > > > If
> > > > > > > >> > we add them,
> > > > > > > >> > >> >> then the need to
> > use attributes
> > > > becomes
> > > > > quite a
> > > > > > bit
> > > > > > > >> weaker,
> > > > > > > >> > right? The
> > > > > > > >> > >> >> other thing I
> > find slightly odd
> > > > is the
> > > > > fact that
> > > > > > null
> > > > > > > >> > headers has no
> > > > > > > >> > >> >> actual
> > > > > > > >> > >> >> semantic meaning
> > for the message
> > > > > (unlike null
> > > > > > keys and
> > > > > > > >> > values). It is
> > > > > > > >> > >> >> just
> > > > > > > >> > >> >> a space
> > optimization. It seems a
> > > > bit
> > > > > better to
> > > > > > always
> > > > > > > use
> > > > > > > >> > size 0 to
> > > > > > > >> > >> >> indicate having
> > no headers.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> Overall, the
> main
> > point is
> > > > ensuring
> > > > > that the
> > > > > > message
> > > > > > > >> schema
> > > > > > > >> > remains
> > > > > > > >> > >> >> consistent,
> > either within the
> > > > larger
> > > > > protocol, or
> > > > > > at a
> > > > > > > >> > minimum within
> > > > > > > >> > >> the
> > > > > > > >> > >> >> message itself.
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> -Jason
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> On Thu, Feb 16,
> > 2017 at 6:39 AM,
> > > > > Michael Pearce <
> > > > > > > >> > Michael.Pearce@ig.com
> > > > > > > >> > >> >
> > > > > > > >> > >> >> wrote:
> > > > > > > >> > >> >>
> > > > > > > >> > >> >> > Hi Jason,
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 1) in
> > the message
> > > > protocol
> > > > > the headers
> > > > > > are
> > > > > > > >> simply
> > > > > > > >> > a byte
> > > > > > > >> > >> array,
> > > > > > > >> > >> >> > as like the
> key
> > or value, this
> > > > is to
> > > > > clearly
> > > > > > demarcate
> > > > > > > >> the
> > > > > > > >> > header in
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > core message.
> > Then the header
> > > > byte
> > > > > array in the
> > > > > > core
> > > > > > > >> > message is an
> > > > > > > >> > >> array of
> > > > > > > >> > >> >> > key, value
> > pairs. This is what
> > > > it is
> > > > > denoting.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Then this
> would
> > be I guess in
> > > > the
> > > > > given
> > > > > > notation:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [KeyLength, Key,
> > > > > ValueLength, Value]
> > > > > > > >> > >> >> > KeyLength
> > => int32
> > > > > <-----------------NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised key
> > value
> > > > > > > >> > >> >> > Key =>
> bytes
> > > > > <---------------------- NEW
> > > > > > > serialised
> > > > > > > >> > string (UTF8)
> > > > > > > >> > >> >> > bytes of the
> > header key
> > > > > > > >> > >> >> >
> ValueLength
> > => int32
> > > > > <-------------- NEW
> > > > > > size of
> > > > > > > the
> > > > > > > >> > byte[] of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > serialised
> > header value
> > > > > > > >> > >> >> > Value =>
> > bytes
> > > > > <-------------------- NEW
> > > > > > > serialised
> > > > > > > >> > form of the
> > > > > > > >> > >> header
> > > > > > > >> > >> >> > value
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The key length
> > and value
> > > length
> > > > is
> > > > > matching the
> > > > > > way
> > > > > > > the
> > > > > > > >> > protocol is
> > > > > > > >> > >> >> > defined in the
> > core message
> > > > currently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 2)
> > > > > > > >> > >> >> > Var sized
> ints,
> > this was
> > > > discussed
> > > > > much earlier
> > > > > > on, in
> > > > > > > >> > fact I had
> > > > > > > >> > >> >> > suggested it
> > myself (with
> > > Hadoop
> > > > > references),
> > > > > > the
> > > > > > > >> > complexity of this
> > > > > > > >> > >> >> > compared to
> > having a simpler
> > > > protocol
> > > > > was
> > > > > > argued and
> > > > > > > >> > agreed it
> > > > > > > >> > >> wasn’t worth
> > > > > > > >> > >> >> > the complexity
> > as all other
> > > > clients
> > > > > in other
> > > > > > languages
> > > > > > > >> > would need to
> > > > > > > >> > >> ensure
> > > > > > > >> > >> >> > theyre using
> > the right var
> > > size
> > > > > algorithm, as
> > > > > > there
> > > > > > > is a
> > > > > > > >> > few.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On point 3)
> > > > > > > >> > >> >> > We did the
> > attributes,
> > > optional
> > > > > approach as
> > > > > > originally
> > > > > > > >> > there was
> > > > > > > >> > >> marked
> > > > > > > >> > >> >> > concern that
> > headers would
> > > > cause a
> > > > > message size
> > > > > > > overhead
> > > > > > > >> > for others,
> > > > > > > >> > >> who
> > > > > > > >> > >> >> > don’t want
> > them. As such this
> > > > is the
> > > > > clean
> > > > > > solution to
> > > > > > > >> > achieve that.
> > > > > > > >> > >> If
> > > > > > > >> > >> >> > that no longer
> > holds, and we
> > > > don’t
> > > > > care that we
> > > > > > add
> > > > > > > >> 4bytes
> > > > > > > >> > overhead,
> > > > > > > >> > >> then
> > > > > > > >> > >> >> > im happy to
> > remove.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I’m personally
> > in favour of
> > > > keeping
> > > > > the message
> > > > > > as
> > > > > > > small
> > > > > > > >> > as possible
> > > > > > > >> > >> so
> > > > > > > >> > >> >> > people don’t
> > get shocks in
> > > perf
> > > > and
> > > > > throughputs
> > > > > > dues
> > > > > > > to
> > > > > > > >> > message size,
> > > > > > > >> > >> >> > unless they
> > actively use the
> > > > feature,
> > > > > as such I
> > > > > > do
> > > > > > > >> prefer
> > > > > > > >> > the
> > > > > > > >> > >> attribute bit
> > > > > > > >> > >> >> > wise feature
> > flag approach
> > > > myself.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On 16/02/2017,
> > 05:40, "Jason
> > > > > Gustafson" <
> > > > > > > >> > jason@confluent.io> wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > We have
> > proposed a few
> > > > > significant changes
> > > > > > to the
> > > > > > > >> > message format
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > KIP-98
> > > > > > > >> > >> >> > which now
> > seems likely to
> > > > pass
> > > > > (perhaps
> > > > > > with some
> > > > > > > >> > iterations on
> > > > > > > >> > >> >> >
> > implementation details).
> > > It
> > > > would
> > > > > be good
> > > > > > to try
> > > > > > > and
> > > > > > > >> > coordinate
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > changes
> > > > > > > >> > >> >> > in both of
> > the proposals
> > > to
> > > > make
> > > > > sure they
> > > > > > are
> > > > > > > >> > consistent and
> > > > > > > >> > >> >> > compatible.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > I think
> > using the
> > > > attributes to
> > > > > indicate
> > > > > > null
> > > > > > > >> headers
> > > > > > > >> > is a
> > > > > > > >> > >> reasonable
> > > > > > > >> > >> >> > approach.
> > We have proposed
> > > > to do
> > > > > the same
> > > > > > thing
> > > > > > > for
> > > > > > > >> > the message
> > > > > > > >> > >> key and
> > > > > > > >> > >> >> > value.
> That
> > said, I
> > > > sympathize
> > > > > with Jay's
> > > > > > > argument.
> > > > > > > >> > Having
> > > > > > > >> > >> multiple
> > > > > > > >> > >> >> > ways to
> > > > > > > >> > >> >> > specify a
> > null value
> > > > increases
> > > > > the overall
> > > > > > > >> complexity
> > > > > > > >> > of the
> > > > > > > >> > >> protocol.
> > > > > > > >> > >> >> > You
> > > > > > > >> > >> >> > can see
> > this just from the
> > > > fact
> > > > > that you
> > > > > > need the
> > > > > > > >> > extra verbiage
> > > > > > > >> > >> in the
> > > > > > > >> > >> >> > protocol
> > specification in
> > > > this
> > > > > KIP and in
> > > > > > KIP-98
> > > > > > > to
> > > > > > > >> > describe the
> > > > > > > >> > >> >> > dependence
> > > > > > > >> > >> >> > between
> the
> > fields and the
> > > > > attributes. It
> > > > > > seems
> > > > > > > >> like a
> > > > > > > >> > slippery
> > > > > > > >> > >> slope
> > > > > > > >> > >> >> > if
> > > > > > > >> > >> >> > you start
> > allowing
> > > different
> > > > > request types
> > > > > > to
> > > > > > > >> > implement the
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> >
> > specification differently.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > You can
> > also argue that
> > > the
> > > > > messages
> > > > > > already are
> > > > > > > and
> > > > > > > >> > are likely
> > > > > > > >> > >> to
> > > > > > > >> > >> >> > remain a
> > > > > > > >> > >> >> > special
> > case. For example,
> > > > there
> > > > > is
> > > > > > currently no
> > > > > > > >> > generality in
> > > > > > > >> > >> how
> > > > > > > >> > >> >> > compressed
> > message sets
> > > are
> > > > > represented
> > > > > > that would
> > > > > > > >> be
> > > > > > > >> > applicable
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > other
> > > > > > > >> > >> >> > request
> > types. Some might
> > > > see this
> > > > > > divergence as
> > > > > > > an
> > > > > > > >> > unfortunate
> > > > > > > >> > >> >> > protocol
> > > > > > > >> > >> >> > deficiency
> > which should be
> > > > fixed;
> > > > > others
> > > > > > might see
> > > > > > > >> it
> > > > > > > >> > as sort of
> > > > > > > >> > >> the
> > > > > > > >> > >> >> >
> > inevitability of needing
> > > to
> > > > > optimize where
> > > > > > it
> > > > > > > counts
> > > > > > > >> > most. I'm
> > > > > > > >> > >> probably
> > > > > > > >> > >> >> > somewhere
> > in between, but
> > > I
> > > > think
> > > > > we
> > > > > > probably all
> > > > > > > >> > share the
> > > > > > > >> > >> intuition
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> > protocol should be
> > > kept
> > > > as
> > > > > consistent as
> > > > > > > >> possible.
> > > > > > > >> > With that
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > mind,
> > > > > > > >> > >> >> > here are a
> > few comments:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 1. One
> > thing I found a
> > > > little odd
> > > > > when
> > > > > > reading the
> > > > > > > >> > current
> > > > > > > >> > >> proposal is
> > > > > > > >> > >> >> > that
> > > > > > > >> > >> >> > the
> headers
> > are both
> > > > represented
> > > > > as an
> > > > > > array of
> > > > > > > >> bytes
> > > > > > > >> > and as an
> > > > > > > >> > >> array
> > > > > > > >> > >> >> > of
> > > > > > > >> > >> >> > key/value
> > pairs. I'd
> > > > probably
> > > > > suggest
> > > > > > something
> > > > > > > like
> > > > > > > >> > this:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > [HeaderKey
> > > > HeaderValue]
> > > > > > > >> > >> >> > HeaderKey
> > => String
> > > > > > > >> > >> >> >
> > HeaderValue => Bytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > An array
> in
> > the Kafka
> > > > protocol is
> > > > > > represented as a
> > > > > > > >> > 4-byte integer
> > > > > > > >> > >> >> > indicating
> > the number of
> > > > elements
> > > > > in the
> > > > > > array
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> >
> > serialization of the
> > > > elements.
> > > > > Unless I'm
> > > > > > > >> > misunderstanding, what
> > > > > > > >> > >> you
> > > > > > > >> > >> >> > have
> > > > > > > >> > >> >> > instead is
> > the total size
> > > > of the
> > > > > headers in
> > > > > > bytes
> > > > > > > >> > followed by the
> > > > > > > >> > >> >> > elements.
> > > > > > > >> > >> >> > I'm not
> > sure I see any
> > > > reason for
> > > > > this
> > > > > > > >> inconsistency.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 2. In
> > KIP-98, we've
> > > > introduced
> > > > > > variable-length
> > > > > > > >> integer
> > > > > > > >> > fields.
> > > > > > > >> > >> >> > Effectively,
> > > > > > > >> > >> >> > we've
> > enriched (or
> > > > "complicated"
> > > > > as Jay
> > > > > > might say
> > > > > > > ;)
> > > > > > > >> > the protocol
> > > > > > > >> > >> >> >
> > specification to include
> > > the
> > > > > following
> > > > > > types:
> > > > > > > >> VarInt,
> > > > > > > >> > VarLong,
> > > > > > > >> > >> >> >
> > UnsignedVarInt and
> > > > > UnsignedVarLong.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Along with
> > these
> > > > primitives, we
> > > > > could
> > > > > > introduce
> > > > > > > the
> > > > > > > >> > following
> > > > > > > >> > >> types:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeArray =>
> > > > NumberOfItems
> > > > > Item1 Item2
> > > > > > .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItems =>
> > > > UnsignedVarInt
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > VarSizeNullableArray =>
> > > > > NumberOfItemsOrNull
> > > > > > Item1
> > > > > > > >> > Item2 .. ItemN
> > > > > > > >> > >> >> >
> > NumberOfItemsOrNull =>
> > > > VarInt
> > > > > (-1 means
> > > > > > null)
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > And
> > similarly for the
> > > > `String`
> > > > > and `Bytes`
> > > > > > types.
> > > > > > > >> > These types
> > > > > > > >> > >> can save
> > > > > > > >> > >> >> > a
> > > > > > > >> > >> >> >
> > considerable amount of
> > > > space in
> > > > > this
> > > > > > proposal
> > > > > > > >> because
> > > > > > > >> > they can
> > > > > > > >> > >> be used
> > > > > > > >> > >> >> > for
> > > > > > > >> > >> >> > both the
> > number of headers
> > > > > included in the
> > > > > > message
> > > > > > > >> and
> > > > > > > >> > the
> > > > > > > >> > >> lengths of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > header
> keys
> > and values. We
> > > > could
> > > > > do this
> > > > > > instead:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Headers =>
> > > > VarSizeArray[HeaderKey
> > > > > > HeaderValue]
> > > > > > > >> > >> >> >
> HeaderKey
> > =>
> > > VarSizeString
> > > > > > > >> > >> >> >
> > HeaderValue =>
> > > > VarSizeBytes
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > Combining
> > the savings from
> > > > the
> > > > > use of
> > > > > > variable
> > > > > > > >> length
> > > > > > > >> > fields, the
> > > > > > > >> > >> >> > benefit
> > > > > > > >> > >> >> > of using
> > the attributes to
> > > > > represent null
> > > > > > seems
> > > > > > > >> pretty
> > > > > > > >> > small.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > 3.
> > Whichever way we go
> > > > (whether
> > > > > we use the
> > > > > > > >> attributes
> > > > > > > >> > or not), we
> > > > > > > >> > >> >> > should at
> > > > > > > >> > >> >> > least be
> > consistent
> > > between
> > > > this
> > > > > KIP and
> > > > > > KIP-98.
> > > > > > > It
> > > > > > > >> > would be very
> > > > > > > >> > >> >> > strange
> > > > > > > >> > >> >> > to have
> two
> > ways to
> > > > represent
> > > > > null values
> > > > > > in the
> > > > > > > >> same
> > > > > > > >> > schema.
> > > > > > > >> > >> Either
> > > > > > > >> > >> >> > way is
> > > > > > > >> > >> >> > OK with
> me.
> > I think some
> > > > > message-level
> > > > > > > optimizations
> > > > > > > >> > are
> > > > > > > >> > >> justifiable,
> > > > > > > >> > >> >> > but
> > > > > > > >> > >> >> > the
> savings
> > here seem
> > > > minimal (a
> > > > > few bytes
> > > > > > per
> > > > > > > >> > message), so
> > > > > > > >> > >> maybe it's
> > > > > > > >> > >> >> > not
> > > > > > > >> > >> >> > worth the
> > cost of letting
> > > > the
> > > > > message
> > > > > > diverge even
> > > > > > > >> > further from
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > rest of
> > > > > > > >> > >> >> > the
> > protocol.
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > -Jason
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > On Wed,
> Feb
> > 15, 2017 at
> > > > 8:52 AM,
> > > > > radai <
> > > > > > > >> > >>
> > radai.rosenblatt@gmail.com>
> > > > > > > >> > >> >> > wrote:
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > > I've
> > trimmed the inline
> > > > > contents as this
> > > > > > mail is
> > > > > > > >> > getting too
> > > > > > > >> > >> big for
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > apache
> > mailing list
> > > > software to
> > > > > deliver
> > > > > > :-(
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 1. the
> > important thing
> > > for
> > > > > > interoperability is
> > > > > > > for
> > > > > > > >> > different
> > > > > > > >> > >> >> > "interested
> > > > > > > >> > >> >> > > parties"
> > (plugins, infra
> > > > > layers/wrappers,
> > > > > > > >> user-code)
> > > > > > > >> > to be
> > > > > > > >> > >> able to
> > > > > > > >> > >> >> > stick
> > > > > > > >> > >> >> > > pieces
> of
> > metadata onto
> > > > msgs
> > > > > without
> > > > > > getting in
> > > > > > > >> each
> > > > > > > >> > other's
> > > > > > > >> > >> way. a
> > > > > > > >> > >> >> > common
> > > > > > > >> > >> >> > > key
> > scheme (Strings, as
> > > > of the
> > > > > time of
> > > > > > this
> > > > > > > >> > writing?) is all
> > > > > > > >> > >> thats
> > > > > > > >> > >> >> > required
> > > > > > > >> > >> >> > > for
> that.
> > it is assumed
> > > > that
> > > > > the other end
> > > > > > > >> > interested in any
> > > > > > > >> > >> such
> > > > > > > >> > >> >> > piece of
> > > > > > > >> > >> >> > > metadata
> > knows the
> > > > encoding,
> > > > > and byte[]
> > > > > > provides
> > > > > > > >> for
> > > > > > > >> > the most
> > > > > > > >> > >> >> > flexibility.
> > > > > > > >> > >> >> > > i
> believe
> > this is the
> > > same
> > > > > logic behind
> > > > > > core
> > > > > > > kafka
> > > > > > > >> > being
> > > > > > > >> > >> >> > byte[]/byte[]
> -
> > > > > > > >> > >> >> > > Strings
> > are more
> > > "usable"
> > > > but
> > > > > bytes are
> > > > > > flexible
> > > > > > > >> and
> > > > > > > >> > so were
> > > > > > > >> > >> chosen.
> > > > > > > >> > >> >> > > Also -
> > core kafka doesnt
> > > > even
> > > > > do that
> > > > > > good of a
> > > > > > > >> job
> > > > > > > >> > on
> > > > > > > >> > >> usability of
> > > > > > > >> > >> >> > the
> > > > > > > >> > >> >> > > payload
> > (example - i
> > > have
> > > > to
> > > > > specify the
> > > > > > nop
> > > > > > > >> byte[]
> > > > > > > >> > "decoders"
> > > > > > > >> > >> >> > explicitly
> > > > > > > >> > >> >> > > in
> conf),
> > and again
> > > > sacrificies
> > > > > usability
> > > > > > for
> > > > > > > the
> > > > > > > >> > sake of
> > > > > > > >> > >> >> > performance
> (no
> > > > > > > >> > >> >> > >
> > convenient single-record
> > > > > processing as
> > > > > > poll is a
> > > > > > > >> > batch, lots of
> > > > > > > >> > >> >> > obscure
> > > > > > > >> > >> >> > > little
> > config details
> > > > exposing
> > > > > internals
> > > > > > of the
> > > > > > > >> > batching
> > > > > > > >> > >> mechanism,
> > > > > > > >> > >> >> > etc)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > this is
> > also why i
> > > really
> > > > > dislike the
> > > > > > idea of a
> > > > > > > >> > "type system"
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > header
> > > > > > > >> > >> >> > > values,
> > it further
> > > > degrades the
> > > > > > usability, adds
> > > > > > > >> > complexity and
> > > > > > > >> > >> will
> > > > > > > >> > >> >> > >
> > eventually get in
> > > > people's way,
> > > > > also, it
> > > > > > would
> > > > > > > be
> > > > > > > >> > the 2nd/3rd
> > > > > > > >> > >> >> > home-group
> > > > > > > >> > >> >> > >
> > serialization mechanism
> > > > in core
> > > > > kafka
> > > > > > (counting
> > > > > > > 2
> > > > > > > >> > iterations
> > > > > > > >> > >> of the
> > > > > > > >> > >> >> > "type
> > > > > > > >> > >> >> > >
> > definition DSL")
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 2. this
> > is an
> > > > implementation
> > > > > detail, and
> > > > > > not
> > > > > > > even
> > > > > > > >> a
> > > > > > > >> > very "user
> > > > > > > >> > >> >> > facing" one?
> > > > > > > >> > >> >> > > to the
> > best of my
> > > > understanding
> > > > > the vote
> > > > > > process
> > > > > > > >> is
> > > > > > > >> > on proposed
> > > > > > > >> > >> >> > >
> > API/behaviour. also -
> > > > since
> > > > > we're willing
> > > > > > to go
> > > > > > > >> with
> > > > > > > >> > strings
> > > > > > > >> > >> just
> > > > > > > >> > >> >> > serialize
> > > > > > > >> > >> >> > > a
> 0-sized
> > header blob
> > > and
> > > > IIUC
> > > > > you dont
> > > > > > need any
> > > > > > > >> > optionals
> > > > > > > >> > >> anymore.
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > 3. yes,
> > we can :-)
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > On Tue,
> > Feb 14, 2017 at
> > > > 11:56
> > > > > PM, Michael
> > > > > > > Pearce <
> > > > > > > >> > >> >> >
> > Michael.Pearce@ig.com>
> > > > > > > >> > >> >> > > wrote:
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> > > > Hi
> Jay,
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1)
> > There was some
> > > > initial
> > > > > debate on the
> > > > > > value
> > > > > > > >> > part, as youll
> > > > > > > >> > >> note
> > > > > > > >> > >> >> > String,
> > > > > > > >> > >> >> > > > String
> > headers were
> > > > > discounted early
> > > > > > on. The
> > > > > > > >> > reason for this
> > > > > > > >> > >> is
> > > > > > > >> > >> >> > >
> > flexibility
> > > > > > > >> > >> >> > > > and
> > keeping in line
> > > > with the
> > > > > > flexibility of
> > > > > > > key,
> > > > > > > >> > value of the
> > > > > > > >> > >> >> > message
> > > > > > > >> > >> >> > > > object
> > itself. I don’t
> > > > think
> > > > > it takes
> > > > > > away
> > > > > > > from
> > > > > > > >> an
> > > > > > > >> > ecosystem
> > > > > > > >> > >> as
> > > > > > > >> > >> >> > each
> > > > > > > >> > >> >> > > plugin
> > > > > > > >> > >> >> > > > will
> > care for their
> > > own
> > > > key,
> > > > > this way
> > > > > > ints,
> > > > > > > >> > booleans , exotic
> > > > > > > >> > >> >> > custom
> > > > > > > >> > >> >> > > binary
> > > > > > > >> > >> >> > > > can
> all
> > be catered
> > > for=.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > wanted
> > > > to
> > > > > push for a
> > > > > > typed
> > > > > > > >> value
> > > > > > > >> > interface,
> > > > > > > >> > >> I
> > > > > > > >> > >> >> > wouldn’t
> > > > > > > >> > >> >> > > > want
> > just String
> > > values
> > > > > supported, but
> > > > > > the the
> > > > > > > >> > primatives
> > > > > > > >> > >> plus
> > > > > > > >> > >> >> > string and
> > > > > > > >> > >> >> > > > also
> > still keeping the
> > > > > ability to have a
> > > > > > > binary
> > > > > > > >> > for custom
> > > > > > > >> > >> >> > binaries that
> > > > > > > >> > >> >> > > > some
> > organisations may
> > > > have.
> > > > > > > >> > >> >> > > > i. I
> > have written this
> > > > slight
> > > > > > alternative
> > > > > > > here,
> > > > > > > >> > >> >> > >
> > > https://cwiki.apache.org/
> > > > > > > >> > >> >> > > >
> > > > confluence/display/KAFKA/KIP-
> > > > > > > >> > 82+-+Add+Record+Headers+-+
> Typed
> > > > > > > >> > >> >> > > > ii.
> > Essentially the
> > > > value
> > > > > bytes, has a
> > > > > > leading
> > > > > > > >> > byte overhead.
> > > > > > > >> > >> >> > > > 1.
> > This tells you
> > > what
> > > > type
> > > > > the value
> > > > > > is,
> > > > > > > >> before
> > > > > > > >> > reading
> > > > > > > >> > >> the rest
> > > > > > > >> > >> >> > of the
> > > > > > > >> > >> >> > > > bytes,
> > allowing
> > > > > > serialisation/deserialization
> > > > > > > to
> > > > > > > >> > and from the
> > > > > > > >> > >> >> > primitives,
> > > > > > > >> > >> >> > > > string
> > and byte[].
> > > This
> > > > is
> > > > > akin to some
> > > > > > other
> > > > > > > >> > messaging
> > > > > > > >> > >> systems.
> > > > > > > >> > >> >> > > > 2) We
> > are making it
> > > > optional,
> > > > > so that
> > > > > > for
> > > > > > > those
> > > > > > > >> > not wanting
> > > > > > > >> > >> >> > headers have
> > > > > > > >> > >> >> > > 0
> > > > > > > >> > >> >> > > > bytes
> > overhead (think
> > > > of it
> > > > > as a feature
> > > > > > > flag),
> > > > > > > >> I
> > > > > > > >> > don’t
> > > > > > > >> > >> think this
> > > > > > > >> > >> >> > is
> > > > > > > >> > >> >> > > >
> > complex, especially if
> > > > > comparing to
> > > > > > changes
> > > > > > > >> > proposed in
> > > > > > > >> > >> other kips
> > > > > > > >> > >> >> > like
> > > > > > > >> > >> >> > > >
> kip-98.
> > > > > > > >> > >> >> > > > a. If
> > you really
> > > really
> > > > don’t
> > > > > like
> > > > > > this, we
> > > > > > > can
> > > > > > > >> > drop it, but
> > > > > > > >> > >> it
> > > > > > > >> > >> >> > would
> > > > > > > >> > >> >> > > mean
> > > > > > > >> > >> >> > > > buying
> > into 4 bytes
> > > > extra
> > > > > overhead for
> > > > > > users
> > > > > > > who
> > > > > > > >> > do not want
> > > > > > > >> > >> to use
> > > > > > > >> > >> >> > > headers.
> > > > > > > >> > >> >> > > > 3) In
> > the summary yes,
> > > > it is
> > > > > at a higher
> > > > > > > level,
> > > > > > > >> > but I think
> > > > > > > >> > >> this
> > > > > > > >> > >> >> > is well
> > > > > > > >> > >> >> > > >
> > documented in the
> > > > proposed
> > > > > changes
> > > > > > section.
> > > > > > > >> > >> >> > > > a.
> > Added getHeaders
> > > > method to
> > > > > > > Producer/Consumer
> > > > > > > >> > record (that
> > > > > > > >> > >> is it)
> > > > > > > >> > >> >> > > > b.
> > We’ve also detailed
> > > > the
> > > > > new Headers
> > > > > > class
> > > > > > > >> that
> > > > > > > >> > this method
> > > > > > > >> > >> >> > returns
> > > > > > > >> > >> >> > > that
> > > > > > > >> > >> >> > > >
> > encapsulates the
> > > headers
> > > > > protocol and
> > > > > > logic.
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Best,
> > > > > > > >> > >> >> > > > Mike
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > >
> > ==Original questions
> > > > from the
> > > > > vote
> > > > > > thread from
> > > > > > > >> > Jay.==
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > Couple
> > of things I
> > > > think we
> > > > > still need
> > > > > > to work
> > > > > > > >> out:
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > 1.
> I
> > think we agree
> > > > about
> > > > > the key,
> > > > > > but I
> > > > > > > >> think
> > > > > > > >> > we haven't
> > > > > > > >> > >> >> > talked about
> > > > > > > >> > >> >> > > > the
> > value yet. I
> > > > think if
> > > > > our goal
> > > > > > is an
> > > > > > > open
> > > > > > > >> > ecosystem
> > > > > > > >> > >> of these
> > > > > > > >> > >> >> > > header
> > > > > > > >> > >> >> > > >
> > spread across many
> > > > plugins
> > > > > from many
> > > > > > > systems
> > > > > > > >> we
> > > > > > > >> > should
> > > > > > > >> > >> consider
> > > > > > > >> > >> >> > making
> > > > > > > >> > >> >> > > > this
> > > > > > > >> > >> >> > > > a
> > string as well so
> > > > it can
> > > > > be
> > > > > > printed, set
> > > > > > > >> via
> > > > > > > >> > a UI, set
> > > > > > > >> > >> in
> > > > > > > >> > >> >> > config,
> > > > > > > >> > >> >> > > etc.
> > > > > > > >> > >> >> > > >
> > Basically
> > > encouraging
> > > > > pluggable
> > > > > > > serialization
> > > > > > > >> > formats
> > > > > > > >> > >> here will
> > > > > > > >> > >> >> > lead
> > > > > > > >> > >> >> > > to
> > > > > > > >> > >> >> > > > a
> > > > > > > >> > >> >> > > > bit
> > of a tower of
> > > > babel.
> > > > > > > >> > >> >> > > > 2.
> > This proposal
> > > > still
> > > > > includes a
> > > > > > pretty
> > > > > > > big
> > > > > > > >> > change to our
> > > > > > > >> > >> >> > >
> > serialization
> > > > > > > >> > >> >> > > > and
> > protocol
> > > > definition
> > > > > layer.
> > > > > > Essentially
> > > > > > > >> it is
> > > > > > > >> > >> introducing an
> > > > > > > >> > >> >> > > optional
> > > > > > > >> > >> >> > > >
> > type, where the
> > > > format is
> > > > > data
> > > > > > dependent. I
> > > > > > > >> > think this is
> > > > > > > >> > >> >> > actually a
> > > > > > > >> > >> >> > > big
> > > > > > > >> > >> >> > > >
> > change though it
> > > > doesn't
> > > > > seem like
> > > > > > it. It
> > > > > > > >> means
> > > > > > > >> > you can no
> > > > > > > >> > >> >> > longer
> > > > > > > >> > >> >> > > >
> specify
> > > > > > > >> > >> >> > > >
> this
> > type with our
> > > > type
> > > > > definition
> > > > > > DSL, and
> > > > > > > >> > likewise it
> > > > > > > >> > >> requires
> > > > > > > >> > >> >> > > custom
> > > > > > > >> > >> >> > > >
> > handling in client
> > > > libs.
> > > > > This isn't
> > > > > > a huge
> > > > > > > >> > thing, since
> > > > > > > >> > >> the
> > > > > > > >> > >> >> > Record
> > > > > > > >> > >> >> > > >
> > definition is
> > > custom
> > > > > anyway, but I
> > > > > > think
> > > > > > > this
> > > > > > > >> > kind of
> > > > > > > >> > >> protocol
> > > > > > > >> > >> >> > > >
> > inconsistency is
> > > very
> > > > > non-desirable
> > > > > > and
> > > > > > > ties
> > > > > > > >> > you to
> > > > > > > >> > >> hand-coding
> > > > > > > >> > >> >> > > things.
> > > > > > > >> > >> >> > > > I
> > > > > > > >> > >> >> > > >
> > think the type
> > > should
> > > > > instead by [Key
> > > > > > > Value]
> > > > > > > >> in
> > > > > > > >> > our BNF,
> > > > > > > >> > >> where
> > > > > > > >> > >> >> > key and
> > > > > > > >> > >> >> > > >
> > value are both
> > > short
> > > > > strings as used
> > > > > > > >> elsewhere.
> > > > > > > >> > This
> > > > > > > >> > >> brings it
> > > > > > > >> > >> >> > in line
> > > > > > > >> > >> >> > > > with
> > > > > > > >> > >> >> > > > the
> > rest of the
> > > > protocol.
> > > > > > > >> > >> >> > > > 3.
> > Could we get
> > > more
> > > > > specific about
> > > > > > the
> > > > > > > exact
> > > > > > > >> > Java API
> > > > > > > >> > >> change to
> > > > > > > >> > >> >> > > >
> > ProducerRecord,
> > > > > ConsumerRecord,
> > > > > > Record,
> > > > > > > etc?
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > > > -Jay
> > > > > > > >> > >> >> > > >
> > > > > > > >> > >> >> > >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> >
> > > > > > > >> > >> >> > The
> information
> > contained in
> > > > this
> > > > > email is
> > > > > > strictly
> > > > > > > >> > confidential and
> > > > > > > >> > >> for
> > > > > > > >> > >> >> > the use of the
> > addressee only,
> > > > unless
> > > > > otherwise
> > > > > > > >> indicated.
> > > > > > > >> > If you
> > > > > > > >> > >> are not
> > > > > > > >> > >> >> > the intended
> > recipient, please
> > > > do not
> > > > > read,
> > > > > > copy, use
> > > > > > > or
> > > > > > > >> > disclose to
> > > > > > > >> > >> others
> > > > > > > >> > >> >> > this message
> or
> > any
> > > attachment.
> > > > > Please also
> > > > > > notify the
> > > > > > > >> > sender by
> > > > > > > >> > >> replying
> > > > > > > >> > >> >> > to this email
> > or by telephone
> > > > > (+44(020 7896
> > > > > > 0011) and
> > > > > > > >> then
> > > > > > > >> > delete
> > > > > > > >> > >> the email
> > > > > > > >> > >> >> > and any copies
> > of it.
> > > Opinions,
> > > > > conclusion
> > > > > > (etc) that
> > > > > > > do
> > > > > > > >> > not relate
> > > > > > > >> > >> to the
> > > > > > > >> > >> >> > official
> > business of this
> > > > company
> > > > > shall be
> > > > > > understood
> > > > > > > as
> > > > > > > >> > neither
> > > > > > > >> > >> given nor
> > > > > > > >> > >> >> > endorsed by
> it.
> > IG is a
> > > trading
> > > > name
> > > > > of IG
> > > > > > Markets
> > > > > > > >> Limited
> > > > > > > >> > (a company
> > > > > > > >> > >> >> > registered in
> > England and
> > > Wales,
> > > > > company number
> > > > > > > >> 04008957)
> > > > > > > >> > and IG
> > > > > > > >> > >> Index
> > > > > > > >> > >> >> > Limited (a
> > company registered
> > > in
> > > > > England and
> > > > > > Wales,
> > > > > > > >> > company number
> > > > > > > >> > >> >> > 01190902).
> > Registered address
> > > at
> > > > > Cannon Bridge
> > > > > > House,
> > > > > > > 25
> > > > > > > >> > Dowgate
> > > > > > > >> > >> Hill,
> > > > > > > >> > >> >> > London EC4R
> > 2YA. Both IG
> > > Markets
> > > > > Limited
> > > > > > (register
> > > > > > > >> number
> > > > > > > >> > 195355)
> > > > > > > >> > >> and IG
> > > > > > > >> > >> >> > Index Limited
> > (register number
> > > > > 114059) are
> > > > > > authorised
> > > > > > > >> and
> > > > > > > >> > regulated
> > > > > > > >> > >> by the
> > > > > > > >> > >> >> > Financial
> > Conduct Authority.
> > > > > > > >> > >> >> >
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >> --
> > > > > > > >> > >> Nacho - Ignacio
> > Solis -
> > > > isolis@igso.net
> > > > > > > >> > >>
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > The information contained in
> > this email is
> > > > strictly
> > > > > > confidential and
> > > > > > > for
> > > > > > > >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > indicated. If
> > > > > > you are
> > > > > > > >> not
> > > > > > > >> > the intended recipient, please
> > do not read,
> > > > copy, use
> > > > > or
> > > > > > disclose to
> > > > > > > >> others
> > > > > > > >> > this message or any
> attachment.
> > Please also
> > > > notify
> > > > > the sender by
> > > > > > > >> replying
> > > > > > > >> > to this email or by telephone
> > (+44(020 7896
> > > > 0011) and
> > > > > then
> > > > > > delete the
> > > > > > > >> email
> > > > > > > >> > and any copies of it.
> Opinions,
> > conclusion
> > > > (etc) that
> > > > > do not
> > > > > > relate to
> > > > > > > >> the
> > > > > > > >> > official business of this
> > company shall be
> > > > understood
> > > > > as
> > > > > > neither given
> > > > > > > >> nor
> > > > > > > >> > endorsed by it. IG is a
> trading
> > name of IG
> > > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> > registered in England and
> > Wales, company
> > > number
> > > > > 04008957) and
> > > > > > IG Index
> > > > > > > >> > Limited (a company registered
> > in England and
> > > > Wales,
> > > > > company
> > > > > > number
> > > > > > > >> > 01190902). Registered address
> > at Cannon Bridge
> > > > House,
> > > > > 25
> > > > > > Dowgate Hill,
> > > > > > > >> > London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and
> > > > > > > IG
> > > > > > > >> > Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > >> the
> > > > > > > >> > Financial Conduct Authority.
> > > > > > > >> >
> > > > > > > >> The information contained in
> this
> > email is
> > > > strictly
> > > > > confidential
> > > > > > and for
> > > > > > > >> the use of the addressee only,
> > unless otherwise
> > > > > indicated. If you
> > > > > > are
> > > > > > > not
> > > > > > > >> the intended recipient, please
> do
> > not read,
> > > copy,
> > > > use
> > > > > or disclose
> > > > > > to
> > > > > > > others
> > > > > > > >> this message or any attachment.
> > Please also
> > > > notify the
> > > > > sender by
> > > > > > > replying
> > > > > > > >> to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the
> > > > > > > email
> > > > > > > >> and any copies of it. Opinions,
> > conclusion (etc)
> > > > that
> > > > > do not
> > > > > > relate to
> > > > > > > the
> > > > > > > >> official business of this
> company
> > shall be
> > > > understood
> > > > > as neither
> > > > > > given
> > > > > > > nor
> > > > > > > >> endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a
> > > > > > company
> > > > > > > >> registered in England and Wales,
> > company number
> > > > > 04008957) and IG
> > > > > > Index
> > > > > > > >> Limited (a company registered in
> > England and
> > > > Wales,
> > > > > company number
> > > > > > > >> 01190902). Registered address at
> > Cannon Bridge
> > > > House,
> > > > > 25 Dowgate
> > > > > > Hill,
> > > > > > > >> London EC4R 2YA. Both IG Markets
> > Limited
> > > (register
> > > > > number 195355)
> > > > > > and IG
> > > > > > > >> Index Limited (register number
> > 114059) are
> > > > authorised
> > > > > and
> > > > > > regulated by
> > > > > > > the
> > > > > > > >> Financial Conduct Authority.
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > The information contained in this
> > email is strictly
> > > > > confidential and
> > > > > > for the use of the addressee only, unless
> > otherwise
> > > > indicated.
> > > > > If you are
> > > > > > not the intended recipient, please do not
> > read, copy, use
> > > > or
> > > > > disclose to
> > > > > > others this message or any attachment.
> > Please also notify
> > > > the
> > > > > sender by
> > > > > > replying to this email or by telephone
> > (+44(020 7896
> > > 0011)
> > > > and
> > > > > then delete
> > > > > > the email and any copies of it. Opinions,
> > conclusion
> > > (etc)
> > > > that
> > > > > do not
> > > > > > relate to the official business of this
> > company shall be
> > > > > understood as
> > > > > > neither given nor endorsed by it. IG is a
> > trading name of
> > > > IG
> > > > > Markets
> > > > > > Limited (a company registered in England
> > and Wales,
> > > company
> > > > > number
> > > > > > 04008957) and IG Index Limited (a company
> > registered in
> > > > England
> > > > > and Wales,
> > > > > > company number 01190902). Registered
> > address at Cannon
> > > > Bridge
> > > > > House, 25
> > > > > > Dowgate Hill, London EC4R 2YA. Both IG
> > Markets Limited
> > > > (register
> > > > > number
> > > > > > 195355) and IG Index Limited (register
> > number 114059) are
> > > > > authorised and
> > > > > > regulated by the Financial Conduct
> > Authority.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > The information contained in this email is
> > strictly
> > > confidential
> > > > and
> > > > > for the use of the addressee only, unless otherwise
> > indicated. If
> > > > you are
> > > > > not the intended recipient, please do not read,
> > copy, use or
> > > > disclose to
> > > > > others this message or any attachment. Please also
> > notify the
> > > sender
> > > > by
> > > > > replying to this email or by telephone (+44(020
> 7896
> > 0011) and then
> > > > delete
> > > > > the email and any copies of it. Opinions,
> conclusion
> > (etc) that do
> > > > not
> > > > > relate to the official business of this company
> > shall be understood
> > > > as
> > > > > neither given nor endorsed by it. IG is a trading
> > name of IG
> > > Markets
> > > > > Limited (a company registered in England and Wales,
> > company number
> > > > > 04008957) and IG Index Limited (a company
> registered
> > in England and
> > > > Wales,
> > > > > company number 01190902). Registered address at
> > Cannon Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> > Limited (register
> > > > number
> > > > > 195355) and IG Index Limited (register number
> > 114059) are
> > > authorised
> > > > and
> > > > > regulated by the Financial Conduct Authority.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > The information contained in this email is strictly
> > confidential and for
> > > > the use of the addressee only, unless otherwise
> indicated.
> > If you are not
> > > > the intended recipient, please do not read, copy, use or
> > disclose to
> > > others
> > > > this message or any attachment. Please also notify the
> > sender by replying
> > > > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > email
> > > > and any copies of it. Opinions, conclusion (etc) that do
> > not relate to
> > > the
> > > > official business of this company shall be understood as
> > neither given
> > > nor
> > > > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > > > registered in England and Wales, company number 04008957)
> > and IG Index
> > > > Limited (a company registered in England and Wales,
> > company number
> > > > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill,
> > > > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG
> > > > Index Limited (register number 114059) are authorised and
> > regulated by
> > > the
> > > > Financial Conduct Authority.
> > > >
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and
> > for the use of the addressee only, unless otherwise indicated. If you are
> > not the intended recipient, please do not read, copy, use or disclose to
> > others this message or any attachment. Please also notify the sender by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do not
> > relate to the official business of this company shall be understood as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> > 195355) and IG Index Limited (register number 114059) are authorised and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
"(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)"
very. this (i claim :-) ) is the result of very intense opposition to the
usefulness of the feature early on, and not a real design goal
On Sun, Feb 19, 2017 at 2:11 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Whilst if having an array of a Header[] would mean getting this through,
> im happy changing this, as already done.
>
> Just going through how this is, I don’t believe this would provide much
> saving. If issue trying to resolve is around object/garbage creation of
> Map.Entry objects.
>
> All you’re doing here is replacing the equiv of a HashMap.Node (Map.Entry)
> which simply holds reference to key and value objects with a custom variant.
>
>
>
>
>
> On 19/02/2017, 20:22, "Michael Pearce" <Mi...@ig.com> wrote:
>
> On point 1 & 2 Ive updated KIP to show varints (and removed the bit
> flag). (on the assumption KIP 98 is getting the agreement the protocol is
> moving from int32 to varInts as standard)
>
> On point 3 ive updated to use an array of Header class, instead of a
> MultiMap in the Headers class object
>
>
>
> On 19/02/2017, 20:06, "Michael Pearce" <Mi...@ig.com> wrote:
>
> On points 1 and 2 I agree.
>
> This also affects kip-98, I should expect this resolved before
> that vote also passes. If it is accepted there (I’m assuming this is
> getting discussed on that KIP? As you’re driving the move to VarInts), I am
> happy to make this KIP will simply follow suit to whatever is agreed in
> KIP-98.
>
> On 3) Agreed this is a simpler form, as long as no one is
> expecting hashmap lookup performance ( O(1) ), I still would prefer an
> encapsulated class, so if we find that holding it as an array in future is
> causing some perf issues, the internals are not exposed to end users,
> allowing the internal structure to move to a map.
>
> Can we compromise on?
>
> class ConsumerRecord<K, V> {
> K key;
> V value;
> Headers headers;
> }
>
> class Headers {
> Header[] headers;
>
> add(String key, byte[] value)
> Collection<byte[]> get(String key)
> }
>
> class Header {
> String key;
> byte[] value;
> }
>
>
>
> On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io>
> wrote:
>
> >
> > headers dont "leak" into application code. they are useful
> to application
> > code as well.
>
>
> This is exactly what I have been trying to get at. The use
> cases documented
> here are middleware:
> https://cwiki.apache.org/confluence/display/KAFKA/A+
> Case+for+Kafka+Headers.
> If headers are intended for the application space as well, the
> document
> should be updated accordingly so that it is an explicit design
> goal and not
> an unfortunate design byproduct. There may be some
> disagreement on whether
> it _should_ be a design goal, but it may not matter much since
> the current
> interceptor API is probably insufficient for middleware
> applications (I
> haven't thought of a way to do this that isn't cumbersome).
>
> That aside, the three unresolved points for me are the
> following:
>
> 1. The use of varints. As a one-off for use only with record
> headers, the
> case was weak, but if we are using them throughout the message
> format, then
> we should do so here as well. The additional complexity is
> minimal and
> early performance testing fully justifies their use.
>
> 2. If varints are used, the case for using attributes to
> indicate null
> headers also becomes weak. We only add one additional byte in
> each message
> if there are no headers. Whatever the case, let us be
> consistent with how
> we treat null keys and values.
>
> 3. We have apparently agreed that the broker will validate
> headers (please
> add this to the KIP). That being the case, I would prefer to
> use Kafka's
> conventional format for arrays. The downside is that it takes
> more work to
> skip over the headers on the consumer, though it's unclear if
> the cost
> would matter in practice. Some concern was previously
> expressed about the
> allocation of maps. An alternative would be to use arrays, i.e.
>
> class ConsumerRecord<K, V> {
> K key;
> V value;
> Header[] headers;
> }
>
> class Header {
> String key;
> byte[] value;
> }
>
> This would work nicely with the conventional array format and
> my guess is
> it would obviate the need do any lazy initialization. If we
> use the map as
> is currently documented, then it is possible with either
> representation to
> slice the headers and initialize them lazily. Either way, it
> might be a
> good idea to use a separate object to represent the headers in
> case we need
> to modify it in the future in some way.
>
> (By the way, doesn't it feel a bit odd that we seem to be
> designing a
> feature which is optimized for people not using?)
>
>
> If we can resolve these points, then at least you will get my
> vote.
>
> Thanks,
> Jason
>
> On Sun, Feb 19, 2017 at 7:30 AM, radai <
> radai.rosenblatt@gmail.com> wrote:
>
> > headers dont "leak" into application code. they are useful
> to application
> > code as well.
> > IIUC samze currently has headers "in-V" and would just
> switch over to kafka
> > headers if they exist.
> > im sure plenty of other users of kafka would have a use for
> headers.
> > im pretty sure use cases exist around shuffling data
> into/out-of kafka
> > (kafka connect or equivalent) where metadata from one end
> could copied over
> > to the other (S3, for example uses http headers for
> user-accessible
> > metadata). it will be kafka client code getting/setting
> those headers. not
> > an interceptor.
> >
> > On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > wrote:
> >
> > > For APM single event tracing, need access to the header at
> the point of
> > > processing on the processing thread.
> > >
> > > As such interceptors will not work/be suitable for these,
> due to the fact
> > > they act on the ConsumerRecords as a batch, before the
> handling thread
> > can
> > > split out and process per message which is the point these
> tools will
> > need
> > > to continue to transaction tracing.
> > >
> > > Like wise tools and infra pieces will need access to the
> message outside
> > > the interceptor realm.
> > >
> > >
> > >
> > > On 17/02/2017, 21:26, "Jason Gustafson" <
> jason@confluent.io> wrote:
> > >
> > > >
> > > > That’s exactly what we’re doing the headers are a
> slice of bytes,
> > > which
> > > > then gets parsed later if needed, or can be parsed
> right away, the
> > > headers
> > > > is part of the protocol, so can still be validated
> if wanted.
> > > > If you had a header count then you would have to go
> through each
> > > header
> > > > key and value length value to work out how much to
> skip to get to
> > > say the
> > > > value or any future component in the message after
> the headers.
> > > Having it
> > > > as a byte[] with length value makes this a lot
> easier to skip.
> > >
> > >
> > > So the broker will parse the headers and validate
> them. Good. The
> > only
> > > reason remaining that I can see to leave the headers
> as a byte array
> > > is to
> > > make it easier for the client to skip past them. Are
> we sure this is
> > > not
> > > premature optimization? Are there any performance
> results which show
> > > that
> > > this is worthwhile?
> > >
> > > What’s the issue with exposing a method getHeaders on
> the
> > > producer/consumer
> > > > record? It doesn’t break anything. We don’t need any
> special
> > version.
> > >
> > >
> > > See my previous explanation. What I am trying to
> resist is the
> > headers
> > > becoming a general application-level facility. The
> primary use case
> > as
> > > far
> > > as I can tell is middleware, which is the use case
> that the
> > > interceptors
> > > are providing.
> > >
> > > Current batch consumer model and consumer interceptors
> don’t work
> > where
> > > > headers need to be acted on at per message level at
> time of
> > > processing,
> > > > very case is APM (the core one), where the header
> value is used to
> > > continue
> > > > tracing.
> > >
> > >
> > > I still don't understand the point about batching. The
> consumer
> > > records are
> > > exposed as a batch in the consumer interceptor, but
> you can still
> > > iterate
> > > them individually. It is no different for the consumer
> API itself.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > Re:
> > > >
> > > > “ The point about creation of maps seems
> orthogonal. We can
> > still
> > > > represent
> > > > the headers as a slice of bytes until the time
> it is accessed.”
> > > >
> > > > That’s exactly what we’re doing the headers are a
> slice of bytes,
> > > which
> > > > then gets parsed later if needed, or can be parsed
> right away, the
> > > headers
> > > > is part of the protocol, so can still be validated
> if wanted.
> > > >
> > > > If you had a header count then you would have to go
> through each
> > > header
> > > > key and value length value to work out how much to
> skip to get to
> > > say the
> > > > value or any future component in the message after
> the headers.
> > > Having it
> > > > as a byte[] with length value makes this a lot
> easier to skip.
> > > >
> > > >
> > > > On 17/02/2017, 20:37, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > What’s the issue with exposing a method
> getHeaders on the
> > > > producer/consumer record? It doesn’t break anything.
> We don’t need
> > > any
> > > > special version.
> > > >
> > > > Current batch consumer model and consumer
> interceptors don’t
> > work
> > > > where headers need to be acted on at per message
> level at time of
> > > > processing, very case is APM (the core one), where
> the header value
> > > is used
> > > > to continue tracing. JMS/HTTP etc all expose these,
> without issues.
> > > I would
> > > > NOT want to lock this down to only be usable
> accessible via
> > > interceptors,
> > > > as we’d fail on one of the main goals.
> > > >
> > > > Regards
> > > > Mike
> > > >
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 20:21, "Jason Gustafson" <
> jason@confluent.io>
> > > wrote:
> > > >
> > > > The point about creation of maps seems
> orthogonal. We can
> > > still
> > > > represent
> > > > the headers as a slice of bytes until the
> time it is
> > > accessed.
> > > >
> > > >
> > > > > Yes exactly we have access to the records
> thus why the
> > > header
> > > > should be
> > > > > accessible via it and not hidden for only
> interceptors to
> > > access.
> > > >
> > > >
> > > > As explained above, the point is to make the
> intended usage
> > > clear.
> > > > Applications should continue to rely on the
> key/value
> > fields
> > > to
> > > > serialize
> > > > their own headers, and it would be more
> ideal if we can
> > avoid
> > > > leaking
> > > > third-party headers into applications. This
> is difficult to
> > > do
> > > > with the
> > > > current interceptors because they share the
> record objects
> > > with
> > > > the common
> > > > API. What I had in mind is something like an
> extension of
> > the
> > > > current
> > > > interceptors which exposed a different
> object (e.g.
> > > > `RecordAndHeaders`).
> > > > The challenge is for MM-like use cases. Let
> me see if I can
> > > come
> > > > up with a
> > > > concrete proposal for that problem.
> > > >
> > > > -Jason
> > > >
> > > >
> > > >
> > > > On Fri, Feb 17, 2017 at 11:55 AM, Michael
> Pearce <
> > > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > > > I am happy to move the definition of the
> header into the
> > > message
> > > > body, but
> > > > > would cause us not to lazy
> initialise/parse the headers,
> > as
> > > > obviously, we
> > > > > would have to traverse these reading the
> message.
> > > > >
> > > > > This was actually one of Jay’s requests:
> > > > >
> > > > > “ 2. I think we should think about
> creating the Map
> > > lazily to
> > > > avoid
> > > > > parsing out all the headers into
> little objects.
> > > HashMaps
> > > > themselves
> > > > > are
> > > > > kind of expensive and the consumer is
> very perf
> > > sensitive so
> > > > and making
> > > > > gazillions of hashmaps that may or may
> not get used
> > is
> > > > probably a bad
> > > > > idea.”
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > >
> > > > > Yes exactly we have access to the
> records thus why
> > the
> > > > header should
> > > > > be accessible via it and not hidden for
> only interceptors
> > > to
> > > > access.
> > > > >
> > > > > Sent using OWA for iPhone
> > > > > ______________________________
> __________
> > > > > From: Magnus Edenhill <
> magnus@edenhill.se>
> > > > > Sent: Friday, February 17, 2017
> 7:34:49 PM
> > > > > To: dev@kafka.apache.org
> > > > > Subject: Re: [DISCUSS] KIP-82 - Add
> Record Headers
> > > > >
> > > > > Big +1 on VarInts.
> > > > > CPUs are fast, memory is slow.
> > > > >
> > > > > I agree with Jason that we'll want to
> continue
> > > verifying
> > > > messages,
> > > > > including their headers, so while I
> appreciate the
> > > idea of
> > > > the opaque
> > > > > header blob it won't be useful in
> practice.
> > > > >
> > > > > /Magnus
> > > > >
> > > > > 2017-02-17 10:41 GMT-08:00 Jason
> Gustafson <
> > > > jason@confluent.io>:
> > > > >
> > > > > > Sorry, my mistake. The consumer
> interceptor is per
> > > batch,
> > > > though I'm
> > > > > not
> > > > > > sure that's an actual limitation
> since you still
> > have
> > > > access to the
> > > > > > individual records.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 10:39 AM,
> Jason Gustafson <
> > > > > jason@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > Re headers as byte array and
> future use by
> > broker.
> > > This
> > > > doesn't
> > > > > take away
> > > > > > >> from that at all. Nor makes it
> difficult at all
> > > in my
> > > > opinion.
> > > > > > >
> > > > > > >
> > > > > > > Yeah, I didn't say it was
> difficult, only
> > awkward.
> > > You
> > > > wouldn't
> > > > > write the
> > > > > > > schema that way if you were
> planning to use it on
> > > the
> > > > brokers from
> > > > > the
> > > > > > > beginning. Note also that one of
> the benefits of
> > > letting
> > > > the broker
> > > > > > > understand headers is that it can
> validate that
> > > they are
> > > > properly
> > > > > > > formatted. If cost is the only
> concern, we should
> > > > confirm its
> > > > > impact
> > > > > > > through performance testing.
> > > > > > >
> > > > > > > One of the key use cases requires
> access on
> > > consume at
> > > > per
> > > > > event/message
> > > > > > >> level at the point that message
> is being
> > > processed, as
> > > > such the
> > > > > batch
> > > > > > >> interceptors and batch consume
> api isn't
> > > suitable. It
> > > > needs to be
> > > > > at the
> > > > > > >> record level.
> > > > > > >
> > > > > > >
> > > > > > > I'm not sure I understand the
> point about
> > batching.
> > > > Interceptors
> > > > > are
> > > > > > > applied per-message, right?
> > > > > > >
> > > > > > > My intent on interceptors is to
> keep the usage of
> > > headers
> > > > > well-defined so
> > > > > > > that they don't start leaking
> unnecessarily into
> > > > applications. My
> > > > > guess
> > > > > > is
> > > > > > > that it's probably inevitable, but
> isolating it
> > in
> > > the
> > > > > interceptors would
> > > > > > > at least give people a second
> thought before
> > > deciding to
> > > > use it.
> > > > > The main
> > > > > > > challenge in my mind is figuring
> out how an MM
> > use
> > > case
> > > > would
> > > > > work. It
> > > > > > > would be more cumbersome to
> replicate headers
> > > through an
> > > > > interceptor,
> > > > > > > though arguably MM should be
> working at a lower
> > > level
> > > > anyway.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Feb 17, 2017 at 10:16 AM,
> Michael Pearce
> > <
> > > > > Michael.Pearce@ig.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Re headers available on the
> record va
> > > interceptors only
> > > > > > >>
> > > > > > >> One of the key use cases requires
> access on
> > > consume at
> > > > per
> > > > > event/message
> > > > > > >> level at the point that message
> is being
> > > processed, as
> > > > such the
> > > > > batch
> > > > > > >> interceptors and batch consume
> api isn't
> > > suitable. It
> > > > needs to be
> > > > > at the
> > > > > > >> record level.
> > > > > > >>
> > > > > > >> This anyhow is similar to
> jms/http/amqp where
> > > headers
> > > > are
> > > > > available to
> > > > > > >> consuming applications.
> > > > > > >>
> > > > > > >> Re headers as byte array and
> future use by
> > > broker. This
> > > > doesn't
> > > > > take
> > > > > > away
> > > > > > >> from that at all. Nor makes it
> difficult at all
> > > in my
> > > > opinion.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> Sent using OWA for iPhone
> > > > > > >> ______________________________
> __________
> > > > > > >> From: Jason Gustafson <
> jason@confluent.io>
> > > > > > >> Sent: Friday, February 17, 2017
> 5:55:42 PM
> > > > > > >> To: dev@kafka.apache.org
> > > > > > >> Subject: Re: [DISCUSS] KIP-82 -
> Add Record
> > Headers
> > > > > > >>
> > > > > > >> >
> > > > > > >> > Would you be proposing in
> KIP-98 to convert
> > the
> > > other
> > > > message
> > > > > int’s
> > > > > > (key
> > > > > > >> > length, value length) also to
> varint to keep
> > it
> > > > uniform.
> > > > > > >> > Also I assume there will be a
> static or helper
> > > method
> > > > made to
> > > > > > write/read
> > > > > > >> > these in the client and server.
> > > > > > >>
> > > > > > >>
> > > > > > >> Yes, that is what we are
> proposing, so using
> > > varints
> > > > for headers
> > > > > would
> > > > > > be
> > > > > > >> consistent with the rest of the
> message. We have
> > > used
> > > > static
> > > > > helper
> > > > > > >> methods
> > > > > > >> in our prototype implementation.
> > > > > > >>
> > > > > > >> The cost of parsing, we want to
> parse/interpret
> > > the
> > > > headers
> > > > > lazily (this
> > > > > > >> is
> > > > > > >> > a key point brought up earlier
> in discussions)
> > > > > > >>
> > > > > > >>
> > > > > > >> I'm a bit skeptical of this. Has
> anyone done the
> > > > performance
> > > > > testing? I
> > > > > > >> can
> > > > > > >> probably implement it and test it
> if no one else
> > > has. I
> > > > was also
> > > > > under
> > > > > > the
> > > > > > >> impression that there may be use
> cases down the
> > > road
> > > > where the
> > > > > broker
> > > > > > >> would
> > > > > > >> need to interpret headers. That
> wouldn't be off
> > > the
> > > > table in the
> > > > > future
> > > > > > if
> > > > > > >> it's represented as bytes, but it
> would be quite
> > > a bit
> > > > more
> > > > > awkward,
> > > > > > >> right?
> > > > > > >>
> > > > > > >> By the way, one question I have
> been wondering
> > > about. My
> > > > > understanding
> > > > > > is
> > > > > > >> that headers are primarily for
> use cases where a
> > > > third-party
> > > > > components
> > > > > > >> wants to enrich messages without
> needing to
> > > understand
> > > > or modify
> > > > > the
> > > > > > >> schema
> > > > > > >> of the message key and value. For
> the
> > applications
> > > > which directly
> > > > > > produce
> > > > > > >> and consume the messages and
> control the
> > key/value
> > > > schema
> > > > > directly, it
> > > > > > >> seems we would rather have them
> implement
> > headers
> > > > directly in
> > > > > their own
> > > > > > >> schema. Supposing for the sake of
> argument that
> > > it was
> > > > possible,
> > > > > my
> > > > > > >> question is whether it be
> sufficient to expose
> > the
> > > > headers in the
> > > > > > >> interceptor API and not in the
> common API?
> > > > > > >>
> > > > > > >> -Jason
> > > > > > >>
> > > > > > >> On Fri, Feb 17, 2017 at 3:26 AM,
> Michael Pearce
> > <
> > > > > Michael.Pearce@ig.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > On the point of varInts
> > > > > > >> >
> > > > > > >> > Would you be proposing in
> KIP-98 to convert
> > the
> > > other
> > > > message
> > > > > int’s
> > > > > > (key
> > > > > > >> > length, value length) also to
> varint to keep
> > it
> > > > uniform.
> > > > > > >> > Also I assume there will be a
> static or helper
> > > method
> > > > made to
> > > > > > write/read
> > > > > > >> > these in the client and server.
> > > > > > >> >
> > > > > > >> > Cheers
> > > > > > >> > Mike
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On 17/02/2017, 11:22, "Michael
> Pearce" <
> > > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > On the point re: headers in
> the message
> > > protocol
> > > > being a
> > > > > byte
> > > > > > array
> > > > > > >> > and not a count of elements
> followed by the
> > > elements.
> > > > Again
> > > > > this was
> > > > > > >> > discussed/argued previously.
> > > > > > >> >
> > > > > > >> > It was agreed on for a few
> reasons some of
> > > which
> > > > you have
> > > > > > obviously
> > > > > > >> > picked up on:
> > > > > > >> >
> > > > > > >> > Broker is able to pass it
> through opaquely
> > > > > > >> > The cost of parsing, we
> want to
> > > parse/interpret
> > > > the headers
> > > > > lazily
> > > > > > >> > (this is a key point brought up
> earlier in
> > > > discussions)
> > > > > > >> > Headers can be copied from
> consumer record
> > > to
> > > > producer
> > > > > record (aka
> > > > > > >> > mirror makers etc) without
> parsing if no
> > > changes are
> > > > being made
> > > > > or
> > > > > > being
> > > > > > >> > looked at.
> > > > > > >> > Keeps the broker agnostic
> to the format
> > > > > > >> > You need an int32 either
> for the byte size
> > > of the
> > > > headers,
> > > > > or for
> > > > > > >> the
> > > > > > >> > count of elements, so overheads
> are the same,
> > > but
> > > > with going
> > > > > with an
> > > > > > >> opaque
> > > > > > >> > byte array has the above
> advantages.
> > > > > > >> >
> > > > > > >> > Cheers
> > > > > > >> > Mike
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On 17/02/2017, 02:50,
> "Jason Gustafson" <
> > > > jason@confluent.io
> > > > > >
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > Sorry, should have
> noted that the
> > > performance
> > > > testing
> > > > > was done
> > > > > > >> > using the
> > > > > > >> > producer performance
> tool shipped with
> > > Kafka.
> > > > > > >> >
> > > > > > >> > -Jason
> > > > > > >> >
> > > > > > >> > On Thu, Feb 16, 2017 at
> 6:44 PM, Jason
> > > > Gustafson <
> > > > > > >> > jason@confluent.io> wrote:
> > > > > > >> >
> > > > > > >> > > Hey Nacho,
> > > > > > >> > >
> > > > > > >> > > I've compared
> performance of our
> > > KIP-98
> > > > > implementation with
> > > > > > >> and
> > > > > > >> > without
> > > > > > >> > > varints. For messages
> around 128
> > > bytes, we
> > > > see an
> > > > > increase
> > > > > > in
> > > > > > >> > throughput of
> > > > > > >> > > about 30% using the
> default
> > > configuration
> > > > settings.
> > > > > At 256
> > > > > > >> > bytes, the
> > > > > > >> > > increase is around
> 16%. Obviously
> > the
> > > > performance
> > > > > converge
> > > > > > as
> > > > > > >> > messages get
> > > > > > >> > > larger, but it seems
> well worth the
> > > cost.
> > > > Note that
> > > > > we are
> > > > > > >> also
> > > > > > >> > seeing a
> > > > > > >> > > substantial
> performance increase
> > > against
> > > > trunk
> > > > > primarily
> > > > > > >> because
> > > > > > >> > of the
> > > > > > >> > > much more efficient
> packing that
> > > varints
> > > > provide us.
> > > > > > Anything
> > > > > > >> > adding to
> > > > > > >> > > message overhead,
> such as record
> > > headers,
> > > > would only
> > > > > > increase
> > > > > > >> > the relative
> > > > > > >> > > difference. (Of
> course take these
> > > numbers
> > > > with a
> > > > > grain of
> > > > > > salt
> > > > > > >> > since I have
> > > > > > >> > > only used the default
> settings with
> > > both
> > > > the producer
> > > > > and
> > > > > > >> broker
> > > > > > >> > on my
> > > > > > >> > > local machine. We
> intend to provide
> > > more
> > > > extensive
> > > > > > performance
> > > > > > >> > details as
> > > > > > >> > > part of the work for
> KIP-98.)
> > > > > > >> > >
> > > > > > >> > > The implementation we
> are using is
> > > from
> > > > protobuf (
> > > > > > >> > >
> https://developers.google.com/
> > > > > > protocol-buffers/docs/encoding
> > > > > > >> ),
> > > > > > >> > which is
> > > > > > >> > > also used in HBase.
> It is trivial to
> > > > implement and as
> > > > > far
> > > > > > as I
> > > > > > >> > know doesn't
> > > > > > >> > > suffer from the
> aliasing problem you
> > > are
> > > > describing. I
> > > > > > checked
> > > > > > >> > with Magnus
> > > > > > >> > > (the author of
> librdkafka) and he
> > > agreed
> > > > that the
> > > > > savings
> > > > > > >> seemed
> > > > > > >> > worth the
> > > > > > >> > > cost of
> implementation.
> > > > > > >> > >
> > > > > > >> > > -Jason
> > > > > > >> > >
> > > > > > >> > > On Thu, Feb 16, 2017
> at 4:32 PM,
> > > Ignacio
> > > > Solis <
> > > > > > >> isolis@igso.net>
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > >> -VarInts
> > > > > > >> > >>
> > > > > > >> > >> I'm one of the
> people (if not the
> > > most)
> > > > opposed to
> > > > > VarInts.
> > > > > > >> > VarInts
> > > > > > >> > >> have a place, but
> this is not it.
> > > (We
> > > > had a large
> > > > > > >> discussion
> > > > > > >> > about
> > > > > > >> > >> them at the
> beginning of KIP-82
> > time)
> > > > > > >> > >>
> > > > > > >> > >> If anybody has real
> life
> > performance
> > > > numbers of
> > > > > VarInts
> > > > > > >> > improving
> > > > > > >> > >> things or
> significantly reducing
> > > resources
> > > > I would
> > > > > like to
> > > > > > >> know
> > > > > > >> > what
> > > > > > >> > >> that case may be.
> Yes, you can save
> > > some
> > > > bytes here
> > > > > and
> > > > > > >> there,
> > > > > > >> > but
> > > > > > >> > >> this is probably
> insignificant to
> > the
> > > > overall system
> > > > > > behavior
> > > > > > >> > and
> > > > > > >> > >> storage
> requirements. -- I say
> > this
> > > with
> > > > respect to
> > > > > using
> > > > > > >> > VarInts in
> > > > > > >> > >> the protocol itself,
> not as part of
> > > the
> > > > data.
> > > > > > >> > >>
> > > > > > >> > >> VarInts require you
> to parse the
> > Int
> > > > before using it
> > > > > and
> > > > > > >> > depending on
> > > > > > >> > >> the encoding they
> can suffer from
> > > aliasing
> > > > (multiple
> > > > > > >> > representations
> > > > > > >> > >> for the same value).
> > > > > > >> > >>
> > > > > > >> > >> Why add complexity?
> > > > > > >> > >>
> > > > > > >> > >> Nacho
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >> On Thu, Feb 16, 2017
> at 10:29 AM,
> > > Colin
> > > > McCabe <
> > > > > > >> > cmccabe@apache.org>
> > > > > > >> > >> wrote:
> > > > > > >> > >> > +1 for varints
> here-- it would
> > save
> > > > quite a bit of
> > > > > space.
> > > > > > >> > They are
> > > > > > >> > >> > pretty quick to
> implement as
> > well.
> > > > > > >> > >> >
> > > > > > >> > >> > I think it makes
> sense for values
> > > to be
> > > > byte
> > > > > arrays.
> > > > > > Users
> > > > > > >> > might want
> > > > > > >> > >> > to attach
> arbitrary payloads;
> > they
> > > > shouldn't be
> > > > > forced to
> > > > > > >> > serialize
> > > > > > >> > >> > everything to Java
> strings.
> > > > > > >> > >> >
> > > > > > >> > >> > best,
> > > > > > >> > >> > Colin
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > On Thu, Feb 16,
> 2017, at 09:52,
> > > Jason
> > > > Gustafson
> > > > > wrote:
> > > > > > >> > >> >> Hey Michael,
> > > > > > >> > >> >>
> > > > > > >> > >> >> Hmm, I guess the
> point of
> > > representing
> > > > it as
> > > > > bytes is to
> > > > > > >> > allow the
> > > > > > >> > >> broker
> > > > > > >> > >> >> to pass it
> through opaquely? Is
> > > the
> > > > cost of
> > > > > parsing
> > > > > > them a
> > > > > > >> > concern, or
> > > > > > >> > >> >> are
> > > > > > >> > >> >> we simply trying
> to ensure that
> > > the
> > > > broker stays
> > > > > > agnostic
> > > > > > >> to
> > > > > > >> > the
> > > > > > >> > >> format?
> > > > > > >> > >> >>
> > > > > > >> > >> >> On varints, I
> think adding
> > > support for
> > > > them makes
> > > > > less
> > > > > > >> sense
> > > > > > >> > for an
> > > > > > >> > >> >> isolated use
> case, but as part
> > of
> > > a
> > > > more holistic
> > > > > change
> > > > > > >> > (such as what
> > > > > > >> > >> we
> > > > > > >> > >> >> have proposed in
> KIP-98), I
> > think
> > > they
> > > > are
> > > > > justifiable.
> > > > > > If
> > > > > > >> > we add them,
> > > > > > >> > >> >> then the need to
> use attributes
> > > becomes
> > > > quite a
> > > > > bit
> > > > > > >> weaker,
> > > > > > >> > right? The
> > > > > > >> > >> >> other thing I
> find slightly odd
> > > is the
> > > > fact that
> > > > > null
> > > > > > >> > headers has no
> > > > > > >> > >> >> actual
> > > > > > >> > >> >> semantic meaning
> for the message
> > > > (unlike null
> > > > > keys and
> > > > > > >> > values). It is
> > > > > > >> > >> >> just
> > > > > > >> > >> >> a space
> optimization. It seems a
> > > bit
> > > > better to
> > > > > always
> > > > > > use
> > > > > > >> > size 0 to
> > > > > > >> > >> >> indicate having
> no headers.
> > > > > > >> > >> >>
> > > > > > >> > >> >> Overall, the main
> point is
> > > ensuring
> > > > that the
> > > > > message
> > > > > > >> schema
> > > > > > >> > remains
> > > > > > >> > >> >> consistent,
> either within the
> > > larger
> > > > protocol, or
> > > > > at a
> > > > > > >> > minimum within
> > > > > > >> > >> the
> > > > > > >> > >> >> message itself.
> > > > > > >> > >> >>
> > > > > > >> > >> >> -Jason
> > > > > > >> > >> >>
> > > > > > >> > >> >> On Thu, Feb 16,
> 2017 at 6:39 AM,
> > > > Michael Pearce <
> > > > > > >> > Michael.Pearce@ig.com
> > > > > > >> > >> >
> > > > > > >> > >> >> wrote:
> > > > > > >> > >> >>
> > > > > > >> > >> >> > Hi Jason,
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On point 1) in
> the message
> > > protocol
> > > > the headers
> > > > > are
> > > > > > >> simply
> > > > > > >> > a byte
> > > > > > >> > >> array,
> > > > > > >> > >> >> > as like the key
> or value, this
> > > is to
> > > > clearly
> > > > > demarcate
> > > > > > >> the
> > > > > > >> > header in
> > > > > > >> > >> the
> > > > > > >> > >> >> > core message.
> Then the header
> > > byte
> > > > array in the
> > > > > core
> > > > > > >> > message is an
> > > > > > >> > >> array of
> > > > > > >> > >> >> > key, value
> pairs. This is what
> > > it is
> > > > denoting.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Then this would
> be I guess in
> > > the
> > > > given
> > > > > notation:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Headers =>
> [KeyLength, Key,
> > > > ValueLength, Value]
> > > > > > >> > >> >> > KeyLength
> => int32
> > > > <-----------------NEW
> > > > > size of
> > > > > > the
> > > > > > >> > byte[] of
> > > > > > >> > >> the
> > > > > > >> > >> >> > serialised key
> value
> > > > > > >> > >> >> > Key => bytes
> > > > <---------------------- NEW
> > > > > > serialised
> > > > > > >> > string (UTF8)
> > > > > > >> > >> >> > bytes of the
> header key
> > > > > > >> > >> >> > ValueLength
> => int32
> > > > <-------------- NEW
> > > > > size of
> > > > > > the
> > > > > > >> > byte[] of
> > > > > > >> > >> the
> > > > > > >> > >> >> > serialised
> header value
> > > > > > >> > >> >> > Value =>
> bytes
> > > > <-------------------- NEW
> > > > > > serialised
> > > > > > >> > form of the
> > > > > > >> > >> header
> > > > > > >> > >> >> > value
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > The key length
> and value
> > length
> > > is
> > > > matching the
> > > > > way
> > > > > > the
> > > > > > >> > protocol is
> > > > > > >> > >> >> > defined in the
> core message
> > > currently.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On point 2)
> > > > > > >> > >> >> > Var sized ints,
> this was
> > > discussed
> > > > much earlier
> > > > > on, in
> > > > > > >> > fact I had
> > > > > > >> > >> >> > suggested it
> myself (with
> > Hadoop
> > > > references),
> > > > > the
> > > > > > >> > complexity of this
> > > > > > >> > >> >> > compared to
> having a simpler
> > > protocol
> > > > was
> > > > > argued and
> > > > > > >> > agreed it
> > > > > > >> > >> wasn’t worth
> > > > > > >> > >> >> > the complexity
> as all other
> > > clients
> > > > in other
> > > > > languages
> > > > > > >> > would need to
> > > > > > >> > >> ensure
> > > > > > >> > >> >> > theyre using
> the right var
> > size
> > > > algorithm, as
> > > > > there
> > > > > > is a
> > > > > > >> > few.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On point 3)
> > > > > > >> > >> >> > We did the
> attributes,
> > optional
> > > > approach as
> > > > > originally
> > > > > > >> > there was
> > > > > > >> > >> marked
> > > > > > >> > >> >> > concern that
> headers would
> > > cause a
> > > > message size
> > > > > > overhead
> > > > > > >> > for others,
> > > > > > >> > >> who
> > > > > > >> > >> >> > don’t want
> them. As such this
> > > is the
> > > > clean
> > > > > solution to
> > > > > > >> > achieve that.
> > > > > > >> > >> If
> > > > > > >> > >> >> > that no longer
> holds, and we
> > > don’t
> > > > care that we
> > > > > add
> > > > > > >> 4bytes
> > > > > > >> > overhead,
> > > > > > >> > >> then
> > > > > > >> > >> >> > im happy to
> remove.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > I’m personally
> in favour of
> > > keeping
> > > > the message
> > > > > as
> > > > > > small
> > > > > > >> > as possible
> > > > > > >> > >> so
> > > > > > >> > >> >> > people don’t
> get shocks in
> > perf
> > > and
> > > > throughputs
> > > > > dues
> > > > > > to
> > > > > > >> > message size,
> > > > > > >> > >> >> > unless they
> actively use the
> > > feature,
> > > > as such I
> > > > > do
> > > > > > >> prefer
> > > > > > >> > the
> > > > > > >> > >> attribute bit
> > > > > > >> > >> >> > wise feature
> flag approach
> > > myself.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On 16/02/2017,
> 05:40, "Jason
> > > > Gustafson" <
> > > > > > >> > jason@confluent.io> wrote:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > We have
> proposed a few
> > > > significant changes
> > > > > to the
> > > > > > >> > message format
> > > > > > >> > >> in
> > > > > > >> > >> >> > KIP-98
> > > > > > >> > >> >> > which now
> seems likely to
> > > pass
> > > > (perhaps
> > > > > with some
> > > > > > >> > iterations on
> > > > > > >> > >> >> >
> implementation details).
> > It
> > > would
> > > > be good
> > > > > to try
> > > > > > and
> > > > > > >> > coordinate
> > > > > > >> > >> the
> > > > > > >> > >> >> > changes
> > > > > > >> > >> >> > in both of
> the proposals
> > to
> > > make
> > > > sure they
> > > > > are
> > > > > > >> > consistent and
> > > > > > >> > >> >> > compatible.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > I think
> using the
> > > attributes to
> > > > indicate
> > > > > null
> > > > > > >> headers
> > > > > > >> > is a
> > > > > > >> > >> reasonable
> > > > > > >> > >> >> > approach.
> We have proposed
> > > to do
> > > > the same
> > > > > thing
> > > > > > for
> > > > > > >> > the message
> > > > > > >> > >> key and
> > > > > > >> > >> >> > value. That
> said, I
> > > sympathize
> > > > with Jay's
> > > > > > argument.
> > > > > > >> > Having
> > > > > > >> > >> multiple
> > > > > > >> > >> >> > ways to
> > > > > > >> > >> >> > specify a
> null value
> > > increases
> > > > the overall
> > > > > > >> complexity
> > > > > > >> > of the
> > > > > > >> > >> protocol.
> > > > > > >> > >> >> > You
> > > > > > >> > >> >> > can see
> this just from the
> > > fact
> > > > that you
> > > > > need the
> > > > > > >> > extra verbiage
> > > > > > >> > >> in the
> > > > > > >> > >> >> > protocol
> specification in
> > > this
> > > > KIP and in
> > > > > KIP-98
> > > > > > to
> > > > > > >> > describe the
> > > > > > >> > >> >> > dependence
> > > > > > >> > >> >> > between the
> fields and the
> > > > attributes. It
> > > > > seems
> > > > > > >> like a
> > > > > > >> > slippery
> > > > > > >> > >> slope
> > > > > > >> > >> >> > if
> > > > > > >> > >> >> > you start
> allowing
> > different
> > > > request types
> > > > > to
> > > > > > >> > implement the
> > > > > > >> > >> protocol
> > > > > > >> > >> >> >
> specification differently.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > You can
> also argue that
> > the
> > > > messages
> > > > > already are
> > > > > > and
> > > > > > >> > are likely
> > > > > > >> > >> to
> > > > > > >> > >> >> > remain a
> > > > > > >> > >> >> > special
> case. For example,
> > > there
> > > > is
> > > > > currently no
> > > > > > >> > generality in
> > > > > > >> > >> how
> > > > > > >> > >> >> > compressed
> message sets
> > are
> > > > represented
> > > > > that would
> > > > > > >> be
> > > > > > >> > applicable
> > > > > > >> > >> for
> > > > > > >> > >> >> > other
> > > > > > >> > >> >> > request
> types. Some might
> > > see this
> > > > > divergence as
> > > > > > an
> > > > > > >> > unfortunate
> > > > > > >> > >> >> > protocol
> > > > > > >> > >> >> > deficiency
> which should be
> > > fixed;
> > > > others
> > > > > might see
> > > > > > >> it
> > > > > > >> > as sort of
> > > > > > >> > >> the
> > > > > > >> > >> >> >
> inevitability of needing
> > to
> > > > optimize where
> > > > > it
> > > > > > counts
> > > > > > >> > most. I'm
> > > > > > >> > >> probably
> > > > > > >> > >> >> > somewhere
> in between, but
> > I
> > > think
> > > > we
> > > > > probably all
> > > > > > >> > share the
> > > > > > >> > >> intuition
> > > > > > >> > >> >> > that
> > > > > > >> > >> >> > the
> protocol should be
> > kept
> > > as
> > > > consistent as
> > > > > > >> possible.
> > > > > > >> > With that
> > > > > > >> > >> in
> > > > > > >> > >> >> > mind,
> > > > > > >> > >> >> > here are a
> few comments:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > 1. One
> thing I found a
> > > little odd
> > > > when
> > > > > reading the
> > > > > > >> > current
> > > > > > >> > >> proposal is
> > > > > > >> > >> >> > that
> > > > > > >> > >> >> > the headers
> are both
> > > represented
> > > > as an
> > > > > array of
> > > > > > >> bytes
> > > > > > >> > and as an
> > > > > > >> > >> array
> > > > > > >> > >> >> > of
> > > > > > >> > >> >> > key/value
> pairs. I'd
> > > probably
> > > > suggest
> > > > > something
> > > > > > like
> > > > > > >> > this:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Headers =>
> [HeaderKey
> > > HeaderValue]
> > > > > > >> > >> >> > HeaderKey
> => String
> > > > > > >> > >> >> >
> HeaderValue => Bytes
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > An array in
> the Kafka
> > > protocol is
> > > > > represented as a
> > > > > > >> > 4-byte integer
> > > > > > >> > >> >> > indicating
> the number of
> > > elements
> > > > in the
> > > > > array
> > > > > > >> > followed by the
> > > > > > >> > >> >> >
> serialization of the
> > > elements.
> > > > Unless I'm
> > > > > > >> > misunderstanding, what
> > > > > > >> > >> you
> > > > > > >> > >> >> > have
> > > > > > >> > >> >> > instead is
> the total size
> > > of the
> > > > headers in
> > > > > bytes
> > > > > > >> > followed by the
> > > > > > >> > >> >> > elements.
> > > > > > >> > >> >> > I'm not
> sure I see any
> > > reason for
> > > > this
> > > > > > >> inconsistency.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > 2. In
> KIP-98, we've
> > > introduced
> > > > > variable-length
> > > > > > >> integer
> > > > > > >> > fields.
> > > > > > >> > >> >> > Effectively,
> > > > > > >> > >> >> > we've
> enriched (or
> > > "complicated"
> > > > as Jay
> > > > > might say
> > > > > > ;)
> > > > > > >> > the protocol
> > > > > > >> > >> >> >
> specification to include
> > the
> > > > following
> > > > > types:
> > > > > > >> VarInt,
> > > > > > >> > VarLong,
> > > > > > >> > >> >> >
> UnsignedVarInt and
> > > > UnsignedVarLong.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Along with
> these
> > > primitives, we
> > > > could
> > > > > introduce
> > > > > > the
> > > > > > >> > following
> > > > > > >> > >> types:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> VarSizeArray =>
> > > NumberOfItems
> > > > Item1 Item2
> > > > > .. ItemN
> > > > > > >> > >> >> >
> NumberOfItems =>
> > > UnsignedVarInt
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> VarSizeNullableArray =>
> > > > NumberOfItemsOrNull
> > > > > Item1
> > > > > > >> > Item2 .. ItemN
> > > > > > >> > >> >> >
> NumberOfItemsOrNull =>
> > > VarInt
> > > > (-1 means
> > > > > null)
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > And
> similarly for the
> > > `String`
> > > > and `Bytes`
> > > > > types.
> > > > > > >> > These types
> > > > > > >> > >> can save
> > > > > > >> > >> >> > a
> > > > > > >> > >> >> >
> considerable amount of
> > > space in
> > > > this
> > > > > proposal
> > > > > > >> because
> > > > > > >> > they can
> > > > > > >> > >> be used
> > > > > > >> > >> >> > for
> > > > > > >> > >> >> > both the
> number of headers
> > > > included in the
> > > > > message
> > > > > > >> and
> > > > > > >> > the
> > > > > > >> > >> lengths of
> > > > > > >> > >> >> > the
> > > > > > >> > >> >> > header keys
> and values. We
> > > could
> > > > do this
> > > > > instead:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Headers =>
> > > VarSizeArray[HeaderKey
> > > > > HeaderValue]
> > > > > > >> > >> >> > HeaderKey
> =>
> > VarSizeString
> > > > > > >> > >> >> >
> HeaderValue =>
> > > VarSizeBytes
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > Combining
> the savings from
> > > the
> > > > use of
> > > > > variable
> > > > > > >> length
> > > > > > >> > fields, the
> > > > > > >> > >> >> > benefit
> > > > > > >> > >> >> > of using
> the attributes to
> > > > represent null
> > > > > seems
> > > > > > >> pretty
> > > > > > >> > small.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > 3.
> Whichever way we go
> > > (whether
> > > > we use the
> > > > > > >> attributes
> > > > > > >> > or not), we
> > > > > > >> > >> >> > should at
> > > > > > >> > >> >> > least be
> consistent
> > between
> > > this
> > > > KIP and
> > > > > KIP-98.
> > > > > > It
> > > > > > >> > would be very
> > > > > > >> > >> >> > strange
> > > > > > >> > >> >> > to have two
> ways to
> > > represent
> > > > null values
> > > > > in the
> > > > > > >> same
> > > > > > >> > schema.
> > > > > > >> > >> Either
> > > > > > >> > >> >> > way is
> > > > > > >> > >> >> > OK with me.
> I think some
> > > > message-level
> > > > > > optimizations
> > > > > > >> > are
> > > > > > >> > >> justifiable,
> > > > > > >> > >> >> > but
> > > > > > >> > >> >> > the savings
> here seem
> > > minimal (a
> > > > few bytes
> > > > > per
> > > > > > >> > message), so
> > > > > > >> > >> maybe it's
> > > > > > >> > >> >> > not
> > > > > > >> > >> >> > worth the
> cost of letting
> > > the
> > > > message
> > > > > diverge even
> > > > > > >> > further from
> > > > > > >> > >> the
> > > > > > >> > >> >> > rest of
> > > > > > >> > >> >> > the
> protocol.
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > -Jason
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > On Wed, Feb
> 15, 2017 at
> > > 8:52 AM,
> > > > radai <
> > > > > > >> > >>
> radai.rosenblatt@gmail.com>
> > > > > > >> > >> >> > wrote:
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > > I've
> trimmed the inline
> > > > contents as this
> > > > > mail is
> > > > > > >> > getting too
> > > > > > >> > >> big for
> > > > > > >> > >> >> > the
> > > > > > >> > >> >> > > apache
> mailing list
> > > software to
> > > > deliver
> > > > > :-(
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > 1. the
> important thing
> > for
> > > > > interoperability is
> > > > > > for
> > > > > > >> > different
> > > > > > >> > >> >> > "interested
> > > > > > >> > >> >> > > parties"
> (plugins, infra
> > > > layers/wrappers,
> > > > > > >> user-code)
> > > > > > >> > to be
> > > > > > >> > >> able to
> > > > > > >> > >> >> > stick
> > > > > > >> > >> >> > > pieces of
> metadata onto
> > > msgs
> > > > without
> > > > > getting in
> > > > > > >> each
> > > > > > >> > other's
> > > > > > >> > >> way. a
> > > > > > >> > >> >> > common
> > > > > > >> > >> >> > > key
> scheme (Strings, as
> > > of the
> > > > time of
> > > > > this
> > > > > > >> > writing?) is all
> > > > > > >> > >> thats
> > > > > > >> > >> >> > required
> > > > > > >> > >> >> > > for that.
> it is assumed
> > > that
> > > > the other end
> > > > > > >> > interested in any
> > > > > > >> > >> such
> > > > > > >> > >> >> > piece of
> > > > > > >> > >> >> > > metadata
> knows the
> > > encoding,
> > > > and byte[]
> > > > > provides
> > > > > > >> for
> > > > > > >> > the most
> > > > > > >> > >> >> > flexibility.
> > > > > > >> > >> >> > > i believe
> this is the
> > same
> > > > logic behind
> > > > > core
> > > > > > kafka
> > > > > > >> > being
> > > > > > >> > >> >> > byte[]/byte[] -
> > > > > > >> > >> >> > > Strings
> are more
> > "usable"
> > > but
> > > > bytes are
> > > > > flexible
> > > > > > >> and
> > > > > > >> > so were
> > > > > > >> > >> chosen.
> > > > > > >> > >> >> > > Also -
> core kafka doesnt
> > > even
> > > > do that
> > > > > good of a
> > > > > > >> job
> > > > > > >> > on
> > > > > > >> > >> usability of
> > > > > > >> > >> >> > the
> > > > > > >> > >> >> > > payload
> (example - i
> > have
> > > to
> > > > specify the
> > > > > nop
> > > > > > >> byte[]
> > > > > > >> > "decoders"
> > > > > > >> > >> >> > explicitly
> > > > > > >> > >> >> > > in conf),
> and again
> > > sacrificies
> > > > usability
> > > > > for
> > > > > > the
> > > > > > >> > sake of
> > > > > > >> > >> >> > performance (no
> > > > > > >> > >> >> > >
> convenient single-record
> > > > processing as
> > > > > poll is a
> > > > > > >> > batch, lots of
> > > > > > >> > >> >> > obscure
> > > > > > >> > >> >> > > little
> config details
> > > exposing
> > > > internals
> > > > > of the
> > > > > > >> > batching
> > > > > > >> > >> mechanism,
> > > > > > >> > >> >> > etc)
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > this is
> also why i
> > really
> > > > dislike the
> > > > > idea of a
> > > > > > >> > "type system"
> > > > > > >> > >> for
> > > > > > >> > >> >> > header
> > > > > > >> > >> >> > > values,
> it further
> > > degrades the
> > > > > usability, adds
> > > > > > >> > complexity and
> > > > > > >> > >> will
> > > > > > >> > >> >> > >
> eventually get in
> > > people's way,
> > > > also, it
> > > > > would
> > > > > > be
> > > > > > >> > the 2nd/3rd
> > > > > > >> > >> >> > home-group
> > > > > > >> > >> >> > >
> serialization mechanism
> > > in core
> > > > kafka
> > > > > (counting
> > > > > > 2
> > > > > > >> > iterations
> > > > > > >> > >> of the
> > > > > > >> > >> >> > "type
> > > > > > >> > >> >> > >
> definition DSL")
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > 2. this
> is an
> > > implementation
> > > > detail, and
> > > > > not
> > > > > > even
> > > > > > >> a
> > > > > > >> > very "user
> > > > > > >> > >> >> > facing" one?
> > > > > > >> > >> >> > > to the
> best of my
> > > understanding
> > > > the vote
> > > > > process
> > > > > > >> is
> > > > > > >> > on proposed
> > > > > > >> > >> >> > >
> API/behaviour. also -
> > > since
> > > > we're willing
> > > > > to go
> > > > > > >> with
> > > > > > >> > strings
> > > > > > >> > >> just
> > > > > > >> > >> >> > serialize
> > > > > > >> > >> >> > > a 0-sized
> header blob
> > and
> > > IIUC
> > > > you dont
> > > > > need any
> > > > > > >> > optionals
> > > > > > >> > >> anymore.
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > 3. yes,
> we can :-)
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > On Tue,
> Feb 14, 2017 at
> > > 11:56
> > > > PM, Michael
> > > > > > Pearce <
> > > > > > >> > >> >> >
> Michael.Pearce@ig.com>
> > > > > > >> > >> >> > > wrote:
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> > > > Hi Jay,
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > 1)
> There was some
> > > initial
> > > > debate on the
> > > > > value
> > > > > > >> > part, as youll
> > > > > > >> > >> note
> > > > > > >> > >> >> > String,
> > > > > > >> > >> >> > > > String
> headers were
> > > > discounted early
> > > > > on. The
> > > > > > >> > reason for this
> > > > > > >> > >> is
> > > > > > >> > >> >> > >
> flexibility
> > > > > > >> > >> >> > > > and
> keeping in line
> > > with the
> > > > > flexibility of
> > > > > > key,
> > > > > > >> > value of the
> > > > > > >> > >> >> > message
> > > > > > >> > >> >> > > > object
> itself. I don’t
> > > think
> > > > it takes
> > > > > away
> > > > > > from
> > > > > > >> an
> > > > > > >> > ecosystem
> > > > > > >> > >> as
> > > > > > >> > >> >> > each
> > > > > > >> > >> >> > > plugin
> > > > > > >> > >> >> > > > will
> care for their
> > own
> > > key,
> > > > this way
> > > > > ints,
> > > > > > >> > booleans , exotic
> > > > > > >> > >> >> > custom
> > > > > > >> > >> >> > > binary
> > > > > > >> > >> >> > > > can all
> be catered
> > for=.
> > > > > > >> > >> >> > > > a. If
> you really
> > wanted
> > > to
> > > > push for a
> > > > > typed
> > > > > > >> value
> > > > > > >> > interface,
> > > > > > >> > >> I
> > > > > > >> > >> >> > wouldn’t
> > > > > > >> > >> >> > > > want
> just String
> > values
> > > > supported, but
> > > > > the the
> > > > > > >> > primatives
> > > > > > >> > >> plus
> > > > > > >> > >> >> > string and
> > > > > > >> > >> >> > > > also
> still keeping the
> > > > ability to have a
> > > > > > binary
> > > > > > >> > for custom
> > > > > > >> > >> >> > binaries that
> > > > > > >> > >> >> > > > some
> organisations may
> > > have.
> > > > > > >> > >> >> > > > i. I
> have written this
> > > slight
> > > > > alternative
> > > > > > here,
> > > > > > >> > >> >> > >
> > https://cwiki.apache.org/
> > > > > > >> > >> >> > > >
> > > confluence/display/KAFKA/KIP-
> > > > > > >> > 82+-+Add+Record+Headers+-+Typed
> > > > > > >> > >> >> > > > ii.
> Essentially the
> > > value
> > > > bytes, has a
> > > > > leading
> > > > > > >> > byte overhead.
> > > > > > >> > >> >> > > > 1.
> This tells you
> > what
> > > type
> > > > the value
> > > > > is,
> > > > > > >> before
> > > > > > >> > reading
> > > > > > >> > >> the rest
> > > > > > >> > >> >> > of the
> > > > > > >> > >> >> > > > bytes,
> allowing
> > > > > serialisation/deserialization
> > > > > > to
> > > > > > >> > and from the
> > > > > > >> > >> >> > primitives,
> > > > > > >> > >> >> > > > string
> and byte[].
> > This
> > > is
> > > > akin to some
> > > > > other
> > > > > > >> > messaging
> > > > > > >> > >> systems.
> > > > > > >> > >> >> > > > 2) We
> are making it
> > > optional,
> > > > so that
> > > > > for
> > > > > > those
> > > > > > >> > not wanting
> > > > > > >> > >> >> > headers have
> > > > > > >> > >> >> > > 0
> > > > > > >> > >> >> > > > bytes
> overhead (think
> > > of it
> > > > as a feature
> > > > > > flag),
> > > > > > >> I
> > > > > > >> > don’t
> > > > > > >> > >> think this
> > > > > > >> > >> >> > is
> > > > > > >> > >> >> > > >
> complex, especially if
> > > > comparing to
> > > > > changes
> > > > > > >> > proposed in
> > > > > > >> > >> other kips
> > > > > > >> > >> >> > like
> > > > > > >> > >> >> > > > kip-98.
> > > > > > >> > >> >> > > > a. If
> you really
> > really
> > > don’t
> > > > like
> > > > > this, we
> > > > > > can
> > > > > > >> > drop it, but
> > > > > > >> > >> it
> > > > > > >> > >> >> > would
> > > > > > >> > >> >> > > mean
> > > > > > >> > >> >> > > > buying
> into 4 bytes
> > > extra
> > > > overhead for
> > > > > users
> > > > > > who
> > > > > > >> > do not want
> > > > > > >> > >> to use
> > > > > > >> > >> >> > > headers.
> > > > > > >> > >> >> > > > 3) In
> the summary yes,
> > > it is
> > > > at a higher
> > > > > > level,
> > > > > > >> > but I think
> > > > > > >> > >> this
> > > > > > >> > >> >> > is well
> > > > > > >> > >> >> > > >
> documented in the
> > > proposed
> > > > changes
> > > > > section.
> > > > > > >> > >> >> > > > a.
> Added getHeaders
> > > method to
> > > > > > Producer/Consumer
> > > > > > >> > record (that
> > > > > > >> > >> is it)
> > > > > > >> > >> >> > > > b.
> We’ve also detailed
> > > the
> > > > new Headers
> > > > > class
> > > > > > >> that
> > > > > > >> > this method
> > > > > > >> > >> >> > returns
> > > > > > >> > >> >> > > that
> > > > > > >> > >> >> > > >
> encapsulates the
> > headers
> > > > protocol and
> > > > > logic.
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > Best,
> > > > > > >> > >> >> > > > Mike
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > >
> ==Original questions
> > > from the
> > > > vote
> > > > > thread from
> > > > > > >> > Jay.==
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > Couple
> of things I
> > > think we
> > > > still need
> > > > > to work
> > > > > > >> out:
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > 1. I
> think we agree
> > > about
> > > > the key,
> > > > > but I
> > > > > > >> think
> > > > > > >> > we haven't
> > > > > > >> > >> >> > talked about
> > > > > > >> > >> >> > > > the
> value yet. I
> > > think if
> > > > our goal
> > > > > is an
> > > > > > open
> > > > > > >> > ecosystem
> > > > > > >> > >> of these
> > > > > > >> > >> >> > > header
> > > > > > >> > >> >> > > >
> spread across many
> > > plugins
> > > > from many
> > > > > > systems
> > > > > > >> we
> > > > > > >> > should
> > > > > > >> > >> consider
> > > > > > >> > >> >> > making
> > > > > > >> > >> >> > > > this
> > > > > > >> > >> >> > > > a
> string as well so
> > > it can
> > > > be
> > > > > printed, set
> > > > > > >> via
> > > > > > >> > a UI, set
> > > > > > >> > >> in
> > > > > > >> > >> >> > config,
> > > > > > >> > >> >> > > etc.
> > > > > > >> > >> >> > > >
> Basically
> > encouraging
> > > > pluggable
> > > > > > serialization
> > > > > > >> > formats
> > > > > > >> > >> here will
> > > > > > >> > >> >> > lead
> > > > > > >> > >> >> > > to
> > > > > > >> > >> >> > > > a
> > > > > > >> > >> >> > > > bit
> of a tower of
> > > babel.
> > > > > > >> > >> >> > > > 2.
> This proposal
> > > still
> > > > includes a
> > > > > pretty
> > > > > > big
> > > > > > >> > change to our
> > > > > > >> > >> >> > >
> serialization
> > > > > > >> > >> >> > > > and
> protocol
> > > definition
> > > > layer.
> > > > > Essentially
> > > > > > >> it is
> > > > > > >> > >> introducing an
> > > > > > >> > >> >> > > optional
> > > > > > >> > >> >> > > >
> type, where the
> > > format is
> > > > data
> > > > > dependent. I
> > > > > > >> > think this is
> > > > > > >> > >> >> > actually a
> > > > > > >> > >> >> > > big
> > > > > > >> > >> >> > > >
> change though it
> > > doesn't
> > > > seem like
> > > > > it. It
> > > > > > >> means
> > > > > > >> > you can no
> > > > > > >> > >> >> > longer
> > > > > > >> > >> >> > > > specify
> > > > > > >> > >> >> > > > this
> type with our
> > > type
> > > > definition
> > > > > DSL, and
> > > > > > >> > likewise it
> > > > > > >> > >> requires
> > > > > > >> > >> >> > > custom
> > > > > > >> > >> >> > > >
> handling in client
> > > libs.
> > > > This isn't
> > > > > a huge
> > > > > > >> > thing, since
> > > > > > >> > >> the
> > > > > > >> > >> >> > Record
> > > > > > >> > >> >> > > >
> definition is
> > custom
> > > > anyway, but I
> > > > > think
> > > > > > this
> > > > > > >> > kind of
> > > > > > >> > >> protocol
> > > > > > >> > >> >> > > >
> inconsistency is
> > very
> > > > non-desirable
> > > > > and
> > > > > > ties
> > > > > > >> > you to
> > > > > > >> > >> hand-coding
> > > > > > >> > >> >> > > things.
> > > > > > >> > >> >> > > > I
> > > > > > >> > >> >> > > >
> think the type
> > should
> > > > instead by [Key
> > > > > > Value]
> > > > > > >> in
> > > > > > >> > our BNF,
> > > > > > >> > >> where
> > > > > > >> > >> >> > key and
> > > > > > >> > >> >> > > >
> value are both
> > short
> > > > strings as used
> > > > > > >> elsewhere.
> > > > > > >> > This
> > > > > > >> > >> brings it
> > > > > > >> > >> >> > in line
> > > > > > >> > >> >> > > > with
> > > > > > >> > >> >> > > > the
> rest of the
> > > protocol.
> > > > > > >> > >> >> > > > 3.
> Could we get
> > more
> > > > specific about
> > > > > the
> > > > > > exact
> > > > > > >> > Java API
> > > > > > >> > >> change to
> > > > > > >> > >> >> > > >
> ProducerRecord,
> > > > ConsumerRecord,
> > > > > Record,
> > > > > > etc?
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > > > -Jay
> > > > > > >> > >> >> > > >
> > > > > > >> > >> >> > >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> >
> > > > > > >> > >> >> > The information
> contained in
> > > this
> > > > email is
> > > > > strictly
> > > > > > >> > confidential and
> > > > > > >> > >> for
> > > > > > >> > >> >> > the use of the
> addressee only,
> > > unless
> > > > otherwise
> > > > > > >> indicated.
> > > > > > >> > If you
> > > > > > >> > >> are not
> > > > > > >> > >> >> > the intended
> recipient, please
> > > do not
> > > > read,
> > > > > copy, use
> > > > > > or
> > > > > > >> > disclose to
> > > > > > >> > >> others
> > > > > > >> > >> >> > this message or
> any
> > attachment.
> > > > Please also
> > > > > notify the
> > > > > > >> > sender by
> > > > > > >> > >> replying
> > > > > > >> > >> >> > to this email
> or by telephone
> > > > (+44(020 7896
> > > > > 0011) and
> > > > > > >> then
> > > > > > >> > delete
> > > > > > >> > >> the email
> > > > > > >> > >> >> > and any copies
> of it.
> > Opinions,
> > > > conclusion
> > > > > (etc) that
> > > > > > do
> > > > > > >> > not relate
> > > > > > >> > >> to the
> > > > > > >> > >> >> > official
> business of this
> > > company
> > > > shall be
> > > > > understood
> > > > > > as
> > > > > > >> > neither
> > > > > > >> > >> given nor
> > > > > > >> > >> >> > endorsed by it.
> IG is a
> > trading
> > > name
> > > > of IG
> > > > > Markets
> > > > > > >> Limited
> > > > > > >> > (a company
> > > > > > >> > >> >> > registered in
> England and
> > Wales,
> > > > company number
> > > > > > >> 04008957)
> > > > > > >> > and IG
> > > > > > >> > >> Index
> > > > > > >> > >> >> > Limited (a
> company registered
> > in
> > > > England and
> > > > > Wales,
> > > > > > >> > company number
> > > > > > >> > >> >> > 01190902).
> Registered address
> > at
> > > > Cannon Bridge
> > > > > House,
> > > > > > 25
> > > > > > >> > Dowgate
> > > > > > >> > >> Hill,
> > > > > > >> > >> >> > London EC4R
> 2YA. Both IG
> > Markets
> > > > Limited
> > > > > (register
> > > > > > >> number
> > > > > > >> > 195355)
> > > > > > >> > >> and IG
> > > > > > >> > >> >> > Index Limited
> (register number
> > > > 114059) are
> > > > > authorised
> > > > > > >> and
> > > > > > >> > regulated
> > > > > > >> > >> by the
> > > > > > >> > >> >> > Financial
> Conduct Authority.
> > > > > > >> > >> >> >
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >> --
> > > > > > >> > >> Nacho - Ignacio
> Solis -
> > > isolis@igso.net
> > > > > > >> > >>
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > The information contained in
> this email is
> > > strictly
> > > > > confidential and
> > > > > > for
> > > > > > >> > the use of the addressee only,
> unless
> > otherwise
> > > > indicated. If
> > > > > you are
> > > > > > >> not
> > > > > > >> > the intended recipient, please
> do not read,
> > > copy, use
> > > > or
> > > > > disclose to
> > > > > > >> others
> > > > > > >> > this message or any attachment.
> Please also
> > > notify
> > > > the sender by
> > > > > > >> replying
> > > > > > >> > to this email or by telephone
> (+44(020 7896
> > > 0011) and
> > > > then
> > > > > delete the
> > > > > > >> email
> > > > > > >> > and any copies of it. Opinions,
> conclusion
> > > (etc) that
> > > > do not
> > > > > relate to
> > > > > > >> the
> > > > > > >> > official business of this
> company shall be
> > > understood
> > > > as
> > > > > neither given
> > > > > > >> nor
> > > > > > >> > endorsed by it. IG is a trading
> name of IG
> > > Markets
> > > > Limited (a
> > > > > company
> > > > > > >> > registered in England and
> Wales, company
> > number
> > > > 04008957) and
> > > > > IG Index
> > > > > > >> > Limited (a company registered
> in England and
> > > Wales,
> > > > company
> > > > > number
> > > > > > >> > 01190902). Registered address
> at Cannon Bridge
> > > House,
> > > > 25
> > > > > Dowgate Hill,
> > > > > > >> > London EC4R 2YA. Both IG
> Markets Limited
> > > (register
> > > > number
> > > > > 195355) and
> > > > > > IG
> > > > > > >> > Index Limited (register number
> 114059) are
> > > authorised
> > > > and
> > > > > regulated by
> > > > > > >> the
> > > > > > >> > Financial Conduct Authority.
> > > > > > >> >
> > > > > > >> The information contained in this
> email is
> > > strictly
> > > > confidential
> > > > > and for
> > > > > > >> the use of the addressee only,
> unless otherwise
> > > > indicated. If you
> > > > > are
> > > > > > not
> > > > > > >> the intended recipient, please do
> not read,
> > copy,
> > > use
> > > > or disclose
> > > > > to
> > > > > > others
> > > > > > >> this message or any attachment.
> Please also
> > > notify the
> > > > sender by
> > > > > > replying
> > > > > > >> to this email or by telephone
> (+44(020 7896
> > 0011)
> > > and
> > > > then delete
> > > > > the
> > > > > > email
> > > > > > >> and any copies of it. Opinions,
> conclusion (etc)
> > > that
> > > > do not
> > > > > relate to
> > > > > > the
> > > > > > >> official business of this company
> shall be
> > > understood
> > > > as neither
> > > > > given
> > > > > > nor
> > > > > > >> endorsed by it. IG is a trading
> name of IG
> > Markets
> > > > Limited (a
> > > > > company
> > > > > > >> registered in England and Wales,
> company number
> > > > 04008957) and IG
> > > > > Index
> > > > > > >> Limited (a company registered in
> England and
> > > Wales,
> > > > company number
> > > > > > >> 01190902). Registered address at
> Cannon Bridge
> > > House,
> > > > 25 Dowgate
> > > > > Hill,
> > > > > > >> London EC4R 2YA. Both IG Markets
> Limited
> > (register
> > > > number 195355)
> > > > > and IG
> > > > > > >> Index Limited (register number
> 114059) are
> > > authorised
> > > > and
> > > > > regulated by
> > > > > > the
> > > > > > >> Financial Conduct Authority.
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > The information contained in this
> email is strictly
> > > > confidential and
> > > > > for the use of the addressee only, unless
> otherwise
> > > indicated.
> > > > If you are
> > > > > not the intended recipient, please do not
> read, copy, use
> > > or
> > > > disclose to
> > > > > others this message or any attachment.
> Please also notify
> > > the
> > > > sender by
> > > > > replying to this email or by telephone
> (+44(020 7896
> > 0011)
> > > and
> > > > then delete
> > > > > the email and any copies of it. Opinions,
> conclusion
> > (etc)
> > > that
> > > > do not
> > > > > relate to the official business of this
> company shall be
> > > > understood as
> > > > > neither given nor endorsed by it. IG is a
> trading name of
> > > IG
> > > > Markets
> > > > > Limited (a company registered in England
> and Wales,
> > company
> > > > number
> > > > > 04008957) and IG Index Limited (a company
> registered in
> > > England
> > > > and Wales,
> > > > > company number 01190902). Registered
> address at Cannon
> > > Bridge
> > > > House, 25
> > > > > Dowgate Hill, London EC4R 2YA. Both IG
> Markets Limited
> > > (register
> > > > number
> > > > > 195355) and IG Index Limited (register
> number 114059) are
> > > > authorised and
> > > > > regulated by the Financial Conduct
> Authority.
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > The information contained in this email is
> strictly
> > confidential
> > > and
> > > > for the use of the addressee only, unless otherwise
> indicated. If
> > > you are
> > > > not the intended recipient, please do not read,
> copy, use or
> > > disclose to
> > > > others this message or any attachment. Please also
> notify the
> > sender
> > > by
> > > > replying to this email or by telephone (+44(020 7896
> 0011) and then
> > > delete
> > > > the email and any copies of it. Opinions, conclusion
> (etc) that do
> > > not
> > > > relate to the official business of this company
> shall be understood
> > > as
> > > > neither given nor endorsed by it. IG is a trading
> name of IG
> > Markets
> > > > Limited (a company registered in England and Wales,
> company number
> > > > 04008957) and IG Index Limited (a company registered
> in England and
> > > Wales,
> > > > company number 01190902). Registered address at
> Cannon Bridge
> > House,
> > > 25
> > > > Dowgate Hill, London EC4R 2YA. Both IG Markets
> Limited (register
> > > number
> > > > 195355) and IG Index Limited (register number
> 114059) are
> > authorised
> > > and
> > > > regulated by the Financial Conduct Authority.
> > > >
> > > >
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential and for
> > > the use of the addressee only, unless otherwise indicated.
> If you are not
> > > the intended recipient, please do not read, copy, use or
> disclose to
> > others
> > > this message or any attachment. Please also notify the
> sender by replying
> > > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do
> not relate to
> > the
> > > official business of this company shall be understood as
> neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited
> (a company
> > > registered in England and Wales, company number 04008957)
> and IG Index
> > > Limited (a company registered in England and Wales,
> company number
> > > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG
> > > Index Limited (register number 114059) are authorised and
> regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
>
>
>
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Whilst if having an array of a Header[] would mean getting this through, im happy changing this, as already done.
Just going through how this is, I don’t believe this would provide much saving. If issue trying to resolve is around object/garbage creation of Map.Entry objects.
All you’re doing here is replacing the equiv of a HashMap.Node (Map.Entry) which simply holds reference to key and value objects with a custom variant.
On 19/02/2017, 20:22, "Michael Pearce" <Mi...@ig.com> wrote:
On point 1 & 2 Ive updated KIP to show varints (and removed the bit flag). (on the assumption KIP 98 is getting the agreement the protocol is moving from int32 to varInts as standard)
On point 3 ive updated to use an array of Header class, instead of a MultiMap in the Headers class object
On 19/02/2017, 20:06, "Michael Pearce" <Mi...@ig.com> wrote:
On points 1 and 2 I agree.
This also affects kip-98, I should expect this resolved before that vote also passes. If it is accepted there (I’m assuming this is getting discussed on that KIP? As you’re driving the move to VarInts), I am happy to make this KIP will simply follow suit to whatever is agreed in KIP-98.
On 3) Agreed this is a simpler form, as long as no one is expecting hashmap lookup performance ( O(1) ), I still would prefer an encapsulated class, so if we find that holding it as an array in future is causing some perf issues, the internals are not exposed to end users, allowing the internal structure to move to a map.
Can we compromise on?
class ConsumerRecord<K, V> {
K key;
V value;
Headers headers;
}
class Headers {
Header[] headers;
add(String key, byte[] value)
Collection<byte[]> get(String key)
}
class Header {
String key;
byte[] value;
}
On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> headers dont "leak" into application code. they are useful to application
> code as well.
This is exactly what I have been trying to get at. The use cases documented
here are middleware:
https://cwiki.apache.org/confluence/display/KAFKA/A+Case+for+Kafka+Headers.
If headers are intended for the application space as well, the document
should be updated accordingly so that it is an explicit design goal and not
an unfortunate design byproduct. There may be some disagreement on whether
it _should_ be a design goal, but it may not matter much since the current
interceptor API is probably insufficient for middleware applications (I
haven't thought of a way to do this that isn't cumbersome).
That aside, the three unresolved points for me are the following:
1. The use of varints. As a one-off for use only with record headers, the
case was weak, but if we are using them throughout the message format, then
we should do so here as well. The additional complexity is minimal and
early performance testing fully justifies their use.
2. If varints are used, the case for using attributes to indicate null
headers also becomes weak. We only add one additional byte in each message
if there are no headers. Whatever the case, let us be consistent with how
we treat null keys and values.
3. We have apparently agreed that the broker will validate headers (please
add this to the KIP). That being the case, I would prefer to use Kafka's
conventional format for arrays. The downside is that it takes more work to
skip over the headers on the consumer, though it's unclear if the cost
would matter in practice. Some concern was previously expressed about the
allocation of maps. An alternative would be to use arrays, i.e.
class ConsumerRecord<K, V> {
K key;
V value;
Header[] headers;
}
class Header {
String key;
byte[] value;
}
This would work nicely with the conventional array format and my guess is
it would obviate the need do any lazy initialization. If we use the map as
is currently documented, then it is possible with either representation to
slice the headers and initialize them lazily. Either way, it might be a
good idea to use a separate object to represent the headers in case we need
to modify it in the future in some way.
(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)
If we can resolve these points, then at least you will get my vote.
Thanks,
Jason
On Sun, Feb 19, 2017 at 7:30 AM, radai <ra...@gmail.com> wrote:
> headers dont "leak" into application code. they are useful to application
> code as well.
> IIUC samze currently has headers "in-V" and would just switch over to kafka
> headers if they exist.
> im sure plenty of other users of kafka would have a use for headers.
> im pretty sure use cases exist around shuffling data into/out-of kafka
> (kafka connect or equivalent) where metadata from one end could copied over
> to the other (S3, for example uses http headers for user-accessible
> metadata). it will be kafka client code getting/setting those headers. not
> an interceptor.
>
> On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > For APM single event tracing, need access to the header at the point of
> > processing on the processing thread.
> >
> > As such interceptors will not work/be suitable for these, due to the fact
> > they act on the ConsumerRecords as a batch, before the handling thread
> can
> > split out and process per message which is the point these tools will
> need
> > to continue to transaction tracing.
> >
> > Like wise tools and infra pieces will need access to the message outside
> > the interceptor realm.
> >
> >
> >
> > On 17/02/2017, 21:26, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> >
> >
> > So the broker will parse the headers and validate them. Good. The
> only
> > reason remaining that I can see to leave the headers as a byte array
> > is to
> > make it easier for the client to skip past them. Are we sure this is
> > not
> > premature optimization? Are there any performance results which show
> > that
> > this is worthwhile?
> >
> > What’s the issue with exposing a method getHeaders on the
> > producer/consumer
> > > record? It doesn’t break anything. We don’t need any special
> version.
> >
> >
> > See my previous explanation. What I am trying to resist is the
> headers
> > becoming a general application-level facility. The primary use case
> as
> > far
> > as I can tell is middleware, which is the use case that the
> > interceptors
> > are providing.
> >
> > Current batch consumer model and consumer interceptors don’t work
> where
> > > headers need to be acted on at per message level at time of
> > processing,
> > > very case is APM (the core one), where the header value is used to
> > continue
> > > tracing.
> >
> >
> > I still don't understand the point about batching. The consumer
> > records are
> > exposed as a batch in the consumer interceptor, but you can still
> > iterate
> > them individually. It is no different for the consumer API itself.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > Re:
> > >
> > > “ The point about creation of maps seems orthogonal. We can
> still
> > > represent
> > > the headers as a slice of bytes until the time it is accessed.”
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > >
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> > >
> > >
> > > On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > What’s the issue with exposing a method getHeaders on the
> > > producer/consumer record? It doesn’t break anything. We don’t need
> > any
> > > special version.
> > >
> > > Current batch consumer model and consumer interceptors don’t
> work
> > > where headers need to be acted on at per message level at time of
> > > processing, very case is APM (the core one), where the header value
> > is used
> > > to continue tracing. JMS/HTTP etc all expose these, without issues.
> > I would
> > > NOT want to lock this down to only be usable accessible via
> > interceptors,
> > > as we’d fail on one of the main goals.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > >
> > > On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > >
> > > The point about creation of maps seems orthogonal. We can
> > still
> > > represent
> > > the headers as a slice of bytes until the time it is
> > accessed.
> > >
> > >
> > > > Yes exactly we have access to the records thus why the
> > header
> > > should be
> > > > accessible via it and not hidden for only interceptors to
> > access.
> > >
> > >
> > > As explained above, the point is to make the intended usage
> > clear.
> > > Applications should continue to rely on the key/value
> fields
> > to
> > > serialize
> > > their own headers, and it would be more ideal if we can
> avoid
> > > leaking
> > > third-party headers into applications. This is difficult to
> > do
> > > with the
> > > current interceptors because they share the record objects
> > with
> > > the common
> > > API. What I had in mind is something like an extension of
> the
> > > current
> > > interceptors which exposed a different object (e.g.
> > > `RecordAndHeaders`).
> > > The challenge is for MM-like use cases. Let me see if I can
> > come
> > > up with a
> > > concrete proposal for that problem.
> > >
> > > -Jason
> > >
> > >
> > >
> > > On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > I am happy to move the definition of the header into the
> > message
> > > body, but
> > > > would cause us not to lazy initialise/parse the headers,
> as
> > > obviously, we
> > > > would have to traverse these reading the message.
> > > >
> > > > This was actually one of Jay’s requests:
> > > >
> > > > “ 2. I think we should think about creating the Map
> > lazily to
> > > avoid
> > > > parsing out all the headers into little objects.
> > HashMaps
> > > themselves
> > > > are
> > > > kind of expensive and the consumer is very perf
> > sensitive so
> > > and making
> > > > gazillions of hashmaps that may or may not get used
> is
> > > probably a bad
> > > > idea.”
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > Yes exactly we have access to the records thus why
> the
> > > header should
> > > > be accessible via it and not hidden for only interceptors
> > to
> > > access.
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Magnus Edenhill <ma...@edenhill.se>
> > > > Sent: Friday, February 17, 2017 7:34:49 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > Big +1 on VarInts.
> > > > CPUs are fast, memory is slow.
> > > >
> > > > I agree with Jason that we'll want to continue
> > verifying
> > > messages,
> > > > including their headers, so while I appreciate the
> > idea of
> > > the opaque
> > > > header blob it won't be useful in practice.
> > > >
> > > > /Magnus
> > > >
> > > > 2017-02-17 10:41 GMT-08:00 Jason Gustafson <
> > > jason@confluent.io>:
> > > >
> > > > > Sorry, my mistake. The consumer interceptor is per
> > batch,
> > > though I'm
> > > > not
> > > > > sure that's an actual limitation since you still
> have
> > > access to the
> > > > > individual records.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Re headers as byte array and future use by
> broker.
> > This
> > > doesn't
> > > > take away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >
> > > > > >
> > > > > > Yeah, I didn't say it was difficult, only
> awkward.
> > You
> > > wouldn't
> > > > write the
> > > > > > schema that way if you were planning to use it on
> > the
> > > brokers from
> > > > the
> > > > > > beginning. Note also that one of the benefits of
> > letting
> > > the broker
> > > > > > understand headers is that it can validate that
> > they are
> > > properly
> > > > > > formatted. If cost is the only concern, we should
> > > confirm its
> > > > impact
> > > > > > through performance testing.
> > > > > >
> > > > > > One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >
> > > > > >
> > > > > > I'm not sure I understand the point about
> batching.
> > > Interceptors
> > > > are
> > > > > > applied per-message, right?
> > > > > >
> > > > > > My intent on interceptors is to keep the usage of
> > headers
> > > > well-defined so
> > > > > > that they don't start leaking unnecessarily into
> > > applications. My
> > > > guess
> > > > > is
> > > > > > that it's probably inevitable, but isolating it
> in
> > the
> > > > interceptors would
> > > > > > at least give people a second thought before
> > deciding to
> > > use it.
> > > > The main
> > > > > > challenge in my mind is figuring out how an MM
> use
> > case
> > > would
> > > > work. It
> > > > > > would be more cumbersome to replicate headers
> > through an
> > > > interceptor,
> > > > > > though arguably MM should be working at a lower
> > level
> > > anyway.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Re headers available on the record va
> > interceptors only
> > > > > >>
> > > > > >> One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >>
> > > > > >> This anyhow is similar to jms/http/amqp where
> > headers
> > > are
> > > > available to
> > > > > >> consuming applications.
> > > > > >>
> > > > > >> Re headers as byte array and future use by
> > broker. This
> > > doesn't
> > > > take
> > > > > away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Sent using OWA for iPhone
> > > > > >> ________________________________________
> > > > > >> From: Jason Gustafson <ja...@confluent.io>
> > > > > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > > > > >> To: dev@kafka.apache.org
> > > > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > > >>
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >>
> > > > > >>
> > > > > >> Yes, that is what we are proposing, so using
> > varints
> > > for headers
> > > > would
> > > > > be
> > > > > >> consistent with the rest of the message. We have
> > used
> > > static
> > > > helper
> > > > > >> methods
> > > > > >> in our prototype implementation.
> > > > > >>
> > > > > >> The cost of parsing, we want to parse/interpret
> > the
> > > headers
> > > > lazily (this
> > > > > >> is
> > > > > >> > a key point brought up earlier in discussions)
> > > > > >>
> > > > > >>
> > > > > >> I'm a bit skeptical of this. Has anyone done the
> > > performance
> > > > testing? I
> > > > > >> can
> > > > > >> probably implement it and test it if no one else
> > has. I
> > > was also
> > > > under
> > > > > the
> > > > > >> impression that there may be use cases down the
> > road
> > > where the
> > > > broker
> > > > > >> would
> > > > > >> need to interpret headers. That wouldn't be off
> > the
> > > table in the
> > > > future
> > > > > if
> > > > > >> it's represented as bytes, but it would be quite
> > a bit
> > > more
> > > > awkward,
> > > > > >> right?
> > > > > >>
> > > > > >> By the way, one question I have been wondering
> > about. My
> > > > understanding
> > > > > is
> > > > > >> that headers are primarily for use cases where a
> > > third-party
> > > > components
> > > > > >> wants to enrich messages without needing to
> > understand
> > > or modify
> > > > the
> > > > > >> schema
> > > > > >> of the message key and value. For the
> applications
> > > which directly
> > > > > produce
> > > > > >> and consume the messages and control the
> key/value
> > > schema
> > > > directly, it
> > > > > >> seems we would rather have them implement
> headers
> > > directly in
> > > > their own
> > > > > >> schema. Supposing for the sake of argument that
> > it was
> > > possible,
> > > > my
> > > > > >> question is whether it be sufficient to expose
> the
> > > headers in the
> > > > > >> interceptor API and not in the common API?
> > > > > >>
> > > > > >> -Jason
> > > > > >>
> > > > > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On the point of varInts
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 11:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > On the point re: headers in the message
> > protocol
> > > being a
> > > > byte
> > > > > array
> > > > > >> > and not a count of elements followed by the
> > elements.
> > > Again
> > > > this was
> > > > > >> > discussed/argued previously.
> > > > > >> >
> > > > > >> > It was agreed on for a few reasons some of
> > which
> > > you have
> > > > > obviously
> > > > > >> > picked up on:
> > > > > >> >
> > > > > >> > Broker is able to pass it through opaquely
> > > > > >> > The cost of parsing, we want to
> > parse/interpret
> > > the headers
> > > > lazily
> > > > > >> > (this is a key point brought up earlier in
> > > discussions)
> > > > > >> > Headers can be copied from consumer record
> > to
> > > producer
> > > > record (aka
> > > > > >> > mirror makers etc) without parsing if no
> > changes are
> > > being made
> > > > or
> > > > > being
> > > > > >> > looked at.
> > > > > >> > Keeps the broker agnostic to the format
> > > > > >> > You need an int32 either for the byte size
> > of the
> > > headers,
> > > > or for
> > > > > >> the
> > > > > >> > count of elements, so overheads are the same,
> > but
> > > with going
> > > > with an
> > > > > >> opaque
> > > > > >> > byte array has the above advantages.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 02:50, "Jason Gustafson" <
> > > jason@confluent.io
> > > > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > Sorry, should have noted that the
> > performance
> > > testing
> > > > was done
> > > > > >> > using the
> > > > > >> > producer performance tool shipped with
> > Kafka.
> > > > > >> >
> > > > > >> > -Jason
> > > > > >> >
> > > > > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason
> > > Gustafson <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> >
> > > > > >> > > Hey Nacho,
> > > > > >> > >
> > > > > >> > > I've compared performance of our
> > KIP-98
> > > > implementation with
> > > > > >> and
> > > > > >> > without
> > > > > >> > > varints. For messages around 128
> > bytes, we
> > > see an
> > > > increase
> > > > > in
> > > > > >> > throughput of
> > > > > >> > > about 30% using the default
> > configuration
> > > settings.
> > > > At 256
> > > > > >> > bytes, the
> > > > > >> > > increase is around 16%. Obviously
> the
> > > performance
> > > > converge
> > > > > as
> > > > > >> > messages get
> > > > > >> > > larger, but it seems well worth the
> > cost.
> > > Note that
> > > > we are
> > > > > >> also
> > > > > >> > seeing a
> > > > > >> > > substantial performance increase
> > against
> > > trunk
> > > > primarily
> > > > > >> because
> > > > > >> > of the
> > > > > >> > > much more efficient packing that
> > varints
> > > provide us.
> > > > > Anything
> > > > > >> > adding to
> > > > > >> > > message overhead, such as record
> > headers,
> > > would only
> > > > > increase
> > > > > >> > the relative
> > > > > >> > > difference. (Of course take these
> > numbers
> > > with a
> > > > grain of
> > > > > salt
> > > > > >> > since I have
> > > > > >> > > only used the default settings with
> > both
> > > the producer
> > > > and
> > > > > >> broker
> > > > > >> > on my
> > > > > >> > > local machine. We intend to provide
> > more
> > > extensive
> > > > > performance
> > > > > >> > details as
> > > > > >> > > part of the work for KIP-98.)
> > > > > >> > >
> > > > > >> > > The implementation we are using is
> > from
> > > protobuf (
> > > > > >> > > https://developers.google.com/
> > > > > protocol-buffers/docs/encoding
> > > > > >> ),
> > > > > >> > which is
> > > > > >> > > also used in HBase. It is trivial to
> > > implement and as
> > > > far
> > > > > as I
> > > > > >> > know doesn't
> > > > > >> > > suffer from the aliasing problem you
> > are
> > > describing. I
> > > > > checked
> > > > > >> > with Magnus
> > > > > >> > > (the author of librdkafka) and he
> > agreed
> > > that the
> > > > savings
> > > > > >> seemed
> > > > > >> > worth the
> > > > > >> > > cost of implementation.
> > > > > >> > >
> > > > > >> > > -Jason
> > > > > >> > >
> > > > > >> > > On Thu, Feb 16, 2017 at 4:32 PM,
> > Ignacio
> > > Solis <
> > > > > >> isolis@igso.net>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> -VarInts
> > > > > >> > >>
> > > > > >> > >> I'm one of the people (if not the
> > most)
> > > opposed to
> > > > VarInts.
> > > > > >> > VarInts
> > > > > >> > >> have a place, but this is not it.
> > (We
> > > had a large
> > > > > >> discussion
> > > > > >> > about
> > > > > >> > >> them at the beginning of KIP-82
> time)
> > > > > >> > >>
> > > > > >> > >> If anybody has real life
> performance
> > > numbers of
> > > > VarInts
> > > > > >> > improving
> > > > > >> > >> things or significantly reducing
> > resources
> > > I would
> > > > like to
> > > > > >> know
> > > > > >> > what
> > > > > >> > >> that case may be. Yes, you can save
> > some
> > > bytes here
> > > > and
> > > > > >> there,
> > > > > >> > but
> > > > > >> > >> this is probably insignificant to
> the
> > > overall system
> > > > > behavior
> > > > > >> > and
> > > > > >> > >> storage requirements. -- I say
> this
> > with
> > > respect to
> > > > using
> > > > > >> > VarInts in
> > > > > >> > >> the protocol itself, not as part of
> > the
> > > data.
> > > > > >> > >>
> > > > > >> > >> VarInts require you to parse the
> Int
> > > before using it
> > > > and
> > > > > >> > depending on
> > > > > >> > >> the encoding they can suffer from
> > aliasing
> > > (multiple
> > > > > >> > representations
> > > > > >> > >> for the same value).
> > > > > >> > >>
> > > > > >> > >> Why add complexity?
> > > > > >> > >>
> > > > > >> > >> Nacho
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> On Thu, Feb 16, 2017 at 10:29 AM,
> > Colin
> > > McCabe <
> > > > > >> > cmccabe@apache.org>
> > > > > >> > >> wrote:
> > > > > >> > >> > +1 for varints here-- it would
> save
> > > quite a bit of
> > > > space.
> > > > > >> > They are
> > > > > >> > >> > pretty quick to implement as
> well.
> > > > > >> > >> >
> > > > > >> > >> > I think it makes sense for values
> > to be
> > > byte
> > > > arrays.
> > > > > Users
> > > > > >> > might want
> > > > > >> > >> > to attach arbitrary payloads;
> they
> > > shouldn't be
> > > > forced to
> > > > > >> > serialize
> > > > > >> > >> > everything to Java strings.
> > > > > >> > >> >
> > > > > >> > >> > best,
> > > > > >> > >> > Colin
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > On Thu, Feb 16, 2017, at 09:52,
> > Jason
> > > Gustafson
> > > > wrote:
> > > > > >> > >> >> Hey Michael,
> > > > > >> > >> >>
> > > > > >> > >> >> Hmm, I guess the point of
> > representing
> > > it as
> > > > bytes is to
> > > > > >> > allow the
> > > > > >> > >> broker
> > > > > >> > >> >> to pass it through opaquely? Is
> > the
> > > cost of
> > > > parsing
> > > > > them a
> > > > > >> > concern, or
> > > > > >> > >> >> are
> > > > > >> > >> >> we simply trying to ensure that
> > the
> > > broker stays
> > > > > agnostic
> > > > > >> to
> > > > > >> > the
> > > > > >> > >> format?
> > > > > >> > >> >>
> > > > > >> > >> >> On varints, I think adding
> > support for
> > > them makes
> > > > less
> > > > > >> sense
> > > > > >> > for an
> > > > > >> > >> >> isolated use case, but as part
> of
> > a
> > > more holistic
> > > > change
> > > > > >> > (such as what
> > > > > >> > >> we
> > > > > >> > >> >> have proposed in KIP-98), I
> think
> > they
> > > are
> > > > justifiable.
> > > > > If
> > > > > >> > we add them,
> > > > > >> > >> >> then the need to use attributes
> > becomes
> > > quite a
> > > > bit
> > > > > >> weaker,
> > > > > >> > right? The
> > > > > >> > >> >> other thing I find slightly odd
> > is the
> > > fact that
> > > > null
> > > > > >> > headers has no
> > > > > >> > >> >> actual
> > > > > >> > >> >> semantic meaning for the message
> > > (unlike null
> > > > keys and
> > > > > >> > values). It is
> > > > > >> > >> >> just
> > > > > >> > >> >> a space optimization. It seems a
> > bit
> > > better to
> > > > always
> > > > > use
> > > > > >> > size 0 to
> > > > > >> > >> >> indicate having no headers.
> > > > > >> > >> >>
> > > > > >> > >> >> Overall, the main point is
> > ensuring
> > > that the
> > > > message
> > > > > >> schema
> > > > > >> > remains
> > > > > >> > >> >> consistent, either within the
> > larger
> > > protocol, or
> > > > at a
> > > > > >> > minimum within
> > > > > >> > >> the
> > > > > >> > >> >> message itself.
> > > > > >> > >> >>
> > > > > >> > >> >> -Jason
> > > > > >> > >> >>
> > > > > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM,
> > > Michael Pearce <
> > > > > >> > Michael.Pearce@ig.com
> > > > > >> > >> >
> > > > > >> > >> >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> > Hi Jason,
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 1) in the message
> > protocol
> > > the headers
> > > > are
> > > > > >> simply
> > > > > >> > a byte
> > > > > >> > >> array,
> > > > > >> > >> >> > as like the key or value, this
> > is to
> > > clearly
> > > > demarcate
> > > > > >> the
> > > > > >> > header in
> > > > > >> > >> the
> > > > > >> > >> >> > core message. Then the header
> > byte
> > > array in the
> > > > core
> > > > > >> > message is an
> > > > > >> > >> array of
> > > > > >> > >> >> > key, value pairs. This is what
> > it is
> > > denoting.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Then this would be I guess in
> > the
> > > given
> > > > notation:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [KeyLength, Key,
> > > ValueLength, Value]
> > > > > >> > >> >> > KeyLength => int32
> > > <-----------------NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised key value
> > > > > >> > >> >> > Key => bytes
> > > <---------------------- NEW
> > > > > serialised
> > > > > >> > string (UTF8)
> > > > > >> > >> >> > bytes of the header key
> > > > > >> > >> >> > ValueLength => int32
> > > <-------------- NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised header value
> > > > > >> > >> >> > Value => bytes
> > > <-------------------- NEW
> > > > > serialised
> > > > > >> > form of the
> > > > > >> > >> header
> > > > > >> > >> >> > value
> > > > > >> > >> >> >
> > > > > >> > >> >> > The key length and value
> length
> > is
> > > matching the
> > > > way
> > > > > the
> > > > > >> > protocol is
> > > > > >> > >> >> > defined in the core message
> > currently.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 2)
> > > > > >> > >> >> > Var sized ints, this was
> > discussed
> > > much earlier
> > > > on, in
> > > > > >> > fact I had
> > > > > >> > >> >> > suggested it myself (with
> Hadoop
> > > references),
> > > > the
> > > > > >> > complexity of this
> > > > > >> > >> >> > compared to having a simpler
> > protocol
> > > was
> > > > argued and
> > > > > >> > agreed it
> > > > > >> > >> wasn’t worth
> > > > > >> > >> >> > the complexity as all other
> > clients
> > > in other
> > > > languages
> > > > > >> > would need to
> > > > > >> > >> ensure
> > > > > >> > >> >> > theyre using the right var
> size
> > > algorithm, as
> > > > there
> > > > > is a
> > > > > >> > few.
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 3)
> > > > > >> > >> >> > We did the attributes,
> optional
> > > approach as
> > > > originally
> > > > > >> > there was
> > > > > >> > >> marked
> > > > > >> > >> >> > concern that headers would
> > cause a
> > > message size
> > > > > overhead
> > > > > >> > for others,
> > > > > >> > >> who
> > > > > >> > >> >> > don’t want them. As such this
> > is the
> > > clean
> > > > solution to
> > > > > >> > achieve that.
> > > > > >> > >> If
> > > > > >> > >> >> > that no longer holds, and we
> > don’t
> > > care that we
> > > > add
> > > > > >> 4bytes
> > > > > >> > overhead,
> > > > > >> > >> then
> > > > > >> > >> >> > im happy to remove.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I’m personally in favour of
> > keeping
> > > the message
> > > > as
> > > > > small
> > > > > >> > as possible
> > > > > >> > >> so
> > > > > >> > >> >> > people don’t get shocks in
> perf
> > and
> > > throughputs
> > > > dues
> > > > > to
> > > > > >> > message size,
> > > > > >> > >> >> > unless they actively use the
> > feature,
> > > as such I
> > > > do
> > > > > >> prefer
> > > > > >> > the
> > > > > >> > >> attribute bit
> > > > > >> > >> >> > wise feature flag approach
> > myself.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On 16/02/2017, 05:40, "Jason
> > > Gustafson" <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > We have proposed a few
> > > significant changes
> > > > to the
> > > > > >> > message format
> > > > > >> > >> in
> > > > > >> > >> >> > KIP-98
> > > > > >> > >> >> > which now seems likely to
> > pass
> > > (perhaps
> > > > with some
> > > > > >> > iterations on
> > > > > >> > >> >> > implementation details).
> It
> > would
> > > be good
> > > > to try
> > > > > and
> > > > > >> > coordinate
> > > > > >> > >> the
> > > > > >> > >> >> > changes
> > > > > >> > >> >> > in both of the proposals
> to
> > make
> > > sure they
> > > > are
> > > > > >> > consistent and
> > > > > >> > >> >> > compatible.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I think using the
> > attributes to
> > > indicate
> > > > null
> > > > > >> headers
> > > > > >> > is a
> > > > > >> > >> reasonable
> > > > > >> > >> >> > approach. We have proposed
> > to do
> > > the same
> > > > thing
> > > > > for
> > > > > >> > the message
> > > > > >> > >> key and
> > > > > >> > >> >> > value. That said, I
> > sympathize
> > > with Jay's
> > > > > argument.
> > > > > >> > Having
> > > > > >> > >> multiple
> > > > > >> > >> >> > ways to
> > > > > >> > >> >> > specify a null value
> > increases
> > > the overall
> > > > > >> complexity
> > > > > >> > of the
> > > > > >> > >> protocol.
> > > > > >> > >> >> > You
> > > > > >> > >> >> > can see this just from the
> > fact
> > > that you
> > > > need the
> > > > > >> > extra verbiage
> > > > > >> > >> in the
> > > > > >> > >> >> > protocol specification in
> > this
> > > KIP and in
> > > > KIP-98
> > > > > to
> > > > > >> > describe the
> > > > > >> > >> >> > dependence
> > > > > >> > >> >> > between the fields and the
> > > attributes. It
> > > > seems
> > > > > >> like a
> > > > > >> > slippery
> > > > > >> > >> slope
> > > > > >> > >> >> > if
> > > > > >> > >> >> > you start allowing
> different
> > > request types
> > > > to
> > > > > >> > implement the
> > > > > >> > >> protocol
> > > > > >> > >> >> > specification differently.
> > > > > >> > >> >> >
> > > > > >> > >> >> > You can also argue that
> the
> > > messages
> > > > already are
> > > > > and
> > > > > >> > are likely
> > > > > >> > >> to
> > > > > >> > >> >> > remain a
> > > > > >> > >> >> > special case. For example,
> > there
> > > is
> > > > currently no
> > > > > >> > generality in
> > > > > >> > >> how
> > > > > >> > >> >> > compressed message sets
> are
> > > represented
> > > > that would
> > > > > >> be
> > > > > >> > applicable
> > > > > >> > >> for
> > > > > >> > >> >> > other
> > > > > >> > >> >> > request types. Some might
> > see this
> > > > divergence as
> > > > > an
> > > > > >> > unfortunate
> > > > > >> > >> >> > protocol
> > > > > >> > >> >> > deficiency which should be
> > fixed;
> > > others
> > > > might see
> > > > > >> it
> > > > > >> > as sort of
> > > > > >> > >> the
> > > > > >> > >> >> > inevitability of needing
> to
> > > optimize where
> > > > it
> > > > > counts
> > > > > >> > most. I'm
> > > > > >> > >> probably
> > > > > >> > >> >> > somewhere in between, but
> I
> > think
> > > we
> > > > probably all
> > > > > >> > share the
> > > > > >> > >> intuition
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the protocol should be
> kept
> > as
> > > consistent as
> > > > > >> possible.
> > > > > >> > With that
> > > > > >> > >> in
> > > > > >> > >> >> > mind,
> > > > > >> > >> >> > here are a few comments:
> > > > > >> > >> >> >
> > > > > >> > >> >> > 1. One thing I found a
> > little odd
> > > when
> > > > reading the
> > > > > >> > current
> > > > > >> > >> proposal is
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the headers are both
> > represented
> > > as an
> > > > array of
> > > > > >> bytes
> > > > > >> > and as an
> > > > > >> > >> array
> > > > > >> > >> >> > of
> > > > > >> > >> >> > key/value pairs. I'd
> > probably
> > > suggest
> > > > something
> > > > > like
> > > > > >> > this:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [HeaderKey
> > HeaderValue]
> > > > > >> > >> >> > HeaderKey => String
> > > > > >> > >> >> > HeaderValue => Bytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > An array in the Kafka
> > protocol is
> > > > represented as a
> > > > > >> > 4-byte integer
> > > > > >> > >> >> > indicating the number of
> > elements
> > > in the
> > > > array
> > > > > >> > followed by the
> > > > > >> > >> >> > serialization of the
> > elements.
> > > Unless I'm
> > > > > >> > misunderstanding, what
> > > > > >> > >> you
> > > > > >> > >> >> > have
> > > > > >> > >> >> > instead is the total size
> > of the
> > > headers in
> > > > bytes
> > > > > >> > followed by the
> > > > > >> > >> >> > elements.
> > > > > >> > >> >> > I'm not sure I see any
> > reason for
> > > this
> > > > > >> inconsistency.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 2. In KIP-98, we've
> > introduced
> > > > variable-length
> > > > > >> integer
> > > > > >> > fields.
> > > > > >> > >> >> > Effectively,
> > > > > >> > >> >> > we've enriched (or
> > "complicated"
> > > as Jay
> > > > might say
> > > > > ;)
> > > > > >> > the protocol
> > > > > >> > >> >> > specification to include
> the
> > > following
> > > > types:
> > > > > >> VarInt,
> > > > > >> > VarLong,
> > > > > >> > >> >> > UnsignedVarInt and
> > > UnsignedVarLong.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Along with these
> > primitives, we
> > > could
> > > > introduce
> > > > > the
> > > > > >> > following
> > > > > >> > >> types:
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeArray =>
> > NumberOfItems
> > > Item1 Item2
> > > > .. ItemN
> > > > > >> > >> >> > NumberOfItems =>
> > UnsignedVarInt
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeNullableArray =>
> > > NumberOfItemsOrNull
> > > > Item1
> > > > > >> > Item2 .. ItemN
> > > > > >> > >> >> > NumberOfItemsOrNull =>
> > VarInt
> > > (-1 means
> > > > null)
> > > > > >> > >> >> >
> > > > > >> > >> >> > And similarly for the
> > `String`
> > > and `Bytes`
> > > > types.
> > > > > >> > These types
> > > > > >> > >> can save
> > > > > >> > >> >> > a
> > > > > >> > >> >> > considerable amount of
> > space in
> > > this
> > > > proposal
> > > > > >> because
> > > > > >> > they can
> > > > > >> > >> be used
> > > > > >> > >> >> > for
> > > > > >> > >> >> > both the number of headers
> > > included in the
> > > > message
> > > > > >> and
> > > > > >> > the
> > > > > >> > >> lengths of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > header keys and values. We
> > could
> > > do this
> > > > instead:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers =>
> > VarSizeArray[HeaderKey
> > > > HeaderValue]
> > > > > >> > >> >> > HeaderKey =>
> VarSizeString
> > > > > >> > >> >> > HeaderValue =>
> > VarSizeBytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > Combining the savings from
> > the
> > > use of
> > > > variable
> > > > > >> length
> > > > > >> > fields, the
> > > > > >> > >> >> > benefit
> > > > > >> > >> >> > of using the attributes to
> > > represent null
> > > > seems
> > > > > >> pretty
> > > > > >> > small.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 3. Whichever way we go
> > (whether
> > > we use the
> > > > > >> attributes
> > > > > >> > or not), we
> > > > > >> > >> >> > should at
> > > > > >> > >> >> > least be consistent
> between
> > this
> > > KIP and
> > > > KIP-98.
> > > > > It
> > > > > >> > would be very
> > > > > >> > >> >> > strange
> > > > > >> > >> >> > to have two ways to
> > represent
> > > null values
> > > > in the
> > > > > >> same
> > > > > >> > schema.
> > > > > >> > >> Either
> > > > > >> > >> >> > way is
> > > > > >> > >> >> > OK with me. I think some
> > > message-level
> > > > > optimizations
> > > > > >> > are
> > > > > >> > >> justifiable,
> > > > > >> > >> >> > but
> > > > > >> > >> >> > the savings here seem
> > minimal (a
> > > few bytes
> > > > per
> > > > > >> > message), so
> > > > > >> > >> maybe it's
> > > > > >> > >> >> > not
> > > > > >> > >> >> > worth the cost of letting
> > the
> > > message
> > > > diverge even
> > > > > >> > further from
> > > > > >> > >> the
> > > > > >> > >> >> > rest of
> > > > > >> > >> >> > the protocol.
> > > > > >> > >> >> >
> > > > > >> > >> >> > -Jason
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On Wed, Feb 15, 2017 at
> > 8:52 AM,
> > > radai <
> > > > > >> > >> radai.rosenblatt@gmail.com>
> > > > > >> > >> >> > wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > > I've trimmed the inline
> > > contents as this
> > > > mail is
> > > > > >> > getting too
> > > > > >> > >> big for
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > apache mailing list
> > software to
> > > deliver
> > > > :-(
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 1. the important thing
> for
> > > > interoperability is
> > > > > for
> > > > > >> > different
> > > > > >> > >> >> > "interested
> > > > > >> > >> >> > > parties" (plugins, infra
> > > layers/wrappers,
> > > > > >> user-code)
> > > > > >> > to be
> > > > > >> > >> able to
> > > > > >> > >> >> > stick
> > > > > >> > >> >> > > pieces of metadata onto
> > msgs
> > > without
> > > > getting in
> > > > > >> each
> > > > > >> > other's
> > > > > >> > >> way. a
> > > > > >> > >> >> > common
> > > > > >> > >> >> > > key scheme (Strings, as
> > of the
> > > time of
> > > > this
> > > > > >> > writing?) is all
> > > > > >> > >> thats
> > > > > >> > >> >> > required
> > > > > >> > >> >> > > for that. it is assumed
> > that
> > > the other end
> > > > > >> > interested in any
> > > > > >> > >> such
> > > > > >> > >> >> > piece of
> > > > > >> > >> >> > > metadata knows the
> > encoding,
> > > and byte[]
> > > > provides
> > > > > >> for
> > > > > >> > the most
> > > > > >> > >> >> > flexibility.
> > > > > >> > >> >> > > i believe this is the
> same
> > > logic behind
> > > > core
> > > > > kafka
> > > > > >> > being
> > > > > >> > >> >> > byte[]/byte[] -
> > > > > >> > >> >> > > Strings are more
> "usable"
> > but
> > > bytes are
> > > > flexible
> > > > > >> and
> > > > > >> > so were
> > > > > >> > >> chosen.
> > > > > >> > >> >> > > Also - core kafka doesnt
> > even
> > > do that
> > > > good of a
> > > > > >> job
> > > > > >> > on
> > > > > >> > >> usability of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > payload (example - i
> have
> > to
> > > specify the
> > > > nop
> > > > > >> byte[]
> > > > > >> > "decoders"
> > > > > >> > >> >> > explicitly
> > > > > >> > >> >> > > in conf), and again
> > sacrificies
> > > usability
> > > > for
> > > > > the
> > > > > >> > sake of
> > > > > >> > >> >> > performance (no
> > > > > >> > >> >> > > convenient single-record
> > > processing as
> > > > poll is a
> > > > > >> > batch, lots of
> > > > > >> > >> >> > obscure
> > > > > >> > >> >> > > little config details
> > exposing
> > > internals
> > > > of the
> > > > > >> > batching
> > > > > >> > >> mechanism,
> > > > > >> > >> >> > etc)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > this is also why i
> really
> > > dislike the
> > > > idea of a
> > > > > >> > "type system"
> > > > > >> > >> for
> > > > > >> > >> >> > header
> > > > > >> > >> >> > > values, it further
> > degrades the
> > > > usability, adds
> > > > > >> > complexity and
> > > > > >> > >> will
> > > > > >> > >> >> > > eventually get in
> > people's way,
> > > also, it
> > > > would
> > > > > be
> > > > > >> > the 2nd/3rd
> > > > > >> > >> >> > home-group
> > > > > >> > >> >> > > serialization mechanism
> > in core
> > > kafka
> > > > (counting
> > > > > 2
> > > > > >> > iterations
> > > > > >> > >> of the
> > > > > >> > >> >> > "type
> > > > > >> > >> >> > > definition DSL")
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 2. this is an
> > implementation
> > > detail, and
> > > > not
> > > > > even
> > > > > >> a
> > > > > >> > very "user
> > > > > >> > >> >> > facing" one?
> > > > > >> > >> >> > > to the best of my
> > understanding
> > > the vote
> > > > process
> > > > > >> is
> > > > > >> > on proposed
> > > > > >> > >> >> > > API/behaviour. also -
> > since
> > > we're willing
> > > > to go
> > > > > >> with
> > > > > >> > strings
> > > > > >> > >> just
> > > > > >> > >> >> > serialize
> > > > > >> > >> >> > > a 0-sized header blob
> and
> > IIUC
> > > you dont
> > > > need any
> > > > > >> > optionals
> > > > > >> > >> anymore.
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 3. yes, we can :-)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > On Tue, Feb 14, 2017 at
> > 11:56
> > > PM, Michael
> > > > > Pearce <
> > > > > >> > >> >> > Michael.Pearce@ig.com>
> > > > > >> > >> >> > > wrote:
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > > Hi Jay,
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1) There was some
> > initial
> > > debate on the
> > > > value
> > > > > >> > part, as youll
> > > > > >> > >> note
> > > > > >> > >> >> > String,
> > > > > >> > >> >> > > > String headers were
> > > discounted early
> > > > on. The
> > > > > >> > reason for this
> > > > > >> > >> is
> > > > > >> > >> >> > > flexibility
> > > > > >> > >> >> > > > and keeping in line
> > with the
> > > > flexibility of
> > > > > key,
> > > > > >> > value of the
> > > > > >> > >> >> > message
> > > > > >> > >> >> > > > object itself. I don’t
> > think
> > > it takes
> > > > away
> > > > > from
> > > > > >> an
> > > > > >> > ecosystem
> > > > > >> > >> as
> > > > > >> > >> >> > each
> > > > > >> > >> >> > > plugin
> > > > > >> > >> >> > > > will care for their
> own
> > key,
> > > this way
> > > > ints,
> > > > > >> > booleans , exotic
> > > > > >> > >> >> > custom
> > > > > >> > >> >> > > binary
> > > > > >> > >> >> > > > can all be catered
> for=.
> > > > > >> > >> >> > > > a. If you really
> wanted
> > to
> > > push for a
> > > > typed
> > > > > >> value
> > > > > >> > interface,
> > > > > >> > >> I
> > > > > >> > >> >> > wouldn’t
> > > > > >> > >> >> > > > want just String
> values
> > > supported, but
> > > > the the
> > > > > >> > primatives
> > > > > >> > >> plus
> > > > > >> > >> >> > string and
> > > > > >> > >> >> > > > also still keeping the
> > > ability to have a
> > > > > binary
> > > > > >> > for custom
> > > > > >> > >> >> > binaries that
> > > > > >> > >> >> > > > some organisations may
> > have.
> > > > > >> > >> >> > > > i. I have written this
> > slight
> > > > alternative
> > > > > here,
> > > > > >> > >> >> > >
> https://cwiki.apache.org/
> > > > > >> > >> >> > > >
> > confluence/display/KAFKA/KIP-
> > > > > >> > 82+-+Add+Record+Headers+-+Typed
> > > > > >> > >> >> > > > ii. Essentially the
> > value
> > > bytes, has a
> > > > leading
> > > > > >> > byte overhead.
> > > > > >> > >> >> > > > 1. This tells you
> what
> > type
> > > the value
> > > > is,
> > > > > >> before
> > > > > >> > reading
> > > > > >> > >> the rest
> > > > > >> > >> >> > of the
> > > > > >> > >> >> > > > bytes, allowing
> > > > serialisation/deserialization
> > > > > to
> > > > > >> > and from the
> > > > > >> > >> >> > primitives,
> > > > > >> > >> >> > > > string and byte[].
> This
> > is
> > > akin to some
> > > > other
> > > > > >> > messaging
> > > > > >> > >> systems.
> > > > > >> > >> >> > > > 2) We are making it
> > optional,
> > > so that
> > > > for
> > > > > those
> > > > > >> > not wanting
> > > > > >> > >> >> > headers have
> > > > > >> > >> >> > > 0
> > > > > >> > >> >> > > > bytes overhead (think
> > of it
> > > as a feature
> > > > > flag),
> > > > > >> I
> > > > > >> > don’t
> > > > > >> > >> think this
> > > > > >> > >> >> > is
> > > > > >> > >> >> > > > complex, especially if
> > > comparing to
> > > > changes
> > > > > >> > proposed in
> > > > > >> > >> other kips
> > > > > >> > >> >> > like
> > > > > >> > >> >> > > > kip-98.
> > > > > >> > >> >> > > > a. If you really
> really
> > don’t
> > > like
> > > > this, we
> > > > > can
> > > > > >> > drop it, but
> > > > > >> > >> it
> > > > > >> > >> >> > would
> > > > > >> > >> >> > > mean
> > > > > >> > >> >> > > > buying into 4 bytes
> > extra
> > > overhead for
> > > > users
> > > > > who
> > > > > >> > do not want
> > > > > >> > >> to use
> > > > > >> > >> >> > > headers.
> > > > > >> > >> >> > > > 3) In the summary yes,
> > it is
> > > at a higher
> > > > > level,
> > > > > >> > but I think
> > > > > >> > >> this
> > > > > >> > >> >> > is well
> > > > > >> > >> >> > > > documented in the
> > proposed
> > > changes
> > > > section.
> > > > > >> > >> >> > > > a. Added getHeaders
> > method to
> > > > > Producer/Consumer
> > > > > >> > record (that
> > > > > >> > >> is it)
> > > > > >> > >> >> > > > b. We’ve also detailed
> > the
> > > new Headers
> > > > class
> > > > > >> that
> > > > > >> > this method
> > > > > >> > >> >> > returns
> > > > > >> > >> >> > > that
> > > > > >> > >> >> > > > encapsulates the
> headers
> > > protocol and
> > > > logic.
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Best,
> > > > > >> > >> >> > > > Mike
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > ==Original questions
> > from the
> > > vote
> > > > thread from
> > > > > >> > Jay.==
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Couple of things I
> > think we
> > > still need
> > > > to work
> > > > > >> out:
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1. I think we agree
> > about
> > > the key,
> > > > but I
> > > > > >> think
> > > > > >> > we haven't
> > > > > >> > >> >> > talked about
> > > > > >> > >> >> > > > the value yet. I
> > think if
> > > our goal
> > > > is an
> > > > > open
> > > > > >> > ecosystem
> > > > > >> > >> of these
> > > > > >> > >> >> > > header
> > > > > >> > >> >> > > > spread across many
> > plugins
> > > from many
> > > > > systems
> > > > > >> we
> > > > > >> > should
> > > > > >> > >> consider
> > > > > >> > >> >> > making
> > > > > >> > >> >> > > > this
> > > > > >> > >> >> > > > a string as well so
> > it can
> > > be
> > > > printed, set
> > > > > >> via
> > > > > >> > a UI, set
> > > > > >> > >> in
> > > > > >> > >> >> > config,
> > > > > >> > >> >> > > etc.
> > > > > >> > >> >> > > > Basically
> encouraging
> > > pluggable
> > > > > serialization
> > > > > >> > formats
> > > > > >> > >> here will
> > > > > >> > >> >> > lead
> > > > > >> > >> >> > > to
> > > > > >> > >> >> > > > a
> > > > > >> > >> >> > > > bit of a tower of
> > babel.
> > > > > >> > >> >> > > > 2. This proposal
> > still
> > > includes a
> > > > pretty
> > > > > big
> > > > > >> > change to our
> > > > > >> > >> >> > > serialization
> > > > > >> > >> >> > > > and protocol
> > definition
> > > layer.
> > > > Essentially
> > > > > >> it is
> > > > > >> > >> introducing an
> > > > > >> > >> >> > > optional
> > > > > >> > >> >> > > > type, where the
> > format is
> > > data
> > > > dependent. I
> > > > > >> > think this is
> > > > > >> > >> >> > actually a
> > > > > >> > >> >> > > big
> > > > > >> > >> >> > > > change though it
> > doesn't
> > > seem like
> > > > it. It
> > > > > >> means
> > > > > >> > you can no
> > > > > >> > >> >> > longer
> > > > > >> > >> >> > > > specify
> > > > > >> > >> >> > > > this type with our
> > type
> > > definition
> > > > DSL, and
> > > > > >> > likewise it
> > > > > >> > >> requires
> > > > > >> > >> >> > > custom
> > > > > >> > >> >> > > > handling in client
> > libs.
> > > This isn't
> > > > a huge
> > > > > >> > thing, since
> > > > > >> > >> the
> > > > > >> > >> >> > Record
> > > > > >> > >> >> > > > definition is
> custom
> > > anyway, but I
> > > > think
> > > > > this
> > > > > >> > kind of
> > > > > >> > >> protocol
> > > > > >> > >> >> > > > inconsistency is
> very
> > > non-desirable
> > > > and
> > > > > ties
> > > > > >> > you to
> > > > > >> > >> hand-coding
> > > > > >> > >> >> > > things.
> > > > > >> > >> >> > > > I
> > > > > >> > >> >> > > > think the type
> should
> > > instead by [Key
> > > > > Value]
> > > > > >> in
> > > > > >> > our BNF,
> > > > > >> > >> where
> > > > > >> > >> >> > key and
> > > > > >> > >> >> > > > value are both
> short
> > > strings as used
> > > > > >> elsewhere.
> > > > > >> > This
> > > > > >> > >> brings it
> > > > > >> > >> >> > in line
> > > > > >> > >> >> > > > with
> > > > > >> > >> >> > > > the rest of the
> > protocol.
> > > > > >> > >> >> > > > 3. Could we get
> more
> > > specific about
> > > > the
> > > > > exact
> > > > > >> > Java API
> > > > > >> > >> change to
> > > > > >> > >> >> > > > ProducerRecord,
> > > ConsumerRecord,
> > > > Record,
> > > > > etc?
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > -Jay
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > The information contained in
> > this
> > > email is
> > > > strictly
> > > > > >> > confidential and
> > > > > >> > >> for
> > > > > >> > >> >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > >> indicated.
> > > > > >> > If you
> > > > > >> > >> are not
> > > > > >> > >> >> > the intended recipient, please
> > do not
> > > read,
> > > > copy, use
> > > > > or
> > > > > >> > disclose to
> > > > > >> > >> others
> > > > > >> > >> >> > this message or any
> attachment.
> > > Please also
> > > > notify the
> > > > > >> > sender by
> > > > > >> > >> replying
> > > > > >> > >> >> > to this email or by telephone
> > > (+44(020 7896
> > > > 0011) and
> > > > > >> then
> > > > > >> > delete
> > > > > >> > >> the email
> > > > > >> > >> >> > and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc) that
> > > > > do
> > > > > >> > not relate
> > > > > >> > >> to the
> > > > > >> > >> >> > official business of this
> > company
> > > shall be
> > > > understood
> > > > > as
> > > > > >> > neither
> > > > > >> > >> given nor
> > > > > >> > >> >> > endorsed by it. IG is a
> trading
> > name
> > > of IG
> > > > Markets
> > > > > >> Limited
> > > > > >> > (a company
> > > > > >> > >> >> > registered in England and
> Wales,
> > > company number
> > > > > >> 04008957)
> > > > > >> > and IG
> > > > > >> > >> Index
> > > > > >> > >> >> > Limited (a company registered
> in
> > > England and
> > > > Wales,
> > > > > >> > company number
> > > > > >> > >> >> > 01190902). Registered address
> at
> > > Cannon Bridge
> > > > House,
> > > > > 25
> > > > > >> > Dowgate
> > > > > >> > >> Hill,
> > > > > >> > >> >> > London EC4R 2YA. Both IG
> Markets
> > > Limited
> > > > (register
> > > > > >> number
> > > > > >> > 195355)
> > > > > >> > >> and IG
> > > > > >> > >> >> > Index Limited (register number
> > > 114059) are
> > > > authorised
> > > > > >> and
> > > > > >> > regulated
> > > > > >> > >> by the
> > > > > >> > >> >> > Financial Conduct Authority.
> > > > > >> > >> >> >
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> --
> > > > > >> > >> Nacho - Ignacio Solis -
> > isolis@igso.net
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > The information contained in this email is
> > strictly
> > > > confidential and
> > > > > for
> > > > > >> > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you are
> > > > > >> not
> > > > > >> > the intended recipient, please do not read,
> > copy, use
> > > or
> > > > disclose to
> > > > > >> others
> > > > > >> > this message or any attachment. Please also
> > notify
> > > the sender by
> > > > > >> replying
> > > > > >> > to this email or by telephone (+44(020 7896
> > 0011) and
> > > then
> > > > delete the
> > > > > >> email
> > > > > >> > and any copies of it. Opinions, conclusion
> > (etc) that
> > > do not
> > > > relate to
> > > > > >> the
> > > > > >> > official business of this company shall be
> > understood
> > > as
> > > > neither given
> > > > > >> nor
> > > > > >> > endorsed by it. IG is a trading name of IG
> > Markets
> > > Limited (a
> > > > company
> > > > > >> > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG Index
> > > > > >> > Limited (a company registered in England and
> > Wales,
> > > company
> > > > number
> > > > > >> > 01190902). Registered address at Cannon Bridge
> > House,
> > > 25
> > > > Dowgate Hill,
> > > > > >> > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and
> > > > > IG
> > > > > >> > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > >> the
> > > > > >> > Financial Conduct Authority.
> > > > > >> >
> > > > > >> The information contained in this email is
> > strictly
> > > confidential
> > > > and for
> > > > > >> the use of the addressee only, unless otherwise
> > > indicated. If you
> > > > are
> > > > > not
> > > > > >> the intended recipient, please do not read,
> copy,
> > use
> > > or disclose
> > > > to
> > > > > others
> > > > > >> this message or any attachment. Please also
> > notify the
> > > sender by
> > > > > replying
> > > > > >> to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the
> > > > > email
> > > > > >> and any copies of it. Opinions, conclusion (etc)
> > that
> > > do not
> > > > relate to
> > > > > the
> > > > > >> official business of this company shall be
> > understood
> > > as neither
> > > > given
> > > > > nor
> > > > > >> endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > >> registered in England and Wales, company number
> > > 04008957) and IG
> > > > Index
> > > > > >> Limited (a company registered in England and
> > Wales,
> > > company number
> > > > > >> 01190902). Registered address at Cannon Bridge
> > House,
> > > 25 Dowgate
> > > > Hill,
> > > > > >> London EC4R 2YA. Both IG Markets Limited
> (register
> > > number 195355)
> > > > and IG
> > > > > >> Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > the
> > > > > >> Financial Conduct Authority.
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > The information contained in this email is strictly
> > > confidential and
> > > > for the use of the addressee only, unless otherwise
> > indicated.
> > > If you are
> > > > not the intended recipient, please do not read, copy, use
> > or
> > > disclose to
> > > > others this message or any attachment. Please also notify
> > the
> > > sender by
> > > > replying to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the email and any copies of it. Opinions, conclusion
> (etc)
> > that
> > > do not
> > > > relate to the official business of this company shall be
> > > understood as
> > > > neither given nor endorsed by it. IG is a trading name of
> > IG
> > > Markets
> > > > Limited (a company registered in England and Wales,
> company
> > > number
> > > > 04008957) and IG Index Limited (a company registered in
> > England
> > > and Wales,
> > > > company number 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and IG Index Limited (register number 114059) are
> > > authorised and
> > > > regulated by the Financial Conduct Authority.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential
> > and
> > > for the use of the addressee only, unless otherwise indicated. If
> > you are
> > > not the intended recipient, please do not read, copy, use or
> > disclose to
> > > others this message or any attachment. Please also notify the
> sender
> > by
> > > replying to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > relate to the official business of this company shall be understood
> > as
> > > neither given nor endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a company registered in England and Wales, company number
> > > 04008957) and IG Index Limited (a company registered in England and
> > Wales,
> > > company number 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > 195355) and IG Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
On point 1 & 2 Ive updated KIP to show varints (and removed the bit flag). (on the assumption KIP 98 is getting the agreement the protocol is moving from int32 to varInts as standard)
On point 3 ive updated to use an array of Header class, instead of a MultiMap in the Headers class object
On 19/02/2017, 20:06, "Michael Pearce" <Mi...@ig.com> wrote:
On points 1 and 2 I agree.
This also affects kip-98, I should expect this resolved before that vote also passes. If it is accepted there (I’m assuming this is getting discussed on that KIP? As you’re driving the move to VarInts), I am happy to make this KIP will simply follow suit to whatever is agreed in KIP-98.
On 3) Agreed this is a simpler form, as long as no one is expecting hashmap lookup performance ( O(1) ), I still would prefer an encapsulated class, so if we find that holding it as an array in future is causing some perf issues, the internals are not exposed to end users, allowing the internal structure to move to a map.
Can we compromise on?
class ConsumerRecord<K, V> {
K key;
V value;
Headers headers;
}
class Headers {
Header[] headers;
add(String key, byte[] value)
Collection<byte[]> get(String key)
}
class Header {
String key;
byte[] value;
}
On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> headers dont "leak" into application code. they are useful to application
> code as well.
This is exactly what I have been trying to get at. The use cases documented
here are middleware:
https://cwiki.apache.org/confluence/display/KAFKA/A+Case+for+Kafka+Headers.
If headers are intended for the application space as well, the document
should be updated accordingly so that it is an explicit design goal and not
an unfortunate design byproduct. There may be some disagreement on whether
it _should_ be a design goal, but it may not matter much since the current
interceptor API is probably insufficient for middleware applications (I
haven't thought of a way to do this that isn't cumbersome).
That aside, the three unresolved points for me are the following:
1. The use of varints. As a one-off for use only with record headers, the
case was weak, but if we are using them throughout the message format, then
we should do so here as well. The additional complexity is minimal and
early performance testing fully justifies their use.
2. If varints are used, the case for using attributes to indicate null
headers also becomes weak. We only add one additional byte in each message
if there are no headers. Whatever the case, let us be consistent with how
we treat null keys and values.
3. We have apparently agreed that the broker will validate headers (please
add this to the KIP). That being the case, I would prefer to use Kafka's
conventional format for arrays. The downside is that it takes more work to
skip over the headers on the consumer, though it's unclear if the cost
would matter in practice. Some concern was previously expressed about the
allocation of maps. An alternative would be to use arrays, i.e.
class ConsumerRecord<K, V> {
K key;
V value;
Header[] headers;
}
class Header {
String key;
byte[] value;
}
This would work nicely with the conventional array format and my guess is
it would obviate the need do any lazy initialization. If we use the map as
is currently documented, then it is possible with either representation to
slice the headers and initialize them lazily. Either way, it might be a
good idea to use a separate object to represent the headers in case we need
to modify it in the future in some way.
(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)
If we can resolve these points, then at least you will get my vote.
Thanks,
Jason
On Sun, Feb 19, 2017 at 7:30 AM, radai <ra...@gmail.com> wrote:
> headers dont "leak" into application code. they are useful to application
> code as well.
> IIUC samze currently has headers "in-V" and would just switch over to kafka
> headers if they exist.
> im sure plenty of other users of kafka would have a use for headers.
> im pretty sure use cases exist around shuffling data into/out-of kafka
> (kafka connect or equivalent) where metadata from one end could copied over
> to the other (S3, for example uses http headers for user-accessible
> metadata). it will be kafka client code getting/setting those headers. not
> an interceptor.
>
> On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > For APM single event tracing, need access to the header at the point of
> > processing on the processing thread.
> >
> > As such interceptors will not work/be suitable for these, due to the fact
> > they act on the ConsumerRecords as a batch, before the handling thread
> can
> > split out and process per message which is the point these tools will
> need
> > to continue to transaction tracing.
> >
> > Like wise tools and infra pieces will need access to the message outside
> > the interceptor realm.
> >
> >
> >
> > On 17/02/2017, 21:26, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> >
> >
> > So the broker will parse the headers and validate them. Good. The
> only
> > reason remaining that I can see to leave the headers as a byte array
> > is to
> > make it easier for the client to skip past them. Are we sure this is
> > not
> > premature optimization? Are there any performance results which show
> > that
> > this is worthwhile?
> >
> > What’s the issue with exposing a method getHeaders on the
> > producer/consumer
> > > record? It doesn’t break anything. We don’t need any special
> version.
> >
> >
> > See my previous explanation. What I am trying to resist is the
> headers
> > becoming a general application-level facility. The primary use case
> as
> > far
> > as I can tell is middleware, which is the use case that the
> > interceptors
> > are providing.
> >
> > Current batch consumer model and consumer interceptors don’t work
> where
> > > headers need to be acted on at per message level at time of
> > processing,
> > > very case is APM (the core one), where the header value is used to
> > continue
> > > tracing.
> >
> >
> > I still don't understand the point about batching. The consumer
> > records are
> > exposed as a batch in the consumer interceptor, but you can still
> > iterate
> > them individually. It is no different for the consumer API itself.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > Re:
> > >
> > > “ The point about creation of maps seems orthogonal. We can
> still
> > > represent
> > > the headers as a slice of bytes until the time it is accessed.”
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > >
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> > >
> > >
> > > On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > What’s the issue with exposing a method getHeaders on the
> > > producer/consumer record? It doesn’t break anything. We don’t need
> > any
> > > special version.
> > >
> > > Current batch consumer model and consumer interceptors don’t
> work
> > > where headers need to be acted on at per message level at time of
> > > processing, very case is APM (the core one), where the header value
> > is used
> > > to continue tracing. JMS/HTTP etc all expose these, without issues.
> > I would
> > > NOT want to lock this down to only be usable accessible via
> > interceptors,
> > > as we’d fail on one of the main goals.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > >
> > > On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > >
> > > The point about creation of maps seems orthogonal. We can
> > still
> > > represent
> > > the headers as a slice of bytes until the time it is
> > accessed.
> > >
> > >
> > > > Yes exactly we have access to the records thus why the
> > header
> > > should be
> > > > accessible via it and not hidden for only interceptors to
> > access.
> > >
> > >
> > > As explained above, the point is to make the intended usage
> > clear.
> > > Applications should continue to rely on the key/value
> fields
> > to
> > > serialize
> > > their own headers, and it would be more ideal if we can
> avoid
> > > leaking
> > > third-party headers into applications. This is difficult to
> > do
> > > with the
> > > current interceptors because they share the record objects
> > with
> > > the common
> > > API. What I had in mind is something like an extension of
> the
> > > current
> > > interceptors which exposed a different object (e.g.
> > > `RecordAndHeaders`).
> > > The challenge is for MM-like use cases. Let me see if I can
> > come
> > > up with a
> > > concrete proposal for that problem.
> > >
> > > -Jason
> > >
> > >
> > >
> > > On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > I am happy to move the definition of the header into the
> > message
> > > body, but
> > > > would cause us not to lazy initialise/parse the headers,
> as
> > > obviously, we
> > > > would have to traverse these reading the message.
> > > >
> > > > This was actually one of Jay’s requests:
> > > >
> > > > “ 2. I think we should think about creating the Map
> > lazily to
> > > avoid
> > > > parsing out all the headers into little objects.
> > HashMaps
> > > themselves
> > > > are
> > > > kind of expensive and the consumer is very perf
> > sensitive so
> > > and making
> > > > gazillions of hashmaps that may or may not get used
> is
> > > probably a bad
> > > > idea.”
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > Yes exactly we have access to the records thus why
> the
> > > header should
> > > > be accessible via it and not hidden for only interceptors
> > to
> > > access.
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Magnus Edenhill <ma...@edenhill.se>
> > > > Sent: Friday, February 17, 2017 7:34:49 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > Big +1 on VarInts.
> > > > CPUs are fast, memory is slow.
> > > >
> > > > I agree with Jason that we'll want to continue
> > verifying
> > > messages,
> > > > including their headers, so while I appreciate the
> > idea of
> > > the opaque
> > > > header blob it won't be useful in practice.
> > > >
> > > > /Magnus
> > > >
> > > > 2017-02-17 10:41 GMT-08:00 Jason Gustafson <
> > > jason@confluent.io>:
> > > >
> > > > > Sorry, my mistake. The consumer interceptor is per
> > batch,
> > > though I'm
> > > > not
> > > > > sure that's an actual limitation since you still
> have
> > > access to the
> > > > > individual records.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Re headers as byte array and future use by
> broker.
> > This
> > > doesn't
> > > > take away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >
> > > > > >
> > > > > > Yeah, I didn't say it was difficult, only
> awkward.
> > You
> > > wouldn't
> > > > write the
> > > > > > schema that way if you were planning to use it on
> > the
> > > brokers from
> > > > the
> > > > > > beginning. Note also that one of the benefits of
> > letting
> > > the broker
> > > > > > understand headers is that it can validate that
> > they are
> > > properly
> > > > > > formatted. If cost is the only concern, we should
> > > confirm its
> > > > impact
> > > > > > through performance testing.
> > > > > >
> > > > > > One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >
> > > > > >
> > > > > > I'm not sure I understand the point about
> batching.
> > > Interceptors
> > > > are
> > > > > > applied per-message, right?
> > > > > >
> > > > > > My intent on interceptors is to keep the usage of
> > headers
> > > > well-defined so
> > > > > > that they don't start leaking unnecessarily into
> > > applications. My
> > > > guess
> > > > > is
> > > > > > that it's probably inevitable, but isolating it
> in
> > the
> > > > interceptors would
> > > > > > at least give people a second thought before
> > deciding to
> > > use it.
> > > > The main
> > > > > > challenge in my mind is figuring out how an MM
> use
> > case
> > > would
> > > > work. It
> > > > > > would be more cumbersome to replicate headers
> > through an
> > > > interceptor,
> > > > > > though arguably MM should be working at a lower
> > level
> > > anyway.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Re headers available on the record va
> > interceptors only
> > > > > >>
> > > > > >> One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >>
> > > > > >> This anyhow is similar to jms/http/amqp where
> > headers
> > > are
> > > > available to
> > > > > >> consuming applications.
> > > > > >>
> > > > > >> Re headers as byte array and future use by
> > broker. This
> > > doesn't
> > > > take
> > > > > away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Sent using OWA for iPhone
> > > > > >> ________________________________________
> > > > > >> From: Jason Gustafson <ja...@confluent.io>
> > > > > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > > > > >> To: dev@kafka.apache.org
> > > > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > > >>
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >>
> > > > > >>
> > > > > >> Yes, that is what we are proposing, so using
> > varints
> > > for headers
> > > > would
> > > > > be
> > > > > >> consistent with the rest of the message. We have
> > used
> > > static
> > > > helper
> > > > > >> methods
> > > > > >> in our prototype implementation.
> > > > > >>
> > > > > >> The cost of parsing, we want to parse/interpret
> > the
> > > headers
> > > > lazily (this
> > > > > >> is
> > > > > >> > a key point brought up earlier in discussions)
> > > > > >>
> > > > > >>
> > > > > >> I'm a bit skeptical of this. Has anyone done the
> > > performance
> > > > testing? I
> > > > > >> can
> > > > > >> probably implement it and test it if no one else
> > has. I
> > > was also
> > > > under
> > > > > the
> > > > > >> impression that there may be use cases down the
> > road
> > > where the
> > > > broker
> > > > > >> would
> > > > > >> need to interpret headers. That wouldn't be off
> > the
> > > table in the
> > > > future
> > > > > if
> > > > > >> it's represented as bytes, but it would be quite
> > a bit
> > > more
> > > > awkward,
> > > > > >> right?
> > > > > >>
> > > > > >> By the way, one question I have been wondering
> > about. My
> > > > understanding
> > > > > is
> > > > > >> that headers are primarily for use cases where a
> > > third-party
> > > > components
> > > > > >> wants to enrich messages without needing to
> > understand
> > > or modify
> > > > the
> > > > > >> schema
> > > > > >> of the message key and value. For the
> applications
> > > which directly
> > > > > produce
> > > > > >> and consume the messages and control the
> key/value
> > > schema
> > > > directly, it
> > > > > >> seems we would rather have them implement
> headers
> > > directly in
> > > > their own
> > > > > >> schema. Supposing for the sake of argument that
> > it was
> > > possible,
> > > > my
> > > > > >> question is whether it be sufficient to expose
> the
> > > headers in the
> > > > > >> interceptor API and not in the common API?
> > > > > >>
> > > > > >> -Jason
> > > > > >>
> > > > > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On the point of varInts
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 11:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > On the point re: headers in the message
> > protocol
> > > being a
> > > > byte
> > > > > array
> > > > > >> > and not a count of elements followed by the
> > elements.
> > > Again
> > > > this was
> > > > > >> > discussed/argued previously.
> > > > > >> >
> > > > > >> > It was agreed on for a few reasons some of
> > which
> > > you have
> > > > > obviously
> > > > > >> > picked up on:
> > > > > >> >
> > > > > >> > Broker is able to pass it through opaquely
> > > > > >> > The cost of parsing, we want to
> > parse/interpret
> > > the headers
> > > > lazily
> > > > > >> > (this is a key point brought up earlier in
> > > discussions)
> > > > > >> > Headers can be copied from consumer record
> > to
> > > producer
> > > > record (aka
> > > > > >> > mirror makers etc) without parsing if no
> > changes are
> > > being made
> > > > or
> > > > > being
> > > > > >> > looked at.
> > > > > >> > Keeps the broker agnostic to the format
> > > > > >> > You need an int32 either for the byte size
> > of the
> > > headers,
> > > > or for
> > > > > >> the
> > > > > >> > count of elements, so overheads are the same,
> > but
> > > with going
> > > > with an
> > > > > >> opaque
> > > > > >> > byte array has the above advantages.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 02:50, "Jason Gustafson" <
> > > jason@confluent.io
> > > > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > Sorry, should have noted that the
> > performance
> > > testing
> > > > was done
> > > > > >> > using the
> > > > > >> > producer performance tool shipped with
> > Kafka.
> > > > > >> >
> > > > > >> > -Jason
> > > > > >> >
> > > > > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason
> > > Gustafson <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> >
> > > > > >> > > Hey Nacho,
> > > > > >> > >
> > > > > >> > > I've compared performance of our
> > KIP-98
> > > > implementation with
> > > > > >> and
> > > > > >> > without
> > > > > >> > > varints. For messages around 128
> > bytes, we
> > > see an
> > > > increase
> > > > > in
> > > > > >> > throughput of
> > > > > >> > > about 30% using the default
> > configuration
> > > settings.
> > > > At 256
> > > > > >> > bytes, the
> > > > > >> > > increase is around 16%. Obviously
> the
> > > performance
> > > > converge
> > > > > as
> > > > > >> > messages get
> > > > > >> > > larger, but it seems well worth the
> > cost.
> > > Note that
> > > > we are
> > > > > >> also
> > > > > >> > seeing a
> > > > > >> > > substantial performance increase
> > against
> > > trunk
> > > > primarily
> > > > > >> because
> > > > > >> > of the
> > > > > >> > > much more efficient packing that
> > varints
> > > provide us.
> > > > > Anything
> > > > > >> > adding to
> > > > > >> > > message overhead, such as record
> > headers,
> > > would only
> > > > > increase
> > > > > >> > the relative
> > > > > >> > > difference. (Of course take these
> > numbers
> > > with a
> > > > grain of
> > > > > salt
> > > > > >> > since I have
> > > > > >> > > only used the default settings with
> > both
> > > the producer
> > > > and
> > > > > >> broker
> > > > > >> > on my
> > > > > >> > > local machine. We intend to provide
> > more
> > > extensive
> > > > > performance
> > > > > >> > details as
> > > > > >> > > part of the work for KIP-98.)
> > > > > >> > >
> > > > > >> > > The implementation we are using is
> > from
> > > protobuf (
> > > > > >> > > https://developers.google.com/
> > > > > protocol-buffers/docs/encoding
> > > > > >> ),
> > > > > >> > which is
> > > > > >> > > also used in HBase. It is trivial to
> > > implement and as
> > > > far
> > > > > as I
> > > > > >> > know doesn't
> > > > > >> > > suffer from the aliasing problem you
> > are
> > > describing. I
> > > > > checked
> > > > > >> > with Magnus
> > > > > >> > > (the author of librdkafka) and he
> > agreed
> > > that the
> > > > savings
> > > > > >> seemed
> > > > > >> > worth the
> > > > > >> > > cost of implementation.
> > > > > >> > >
> > > > > >> > > -Jason
> > > > > >> > >
> > > > > >> > > On Thu, Feb 16, 2017 at 4:32 PM,
> > Ignacio
> > > Solis <
> > > > > >> isolis@igso.net>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> -VarInts
> > > > > >> > >>
> > > > > >> > >> I'm one of the people (if not the
> > most)
> > > opposed to
> > > > VarInts.
> > > > > >> > VarInts
> > > > > >> > >> have a place, but this is not it.
> > (We
> > > had a large
> > > > > >> discussion
> > > > > >> > about
> > > > > >> > >> them at the beginning of KIP-82
> time)
> > > > > >> > >>
> > > > > >> > >> If anybody has real life
> performance
> > > numbers of
> > > > VarInts
> > > > > >> > improving
> > > > > >> > >> things or significantly reducing
> > resources
> > > I would
> > > > like to
> > > > > >> know
> > > > > >> > what
> > > > > >> > >> that case may be. Yes, you can save
> > some
> > > bytes here
> > > > and
> > > > > >> there,
> > > > > >> > but
> > > > > >> > >> this is probably insignificant to
> the
> > > overall system
> > > > > behavior
> > > > > >> > and
> > > > > >> > >> storage requirements. -- I say
> this
> > with
> > > respect to
> > > > using
> > > > > >> > VarInts in
> > > > > >> > >> the protocol itself, not as part of
> > the
> > > data.
> > > > > >> > >>
> > > > > >> > >> VarInts require you to parse the
> Int
> > > before using it
> > > > and
> > > > > >> > depending on
> > > > > >> > >> the encoding they can suffer from
> > aliasing
> > > (multiple
> > > > > >> > representations
> > > > > >> > >> for the same value).
> > > > > >> > >>
> > > > > >> > >> Why add complexity?
> > > > > >> > >>
> > > > > >> > >> Nacho
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> On Thu, Feb 16, 2017 at 10:29 AM,
> > Colin
> > > McCabe <
> > > > > >> > cmccabe@apache.org>
> > > > > >> > >> wrote:
> > > > > >> > >> > +1 for varints here-- it would
> save
> > > quite a bit of
> > > > space.
> > > > > >> > They are
> > > > > >> > >> > pretty quick to implement as
> well.
> > > > > >> > >> >
> > > > > >> > >> > I think it makes sense for values
> > to be
> > > byte
> > > > arrays.
> > > > > Users
> > > > > >> > might want
> > > > > >> > >> > to attach arbitrary payloads;
> they
> > > shouldn't be
> > > > forced to
> > > > > >> > serialize
> > > > > >> > >> > everything to Java strings.
> > > > > >> > >> >
> > > > > >> > >> > best,
> > > > > >> > >> > Colin
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > On Thu, Feb 16, 2017, at 09:52,
> > Jason
> > > Gustafson
> > > > wrote:
> > > > > >> > >> >> Hey Michael,
> > > > > >> > >> >>
> > > > > >> > >> >> Hmm, I guess the point of
> > representing
> > > it as
> > > > bytes is to
> > > > > >> > allow the
> > > > > >> > >> broker
> > > > > >> > >> >> to pass it through opaquely? Is
> > the
> > > cost of
> > > > parsing
> > > > > them a
> > > > > >> > concern, or
> > > > > >> > >> >> are
> > > > > >> > >> >> we simply trying to ensure that
> > the
> > > broker stays
> > > > > agnostic
> > > > > >> to
> > > > > >> > the
> > > > > >> > >> format?
> > > > > >> > >> >>
> > > > > >> > >> >> On varints, I think adding
> > support for
> > > them makes
> > > > less
> > > > > >> sense
> > > > > >> > for an
> > > > > >> > >> >> isolated use case, but as part
> of
> > a
> > > more holistic
> > > > change
> > > > > >> > (such as what
> > > > > >> > >> we
> > > > > >> > >> >> have proposed in KIP-98), I
> think
> > they
> > > are
> > > > justifiable.
> > > > > If
> > > > > >> > we add them,
> > > > > >> > >> >> then the need to use attributes
> > becomes
> > > quite a
> > > > bit
> > > > > >> weaker,
> > > > > >> > right? The
> > > > > >> > >> >> other thing I find slightly odd
> > is the
> > > fact that
> > > > null
> > > > > >> > headers has no
> > > > > >> > >> >> actual
> > > > > >> > >> >> semantic meaning for the message
> > > (unlike null
> > > > keys and
> > > > > >> > values). It is
> > > > > >> > >> >> just
> > > > > >> > >> >> a space optimization. It seems a
> > bit
> > > better to
> > > > always
> > > > > use
> > > > > >> > size 0 to
> > > > > >> > >> >> indicate having no headers.
> > > > > >> > >> >>
> > > > > >> > >> >> Overall, the main point is
> > ensuring
> > > that the
> > > > message
> > > > > >> schema
> > > > > >> > remains
> > > > > >> > >> >> consistent, either within the
> > larger
> > > protocol, or
> > > > at a
> > > > > >> > minimum within
> > > > > >> > >> the
> > > > > >> > >> >> message itself.
> > > > > >> > >> >>
> > > > > >> > >> >> -Jason
> > > > > >> > >> >>
> > > > > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM,
> > > Michael Pearce <
> > > > > >> > Michael.Pearce@ig.com
> > > > > >> > >> >
> > > > > >> > >> >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> > Hi Jason,
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 1) in the message
> > protocol
> > > the headers
> > > > are
> > > > > >> simply
> > > > > >> > a byte
> > > > > >> > >> array,
> > > > > >> > >> >> > as like the key or value, this
> > is to
> > > clearly
> > > > demarcate
> > > > > >> the
> > > > > >> > header in
> > > > > >> > >> the
> > > > > >> > >> >> > core message. Then the header
> > byte
> > > array in the
> > > > core
> > > > > >> > message is an
> > > > > >> > >> array of
> > > > > >> > >> >> > key, value pairs. This is what
> > it is
> > > denoting.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Then this would be I guess in
> > the
> > > given
> > > > notation:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [KeyLength, Key,
> > > ValueLength, Value]
> > > > > >> > >> >> > KeyLength => int32
> > > <-----------------NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised key value
> > > > > >> > >> >> > Key => bytes
> > > <---------------------- NEW
> > > > > serialised
> > > > > >> > string (UTF8)
> > > > > >> > >> >> > bytes of the header key
> > > > > >> > >> >> > ValueLength => int32
> > > <-------------- NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised header value
> > > > > >> > >> >> > Value => bytes
> > > <-------------------- NEW
> > > > > serialised
> > > > > >> > form of the
> > > > > >> > >> header
> > > > > >> > >> >> > value
> > > > > >> > >> >> >
> > > > > >> > >> >> > The key length and value
> length
> > is
> > > matching the
> > > > way
> > > > > the
> > > > > >> > protocol is
> > > > > >> > >> >> > defined in the core message
> > currently.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 2)
> > > > > >> > >> >> > Var sized ints, this was
> > discussed
> > > much earlier
> > > > on, in
> > > > > >> > fact I had
> > > > > >> > >> >> > suggested it myself (with
> Hadoop
> > > references),
> > > > the
> > > > > >> > complexity of this
> > > > > >> > >> >> > compared to having a simpler
> > protocol
> > > was
> > > > argued and
> > > > > >> > agreed it
> > > > > >> > >> wasn’t worth
> > > > > >> > >> >> > the complexity as all other
> > clients
> > > in other
> > > > languages
> > > > > >> > would need to
> > > > > >> > >> ensure
> > > > > >> > >> >> > theyre using the right var
> size
> > > algorithm, as
> > > > there
> > > > > is a
> > > > > >> > few.
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 3)
> > > > > >> > >> >> > We did the attributes,
> optional
> > > approach as
> > > > originally
> > > > > >> > there was
> > > > > >> > >> marked
> > > > > >> > >> >> > concern that headers would
> > cause a
> > > message size
> > > > > overhead
> > > > > >> > for others,
> > > > > >> > >> who
> > > > > >> > >> >> > don’t want them. As such this
> > is the
> > > clean
> > > > solution to
> > > > > >> > achieve that.
> > > > > >> > >> If
> > > > > >> > >> >> > that no longer holds, and we
> > don’t
> > > care that we
> > > > add
> > > > > >> 4bytes
> > > > > >> > overhead,
> > > > > >> > >> then
> > > > > >> > >> >> > im happy to remove.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I’m personally in favour of
> > keeping
> > > the message
> > > > as
> > > > > small
> > > > > >> > as possible
> > > > > >> > >> so
> > > > > >> > >> >> > people don’t get shocks in
> perf
> > and
> > > throughputs
> > > > dues
> > > > > to
> > > > > >> > message size,
> > > > > >> > >> >> > unless they actively use the
> > feature,
> > > as such I
> > > > do
> > > > > >> prefer
> > > > > >> > the
> > > > > >> > >> attribute bit
> > > > > >> > >> >> > wise feature flag approach
> > myself.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On 16/02/2017, 05:40, "Jason
> > > Gustafson" <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > We have proposed a few
> > > significant changes
> > > > to the
> > > > > >> > message format
> > > > > >> > >> in
> > > > > >> > >> >> > KIP-98
> > > > > >> > >> >> > which now seems likely to
> > pass
> > > (perhaps
> > > > with some
> > > > > >> > iterations on
> > > > > >> > >> >> > implementation details).
> It
> > would
> > > be good
> > > > to try
> > > > > and
> > > > > >> > coordinate
> > > > > >> > >> the
> > > > > >> > >> >> > changes
> > > > > >> > >> >> > in both of the proposals
> to
> > make
> > > sure they
> > > > are
> > > > > >> > consistent and
> > > > > >> > >> >> > compatible.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I think using the
> > attributes to
> > > indicate
> > > > null
> > > > > >> headers
> > > > > >> > is a
> > > > > >> > >> reasonable
> > > > > >> > >> >> > approach. We have proposed
> > to do
> > > the same
> > > > thing
> > > > > for
> > > > > >> > the message
> > > > > >> > >> key and
> > > > > >> > >> >> > value. That said, I
> > sympathize
> > > with Jay's
> > > > > argument.
> > > > > >> > Having
> > > > > >> > >> multiple
> > > > > >> > >> >> > ways to
> > > > > >> > >> >> > specify a null value
> > increases
> > > the overall
> > > > > >> complexity
> > > > > >> > of the
> > > > > >> > >> protocol.
> > > > > >> > >> >> > You
> > > > > >> > >> >> > can see this just from the
> > fact
> > > that you
> > > > need the
> > > > > >> > extra verbiage
> > > > > >> > >> in the
> > > > > >> > >> >> > protocol specification in
> > this
> > > KIP and in
> > > > KIP-98
> > > > > to
> > > > > >> > describe the
> > > > > >> > >> >> > dependence
> > > > > >> > >> >> > between the fields and the
> > > attributes. It
> > > > seems
> > > > > >> like a
> > > > > >> > slippery
> > > > > >> > >> slope
> > > > > >> > >> >> > if
> > > > > >> > >> >> > you start allowing
> different
> > > request types
> > > > to
> > > > > >> > implement the
> > > > > >> > >> protocol
> > > > > >> > >> >> > specification differently.
> > > > > >> > >> >> >
> > > > > >> > >> >> > You can also argue that
> the
> > > messages
> > > > already are
> > > > > and
> > > > > >> > are likely
> > > > > >> > >> to
> > > > > >> > >> >> > remain a
> > > > > >> > >> >> > special case. For example,
> > there
> > > is
> > > > currently no
> > > > > >> > generality in
> > > > > >> > >> how
> > > > > >> > >> >> > compressed message sets
> are
> > > represented
> > > > that would
> > > > > >> be
> > > > > >> > applicable
> > > > > >> > >> for
> > > > > >> > >> >> > other
> > > > > >> > >> >> > request types. Some might
> > see this
> > > > divergence as
> > > > > an
> > > > > >> > unfortunate
> > > > > >> > >> >> > protocol
> > > > > >> > >> >> > deficiency which should be
> > fixed;
> > > others
> > > > might see
> > > > > >> it
> > > > > >> > as sort of
> > > > > >> > >> the
> > > > > >> > >> >> > inevitability of needing
> to
> > > optimize where
> > > > it
> > > > > counts
> > > > > >> > most. I'm
> > > > > >> > >> probably
> > > > > >> > >> >> > somewhere in between, but
> I
> > think
> > > we
> > > > probably all
> > > > > >> > share the
> > > > > >> > >> intuition
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the protocol should be
> kept
> > as
> > > consistent as
> > > > > >> possible.
> > > > > >> > With that
> > > > > >> > >> in
> > > > > >> > >> >> > mind,
> > > > > >> > >> >> > here are a few comments:
> > > > > >> > >> >> >
> > > > > >> > >> >> > 1. One thing I found a
> > little odd
> > > when
> > > > reading the
> > > > > >> > current
> > > > > >> > >> proposal is
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the headers are both
> > represented
> > > as an
> > > > array of
> > > > > >> bytes
> > > > > >> > and as an
> > > > > >> > >> array
> > > > > >> > >> >> > of
> > > > > >> > >> >> > key/value pairs. I'd
> > probably
> > > suggest
> > > > something
> > > > > like
> > > > > >> > this:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [HeaderKey
> > HeaderValue]
> > > > > >> > >> >> > HeaderKey => String
> > > > > >> > >> >> > HeaderValue => Bytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > An array in the Kafka
> > protocol is
> > > > represented as a
> > > > > >> > 4-byte integer
> > > > > >> > >> >> > indicating the number of
> > elements
> > > in the
> > > > array
> > > > > >> > followed by the
> > > > > >> > >> >> > serialization of the
> > elements.
> > > Unless I'm
> > > > > >> > misunderstanding, what
> > > > > >> > >> you
> > > > > >> > >> >> > have
> > > > > >> > >> >> > instead is the total size
> > of the
> > > headers in
> > > > bytes
> > > > > >> > followed by the
> > > > > >> > >> >> > elements.
> > > > > >> > >> >> > I'm not sure I see any
> > reason for
> > > this
> > > > > >> inconsistency.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 2. In KIP-98, we've
> > introduced
> > > > variable-length
> > > > > >> integer
> > > > > >> > fields.
> > > > > >> > >> >> > Effectively,
> > > > > >> > >> >> > we've enriched (or
> > "complicated"
> > > as Jay
> > > > might say
> > > > > ;)
> > > > > >> > the protocol
> > > > > >> > >> >> > specification to include
> the
> > > following
> > > > types:
> > > > > >> VarInt,
> > > > > >> > VarLong,
> > > > > >> > >> >> > UnsignedVarInt and
> > > UnsignedVarLong.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Along with these
> > primitives, we
> > > could
> > > > introduce
> > > > > the
> > > > > >> > following
> > > > > >> > >> types:
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeArray =>
> > NumberOfItems
> > > Item1 Item2
> > > > .. ItemN
> > > > > >> > >> >> > NumberOfItems =>
> > UnsignedVarInt
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeNullableArray =>
> > > NumberOfItemsOrNull
> > > > Item1
> > > > > >> > Item2 .. ItemN
> > > > > >> > >> >> > NumberOfItemsOrNull =>
> > VarInt
> > > (-1 means
> > > > null)
> > > > > >> > >> >> >
> > > > > >> > >> >> > And similarly for the
> > `String`
> > > and `Bytes`
> > > > types.
> > > > > >> > These types
> > > > > >> > >> can save
> > > > > >> > >> >> > a
> > > > > >> > >> >> > considerable amount of
> > space in
> > > this
> > > > proposal
> > > > > >> because
> > > > > >> > they can
> > > > > >> > >> be used
> > > > > >> > >> >> > for
> > > > > >> > >> >> > both the number of headers
> > > included in the
> > > > message
> > > > > >> and
> > > > > >> > the
> > > > > >> > >> lengths of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > header keys and values. We
> > could
> > > do this
> > > > instead:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers =>
> > VarSizeArray[HeaderKey
> > > > HeaderValue]
> > > > > >> > >> >> > HeaderKey =>
> VarSizeString
> > > > > >> > >> >> > HeaderValue =>
> > VarSizeBytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > Combining the savings from
> > the
> > > use of
> > > > variable
> > > > > >> length
> > > > > >> > fields, the
> > > > > >> > >> >> > benefit
> > > > > >> > >> >> > of using the attributes to
> > > represent null
> > > > seems
> > > > > >> pretty
> > > > > >> > small.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 3. Whichever way we go
> > (whether
> > > we use the
> > > > > >> attributes
> > > > > >> > or not), we
> > > > > >> > >> >> > should at
> > > > > >> > >> >> > least be consistent
> between
> > this
> > > KIP and
> > > > KIP-98.
> > > > > It
> > > > > >> > would be very
> > > > > >> > >> >> > strange
> > > > > >> > >> >> > to have two ways to
> > represent
> > > null values
> > > > in the
> > > > > >> same
> > > > > >> > schema.
> > > > > >> > >> Either
> > > > > >> > >> >> > way is
> > > > > >> > >> >> > OK with me. I think some
> > > message-level
> > > > > optimizations
> > > > > >> > are
> > > > > >> > >> justifiable,
> > > > > >> > >> >> > but
> > > > > >> > >> >> > the savings here seem
> > minimal (a
> > > few bytes
> > > > per
> > > > > >> > message), so
> > > > > >> > >> maybe it's
> > > > > >> > >> >> > not
> > > > > >> > >> >> > worth the cost of letting
> > the
> > > message
> > > > diverge even
> > > > > >> > further from
> > > > > >> > >> the
> > > > > >> > >> >> > rest of
> > > > > >> > >> >> > the protocol.
> > > > > >> > >> >> >
> > > > > >> > >> >> > -Jason
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On Wed, Feb 15, 2017 at
> > 8:52 AM,
> > > radai <
> > > > > >> > >> radai.rosenblatt@gmail.com>
> > > > > >> > >> >> > wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > > I've trimmed the inline
> > > contents as this
> > > > mail is
> > > > > >> > getting too
> > > > > >> > >> big for
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > apache mailing list
> > software to
> > > deliver
> > > > :-(
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 1. the important thing
> for
> > > > interoperability is
> > > > > for
> > > > > >> > different
> > > > > >> > >> >> > "interested
> > > > > >> > >> >> > > parties" (plugins, infra
> > > layers/wrappers,
> > > > > >> user-code)
> > > > > >> > to be
> > > > > >> > >> able to
> > > > > >> > >> >> > stick
> > > > > >> > >> >> > > pieces of metadata onto
> > msgs
> > > without
> > > > getting in
> > > > > >> each
> > > > > >> > other's
> > > > > >> > >> way. a
> > > > > >> > >> >> > common
> > > > > >> > >> >> > > key scheme (Strings, as
> > of the
> > > time of
> > > > this
> > > > > >> > writing?) is all
> > > > > >> > >> thats
> > > > > >> > >> >> > required
> > > > > >> > >> >> > > for that. it is assumed
> > that
> > > the other end
> > > > > >> > interested in any
> > > > > >> > >> such
> > > > > >> > >> >> > piece of
> > > > > >> > >> >> > > metadata knows the
> > encoding,
> > > and byte[]
> > > > provides
> > > > > >> for
> > > > > >> > the most
> > > > > >> > >> >> > flexibility.
> > > > > >> > >> >> > > i believe this is the
> same
> > > logic behind
> > > > core
> > > > > kafka
> > > > > >> > being
> > > > > >> > >> >> > byte[]/byte[] -
> > > > > >> > >> >> > > Strings are more
> "usable"
> > but
> > > bytes are
> > > > flexible
> > > > > >> and
> > > > > >> > so were
> > > > > >> > >> chosen.
> > > > > >> > >> >> > > Also - core kafka doesnt
> > even
> > > do that
> > > > good of a
> > > > > >> job
> > > > > >> > on
> > > > > >> > >> usability of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > payload (example - i
> have
> > to
> > > specify the
> > > > nop
> > > > > >> byte[]
> > > > > >> > "decoders"
> > > > > >> > >> >> > explicitly
> > > > > >> > >> >> > > in conf), and again
> > sacrificies
> > > usability
> > > > for
> > > > > the
> > > > > >> > sake of
> > > > > >> > >> >> > performance (no
> > > > > >> > >> >> > > convenient single-record
> > > processing as
> > > > poll is a
> > > > > >> > batch, lots of
> > > > > >> > >> >> > obscure
> > > > > >> > >> >> > > little config details
> > exposing
> > > internals
> > > > of the
> > > > > >> > batching
> > > > > >> > >> mechanism,
> > > > > >> > >> >> > etc)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > this is also why i
> really
> > > dislike the
> > > > idea of a
> > > > > >> > "type system"
> > > > > >> > >> for
> > > > > >> > >> >> > header
> > > > > >> > >> >> > > values, it further
> > degrades the
> > > > usability, adds
> > > > > >> > complexity and
> > > > > >> > >> will
> > > > > >> > >> >> > > eventually get in
> > people's way,
> > > also, it
> > > > would
> > > > > be
> > > > > >> > the 2nd/3rd
> > > > > >> > >> >> > home-group
> > > > > >> > >> >> > > serialization mechanism
> > in core
> > > kafka
> > > > (counting
> > > > > 2
> > > > > >> > iterations
> > > > > >> > >> of the
> > > > > >> > >> >> > "type
> > > > > >> > >> >> > > definition DSL")
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 2. this is an
> > implementation
> > > detail, and
> > > > not
> > > > > even
> > > > > >> a
> > > > > >> > very "user
> > > > > >> > >> >> > facing" one?
> > > > > >> > >> >> > > to the best of my
> > understanding
> > > the vote
> > > > process
> > > > > >> is
> > > > > >> > on proposed
> > > > > >> > >> >> > > API/behaviour. also -
> > since
> > > we're willing
> > > > to go
> > > > > >> with
> > > > > >> > strings
> > > > > >> > >> just
> > > > > >> > >> >> > serialize
> > > > > >> > >> >> > > a 0-sized header blob
> and
> > IIUC
> > > you dont
> > > > need any
> > > > > >> > optionals
> > > > > >> > >> anymore.
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 3. yes, we can :-)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > On Tue, Feb 14, 2017 at
> > 11:56
> > > PM, Michael
> > > > > Pearce <
> > > > > >> > >> >> > Michael.Pearce@ig.com>
> > > > > >> > >> >> > > wrote:
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > > Hi Jay,
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1) There was some
> > initial
> > > debate on the
> > > > value
> > > > > >> > part, as youll
> > > > > >> > >> note
> > > > > >> > >> >> > String,
> > > > > >> > >> >> > > > String headers were
> > > discounted early
> > > > on. The
> > > > > >> > reason for this
> > > > > >> > >> is
> > > > > >> > >> >> > > flexibility
> > > > > >> > >> >> > > > and keeping in line
> > with the
> > > > flexibility of
> > > > > key,
> > > > > >> > value of the
> > > > > >> > >> >> > message
> > > > > >> > >> >> > > > object itself. I don’t
> > think
> > > it takes
> > > > away
> > > > > from
> > > > > >> an
> > > > > >> > ecosystem
> > > > > >> > >> as
> > > > > >> > >> >> > each
> > > > > >> > >> >> > > plugin
> > > > > >> > >> >> > > > will care for their
> own
> > key,
> > > this way
> > > > ints,
> > > > > >> > booleans , exotic
> > > > > >> > >> >> > custom
> > > > > >> > >> >> > > binary
> > > > > >> > >> >> > > > can all be catered
> for=.
> > > > > >> > >> >> > > > a. If you really
> wanted
> > to
> > > push for a
> > > > typed
> > > > > >> value
> > > > > >> > interface,
> > > > > >> > >> I
> > > > > >> > >> >> > wouldn’t
> > > > > >> > >> >> > > > want just String
> values
> > > supported, but
> > > > the the
> > > > > >> > primatives
> > > > > >> > >> plus
> > > > > >> > >> >> > string and
> > > > > >> > >> >> > > > also still keeping the
> > > ability to have a
> > > > > binary
> > > > > >> > for custom
> > > > > >> > >> >> > binaries that
> > > > > >> > >> >> > > > some organisations may
> > have.
> > > > > >> > >> >> > > > i. I have written this
> > slight
> > > > alternative
> > > > > here,
> > > > > >> > >> >> > >
> https://cwiki.apache.org/
> > > > > >> > >> >> > > >
> > confluence/display/KAFKA/KIP-
> > > > > >> > 82+-+Add+Record+Headers+-+Typed
> > > > > >> > >> >> > > > ii. Essentially the
> > value
> > > bytes, has a
> > > > leading
> > > > > >> > byte overhead.
> > > > > >> > >> >> > > > 1. This tells you
> what
> > type
> > > the value
> > > > is,
> > > > > >> before
> > > > > >> > reading
> > > > > >> > >> the rest
> > > > > >> > >> >> > of the
> > > > > >> > >> >> > > > bytes, allowing
> > > > serialisation/deserialization
> > > > > to
> > > > > >> > and from the
> > > > > >> > >> >> > primitives,
> > > > > >> > >> >> > > > string and byte[].
> This
> > is
> > > akin to some
> > > > other
> > > > > >> > messaging
> > > > > >> > >> systems.
> > > > > >> > >> >> > > > 2) We are making it
> > optional,
> > > so that
> > > > for
> > > > > those
> > > > > >> > not wanting
> > > > > >> > >> >> > headers have
> > > > > >> > >> >> > > 0
> > > > > >> > >> >> > > > bytes overhead (think
> > of it
> > > as a feature
> > > > > flag),
> > > > > >> I
> > > > > >> > don’t
> > > > > >> > >> think this
> > > > > >> > >> >> > is
> > > > > >> > >> >> > > > complex, especially if
> > > comparing to
> > > > changes
> > > > > >> > proposed in
> > > > > >> > >> other kips
> > > > > >> > >> >> > like
> > > > > >> > >> >> > > > kip-98.
> > > > > >> > >> >> > > > a. If you really
> really
> > don’t
> > > like
> > > > this, we
> > > > > can
> > > > > >> > drop it, but
> > > > > >> > >> it
> > > > > >> > >> >> > would
> > > > > >> > >> >> > > mean
> > > > > >> > >> >> > > > buying into 4 bytes
> > extra
> > > overhead for
> > > > users
> > > > > who
> > > > > >> > do not want
> > > > > >> > >> to use
> > > > > >> > >> >> > > headers.
> > > > > >> > >> >> > > > 3) In the summary yes,
> > it is
> > > at a higher
> > > > > level,
> > > > > >> > but I think
> > > > > >> > >> this
> > > > > >> > >> >> > is well
> > > > > >> > >> >> > > > documented in the
> > proposed
> > > changes
> > > > section.
> > > > > >> > >> >> > > > a. Added getHeaders
> > method to
> > > > > Producer/Consumer
> > > > > >> > record (that
> > > > > >> > >> is it)
> > > > > >> > >> >> > > > b. We’ve also detailed
> > the
> > > new Headers
> > > > class
> > > > > >> that
> > > > > >> > this method
> > > > > >> > >> >> > returns
> > > > > >> > >> >> > > that
> > > > > >> > >> >> > > > encapsulates the
> headers
> > > protocol and
> > > > logic.
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Best,
> > > > > >> > >> >> > > > Mike
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > ==Original questions
> > from the
> > > vote
> > > > thread from
> > > > > >> > Jay.==
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Couple of things I
> > think we
> > > still need
> > > > to work
> > > > > >> out:
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1. I think we agree
> > about
> > > the key,
> > > > but I
> > > > > >> think
> > > > > >> > we haven't
> > > > > >> > >> >> > talked about
> > > > > >> > >> >> > > > the value yet. I
> > think if
> > > our goal
> > > > is an
> > > > > open
> > > > > >> > ecosystem
> > > > > >> > >> of these
> > > > > >> > >> >> > > header
> > > > > >> > >> >> > > > spread across many
> > plugins
> > > from many
> > > > > systems
> > > > > >> we
> > > > > >> > should
> > > > > >> > >> consider
> > > > > >> > >> >> > making
> > > > > >> > >> >> > > > this
> > > > > >> > >> >> > > > a string as well so
> > it can
> > > be
> > > > printed, set
> > > > > >> via
> > > > > >> > a UI, set
> > > > > >> > >> in
> > > > > >> > >> >> > config,
> > > > > >> > >> >> > > etc.
> > > > > >> > >> >> > > > Basically
> encouraging
> > > pluggable
> > > > > serialization
> > > > > >> > formats
> > > > > >> > >> here will
> > > > > >> > >> >> > lead
> > > > > >> > >> >> > > to
> > > > > >> > >> >> > > > a
> > > > > >> > >> >> > > > bit of a tower of
> > babel.
> > > > > >> > >> >> > > > 2. This proposal
> > still
> > > includes a
> > > > pretty
> > > > > big
> > > > > >> > change to our
> > > > > >> > >> >> > > serialization
> > > > > >> > >> >> > > > and protocol
> > definition
> > > layer.
> > > > Essentially
> > > > > >> it is
> > > > > >> > >> introducing an
> > > > > >> > >> >> > > optional
> > > > > >> > >> >> > > > type, where the
> > format is
> > > data
> > > > dependent. I
> > > > > >> > think this is
> > > > > >> > >> >> > actually a
> > > > > >> > >> >> > > big
> > > > > >> > >> >> > > > change though it
> > doesn't
> > > seem like
> > > > it. It
> > > > > >> means
> > > > > >> > you can no
> > > > > >> > >> >> > longer
> > > > > >> > >> >> > > > specify
> > > > > >> > >> >> > > > this type with our
> > type
> > > definition
> > > > DSL, and
> > > > > >> > likewise it
> > > > > >> > >> requires
> > > > > >> > >> >> > > custom
> > > > > >> > >> >> > > > handling in client
> > libs.
> > > This isn't
> > > > a huge
> > > > > >> > thing, since
> > > > > >> > >> the
> > > > > >> > >> >> > Record
> > > > > >> > >> >> > > > definition is
> custom
> > > anyway, but I
> > > > think
> > > > > this
> > > > > >> > kind of
> > > > > >> > >> protocol
> > > > > >> > >> >> > > > inconsistency is
> very
> > > non-desirable
> > > > and
> > > > > ties
> > > > > >> > you to
> > > > > >> > >> hand-coding
> > > > > >> > >> >> > > things.
> > > > > >> > >> >> > > > I
> > > > > >> > >> >> > > > think the type
> should
> > > instead by [Key
> > > > > Value]
> > > > > >> in
> > > > > >> > our BNF,
> > > > > >> > >> where
> > > > > >> > >> >> > key and
> > > > > >> > >> >> > > > value are both
> short
> > > strings as used
> > > > > >> elsewhere.
> > > > > >> > This
> > > > > >> > >> brings it
> > > > > >> > >> >> > in line
> > > > > >> > >> >> > > > with
> > > > > >> > >> >> > > > the rest of the
> > protocol.
> > > > > >> > >> >> > > > 3. Could we get
> more
> > > specific about
> > > > the
> > > > > exact
> > > > > >> > Java API
> > > > > >> > >> change to
> > > > > >> > >> >> > > > ProducerRecord,
> > > ConsumerRecord,
> > > > Record,
> > > > > etc?
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > -Jay
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > The information contained in
> > this
> > > email is
> > > > strictly
> > > > > >> > confidential and
> > > > > >> > >> for
> > > > > >> > >> >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > >> indicated.
> > > > > >> > If you
> > > > > >> > >> are not
> > > > > >> > >> >> > the intended recipient, please
> > do not
> > > read,
> > > > copy, use
> > > > > or
> > > > > >> > disclose to
> > > > > >> > >> others
> > > > > >> > >> >> > this message or any
> attachment.
> > > Please also
> > > > notify the
> > > > > >> > sender by
> > > > > >> > >> replying
> > > > > >> > >> >> > to this email or by telephone
> > > (+44(020 7896
> > > > 0011) and
> > > > > >> then
> > > > > >> > delete
> > > > > >> > >> the email
> > > > > >> > >> >> > and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc) that
> > > > > do
> > > > > >> > not relate
> > > > > >> > >> to the
> > > > > >> > >> >> > official business of this
> > company
> > > shall be
> > > > understood
> > > > > as
> > > > > >> > neither
> > > > > >> > >> given nor
> > > > > >> > >> >> > endorsed by it. IG is a
> trading
> > name
> > > of IG
> > > > Markets
> > > > > >> Limited
> > > > > >> > (a company
> > > > > >> > >> >> > registered in England and
> Wales,
> > > company number
> > > > > >> 04008957)
> > > > > >> > and IG
> > > > > >> > >> Index
> > > > > >> > >> >> > Limited (a company registered
> in
> > > England and
> > > > Wales,
> > > > > >> > company number
> > > > > >> > >> >> > 01190902). Registered address
> at
> > > Cannon Bridge
> > > > House,
> > > > > 25
> > > > > >> > Dowgate
> > > > > >> > >> Hill,
> > > > > >> > >> >> > London EC4R 2YA. Both IG
> Markets
> > > Limited
> > > > (register
> > > > > >> number
> > > > > >> > 195355)
> > > > > >> > >> and IG
> > > > > >> > >> >> > Index Limited (register number
> > > 114059) are
> > > > authorised
> > > > > >> and
> > > > > >> > regulated
> > > > > >> > >> by the
> > > > > >> > >> >> > Financial Conduct Authority.
> > > > > >> > >> >> >
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> --
> > > > > >> > >> Nacho - Ignacio Solis -
> > isolis@igso.net
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > The information contained in this email is
> > strictly
> > > > confidential and
> > > > > for
> > > > > >> > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you are
> > > > > >> not
> > > > > >> > the intended recipient, please do not read,
> > copy, use
> > > or
> > > > disclose to
> > > > > >> others
> > > > > >> > this message or any attachment. Please also
> > notify
> > > the sender by
> > > > > >> replying
> > > > > >> > to this email or by telephone (+44(020 7896
> > 0011) and
> > > then
> > > > delete the
> > > > > >> email
> > > > > >> > and any copies of it. Opinions, conclusion
> > (etc) that
> > > do not
> > > > relate to
> > > > > >> the
> > > > > >> > official business of this company shall be
> > understood
> > > as
> > > > neither given
> > > > > >> nor
> > > > > >> > endorsed by it. IG is a trading name of IG
> > Markets
> > > Limited (a
> > > > company
> > > > > >> > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG Index
> > > > > >> > Limited (a company registered in England and
> > Wales,
> > > company
> > > > number
> > > > > >> > 01190902). Registered address at Cannon Bridge
> > House,
> > > 25
> > > > Dowgate Hill,
> > > > > >> > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and
> > > > > IG
> > > > > >> > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > >> the
> > > > > >> > Financial Conduct Authority.
> > > > > >> >
> > > > > >> The information contained in this email is
> > strictly
> > > confidential
> > > > and for
> > > > > >> the use of the addressee only, unless otherwise
> > > indicated. If you
> > > > are
> > > > > not
> > > > > >> the intended recipient, please do not read,
> copy,
> > use
> > > or disclose
> > > > to
> > > > > others
> > > > > >> this message or any attachment. Please also
> > notify the
> > > sender by
> > > > > replying
> > > > > >> to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the
> > > > > email
> > > > > >> and any copies of it. Opinions, conclusion (etc)
> > that
> > > do not
> > > > relate to
> > > > > the
> > > > > >> official business of this company shall be
> > understood
> > > as neither
> > > > given
> > > > > nor
> > > > > >> endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > >> registered in England and Wales, company number
> > > 04008957) and IG
> > > > Index
> > > > > >> Limited (a company registered in England and
> > Wales,
> > > company number
> > > > > >> 01190902). Registered address at Cannon Bridge
> > House,
> > > 25 Dowgate
> > > > Hill,
> > > > > >> London EC4R 2YA. Both IG Markets Limited
> (register
> > > number 195355)
> > > > and IG
> > > > > >> Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > the
> > > > > >> Financial Conduct Authority.
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > The information contained in this email is strictly
> > > confidential and
> > > > for the use of the addressee only, unless otherwise
> > indicated.
> > > If you are
> > > > not the intended recipient, please do not read, copy, use
> > or
> > > disclose to
> > > > others this message or any attachment. Please also notify
> > the
> > > sender by
> > > > replying to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the email and any copies of it. Opinions, conclusion
> (etc)
> > that
> > > do not
> > > > relate to the official business of this company shall be
> > > understood as
> > > > neither given nor endorsed by it. IG is a trading name of
> > IG
> > > Markets
> > > > Limited (a company registered in England and Wales,
> company
> > > number
> > > > 04008957) and IG Index Limited (a company registered in
> > England
> > > and Wales,
> > > > company number 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and IG Index Limited (register number 114059) are
> > > authorised and
> > > > regulated by the Financial Conduct Authority.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential
> > and
> > > for the use of the addressee only, unless otherwise indicated. If
> > you are
> > > not the intended recipient, please do not read, copy, use or
> > disclose to
> > > others this message or any attachment. Please also notify the
> sender
> > by
> > > replying to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > relate to the official business of this company shall be understood
> > as
> > > neither given nor endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a company registered in England and Wales, company number
> > > 04008957) and IG Index Limited (a company registered in England and
> > Wales,
> > > company number 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > 195355) and IG Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
On points 1 and 2 I agree.
This also affects kip-98, I should expect this resolved before that vote also passes. If it is accepted there (I’m assuming this is getting discussed on that KIP? As you’re driving the move to VarInts), I am happy to make this KIP will simply follow suit to whatever is agreed in KIP-98.
On 3) Agreed this is a simpler form, as long as no one is expecting hashmap lookup performance ( O(1) ), I still would prefer an encapsulated class, so if we find that holding it as an array in future is causing some perf issues, the internals are not exposed to end users, allowing the internal structure to move to a map.
Can we compromise on?
class ConsumerRecord<K, V> {
K key;
V value;
Headers headers;
}
class Headers {
Header[] headers;
add(String key, byte[] value)
Collection<byte[]> get(String key)
}
class Header {
String key;
byte[] value;
}
On 19/02/2017, 18:54, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> headers dont "leak" into application code. they are useful to application
> code as well.
This is exactly what I have been trying to get at. The use cases documented
here are middleware:
https://cwiki.apache.org/confluence/display/KAFKA/A+Case+for+Kafka+Headers.
If headers are intended for the application space as well, the document
should be updated accordingly so that it is an explicit design goal and not
an unfortunate design byproduct. There may be some disagreement on whether
it _should_ be a design goal, but it may not matter much since the current
interceptor API is probably insufficient for middleware applications (I
haven't thought of a way to do this that isn't cumbersome).
That aside, the three unresolved points for me are the following:
1. The use of varints. As a one-off for use only with record headers, the
case was weak, but if we are using them throughout the message format, then
we should do so here as well. The additional complexity is minimal and
early performance testing fully justifies their use.
2. If varints are used, the case for using attributes to indicate null
headers also becomes weak. We only add one additional byte in each message
if there are no headers. Whatever the case, let us be consistent with how
we treat null keys and values.
3. We have apparently agreed that the broker will validate headers (please
add this to the KIP). That being the case, I would prefer to use Kafka's
conventional format for arrays. The downside is that it takes more work to
skip over the headers on the consumer, though it's unclear if the cost
would matter in practice. Some concern was previously expressed about the
allocation of maps. An alternative would be to use arrays, i.e.
class ConsumerRecord<K, V> {
K key;
V value;
Header[] headers;
}
class Header {
String key;
byte[] value;
}
This would work nicely with the conventional array format and my guess is
it would obviate the need do any lazy initialization. If we use the map as
is currently documented, then it is possible with either representation to
slice the headers and initialize them lazily. Either way, it might be a
good idea to use a separate object to represent the headers in case we need
to modify it in the future in some way.
(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)
If we can resolve these points, then at least you will get my vote.
Thanks,
Jason
On Sun, Feb 19, 2017 at 7:30 AM, radai <ra...@gmail.com> wrote:
> headers dont "leak" into application code. they are useful to application
> code as well.
> IIUC samze currently has headers "in-V" and would just switch over to kafka
> headers if they exist.
> im sure plenty of other users of kafka would have a use for headers.
> im pretty sure use cases exist around shuffling data into/out-of kafka
> (kafka connect or equivalent) where metadata from one end could copied over
> to the other (S3, for example uses http headers for user-accessible
> metadata). it will be kafka client code getting/setting those headers. not
> an interceptor.
>
> On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > For APM single event tracing, need access to the header at the point of
> > processing on the processing thread.
> >
> > As such interceptors will not work/be suitable for these, due to the fact
> > they act on the ConsumerRecords as a batch, before the handling thread
> can
> > split out and process per message which is the point these tools will
> need
> > to continue to transaction tracing.
> >
> > Like wise tools and infra pieces will need access to the message outside
> > the interceptor realm.
> >
> >
> >
> > On 17/02/2017, 21:26, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> >
> >
> > So the broker will parse the headers and validate them. Good. The
> only
> > reason remaining that I can see to leave the headers as a byte array
> > is to
> > make it easier for the client to skip past them. Are we sure this is
> > not
> > premature optimization? Are there any performance results which show
> > that
> > this is worthwhile?
> >
> > What’s the issue with exposing a method getHeaders on the
> > producer/consumer
> > > record? It doesn’t break anything. We don’t need any special
> version.
> >
> >
> > See my previous explanation. What I am trying to resist is the
> headers
> > becoming a general application-level facility. The primary use case
> as
> > far
> > as I can tell is middleware, which is the use case that the
> > interceptors
> > are providing.
> >
> > Current batch consumer model and consumer interceptors don’t work
> where
> > > headers need to be acted on at per message level at time of
> > processing,
> > > very case is APM (the core one), where the header value is used to
> > continue
> > > tracing.
> >
> >
> > I still don't understand the point about batching. The consumer
> > records are
> > exposed as a batch in the consumer interceptor, but you can still
> > iterate
> > them individually. It is no different for the consumer API itself.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > Re:
> > >
> > > “ The point about creation of maps seems orthogonal. We can
> still
> > > represent
> > > the headers as a slice of bytes until the time it is accessed.”
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > >
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> > >
> > >
> > > On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > What’s the issue with exposing a method getHeaders on the
> > > producer/consumer record? It doesn’t break anything. We don’t need
> > any
> > > special version.
> > >
> > > Current batch consumer model and consumer interceptors don’t
> work
> > > where headers need to be acted on at per message level at time of
> > > processing, very case is APM (the core one), where the header value
> > is used
> > > to continue tracing. JMS/HTTP etc all expose these, without issues.
> > I would
> > > NOT want to lock this down to only be usable accessible via
> > interceptors,
> > > as we’d fail on one of the main goals.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > >
> > > On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > >
> > > The point about creation of maps seems orthogonal. We can
> > still
> > > represent
> > > the headers as a slice of bytes until the time it is
> > accessed.
> > >
> > >
> > > > Yes exactly we have access to the records thus why the
> > header
> > > should be
> > > > accessible via it and not hidden for only interceptors to
> > access.
> > >
> > >
> > > As explained above, the point is to make the intended usage
> > clear.
> > > Applications should continue to rely on the key/value
> fields
> > to
> > > serialize
> > > their own headers, and it would be more ideal if we can
> avoid
> > > leaking
> > > third-party headers into applications. This is difficult to
> > do
> > > with the
> > > current interceptors because they share the record objects
> > with
> > > the common
> > > API. What I had in mind is something like an extension of
> the
> > > current
> > > interceptors which exposed a different object (e.g.
> > > `RecordAndHeaders`).
> > > The challenge is for MM-like use cases. Let me see if I can
> > come
> > > up with a
> > > concrete proposal for that problem.
> > >
> > > -Jason
> > >
> > >
> > >
> > > On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > I am happy to move the definition of the header into the
> > message
> > > body, but
> > > > would cause us not to lazy initialise/parse the headers,
> as
> > > obviously, we
> > > > would have to traverse these reading the message.
> > > >
> > > > This was actually one of Jay’s requests:
> > > >
> > > > “ 2. I think we should think about creating the Map
> > lazily to
> > > avoid
> > > > parsing out all the headers into little objects.
> > HashMaps
> > > themselves
> > > > are
> > > > kind of expensive and the consumer is very perf
> > sensitive so
> > > and making
> > > > gazillions of hashmaps that may or may not get used
> is
> > > probably a bad
> > > > idea.”
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > Yes exactly we have access to the records thus why
> the
> > > header should
> > > > be accessible via it and not hidden for only interceptors
> > to
> > > access.
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Magnus Edenhill <ma...@edenhill.se>
> > > > Sent: Friday, February 17, 2017 7:34:49 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > Big +1 on VarInts.
> > > > CPUs are fast, memory is slow.
> > > >
> > > > I agree with Jason that we'll want to continue
> > verifying
> > > messages,
> > > > including their headers, so while I appreciate the
> > idea of
> > > the opaque
> > > > header blob it won't be useful in practice.
> > > >
> > > > /Magnus
> > > >
> > > > 2017-02-17 10:41 GMT-08:00 Jason Gustafson <
> > > jason@confluent.io>:
> > > >
> > > > > Sorry, my mistake. The consumer interceptor is per
> > batch,
> > > though I'm
> > > > not
> > > > > sure that's an actual limitation since you still
> have
> > > access to the
> > > > > individual records.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Re headers as byte array and future use by
> broker.
> > This
> > > doesn't
> > > > take away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >
> > > > > >
> > > > > > Yeah, I didn't say it was difficult, only
> awkward.
> > You
> > > wouldn't
> > > > write the
> > > > > > schema that way if you were planning to use it on
> > the
> > > brokers from
> > > > the
> > > > > > beginning. Note also that one of the benefits of
> > letting
> > > the broker
> > > > > > understand headers is that it can validate that
> > they are
> > > properly
> > > > > > formatted. If cost is the only concern, we should
> > > confirm its
> > > > impact
> > > > > > through performance testing.
> > > > > >
> > > > > > One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >
> > > > > >
> > > > > > I'm not sure I understand the point about
> batching.
> > > Interceptors
> > > > are
> > > > > > applied per-message, right?
> > > > > >
> > > > > > My intent on interceptors is to keep the usage of
> > headers
> > > > well-defined so
> > > > > > that they don't start leaking unnecessarily into
> > > applications. My
> > > > guess
> > > > > is
> > > > > > that it's probably inevitable, but isolating it
> in
> > the
> > > > interceptors would
> > > > > > at least give people a second thought before
> > deciding to
> > > use it.
> > > > The main
> > > > > > challenge in my mind is figuring out how an MM
> use
> > case
> > > would
> > > > work. It
> > > > > > would be more cumbersome to replicate headers
> > through an
> > > > interceptor,
> > > > > > though arguably MM should be working at a lower
> > level
> > > anyway.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Re headers available on the record va
> > interceptors only
> > > > > >>
> > > > > >> One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >>
> > > > > >> This anyhow is similar to jms/http/amqp where
> > headers
> > > are
> > > > available to
> > > > > >> consuming applications.
> > > > > >>
> > > > > >> Re headers as byte array and future use by
> > broker. This
> > > doesn't
> > > > take
> > > > > away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Sent using OWA for iPhone
> > > > > >> ________________________________________
> > > > > >> From: Jason Gustafson <ja...@confluent.io>
> > > > > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > > > > >> To: dev@kafka.apache.org
> > > > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > > >>
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >>
> > > > > >>
> > > > > >> Yes, that is what we are proposing, so using
> > varints
> > > for headers
> > > > would
> > > > > be
> > > > > >> consistent with the rest of the message. We have
> > used
> > > static
> > > > helper
> > > > > >> methods
> > > > > >> in our prototype implementation.
> > > > > >>
> > > > > >> The cost of parsing, we want to parse/interpret
> > the
> > > headers
> > > > lazily (this
> > > > > >> is
> > > > > >> > a key point brought up earlier in discussions)
> > > > > >>
> > > > > >>
> > > > > >> I'm a bit skeptical of this. Has anyone done the
> > > performance
> > > > testing? I
> > > > > >> can
> > > > > >> probably implement it and test it if no one else
> > has. I
> > > was also
> > > > under
> > > > > the
> > > > > >> impression that there may be use cases down the
> > road
> > > where the
> > > > broker
> > > > > >> would
> > > > > >> need to interpret headers. That wouldn't be off
> > the
> > > table in the
> > > > future
> > > > > if
> > > > > >> it's represented as bytes, but it would be quite
> > a bit
> > > more
> > > > awkward,
> > > > > >> right?
> > > > > >>
> > > > > >> By the way, one question I have been wondering
> > about. My
> > > > understanding
> > > > > is
> > > > > >> that headers are primarily for use cases where a
> > > third-party
> > > > components
> > > > > >> wants to enrich messages without needing to
> > understand
> > > or modify
> > > > the
> > > > > >> schema
> > > > > >> of the message key and value. For the
> applications
> > > which directly
> > > > > produce
> > > > > >> and consume the messages and control the
> key/value
> > > schema
> > > > directly, it
> > > > > >> seems we would rather have them implement
> headers
> > > directly in
> > > > their own
> > > > > >> schema. Supposing for the sake of argument that
> > it was
> > > possible,
> > > > my
> > > > > >> question is whether it be sufficient to expose
> the
> > > headers in the
> > > > > >> interceptor API and not in the common API?
> > > > > >>
> > > > > >> -Jason
> > > > > >>
> > > > > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On the point of varInts
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 11:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > On the point re: headers in the message
> > protocol
> > > being a
> > > > byte
> > > > > array
> > > > > >> > and not a count of elements followed by the
> > elements.
> > > Again
> > > > this was
> > > > > >> > discussed/argued previously.
> > > > > >> >
> > > > > >> > It was agreed on for a few reasons some of
> > which
> > > you have
> > > > > obviously
> > > > > >> > picked up on:
> > > > > >> >
> > > > > >> > Broker is able to pass it through opaquely
> > > > > >> > The cost of parsing, we want to
> > parse/interpret
> > > the headers
> > > > lazily
> > > > > >> > (this is a key point brought up earlier in
> > > discussions)
> > > > > >> > Headers can be copied from consumer record
> > to
> > > producer
> > > > record (aka
> > > > > >> > mirror makers etc) without parsing if no
> > changes are
> > > being made
> > > > or
> > > > > being
> > > > > >> > looked at.
> > > > > >> > Keeps the broker agnostic to the format
> > > > > >> > You need an int32 either for the byte size
> > of the
> > > headers,
> > > > or for
> > > > > >> the
> > > > > >> > count of elements, so overheads are the same,
> > but
> > > with going
> > > > with an
> > > > > >> opaque
> > > > > >> > byte array has the above advantages.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 02:50, "Jason Gustafson" <
> > > jason@confluent.io
> > > > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > Sorry, should have noted that the
> > performance
> > > testing
> > > > was done
> > > > > >> > using the
> > > > > >> > producer performance tool shipped with
> > Kafka.
> > > > > >> >
> > > > > >> > -Jason
> > > > > >> >
> > > > > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason
> > > Gustafson <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> >
> > > > > >> > > Hey Nacho,
> > > > > >> > >
> > > > > >> > > I've compared performance of our
> > KIP-98
> > > > implementation with
> > > > > >> and
> > > > > >> > without
> > > > > >> > > varints. For messages around 128
> > bytes, we
> > > see an
> > > > increase
> > > > > in
> > > > > >> > throughput of
> > > > > >> > > about 30% using the default
> > configuration
> > > settings.
> > > > At 256
> > > > > >> > bytes, the
> > > > > >> > > increase is around 16%. Obviously
> the
> > > performance
> > > > converge
> > > > > as
> > > > > >> > messages get
> > > > > >> > > larger, but it seems well worth the
> > cost.
> > > Note that
> > > > we are
> > > > > >> also
> > > > > >> > seeing a
> > > > > >> > > substantial performance increase
> > against
> > > trunk
> > > > primarily
> > > > > >> because
> > > > > >> > of the
> > > > > >> > > much more efficient packing that
> > varints
> > > provide us.
> > > > > Anything
> > > > > >> > adding to
> > > > > >> > > message overhead, such as record
> > headers,
> > > would only
> > > > > increase
> > > > > >> > the relative
> > > > > >> > > difference. (Of course take these
> > numbers
> > > with a
> > > > grain of
> > > > > salt
> > > > > >> > since I have
> > > > > >> > > only used the default settings with
> > both
> > > the producer
> > > > and
> > > > > >> broker
> > > > > >> > on my
> > > > > >> > > local machine. We intend to provide
> > more
> > > extensive
> > > > > performance
> > > > > >> > details as
> > > > > >> > > part of the work for KIP-98.)
> > > > > >> > >
> > > > > >> > > The implementation we are using is
> > from
> > > protobuf (
> > > > > >> > > https://developers.google.com/
> > > > > protocol-buffers/docs/encoding
> > > > > >> ),
> > > > > >> > which is
> > > > > >> > > also used in HBase. It is trivial to
> > > implement and as
> > > > far
> > > > > as I
> > > > > >> > know doesn't
> > > > > >> > > suffer from the aliasing problem you
> > are
> > > describing. I
> > > > > checked
> > > > > >> > with Magnus
> > > > > >> > > (the author of librdkafka) and he
> > agreed
> > > that the
> > > > savings
> > > > > >> seemed
> > > > > >> > worth the
> > > > > >> > > cost of implementation.
> > > > > >> > >
> > > > > >> > > -Jason
> > > > > >> > >
> > > > > >> > > On Thu, Feb 16, 2017 at 4:32 PM,
> > Ignacio
> > > Solis <
> > > > > >> isolis@igso.net>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> -VarInts
> > > > > >> > >>
> > > > > >> > >> I'm one of the people (if not the
> > most)
> > > opposed to
> > > > VarInts.
> > > > > >> > VarInts
> > > > > >> > >> have a place, but this is not it.
> > (We
> > > had a large
> > > > > >> discussion
> > > > > >> > about
> > > > > >> > >> them at the beginning of KIP-82
> time)
> > > > > >> > >>
> > > > > >> > >> If anybody has real life
> performance
> > > numbers of
> > > > VarInts
> > > > > >> > improving
> > > > > >> > >> things or significantly reducing
> > resources
> > > I would
> > > > like to
> > > > > >> know
> > > > > >> > what
> > > > > >> > >> that case may be. Yes, you can save
> > some
> > > bytes here
> > > > and
> > > > > >> there,
> > > > > >> > but
> > > > > >> > >> this is probably insignificant to
> the
> > > overall system
> > > > > behavior
> > > > > >> > and
> > > > > >> > >> storage requirements. -- I say
> this
> > with
> > > respect to
> > > > using
> > > > > >> > VarInts in
> > > > > >> > >> the protocol itself, not as part of
> > the
> > > data.
> > > > > >> > >>
> > > > > >> > >> VarInts require you to parse the
> Int
> > > before using it
> > > > and
> > > > > >> > depending on
> > > > > >> > >> the encoding they can suffer from
> > aliasing
> > > (multiple
> > > > > >> > representations
> > > > > >> > >> for the same value).
> > > > > >> > >>
> > > > > >> > >> Why add complexity?
> > > > > >> > >>
> > > > > >> > >> Nacho
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> On Thu, Feb 16, 2017 at 10:29 AM,
> > Colin
> > > McCabe <
> > > > > >> > cmccabe@apache.org>
> > > > > >> > >> wrote:
> > > > > >> > >> > +1 for varints here-- it would
> save
> > > quite a bit of
> > > > space.
> > > > > >> > They are
> > > > > >> > >> > pretty quick to implement as
> well.
> > > > > >> > >> >
> > > > > >> > >> > I think it makes sense for values
> > to be
> > > byte
> > > > arrays.
> > > > > Users
> > > > > >> > might want
> > > > > >> > >> > to attach arbitrary payloads;
> they
> > > shouldn't be
> > > > forced to
> > > > > >> > serialize
> > > > > >> > >> > everything to Java strings.
> > > > > >> > >> >
> > > > > >> > >> > best,
> > > > > >> > >> > Colin
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > On Thu, Feb 16, 2017, at 09:52,
> > Jason
> > > Gustafson
> > > > wrote:
> > > > > >> > >> >> Hey Michael,
> > > > > >> > >> >>
> > > > > >> > >> >> Hmm, I guess the point of
> > representing
> > > it as
> > > > bytes is to
> > > > > >> > allow the
> > > > > >> > >> broker
> > > > > >> > >> >> to pass it through opaquely? Is
> > the
> > > cost of
> > > > parsing
> > > > > them a
> > > > > >> > concern, or
> > > > > >> > >> >> are
> > > > > >> > >> >> we simply trying to ensure that
> > the
> > > broker stays
> > > > > agnostic
> > > > > >> to
> > > > > >> > the
> > > > > >> > >> format?
> > > > > >> > >> >>
> > > > > >> > >> >> On varints, I think adding
> > support for
> > > them makes
> > > > less
> > > > > >> sense
> > > > > >> > for an
> > > > > >> > >> >> isolated use case, but as part
> of
> > a
> > > more holistic
> > > > change
> > > > > >> > (such as what
> > > > > >> > >> we
> > > > > >> > >> >> have proposed in KIP-98), I
> think
> > they
> > > are
> > > > justifiable.
> > > > > If
> > > > > >> > we add them,
> > > > > >> > >> >> then the need to use attributes
> > becomes
> > > quite a
> > > > bit
> > > > > >> weaker,
> > > > > >> > right? The
> > > > > >> > >> >> other thing I find slightly odd
> > is the
> > > fact that
> > > > null
> > > > > >> > headers has no
> > > > > >> > >> >> actual
> > > > > >> > >> >> semantic meaning for the message
> > > (unlike null
> > > > keys and
> > > > > >> > values). It is
> > > > > >> > >> >> just
> > > > > >> > >> >> a space optimization. It seems a
> > bit
> > > better to
> > > > always
> > > > > use
> > > > > >> > size 0 to
> > > > > >> > >> >> indicate having no headers.
> > > > > >> > >> >>
> > > > > >> > >> >> Overall, the main point is
> > ensuring
> > > that the
> > > > message
> > > > > >> schema
> > > > > >> > remains
> > > > > >> > >> >> consistent, either within the
> > larger
> > > protocol, or
> > > > at a
> > > > > >> > minimum within
> > > > > >> > >> the
> > > > > >> > >> >> message itself.
> > > > > >> > >> >>
> > > > > >> > >> >> -Jason
> > > > > >> > >> >>
> > > > > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM,
> > > Michael Pearce <
> > > > > >> > Michael.Pearce@ig.com
> > > > > >> > >> >
> > > > > >> > >> >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> > Hi Jason,
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 1) in the message
> > protocol
> > > the headers
> > > > are
> > > > > >> simply
> > > > > >> > a byte
> > > > > >> > >> array,
> > > > > >> > >> >> > as like the key or value, this
> > is to
> > > clearly
> > > > demarcate
> > > > > >> the
> > > > > >> > header in
> > > > > >> > >> the
> > > > > >> > >> >> > core message. Then the header
> > byte
> > > array in the
> > > > core
> > > > > >> > message is an
> > > > > >> > >> array of
> > > > > >> > >> >> > key, value pairs. This is what
> > it is
> > > denoting.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Then this would be I guess in
> > the
> > > given
> > > > notation:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [KeyLength, Key,
> > > ValueLength, Value]
> > > > > >> > >> >> > KeyLength => int32
> > > <-----------------NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised key value
> > > > > >> > >> >> > Key => bytes
> > > <---------------------- NEW
> > > > > serialised
> > > > > >> > string (UTF8)
> > > > > >> > >> >> > bytes of the header key
> > > > > >> > >> >> > ValueLength => int32
> > > <-------------- NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised header value
> > > > > >> > >> >> > Value => bytes
> > > <-------------------- NEW
> > > > > serialised
> > > > > >> > form of the
> > > > > >> > >> header
> > > > > >> > >> >> > value
> > > > > >> > >> >> >
> > > > > >> > >> >> > The key length and value
> length
> > is
> > > matching the
> > > > way
> > > > > the
> > > > > >> > protocol is
> > > > > >> > >> >> > defined in the core message
> > currently.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 2)
> > > > > >> > >> >> > Var sized ints, this was
> > discussed
> > > much earlier
> > > > on, in
> > > > > >> > fact I had
> > > > > >> > >> >> > suggested it myself (with
> Hadoop
> > > references),
> > > > the
> > > > > >> > complexity of this
> > > > > >> > >> >> > compared to having a simpler
> > protocol
> > > was
> > > > argued and
> > > > > >> > agreed it
> > > > > >> > >> wasn’t worth
> > > > > >> > >> >> > the complexity as all other
> > clients
> > > in other
> > > > languages
> > > > > >> > would need to
> > > > > >> > >> ensure
> > > > > >> > >> >> > theyre using the right var
> size
> > > algorithm, as
> > > > there
> > > > > is a
> > > > > >> > few.
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 3)
> > > > > >> > >> >> > We did the attributes,
> optional
> > > approach as
> > > > originally
> > > > > >> > there was
> > > > > >> > >> marked
> > > > > >> > >> >> > concern that headers would
> > cause a
> > > message size
> > > > > overhead
> > > > > >> > for others,
> > > > > >> > >> who
> > > > > >> > >> >> > don’t want them. As such this
> > is the
> > > clean
> > > > solution to
> > > > > >> > achieve that.
> > > > > >> > >> If
> > > > > >> > >> >> > that no longer holds, and we
> > don’t
> > > care that we
> > > > add
> > > > > >> 4bytes
> > > > > >> > overhead,
> > > > > >> > >> then
> > > > > >> > >> >> > im happy to remove.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I’m personally in favour of
> > keeping
> > > the message
> > > > as
> > > > > small
> > > > > >> > as possible
> > > > > >> > >> so
> > > > > >> > >> >> > people don’t get shocks in
> perf
> > and
> > > throughputs
> > > > dues
> > > > > to
> > > > > >> > message size,
> > > > > >> > >> >> > unless they actively use the
> > feature,
> > > as such I
> > > > do
> > > > > >> prefer
> > > > > >> > the
> > > > > >> > >> attribute bit
> > > > > >> > >> >> > wise feature flag approach
> > myself.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On 16/02/2017, 05:40, "Jason
> > > Gustafson" <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > We have proposed a few
> > > significant changes
> > > > to the
> > > > > >> > message format
> > > > > >> > >> in
> > > > > >> > >> >> > KIP-98
> > > > > >> > >> >> > which now seems likely to
> > pass
> > > (perhaps
> > > > with some
> > > > > >> > iterations on
> > > > > >> > >> >> > implementation details).
> It
> > would
> > > be good
> > > > to try
> > > > > and
> > > > > >> > coordinate
> > > > > >> > >> the
> > > > > >> > >> >> > changes
> > > > > >> > >> >> > in both of the proposals
> to
> > make
> > > sure they
> > > > are
> > > > > >> > consistent and
> > > > > >> > >> >> > compatible.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I think using the
> > attributes to
> > > indicate
> > > > null
> > > > > >> headers
> > > > > >> > is a
> > > > > >> > >> reasonable
> > > > > >> > >> >> > approach. We have proposed
> > to do
> > > the same
> > > > thing
> > > > > for
> > > > > >> > the message
> > > > > >> > >> key and
> > > > > >> > >> >> > value. That said, I
> > sympathize
> > > with Jay's
> > > > > argument.
> > > > > >> > Having
> > > > > >> > >> multiple
> > > > > >> > >> >> > ways to
> > > > > >> > >> >> > specify a null value
> > increases
> > > the overall
> > > > > >> complexity
> > > > > >> > of the
> > > > > >> > >> protocol.
> > > > > >> > >> >> > You
> > > > > >> > >> >> > can see this just from the
> > fact
> > > that you
> > > > need the
> > > > > >> > extra verbiage
> > > > > >> > >> in the
> > > > > >> > >> >> > protocol specification in
> > this
> > > KIP and in
> > > > KIP-98
> > > > > to
> > > > > >> > describe the
> > > > > >> > >> >> > dependence
> > > > > >> > >> >> > between the fields and the
> > > attributes. It
> > > > seems
> > > > > >> like a
> > > > > >> > slippery
> > > > > >> > >> slope
> > > > > >> > >> >> > if
> > > > > >> > >> >> > you start allowing
> different
> > > request types
> > > > to
> > > > > >> > implement the
> > > > > >> > >> protocol
> > > > > >> > >> >> > specification differently.
> > > > > >> > >> >> >
> > > > > >> > >> >> > You can also argue that
> the
> > > messages
> > > > already are
> > > > > and
> > > > > >> > are likely
> > > > > >> > >> to
> > > > > >> > >> >> > remain a
> > > > > >> > >> >> > special case. For example,
> > there
> > > is
> > > > currently no
> > > > > >> > generality in
> > > > > >> > >> how
> > > > > >> > >> >> > compressed message sets
> are
> > > represented
> > > > that would
> > > > > >> be
> > > > > >> > applicable
> > > > > >> > >> for
> > > > > >> > >> >> > other
> > > > > >> > >> >> > request types. Some might
> > see this
> > > > divergence as
> > > > > an
> > > > > >> > unfortunate
> > > > > >> > >> >> > protocol
> > > > > >> > >> >> > deficiency which should be
> > fixed;
> > > others
> > > > might see
> > > > > >> it
> > > > > >> > as sort of
> > > > > >> > >> the
> > > > > >> > >> >> > inevitability of needing
> to
> > > optimize where
> > > > it
> > > > > counts
> > > > > >> > most. I'm
> > > > > >> > >> probably
> > > > > >> > >> >> > somewhere in between, but
> I
> > think
> > > we
> > > > probably all
> > > > > >> > share the
> > > > > >> > >> intuition
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the protocol should be
> kept
> > as
> > > consistent as
> > > > > >> possible.
> > > > > >> > With that
> > > > > >> > >> in
> > > > > >> > >> >> > mind,
> > > > > >> > >> >> > here are a few comments:
> > > > > >> > >> >> >
> > > > > >> > >> >> > 1. One thing I found a
> > little odd
> > > when
> > > > reading the
> > > > > >> > current
> > > > > >> > >> proposal is
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the headers are both
> > represented
> > > as an
> > > > array of
> > > > > >> bytes
> > > > > >> > and as an
> > > > > >> > >> array
> > > > > >> > >> >> > of
> > > > > >> > >> >> > key/value pairs. I'd
> > probably
> > > suggest
> > > > something
> > > > > like
> > > > > >> > this:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [HeaderKey
> > HeaderValue]
> > > > > >> > >> >> > HeaderKey => String
> > > > > >> > >> >> > HeaderValue => Bytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > An array in the Kafka
> > protocol is
> > > > represented as a
> > > > > >> > 4-byte integer
> > > > > >> > >> >> > indicating the number of
> > elements
> > > in the
> > > > array
> > > > > >> > followed by the
> > > > > >> > >> >> > serialization of the
> > elements.
> > > Unless I'm
> > > > > >> > misunderstanding, what
> > > > > >> > >> you
> > > > > >> > >> >> > have
> > > > > >> > >> >> > instead is the total size
> > of the
> > > headers in
> > > > bytes
> > > > > >> > followed by the
> > > > > >> > >> >> > elements.
> > > > > >> > >> >> > I'm not sure I see any
> > reason for
> > > this
> > > > > >> inconsistency.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 2. In KIP-98, we've
> > introduced
> > > > variable-length
> > > > > >> integer
> > > > > >> > fields.
> > > > > >> > >> >> > Effectively,
> > > > > >> > >> >> > we've enriched (or
> > "complicated"
> > > as Jay
> > > > might say
> > > > > ;)
> > > > > >> > the protocol
> > > > > >> > >> >> > specification to include
> the
> > > following
> > > > types:
> > > > > >> VarInt,
> > > > > >> > VarLong,
> > > > > >> > >> >> > UnsignedVarInt and
> > > UnsignedVarLong.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Along with these
> > primitives, we
> > > could
> > > > introduce
> > > > > the
> > > > > >> > following
> > > > > >> > >> types:
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeArray =>
> > NumberOfItems
> > > Item1 Item2
> > > > .. ItemN
> > > > > >> > >> >> > NumberOfItems =>
> > UnsignedVarInt
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeNullableArray =>
> > > NumberOfItemsOrNull
> > > > Item1
> > > > > >> > Item2 .. ItemN
> > > > > >> > >> >> > NumberOfItemsOrNull =>
> > VarInt
> > > (-1 means
> > > > null)
> > > > > >> > >> >> >
> > > > > >> > >> >> > And similarly for the
> > `String`
> > > and `Bytes`
> > > > types.
> > > > > >> > These types
> > > > > >> > >> can save
> > > > > >> > >> >> > a
> > > > > >> > >> >> > considerable amount of
> > space in
> > > this
> > > > proposal
> > > > > >> because
> > > > > >> > they can
> > > > > >> > >> be used
> > > > > >> > >> >> > for
> > > > > >> > >> >> > both the number of headers
> > > included in the
> > > > message
> > > > > >> and
> > > > > >> > the
> > > > > >> > >> lengths of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > header keys and values. We
> > could
> > > do this
> > > > instead:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers =>
> > VarSizeArray[HeaderKey
> > > > HeaderValue]
> > > > > >> > >> >> > HeaderKey =>
> VarSizeString
> > > > > >> > >> >> > HeaderValue =>
> > VarSizeBytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > Combining the savings from
> > the
> > > use of
> > > > variable
> > > > > >> length
> > > > > >> > fields, the
> > > > > >> > >> >> > benefit
> > > > > >> > >> >> > of using the attributes to
> > > represent null
> > > > seems
> > > > > >> pretty
> > > > > >> > small.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 3. Whichever way we go
> > (whether
> > > we use the
> > > > > >> attributes
> > > > > >> > or not), we
> > > > > >> > >> >> > should at
> > > > > >> > >> >> > least be consistent
> between
> > this
> > > KIP and
> > > > KIP-98.
> > > > > It
> > > > > >> > would be very
> > > > > >> > >> >> > strange
> > > > > >> > >> >> > to have two ways to
> > represent
> > > null values
> > > > in the
> > > > > >> same
> > > > > >> > schema.
> > > > > >> > >> Either
> > > > > >> > >> >> > way is
> > > > > >> > >> >> > OK with me. I think some
> > > message-level
> > > > > optimizations
> > > > > >> > are
> > > > > >> > >> justifiable,
> > > > > >> > >> >> > but
> > > > > >> > >> >> > the savings here seem
> > minimal (a
> > > few bytes
> > > > per
> > > > > >> > message), so
> > > > > >> > >> maybe it's
> > > > > >> > >> >> > not
> > > > > >> > >> >> > worth the cost of letting
> > the
> > > message
> > > > diverge even
> > > > > >> > further from
> > > > > >> > >> the
> > > > > >> > >> >> > rest of
> > > > > >> > >> >> > the protocol.
> > > > > >> > >> >> >
> > > > > >> > >> >> > -Jason
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On Wed, Feb 15, 2017 at
> > 8:52 AM,
> > > radai <
> > > > > >> > >> radai.rosenblatt@gmail.com>
> > > > > >> > >> >> > wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > > I've trimmed the inline
> > > contents as this
> > > > mail is
> > > > > >> > getting too
> > > > > >> > >> big for
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > apache mailing list
> > software to
> > > deliver
> > > > :-(
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 1. the important thing
> for
> > > > interoperability is
> > > > > for
> > > > > >> > different
> > > > > >> > >> >> > "interested
> > > > > >> > >> >> > > parties" (plugins, infra
> > > layers/wrappers,
> > > > > >> user-code)
> > > > > >> > to be
> > > > > >> > >> able to
> > > > > >> > >> >> > stick
> > > > > >> > >> >> > > pieces of metadata onto
> > msgs
> > > without
> > > > getting in
> > > > > >> each
> > > > > >> > other's
> > > > > >> > >> way. a
> > > > > >> > >> >> > common
> > > > > >> > >> >> > > key scheme (Strings, as
> > of the
> > > time of
> > > > this
> > > > > >> > writing?) is all
> > > > > >> > >> thats
> > > > > >> > >> >> > required
> > > > > >> > >> >> > > for that. it is assumed
> > that
> > > the other end
> > > > > >> > interested in any
> > > > > >> > >> such
> > > > > >> > >> >> > piece of
> > > > > >> > >> >> > > metadata knows the
> > encoding,
> > > and byte[]
> > > > provides
> > > > > >> for
> > > > > >> > the most
> > > > > >> > >> >> > flexibility.
> > > > > >> > >> >> > > i believe this is the
> same
> > > logic behind
> > > > core
> > > > > kafka
> > > > > >> > being
> > > > > >> > >> >> > byte[]/byte[] -
> > > > > >> > >> >> > > Strings are more
> "usable"
> > but
> > > bytes are
> > > > flexible
> > > > > >> and
> > > > > >> > so were
> > > > > >> > >> chosen.
> > > > > >> > >> >> > > Also - core kafka doesnt
> > even
> > > do that
> > > > good of a
> > > > > >> job
> > > > > >> > on
> > > > > >> > >> usability of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > payload (example - i
> have
> > to
> > > specify the
> > > > nop
> > > > > >> byte[]
> > > > > >> > "decoders"
> > > > > >> > >> >> > explicitly
> > > > > >> > >> >> > > in conf), and again
> > sacrificies
> > > usability
> > > > for
> > > > > the
> > > > > >> > sake of
> > > > > >> > >> >> > performance (no
> > > > > >> > >> >> > > convenient single-record
> > > processing as
> > > > poll is a
> > > > > >> > batch, lots of
> > > > > >> > >> >> > obscure
> > > > > >> > >> >> > > little config details
> > exposing
> > > internals
> > > > of the
> > > > > >> > batching
> > > > > >> > >> mechanism,
> > > > > >> > >> >> > etc)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > this is also why i
> really
> > > dislike the
> > > > idea of a
> > > > > >> > "type system"
> > > > > >> > >> for
> > > > > >> > >> >> > header
> > > > > >> > >> >> > > values, it further
> > degrades the
> > > > usability, adds
> > > > > >> > complexity and
> > > > > >> > >> will
> > > > > >> > >> >> > > eventually get in
> > people's way,
> > > also, it
> > > > would
> > > > > be
> > > > > >> > the 2nd/3rd
> > > > > >> > >> >> > home-group
> > > > > >> > >> >> > > serialization mechanism
> > in core
> > > kafka
> > > > (counting
> > > > > 2
> > > > > >> > iterations
> > > > > >> > >> of the
> > > > > >> > >> >> > "type
> > > > > >> > >> >> > > definition DSL")
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 2. this is an
> > implementation
> > > detail, and
> > > > not
> > > > > even
> > > > > >> a
> > > > > >> > very "user
> > > > > >> > >> >> > facing" one?
> > > > > >> > >> >> > > to the best of my
> > understanding
> > > the vote
> > > > process
> > > > > >> is
> > > > > >> > on proposed
> > > > > >> > >> >> > > API/behaviour. also -
> > since
> > > we're willing
> > > > to go
> > > > > >> with
> > > > > >> > strings
> > > > > >> > >> just
> > > > > >> > >> >> > serialize
> > > > > >> > >> >> > > a 0-sized header blob
> and
> > IIUC
> > > you dont
> > > > need any
> > > > > >> > optionals
> > > > > >> > >> anymore.
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 3. yes, we can :-)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > On Tue, Feb 14, 2017 at
> > 11:56
> > > PM, Michael
> > > > > Pearce <
> > > > > >> > >> >> > Michael.Pearce@ig.com>
> > > > > >> > >> >> > > wrote:
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > > Hi Jay,
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1) There was some
> > initial
> > > debate on the
> > > > value
> > > > > >> > part, as youll
> > > > > >> > >> note
> > > > > >> > >> >> > String,
> > > > > >> > >> >> > > > String headers were
> > > discounted early
> > > > on. The
> > > > > >> > reason for this
> > > > > >> > >> is
> > > > > >> > >> >> > > flexibility
> > > > > >> > >> >> > > > and keeping in line
> > with the
> > > > flexibility of
> > > > > key,
> > > > > >> > value of the
> > > > > >> > >> >> > message
> > > > > >> > >> >> > > > object itself. I don’t
> > think
> > > it takes
> > > > away
> > > > > from
> > > > > >> an
> > > > > >> > ecosystem
> > > > > >> > >> as
> > > > > >> > >> >> > each
> > > > > >> > >> >> > > plugin
> > > > > >> > >> >> > > > will care for their
> own
> > key,
> > > this way
> > > > ints,
> > > > > >> > booleans , exotic
> > > > > >> > >> >> > custom
> > > > > >> > >> >> > > binary
> > > > > >> > >> >> > > > can all be catered
> for=.
> > > > > >> > >> >> > > > a. If you really
> wanted
> > to
> > > push for a
> > > > typed
> > > > > >> value
> > > > > >> > interface,
> > > > > >> > >> I
> > > > > >> > >> >> > wouldn’t
> > > > > >> > >> >> > > > want just String
> values
> > > supported, but
> > > > the the
> > > > > >> > primatives
> > > > > >> > >> plus
> > > > > >> > >> >> > string and
> > > > > >> > >> >> > > > also still keeping the
> > > ability to have a
> > > > > binary
> > > > > >> > for custom
> > > > > >> > >> >> > binaries that
> > > > > >> > >> >> > > > some organisations may
> > have.
> > > > > >> > >> >> > > > i. I have written this
> > slight
> > > > alternative
> > > > > here,
> > > > > >> > >> >> > >
> https://cwiki.apache.org/
> > > > > >> > >> >> > > >
> > confluence/display/KAFKA/KIP-
> > > > > >> > 82+-+Add+Record+Headers+-+Typed
> > > > > >> > >> >> > > > ii. Essentially the
> > value
> > > bytes, has a
> > > > leading
> > > > > >> > byte overhead.
> > > > > >> > >> >> > > > 1. This tells you
> what
> > type
> > > the value
> > > > is,
> > > > > >> before
> > > > > >> > reading
> > > > > >> > >> the rest
> > > > > >> > >> >> > of the
> > > > > >> > >> >> > > > bytes, allowing
> > > > serialisation/deserialization
> > > > > to
> > > > > >> > and from the
> > > > > >> > >> >> > primitives,
> > > > > >> > >> >> > > > string and byte[].
> This
> > is
> > > akin to some
> > > > other
> > > > > >> > messaging
> > > > > >> > >> systems.
> > > > > >> > >> >> > > > 2) We are making it
> > optional,
> > > so that
> > > > for
> > > > > those
> > > > > >> > not wanting
> > > > > >> > >> >> > headers have
> > > > > >> > >> >> > > 0
> > > > > >> > >> >> > > > bytes overhead (think
> > of it
> > > as a feature
> > > > > flag),
> > > > > >> I
> > > > > >> > don’t
> > > > > >> > >> think this
> > > > > >> > >> >> > is
> > > > > >> > >> >> > > > complex, especially if
> > > comparing to
> > > > changes
> > > > > >> > proposed in
> > > > > >> > >> other kips
> > > > > >> > >> >> > like
> > > > > >> > >> >> > > > kip-98.
> > > > > >> > >> >> > > > a. If you really
> really
> > don’t
> > > like
> > > > this, we
> > > > > can
> > > > > >> > drop it, but
> > > > > >> > >> it
> > > > > >> > >> >> > would
> > > > > >> > >> >> > > mean
> > > > > >> > >> >> > > > buying into 4 bytes
> > extra
> > > overhead for
> > > > users
> > > > > who
> > > > > >> > do not want
> > > > > >> > >> to use
> > > > > >> > >> >> > > headers.
> > > > > >> > >> >> > > > 3) In the summary yes,
> > it is
> > > at a higher
> > > > > level,
> > > > > >> > but I think
> > > > > >> > >> this
> > > > > >> > >> >> > is well
> > > > > >> > >> >> > > > documented in the
> > proposed
> > > changes
> > > > section.
> > > > > >> > >> >> > > > a. Added getHeaders
> > method to
> > > > > Producer/Consumer
> > > > > >> > record (that
> > > > > >> > >> is it)
> > > > > >> > >> >> > > > b. We’ve also detailed
> > the
> > > new Headers
> > > > class
> > > > > >> that
> > > > > >> > this method
> > > > > >> > >> >> > returns
> > > > > >> > >> >> > > that
> > > > > >> > >> >> > > > encapsulates the
> headers
> > > protocol and
> > > > logic.
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Best,
> > > > > >> > >> >> > > > Mike
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > ==Original questions
> > from the
> > > vote
> > > > thread from
> > > > > >> > Jay.==
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Couple of things I
> > think we
> > > still need
> > > > to work
> > > > > >> out:
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1. I think we agree
> > about
> > > the key,
> > > > but I
> > > > > >> think
> > > > > >> > we haven't
> > > > > >> > >> >> > talked about
> > > > > >> > >> >> > > > the value yet. I
> > think if
> > > our goal
> > > > is an
> > > > > open
> > > > > >> > ecosystem
> > > > > >> > >> of these
> > > > > >> > >> >> > > header
> > > > > >> > >> >> > > > spread across many
> > plugins
> > > from many
> > > > > systems
> > > > > >> we
> > > > > >> > should
> > > > > >> > >> consider
> > > > > >> > >> >> > making
> > > > > >> > >> >> > > > this
> > > > > >> > >> >> > > > a string as well so
> > it can
> > > be
> > > > printed, set
> > > > > >> via
> > > > > >> > a UI, set
> > > > > >> > >> in
> > > > > >> > >> >> > config,
> > > > > >> > >> >> > > etc.
> > > > > >> > >> >> > > > Basically
> encouraging
> > > pluggable
> > > > > serialization
> > > > > >> > formats
> > > > > >> > >> here will
> > > > > >> > >> >> > lead
> > > > > >> > >> >> > > to
> > > > > >> > >> >> > > > a
> > > > > >> > >> >> > > > bit of a tower of
> > babel.
> > > > > >> > >> >> > > > 2. This proposal
> > still
> > > includes a
> > > > pretty
> > > > > big
> > > > > >> > change to our
> > > > > >> > >> >> > > serialization
> > > > > >> > >> >> > > > and protocol
> > definition
> > > layer.
> > > > Essentially
> > > > > >> it is
> > > > > >> > >> introducing an
> > > > > >> > >> >> > > optional
> > > > > >> > >> >> > > > type, where the
> > format is
> > > data
> > > > dependent. I
> > > > > >> > think this is
> > > > > >> > >> >> > actually a
> > > > > >> > >> >> > > big
> > > > > >> > >> >> > > > change though it
> > doesn't
> > > seem like
> > > > it. It
> > > > > >> means
> > > > > >> > you can no
> > > > > >> > >> >> > longer
> > > > > >> > >> >> > > > specify
> > > > > >> > >> >> > > > this type with our
> > type
> > > definition
> > > > DSL, and
> > > > > >> > likewise it
> > > > > >> > >> requires
> > > > > >> > >> >> > > custom
> > > > > >> > >> >> > > > handling in client
> > libs.
> > > This isn't
> > > > a huge
> > > > > >> > thing, since
> > > > > >> > >> the
> > > > > >> > >> >> > Record
> > > > > >> > >> >> > > > definition is
> custom
> > > anyway, but I
> > > > think
> > > > > this
> > > > > >> > kind of
> > > > > >> > >> protocol
> > > > > >> > >> >> > > > inconsistency is
> very
> > > non-desirable
> > > > and
> > > > > ties
> > > > > >> > you to
> > > > > >> > >> hand-coding
> > > > > >> > >> >> > > things.
> > > > > >> > >> >> > > > I
> > > > > >> > >> >> > > > think the type
> should
> > > instead by [Key
> > > > > Value]
> > > > > >> in
> > > > > >> > our BNF,
> > > > > >> > >> where
> > > > > >> > >> >> > key and
> > > > > >> > >> >> > > > value are both
> short
> > > strings as used
> > > > > >> elsewhere.
> > > > > >> > This
> > > > > >> > >> brings it
> > > > > >> > >> >> > in line
> > > > > >> > >> >> > > > with
> > > > > >> > >> >> > > > the rest of the
> > protocol.
> > > > > >> > >> >> > > > 3. Could we get
> more
> > > specific about
> > > > the
> > > > > exact
> > > > > >> > Java API
> > > > > >> > >> change to
> > > > > >> > >> >> > > > ProducerRecord,
> > > ConsumerRecord,
> > > > Record,
> > > > > etc?
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > -Jay
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > The information contained in
> > this
> > > email is
> > > > strictly
> > > > > >> > confidential and
> > > > > >> > >> for
> > > > > >> > >> >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > >> indicated.
> > > > > >> > If you
> > > > > >> > >> are not
> > > > > >> > >> >> > the intended recipient, please
> > do not
> > > read,
> > > > copy, use
> > > > > or
> > > > > >> > disclose to
> > > > > >> > >> others
> > > > > >> > >> >> > this message or any
> attachment.
> > > Please also
> > > > notify the
> > > > > >> > sender by
> > > > > >> > >> replying
> > > > > >> > >> >> > to this email or by telephone
> > > (+44(020 7896
> > > > 0011) and
> > > > > >> then
> > > > > >> > delete
> > > > > >> > >> the email
> > > > > >> > >> >> > and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc) that
> > > > > do
> > > > > >> > not relate
> > > > > >> > >> to the
> > > > > >> > >> >> > official business of this
> > company
> > > shall be
> > > > understood
> > > > > as
> > > > > >> > neither
> > > > > >> > >> given nor
> > > > > >> > >> >> > endorsed by it. IG is a
> trading
> > name
> > > of IG
> > > > Markets
> > > > > >> Limited
> > > > > >> > (a company
> > > > > >> > >> >> > registered in England and
> Wales,
> > > company number
> > > > > >> 04008957)
> > > > > >> > and IG
> > > > > >> > >> Index
> > > > > >> > >> >> > Limited (a company registered
> in
> > > England and
> > > > Wales,
> > > > > >> > company number
> > > > > >> > >> >> > 01190902). Registered address
> at
> > > Cannon Bridge
> > > > House,
> > > > > 25
> > > > > >> > Dowgate
> > > > > >> > >> Hill,
> > > > > >> > >> >> > London EC4R 2YA. Both IG
> Markets
> > > Limited
> > > > (register
> > > > > >> number
> > > > > >> > 195355)
> > > > > >> > >> and IG
> > > > > >> > >> >> > Index Limited (register number
> > > 114059) are
> > > > authorised
> > > > > >> and
> > > > > >> > regulated
> > > > > >> > >> by the
> > > > > >> > >> >> > Financial Conduct Authority.
> > > > > >> > >> >> >
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> --
> > > > > >> > >> Nacho - Ignacio Solis -
> > isolis@igso.net
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > The information contained in this email is
> > strictly
> > > > confidential and
> > > > > for
> > > > > >> > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you are
> > > > > >> not
> > > > > >> > the intended recipient, please do not read,
> > copy, use
> > > or
> > > > disclose to
> > > > > >> others
> > > > > >> > this message or any attachment. Please also
> > notify
> > > the sender by
> > > > > >> replying
> > > > > >> > to this email or by telephone (+44(020 7896
> > 0011) and
> > > then
> > > > delete the
> > > > > >> email
> > > > > >> > and any copies of it. Opinions, conclusion
> > (etc) that
> > > do not
> > > > relate to
> > > > > >> the
> > > > > >> > official business of this company shall be
> > understood
> > > as
> > > > neither given
> > > > > >> nor
> > > > > >> > endorsed by it. IG is a trading name of IG
> > Markets
> > > Limited (a
> > > > company
> > > > > >> > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG Index
> > > > > >> > Limited (a company registered in England and
> > Wales,
> > > company
> > > > number
> > > > > >> > 01190902). Registered address at Cannon Bridge
> > House,
> > > 25
> > > > Dowgate Hill,
> > > > > >> > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and
> > > > > IG
> > > > > >> > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > >> the
> > > > > >> > Financial Conduct Authority.
> > > > > >> >
> > > > > >> The information contained in this email is
> > strictly
> > > confidential
> > > > and for
> > > > > >> the use of the addressee only, unless otherwise
> > > indicated. If you
> > > > are
> > > > > not
> > > > > >> the intended recipient, please do not read,
> copy,
> > use
> > > or disclose
> > > > to
> > > > > others
> > > > > >> this message or any attachment. Please also
> > notify the
> > > sender by
> > > > > replying
> > > > > >> to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the
> > > > > email
> > > > > >> and any copies of it. Opinions, conclusion (etc)
> > that
> > > do not
> > > > relate to
> > > > > the
> > > > > >> official business of this company shall be
> > understood
> > > as neither
> > > > given
> > > > > nor
> > > > > >> endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > >> registered in England and Wales, company number
> > > 04008957) and IG
> > > > Index
> > > > > >> Limited (a company registered in England and
> > Wales,
> > > company number
> > > > > >> 01190902). Registered address at Cannon Bridge
> > House,
> > > 25 Dowgate
> > > > Hill,
> > > > > >> London EC4R 2YA. Both IG Markets Limited
> (register
> > > number 195355)
> > > > and IG
> > > > > >> Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > the
> > > > > >> Financial Conduct Authority.
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > The information contained in this email is strictly
> > > confidential and
> > > > for the use of the addressee only, unless otherwise
> > indicated.
> > > If you are
> > > > not the intended recipient, please do not read, copy, use
> > or
> > > disclose to
> > > > others this message or any attachment. Please also notify
> > the
> > > sender by
> > > > replying to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the email and any copies of it. Opinions, conclusion
> (etc)
> > that
> > > do not
> > > > relate to the official business of this company shall be
> > > understood as
> > > > neither given nor endorsed by it. IG is a trading name of
> > IG
> > > Markets
> > > > Limited (a company registered in England and Wales,
> company
> > > number
> > > > 04008957) and IG Index Limited (a company registered in
> > England
> > > and Wales,
> > > > company number 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and IG Index Limited (register number 114059) are
> > > authorised and
> > > > regulated by the Financial Conduct Authority.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential
> > and
> > > for the use of the addressee only, unless otherwise indicated. If
> > you are
> > > not the intended recipient, please do not read, copy, use or
> > disclose to
> > > others this message or any attachment. Please also notify the
> sender
> > by
> > > replying to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > relate to the official business of this company shall be understood
> > as
> > > neither given nor endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a company registered in England and Wales, company number
> > > 04008957) and IG Index Limited (a company registered in England and
> > Wales,
> > > company number 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > 195355) and IG Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
>
> headers dont "leak" into application code. they are useful to application
> code as well.
This is exactly what I have been trying to get at. The use cases documented
here are middleware:
https://cwiki.apache.org/confluence/display/KAFKA/A+Case+for+Kafka+Headers.
If headers are intended for the application space as well, the document
should be updated accordingly so that it is an explicit design goal and not
an unfortunate design byproduct. There may be some disagreement on whether
it _should_ be a design goal, but it may not matter much since the current
interceptor API is probably insufficient for middleware applications (I
haven't thought of a way to do this that isn't cumbersome).
That aside, the three unresolved points for me are the following:
1. The use of varints. As a one-off for use only with record headers, the
case was weak, but if we are using them throughout the message format, then
we should do so here as well. The additional complexity is minimal and
early performance testing fully justifies their use.
2. If varints are used, the case for using attributes to indicate null
headers also becomes weak. We only add one additional byte in each message
if there are no headers. Whatever the case, let us be consistent with how
we treat null keys and values.
3. We have apparently agreed that the broker will validate headers (please
add this to the KIP). That being the case, I would prefer to use Kafka's
conventional format for arrays. The downside is that it takes more work to
skip over the headers on the consumer, though it's unclear if the cost
would matter in practice. Some concern was previously expressed about the
allocation of maps. An alternative would be to use arrays, i.e.
class ConsumerRecord<K, V> {
K key;
V value;
Header[] headers;
}
class Header {
String key;
byte[] value;
}
This would work nicely with the conventional array format and my guess is
it would obviate the need do any lazy initialization. If we use the map as
is currently documented, then it is possible with either representation to
slice the headers and initialize them lazily. Either way, it might be a
good idea to use a separate object to represent the headers in case we need
to modify it in the future in some way.
(By the way, doesn't it feel a bit odd that we seem to be designing a
feature which is optimized for people not using?)
If we can resolve these points, then at least you will get my vote.
Thanks,
Jason
On Sun, Feb 19, 2017 at 7:30 AM, radai <ra...@gmail.com> wrote:
> headers dont "leak" into application code. they are useful to application
> code as well.
> IIUC samze currently has headers "in-V" and would just switch over to kafka
> headers if they exist.
> im sure plenty of other users of kafka would have a use for headers.
> im pretty sure use cases exist around shuffling data into/out-of kafka
> (kafka connect or equivalent) where metadata from one end could copied over
> to the other (S3, for example uses http headers for user-accessible
> metadata). it will be kafka client code getting/setting those headers. not
> an interceptor.
>
> On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > For APM single event tracing, need access to the header at the point of
> > processing on the processing thread.
> >
> > As such interceptors will not work/be suitable for these, due to the fact
> > they act on the ConsumerRecords as a batch, before the handling thread
> can
> > split out and process per message which is the point these tools will
> need
> > to continue to transaction tracing.
> >
> > Like wise tools and infra pieces will need access to the message outside
> > the interceptor realm.
> >
> >
> >
> > On 17/02/2017, 21:26, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> >
> >
> > So the broker will parse the headers and validate them. Good. The
> only
> > reason remaining that I can see to leave the headers as a byte array
> > is to
> > make it easier for the client to skip past them. Are we sure this is
> > not
> > premature optimization? Are there any performance results which show
> > that
> > this is worthwhile?
> >
> > What’s the issue with exposing a method getHeaders on the
> > producer/consumer
> > > record? It doesn’t break anything. We don’t need any special
> version.
> >
> >
> > See my previous explanation. What I am trying to resist is the
> headers
> > becoming a general application-level facility. The primary use case
> as
> > far
> > as I can tell is middleware, which is the use case that the
> > interceptors
> > are providing.
> >
> > Current batch consumer model and consumer interceptors don’t work
> where
> > > headers need to be acted on at per message level at time of
> > processing,
> > > very case is APM (the core one), where the header value is used to
> > continue
> > > tracing.
> >
> >
> > I still don't understand the point about batching. The consumer
> > records are
> > exposed as a batch in the consumer interceptor, but you can still
> > iterate
> > them individually. It is no different for the consumer API itself.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > Re:
> > >
> > > “ The point about creation of maps seems orthogonal. We can
> still
> > > represent
> > > the headers as a slice of bytes until the time it is accessed.”
> > >
> > > That’s exactly what we’re doing the headers are a slice of bytes,
> > which
> > > then gets parsed later if needed, or can be parsed right away, the
> > headers
> > > is part of the protocol, so can still be validated if wanted.
> > >
> > > If you had a header count then you would have to go through each
> > header
> > > key and value length value to work out how much to skip to get to
> > say the
> > > value or any future component in the message after the headers.
> > Having it
> > > as a byte[] with length value makes this a lot easier to skip.
> > >
> > >
> > > On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com>
> > wrote:
> > >
> > > What’s the issue with exposing a method getHeaders on the
> > > producer/consumer record? It doesn’t break anything. We don’t need
> > any
> > > special version.
> > >
> > > Current batch consumer model and consumer interceptors don’t
> work
> > > where headers need to be acted on at per message level at time of
> > > processing, very case is APM (the core one), where the header value
> > is used
> > > to continue tracing. JMS/HTTP etc all expose these, without issues.
> > I would
> > > NOT want to lock this down to only be usable accessible via
> > interceptors,
> > > as we’d fail on one of the main goals.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > >
> > >
> > > On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io>
> > wrote:
> > >
> > > The point about creation of maps seems orthogonal. We can
> > still
> > > represent
> > > the headers as a slice of bytes until the time it is
> > accessed.
> > >
> > >
> > > > Yes exactly we have access to the records thus why the
> > header
> > > should be
> > > > accessible via it and not hidden for only interceptors to
> > access.
> > >
> > >
> > > As explained above, the point is to make the intended usage
> > clear.
> > > Applications should continue to rely on the key/value
> fields
> > to
> > > serialize
> > > their own headers, and it would be more ideal if we can
> avoid
> > > leaking
> > > third-party headers into applications. This is difficult to
> > do
> > > with the
> > > current interceptors because they share the record objects
> > with
> > > the common
> > > API. What I had in mind is something like an extension of
> the
> > > current
> > > interceptors which exposed a different object (e.g.
> > > `RecordAndHeaders`).
> > > The challenge is for MM-like use cases. Let me see if I can
> > come
> > > up with a
> > > concrete proposal for that problem.
> > >
> > > -Jason
> > >
> > >
> > >
> > > On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > I am happy to move the definition of the header into the
> > message
> > > body, but
> > > > would cause us not to lazy initialise/parse the headers,
> as
> > > obviously, we
> > > > would have to traverse these reading the message.
> > > >
> > > > This was actually one of Jay’s requests:
> > > >
> > > > “ 2. I think we should think about creating the Map
> > lazily to
> > > avoid
> > > > parsing out all the headers into little objects.
> > HashMaps
> > > themselves
> > > > are
> > > > kind of expensive and the consumer is very perf
> > sensitive so
> > > and making
> > > > gazillions of hashmaps that may or may not get used
> is
> > > probably a bad
> > > > idea.”
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On 17/02/2017, 19:44, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > wrote:
> > > >
> > > > Yes exactly we have access to the records thus why
> the
> > > header should
> > > > be accessible via it and not hidden for only interceptors
> > to
> > > access.
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Magnus Edenhill <ma...@edenhill.se>
> > > > Sent: Friday, February 17, 2017 7:34:49 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >
> > > > Big +1 on VarInts.
> > > > CPUs are fast, memory is slow.
> > > >
> > > > I agree with Jason that we'll want to continue
> > verifying
> > > messages,
> > > > including their headers, so while I appreciate the
> > idea of
> > > the opaque
> > > > header blob it won't be useful in practice.
> > > >
> > > > /Magnus
> > > >
> > > > 2017-02-17 10:41 GMT-08:00 Jason Gustafson <
> > > jason@confluent.io>:
> > > >
> > > > > Sorry, my mistake. The consumer interceptor is per
> > batch,
> > > though I'm
> > > > not
> > > > > sure that's an actual limitation since you still
> have
> > > access to the
> > > > > individual records.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> > > > jason@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > Re headers as byte array and future use by
> broker.
> > This
> > > doesn't
> > > > take away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >
> > > > > >
> > > > > > Yeah, I didn't say it was difficult, only
> awkward.
> > You
> > > wouldn't
> > > > write the
> > > > > > schema that way if you were planning to use it on
> > the
> > > brokers from
> > > > the
> > > > > > beginning. Note also that one of the benefits of
> > letting
> > > the broker
> > > > > > understand headers is that it can validate that
> > they are
> > > properly
> > > > > > formatted. If cost is the only concern, we should
> > > confirm its
> > > > impact
> > > > > > through performance testing.
> > > > > >
> > > > > > One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >
> > > > > >
> > > > > > I'm not sure I understand the point about
> batching.
> > > Interceptors
> > > > are
> > > > > > applied per-message, right?
> > > > > >
> > > > > > My intent on interceptors is to keep the usage of
> > headers
> > > > well-defined so
> > > > > > that they don't start leaking unnecessarily into
> > > applications. My
> > > > guess
> > > > > is
> > > > > > that it's probably inevitable, but isolating it
> in
> > the
> > > > interceptors would
> > > > > > at least give people a second thought before
> > deciding to
> > > use it.
> > > > The main
> > > > > > challenge in my mind is figuring out how an MM
> use
> > case
> > > would
> > > > work. It
> > > > > > would be more cumbersome to replicate headers
> > through an
> > > > interceptor,
> > > > > > though arguably MM should be working at a lower
> > level
> > > anyway.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Re headers available on the record va
> > interceptors only
> > > > > >>
> > > > > >> One of the key use cases requires access on
> > consume at
> > > per
> > > > event/message
> > > > > >> level at the point that message is being
> > processed, as
> > > such the
> > > > batch
> > > > > >> interceptors and batch consume api isn't
> > suitable. It
> > > needs to be
> > > > at the
> > > > > >> record level.
> > > > > >>
> > > > > >> This anyhow is similar to jms/http/amqp where
> > headers
> > > are
> > > > available to
> > > > > >> consuming applications.
> > > > > >>
> > > > > >> Re headers as byte array and future use by
> > broker. This
> > > doesn't
> > > > take
> > > > > away
> > > > > >> from that at all. Nor makes it difficult at all
> > in my
> > > opinion.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Sent using OWA for iPhone
> > > > > >> ________________________________________
> > > > > >> From: Jason Gustafson <ja...@confluent.io>
> > > > > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > > > > >> To: dev@kafka.apache.org
> > > > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> > > > > >>
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >>
> > > > > >>
> > > > > >> Yes, that is what we are proposing, so using
> > varints
> > > for headers
> > > > would
> > > > > be
> > > > > >> consistent with the rest of the message. We have
> > used
> > > static
> > > > helper
> > > > > >> methods
> > > > > >> in our prototype implementation.
> > > > > >>
> > > > > >> The cost of parsing, we want to parse/interpret
> > the
> > > headers
> > > > lazily (this
> > > > > >> is
> > > > > >> > a key point brought up earlier in discussions)
> > > > > >>
> > > > > >>
> > > > > >> I'm a bit skeptical of this. Has anyone done the
> > > performance
> > > > testing? I
> > > > > >> can
> > > > > >> probably implement it and test it if no one else
> > has. I
> > > was also
> > > > under
> > > > > the
> > > > > >> impression that there may be use cases down the
> > road
> > > where the
> > > > broker
> > > > > >> would
> > > > > >> need to interpret headers. That wouldn't be off
> > the
> > > table in the
> > > > future
> > > > > if
> > > > > >> it's represented as bytes, but it would be quite
> > a bit
> > > more
> > > > awkward,
> > > > > >> right?
> > > > > >>
> > > > > >> By the way, one question I have been wondering
> > about. My
> > > > understanding
> > > > > is
> > > > > >> that headers are primarily for use cases where a
> > > third-party
> > > > components
> > > > > >> wants to enrich messages without needing to
> > understand
> > > or modify
> > > > the
> > > > > >> schema
> > > > > >> of the message key and value. For the
> applications
> > > which directly
> > > > > produce
> > > > > >> and consume the messages and control the
> key/value
> > > schema
> > > > directly, it
> > > > > >> seems we would rather have them implement
> headers
> > > directly in
> > > > their own
> > > > > >> schema. Supposing for the sake of argument that
> > it was
> > > possible,
> > > > my
> > > > > >> question is whether it be sufficient to expose
> the
> > > headers in the
> > > > > >> interceptor API and not in the common API?
> > > > > >>
> > > > > >> -Jason
> > > > > >>
> > > > > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce
> <
> > > > Michael.Pearce@ig.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On the point of varInts
> > > > > >> >
> > > > > >> > Would you be proposing in KIP-98 to convert
> the
> > other
> > > message
> > > > int’s
> > > > > (key
> > > > > >> > length, value length) also to varint to keep
> it
> > > uniform.
> > > > > >> > Also I assume there will be a static or helper
> > method
> > > made to
> > > > > write/read
> > > > > >> > these in the client and server.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 11:22, "Michael Pearce" <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > > > >> >
> > > > > >> > On the point re: headers in the message
> > protocol
> > > being a
> > > > byte
> > > > > array
> > > > > >> > and not a count of elements followed by the
> > elements.
> > > Again
> > > > this was
> > > > > >> > discussed/argued previously.
> > > > > >> >
> > > > > >> > It was agreed on for a few reasons some of
> > which
> > > you have
> > > > > obviously
> > > > > >> > picked up on:
> > > > > >> >
> > > > > >> > Broker is able to pass it through opaquely
> > > > > >> > The cost of parsing, we want to
> > parse/interpret
> > > the headers
> > > > lazily
> > > > > >> > (this is a key point brought up earlier in
> > > discussions)
> > > > > >> > Headers can be copied from consumer record
> > to
> > > producer
> > > > record (aka
> > > > > >> > mirror makers etc) without parsing if no
> > changes are
> > > being made
> > > > or
> > > > > being
> > > > > >> > looked at.
> > > > > >> > Keeps the broker agnostic to the format
> > > > > >> > You need an int32 either for the byte size
> > of the
> > > headers,
> > > > or for
> > > > > >> the
> > > > > >> > count of elements, so overheads are the same,
> > but
> > > with going
> > > > with an
> > > > > >> opaque
> > > > > >> > byte array has the above advantages.
> > > > > >> >
> > > > > >> > Cheers
> > > > > >> > Mike
> > > > > >> >
> > > > > >> >
> > > > > >> > On 17/02/2017, 02:50, "Jason Gustafson" <
> > > jason@confluent.io
> > > > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > Sorry, should have noted that the
> > performance
> > > testing
> > > > was done
> > > > > >> > using the
> > > > > >> > producer performance tool shipped with
> > Kafka.
> > > > > >> >
> > > > > >> > -Jason
> > > > > >> >
> > > > > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason
> > > Gustafson <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> >
> > > > > >> > > Hey Nacho,
> > > > > >> > >
> > > > > >> > > I've compared performance of our
> > KIP-98
> > > > implementation with
> > > > > >> and
> > > > > >> > without
> > > > > >> > > varints. For messages around 128
> > bytes, we
> > > see an
> > > > increase
> > > > > in
> > > > > >> > throughput of
> > > > > >> > > about 30% using the default
> > configuration
> > > settings.
> > > > At 256
> > > > > >> > bytes, the
> > > > > >> > > increase is around 16%. Obviously
> the
> > > performance
> > > > converge
> > > > > as
> > > > > >> > messages get
> > > > > >> > > larger, but it seems well worth the
> > cost.
> > > Note that
> > > > we are
> > > > > >> also
> > > > > >> > seeing a
> > > > > >> > > substantial performance increase
> > against
> > > trunk
> > > > primarily
> > > > > >> because
> > > > > >> > of the
> > > > > >> > > much more efficient packing that
> > varints
> > > provide us.
> > > > > Anything
> > > > > >> > adding to
> > > > > >> > > message overhead, such as record
> > headers,
> > > would only
> > > > > increase
> > > > > >> > the relative
> > > > > >> > > difference. (Of course take these
> > numbers
> > > with a
> > > > grain of
> > > > > salt
> > > > > >> > since I have
> > > > > >> > > only used the default settings with
> > both
> > > the producer
> > > > and
> > > > > >> broker
> > > > > >> > on my
> > > > > >> > > local machine. We intend to provide
> > more
> > > extensive
> > > > > performance
> > > > > >> > details as
> > > > > >> > > part of the work for KIP-98.)
> > > > > >> > >
> > > > > >> > > The implementation we are using is
> > from
> > > protobuf (
> > > > > >> > > https://developers.google.com/
> > > > > protocol-buffers/docs/encoding
> > > > > >> ),
> > > > > >> > which is
> > > > > >> > > also used in HBase. It is trivial to
> > > implement and as
> > > > far
> > > > > as I
> > > > > >> > know doesn't
> > > > > >> > > suffer from the aliasing problem you
> > are
> > > describing. I
> > > > > checked
> > > > > >> > with Magnus
> > > > > >> > > (the author of librdkafka) and he
> > agreed
> > > that the
> > > > savings
> > > > > >> seemed
> > > > > >> > worth the
> > > > > >> > > cost of implementation.
> > > > > >> > >
> > > > > >> > > -Jason
> > > > > >> > >
> > > > > >> > > On Thu, Feb 16, 2017 at 4:32 PM,
> > Ignacio
> > > Solis <
> > > > > >> isolis@igso.net>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> -VarInts
> > > > > >> > >>
> > > > > >> > >> I'm one of the people (if not the
> > most)
> > > opposed to
> > > > VarInts.
> > > > > >> > VarInts
> > > > > >> > >> have a place, but this is not it.
> > (We
> > > had a large
> > > > > >> discussion
> > > > > >> > about
> > > > > >> > >> them at the beginning of KIP-82
> time)
> > > > > >> > >>
> > > > > >> > >> If anybody has real life
> performance
> > > numbers of
> > > > VarInts
> > > > > >> > improving
> > > > > >> > >> things or significantly reducing
> > resources
> > > I would
> > > > like to
> > > > > >> know
> > > > > >> > what
> > > > > >> > >> that case may be. Yes, you can save
> > some
> > > bytes here
> > > > and
> > > > > >> there,
> > > > > >> > but
> > > > > >> > >> this is probably insignificant to
> the
> > > overall system
> > > > > behavior
> > > > > >> > and
> > > > > >> > >> storage requirements. -- I say
> this
> > with
> > > respect to
> > > > using
> > > > > >> > VarInts in
> > > > > >> > >> the protocol itself, not as part of
> > the
> > > data.
> > > > > >> > >>
> > > > > >> > >> VarInts require you to parse the
> Int
> > > before using it
> > > > and
> > > > > >> > depending on
> > > > > >> > >> the encoding they can suffer from
> > aliasing
> > > (multiple
> > > > > >> > representations
> > > > > >> > >> for the same value).
> > > > > >> > >>
> > > > > >> > >> Why add complexity?
> > > > > >> > >>
> > > > > >> > >> Nacho
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> On Thu, Feb 16, 2017 at 10:29 AM,
> > Colin
> > > McCabe <
> > > > > >> > cmccabe@apache.org>
> > > > > >> > >> wrote:
> > > > > >> > >> > +1 for varints here-- it would
> save
> > > quite a bit of
> > > > space.
> > > > > >> > They are
> > > > > >> > >> > pretty quick to implement as
> well.
> > > > > >> > >> >
> > > > > >> > >> > I think it makes sense for values
> > to be
> > > byte
> > > > arrays.
> > > > > Users
> > > > > >> > might want
> > > > > >> > >> > to attach arbitrary payloads;
> they
> > > shouldn't be
> > > > forced to
> > > > > >> > serialize
> > > > > >> > >> > everything to Java strings.
> > > > > >> > >> >
> > > > > >> > >> > best,
> > > > > >> > >> > Colin
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > On Thu, Feb 16, 2017, at 09:52,
> > Jason
> > > Gustafson
> > > > wrote:
> > > > > >> > >> >> Hey Michael,
> > > > > >> > >> >>
> > > > > >> > >> >> Hmm, I guess the point of
> > representing
> > > it as
> > > > bytes is to
> > > > > >> > allow the
> > > > > >> > >> broker
> > > > > >> > >> >> to pass it through opaquely? Is
> > the
> > > cost of
> > > > parsing
> > > > > them a
> > > > > >> > concern, or
> > > > > >> > >> >> are
> > > > > >> > >> >> we simply trying to ensure that
> > the
> > > broker stays
> > > > > agnostic
> > > > > >> to
> > > > > >> > the
> > > > > >> > >> format?
> > > > > >> > >> >>
> > > > > >> > >> >> On varints, I think adding
> > support for
> > > them makes
> > > > less
> > > > > >> sense
> > > > > >> > for an
> > > > > >> > >> >> isolated use case, but as part
> of
> > a
> > > more holistic
> > > > change
> > > > > >> > (such as what
> > > > > >> > >> we
> > > > > >> > >> >> have proposed in KIP-98), I
> think
> > they
> > > are
> > > > justifiable.
> > > > > If
> > > > > >> > we add them,
> > > > > >> > >> >> then the need to use attributes
> > becomes
> > > quite a
> > > > bit
> > > > > >> weaker,
> > > > > >> > right? The
> > > > > >> > >> >> other thing I find slightly odd
> > is the
> > > fact that
> > > > null
> > > > > >> > headers has no
> > > > > >> > >> >> actual
> > > > > >> > >> >> semantic meaning for the message
> > > (unlike null
> > > > keys and
> > > > > >> > values). It is
> > > > > >> > >> >> just
> > > > > >> > >> >> a space optimization. It seems a
> > bit
> > > better to
> > > > always
> > > > > use
> > > > > >> > size 0 to
> > > > > >> > >> >> indicate having no headers.
> > > > > >> > >> >>
> > > > > >> > >> >> Overall, the main point is
> > ensuring
> > > that the
> > > > message
> > > > > >> schema
> > > > > >> > remains
> > > > > >> > >> >> consistent, either within the
> > larger
> > > protocol, or
> > > > at a
> > > > > >> > minimum within
> > > > > >> > >> the
> > > > > >> > >> >> message itself.
> > > > > >> > >> >>
> > > > > >> > >> >> -Jason
> > > > > >> > >> >>
> > > > > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM,
> > > Michael Pearce <
> > > > > >> > Michael.Pearce@ig.com
> > > > > >> > >> >
> > > > > >> > >> >> wrote:
> > > > > >> > >> >>
> > > > > >> > >> >> > Hi Jason,
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 1) in the message
> > protocol
> > > the headers
> > > > are
> > > > > >> simply
> > > > > >> > a byte
> > > > > >> > >> array,
> > > > > >> > >> >> > as like the key or value, this
> > is to
> > > clearly
> > > > demarcate
> > > > > >> the
> > > > > >> > header in
> > > > > >> > >> the
> > > > > >> > >> >> > core message. Then the header
> > byte
> > > array in the
> > > > core
> > > > > >> > message is an
> > > > > >> > >> array of
> > > > > >> > >> >> > key, value pairs. This is what
> > it is
> > > denoting.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Then this would be I guess in
> > the
> > > given
> > > > notation:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [KeyLength, Key,
> > > ValueLength, Value]
> > > > > >> > >> >> > KeyLength => int32
> > > <-----------------NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised key value
> > > > > >> > >> >> > Key => bytes
> > > <---------------------- NEW
> > > > > serialised
> > > > > >> > string (UTF8)
> > > > > >> > >> >> > bytes of the header key
> > > > > >> > >> >> > ValueLength => int32
> > > <-------------- NEW
> > > > size of
> > > > > the
> > > > > >> > byte[] of
> > > > > >> > >> the
> > > > > >> > >> >> > serialised header value
> > > > > >> > >> >> > Value => bytes
> > > <-------------------- NEW
> > > > > serialised
> > > > > >> > form of the
> > > > > >> > >> header
> > > > > >> > >> >> > value
> > > > > >> > >> >> >
> > > > > >> > >> >> > The key length and value
> length
> > is
> > > matching the
> > > > way
> > > > > the
> > > > > >> > protocol is
> > > > > >> > >> >> > defined in the core message
> > currently.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 2)
> > > > > >> > >> >> > Var sized ints, this was
> > discussed
> > > much earlier
> > > > on, in
> > > > > >> > fact I had
> > > > > >> > >> >> > suggested it myself (with
> Hadoop
> > > references),
> > > > the
> > > > > >> > complexity of this
> > > > > >> > >> >> > compared to having a simpler
> > protocol
> > > was
> > > > argued and
> > > > > >> > agreed it
> > > > > >> > >> wasn’t worth
> > > > > >> > >> >> > the complexity as all other
> > clients
> > > in other
> > > > languages
> > > > > >> > would need to
> > > > > >> > >> ensure
> > > > > >> > >> >> > theyre using the right var
> size
> > > algorithm, as
> > > > there
> > > > > is a
> > > > > >> > few.
> > > > > >> > >> >> >
> > > > > >> > >> >> > On point 3)
> > > > > >> > >> >> > We did the attributes,
> optional
> > > approach as
> > > > originally
> > > > > >> > there was
> > > > > >> > >> marked
> > > > > >> > >> >> > concern that headers would
> > cause a
> > > message size
> > > > > overhead
> > > > > >> > for others,
> > > > > >> > >> who
> > > > > >> > >> >> > don’t want them. As such this
> > is the
> > > clean
> > > > solution to
> > > > > >> > achieve that.
> > > > > >> > >> If
> > > > > >> > >> >> > that no longer holds, and we
> > don’t
> > > care that we
> > > > add
> > > > > >> 4bytes
> > > > > >> > overhead,
> > > > > >> > >> then
> > > > > >> > >> >> > im happy to remove.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I’m personally in favour of
> > keeping
> > > the message
> > > > as
> > > > > small
> > > > > >> > as possible
> > > > > >> > >> so
> > > > > >> > >> >> > people don’t get shocks in
> perf
> > and
> > > throughputs
> > > > dues
> > > > > to
> > > > > >> > message size,
> > > > > >> > >> >> > unless they actively use the
> > feature,
> > > as such I
> > > > do
> > > > > >> prefer
> > > > > >> > the
> > > > > >> > >> attribute bit
> > > > > >> > >> >> > wise feature flag approach
> > myself.
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On 16/02/2017, 05:40, "Jason
> > > Gustafson" <
> > > > > >> > jason@confluent.io> wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > We have proposed a few
> > > significant changes
> > > > to the
> > > > > >> > message format
> > > > > >> > >> in
> > > > > >> > >> >> > KIP-98
> > > > > >> > >> >> > which now seems likely to
> > pass
> > > (perhaps
> > > > with some
> > > > > >> > iterations on
> > > > > >> > >> >> > implementation details).
> It
> > would
> > > be good
> > > > to try
> > > > > and
> > > > > >> > coordinate
> > > > > >> > >> the
> > > > > >> > >> >> > changes
> > > > > >> > >> >> > in both of the proposals
> to
> > make
> > > sure they
> > > > are
> > > > > >> > consistent and
> > > > > >> > >> >> > compatible.
> > > > > >> > >> >> >
> > > > > >> > >> >> > I think using the
> > attributes to
> > > indicate
> > > > null
> > > > > >> headers
> > > > > >> > is a
> > > > > >> > >> reasonable
> > > > > >> > >> >> > approach. We have proposed
> > to do
> > > the same
> > > > thing
> > > > > for
> > > > > >> > the message
> > > > > >> > >> key and
> > > > > >> > >> >> > value. That said, I
> > sympathize
> > > with Jay's
> > > > > argument.
> > > > > >> > Having
> > > > > >> > >> multiple
> > > > > >> > >> >> > ways to
> > > > > >> > >> >> > specify a null value
> > increases
> > > the overall
> > > > > >> complexity
> > > > > >> > of the
> > > > > >> > >> protocol.
> > > > > >> > >> >> > You
> > > > > >> > >> >> > can see this just from the
> > fact
> > > that you
> > > > need the
> > > > > >> > extra verbiage
> > > > > >> > >> in the
> > > > > >> > >> >> > protocol specification in
> > this
> > > KIP and in
> > > > KIP-98
> > > > > to
> > > > > >> > describe the
> > > > > >> > >> >> > dependence
> > > > > >> > >> >> > between the fields and the
> > > attributes. It
> > > > seems
> > > > > >> like a
> > > > > >> > slippery
> > > > > >> > >> slope
> > > > > >> > >> >> > if
> > > > > >> > >> >> > you start allowing
> different
> > > request types
> > > > to
> > > > > >> > implement the
> > > > > >> > >> protocol
> > > > > >> > >> >> > specification differently.
> > > > > >> > >> >> >
> > > > > >> > >> >> > You can also argue that
> the
> > > messages
> > > > already are
> > > > > and
> > > > > >> > are likely
> > > > > >> > >> to
> > > > > >> > >> >> > remain a
> > > > > >> > >> >> > special case. For example,
> > there
> > > is
> > > > currently no
> > > > > >> > generality in
> > > > > >> > >> how
> > > > > >> > >> >> > compressed message sets
> are
> > > represented
> > > > that would
> > > > > >> be
> > > > > >> > applicable
> > > > > >> > >> for
> > > > > >> > >> >> > other
> > > > > >> > >> >> > request types. Some might
> > see this
> > > > divergence as
> > > > > an
> > > > > >> > unfortunate
> > > > > >> > >> >> > protocol
> > > > > >> > >> >> > deficiency which should be
> > fixed;
> > > others
> > > > might see
> > > > > >> it
> > > > > >> > as sort of
> > > > > >> > >> the
> > > > > >> > >> >> > inevitability of needing
> to
> > > optimize where
> > > > it
> > > > > counts
> > > > > >> > most. I'm
> > > > > >> > >> probably
> > > > > >> > >> >> > somewhere in between, but
> I
> > think
> > > we
> > > > probably all
> > > > > >> > share the
> > > > > >> > >> intuition
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the protocol should be
> kept
> > as
> > > consistent as
> > > > > >> possible.
> > > > > >> > With that
> > > > > >> > >> in
> > > > > >> > >> >> > mind,
> > > > > >> > >> >> > here are a few comments:
> > > > > >> > >> >> >
> > > > > >> > >> >> > 1. One thing I found a
> > little odd
> > > when
> > > > reading the
> > > > > >> > current
> > > > > >> > >> proposal is
> > > > > >> > >> >> > that
> > > > > >> > >> >> > the headers are both
> > represented
> > > as an
> > > > array of
> > > > > >> bytes
> > > > > >> > and as an
> > > > > >> > >> array
> > > > > >> > >> >> > of
> > > > > >> > >> >> > key/value pairs. I'd
> > probably
> > > suggest
> > > > something
> > > > > like
> > > > > >> > this:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers => [HeaderKey
> > HeaderValue]
> > > > > >> > >> >> > HeaderKey => String
> > > > > >> > >> >> > HeaderValue => Bytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > An array in the Kafka
> > protocol is
> > > > represented as a
> > > > > >> > 4-byte integer
> > > > > >> > >> >> > indicating the number of
> > elements
> > > in the
> > > > array
> > > > > >> > followed by the
> > > > > >> > >> >> > serialization of the
> > elements.
> > > Unless I'm
> > > > > >> > misunderstanding, what
> > > > > >> > >> you
> > > > > >> > >> >> > have
> > > > > >> > >> >> > instead is the total size
> > of the
> > > headers in
> > > > bytes
> > > > > >> > followed by the
> > > > > >> > >> >> > elements.
> > > > > >> > >> >> > I'm not sure I see any
> > reason for
> > > this
> > > > > >> inconsistency.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 2. In KIP-98, we've
> > introduced
> > > > variable-length
> > > > > >> integer
> > > > > >> > fields.
> > > > > >> > >> >> > Effectively,
> > > > > >> > >> >> > we've enriched (or
> > "complicated"
> > > as Jay
> > > > might say
> > > > > ;)
> > > > > >> > the protocol
> > > > > >> > >> >> > specification to include
> the
> > > following
> > > > types:
> > > > > >> VarInt,
> > > > > >> > VarLong,
> > > > > >> > >> >> > UnsignedVarInt and
> > > UnsignedVarLong.
> > > > > >> > >> >> >
> > > > > >> > >> >> > Along with these
> > primitives, we
> > > could
> > > > introduce
> > > > > the
> > > > > >> > following
> > > > > >> > >> types:
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeArray =>
> > NumberOfItems
> > > Item1 Item2
> > > > .. ItemN
> > > > > >> > >> >> > NumberOfItems =>
> > UnsignedVarInt
> > > > > >> > >> >> >
> > > > > >> > >> >> > VarSizeNullableArray =>
> > > NumberOfItemsOrNull
> > > > Item1
> > > > > >> > Item2 .. ItemN
> > > > > >> > >> >> > NumberOfItemsOrNull =>
> > VarInt
> > > (-1 means
> > > > null)
> > > > > >> > >> >> >
> > > > > >> > >> >> > And similarly for the
> > `String`
> > > and `Bytes`
> > > > types.
> > > > > >> > These types
> > > > > >> > >> can save
> > > > > >> > >> >> > a
> > > > > >> > >> >> > considerable amount of
> > space in
> > > this
> > > > proposal
> > > > > >> because
> > > > > >> > they can
> > > > > >> > >> be used
> > > > > >> > >> >> > for
> > > > > >> > >> >> > both the number of headers
> > > included in the
> > > > message
> > > > > >> and
> > > > > >> > the
> > > > > >> > >> lengths of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > header keys and values. We
> > could
> > > do this
> > > > instead:
> > > > > >> > >> >> >
> > > > > >> > >> >> > Headers =>
> > VarSizeArray[HeaderKey
> > > > HeaderValue]
> > > > > >> > >> >> > HeaderKey =>
> VarSizeString
> > > > > >> > >> >> > HeaderValue =>
> > VarSizeBytes
> > > > > >> > >> >> >
> > > > > >> > >> >> > Combining the savings from
> > the
> > > use of
> > > > variable
> > > > > >> length
> > > > > >> > fields, the
> > > > > >> > >> >> > benefit
> > > > > >> > >> >> > of using the attributes to
> > > represent null
> > > > seems
> > > > > >> pretty
> > > > > >> > small.
> > > > > >> > >> >> >
> > > > > >> > >> >> > 3. Whichever way we go
> > (whether
> > > we use the
> > > > > >> attributes
> > > > > >> > or not), we
> > > > > >> > >> >> > should at
> > > > > >> > >> >> > least be consistent
> between
> > this
> > > KIP and
> > > > KIP-98.
> > > > > It
> > > > > >> > would be very
> > > > > >> > >> >> > strange
> > > > > >> > >> >> > to have two ways to
> > represent
> > > null values
> > > > in the
> > > > > >> same
> > > > > >> > schema.
> > > > > >> > >> Either
> > > > > >> > >> >> > way is
> > > > > >> > >> >> > OK with me. I think some
> > > message-level
> > > > > optimizations
> > > > > >> > are
> > > > > >> > >> justifiable,
> > > > > >> > >> >> > but
> > > > > >> > >> >> > the savings here seem
> > minimal (a
> > > few bytes
> > > > per
> > > > > >> > message), so
> > > > > >> > >> maybe it's
> > > > > >> > >> >> > not
> > > > > >> > >> >> > worth the cost of letting
> > the
> > > message
> > > > diverge even
> > > > > >> > further from
> > > > > >> > >> the
> > > > > >> > >> >> > rest of
> > > > > >> > >> >> > the protocol.
> > > > > >> > >> >> >
> > > > > >> > >> >> > -Jason
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > On Wed, Feb 15, 2017 at
> > 8:52 AM,
> > > radai <
> > > > > >> > >> radai.rosenblatt@gmail.com>
> > > > > >> > >> >> > wrote:
> > > > > >> > >> >> >
> > > > > >> > >> >> > > I've trimmed the inline
> > > contents as this
> > > > mail is
> > > > > >> > getting too
> > > > > >> > >> big for
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > apache mailing list
> > software to
> > > deliver
> > > > :-(
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 1. the important thing
> for
> > > > interoperability is
> > > > > for
> > > > > >> > different
> > > > > >> > >> >> > "interested
> > > > > >> > >> >> > > parties" (plugins, infra
> > > layers/wrappers,
> > > > > >> user-code)
> > > > > >> > to be
> > > > > >> > >> able to
> > > > > >> > >> >> > stick
> > > > > >> > >> >> > > pieces of metadata onto
> > msgs
> > > without
> > > > getting in
> > > > > >> each
> > > > > >> > other's
> > > > > >> > >> way. a
> > > > > >> > >> >> > common
> > > > > >> > >> >> > > key scheme (Strings, as
> > of the
> > > time of
> > > > this
> > > > > >> > writing?) is all
> > > > > >> > >> thats
> > > > > >> > >> >> > required
> > > > > >> > >> >> > > for that. it is assumed
> > that
> > > the other end
> > > > > >> > interested in any
> > > > > >> > >> such
> > > > > >> > >> >> > piece of
> > > > > >> > >> >> > > metadata knows the
> > encoding,
> > > and byte[]
> > > > provides
> > > > > >> for
> > > > > >> > the most
> > > > > >> > >> >> > flexibility.
> > > > > >> > >> >> > > i believe this is the
> same
> > > logic behind
> > > > core
> > > > > kafka
> > > > > >> > being
> > > > > >> > >> >> > byte[]/byte[] -
> > > > > >> > >> >> > > Strings are more
> "usable"
> > but
> > > bytes are
> > > > flexible
> > > > > >> and
> > > > > >> > so were
> > > > > >> > >> chosen.
> > > > > >> > >> >> > > Also - core kafka doesnt
> > even
> > > do that
> > > > good of a
> > > > > >> job
> > > > > >> > on
> > > > > >> > >> usability of
> > > > > >> > >> >> > the
> > > > > >> > >> >> > > payload (example - i
> have
> > to
> > > specify the
> > > > nop
> > > > > >> byte[]
> > > > > >> > "decoders"
> > > > > >> > >> >> > explicitly
> > > > > >> > >> >> > > in conf), and again
> > sacrificies
> > > usability
> > > > for
> > > > > the
> > > > > >> > sake of
> > > > > >> > >> >> > performance (no
> > > > > >> > >> >> > > convenient single-record
> > > processing as
> > > > poll is a
> > > > > >> > batch, lots of
> > > > > >> > >> >> > obscure
> > > > > >> > >> >> > > little config details
> > exposing
> > > internals
> > > > of the
> > > > > >> > batching
> > > > > >> > >> mechanism,
> > > > > >> > >> >> > etc)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > this is also why i
> really
> > > dislike the
> > > > idea of a
> > > > > >> > "type system"
> > > > > >> > >> for
> > > > > >> > >> >> > header
> > > > > >> > >> >> > > values, it further
> > degrades the
> > > > usability, adds
> > > > > >> > complexity and
> > > > > >> > >> will
> > > > > >> > >> >> > > eventually get in
> > people's way,
> > > also, it
> > > > would
> > > > > be
> > > > > >> > the 2nd/3rd
> > > > > >> > >> >> > home-group
> > > > > >> > >> >> > > serialization mechanism
> > in core
> > > kafka
> > > > (counting
> > > > > 2
> > > > > >> > iterations
> > > > > >> > >> of the
> > > > > >> > >> >> > "type
> > > > > >> > >> >> > > definition DSL")
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 2. this is an
> > implementation
> > > detail, and
> > > > not
> > > > > even
> > > > > >> a
> > > > > >> > very "user
> > > > > >> > >> >> > facing" one?
> > > > > >> > >> >> > > to the best of my
> > understanding
> > > the vote
> > > > process
> > > > > >> is
> > > > > >> > on proposed
> > > > > >> > >> >> > > API/behaviour. also -
> > since
> > > we're willing
> > > > to go
> > > > > >> with
> > > > > >> > strings
> > > > > >> > >> just
> > > > > >> > >> >> > serialize
> > > > > >> > >> >> > > a 0-sized header blob
> and
> > IIUC
> > > you dont
> > > > need any
> > > > > >> > optionals
> > > > > >> > >> anymore.
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > 3. yes, we can :-)
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > On Tue, Feb 14, 2017 at
> > 11:56
> > > PM, Michael
> > > > > Pearce <
> > > > > >> > >> >> > Michael.Pearce@ig.com>
> > > > > >> > >> >> > > wrote:
> > > > > >> > >> >> > >
> > > > > >> > >> >> > > > Hi Jay,
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1) There was some
> > initial
> > > debate on the
> > > > value
> > > > > >> > part, as youll
> > > > > >> > >> note
> > > > > >> > >> >> > String,
> > > > > >> > >> >> > > > String headers were
> > > discounted early
> > > > on. The
> > > > > >> > reason for this
> > > > > >> > >> is
> > > > > >> > >> >> > > flexibility
> > > > > >> > >> >> > > > and keeping in line
> > with the
> > > > flexibility of
> > > > > key,
> > > > > >> > value of the
> > > > > >> > >> >> > message
> > > > > >> > >> >> > > > object itself. I don’t
> > think
> > > it takes
> > > > away
> > > > > from
> > > > > >> an
> > > > > >> > ecosystem
> > > > > >> > >> as
> > > > > >> > >> >> > each
> > > > > >> > >> >> > > plugin
> > > > > >> > >> >> > > > will care for their
> own
> > key,
> > > this way
> > > > ints,
> > > > > >> > booleans , exotic
> > > > > >> > >> >> > custom
> > > > > >> > >> >> > > binary
> > > > > >> > >> >> > > > can all be catered
> for=.
> > > > > >> > >> >> > > > a. If you really
> wanted
> > to
> > > push for a
> > > > typed
> > > > > >> value
> > > > > >> > interface,
> > > > > >> > >> I
> > > > > >> > >> >> > wouldn’t
> > > > > >> > >> >> > > > want just String
> values
> > > supported, but
> > > > the the
> > > > > >> > primatives
> > > > > >> > >> plus
> > > > > >> > >> >> > string and
> > > > > >> > >> >> > > > also still keeping the
> > > ability to have a
> > > > > binary
> > > > > >> > for custom
> > > > > >> > >> >> > binaries that
> > > > > >> > >> >> > > > some organisations may
> > have.
> > > > > >> > >> >> > > > i. I have written this
> > slight
> > > > alternative
> > > > > here,
> > > > > >> > >> >> > >
> https://cwiki.apache.org/
> > > > > >> > >> >> > > >
> > confluence/display/KAFKA/KIP-
> > > > > >> > 82+-+Add+Record+Headers+-+Typed
> > > > > >> > >> >> > > > ii. Essentially the
> > value
> > > bytes, has a
> > > > leading
> > > > > >> > byte overhead.
> > > > > >> > >> >> > > > 1. This tells you
> what
> > type
> > > the value
> > > > is,
> > > > > >> before
> > > > > >> > reading
> > > > > >> > >> the rest
> > > > > >> > >> >> > of the
> > > > > >> > >> >> > > > bytes, allowing
> > > > serialisation/deserialization
> > > > > to
> > > > > >> > and from the
> > > > > >> > >> >> > primitives,
> > > > > >> > >> >> > > > string and byte[].
> This
> > is
> > > akin to some
> > > > other
> > > > > >> > messaging
> > > > > >> > >> systems.
> > > > > >> > >> >> > > > 2) We are making it
> > optional,
> > > so that
> > > > for
> > > > > those
> > > > > >> > not wanting
> > > > > >> > >> >> > headers have
> > > > > >> > >> >> > > 0
> > > > > >> > >> >> > > > bytes overhead (think
> > of it
> > > as a feature
> > > > > flag),
> > > > > >> I
> > > > > >> > don’t
> > > > > >> > >> think this
> > > > > >> > >> >> > is
> > > > > >> > >> >> > > > complex, especially if
> > > comparing to
> > > > changes
> > > > > >> > proposed in
> > > > > >> > >> other kips
> > > > > >> > >> >> > like
> > > > > >> > >> >> > > > kip-98.
> > > > > >> > >> >> > > > a. If you really
> really
> > don’t
> > > like
> > > > this, we
> > > > > can
> > > > > >> > drop it, but
> > > > > >> > >> it
> > > > > >> > >> >> > would
> > > > > >> > >> >> > > mean
> > > > > >> > >> >> > > > buying into 4 bytes
> > extra
> > > overhead for
> > > > users
> > > > > who
> > > > > >> > do not want
> > > > > >> > >> to use
> > > > > >> > >> >> > > headers.
> > > > > >> > >> >> > > > 3) In the summary yes,
> > it is
> > > at a higher
> > > > > level,
> > > > > >> > but I think
> > > > > >> > >> this
> > > > > >> > >> >> > is well
> > > > > >> > >> >> > > > documented in the
> > proposed
> > > changes
> > > > section.
> > > > > >> > >> >> > > > a. Added getHeaders
> > method to
> > > > > Producer/Consumer
> > > > > >> > record (that
> > > > > >> > >> is it)
> > > > > >> > >> >> > > > b. We’ve also detailed
> > the
> > > new Headers
> > > > class
> > > > > >> that
> > > > > >> > this method
> > > > > >> > >> >> > returns
> > > > > >> > >> >> > > that
> > > > > >> > >> >> > > > encapsulates the
> headers
> > > protocol and
> > > > logic.
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Best,
> > > > > >> > >> >> > > > Mike
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > ==Original questions
> > from the
> > > vote
> > > > thread from
> > > > > >> > Jay.==
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > Couple of things I
> > think we
> > > still need
> > > > to work
> > > > > >> out:
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > 1. I think we agree
> > about
> > > the key,
> > > > but I
> > > > > >> think
> > > > > >> > we haven't
> > > > > >> > >> >> > talked about
> > > > > >> > >> >> > > > the value yet. I
> > think if
> > > our goal
> > > > is an
> > > > > open
> > > > > >> > ecosystem
> > > > > >> > >> of these
> > > > > >> > >> >> > > header
> > > > > >> > >> >> > > > spread across many
> > plugins
> > > from many
> > > > > systems
> > > > > >> we
> > > > > >> > should
> > > > > >> > >> consider
> > > > > >> > >> >> > making
> > > > > >> > >> >> > > > this
> > > > > >> > >> >> > > > a string as well so
> > it can
> > > be
> > > > printed, set
> > > > > >> via
> > > > > >> > a UI, set
> > > > > >> > >> in
> > > > > >> > >> >> > config,
> > > > > >> > >> >> > > etc.
> > > > > >> > >> >> > > > Basically
> encouraging
> > > pluggable
> > > > > serialization
> > > > > >> > formats
> > > > > >> > >> here will
> > > > > >> > >> >> > lead
> > > > > >> > >> >> > > to
> > > > > >> > >> >> > > > a
> > > > > >> > >> >> > > > bit of a tower of
> > babel.
> > > > > >> > >> >> > > > 2. This proposal
> > still
> > > includes a
> > > > pretty
> > > > > big
> > > > > >> > change to our
> > > > > >> > >> >> > > serialization
> > > > > >> > >> >> > > > and protocol
> > definition
> > > layer.
> > > > Essentially
> > > > > >> it is
> > > > > >> > >> introducing an
> > > > > >> > >> >> > > optional
> > > > > >> > >> >> > > > type, where the
> > format is
> > > data
> > > > dependent. I
> > > > > >> > think this is
> > > > > >> > >> >> > actually a
> > > > > >> > >> >> > > big
> > > > > >> > >> >> > > > change though it
> > doesn't
> > > seem like
> > > > it. It
> > > > > >> means
> > > > > >> > you can no
> > > > > >> > >> >> > longer
> > > > > >> > >> >> > > > specify
> > > > > >> > >> >> > > > this type with our
> > type
> > > definition
> > > > DSL, and
> > > > > >> > likewise it
> > > > > >> > >> requires
> > > > > >> > >> >> > > custom
> > > > > >> > >> >> > > > handling in client
> > libs.
> > > This isn't
> > > > a huge
> > > > > >> > thing, since
> > > > > >> > >> the
> > > > > >> > >> >> > Record
> > > > > >> > >> >> > > > definition is
> custom
> > > anyway, but I
> > > > think
> > > > > this
> > > > > >> > kind of
> > > > > >> > >> protocol
> > > > > >> > >> >> > > > inconsistency is
> very
> > > non-desirable
> > > > and
> > > > > ties
> > > > > >> > you to
> > > > > >> > >> hand-coding
> > > > > >> > >> >> > > things.
> > > > > >> > >> >> > > > I
> > > > > >> > >> >> > > > think the type
> should
> > > instead by [Key
> > > > > Value]
> > > > > >> in
> > > > > >> > our BNF,
> > > > > >> > >> where
> > > > > >> > >> >> > key and
> > > > > >> > >> >> > > > value are both
> short
> > > strings as used
> > > > > >> elsewhere.
> > > > > >> > This
> > > > > >> > >> brings it
> > > > > >> > >> >> > in line
> > > > > >> > >> >> > > > with
> > > > > >> > >> >> > > > the rest of the
> > protocol.
> > > > > >> > >> >> > > > 3. Could we get
> more
> > > specific about
> > > > the
> > > > > exact
> > > > > >> > Java API
> > > > > >> > >> change to
> > > > > >> > >> >> > > > ProducerRecord,
> > > ConsumerRecord,
> > > > Record,
> > > > > etc?
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > > > -Jay
> > > > > >> > >> >> > > >
> > > > > >> > >> >> > >
> > > > > >> > >> >> >
> > > > > >> > >> >> >
> > > > > >> > >> >> > The information contained in
> > this
> > > email is
> > > > strictly
> > > > > >> > confidential and
> > > > > >> > >> for
> > > > > >> > >> >> > the use of the addressee only,
> > unless
> > > otherwise
> > > > > >> indicated.
> > > > > >> > If you
> > > > > >> > >> are not
> > > > > >> > >> >> > the intended recipient, please
> > do not
> > > read,
> > > > copy, use
> > > > > or
> > > > > >> > disclose to
> > > > > >> > >> others
> > > > > >> > >> >> > this message or any
> attachment.
> > > Please also
> > > > notify the
> > > > > >> > sender by
> > > > > >> > >> replying
> > > > > >> > >> >> > to this email or by telephone
> > > (+44(020 7896
> > > > 0011) and
> > > > > >> then
> > > > > >> > delete
> > > > > >> > >> the email
> > > > > >> > >> >> > and any copies of it.
> Opinions,
> > > conclusion
> > > > (etc) that
> > > > > do
> > > > > >> > not relate
> > > > > >> > >> to the
> > > > > >> > >> >> > official business of this
> > company
> > > shall be
> > > > understood
> > > > > as
> > > > > >> > neither
> > > > > >> > >> given nor
> > > > > >> > >> >> > endorsed by it. IG is a
> trading
> > name
> > > of IG
> > > > Markets
> > > > > >> Limited
> > > > > >> > (a company
> > > > > >> > >> >> > registered in England and
> Wales,
> > > company number
> > > > > >> 04008957)
> > > > > >> > and IG
> > > > > >> > >> Index
> > > > > >> > >> >> > Limited (a company registered
> in
> > > England and
> > > > Wales,
> > > > > >> > company number
> > > > > >> > >> >> > 01190902). Registered address
> at
> > > Cannon Bridge
> > > > House,
> > > > > 25
> > > > > >> > Dowgate
> > > > > >> > >> Hill,
> > > > > >> > >> >> > London EC4R 2YA. Both IG
> Markets
> > > Limited
> > > > (register
> > > > > >> number
> > > > > >> > 195355)
> > > > > >> > >> and IG
> > > > > >> > >> >> > Index Limited (register number
> > > 114059) are
> > > > authorised
> > > > > >> and
> > > > > >> > regulated
> > > > > >> > >> by the
> > > > > >> > >> >> > Financial Conduct Authority.
> > > > > >> > >> >> >
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >> --
> > > > > >> > >> Nacho - Ignacio Solis -
> > isolis@igso.net
> > > > > >> > >>
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > The information contained in this email is
> > strictly
> > > > confidential and
> > > > > for
> > > > > >> > the use of the addressee only, unless
> otherwise
> > > indicated. If
> > > > you are
> > > > > >> not
> > > > > >> > the intended recipient, please do not read,
> > copy, use
> > > or
> > > > disclose to
> > > > > >> others
> > > > > >> > this message or any attachment. Please also
> > notify
> > > the sender by
> > > > > >> replying
> > > > > >> > to this email or by telephone (+44(020 7896
> > 0011) and
> > > then
> > > > delete the
> > > > > >> email
> > > > > >> > and any copies of it. Opinions, conclusion
> > (etc) that
> > > do not
> > > > relate to
> > > > > >> the
> > > > > >> > official business of this company shall be
> > understood
> > > as
> > > > neither given
> > > > > >> nor
> > > > > >> > endorsed by it. IG is a trading name of IG
> > Markets
> > > Limited (a
> > > > company
> > > > > >> > registered in England and Wales, company
> number
> > > 04008957) and
> > > > IG Index
> > > > > >> > Limited (a company registered in England and
> > Wales,
> > > company
> > > > number
> > > > > >> > 01190902). Registered address at Cannon Bridge
> > House,
> > > 25
> > > > Dowgate Hill,
> > > > > >> > London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and
> > > > > IG
> > > > > >> > Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > >> the
> > > > > >> > Financial Conduct Authority.
> > > > > >> >
> > > > > >> The information contained in this email is
> > strictly
> > > confidential
> > > > and for
> > > > > >> the use of the addressee only, unless otherwise
> > > indicated. If you
> > > > are
> > > > > not
> > > > > >> the intended recipient, please do not read,
> copy,
> > use
> > > or disclose
> > > > to
> > > > > others
> > > > > >> this message or any attachment. Please also
> > notify the
> > > sender by
> > > > > replying
> > > > > >> to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the
> > > > > email
> > > > > >> and any copies of it. Opinions, conclusion (etc)
> > that
> > > do not
> > > > relate to
> > > > > the
> > > > > >> official business of this company shall be
> > understood
> > > as neither
> > > > given
> > > > > nor
> > > > > >> endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a
> > > > company
> > > > > >> registered in England and Wales, company number
> > > 04008957) and IG
> > > > Index
> > > > > >> Limited (a company registered in England and
> > Wales,
> > > company number
> > > > > >> 01190902). Registered address at Cannon Bridge
> > House,
> > > 25 Dowgate
> > > > Hill,
> > > > > >> London EC4R 2YA. Both IG Markets Limited
> (register
> > > number 195355)
> > > > and IG
> > > > > >> Index Limited (register number 114059) are
> > authorised
> > > and
> > > > regulated by
> > > > > the
> > > > > >> Financial Conduct Authority.
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > The information contained in this email is strictly
> > > confidential and
> > > > for the use of the addressee only, unless otherwise
> > indicated.
> > > If you are
> > > > not the intended recipient, please do not read, copy, use
> > or
> > > disclose to
> > > > others this message or any attachment. Please also notify
> > the
> > > sender by
> > > > replying to this email or by telephone (+44(020 7896
> 0011)
> > and
> > > then delete
> > > > the email and any copies of it. Opinions, conclusion
> (etc)
> > that
> > > do not
> > > > relate to the official business of this company shall be
> > > understood as
> > > > neither given nor endorsed by it. IG is a trading name of
> > IG
> > > Markets
> > > > Limited (a company registered in England and Wales,
> company
> > > number
> > > > 04008957) and IG Index Limited (a company registered in
> > England
> > > and Wales,
> > > > company number 01190902). Registered address at Cannon
> > Bridge
> > > House, 25
> > > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> > (register
> > > number
> > > > 195355) and IG Index Limited (register number 114059) are
> > > authorised and
> > > > regulated by the Financial Conduct Authority.
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > > The information contained in this email is strictly
> confidential
> > and
> > > for the use of the addressee only, unless otherwise indicated. If
> > you are
> > > not the intended recipient, please do not read, copy, use or
> > disclose to
> > > others this message or any attachment. Please also notify the
> sender
> > by
> > > replying to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > > the email and any copies of it. Opinions, conclusion (etc) that do
> > not
> > > relate to the official business of this company shall be understood
> > as
> > > neither given nor endorsed by it. IG is a trading name of IG
> Markets
> > > Limited (a company registered in England and Wales, company number
> > > 04008957) and IG Index Limited (a company registered in England and
> > Wales,
> > > company number 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> > number
> > > 195355) and IG Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
headers dont "leak" into application code. they are useful to application
code as well.
IIUC samze currently has headers "in-V" and would just switch over to kafka
headers if they exist.
im sure plenty of other users of kafka would have a use for headers.
im pretty sure use cases exist around shuffling data into/out-of kafka
(kafka connect or equivalent) where metadata from one end could copied over
to the other (S3, for example uses http headers for user-accessible
metadata). it will be kafka client code getting/setting those headers. not
an interceptor.
On Fri, Feb 17, 2017 at 1:41 PM, Michael Pearce <Mi...@ig.com>
wrote:
> For APM single event tracing, need access to the header at the point of
> processing on the processing thread.
>
> As such interceptors will not work/be suitable for these, due to the fact
> they act on the ConsumerRecords as a batch, before the handling thread can
> split out and process per message which is the point these tools will need
> to continue to transaction tracing.
>
> Like wise tools and infra pieces will need access to the message outside
> the interceptor realm.
>
>
>
> On 17/02/2017, 21:26, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> >
> > That’s exactly what we’re doing the headers are a slice of bytes,
> which
> > then gets parsed later if needed, or can be parsed right away, the
> headers
> > is part of the protocol, so can still be validated if wanted.
> > If you had a header count then you would have to go through each
> header
> > key and value length value to work out how much to skip to get to
> say the
> > value or any future component in the message after the headers.
> Having it
> > as a byte[] with length value makes this a lot easier to skip.
>
>
> So the broker will parse the headers and validate them. Good. The only
> reason remaining that I can see to leave the headers as a byte array
> is to
> make it easier for the client to skip past them. Are we sure this is
> not
> premature optimization? Are there any performance results which show
> that
> this is worthwhile?
>
> What’s the issue with exposing a method getHeaders on the
> producer/consumer
> > record? It doesn’t break anything. We don’t need any special version.
>
>
> See my previous explanation. What I am trying to resist is the headers
> becoming a general application-level facility. The primary use case as
> far
> as I can tell is middleware, which is the use case that the
> interceptors
> are providing.
>
> Current batch consumer model and consumer interceptors don’t work where
> > headers need to be acted on at per message level at time of
> processing,
> > very case is APM (the core one), where the header value is used to
> continue
> > tracing.
>
>
> I still don't understand the point about batching. The consumer
> records are
> exposed as a batch in the consumer interceptor, but you can still
> iterate
> them individually. It is no different for the consumer API itself.
>
> -Jason
>
> On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > Re:
> >
> > “ The point about creation of maps seems orthogonal. We can still
> > represent
> > the headers as a slice of bytes until the time it is accessed.”
> >
> > That’s exactly what we’re doing the headers are a slice of bytes,
> which
> > then gets parsed later if needed, or can be parsed right away, the
> headers
> > is part of the protocol, so can still be validated if wanted.
> >
> > If you had a header count then you would have to go through each
> header
> > key and value length value to work out how much to skip to get to
> say the
> > value or any future component in the message after the headers.
> Having it
> > as a byte[] with length value makes this a lot easier to skip.
> >
> >
> > On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > What’s the issue with exposing a method getHeaders on the
> > producer/consumer record? It doesn’t break anything. We don’t need
> any
> > special version.
> >
> > Current batch consumer model and consumer interceptors don’t work
> > where headers need to be acted on at per message level at time of
> > processing, very case is APM (the core one), where the header value
> is used
> > to continue tracing. JMS/HTTP etc all expose these, without issues.
> I would
> > NOT want to lock this down to only be usable accessible via
> interceptors,
> > as we’d fail on one of the main goals.
> >
> > Regards
> > Mike
> >
> >
> >
> >
> > On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >
> > The point about creation of maps seems orthogonal. We can
> still
> > represent
> > the headers as a slice of bytes until the time it is
> accessed.
> >
> >
> > > Yes exactly we have access to the records thus why the
> header
> > should be
> > > accessible via it and not hidden for only interceptors to
> access.
> >
> >
> > As explained above, the point is to make the intended usage
> clear.
> > Applications should continue to rely on the key/value fields
> to
> > serialize
> > their own headers, and it would be more ideal if we can avoid
> > leaking
> > third-party headers into applications. This is difficult to
> do
> > with the
> > current interceptors because they share the record objects
> with
> > the common
> > API. What I had in mind is something like an extension of the
> > current
> > interceptors which exposed a different object (e.g.
> > `RecordAndHeaders`).
> > The challenge is for MM-like use cases. Let me see if I can
> come
> > up with a
> > concrete proposal for that problem.
> >
> > -Jason
> >
> >
> >
> > On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > wrote:
> >
> > > I am happy to move the definition of the header into the
> message
> > body, but
> > > would cause us not to lazy initialise/parse the headers, as
> > obviously, we
> > > would have to traverse these reading the message.
> > >
> > > This was actually one of Jay’s requests:
> > >
> > > “ 2. I think we should think about creating the Map
> lazily to
> > avoid
> > > parsing out all the headers into little objects.
> HashMaps
> > themselves
> > > are
> > > kind of expensive and the consumer is very perf
> sensitive so
> > and making
> > > gazillions of hashmaps that may or may not get used is
> > probably a bad
> > > idea.”
> > >
> > >
> > >
> > >
> > >
> > > On 17/02/2017, 19:44, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > wrote:
> > >
> > > Yes exactly we have access to the records thus why the
> > header should
> > > be accessible via it and not hidden for only interceptors
> to
> > access.
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Magnus Edenhill <ma...@edenhill.se>
> > > Sent: Friday, February 17, 2017 7:34:49 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >
> > > Big +1 on VarInts.
> > > CPUs are fast, memory is slow.
> > >
> > > I agree with Jason that we'll want to continue
> verifying
> > messages,
> > > including their headers, so while I appreciate the
> idea of
> > the opaque
> > > header blob it won't be useful in practice.
> > >
> > > /Magnus
> > >
> > > 2017-02-17 10:41 GMT-08:00 Jason Gustafson <
> > jason@confluent.io>:
> > >
> > > > Sorry, my mistake. The consumer interceptor is per
> batch,
> > though I'm
> > > not
> > > > sure that's an actual limitation since you still have
> > access to the
> > > > individual records.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> > > jason@confluent.io>
> > > > wrote:
> > > >
> > > > > Re headers as byte array and future use by broker.
> This
> > doesn't
> > > take away
> > > > >> from that at all. Nor makes it difficult at all
> in my
> > opinion.
> > > > >
> > > > >
> > > > > Yeah, I didn't say it was difficult, only awkward.
> You
> > wouldn't
> > > write the
> > > > > schema that way if you were planning to use it on
> the
> > brokers from
> > > the
> > > > > beginning. Note also that one of the benefits of
> letting
> > the broker
> > > > > understand headers is that it can validate that
> they are
> > properly
> > > > > formatted. If cost is the only concern, we should
> > confirm its
> > > impact
> > > > > through performance testing.
> > > > >
> > > > > One of the key use cases requires access on
> consume at
> > per
> > > event/message
> > > > >> level at the point that message is being
> processed, as
> > such the
> > > batch
> > > > >> interceptors and batch consume api isn't
> suitable. It
> > needs to be
> > > at the
> > > > >> record level.
> > > > >
> > > > >
> > > > > I'm not sure I understand the point about batching.
> > Interceptors
> > > are
> > > > > applied per-message, right?
> > > > >
> > > > > My intent on interceptors is to keep the usage of
> headers
> > > well-defined so
> > > > > that they don't start leaking unnecessarily into
> > applications. My
> > > guess
> > > > is
> > > > > that it's probably inevitable, but isolating it in
> the
> > > interceptors would
> > > > > at least give people a second thought before
> deciding to
> > use it.
> > > The main
> > > > > challenge in my mind is figuring out how an MM use
> case
> > would
> > > work. It
> > > > > would be more cumbersome to replicate headers
> through an
> > > interceptor,
> > > > > though arguably MM should be working at a lower
> level
> > anyway.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > > wrote:
> > > > >
> > > > >> Re headers available on the record va
> interceptors only
> > > > >>
> > > > >> One of the key use cases requires access on
> consume at
> > per
> > > event/message
> > > > >> level at the point that message is being
> processed, as
> > such the
> > > batch
> > > > >> interceptors and batch consume api isn't
> suitable. It
> > needs to be
> > > at the
> > > > >> record level.
> > > > >>
> > > > >> This anyhow is similar to jms/http/amqp where
> headers
> > are
> > > available to
> > > > >> consuming applications.
> > > > >>
> > > > >> Re headers as byte array and future use by
> broker. This
> > doesn't
> > > take
> > > > away
> > > > >> from that at all. Nor makes it difficult at all
> in my
> > opinion.
> > > > >>
> > > > >>
> > > > >>
> > > > >> Sent using OWA for iPhone
> > > > >> ________________________________________
> > > > >> From: Jason Gustafson <ja...@confluent.io>
> > > > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > > > >> To: dev@kafka.apache.org
> > > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > > >>
> > > > >> >
> > > > >> > Would you be proposing in KIP-98 to convert the
> other
> > message
> > > int’s
> > > > (key
> > > > >> > length, value length) also to varint to keep it
> > uniform.
> > > > >> > Also I assume there will be a static or helper
> method
> > made to
> > > > write/read
> > > > >> > these in the client and server.
> > > > >>
> > > > >>
> > > > >> Yes, that is what we are proposing, so using
> varints
> > for headers
> > > would
> > > > be
> > > > >> consistent with the rest of the message. We have
> used
> > static
> > > helper
> > > > >> methods
> > > > >> in our prototype implementation.
> > > > >>
> > > > >> The cost of parsing, we want to parse/interpret
> the
> > headers
> > > lazily (this
> > > > >> is
> > > > >> > a key point brought up earlier in discussions)
> > > > >>
> > > > >>
> > > > >> I'm a bit skeptical of this. Has anyone done the
> > performance
> > > testing? I
> > > > >> can
> > > > >> probably implement it and test it if no one else
> has. I
> > was also
> > > under
> > > > the
> > > > >> impression that there may be use cases down the
> road
> > where the
> > > broker
> > > > >> would
> > > > >> need to interpret headers. That wouldn't be off
> the
> > table in the
> > > future
> > > > if
> > > > >> it's represented as bytes, but it would be quite
> a bit
> > more
> > > awkward,
> > > > >> right?
> > > > >>
> > > > >> By the way, one question I have been wondering
> about. My
> > > understanding
> > > > is
> > > > >> that headers are primarily for use cases where a
> > third-party
> > > components
> > > > >> wants to enrich messages without needing to
> understand
> > or modify
> > > the
> > > > >> schema
> > > > >> of the message key and value. For the applications
> > which directly
> > > > produce
> > > > >> and consume the messages and control the key/value
> > schema
> > > directly, it
> > > > >> seems we would rather have them implement headers
> > directly in
> > > their own
> > > > >> schema. Supposing for the sake of argument that
> it was
> > possible,
> > > my
> > > > >> question is whether it be sufficient to expose the
> > headers in the
> > > > >> interceptor API and not in the common API?
> > > > >>
> > > > >> -Jason
> > > > >>
> > > > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > >> wrote:
> > > > >>
> > > > >> > On the point of varInts
> > > > >> >
> > > > >> > Would you be proposing in KIP-98 to convert the
> other
> > message
> > > int’s
> > > > (key
> > > > >> > length, value length) also to varint to keep it
> > uniform.
> > > > >> > Also I assume there will be a static or helper
> method
> > made to
> > > > write/read
> > > > >> > these in the client and server.
> > > > >> >
> > > > >> > Cheers
> > > > >> > Mike
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On 17/02/2017, 11:22, "Michael Pearce" <
> > Michael.Pearce@ig.com>
> > > wrote:
> > > > >> >
> > > > >> > On the point re: headers in the message
> protocol
> > being a
> > > byte
> > > > array
> > > > >> > and not a count of elements followed by the
> elements.
> > Again
> > > this was
> > > > >> > discussed/argued previously.
> > > > >> >
> > > > >> > It was agreed on for a few reasons some of
> which
> > you have
> > > > obviously
> > > > >> > picked up on:
> > > > >> >
> > > > >> > Broker is able to pass it through opaquely
> > > > >> > The cost of parsing, we want to
> parse/interpret
> > the headers
> > > lazily
> > > > >> > (this is a key point brought up earlier in
> > discussions)
> > > > >> > Headers can be copied from consumer record
> to
> > producer
> > > record (aka
> > > > >> > mirror makers etc) without parsing if no
> changes are
> > being made
> > > or
> > > > being
> > > > >> > looked at.
> > > > >> > Keeps the broker agnostic to the format
> > > > >> > You need an int32 either for the byte size
> of the
> > headers,
> > > or for
> > > > >> the
> > > > >> > count of elements, so overheads are the same,
> but
> > with going
> > > with an
> > > > >> opaque
> > > > >> > byte array has the above advantages.
> > > > >> >
> > > > >> > Cheers
> > > > >> > Mike
> > > > >> >
> > > > >> >
> > > > >> > On 17/02/2017, 02:50, "Jason Gustafson" <
> > jason@confluent.io
> > > >
> > > > wrote:
> > > > >> >
> > > > >> > Sorry, should have noted that the
> performance
> > testing
> > > was done
> > > > >> > using the
> > > > >> > producer performance tool shipped with
> Kafka.
> > > > >> >
> > > > >> > -Jason
> > > > >> >
> > > > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason
> > Gustafson <
> > > > >> > jason@confluent.io> wrote:
> > > > >> >
> > > > >> > > Hey Nacho,
> > > > >> > >
> > > > >> > > I've compared performance of our
> KIP-98
> > > implementation with
> > > > >> and
> > > > >> > without
> > > > >> > > varints. For messages around 128
> bytes, we
> > see an
> > > increase
> > > > in
> > > > >> > throughput of
> > > > >> > > about 30% using the default
> configuration
> > settings.
> > > At 256
> > > > >> > bytes, the
> > > > >> > > increase is around 16%. Obviously the
> > performance
> > > converge
> > > > as
> > > > >> > messages get
> > > > >> > > larger, but it seems well worth the
> cost.
> > Note that
> > > we are
> > > > >> also
> > > > >> > seeing a
> > > > >> > > substantial performance increase
> against
> > trunk
> > > primarily
> > > > >> because
> > > > >> > of the
> > > > >> > > much more efficient packing that
> varints
> > provide us.
> > > > Anything
> > > > >> > adding to
> > > > >> > > message overhead, such as record
> headers,
> > would only
> > > > increase
> > > > >> > the relative
> > > > >> > > difference. (Of course take these
> numbers
> > with a
> > > grain of
> > > > salt
> > > > >> > since I have
> > > > >> > > only used the default settings with
> both
> > the producer
> > > and
> > > > >> broker
> > > > >> > on my
> > > > >> > > local machine. We intend to provide
> more
> > extensive
> > > > performance
> > > > >> > details as
> > > > >> > > part of the work for KIP-98.)
> > > > >> > >
> > > > >> > > The implementation we are using is
> from
> > protobuf (
> > > > >> > > https://developers.google.com/
> > > > protocol-buffers/docs/encoding
> > > > >> ),
> > > > >> > which is
> > > > >> > > also used in HBase. It is trivial to
> > implement and as
> > > far
> > > > as I
> > > > >> > know doesn't
> > > > >> > > suffer from the aliasing problem you
> are
> > describing. I
> > > > checked
> > > > >> > with Magnus
> > > > >> > > (the author of librdkafka) and he
> agreed
> > that the
> > > savings
> > > > >> seemed
> > > > >> > worth the
> > > > >> > > cost of implementation.
> > > > >> > >
> > > > >> > > -Jason
> > > > >> > >
> > > > >> > > On Thu, Feb 16, 2017 at 4:32 PM,
> Ignacio
> > Solis <
> > > > >> isolis@igso.net>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > >> -VarInts
> > > > >> > >>
> > > > >> > >> I'm one of the people (if not the
> most)
> > opposed to
> > > VarInts.
> > > > >> > VarInts
> > > > >> > >> have a place, but this is not it.
> (We
> > had a large
> > > > >> discussion
> > > > >> > about
> > > > >> > >> them at the beginning of KIP-82 time)
> > > > >> > >>
> > > > >> > >> If anybody has real life performance
> > numbers of
> > > VarInts
> > > > >> > improving
> > > > >> > >> things or significantly reducing
> resources
> > I would
> > > like to
> > > > >> know
> > > > >> > what
> > > > >> > >> that case may be. Yes, you can save
> some
> > bytes here
> > > and
> > > > >> there,
> > > > >> > but
> > > > >> > >> this is probably insignificant to the
> > overall system
> > > > behavior
> > > > >> > and
> > > > >> > >> storage requirements. -- I say this
> with
> > respect to
> > > using
> > > > >> > VarInts in
> > > > >> > >> the protocol itself, not as part of
> the
> > data.
> > > > >> > >>
> > > > >> > >> VarInts require you to parse the Int
> > before using it
> > > and
> > > > >> > depending on
> > > > >> > >> the encoding they can suffer from
> aliasing
> > (multiple
> > > > >> > representations
> > > > >> > >> for the same value).
> > > > >> > >>
> > > > >> > >> Why add complexity?
> > > > >> > >>
> > > > >> > >> Nacho
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> On Thu, Feb 16, 2017 at 10:29 AM,
> Colin
> > McCabe <
> > > > >> > cmccabe@apache.org>
> > > > >> > >> wrote:
> > > > >> > >> > +1 for varints here-- it would save
> > quite a bit of
> > > space.
> > > > >> > They are
> > > > >> > >> > pretty quick to implement as well.
> > > > >> > >> >
> > > > >> > >> > I think it makes sense for values
> to be
> > byte
> > > arrays.
> > > > Users
> > > > >> > might want
> > > > >> > >> > to attach arbitrary payloads; they
> > shouldn't be
> > > forced to
> > > > >> > serialize
> > > > >> > >> > everything to Java strings.
> > > > >> > >> >
> > > > >> > >> > best,
> > > > >> > >> > Colin
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Thu, Feb 16, 2017, at 09:52,
> Jason
> > Gustafson
> > > wrote:
> > > > >> > >> >> Hey Michael,
> > > > >> > >> >>
> > > > >> > >> >> Hmm, I guess the point of
> representing
> > it as
> > > bytes is to
> > > > >> > allow the
> > > > >> > >> broker
> > > > >> > >> >> to pass it through opaquely? Is
> the
> > cost of
> > > parsing
> > > > them a
> > > > >> > concern, or
> > > > >> > >> >> are
> > > > >> > >> >> we simply trying to ensure that
> the
> > broker stays
> > > > agnostic
> > > > >> to
> > > > >> > the
> > > > >> > >> format?
> > > > >> > >> >>
> > > > >> > >> >> On varints, I think adding
> support for
> > them makes
> > > less
> > > > >> sense
> > > > >> > for an
> > > > >> > >> >> isolated use case, but as part of
> a
> > more holistic
> > > change
> > > > >> > (such as what
> > > > >> > >> we
> > > > >> > >> >> have proposed in KIP-98), I think
> they
> > are
> > > justifiable.
> > > > If
> > > > >> > we add them,
> > > > >> > >> >> then the need to use attributes
> becomes
> > quite a
> > > bit
> > > > >> weaker,
> > > > >> > right? The
> > > > >> > >> >> other thing I find slightly odd
> is the
> > fact that
> > > null
> > > > >> > headers has no
> > > > >> > >> >> actual
> > > > >> > >> >> semantic meaning for the message
> > (unlike null
> > > keys and
> > > > >> > values). It is
> > > > >> > >> >> just
> > > > >> > >> >> a space optimization. It seems a
> bit
> > better to
> > > always
> > > > use
> > > > >> > size 0 to
> > > > >> > >> >> indicate having no headers.
> > > > >> > >> >>
> > > > >> > >> >> Overall, the main point is
> ensuring
> > that the
> > > message
> > > > >> schema
> > > > >> > remains
> > > > >> > >> >> consistent, either within the
> larger
> > protocol, or
> > > at a
> > > > >> > minimum within
> > > > >> > >> the
> > > > >> > >> >> message itself.
> > > > >> > >> >>
> > > > >> > >> >> -Jason
> > > > >> > >> >>
> > > > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM,
> > Michael Pearce <
> > > > >> > Michael.Pearce@ig.com
> > > > >> > >> >
> > > > >> > >> >> wrote:
> > > > >> > >> >>
> > > > >> > >> >> > Hi Jason,
> > > > >> > >> >> >
> > > > >> > >> >> > On point 1) in the message
> protocol
> > the headers
> > > are
> > > > >> simply
> > > > >> > a byte
> > > > >> > >> array,
> > > > >> > >> >> > as like the key or value, this
> is to
> > clearly
> > > demarcate
> > > > >> the
> > > > >> > header in
> > > > >> > >> the
> > > > >> > >> >> > core message. Then the header
> byte
> > array in the
> > > core
> > > > >> > message is an
> > > > >> > >> array of
> > > > >> > >> >> > key, value pairs. This is what
> it is
> > denoting.
> > > > >> > >> >> >
> > > > >> > >> >> > Then this would be I guess in
> the
> > given
> > > notation:
> > > > >> > >> >> >
> > > > >> > >> >> > Headers => [KeyLength, Key,
> > ValueLength, Value]
> > > > >> > >> >> > KeyLength => int32
> > <-----------------NEW
> > > size of
> > > > the
> > > > >> > byte[] of
> > > > >> > >> the
> > > > >> > >> >> > serialised key value
> > > > >> > >> >> > Key => bytes
> > <---------------------- NEW
> > > > serialised
> > > > >> > string (UTF8)
> > > > >> > >> >> > bytes of the header key
> > > > >> > >> >> > ValueLength => int32
> > <-------------- NEW
> > > size of
> > > > the
> > > > >> > byte[] of
> > > > >> > >> the
> > > > >> > >> >> > serialised header value
> > > > >> > >> >> > Value => bytes
> > <-------------------- NEW
> > > > serialised
> > > > >> > form of the
> > > > >> > >> header
> > > > >> > >> >> > value
> > > > >> > >> >> >
> > > > >> > >> >> > The key length and value length
> is
> > matching the
> > > way
> > > > the
> > > > >> > protocol is
> > > > >> > >> >> > defined in the core message
> currently.
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> > On point 2)
> > > > >> > >> >> > Var sized ints, this was
> discussed
> > much earlier
> > > on, in
> > > > >> > fact I had
> > > > >> > >> >> > suggested it myself (with Hadoop
> > references),
> > > the
> > > > >> > complexity of this
> > > > >> > >> >> > compared to having a simpler
> protocol
> > was
> > > argued and
> > > > >> > agreed it
> > > > >> > >> wasn’t worth
> > > > >> > >> >> > the complexity as all other
> clients
> > in other
> > > languages
> > > > >> > would need to
> > > > >> > >> ensure
> > > > >> > >> >> > theyre using the right var size
> > algorithm, as
> > > there
> > > > is a
> > > > >> > few.
> > > > >> > >> >> >
> > > > >> > >> >> > On point 3)
> > > > >> > >> >> > We did the attributes, optional
> > approach as
> > > originally
> > > > >> > there was
> > > > >> > >> marked
> > > > >> > >> >> > concern that headers would
> cause a
> > message size
> > > > overhead
> > > > >> > for others,
> > > > >> > >> who
> > > > >> > >> >> > don’t want them. As such this
> is the
> > clean
> > > solution to
> > > > >> > achieve that.
> > > > >> > >> If
> > > > >> > >> >> > that no longer holds, and we
> don’t
> > care that we
> > > add
> > > > >> 4bytes
> > > > >> > overhead,
> > > > >> > >> then
> > > > >> > >> >> > im happy to remove.
> > > > >> > >> >> >
> > > > >> > >> >> > I’m personally in favour of
> keeping
> > the message
> > > as
> > > > small
> > > > >> > as possible
> > > > >> > >> so
> > > > >> > >> >> > people don’t get shocks in perf
> and
> > throughputs
> > > dues
> > > > to
> > > > >> > message size,
> > > > >> > >> >> > unless they actively use the
> feature,
> > as such I
> > > do
> > > > >> prefer
> > > > >> > the
> > > > >> > >> attribute bit
> > > > >> > >> >> > wise feature flag approach
> myself.
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> > On 16/02/2017, 05:40, "Jason
> > Gustafson" <
> > > > >> > jason@confluent.io> wrote:
> > > > >> > >> >> >
> > > > >> > >> >> > We have proposed a few
> > significant changes
> > > to the
> > > > >> > message format
> > > > >> > >> in
> > > > >> > >> >> > KIP-98
> > > > >> > >> >> > which now seems likely to
> pass
> > (perhaps
> > > with some
> > > > >> > iterations on
> > > > >> > >> >> > implementation details). It
> would
> > be good
> > > to try
> > > > and
> > > > >> > coordinate
> > > > >> > >> the
> > > > >> > >> >> > changes
> > > > >> > >> >> > in both of the proposals to
> make
> > sure they
> > > are
> > > > >> > consistent and
> > > > >> > >> >> > compatible.
> > > > >> > >> >> >
> > > > >> > >> >> > I think using the
> attributes to
> > indicate
> > > null
> > > > >> headers
> > > > >> > is a
> > > > >> > >> reasonable
> > > > >> > >> >> > approach. We have proposed
> to do
> > the same
> > > thing
> > > > for
> > > > >> > the message
> > > > >> > >> key and
> > > > >> > >> >> > value. That said, I
> sympathize
> > with Jay's
> > > > argument.
> > > > >> > Having
> > > > >> > >> multiple
> > > > >> > >> >> > ways to
> > > > >> > >> >> > specify a null value
> increases
> > the overall
> > > > >> complexity
> > > > >> > of the
> > > > >> > >> protocol.
> > > > >> > >> >> > You
> > > > >> > >> >> > can see this just from the
> fact
> > that you
> > > need the
> > > > >> > extra verbiage
> > > > >> > >> in the
> > > > >> > >> >> > protocol specification in
> this
> > KIP and in
> > > KIP-98
> > > > to
> > > > >> > describe the
> > > > >> > >> >> > dependence
> > > > >> > >> >> > between the fields and the
> > attributes. It
> > > seems
> > > > >> like a
> > > > >> > slippery
> > > > >> > >> slope
> > > > >> > >> >> > if
> > > > >> > >> >> > you start allowing different
> > request types
> > > to
> > > > >> > implement the
> > > > >> > >> protocol
> > > > >> > >> >> > specification differently.
> > > > >> > >> >> >
> > > > >> > >> >> > You can also argue that the
> > messages
> > > already are
> > > > and
> > > > >> > are likely
> > > > >> > >> to
> > > > >> > >> >> > remain a
> > > > >> > >> >> > special case. For example,
> there
> > is
> > > currently no
> > > > >> > generality in
> > > > >> > >> how
> > > > >> > >> >> > compressed message sets are
> > represented
> > > that would
> > > > >> be
> > > > >> > applicable
> > > > >> > >> for
> > > > >> > >> >> > other
> > > > >> > >> >> > request types. Some might
> see this
> > > divergence as
> > > > an
> > > > >> > unfortunate
> > > > >> > >> >> > protocol
> > > > >> > >> >> > deficiency which should be
> fixed;
> > others
> > > might see
> > > > >> it
> > > > >> > as sort of
> > > > >> > >> the
> > > > >> > >> >> > inevitability of needing to
> > optimize where
> > > it
> > > > counts
> > > > >> > most. I'm
> > > > >> > >> probably
> > > > >> > >> >> > somewhere in between, but I
> think
> > we
> > > probably all
> > > > >> > share the
> > > > >> > >> intuition
> > > > >> > >> >> > that
> > > > >> > >> >> > the protocol should be kept
> as
> > consistent as
> > > > >> possible.
> > > > >> > With that
> > > > >> > >> in
> > > > >> > >> >> > mind,
> > > > >> > >> >> > here are a few comments:
> > > > >> > >> >> >
> > > > >> > >> >> > 1. One thing I found a
> little odd
> > when
> > > reading the
> > > > >> > current
> > > > >> > >> proposal is
> > > > >> > >> >> > that
> > > > >> > >> >> > the headers are both
> represented
> > as an
> > > array of
> > > > >> bytes
> > > > >> > and as an
> > > > >> > >> array
> > > > >> > >> >> > of
> > > > >> > >> >> > key/value pairs. I'd
> probably
> > suggest
> > > something
> > > > like
> > > > >> > this:
> > > > >> > >> >> >
> > > > >> > >> >> > Headers => [HeaderKey
> HeaderValue]
> > > > >> > >> >> > HeaderKey => String
> > > > >> > >> >> > HeaderValue => Bytes
> > > > >> > >> >> >
> > > > >> > >> >> > An array in the Kafka
> protocol is
> > > represented as a
> > > > >> > 4-byte integer
> > > > >> > >> >> > indicating the number of
> elements
> > in the
> > > array
> > > > >> > followed by the
> > > > >> > >> >> > serialization of the
> elements.
> > Unless I'm
> > > > >> > misunderstanding, what
> > > > >> > >> you
> > > > >> > >> >> > have
> > > > >> > >> >> > instead is the total size
> of the
> > headers in
> > > bytes
> > > > >> > followed by the
> > > > >> > >> >> > elements.
> > > > >> > >> >> > I'm not sure I see any
> reason for
> > this
> > > > >> inconsistency.
> > > > >> > >> >> >
> > > > >> > >> >> > 2. In KIP-98, we've
> introduced
> > > variable-length
> > > > >> integer
> > > > >> > fields.
> > > > >> > >> >> > Effectively,
> > > > >> > >> >> > we've enriched (or
> "complicated"
> > as Jay
> > > might say
> > > > ;)
> > > > >> > the protocol
> > > > >> > >> >> > specification to include the
> > following
> > > types:
> > > > >> VarInt,
> > > > >> > VarLong,
> > > > >> > >> >> > UnsignedVarInt and
> > UnsignedVarLong.
> > > > >> > >> >> >
> > > > >> > >> >> > Along with these
> primitives, we
> > could
> > > introduce
> > > > the
> > > > >> > following
> > > > >> > >> types:
> > > > >> > >> >> >
> > > > >> > >> >> > VarSizeArray =>
> NumberOfItems
> > Item1 Item2
> > > .. ItemN
> > > > >> > >> >> > NumberOfItems =>
> UnsignedVarInt
> > > > >> > >> >> >
> > > > >> > >> >> > VarSizeNullableArray =>
> > NumberOfItemsOrNull
> > > Item1
> > > > >> > Item2 .. ItemN
> > > > >> > >> >> > NumberOfItemsOrNull =>
> VarInt
> > (-1 means
> > > null)
> > > > >> > >> >> >
> > > > >> > >> >> > And similarly for the
> `String`
> > and `Bytes`
> > > types.
> > > > >> > These types
> > > > >> > >> can save
> > > > >> > >> >> > a
> > > > >> > >> >> > considerable amount of
> space in
> > this
> > > proposal
> > > > >> because
> > > > >> > they can
> > > > >> > >> be used
> > > > >> > >> >> > for
> > > > >> > >> >> > both the number of headers
> > included in the
> > > message
> > > > >> and
> > > > >> > the
> > > > >> > >> lengths of
> > > > >> > >> >> > the
> > > > >> > >> >> > header keys and values. We
> could
> > do this
> > > instead:
> > > > >> > >> >> >
> > > > >> > >> >> > Headers =>
> VarSizeArray[HeaderKey
> > > HeaderValue]
> > > > >> > >> >> > HeaderKey => VarSizeString
> > > > >> > >> >> > HeaderValue =>
> VarSizeBytes
> > > > >> > >> >> >
> > > > >> > >> >> > Combining the savings from
> the
> > use of
> > > variable
> > > > >> length
> > > > >> > fields, the
> > > > >> > >> >> > benefit
> > > > >> > >> >> > of using the attributes to
> > represent null
> > > seems
> > > > >> pretty
> > > > >> > small.
> > > > >> > >> >> >
> > > > >> > >> >> > 3. Whichever way we go
> (whether
> > we use the
> > > > >> attributes
> > > > >> > or not), we
> > > > >> > >> >> > should at
> > > > >> > >> >> > least be consistent between
> this
> > KIP and
> > > KIP-98.
> > > > It
> > > > >> > would be very
> > > > >> > >> >> > strange
> > > > >> > >> >> > to have two ways to
> represent
> > null values
> > > in the
> > > > >> same
> > > > >> > schema.
> > > > >> > >> Either
> > > > >> > >> >> > way is
> > > > >> > >> >> > OK with me. I think some
> > message-level
> > > > optimizations
> > > > >> > are
> > > > >> > >> justifiable,
> > > > >> > >> >> > but
> > > > >> > >> >> > the savings here seem
> minimal (a
> > few bytes
> > > per
> > > > >> > message), so
> > > > >> > >> maybe it's
> > > > >> > >> >> > not
> > > > >> > >> >> > worth the cost of letting
> the
> > message
> > > diverge even
> > > > >> > further from
> > > > >> > >> the
> > > > >> > >> >> > rest of
> > > > >> > >> >> > the protocol.
> > > > >> > >> >> >
> > > > >> > >> >> > -Jason
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> > On Wed, Feb 15, 2017 at
> 8:52 AM,
> > radai <
> > > > >> > >> radai.rosenblatt@gmail.com>
> > > > >> > >> >> > wrote:
> > > > >> > >> >> >
> > > > >> > >> >> > > I've trimmed the inline
> > contents as this
> > > mail is
> > > > >> > getting too
> > > > >> > >> big for
> > > > >> > >> >> > the
> > > > >> > >> >> > > apache mailing list
> software to
> > deliver
> > > :-(
> > > > >> > >> >> > >
> > > > >> > >> >> > > 1. the important thing for
> > > interoperability is
> > > > for
> > > > >> > different
> > > > >> > >> >> > "interested
> > > > >> > >> >> > > parties" (plugins, infra
> > layers/wrappers,
> > > > >> user-code)
> > > > >> > to be
> > > > >> > >> able to
> > > > >> > >> >> > stick
> > > > >> > >> >> > > pieces of metadata onto
> msgs
> > without
> > > getting in
> > > > >> each
> > > > >> > other's
> > > > >> > >> way. a
> > > > >> > >> >> > common
> > > > >> > >> >> > > key scheme (Strings, as
> of the
> > time of
> > > this
> > > > >> > writing?) is all
> > > > >> > >> thats
> > > > >> > >> >> > required
> > > > >> > >> >> > > for that. it is assumed
> that
> > the other end
> > > > >> > interested in any
> > > > >> > >> such
> > > > >> > >> >> > piece of
> > > > >> > >> >> > > metadata knows the
> encoding,
> > and byte[]
> > > provides
> > > > >> for
> > > > >> > the most
> > > > >> > >> >> > flexibility.
> > > > >> > >> >> > > i believe this is the same
> > logic behind
> > > core
> > > > kafka
> > > > >> > being
> > > > >> > >> >> > byte[]/byte[] -
> > > > >> > >> >> > > Strings are more "usable"
> but
> > bytes are
> > > flexible
> > > > >> and
> > > > >> > so were
> > > > >> > >> chosen.
> > > > >> > >> >> > > Also - core kafka doesnt
> even
> > do that
> > > good of a
> > > > >> job
> > > > >> > on
> > > > >> > >> usability of
> > > > >> > >> >> > the
> > > > >> > >> >> > > payload (example - i have
> to
> > specify the
> > > nop
> > > > >> byte[]
> > > > >> > "decoders"
> > > > >> > >> >> > explicitly
> > > > >> > >> >> > > in conf), and again
> sacrificies
> > usability
> > > for
> > > > the
> > > > >> > sake of
> > > > >> > >> >> > performance (no
> > > > >> > >> >> > > convenient single-record
> > processing as
> > > poll is a
> > > > >> > batch, lots of
> > > > >> > >> >> > obscure
> > > > >> > >> >> > > little config details
> exposing
> > internals
> > > of the
> > > > >> > batching
> > > > >> > >> mechanism,
> > > > >> > >> >> > etc)
> > > > >> > >> >> > >
> > > > >> > >> >> > > this is also why i really
> > dislike the
> > > idea of a
> > > > >> > "type system"
> > > > >> > >> for
> > > > >> > >> >> > header
> > > > >> > >> >> > > values, it further
> degrades the
> > > usability, adds
> > > > >> > complexity and
> > > > >> > >> will
> > > > >> > >> >> > > eventually get in
> people's way,
> > also, it
> > > would
> > > > be
> > > > >> > the 2nd/3rd
> > > > >> > >> >> > home-group
> > > > >> > >> >> > > serialization mechanism
> in core
> > kafka
> > > (counting
> > > > 2
> > > > >> > iterations
> > > > >> > >> of the
> > > > >> > >> >> > "type
> > > > >> > >> >> > > definition DSL")
> > > > >> > >> >> > >
> > > > >> > >> >> > > 2. this is an
> implementation
> > detail, and
> > > not
> > > > even
> > > > >> a
> > > > >> > very "user
> > > > >> > >> >> > facing" one?
> > > > >> > >> >> > > to the best of my
> understanding
> > the vote
> > > process
> > > > >> is
> > > > >> > on proposed
> > > > >> > >> >> > > API/behaviour. also -
> since
> > we're willing
> > > to go
> > > > >> with
> > > > >> > strings
> > > > >> > >> just
> > > > >> > >> >> > serialize
> > > > >> > >> >> > > a 0-sized header blob and
> IIUC
> > you dont
> > > need any
> > > > >> > optionals
> > > > >> > >> anymore.
> > > > >> > >> >> > >
> > > > >> > >> >> > > 3. yes, we can :-)
> > > > >> > >> >> > >
> > > > >> > >> >> > > On Tue, Feb 14, 2017 at
> 11:56
> > PM, Michael
> > > > Pearce <
> > > > >> > >> >> > Michael.Pearce@ig.com>
> > > > >> > >> >> > > wrote:
> > > > >> > >> >> > >
> > > > >> > >> >> > > > Hi Jay,
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > 1) There was some
> initial
> > debate on the
> > > value
> > > > >> > part, as youll
> > > > >> > >> note
> > > > >> > >> >> > String,
> > > > >> > >> >> > > > String headers were
> > discounted early
> > > on. The
> > > > >> > reason for this
> > > > >> > >> is
> > > > >> > >> >> > > flexibility
> > > > >> > >> >> > > > and keeping in line
> with the
> > > flexibility of
> > > > key,
> > > > >> > value of the
> > > > >> > >> >> > message
> > > > >> > >> >> > > > object itself. I don’t
> think
> > it takes
> > > away
> > > > from
> > > > >> an
> > > > >> > ecosystem
> > > > >> > >> as
> > > > >> > >> >> > each
> > > > >> > >> >> > > plugin
> > > > >> > >> >> > > > will care for their own
> key,
> > this way
> > > ints,
> > > > >> > booleans , exotic
> > > > >> > >> >> > custom
> > > > >> > >> >> > > binary
> > > > >> > >> >> > > > can all be catered for=.
> > > > >> > >> >> > > > a. If you really wanted
> to
> > push for a
> > > typed
> > > > >> value
> > > > >> > interface,
> > > > >> > >> I
> > > > >> > >> >> > wouldn’t
> > > > >> > >> >> > > > want just String values
> > supported, but
> > > the the
> > > > >> > primatives
> > > > >> > >> plus
> > > > >> > >> >> > string and
> > > > >> > >> >> > > > also still keeping the
> > ability to have a
> > > > binary
> > > > >> > for custom
> > > > >> > >> >> > binaries that
> > > > >> > >> >> > > > some organisations may
> have.
> > > > >> > >> >> > > > i. I have written this
> slight
> > > alternative
> > > > here,
> > > > >> > >> >> > > https://cwiki.apache.org/
> > > > >> > >> >> > > >
> confluence/display/KAFKA/KIP-
> > > > >> > 82+-+Add+Record+Headers+-+Typed
> > > > >> > >> >> > > > ii. Essentially the
> value
> > bytes, has a
> > > leading
> > > > >> > byte overhead.
> > > > >> > >> >> > > > 1. This tells you what
> type
> > the value
> > > is,
> > > > >> before
> > > > >> > reading
> > > > >> > >> the rest
> > > > >> > >> >> > of the
> > > > >> > >> >> > > > bytes, allowing
> > > serialisation/deserialization
> > > > to
> > > > >> > and from the
> > > > >> > >> >> > primitives,
> > > > >> > >> >> > > > string and byte[]. This
> is
> > akin to some
> > > other
> > > > >> > messaging
> > > > >> > >> systems.
> > > > >> > >> >> > > > 2) We are making it
> optional,
> > so that
> > > for
> > > > those
> > > > >> > not wanting
> > > > >> > >> >> > headers have
> > > > >> > >> >> > > 0
> > > > >> > >> >> > > > bytes overhead (think
> of it
> > as a feature
> > > > flag),
> > > > >> I
> > > > >> > don’t
> > > > >> > >> think this
> > > > >> > >> >> > is
> > > > >> > >> >> > > > complex, especially if
> > comparing to
> > > changes
> > > > >> > proposed in
> > > > >> > >> other kips
> > > > >> > >> >> > like
> > > > >> > >> >> > > > kip-98.
> > > > >> > >> >> > > > a. If you really really
> don’t
> > like
> > > this, we
> > > > can
> > > > >> > drop it, but
> > > > >> > >> it
> > > > >> > >> >> > would
> > > > >> > >> >> > > mean
> > > > >> > >> >> > > > buying into 4 bytes
> extra
> > overhead for
> > > users
> > > > who
> > > > >> > do not want
> > > > >> > >> to use
> > > > >> > >> >> > > headers.
> > > > >> > >> >> > > > 3) In the summary yes,
> it is
> > at a higher
> > > > level,
> > > > >> > but I think
> > > > >> > >> this
> > > > >> > >> >> > is well
> > > > >> > >> >> > > > documented in the
> proposed
> > changes
> > > section.
> > > > >> > >> >> > > > a. Added getHeaders
> method to
> > > > Producer/Consumer
> > > > >> > record (that
> > > > >> > >> is it)
> > > > >> > >> >> > > > b. We’ve also detailed
> the
> > new Headers
> > > class
> > > > >> that
> > > > >> > this method
> > > > >> > >> >> > returns
> > > > >> > >> >> > > that
> > > > >> > >> >> > > > encapsulates the headers
> > protocol and
> > > logic.
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > Best,
> > > > >> > >> >> > > > Mike
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > ==Original questions
> from the
> > vote
> > > thread from
> > > > >> > Jay.==
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > Couple of things I
> think we
> > still need
> > > to work
> > > > >> out:
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > 1. I think we agree
> about
> > the key,
> > > but I
> > > > >> think
> > > > >> > we haven't
> > > > >> > >> >> > talked about
> > > > >> > >> >> > > > the value yet. I
> think if
> > our goal
> > > is an
> > > > open
> > > > >> > ecosystem
> > > > >> > >> of these
> > > > >> > >> >> > > header
> > > > >> > >> >> > > > spread across many
> plugins
> > from many
> > > > systems
> > > > >> we
> > > > >> > should
> > > > >> > >> consider
> > > > >> > >> >> > making
> > > > >> > >> >> > > > this
> > > > >> > >> >> > > > a string as well so
> it can
> > be
> > > printed, set
> > > > >> via
> > > > >> > a UI, set
> > > > >> > >> in
> > > > >> > >> >> > config,
> > > > >> > >> >> > > etc.
> > > > >> > >> >> > > > Basically encouraging
> > pluggable
> > > > serialization
> > > > >> > formats
> > > > >> > >> here will
> > > > >> > >> >> > lead
> > > > >> > >> >> > > to
> > > > >> > >> >> > > > a
> > > > >> > >> >> > > > bit of a tower of
> babel.
> > > > >> > >> >> > > > 2. This proposal
> still
> > includes a
> > > pretty
> > > > big
> > > > >> > change to our
> > > > >> > >> >> > > serialization
> > > > >> > >> >> > > > and protocol
> definition
> > layer.
> > > Essentially
> > > > >> it is
> > > > >> > >> introducing an
> > > > >> > >> >> > > optional
> > > > >> > >> >> > > > type, where the
> format is
> > data
> > > dependent. I
> > > > >> > think this is
> > > > >> > >> >> > actually a
> > > > >> > >> >> > > big
> > > > >> > >> >> > > > change though it
> doesn't
> > seem like
> > > it. It
> > > > >> means
> > > > >> > you can no
> > > > >> > >> >> > longer
> > > > >> > >> >> > > > specify
> > > > >> > >> >> > > > this type with our
> type
> > definition
> > > DSL, and
> > > > >> > likewise it
> > > > >> > >> requires
> > > > >> > >> >> > > custom
> > > > >> > >> >> > > > handling in client
> libs.
> > This isn't
> > > a huge
> > > > >> > thing, since
> > > > >> > >> the
> > > > >> > >> >> > Record
> > > > >> > >> >> > > > definition is custom
> > anyway, but I
> > > think
> > > > this
> > > > >> > kind of
> > > > >> > >> protocol
> > > > >> > >> >> > > > inconsistency is very
> > non-desirable
> > > and
> > > > ties
> > > > >> > you to
> > > > >> > >> hand-coding
> > > > >> > >> >> > > things.
> > > > >> > >> >> > > > I
> > > > >> > >> >> > > > think the type should
> > instead by [Key
> > > > Value]
> > > > >> in
> > > > >> > our BNF,
> > > > >> > >> where
> > > > >> > >> >> > key and
> > > > >> > >> >> > > > value are both short
> > strings as used
> > > > >> elsewhere.
> > > > >> > This
> > > > >> > >> brings it
> > > > >> > >> >> > in line
> > > > >> > >> >> > > > with
> > > > >> > >> >> > > > the rest of the
> protocol.
> > > > >> > >> >> > > > 3. Could we get more
> > specific about
> > > the
> > > > exact
> > > > >> > Java API
> > > > >> > >> change to
> > > > >> > >> >> > > > ProducerRecord,
> > ConsumerRecord,
> > > Record,
> > > > etc?
> > > > >> > >> >> > > >
> > > > >> > >> >> > > > -Jay
> > > > >> > >> >> > > >
> > > > >> > >> >> > >
> > > > >> > >> >> >
> > > > >> > >> >> >
> > > > >> > >> >> > The information contained in
> this
> > email is
> > > strictly
> > > > >> > confidential and
> > > > >> > >> for
> > > > >> > >> >> > the use of the addressee only,
> unless
> > otherwise
> > > > >> indicated.
> > > > >> > If you
> > > > >> > >> are not
> > > > >> > >> >> > the intended recipient, please
> do not
> > read,
> > > copy, use
> > > > or
> > > > >> > disclose to
> > > > >> > >> others
> > > > >> > >> >> > this message or any attachment.
> > Please also
> > > notify the
> > > > >> > sender by
> > > > >> > >> replying
> > > > >> > >> >> > to this email or by telephone
> > (+44(020 7896
> > > 0011) and
> > > > >> then
> > > > >> > delete
> > > > >> > >> the email
> > > > >> > >> >> > and any copies of it. Opinions,
> > conclusion
> > > (etc) that
> > > > do
> > > > >> > not relate
> > > > >> > >> to the
> > > > >> > >> >> > official business of this
> company
> > shall be
> > > understood
> > > > as
> > > > >> > neither
> > > > >> > >> given nor
> > > > >> > >> >> > endorsed by it. IG is a trading
> name
> > of IG
> > > Markets
> > > > >> Limited
> > > > >> > (a company
> > > > >> > >> >> > registered in England and Wales,
> > company number
> > > > >> 04008957)
> > > > >> > and IG
> > > > >> > >> Index
> > > > >> > >> >> > Limited (a company registered in
> > England and
> > > Wales,
> > > > >> > company number
> > > > >> > >> >> > 01190902). Registered address at
> > Cannon Bridge
> > > House,
> > > > 25
> > > > >> > Dowgate
> > > > >> > >> Hill,
> > > > >> > >> >> > London EC4R 2YA. Both IG Markets
> > Limited
> > > (register
> > > > >> number
> > > > >> > 195355)
> > > > >> > >> and IG
> > > > >> > >> >> > Index Limited (register number
> > 114059) are
> > > authorised
> > > > >> and
> > > > >> > regulated
> > > > >> > >> by the
> > > > >> > >> >> > Financial Conduct Authority.
> > > > >> > >> >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Nacho - Ignacio Solis -
> isolis@igso.net
> > > > >> > >>
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > The information contained in this email is
> strictly
> > > confidential and
> > > > for
> > > > >> > the use of the addressee only, unless otherwise
> > indicated. If
> > > you are
> > > > >> not
> > > > >> > the intended recipient, please do not read,
> copy, use
> > or
> > > disclose to
> > > > >> others
> > > > >> > this message or any attachment. Please also
> notify
> > the sender by
> > > > >> replying
> > > > >> > to this email or by telephone (+44(020 7896
> 0011) and
> > then
> > > delete the
> > > > >> email
> > > > >> > and any copies of it. Opinions, conclusion
> (etc) that
> > do not
> > > relate to
> > > > >> the
> > > > >> > official business of this company shall be
> understood
> > as
> > > neither given
> > > > >> nor
> > > > >> > endorsed by it. IG is a trading name of IG
> Markets
> > Limited (a
> > > company
> > > > >> > registered in England and Wales, company number
> > 04008957) and
> > > IG Index
> > > > >> > Limited (a company registered in England and
> Wales,
> > company
> > > number
> > > > >> > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > > Dowgate Hill,
> > > > >> > London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > 195355) and
> > > > IG
> > > > >> > Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by
> > > > >> the
> > > > >> > Financial Conduct Authority.
> > > > >> >
> > > > >> The information contained in this email is
> strictly
> > confidential
> > > and for
> > > > >> the use of the addressee only, unless otherwise
> > indicated. If you
> > > are
> > > > not
> > > > >> the intended recipient, please do not read, copy,
> use
> > or disclose
> > > to
> > > > others
> > > > >> this message or any attachment. Please also
> notify the
> > sender by
> > > > replying
> > > > >> to this email or by telephone (+44(020 7896 0011)
> and
> > then delete
> > > the
> > > > email
> > > > >> and any copies of it. Opinions, conclusion (etc)
> that
> > do not
> > > relate to
> > > > the
> > > > >> official business of this company shall be
> understood
> > as neither
> > > given
> > > > nor
> > > > >> endorsed by it. IG is a trading name of IG Markets
> > Limited (a
> > > company
> > > > >> registered in England and Wales, company number
> > 04008957) and IG
> > > Index
> > > > >> Limited (a company registered in England and
> Wales,
> > company number
> > > > >> 01190902). Registered address at Cannon Bridge
> House,
> > 25 Dowgate
> > > Hill,
> > > > >> London EC4R 2YA. Both IG Markets Limited (register
> > number 195355)
> > > and IG
> > > > >> Index Limited (register number 114059) are
> authorised
> > and
> > > regulated by
> > > > the
> > > > >> Financial Conduct Authority.
> > > > >>
> > > > >
> > > > >
> > > >
> > > The information contained in this email is strictly
> > confidential and
> > > for the use of the addressee only, unless otherwise
> indicated.
> > If you are
> > > not the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > others this message or any attachment. Please also notify
> the
> > sender by
> > > replying to this email or by telephone (+44(020 7896 0011)
> and
> > then delete
> > > the email and any copies of it. Opinions, conclusion (etc)
> that
> > do not
> > > relate to the official business of this company shall be
> > understood as
> > > neither given nor endorsed by it. IG is a trading name of
> IG
> > Markets
> > > Limited (a company registered in England and Wales, company
> > number
> > > 04008957) and IG Index Limited (a company registered in
> England
> > and Wales,
> > > company number 01190902). Registered address at Cannon
> Bridge
> > House, 25
> > > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited
> (register
> > number
> > > 195355) and IG Index Limited (register number 114059) are
> > authorised and
> > > regulated by the Financial Conduct Authority.
> > >
> > >
> > >
> > >
> >
> >
> > The information contained in this email is strictly confidential
> and
> > for the use of the addressee only, unless otherwise indicated. If
> you are
> > not the intended recipient, please do not read, copy, use or
> disclose to
> > others this message or any attachment. Please also notify the sender
> by
> > replying to this email or by telephone (+44(020 7896 0011) and then
> delete
> > the email and any copies of it. Opinions, conclusion (etc) that do
> not
> > relate to the official business of this company shall be understood
> as
> > neither given nor endorsed by it. IG is a trading name of IG Markets
> > Limited (a company registered in England and Wales, company number
> > 04008957) and IG Index Limited (a company registered in England and
> Wales,
> > company number 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and IG Index Limited (register number 114059) are authorised
> and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
For APM single event tracing, need access to the header at the point of processing on the processing thread.
As such interceptors will not work/be suitable for these, due to the fact they act on the ConsumerRecords as a batch, before the handling thread can split out and process per message which is the point these tools will need to continue to transaction tracing.
Like wise tools and infra pieces will need access to the message outside the interceptor realm.
On 17/02/2017, 21:26, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> That’s exactly what we’re doing the headers are a slice of bytes, which
> then gets parsed later if needed, or can be parsed right away, the headers
> is part of the protocol, so can still be validated if wanted.
> If you had a header count then you would have to go through each header
> key and value length value to work out how much to skip to get to say the
> value or any future component in the message after the headers. Having it
> as a byte[] with length value makes this a lot easier to skip.
So the broker will parse the headers and validate them. Good. The only
reason remaining that I can see to leave the headers as a byte array is to
make it easier for the client to skip past them. Are we sure this is not
premature optimization? Are there any performance results which show that
this is worthwhile?
What’s the issue with exposing a method getHeaders on the producer/consumer
> record? It doesn’t break anything. We don’t need any special version.
See my previous explanation. What I am trying to resist is the headers
becoming a general application-level facility. The primary use case as far
as I can tell is middleware, which is the use case that the interceptors
are providing.
Current batch consumer model and consumer interceptors don’t work where
> headers need to be acted on at per message level at time of processing,
> very case is APM (the core one), where the header value is used to continue
> tracing.
I still don't understand the point about batching. The consumer records are
exposed as a batch in the consumer interceptor, but you can still iterate
them individually. It is no different for the consumer API itself.
-Jason
On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Re:
>
> “ The point about creation of maps seems orthogonal. We can still
> represent
> the headers as a slice of bytes until the time it is accessed.”
>
> That’s exactly what we’re doing the headers are a slice of bytes, which
> then gets parsed later if needed, or can be parsed right away, the headers
> is part of the protocol, so can still be validated if wanted.
>
> If you had a header count then you would have to go through each header
> key and value length value to work out how much to skip to get to say the
> value or any future component in the message after the headers. Having it
> as a byte[] with length value makes this a lot easier to skip.
>
>
> On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com> wrote:
>
> What’s the issue with exposing a method getHeaders on the
> producer/consumer record? It doesn’t break anything. We don’t need any
> special version.
>
> Current batch consumer model and consumer interceptors don’t work
> where headers need to be acted on at per message level at time of
> processing, very case is APM (the core one), where the header value is used
> to continue tracing. JMS/HTTP etc all expose these, without issues. I would
> NOT want to lock this down to only be usable accessible via interceptors,
> as we’d fail on one of the main goals.
>
> Regards
> Mike
>
>
>
>
> On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> The point about creation of maps seems orthogonal. We can still
> represent
> the headers as a slice of bytes until the time it is accessed.
>
>
> > Yes exactly we have access to the records thus why the header
> should be
> > accessible via it and not hidden for only interceptors to access.
>
>
> As explained above, the point is to make the intended usage clear.
> Applications should continue to rely on the key/value fields to
> serialize
> their own headers, and it would be more ideal if we can avoid
> leaking
> third-party headers into applications. This is difficult to do
> with the
> current interceptors because they share the record objects with
> the common
> API. What I had in mind is something like an extension of the
> current
> interceptors which exposed a different object (e.g.
> `RecordAndHeaders`).
> The challenge is for MM-like use cases. Let me see if I can come
> up with a
> concrete proposal for that problem.
>
> -Jason
>
>
>
> On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > I am happy to move the definition of the header into the message
> body, but
> > would cause us not to lazy initialise/parse the headers, as
> obviously, we
> > would have to traverse these reading the message.
> >
> > This was actually one of Jay’s requests:
> >
> > “ 2. I think we should think about creating the Map lazily to
> avoid
> > parsing out all the headers into little objects. HashMaps
> themselves
> > are
> > kind of expensive and the consumer is very perf sensitive so
> and making
> > gazillions of hashmaps that may or may not get used is
> probably a bad
> > idea.”
> >
> >
> >
> >
> >
> > On 17/02/2017, 19:44, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Yes exactly we have access to the records thus why the
> header should
> > be accessible via it and not hidden for only interceptors to
> access.
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Magnus Edenhill <ma...@edenhill.se>
> > Sent: Friday, February 17, 2017 7:34:49 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > Big +1 on VarInts.
> > CPUs are fast, memory is slow.
> >
> > I agree with Jason that we'll want to continue verifying
> messages,
> > including their headers, so while I appreciate the idea of
> the opaque
> > header blob it won't be useful in practice.
> >
> > /Magnus
> >
> > 2017-02-17 10:41 GMT-08:00 Jason Gustafson <
> jason@confluent.io>:
> >
> > > Sorry, my mistake. The consumer interceptor is per batch,
> though I'm
> > not
> > > sure that's an actual limitation since you still have
> access to the
> > > individual records.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> > jason@confluent.io>
> > > wrote:
> > >
> > > > Re headers as byte array and future use by broker. This
> doesn't
> > take away
> > > >> from that at all. Nor makes it difficult at all in my
> opinion.
> > > >
> > > >
> > > > Yeah, I didn't say it was difficult, only awkward. You
> wouldn't
> > write the
> > > > schema that way if you were planning to use it on the
> brokers from
> > the
> > > > beginning. Note also that one of the benefits of letting
> the broker
> > > > understand headers is that it can validate that they are
> properly
> > > > formatted. If cost is the only concern, we should
> confirm its
> > impact
> > > > through performance testing.
> > > >
> > > > One of the key use cases requires access on consume at
> per
> > event/message
> > > >> level at the point that message is being processed, as
> such the
> > batch
> > > >> interceptors and batch consume api isn't suitable. It
> needs to be
> > at the
> > > >> record level.
> > > >
> > > >
> > > > I'm not sure I understand the point about batching.
> Interceptors
> > are
> > > > applied per-message, right?
> > > >
> > > > My intent on interceptors is to keep the usage of headers
> > well-defined so
> > > > that they don't start leaking unnecessarily into
> applications. My
> > guess
> > > is
> > > > that it's probably inevitable, but isolating it in the
> > interceptors would
> > > > at least give people a second thought before deciding to
> use it.
> > The main
> > > > challenge in my mind is figuring out how an MM use case
> would
> > work. It
> > > > would be more cumbersome to replicate headers through an
> > interceptor,
> > > > though arguably MM should be working at a lower level
> anyway.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > >> Re headers available on the record va interceptors only
> > > >>
> > > >> One of the key use cases requires access on consume at
> per
> > event/message
> > > >> level at the point that message is being processed, as
> such the
> > batch
> > > >> interceptors and batch consume api isn't suitable. It
> needs to be
> > at the
> > > >> record level.
> > > >>
> > > >> This anyhow is similar to jms/http/amqp where headers
> are
> > available to
> > > >> consuming applications.
> > > >>
> > > >> Re headers as byte array and future use by broker. This
> doesn't
> > take
> > > away
> > > >> from that at all. Nor makes it difficult at all in my
> opinion.
> > > >>
> > > >>
> > > >>
> > > >> Sent using OWA for iPhone
> > > >> ________________________________________
> > > >> From: Jason Gustafson <ja...@confluent.io>
> > > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > > >> To: dev@kafka.apache.org
> > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >>
> > > >> >
> > > >> > Would you be proposing in KIP-98 to convert the other
> message
> > int’s
> > > (key
> > > >> > length, value length) also to varint to keep it
> uniform.
> > > >> > Also I assume there will be a static or helper method
> made to
> > > write/read
> > > >> > these in the client and server.
> > > >>
> > > >>
> > > >> Yes, that is what we are proposing, so using varints
> for headers
> > would
> > > be
> > > >> consistent with the rest of the message. We have used
> static
> > helper
> > > >> methods
> > > >> in our prototype implementation.
> > > >>
> > > >> The cost of parsing, we want to parse/interpret the
> headers
> > lazily (this
> > > >> is
> > > >> > a key point brought up earlier in discussions)
> > > >>
> > > >>
> > > >> I'm a bit skeptical of this. Has anyone done the
> performance
> > testing? I
> > > >> can
> > > >> probably implement it and test it if no one else has. I
> was also
> > under
> > > the
> > > >> impression that there may be use cases down the road
> where the
> > broker
> > > >> would
> > > >> need to interpret headers. That wouldn't be off the
> table in the
> > future
> > > if
> > > >> it's represented as bytes, but it would be quite a bit
> more
> > awkward,
> > > >> right?
> > > >>
> > > >> By the way, one question I have been wondering about. My
> > understanding
> > > is
> > > >> that headers are primarily for use cases where a
> third-party
> > components
> > > >> wants to enrich messages without needing to understand
> or modify
> > the
> > > >> schema
> > > >> of the message key and value. For the applications
> which directly
> > > produce
> > > >> and consume the messages and control the key/value
> schema
> > directly, it
> > > >> seems we would rather have them implement headers
> directly in
> > their own
> > > >> schema. Supposing for the sake of argument that it was
> possible,
> > my
> > > >> question is whether it be sufficient to expose the
> headers in the
> > > >> interceptor API and not in the common API?
> > > >>
> > > >> -Jason
> > > >>
> > > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > >> wrote:
> > > >>
> > > >> > On the point of varInts
> > > >> >
> > > >> > Would you be proposing in KIP-98 to convert the other
> message
> > int’s
> > > (key
> > > >> > length, value length) also to varint to keep it
> uniform.
> > > >> > Also I assume there will be a static or helper method
> made to
> > > write/read
> > > >> > these in the client and server.
> > > >> >
> > > >> > Cheers
> > > >> > Mike
> > > >> >
> > > >> >
> > > >> >
> > > >> > On 17/02/2017, 11:22, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > wrote:
> > > >> >
> > > >> > On the point re: headers in the message protocol
> being a
> > byte
> > > array
> > > >> > and not a count of elements followed by the elements.
> Again
> > this was
> > > >> > discussed/argued previously.
> > > >> >
> > > >> > It was agreed on for a few reasons some of which
> you have
> > > obviously
> > > >> > picked up on:
> > > >> >
> > > >> > Broker is able to pass it through opaquely
> > > >> > The cost of parsing, we want to parse/interpret
> the headers
> > lazily
> > > >> > (this is a key point brought up earlier in
> discussions)
> > > >> > Headers can be copied from consumer record to
> producer
> > record (aka
> > > >> > mirror makers etc) without parsing if no changes are
> being made
> > or
> > > being
> > > >> > looked at.
> > > >> > Keeps the broker agnostic to the format
> > > >> > You need an int32 either for the byte size of the
> headers,
> > or for
> > > >> the
> > > >> > count of elements, so overheads are the same, but
> with going
> > with an
> > > >> opaque
> > > >> > byte array has the above advantages.
> > > >> >
> > > >> > Cheers
> > > >> > Mike
> > > >> >
> > > >> >
> > > >> > On 17/02/2017, 02:50, "Jason Gustafson" <
> jason@confluent.io
> > >
> > > wrote:
> > > >> >
> > > >> > Sorry, should have noted that the performance
> testing
> > was done
> > > >> > using the
> > > >> > producer performance tool shipped with Kafka.
> > > >> >
> > > >> > -Jason
> > > >> >
> > > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason
> Gustafson <
> > > >> > jason@confluent.io> wrote:
> > > >> >
> > > >> > > Hey Nacho,
> > > >> > >
> > > >> > > I've compared performance of our KIP-98
> > implementation with
> > > >> and
> > > >> > without
> > > >> > > varints. For messages around 128 bytes, we
> see an
> > increase
> > > in
> > > >> > throughput of
> > > >> > > about 30% using the default configuration
> settings.
> > At 256
> > > >> > bytes, the
> > > >> > > increase is around 16%. Obviously the
> performance
> > converge
> > > as
> > > >> > messages get
> > > >> > > larger, but it seems well worth the cost.
> Note that
> > we are
> > > >> also
> > > >> > seeing a
> > > >> > > substantial performance increase against
> trunk
> > primarily
> > > >> because
> > > >> > of the
> > > >> > > much more efficient packing that varints
> provide us.
> > > Anything
> > > >> > adding to
> > > >> > > message overhead, such as record headers,
> would only
> > > increase
> > > >> > the relative
> > > >> > > difference. (Of course take these numbers
> with a
> > grain of
> > > salt
> > > >> > since I have
> > > >> > > only used the default settings with both
> the producer
> > and
> > > >> broker
> > > >> > on my
> > > >> > > local machine. We intend to provide more
> extensive
> > > performance
> > > >> > details as
> > > >> > > part of the work for KIP-98.)
> > > >> > >
> > > >> > > The implementation we are using is from
> protobuf (
> > > >> > > https://developers.google.com/
> > > protocol-buffers/docs/encoding
> > > >> ),
> > > >> > which is
> > > >> > > also used in HBase. It is trivial to
> implement and as
> > far
> > > as I
> > > >> > know doesn't
> > > >> > > suffer from the aliasing problem you are
> describing. I
> > > checked
> > > >> > with Magnus
> > > >> > > (the author of librdkafka) and he agreed
> that the
> > savings
> > > >> seemed
> > > >> > worth the
> > > >> > > cost of implementation.
> > > >> > >
> > > >> > > -Jason
> > > >> > >
> > > >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio
> Solis <
> > > >> isolis@igso.net>
> > > >> > wrote:
> > > >> > >
> > > >> > >> -VarInts
> > > >> > >>
> > > >> > >> I'm one of the people (if not the most)
> opposed to
> > VarInts.
> > > >> > VarInts
> > > >> > >> have a place, but this is not it. (We
> had a large
> > > >> discussion
> > > >> > about
> > > >> > >> them at the beginning of KIP-82 time)
> > > >> > >>
> > > >> > >> If anybody has real life performance
> numbers of
> > VarInts
> > > >> > improving
> > > >> > >> things or significantly reducing resources
> I would
> > like to
> > > >> know
> > > >> > what
> > > >> > >> that case may be. Yes, you can save some
> bytes here
> > and
> > > >> there,
> > > >> > but
> > > >> > >> this is probably insignificant to the
> overall system
> > > behavior
> > > >> > and
> > > >> > >> storage requirements. -- I say this with
> respect to
> > using
> > > >> > VarInts in
> > > >> > >> the protocol itself, not as part of the
> data.
> > > >> > >>
> > > >> > >> VarInts require you to parse the Int
> before using it
> > and
> > > >> > depending on
> > > >> > >> the encoding they can suffer from aliasing
> (multiple
> > > >> > representations
> > > >> > >> for the same value).
> > > >> > >>
> > > >> > >> Why add complexity?
> > > >> > >>
> > > >> > >> Nacho
> > > >> > >>
> > > >> > >>
> > > >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin
> McCabe <
> > > >> > cmccabe@apache.org>
> > > >> > >> wrote:
> > > >> > >> > +1 for varints here-- it would save
> quite a bit of
> > space.
> > > >> > They are
> > > >> > >> > pretty quick to implement as well.
> > > >> > >> >
> > > >> > >> > I think it makes sense for values to be
> byte
> > arrays.
> > > Users
> > > >> > might want
> > > >> > >> > to attach arbitrary payloads; they
> shouldn't be
> > forced to
> > > >> > serialize
> > > >> > >> > everything to Java strings.
> > > >> > >> >
> > > >> > >> > best,
> > > >> > >> > Colin
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason
> Gustafson
> > wrote:
> > > >> > >> >> Hey Michael,
> > > >> > >> >>
> > > >> > >> >> Hmm, I guess the point of representing
> it as
> > bytes is to
> > > >> > allow the
> > > >> > >> broker
> > > >> > >> >> to pass it through opaquely? Is the
> cost of
> > parsing
> > > them a
> > > >> > concern, or
> > > >> > >> >> are
> > > >> > >> >> we simply trying to ensure that the
> broker stays
> > > agnostic
> > > >> to
> > > >> > the
> > > >> > >> format?
> > > >> > >> >>
> > > >> > >> >> On varints, I think adding support for
> them makes
> > less
> > > >> sense
> > > >> > for an
> > > >> > >> >> isolated use case, but as part of a
> more holistic
> > change
> > > >> > (such as what
> > > >> > >> we
> > > >> > >> >> have proposed in KIP-98), I think they
> are
> > justifiable.
> > > If
> > > >> > we add them,
> > > >> > >> >> then the need to use attributes becomes
> quite a
> > bit
> > > >> weaker,
> > > >> > right? The
> > > >> > >> >> other thing I find slightly odd is the
> fact that
> > null
> > > >> > headers has no
> > > >> > >> >> actual
> > > >> > >> >> semantic meaning for the message
> (unlike null
> > keys and
> > > >> > values). It is
> > > >> > >> >> just
> > > >> > >> >> a space optimization. It seems a bit
> better to
> > always
> > > use
> > > >> > size 0 to
> > > >> > >> >> indicate having no headers.
> > > >> > >> >>
> > > >> > >> >> Overall, the main point is ensuring
> that the
> > message
> > > >> schema
> > > >> > remains
> > > >> > >> >> consistent, either within the larger
> protocol, or
> > at a
> > > >> > minimum within
> > > >> > >> the
> > > >> > >> >> message itself.
> > > >> > >> >>
> > > >> > >> >> -Jason
> > > >> > >> >>
> > > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM,
> Michael Pearce <
> > > >> > Michael.Pearce@ig.com
> > > >> > >> >
> > > >> > >> >> wrote:
> > > >> > >> >>
> > > >> > >> >> > Hi Jason,
> > > >> > >> >> >
> > > >> > >> >> > On point 1) in the message protocol
> the headers
> > are
> > > >> simply
> > > >> > a byte
> > > >> > >> array,
> > > >> > >> >> > as like the key or value, this is to
> clearly
> > demarcate
> > > >> the
> > > >> > header in
> > > >> > >> the
> > > >> > >> >> > core message. Then the header byte
> array in the
> > core
> > > >> > message is an
> > > >> > >> array of
> > > >> > >> >> > key, value pairs. This is what it is
> denoting.
> > > >> > >> >> >
> > > >> > >> >> > Then this would be I guess in the
> given
> > notation:
> > > >> > >> >> >
> > > >> > >> >> > Headers => [KeyLength, Key,
> ValueLength, Value]
> > > >> > >> >> > KeyLength => int32
> <-----------------NEW
> > size of
> > > the
> > > >> > byte[] of
> > > >> > >> the
> > > >> > >> >> > serialised key value
> > > >> > >> >> > Key => bytes
> <---------------------- NEW
> > > serialised
> > > >> > string (UTF8)
> > > >> > >> >> > bytes of the header key
> > > >> > >> >> > ValueLength => int32
> <-------------- NEW
> > size of
> > > the
> > > >> > byte[] of
> > > >> > >> the
> > > >> > >> >> > serialised header value
> > > >> > >> >> > Value => bytes
> <-------------------- NEW
> > > serialised
> > > >> > form of the
> > > >> > >> header
> > > >> > >> >> > value
> > > >> > >> >> >
> > > >> > >> >> > The key length and value length is
> matching the
> > way
> > > the
> > > >> > protocol is
> > > >> > >> >> > defined in the core message currently.
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > On point 2)
> > > >> > >> >> > Var sized ints, this was discussed
> much earlier
> > on, in
> > > >> > fact I had
> > > >> > >> >> > suggested it myself (with Hadoop
> references),
> > the
> > > >> > complexity of this
> > > >> > >> >> > compared to having a simpler protocol
> was
> > argued and
> > > >> > agreed it
> > > >> > >> wasn’t worth
> > > >> > >> >> > the complexity as all other clients
> in other
> > languages
> > > >> > would need to
> > > >> > >> ensure
> > > >> > >> >> > theyre using the right var size
> algorithm, as
> > there
> > > is a
> > > >> > few.
> > > >> > >> >> >
> > > >> > >> >> > On point 3)
> > > >> > >> >> > We did the attributes, optional
> approach as
> > originally
> > > >> > there was
> > > >> > >> marked
> > > >> > >> >> > concern that headers would cause a
> message size
> > > overhead
> > > >> > for others,
> > > >> > >> who
> > > >> > >> >> > don’t want them. As such this is the
> clean
> > solution to
> > > >> > achieve that.
> > > >> > >> If
> > > >> > >> >> > that no longer holds, and we don’t
> care that we
> > add
> > > >> 4bytes
> > > >> > overhead,
> > > >> > >> then
> > > >> > >> >> > im happy to remove.
> > > >> > >> >> >
> > > >> > >> >> > I’m personally in favour of keeping
> the message
> > as
> > > small
> > > >> > as possible
> > > >> > >> so
> > > >> > >> >> > people don’t get shocks in perf and
> throughputs
> > dues
> > > to
> > > >> > message size,
> > > >> > >> >> > unless they actively use the feature,
> as such I
> > do
> > > >> prefer
> > > >> > the
> > > >> > >> attribute bit
> > > >> > >> >> > wise feature flag approach myself.
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > On 16/02/2017, 05:40, "Jason
> Gustafson" <
> > > >> > jason@confluent.io> wrote:
> > > >> > >> >> >
> > > >> > >> >> > We have proposed a few
> significant changes
> > to the
> > > >> > message format
> > > >> > >> in
> > > >> > >> >> > KIP-98
> > > >> > >> >> > which now seems likely to pass
> (perhaps
> > with some
> > > >> > iterations on
> > > >> > >> >> > implementation details). It would
> be good
> > to try
> > > and
> > > >> > coordinate
> > > >> > >> the
> > > >> > >> >> > changes
> > > >> > >> >> > in both of the proposals to make
> sure they
> > are
> > > >> > consistent and
> > > >> > >> >> > compatible.
> > > >> > >> >> >
> > > >> > >> >> > I think using the attributes to
> indicate
> > null
> > > >> headers
> > > >> > is a
> > > >> > >> reasonable
> > > >> > >> >> > approach. We have proposed to do
> the same
> > thing
> > > for
> > > >> > the message
> > > >> > >> key and
> > > >> > >> >> > value. That said, I sympathize
> with Jay's
> > > argument.
> > > >> > Having
> > > >> > >> multiple
> > > >> > >> >> > ways to
> > > >> > >> >> > specify a null value increases
> the overall
> > > >> complexity
> > > >> > of the
> > > >> > >> protocol.
> > > >> > >> >> > You
> > > >> > >> >> > can see this just from the fact
> that you
> > need the
> > > >> > extra verbiage
> > > >> > >> in the
> > > >> > >> >> > protocol specification in this
> KIP and in
> > KIP-98
> > > to
> > > >> > describe the
> > > >> > >> >> > dependence
> > > >> > >> >> > between the fields and the
> attributes. It
> > seems
> > > >> like a
> > > >> > slippery
> > > >> > >> slope
> > > >> > >> >> > if
> > > >> > >> >> > you start allowing different
> request types
> > to
> > > >> > implement the
> > > >> > >> protocol
> > > >> > >> >> > specification differently.
> > > >> > >> >> >
> > > >> > >> >> > You can also argue that the
> messages
> > already are
> > > and
> > > >> > are likely
> > > >> > >> to
> > > >> > >> >> > remain a
> > > >> > >> >> > special case. For example, there
> is
> > currently no
> > > >> > generality in
> > > >> > >> how
> > > >> > >> >> > compressed message sets are
> represented
> > that would
> > > >> be
> > > >> > applicable
> > > >> > >> for
> > > >> > >> >> > other
> > > >> > >> >> > request types. Some might see this
> > divergence as
> > > an
> > > >> > unfortunate
> > > >> > >> >> > protocol
> > > >> > >> >> > deficiency which should be fixed;
> others
> > might see
> > > >> it
> > > >> > as sort of
> > > >> > >> the
> > > >> > >> >> > inevitability of needing to
> optimize where
> > it
> > > counts
> > > >> > most. I'm
> > > >> > >> probably
> > > >> > >> >> > somewhere in between, but I think
> we
> > probably all
> > > >> > share the
> > > >> > >> intuition
> > > >> > >> >> > that
> > > >> > >> >> > the protocol should be kept as
> consistent as
> > > >> possible.
> > > >> > With that
> > > >> > >> in
> > > >> > >> >> > mind,
> > > >> > >> >> > here are a few comments:
> > > >> > >> >> >
> > > >> > >> >> > 1. One thing I found a little odd
> when
> > reading the
> > > >> > current
> > > >> > >> proposal is
> > > >> > >> >> > that
> > > >> > >> >> > the headers are both represented
> as an
> > array of
> > > >> bytes
> > > >> > and as an
> > > >> > >> array
> > > >> > >> >> > of
> > > >> > >> >> > key/value pairs. I'd probably
> suggest
> > something
> > > like
> > > >> > this:
> > > >> > >> >> >
> > > >> > >> >> > Headers => [HeaderKey HeaderValue]
> > > >> > >> >> > HeaderKey => String
> > > >> > >> >> > HeaderValue => Bytes
> > > >> > >> >> >
> > > >> > >> >> > An array in the Kafka protocol is
> > represented as a
> > > >> > 4-byte integer
> > > >> > >> >> > indicating the number of elements
> in the
> > array
> > > >> > followed by the
> > > >> > >> >> > serialization of the elements.
> Unless I'm
> > > >> > misunderstanding, what
> > > >> > >> you
> > > >> > >> >> > have
> > > >> > >> >> > instead is the total size of the
> headers in
> > bytes
> > > >> > followed by the
> > > >> > >> >> > elements.
> > > >> > >> >> > I'm not sure I see any reason for
> this
> > > >> inconsistency.
> > > >> > >> >> >
> > > >> > >> >> > 2. In KIP-98, we've introduced
> > variable-length
> > > >> integer
> > > >> > fields.
> > > >> > >> >> > Effectively,
> > > >> > >> >> > we've enriched (or "complicated"
> as Jay
> > might say
> > > ;)
> > > >> > the protocol
> > > >> > >> >> > specification to include the
> following
> > types:
> > > >> VarInt,
> > > >> > VarLong,
> > > >> > >> >> > UnsignedVarInt and
> UnsignedVarLong.
> > > >> > >> >> >
> > > >> > >> >> > Along with these primitives, we
> could
> > introduce
> > > the
> > > >> > following
> > > >> > >> types:
> > > >> > >> >> >
> > > >> > >> >> > VarSizeArray => NumberOfItems
> Item1 Item2
> > .. ItemN
> > > >> > >> >> > NumberOfItems => UnsignedVarInt
> > > >> > >> >> >
> > > >> > >> >> > VarSizeNullableArray =>
> NumberOfItemsOrNull
> > Item1
> > > >> > Item2 .. ItemN
> > > >> > >> >> > NumberOfItemsOrNull => VarInt
> (-1 means
> > null)
> > > >> > >> >> >
> > > >> > >> >> > And similarly for the `String`
> and `Bytes`
> > types.
> > > >> > These types
> > > >> > >> can save
> > > >> > >> >> > a
> > > >> > >> >> > considerable amount of space in
> this
> > proposal
> > > >> because
> > > >> > they can
> > > >> > >> be used
> > > >> > >> >> > for
> > > >> > >> >> > both the number of headers
> included in the
> > message
> > > >> and
> > > >> > the
> > > >> > >> lengths of
> > > >> > >> >> > the
> > > >> > >> >> > header keys and values. We could
> do this
> > instead:
> > > >> > >> >> >
> > > >> > >> >> > Headers => VarSizeArray[HeaderKey
> > HeaderValue]
> > > >> > >> >> > HeaderKey => VarSizeString
> > > >> > >> >> > HeaderValue => VarSizeBytes
> > > >> > >> >> >
> > > >> > >> >> > Combining the savings from the
> use of
> > variable
> > > >> length
> > > >> > fields, the
> > > >> > >> >> > benefit
> > > >> > >> >> > of using the attributes to
> represent null
> > seems
> > > >> pretty
> > > >> > small.
> > > >> > >> >> >
> > > >> > >> >> > 3. Whichever way we go (whether
> we use the
> > > >> attributes
> > > >> > or not), we
> > > >> > >> >> > should at
> > > >> > >> >> > least be consistent between this
> KIP and
> > KIP-98.
> > > It
> > > >> > would be very
> > > >> > >> >> > strange
> > > >> > >> >> > to have two ways to represent
> null values
> > in the
> > > >> same
> > > >> > schema.
> > > >> > >> Either
> > > >> > >> >> > way is
> > > >> > >> >> > OK with me. I think some
> message-level
> > > optimizations
> > > >> > are
> > > >> > >> justifiable,
> > > >> > >> >> > but
> > > >> > >> >> > the savings here seem minimal (a
> few bytes
> > per
> > > >> > message), so
> > > >> > >> maybe it's
> > > >> > >> >> > not
> > > >> > >> >> > worth the cost of letting the
> message
> > diverge even
> > > >> > further from
> > > >> > >> the
> > > >> > >> >> > rest of
> > > >> > >> >> > the protocol.
> > > >> > >> >> >
> > > >> > >> >> > -Jason
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM,
> radai <
> > > >> > >> radai.rosenblatt@gmail.com>
> > > >> > >> >> > wrote:
> > > >> > >> >> >
> > > >> > >> >> > > I've trimmed the inline
> contents as this
> > mail is
> > > >> > getting too
> > > >> > >> big for
> > > >> > >> >> > the
> > > >> > >> >> > > apache mailing list software to
> deliver
> > :-(
> > > >> > >> >> > >
> > > >> > >> >> > > 1. the important thing for
> > interoperability is
> > > for
> > > >> > different
> > > >> > >> >> > "interested
> > > >> > >> >> > > parties" (plugins, infra
> layers/wrappers,
> > > >> user-code)
> > > >> > to be
> > > >> > >> able to
> > > >> > >> >> > stick
> > > >> > >> >> > > pieces of metadata onto msgs
> without
> > getting in
> > > >> each
> > > >> > other's
> > > >> > >> way. a
> > > >> > >> >> > common
> > > >> > >> >> > > key scheme (Strings, as of the
> time of
> > this
> > > >> > writing?) is all
> > > >> > >> thats
> > > >> > >> >> > required
> > > >> > >> >> > > for that. it is assumed that
> the other end
> > > >> > interested in any
> > > >> > >> such
> > > >> > >> >> > piece of
> > > >> > >> >> > > metadata knows the encoding,
> and byte[]
> > provides
> > > >> for
> > > >> > the most
> > > >> > >> >> > flexibility.
> > > >> > >> >> > > i believe this is the same
> logic behind
> > core
> > > kafka
> > > >> > being
> > > >> > >> >> > byte[]/byte[] -
> > > >> > >> >> > > Strings are more "usable" but
> bytes are
> > flexible
> > > >> and
> > > >> > so were
> > > >> > >> chosen.
> > > >> > >> >> > > Also - core kafka doesnt even
> do that
> > good of a
> > > >> job
> > > >> > on
> > > >> > >> usability of
> > > >> > >> >> > the
> > > >> > >> >> > > payload (example - i have to
> specify the
> > nop
> > > >> byte[]
> > > >> > "decoders"
> > > >> > >> >> > explicitly
> > > >> > >> >> > > in conf), and again sacrificies
> usability
> > for
> > > the
> > > >> > sake of
> > > >> > >> >> > performance (no
> > > >> > >> >> > > convenient single-record
> processing as
> > poll is a
> > > >> > batch, lots of
> > > >> > >> >> > obscure
> > > >> > >> >> > > little config details exposing
> internals
> > of the
> > > >> > batching
> > > >> > >> mechanism,
> > > >> > >> >> > etc)
> > > >> > >> >> > >
> > > >> > >> >> > > this is also why i really
> dislike the
> > idea of a
> > > >> > "type system"
> > > >> > >> for
> > > >> > >> >> > header
> > > >> > >> >> > > values, it further degrades the
> > usability, adds
> > > >> > complexity and
> > > >> > >> will
> > > >> > >> >> > > eventually get in people's way,
> also, it
> > would
> > > be
> > > >> > the 2nd/3rd
> > > >> > >> >> > home-group
> > > >> > >> >> > > serialization mechanism in core
> kafka
> > (counting
> > > 2
> > > >> > iterations
> > > >> > >> of the
> > > >> > >> >> > "type
> > > >> > >> >> > > definition DSL")
> > > >> > >> >> > >
> > > >> > >> >> > > 2. this is an implementation
> detail, and
> > not
> > > even
> > > >> a
> > > >> > very "user
> > > >> > >> >> > facing" one?
> > > >> > >> >> > > to the best of my understanding
> the vote
> > process
> > > >> is
> > > >> > on proposed
> > > >> > >> >> > > API/behaviour. also - since
> we're willing
> > to go
> > > >> with
> > > >> > strings
> > > >> > >> just
> > > >> > >> >> > serialize
> > > >> > >> >> > > a 0-sized header blob and IIUC
> you dont
> > need any
> > > >> > optionals
> > > >> > >> anymore.
> > > >> > >> >> > >
> > > >> > >> >> > > 3. yes, we can :-)
> > > >> > >> >> > >
> > > >> > >> >> > > On Tue, Feb 14, 2017 at 11:56
> PM, Michael
> > > Pearce <
> > > >> > >> >> > Michael.Pearce@ig.com>
> > > >> > >> >> > > wrote:
> > > >> > >> >> > >
> > > >> > >> >> > > > Hi Jay,
> > > >> > >> >> > > >
> > > >> > >> >> > > > 1) There was some initial
> debate on the
> > value
> > > >> > part, as youll
> > > >> > >> note
> > > >> > >> >> > String,
> > > >> > >> >> > > > String headers were
> discounted early
> > on. The
> > > >> > reason for this
> > > >> > >> is
> > > >> > >> >> > > flexibility
> > > >> > >> >> > > > and keeping in line with the
> > flexibility of
> > > key,
> > > >> > value of the
> > > >> > >> >> > message
> > > >> > >> >> > > > object itself. I don’t think
> it takes
> > away
> > > from
> > > >> an
> > > >> > ecosystem
> > > >> > >> as
> > > >> > >> >> > each
> > > >> > >> >> > > plugin
> > > >> > >> >> > > > will care for their own key,
> this way
> > ints,
> > > >> > booleans , exotic
> > > >> > >> >> > custom
> > > >> > >> >> > > binary
> > > >> > >> >> > > > can all be catered for=.
> > > >> > >> >> > > > a. If you really wanted to
> push for a
> > typed
> > > >> value
> > > >> > interface,
> > > >> > >> I
> > > >> > >> >> > wouldn’t
> > > >> > >> >> > > > want just String values
> supported, but
> > the the
> > > >> > primatives
> > > >> > >> plus
> > > >> > >> >> > string and
> > > >> > >> >> > > > also still keeping the
> ability to have a
> > > binary
> > > >> > for custom
> > > >> > >> >> > binaries that
> > > >> > >> >> > > > some organisations may have.
> > > >> > >> >> > > > i. I have written this slight
> > alternative
> > > here,
> > > >> > >> >> > > https://cwiki.apache.org/
> > > >> > >> >> > > > confluence/display/KAFKA/KIP-
> > > >> > 82+-+Add+Record+Headers+-+Typed
> > > >> > >> >> > > > ii. Essentially the value
> bytes, has a
> > leading
> > > >> > byte overhead.
> > > >> > >> >> > > > 1. This tells you what type
> the value
> > is,
> > > >> before
> > > >> > reading
> > > >> > >> the rest
> > > >> > >> >> > of the
> > > >> > >> >> > > > bytes, allowing
> > serialisation/deserialization
> > > to
> > > >> > and from the
> > > >> > >> >> > primitives,
> > > >> > >> >> > > > string and byte[]. This is
> akin to some
> > other
> > > >> > messaging
> > > >> > >> systems.
> > > >> > >> >> > > > 2) We are making it optional,
> so that
> > for
> > > those
> > > >> > not wanting
> > > >> > >> >> > headers have
> > > >> > >> >> > > 0
> > > >> > >> >> > > > bytes overhead (think of it
> as a feature
> > > flag),
> > > >> I
> > > >> > don’t
> > > >> > >> think this
> > > >> > >> >> > is
> > > >> > >> >> > > > complex, especially if
> comparing to
> > changes
> > > >> > proposed in
> > > >> > >> other kips
> > > >> > >> >> > like
> > > >> > >> >> > > > kip-98.
> > > >> > >> >> > > > a. If you really really don’t
> like
> > this, we
> > > can
> > > >> > drop it, but
> > > >> > >> it
> > > >> > >> >> > would
> > > >> > >> >> > > mean
> > > >> > >> >> > > > buying into 4 bytes extra
> overhead for
> > users
> > > who
> > > >> > do not want
> > > >> > >> to use
> > > >> > >> >> > > headers.
> > > >> > >> >> > > > 3) In the summary yes, it is
> at a higher
> > > level,
> > > >> > but I think
> > > >> > >> this
> > > >> > >> >> > is well
> > > >> > >> >> > > > documented in the proposed
> changes
> > section.
> > > >> > >> >> > > > a. Added getHeaders method to
> > > Producer/Consumer
> > > >> > record (that
> > > >> > >> is it)
> > > >> > >> >> > > > b. We’ve also detailed the
> new Headers
> > class
> > > >> that
> > > >> > this method
> > > >> > >> >> > returns
> > > >> > >> >> > > that
> > > >> > >> >> > > > encapsulates the headers
> protocol and
> > logic.
> > > >> > >> >> > > >
> > > >> > >> >> > > > Best,
> > > >> > >> >> > > > Mike
> > > >> > >> >> > > >
> > > >> > >> >> > > > ==Original questions from the
> vote
> > thread from
> > > >> > Jay.==
> > > >> > >> >> > > >
> > > >> > >> >> > > > Couple of things I think we
> still need
> > to work
> > > >> out:
> > > >> > >> >> > > >
> > > >> > >> >> > > > 1. I think we agree about
> the key,
> > but I
> > > >> think
> > > >> > we haven't
> > > >> > >> >> > talked about
> > > >> > >> >> > > > the value yet. I think if
> our goal
> > is an
> > > open
> > > >> > ecosystem
> > > >> > >> of these
> > > >> > >> >> > > header
> > > >> > >> >> > > > spread across many plugins
> from many
> > > systems
> > > >> we
> > > >> > should
> > > >> > >> consider
> > > >> > >> >> > making
> > > >> > >> >> > > > this
> > > >> > >> >> > > > a string as well so it can
> be
> > printed, set
> > > >> via
> > > >> > a UI, set
> > > >> > >> in
> > > >> > >> >> > config,
> > > >> > >> >> > > etc.
> > > >> > >> >> > > > Basically encouraging
> pluggable
> > > serialization
> > > >> > formats
> > > >> > >> here will
> > > >> > >> >> > lead
> > > >> > >> >> > > to
> > > >> > >> >> > > > a
> > > >> > >> >> > > > bit of a tower of babel.
> > > >> > >> >> > > > 2. This proposal still
> includes a
> > pretty
> > > big
> > > >> > change to our
> > > >> > >> >> > > serialization
> > > >> > >> >> > > > and protocol definition
> layer.
> > Essentially
> > > >> it is
> > > >> > >> introducing an
> > > >> > >> >> > > optional
> > > >> > >> >> > > > type, where the format is
> data
> > dependent. I
> > > >> > think this is
> > > >> > >> >> > actually a
> > > >> > >> >> > > big
> > > >> > >> >> > > > change though it doesn't
> seem like
> > it. It
> > > >> means
> > > >> > you can no
> > > >> > >> >> > longer
> > > >> > >> >> > > > specify
> > > >> > >> >> > > > this type with our type
> definition
> > DSL, and
> > > >> > likewise it
> > > >> > >> requires
> > > >> > >> >> > > custom
> > > >> > >> >> > > > handling in client libs.
> This isn't
> > a huge
> > > >> > thing, since
> > > >> > >> the
> > > >> > >> >> > Record
> > > >> > >> >> > > > definition is custom
> anyway, but I
> > think
> > > this
> > > >> > kind of
> > > >> > >> protocol
> > > >> > >> >> > > > inconsistency is very
> non-desirable
> > and
> > > ties
> > > >> > you to
> > > >> > >> hand-coding
> > > >> > >> >> > > things.
> > > >> > >> >> > > > I
> > > >> > >> >> > > > think the type should
> instead by [Key
> > > Value]
> > > >> in
> > > >> > our BNF,
> > > >> > >> where
> > > >> > >> >> > key and
> > > >> > >> >> > > > value are both short
> strings as used
> > > >> elsewhere.
> > > >> > This
> > > >> > >> brings it
> > > >> > >> >> > in line
> > > >> > >> >> > > > with
> > > >> > >> >> > > > the rest of the protocol.
> > > >> > >> >> > > > 3. Could we get more
> specific about
> > the
> > > exact
> > > >> > Java API
> > > >> > >> change to
> > > >> > >> >> > > > ProducerRecord,
> ConsumerRecord,
> > Record,
> > > etc?
> > > >> > >> >> > > >
> > > >> > >> >> > > > -Jay
> > > >> > >> >> > > >
> > > >> > >> >> > >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > The information contained in this
> email is
> > strictly
> > > >> > confidential and
> > > >> > >> for
> > > >> > >> >> > the use of the addressee only, unless
> otherwise
> > > >> indicated.
> > > >> > If you
> > > >> > >> are not
> > > >> > >> >> > the intended recipient, please do not
> read,
> > copy, use
> > > or
> > > >> > disclose to
> > > >> > >> others
> > > >> > >> >> > this message or any attachment.
> Please also
> > notify the
> > > >> > sender by
> > > >> > >> replying
> > > >> > >> >> > to this email or by telephone
> (+44(020 7896
> > 0011) and
> > > >> then
> > > >> > delete
> > > >> > >> the email
> > > >> > >> >> > and any copies of it. Opinions,
> conclusion
> > (etc) that
> > > do
> > > >> > not relate
> > > >> > >> to the
> > > >> > >> >> > official business of this company
> shall be
> > understood
> > > as
> > > >> > neither
> > > >> > >> given nor
> > > >> > >> >> > endorsed by it. IG is a trading name
> of IG
> > Markets
> > > >> Limited
> > > >> > (a company
> > > >> > >> >> > registered in England and Wales,
> company number
> > > >> 04008957)
> > > >> > and IG
> > > >> > >> Index
> > > >> > >> >> > Limited (a company registered in
> England and
> > Wales,
> > > >> > company number
> > > >> > >> >> > 01190902). Registered address at
> Cannon Bridge
> > House,
> > > 25
> > > >> > Dowgate
> > > >> > >> Hill,
> > > >> > >> >> > London EC4R 2YA. Both IG Markets
> Limited
> > (register
> > > >> number
> > > >> > 195355)
> > > >> > >> and IG
> > > >> > >> >> > Index Limited (register number
> 114059) are
> > authorised
> > > >> and
> > > >> > regulated
> > > >> > >> by the
> > > >> > >> >> > Financial Conduct Authority.
> > > >> > >> >> >
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> --
> > > >> > >> Nacho - Ignacio Solis - isolis@igso.net
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > The information contained in this email is strictly
> > confidential and
> > > for
> > > >> > the use of the addressee only, unless otherwise
> indicated. If
> > you are
> > > >> not
> > > >> > the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > >> others
> > > >> > this message or any attachment. Please also notify
> the sender by
> > > >> replying
> > > >> > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > >> email
> > > >> > and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > >> the
> > > >> > official business of this company shall be understood
> as
> > neither given
> > > >> nor
> > > >> > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > >> > registered in England and Wales, company number
> 04008957) and
> > IG Index
> > > >> > Limited (a company registered in England and Wales,
> company
> > number
> > > >> > 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill,
> > > >> > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and
> > > IG
> > > >> > Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > >> the
> > > >> > Financial Conduct Authority.
> > > >> >
> > > >> The information contained in this email is strictly
> confidential
> > and for
> > > >> the use of the addressee only, unless otherwise
> indicated. If you
> > are
> > > not
> > > >> the intended recipient, please do not read, copy, use
> or disclose
> > to
> > > others
> > > >> this message or any attachment. Please also notify the
> sender by
> > > replying
> > > >> to this email or by telephone (+44(020 7896 0011) and
> then delete
> > the
> > > email
> > > >> and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > the
> > > >> official business of this company shall be understood
> as neither
> > given
> > > nor
> > > >> endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > >> registered in England and Wales, company number
> 04008957) and IG
> > Index
> > > >> Limited (a company registered in England and Wales,
> company number
> > > >> 01190902). Registered address at Cannon Bridge House,
> 25 Dowgate
> > Hill,
> > > >> London EC4R 2YA. Both IG Markets Limited (register
> number 195355)
> > and IG
> > > >> Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > the
> > > >> Financial Conduct Authority.
> > > >>
> > > >
> > > >
> > >
> > The information contained in this email is strictly
> confidential and
> > for the use of the addressee only, unless otherwise indicated.
> If you are
> > not the intended recipient, please do not read, copy, use or
> disclose to
> > others this message or any attachment. Please also notify the
> sender by
> > replying to this email or by telephone (+44(020 7896 0011) and
> then delete
> > the email and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to the official business of this company shall be
> understood as
> > neither given nor endorsed by it. IG is a trading name of IG
> Markets
> > Limited (a company registered in England and Wales, company
> number
> > 04008957) and IG Index Limited (a company registered in England
> and Wales,
> > company number 01190902). Registered address at Cannon Bridge
> House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and IG Index Limited (register number 114059) are
> authorised and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
> >
>
>
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
>
> That’s exactly what we’re doing the headers are a slice of bytes, which
> then gets parsed later if needed, or can be parsed right away, the headers
> is part of the protocol, so can still be validated if wanted.
> If you had a header count then you would have to go through each header
> key and value length value to work out how much to skip to get to say the
> value or any future component in the message after the headers. Having it
> as a byte[] with length value makes this a lot easier to skip.
So the broker will parse the headers and validate them. Good. The only
reason remaining that I can see to leave the headers as a byte array is to
make it easier for the client to skip past them. Are we sure this is not
premature optimization? Are there any performance results which show that
this is worthwhile?
What’s the issue with exposing a method getHeaders on the producer/consumer
> record? It doesn’t break anything. We don’t need any special version.
See my previous explanation. What I am trying to resist is the headers
becoming a general application-level facility. The primary use case as far
as I can tell is middleware, which is the use case that the interceptors
are providing.
Current batch consumer model and consumer interceptors don’t work where
> headers need to be acted on at per message level at time of processing,
> very case is APM (the core one), where the header value is used to continue
> tracing.
I still don't understand the point about batching. The consumer records are
exposed as a batch in the consumer interceptor, but you can still iterate
them individually. It is no different for the consumer API itself.
-Jason
On Fri, Feb 17, 2017 at 12:48 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Re:
>
> “ The point about creation of maps seems orthogonal. We can still
> represent
> the headers as a slice of bytes until the time it is accessed.”
>
> That’s exactly what we’re doing the headers are a slice of bytes, which
> then gets parsed later if needed, or can be parsed right away, the headers
> is part of the protocol, so can still be validated if wanted.
>
> If you had a header count then you would have to go through each header
> key and value length value to work out how much to skip to get to say the
> value or any future component in the message after the headers. Having it
> as a byte[] with length value makes this a lot easier to skip.
>
>
> On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com> wrote:
>
> What’s the issue with exposing a method getHeaders on the
> producer/consumer record? It doesn’t break anything. We don’t need any
> special version.
>
> Current batch consumer model and consumer interceptors don’t work
> where headers need to be acted on at per message level at time of
> processing, very case is APM (the core one), where the header value is used
> to continue tracing. JMS/HTTP etc all expose these, without issues. I would
> NOT want to lock this down to only be usable accessible via interceptors,
> as we’d fail on one of the main goals.
>
> Regards
> Mike
>
>
>
>
> On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> The point about creation of maps seems orthogonal. We can still
> represent
> the headers as a slice of bytes until the time it is accessed.
>
>
> > Yes exactly we have access to the records thus why the header
> should be
> > accessible via it and not hidden for only interceptors to access.
>
>
> As explained above, the point is to make the intended usage clear.
> Applications should continue to rely on the key/value fields to
> serialize
> their own headers, and it would be more ideal if we can avoid
> leaking
> third-party headers into applications. This is difficult to do
> with the
> current interceptors because they share the record objects with
> the common
> API. What I had in mind is something like an extension of the
> current
> interceptors which exposed a different object (e.g.
> `RecordAndHeaders`).
> The challenge is for MM-like use cases. Let me see if I can come
> up with a
> concrete proposal for that problem.
>
> -Jason
>
>
>
> On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> wrote:
>
> > I am happy to move the definition of the header into the message
> body, but
> > would cause us not to lazy initialise/parse the headers, as
> obviously, we
> > would have to traverse these reading the message.
> >
> > This was actually one of Jay’s requests:
> >
> > “ 2. I think we should think about creating the Map lazily to
> avoid
> > parsing out all the headers into little objects. HashMaps
> themselves
> > are
> > kind of expensive and the consumer is very perf sensitive so
> and making
> > gazillions of hashmaps that may or may not get used is
> probably a bad
> > idea.”
> >
> >
> >
> >
> >
> > On 17/02/2017, 19:44, "Michael Pearce" <Mi...@ig.com>
> wrote:
> >
> > Yes exactly we have access to the records thus why the
> header should
> > be accessible via it and not hidden for only interceptors to
> access.
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Magnus Edenhill <ma...@edenhill.se>
> > Sent: Friday, February 17, 2017 7:34:49 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >
> > Big +1 on VarInts.
> > CPUs are fast, memory is slow.
> >
> > I agree with Jason that we'll want to continue verifying
> messages,
> > including their headers, so while I appreciate the idea of
> the opaque
> > header blob it won't be useful in practice.
> >
> > /Magnus
> >
> > 2017-02-17 10:41 GMT-08:00 Jason Gustafson <
> jason@confluent.io>:
> >
> > > Sorry, my mistake. The consumer interceptor is per batch,
> though I'm
> > not
> > > sure that's an actual limitation since you still have
> access to the
> > > individual records.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> > jason@confluent.io>
> > > wrote:
> > >
> > > > Re headers as byte array and future use by broker. This
> doesn't
> > take away
> > > >> from that at all. Nor makes it difficult at all in my
> opinion.
> > > >
> > > >
> > > > Yeah, I didn't say it was difficult, only awkward. You
> wouldn't
> > write the
> > > > schema that way if you were planning to use it on the
> brokers from
> > the
> > > > beginning. Note also that one of the benefits of letting
> the broker
> > > > understand headers is that it can validate that they are
> properly
> > > > formatted. If cost is the only concern, we should
> confirm its
> > impact
> > > > through performance testing.
> > > >
> > > > One of the key use cases requires access on consume at
> per
> > event/message
> > > >> level at the point that message is being processed, as
> such the
> > batch
> > > >> interceptors and batch consume api isn't suitable. It
> needs to be
> > at the
> > > >> record level.
> > > >
> > > >
> > > > I'm not sure I understand the point about batching.
> Interceptors
> > are
> > > > applied per-message, right?
> > > >
> > > > My intent on interceptors is to keep the usage of headers
> > well-defined so
> > > > that they don't start leaking unnecessarily into
> applications. My
> > guess
> > > is
> > > > that it's probably inevitable, but isolating it in the
> > interceptors would
> > > > at least give people a second thought before deciding to
> use it.
> > The main
> > > > challenge in my mind is figuring out how an MM use case
> would
> > work. It
> > > > would be more cumbersome to replicate headers through an
> > interceptor,
> > > > though arguably MM should be working at a lower level
> anyway.
> > > >
> > > > -Jason
> > > >
> > > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > >> Re headers available on the record va interceptors only
> > > >>
> > > >> One of the key use cases requires access on consume at
> per
> > event/message
> > > >> level at the point that message is being processed, as
> such the
> > batch
> > > >> interceptors and batch consume api isn't suitable. It
> needs to be
> > at the
> > > >> record level.
> > > >>
> > > >> This anyhow is similar to jms/http/amqp where headers
> are
> > available to
> > > >> consuming applications.
> > > >>
> > > >> Re headers as byte array and future use by broker. This
> doesn't
> > take
> > > away
> > > >> from that at all. Nor makes it difficult at all in my
> opinion.
> > > >>
> > > >>
> > > >>
> > > >> Sent using OWA for iPhone
> > > >> ________________________________________
> > > >> From: Jason Gustafson <ja...@confluent.io>
> > > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > > >> To: dev@kafka.apache.org
> > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >>
> > > >> >
> > > >> > Would you be proposing in KIP-98 to convert the other
> message
> > int’s
> > > (key
> > > >> > length, value length) also to varint to keep it
> uniform.
> > > >> > Also I assume there will be a static or helper method
> made to
> > > write/read
> > > >> > these in the client and server.
> > > >>
> > > >>
> > > >> Yes, that is what we are proposing, so using varints
> for headers
> > would
> > > be
> > > >> consistent with the rest of the message. We have used
> static
> > helper
> > > >> methods
> > > >> in our prototype implementation.
> > > >>
> > > >> The cost of parsing, we want to parse/interpret the
> headers
> > lazily (this
> > > >> is
> > > >> > a key point brought up earlier in discussions)
> > > >>
> > > >>
> > > >> I'm a bit skeptical of this. Has anyone done the
> performance
> > testing? I
> > > >> can
> > > >> probably implement it and test it if no one else has. I
> was also
> > under
> > > the
> > > >> impression that there may be use cases down the road
> where the
> > broker
> > > >> would
> > > >> need to interpret headers. That wouldn't be off the
> table in the
> > future
> > > if
> > > >> it's represented as bytes, but it would be quite a bit
> more
> > awkward,
> > > >> right?
> > > >>
> > > >> By the way, one question I have been wondering about. My
> > understanding
> > > is
> > > >> that headers are primarily for use cases where a
> third-party
> > components
> > > >> wants to enrich messages without needing to understand
> or modify
> > the
> > > >> schema
> > > >> of the message key and value. For the applications
> which directly
> > > produce
> > > >> and consume the messages and control the key/value
> schema
> > directly, it
> > > >> seems we would rather have them implement headers
> directly in
> > their own
> > > >> schema. Supposing for the sake of argument that it was
> possible,
> > my
> > > >> question is whether it be sufficient to expose the
> headers in the
> > > >> interceptor API and not in the common API?
> > > >>
> > > >> -Jason
> > > >>
> > > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > >> wrote:
> > > >>
> > > >> > On the point of varInts
> > > >> >
> > > >> > Would you be proposing in KIP-98 to convert the other
> message
> > int’s
> > > (key
> > > >> > length, value length) also to varint to keep it
> uniform.
> > > >> > Also I assume there will be a static or helper method
> made to
> > > write/read
> > > >> > these in the client and server.
> > > >> >
> > > >> > Cheers
> > > >> > Mike
> > > >> >
> > > >> >
> > > >> >
> > > >> > On 17/02/2017, 11:22, "Michael Pearce" <
> Michael.Pearce@ig.com>
> > wrote:
> > > >> >
> > > >> > On the point re: headers in the message protocol
> being a
> > byte
> > > array
> > > >> > and not a count of elements followed by the elements.
> Again
> > this was
> > > >> > discussed/argued previously.
> > > >> >
> > > >> > It was agreed on for a few reasons some of which
> you have
> > > obviously
> > > >> > picked up on:
> > > >> >
> > > >> > Broker is able to pass it through opaquely
> > > >> > The cost of parsing, we want to parse/interpret
> the headers
> > lazily
> > > >> > (this is a key point brought up earlier in
> discussions)
> > > >> > Headers can be copied from consumer record to
> producer
> > record (aka
> > > >> > mirror makers etc) without parsing if no changes are
> being made
> > or
> > > being
> > > >> > looked at.
> > > >> > Keeps the broker agnostic to the format
> > > >> > You need an int32 either for the byte size of the
> headers,
> > or for
> > > >> the
> > > >> > count of elements, so overheads are the same, but
> with going
> > with an
> > > >> opaque
> > > >> > byte array has the above advantages.
> > > >> >
> > > >> > Cheers
> > > >> > Mike
> > > >> >
> > > >> >
> > > >> > On 17/02/2017, 02:50, "Jason Gustafson" <
> jason@confluent.io
> > >
> > > wrote:
> > > >> >
> > > >> > Sorry, should have noted that the performance
> testing
> > was done
> > > >> > using the
> > > >> > producer performance tool shipped with Kafka.
> > > >> >
> > > >> > -Jason
> > > >> >
> > > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason
> Gustafson <
> > > >> > jason@confluent.io> wrote:
> > > >> >
> > > >> > > Hey Nacho,
> > > >> > >
> > > >> > > I've compared performance of our KIP-98
> > implementation with
> > > >> and
> > > >> > without
> > > >> > > varints. For messages around 128 bytes, we
> see an
> > increase
> > > in
> > > >> > throughput of
> > > >> > > about 30% using the default configuration
> settings.
> > At 256
> > > >> > bytes, the
> > > >> > > increase is around 16%. Obviously the
> performance
> > converge
> > > as
> > > >> > messages get
> > > >> > > larger, but it seems well worth the cost.
> Note that
> > we are
> > > >> also
> > > >> > seeing a
> > > >> > > substantial performance increase against
> trunk
> > primarily
> > > >> because
> > > >> > of the
> > > >> > > much more efficient packing that varints
> provide us.
> > > Anything
> > > >> > adding to
> > > >> > > message overhead, such as record headers,
> would only
> > > increase
> > > >> > the relative
> > > >> > > difference. (Of course take these numbers
> with a
> > grain of
> > > salt
> > > >> > since I have
> > > >> > > only used the default settings with both
> the producer
> > and
> > > >> broker
> > > >> > on my
> > > >> > > local machine. We intend to provide more
> extensive
> > > performance
> > > >> > details as
> > > >> > > part of the work for KIP-98.)
> > > >> > >
> > > >> > > The implementation we are using is from
> protobuf (
> > > >> > > https://developers.google.com/
> > > protocol-buffers/docs/encoding
> > > >> ),
> > > >> > which is
> > > >> > > also used in HBase. It is trivial to
> implement and as
> > far
> > > as I
> > > >> > know doesn't
> > > >> > > suffer from the aliasing problem you are
> describing. I
> > > checked
> > > >> > with Magnus
> > > >> > > (the author of librdkafka) and he agreed
> that the
> > savings
> > > >> seemed
> > > >> > worth the
> > > >> > > cost of implementation.
> > > >> > >
> > > >> > > -Jason
> > > >> > >
> > > >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio
> Solis <
> > > >> isolis@igso.net>
> > > >> > wrote:
> > > >> > >
> > > >> > >> -VarInts
> > > >> > >>
> > > >> > >> I'm one of the people (if not the most)
> opposed to
> > VarInts.
> > > >> > VarInts
> > > >> > >> have a place, but this is not it. (We
> had a large
> > > >> discussion
> > > >> > about
> > > >> > >> them at the beginning of KIP-82 time)
> > > >> > >>
> > > >> > >> If anybody has real life performance
> numbers of
> > VarInts
> > > >> > improving
> > > >> > >> things or significantly reducing resources
> I would
> > like to
> > > >> know
> > > >> > what
> > > >> > >> that case may be. Yes, you can save some
> bytes here
> > and
> > > >> there,
> > > >> > but
> > > >> > >> this is probably insignificant to the
> overall system
> > > behavior
> > > >> > and
> > > >> > >> storage requirements. -- I say this with
> respect to
> > using
> > > >> > VarInts in
> > > >> > >> the protocol itself, not as part of the
> data.
> > > >> > >>
> > > >> > >> VarInts require you to parse the Int
> before using it
> > and
> > > >> > depending on
> > > >> > >> the encoding they can suffer from aliasing
> (multiple
> > > >> > representations
> > > >> > >> for the same value).
> > > >> > >>
> > > >> > >> Why add complexity?
> > > >> > >>
> > > >> > >> Nacho
> > > >> > >>
> > > >> > >>
> > > >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin
> McCabe <
> > > >> > cmccabe@apache.org>
> > > >> > >> wrote:
> > > >> > >> > +1 for varints here-- it would save
> quite a bit of
> > space.
> > > >> > They are
> > > >> > >> > pretty quick to implement as well.
> > > >> > >> >
> > > >> > >> > I think it makes sense for values to be
> byte
> > arrays.
> > > Users
> > > >> > might want
> > > >> > >> > to attach arbitrary payloads; they
> shouldn't be
> > forced to
> > > >> > serialize
> > > >> > >> > everything to Java strings.
> > > >> > >> >
> > > >> > >> > best,
> > > >> > >> > Colin
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason
> Gustafson
> > wrote:
> > > >> > >> >> Hey Michael,
> > > >> > >> >>
> > > >> > >> >> Hmm, I guess the point of representing
> it as
> > bytes is to
> > > >> > allow the
> > > >> > >> broker
> > > >> > >> >> to pass it through opaquely? Is the
> cost of
> > parsing
> > > them a
> > > >> > concern, or
> > > >> > >> >> are
> > > >> > >> >> we simply trying to ensure that the
> broker stays
> > > agnostic
> > > >> to
> > > >> > the
> > > >> > >> format?
> > > >> > >> >>
> > > >> > >> >> On varints, I think adding support for
> them makes
> > less
> > > >> sense
> > > >> > for an
> > > >> > >> >> isolated use case, but as part of a
> more holistic
> > change
> > > >> > (such as what
> > > >> > >> we
> > > >> > >> >> have proposed in KIP-98), I think they
> are
> > justifiable.
> > > If
> > > >> > we add them,
> > > >> > >> >> then the need to use attributes becomes
> quite a
> > bit
> > > >> weaker,
> > > >> > right? The
> > > >> > >> >> other thing I find slightly odd is the
> fact that
> > null
> > > >> > headers has no
> > > >> > >> >> actual
> > > >> > >> >> semantic meaning for the message
> (unlike null
> > keys and
> > > >> > values). It is
> > > >> > >> >> just
> > > >> > >> >> a space optimization. It seems a bit
> better to
> > always
> > > use
> > > >> > size 0 to
> > > >> > >> >> indicate having no headers.
> > > >> > >> >>
> > > >> > >> >> Overall, the main point is ensuring
> that the
> > message
> > > >> schema
> > > >> > remains
> > > >> > >> >> consistent, either within the larger
> protocol, or
> > at a
> > > >> > minimum within
> > > >> > >> the
> > > >> > >> >> message itself.
> > > >> > >> >>
> > > >> > >> >> -Jason
> > > >> > >> >>
> > > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM,
> Michael Pearce <
> > > >> > Michael.Pearce@ig.com
> > > >> > >> >
> > > >> > >> >> wrote:
> > > >> > >> >>
> > > >> > >> >> > Hi Jason,
> > > >> > >> >> >
> > > >> > >> >> > On point 1) in the message protocol
> the headers
> > are
> > > >> simply
> > > >> > a byte
> > > >> > >> array,
> > > >> > >> >> > as like the key or value, this is to
> clearly
> > demarcate
> > > >> the
> > > >> > header in
> > > >> > >> the
> > > >> > >> >> > core message. Then the header byte
> array in the
> > core
> > > >> > message is an
> > > >> > >> array of
> > > >> > >> >> > key, value pairs. This is what it is
> denoting.
> > > >> > >> >> >
> > > >> > >> >> > Then this would be I guess in the
> given
> > notation:
> > > >> > >> >> >
> > > >> > >> >> > Headers => [KeyLength, Key,
> ValueLength, Value]
> > > >> > >> >> > KeyLength => int32
> <-----------------NEW
> > size of
> > > the
> > > >> > byte[] of
> > > >> > >> the
> > > >> > >> >> > serialised key value
> > > >> > >> >> > Key => bytes
> <---------------------- NEW
> > > serialised
> > > >> > string (UTF8)
> > > >> > >> >> > bytes of the header key
> > > >> > >> >> > ValueLength => int32
> <-------------- NEW
> > size of
> > > the
> > > >> > byte[] of
> > > >> > >> the
> > > >> > >> >> > serialised header value
> > > >> > >> >> > Value => bytes
> <-------------------- NEW
> > > serialised
> > > >> > form of the
> > > >> > >> header
> > > >> > >> >> > value
> > > >> > >> >> >
> > > >> > >> >> > The key length and value length is
> matching the
> > way
> > > the
> > > >> > protocol is
> > > >> > >> >> > defined in the core message currently.
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > On point 2)
> > > >> > >> >> > Var sized ints, this was discussed
> much earlier
> > on, in
> > > >> > fact I had
> > > >> > >> >> > suggested it myself (with Hadoop
> references),
> > the
> > > >> > complexity of this
> > > >> > >> >> > compared to having a simpler protocol
> was
> > argued and
> > > >> > agreed it
> > > >> > >> wasn’t worth
> > > >> > >> >> > the complexity as all other clients
> in other
> > languages
> > > >> > would need to
> > > >> > >> ensure
> > > >> > >> >> > theyre using the right var size
> algorithm, as
> > there
> > > is a
> > > >> > few.
> > > >> > >> >> >
> > > >> > >> >> > On point 3)
> > > >> > >> >> > We did the attributes, optional
> approach as
> > originally
> > > >> > there was
> > > >> > >> marked
> > > >> > >> >> > concern that headers would cause a
> message size
> > > overhead
> > > >> > for others,
> > > >> > >> who
> > > >> > >> >> > don’t want them. As such this is the
> clean
> > solution to
> > > >> > achieve that.
> > > >> > >> If
> > > >> > >> >> > that no longer holds, and we don’t
> care that we
> > add
> > > >> 4bytes
> > > >> > overhead,
> > > >> > >> then
> > > >> > >> >> > im happy to remove.
> > > >> > >> >> >
> > > >> > >> >> > I’m personally in favour of keeping
> the message
> > as
> > > small
> > > >> > as possible
> > > >> > >> so
> > > >> > >> >> > people don’t get shocks in perf and
> throughputs
> > dues
> > > to
> > > >> > message size,
> > > >> > >> >> > unless they actively use the feature,
> as such I
> > do
> > > >> prefer
> > > >> > the
> > > >> > >> attribute bit
> > > >> > >> >> > wise feature flag approach myself.
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > On 16/02/2017, 05:40, "Jason
> Gustafson" <
> > > >> > jason@confluent.io> wrote:
> > > >> > >> >> >
> > > >> > >> >> > We have proposed a few
> significant changes
> > to the
> > > >> > message format
> > > >> > >> in
> > > >> > >> >> > KIP-98
> > > >> > >> >> > which now seems likely to pass
> (perhaps
> > with some
> > > >> > iterations on
> > > >> > >> >> > implementation details). It would
> be good
> > to try
> > > and
> > > >> > coordinate
> > > >> > >> the
> > > >> > >> >> > changes
> > > >> > >> >> > in both of the proposals to make
> sure they
> > are
> > > >> > consistent and
> > > >> > >> >> > compatible.
> > > >> > >> >> >
> > > >> > >> >> > I think using the attributes to
> indicate
> > null
> > > >> headers
> > > >> > is a
> > > >> > >> reasonable
> > > >> > >> >> > approach. We have proposed to do
> the same
> > thing
> > > for
> > > >> > the message
> > > >> > >> key and
> > > >> > >> >> > value. That said, I sympathize
> with Jay's
> > > argument.
> > > >> > Having
> > > >> > >> multiple
> > > >> > >> >> > ways to
> > > >> > >> >> > specify a null value increases
> the overall
> > > >> complexity
> > > >> > of the
> > > >> > >> protocol.
> > > >> > >> >> > You
> > > >> > >> >> > can see this just from the fact
> that you
> > need the
> > > >> > extra verbiage
> > > >> > >> in the
> > > >> > >> >> > protocol specification in this
> KIP and in
> > KIP-98
> > > to
> > > >> > describe the
> > > >> > >> >> > dependence
> > > >> > >> >> > between the fields and the
> attributes. It
> > seems
> > > >> like a
> > > >> > slippery
> > > >> > >> slope
> > > >> > >> >> > if
> > > >> > >> >> > you start allowing different
> request types
> > to
> > > >> > implement the
> > > >> > >> protocol
> > > >> > >> >> > specification differently.
> > > >> > >> >> >
> > > >> > >> >> > You can also argue that the
> messages
> > already are
> > > and
> > > >> > are likely
> > > >> > >> to
> > > >> > >> >> > remain a
> > > >> > >> >> > special case. For example, there
> is
> > currently no
> > > >> > generality in
> > > >> > >> how
> > > >> > >> >> > compressed message sets are
> represented
> > that would
> > > >> be
> > > >> > applicable
> > > >> > >> for
> > > >> > >> >> > other
> > > >> > >> >> > request types. Some might see this
> > divergence as
> > > an
> > > >> > unfortunate
> > > >> > >> >> > protocol
> > > >> > >> >> > deficiency which should be fixed;
> others
> > might see
> > > >> it
> > > >> > as sort of
> > > >> > >> the
> > > >> > >> >> > inevitability of needing to
> optimize where
> > it
> > > counts
> > > >> > most. I'm
> > > >> > >> probably
> > > >> > >> >> > somewhere in between, but I think
> we
> > probably all
> > > >> > share the
> > > >> > >> intuition
> > > >> > >> >> > that
> > > >> > >> >> > the protocol should be kept as
> consistent as
> > > >> possible.
> > > >> > With that
> > > >> > >> in
> > > >> > >> >> > mind,
> > > >> > >> >> > here are a few comments:
> > > >> > >> >> >
> > > >> > >> >> > 1. One thing I found a little odd
> when
> > reading the
> > > >> > current
> > > >> > >> proposal is
> > > >> > >> >> > that
> > > >> > >> >> > the headers are both represented
> as an
> > array of
> > > >> bytes
> > > >> > and as an
> > > >> > >> array
> > > >> > >> >> > of
> > > >> > >> >> > key/value pairs. I'd probably
> suggest
> > something
> > > like
> > > >> > this:
> > > >> > >> >> >
> > > >> > >> >> > Headers => [HeaderKey HeaderValue]
> > > >> > >> >> > HeaderKey => String
> > > >> > >> >> > HeaderValue => Bytes
> > > >> > >> >> >
> > > >> > >> >> > An array in the Kafka protocol is
> > represented as a
> > > >> > 4-byte integer
> > > >> > >> >> > indicating the number of elements
> in the
> > array
> > > >> > followed by the
> > > >> > >> >> > serialization of the elements.
> Unless I'm
> > > >> > misunderstanding, what
> > > >> > >> you
> > > >> > >> >> > have
> > > >> > >> >> > instead is the total size of the
> headers in
> > bytes
> > > >> > followed by the
> > > >> > >> >> > elements.
> > > >> > >> >> > I'm not sure I see any reason for
> this
> > > >> inconsistency.
> > > >> > >> >> >
> > > >> > >> >> > 2. In KIP-98, we've introduced
> > variable-length
> > > >> integer
> > > >> > fields.
> > > >> > >> >> > Effectively,
> > > >> > >> >> > we've enriched (or "complicated"
> as Jay
> > might say
> > > ;)
> > > >> > the protocol
> > > >> > >> >> > specification to include the
> following
> > types:
> > > >> VarInt,
> > > >> > VarLong,
> > > >> > >> >> > UnsignedVarInt and
> UnsignedVarLong.
> > > >> > >> >> >
> > > >> > >> >> > Along with these primitives, we
> could
> > introduce
> > > the
> > > >> > following
> > > >> > >> types:
> > > >> > >> >> >
> > > >> > >> >> > VarSizeArray => NumberOfItems
> Item1 Item2
> > .. ItemN
> > > >> > >> >> > NumberOfItems => UnsignedVarInt
> > > >> > >> >> >
> > > >> > >> >> > VarSizeNullableArray =>
> NumberOfItemsOrNull
> > Item1
> > > >> > Item2 .. ItemN
> > > >> > >> >> > NumberOfItemsOrNull => VarInt
> (-1 means
> > null)
> > > >> > >> >> >
> > > >> > >> >> > And similarly for the `String`
> and `Bytes`
> > types.
> > > >> > These types
> > > >> > >> can save
> > > >> > >> >> > a
> > > >> > >> >> > considerable amount of space in
> this
> > proposal
> > > >> because
> > > >> > they can
> > > >> > >> be used
> > > >> > >> >> > for
> > > >> > >> >> > both the number of headers
> included in the
> > message
> > > >> and
> > > >> > the
> > > >> > >> lengths of
> > > >> > >> >> > the
> > > >> > >> >> > header keys and values. We could
> do this
> > instead:
> > > >> > >> >> >
> > > >> > >> >> > Headers => VarSizeArray[HeaderKey
> > HeaderValue]
> > > >> > >> >> > HeaderKey => VarSizeString
> > > >> > >> >> > HeaderValue => VarSizeBytes
> > > >> > >> >> >
> > > >> > >> >> > Combining the savings from the
> use of
> > variable
> > > >> length
> > > >> > fields, the
> > > >> > >> >> > benefit
> > > >> > >> >> > of using the attributes to
> represent null
> > seems
> > > >> pretty
> > > >> > small.
> > > >> > >> >> >
> > > >> > >> >> > 3. Whichever way we go (whether
> we use the
> > > >> attributes
> > > >> > or not), we
> > > >> > >> >> > should at
> > > >> > >> >> > least be consistent between this
> KIP and
> > KIP-98.
> > > It
> > > >> > would be very
> > > >> > >> >> > strange
> > > >> > >> >> > to have two ways to represent
> null values
> > in the
> > > >> same
> > > >> > schema.
> > > >> > >> Either
> > > >> > >> >> > way is
> > > >> > >> >> > OK with me. I think some
> message-level
> > > optimizations
> > > >> > are
> > > >> > >> justifiable,
> > > >> > >> >> > but
> > > >> > >> >> > the savings here seem minimal (a
> few bytes
> > per
> > > >> > message), so
> > > >> > >> maybe it's
> > > >> > >> >> > not
> > > >> > >> >> > worth the cost of letting the
> message
> > diverge even
> > > >> > further from
> > > >> > >> the
> > > >> > >> >> > rest of
> > > >> > >> >> > the protocol.
> > > >> > >> >> >
> > > >> > >> >> > -Jason
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM,
> radai <
> > > >> > >> radai.rosenblatt@gmail.com>
> > > >> > >> >> > wrote:
> > > >> > >> >> >
> > > >> > >> >> > > I've trimmed the inline
> contents as this
> > mail is
> > > >> > getting too
> > > >> > >> big for
> > > >> > >> >> > the
> > > >> > >> >> > > apache mailing list software to
> deliver
> > :-(
> > > >> > >> >> > >
> > > >> > >> >> > > 1. the important thing for
> > interoperability is
> > > for
> > > >> > different
> > > >> > >> >> > "interested
> > > >> > >> >> > > parties" (plugins, infra
> layers/wrappers,
> > > >> user-code)
> > > >> > to be
> > > >> > >> able to
> > > >> > >> >> > stick
> > > >> > >> >> > > pieces of metadata onto msgs
> without
> > getting in
> > > >> each
> > > >> > other's
> > > >> > >> way. a
> > > >> > >> >> > common
> > > >> > >> >> > > key scheme (Strings, as of the
> time of
> > this
> > > >> > writing?) is all
> > > >> > >> thats
> > > >> > >> >> > required
> > > >> > >> >> > > for that. it is assumed that
> the other end
> > > >> > interested in any
> > > >> > >> such
> > > >> > >> >> > piece of
> > > >> > >> >> > > metadata knows the encoding,
> and byte[]
> > provides
> > > >> for
> > > >> > the most
> > > >> > >> >> > flexibility.
> > > >> > >> >> > > i believe this is the same
> logic behind
> > core
> > > kafka
> > > >> > being
> > > >> > >> >> > byte[]/byte[] -
> > > >> > >> >> > > Strings are more "usable" but
> bytes are
> > flexible
> > > >> and
> > > >> > so were
> > > >> > >> chosen.
> > > >> > >> >> > > Also - core kafka doesnt even
> do that
> > good of a
> > > >> job
> > > >> > on
> > > >> > >> usability of
> > > >> > >> >> > the
> > > >> > >> >> > > payload (example - i have to
> specify the
> > nop
> > > >> byte[]
> > > >> > "decoders"
> > > >> > >> >> > explicitly
> > > >> > >> >> > > in conf), and again sacrificies
> usability
> > for
> > > the
> > > >> > sake of
> > > >> > >> >> > performance (no
> > > >> > >> >> > > convenient single-record
> processing as
> > poll is a
> > > >> > batch, lots of
> > > >> > >> >> > obscure
> > > >> > >> >> > > little config details exposing
> internals
> > of the
> > > >> > batching
> > > >> > >> mechanism,
> > > >> > >> >> > etc)
> > > >> > >> >> > >
> > > >> > >> >> > > this is also why i really
> dislike the
> > idea of a
> > > >> > "type system"
> > > >> > >> for
> > > >> > >> >> > header
> > > >> > >> >> > > values, it further degrades the
> > usability, adds
> > > >> > complexity and
> > > >> > >> will
> > > >> > >> >> > > eventually get in people's way,
> also, it
> > would
> > > be
> > > >> > the 2nd/3rd
> > > >> > >> >> > home-group
> > > >> > >> >> > > serialization mechanism in core
> kafka
> > (counting
> > > 2
> > > >> > iterations
> > > >> > >> of the
> > > >> > >> >> > "type
> > > >> > >> >> > > definition DSL")
> > > >> > >> >> > >
> > > >> > >> >> > > 2. this is an implementation
> detail, and
> > not
> > > even
> > > >> a
> > > >> > very "user
> > > >> > >> >> > facing" one?
> > > >> > >> >> > > to the best of my understanding
> the vote
> > process
> > > >> is
> > > >> > on proposed
> > > >> > >> >> > > API/behaviour. also - since
> we're willing
> > to go
> > > >> with
> > > >> > strings
> > > >> > >> just
> > > >> > >> >> > serialize
> > > >> > >> >> > > a 0-sized header blob and IIUC
> you dont
> > need any
> > > >> > optionals
> > > >> > >> anymore.
> > > >> > >> >> > >
> > > >> > >> >> > > 3. yes, we can :-)
> > > >> > >> >> > >
> > > >> > >> >> > > On Tue, Feb 14, 2017 at 11:56
> PM, Michael
> > > Pearce <
> > > >> > >> >> > Michael.Pearce@ig.com>
> > > >> > >> >> > > wrote:
> > > >> > >> >> > >
> > > >> > >> >> > > > Hi Jay,
> > > >> > >> >> > > >
> > > >> > >> >> > > > 1) There was some initial
> debate on the
> > value
> > > >> > part, as youll
> > > >> > >> note
> > > >> > >> >> > String,
> > > >> > >> >> > > > String headers were
> discounted early
> > on. The
> > > >> > reason for this
> > > >> > >> is
> > > >> > >> >> > > flexibility
> > > >> > >> >> > > > and keeping in line with the
> > flexibility of
> > > key,
> > > >> > value of the
> > > >> > >> >> > message
> > > >> > >> >> > > > object itself. I don’t think
> it takes
> > away
> > > from
> > > >> an
> > > >> > ecosystem
> > > >> > >> as
> > > >> > >> >> > each
> > > >> > >> >> > > plugin
> > > >> > >> >> > > > will care for their own key,
> this way
> > ints,
> > > >> > booleans , exotic
> > > >> > >> >> > custom
> > > >> > >> >> > > binary
> > > >> > >> >> > > > can all be catered for=.
> > > >> > >> >> > > > a. If you really wanted to
> push for a
> > typed
> > > >> value
> > > >> > interface,
> > > >> > >> I
> > > >> > >> >> > wouldn’t
> > > >> > >> >> > > > want just String values
> supported, but
> > the the
> > > >> > primatives
> > > >> > >> plus
> > > >> > >> >> > string and
> > > >> > >> >> > > > also still keeping the
> ability to have a
> > > binary
> > > >> > for custom
> > > >> > >> >> > binaries that
> > > >> > >> >> > > > some organisations may have.
> > > >> > >> >> > > > i. I have written this slight
> > alternative
> > > here,
> > > >> > >> >> > > https://cwiki.apache.org/
> > > >> > >> >> > > > confluence/display/KAFKA/KIP-
> > > >> > 82+-+Add+Record+Headers+-+Typed
> > > >> > >> >> > > > ii. Essentially the value
> bytes, has a
> > leading
> > > >> > byte overhead.
> > > >> > >> >> > > > 1. This tells you what type
> the value
> > is,
> > > >> before
> > > >> > reading
> > > >> > >> the rest
> > > >> > >> >> > of the
> > > >> > >> >> > > > bytes, allowing
> > serialisation/deserialization
> > > to
> > > >> > and from the
> > > >> > >> >> > primitives,
> > > >> > >> >> > > > string and byte[]. This is
> akin to some
> > other
> > > >> > messaging
> > > >> > >> systems.
> > > >> > >> >> > > > 2) We are making it optional,
> so that
> > for
> > > those
> > > >> > not wanting
> > > >> > >> >> > headers have
> > > >> > >> >> > > 0
> > > >> > >> >> > > > bytes overhead (think of it
> as a feature
> > > flag),
> > > >> I
> > > >> > don’t
> > > >> > >> think this
> > > >> > >> >> > is
> > > >> > >> >> > > > complex, especially if
> comparing to
> > changes
> > > >> > proposed in
> > > >> > >> other kips
> > > >> > >> >> > like
> > > >> > >> >> > > > kip-98.
> > > >> > >> >> > > > a. If you really really don’t
> like
> > this, we
> > > can
> > > >> > drop it, but
> > > >> > >> it
> > > >> > >> >> > would
> > > >> > >> >> > > mean
> > > >> > >> >> > > > buying into 4 bytes extra
> overhead for
> > users
> > > who
> > > >> > do not want
> > > >> > >> to use
> > > >> > >> >> > > headers.
> > > >> > >> >> > > > 3) In the summary yes, it is
> at a higher
> > > level,
> > > >> > but I think
> > > >> > >> this
> > > >> > >> >> > is well
> > > >> > >> >> > > > documented in the proposed
> changes
> > section.
> > > >> > >> >> > > > a. Added getHeaders method to
> > > Producer/Consumer
> > > >> > record (that
> > > >> > >> is it)
> > > >> > >> >> > > > b. We’ve also detailed the
> new Headers
> > class
> > > >> that
> > > >> > this method
> > > >> > >> >> > returns
> > > >> > >> >> > > that
> > > >> > >> >> > > > encapsulates the headers
> protocol and
> > logic.
> > > >> > >> >> > > >
> > > >> > >> >> > > > Best,
> > > >> > >> >> > > > Mike
> > > >> > >> >> > > >
> > > >> > >> >> > > > ==Original questions from the
> vote
> > thread from
> > > >> > Jay.==
> > > >> > >> >> > > >
> > > >> > >> >> > > > Couple of things I think we
> still need
> > to work
> > > >> out:
> > > >> > >> >> > > >
> > > >> > >> >> > > > 1. I think we agree about
> the key,
> > but I
> > > >> think
> > > >> > we haven't
> > > >> > >> >> > talked about
> > > >> > >> >> > > > the value yet. I think if
> our goal
> > is an
> > > open
> > > >> > ecosystem
> > > >> > >> of these
> > > >> > >> >> > > header
> > > >> > >> >> > > > spread across many plugins
> from many
> > > systems
> > > >> we
> > > >> > should
> > > >> > >> consider
> > > >> > >> >> > making
> > > >> > >> >> > > > this
> > > >> > >> >> > > > a string as well so it can
> be
> > printed, set
> > > >> via
> > > >> > a UI, set
> > > >> > >> in
> > > >> > >> >> > config,
> > > >> > >> >> > > etc.
> > > >> > >> >> > > > Basically encouraging
> pluggable
> > > serialization
> > > >> > formats
> > > >> > >> here will
> > > >> > >> >> > lead
> > > >> > >> >> > > to
> > > >> > >> >> > > > a
> > > >> > >> >> > > > bit of a tower of babel.
> > > >> > >> >> > > > 2. This proposal still
> includes a
> > pretty
> > > big
> > > >> > change to our
> > > >> > >> >> > > serialization
> > > >> > >> >> > > > and protocol definition
> layer.
> > Essentially
> > > >> it is
> > > >> > >> introducing an
> > > >> > >> >> > > optional
> > > >> > >> >> > > > type, where the format is
> data
> > dependent. I
> > > >> > think this is
> > > >> > >> >> > actually a
> > > >> > >> >> > > big
> > > >> > >> >> > > > change though it doesn't
> seem like
> > it. It
> > > >> means
> > > >> > you can no
> > > >> > >> >> > longer
> > > >> > >> >> > > > specify
> > > >> > >> >> > > > this type with our type
> definition
> > DSL, and
> > > >> > likewise it
> > > >> > >> requires
> > > >> > >> >> > > custom
> > > >> > >> >> > > > handling in client libs.
> This isn't
> > a huge
> > > >> > thing, since
> > > >> > >> the
> > > >> > >> >> > Record
> > > >> > >> >> > > > definition is custom
> anyway, but I
> > think
> > > this
> > > >> > kind of
> > > >> > >> protocol
> > > >> > >> >> > > > inconsistency is very
> non-desirable
> > and
> > > ties
> > > >> > you to
> > > >> > >> hand-coding
> > > >> > >> >> > > things.
> > > >> > >> >> > > > I
> > > >> > >> >> > > > think the type should
> instead by [Key
> > > Value]
> > > >> in
> > > >> > our BNF,
> > > >> > >> where
> > > >> > >> >> > key and
> > > >> > >> >> > > > value are both short
> strings as used
> > > >> elsewhere.
> > > >> > This
> > > >> > >> brings it
> > > >> > >> >> > in line
> > > >> > >> >> > > > with
> > > >> > >> >> > > > the rest of the protocol.
> > > >> > >> >> > > > 3. Could we get more
> specific about
> > the
> > > exact
> > > >> > Java API
> > > >> > >> change to
> > > >> > >> >> > > > ProducerRecord,
> ConsumerRecord,
> > Record,
> > > etc?
> > > >> > >> >> > > >
> > > >> > >> >> > > > -Jay
> > > >> > >> >> > > >
> > > >> > >> >> > >
> > > >> > >> >> >
> > > >> > >> >> >
> > > >> > >> >> > The information contained in this
> email is
> > strictly
> > > >> > confidential and
> > > >> > >> for
> > > >> > >> >> > the use of the addressee only, unless
> otherwise
> > > >> indicated.
> > > >> > If you
> > > >> > >> are not
> > > >> > >> >> > the intended recipient, please do not
> read,
> > copy, use
> > > or
> > > >> > disclose to
> > > >> > >> others
> > > >> > >> >> > this message or any attachment.
> Please also
> > notify the
> > > >> > sender by
> > > >> > >> replying
> > > >> > >> >> > to this email or by telephone
> (+44(020 7896
> > 0011) and
> > > >> then
> > > >> > delete
> > > >> > >> the email
> > > >> > >> >> > and any copies of it. Opinions,
> conclusion
> > (etc) that
> > > do
> > > >> > not relate
> > > >> > >> to the
> > > >> > >> >> > official business of this company
> shall be
> > understood
> > > as
> > > >> > neither
> > > >> > >> given nor
> > > >> > >> >> > endorsed by it. IG is a trading name
> of IG
> > Markets
> > > >> Limited
> > > >> > (a company
> > > >> > >> >> > registered in England and Wales,
> company number
> > > >> 04008957)
> > > >> > and IG
> > > >> > >> Index
> > > >> > >> >> > Limited (a company registered in
> England and
> > Wales,
> > > >> > company number
> > > >> > >> >> > 01190902). Registered address at
> Cannon Bridge
> > House,
> > > 25
> > > >> > Dowgate
> > > >> > >> Hill,
> > > >> > >> >> > London EC4R 2YA. Both IG Markets
> Limited
> > (register
> > > >> number
> > > >> > 195355)
> > > >> > >> and IG
> > > >> > >> >> > Index Limited (register number
> 114059) are
> > authorised
> > > >> and
> > > >> > regulated
> > > >> > >> by the
> > > >> > >> >> > Financial Conduct Authority.
> > > >> > >> >> >
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> --
> > > >> > >> Nacho - Ignacio Solis - isolis@igso.net
> > > >> > >>
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > The information contained in this email is strictly
> > confidential and
> > > for
> > > >> > the use of the addressee only, unless otherwise
> indicated. If
> > you are
> > > >> not
> > > >> > the intended recipient, please do not read, copy, use
> or
> > disclose to
> > > >> others
> > > >> > this message or any attachment. Please also notify
> the sender by
> > > >> replying
> > > >> > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete the
> > > >> email
> > > >> > and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > >> the
> > > >> > official business of this company shall be understood
> as
> > neither given
> > > >> nor
> > > >> > endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > >> > registered in England and Wales, company number
> 04008957) and
> > IG Index
> > > >> > Limited (a company registered in England and Wales,
> company
> > number
> > > >> > 01190902). Registered address at Cannon Bridge House,
> 25
> > Dowgate Hill,
> > > >> > London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and
> > > IG
> > > >> > Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > >> the
> > > >> > Financial Conduct Authority.
> > > >> >
> > > >> The information contained in this email is strictly
> confidential
> > and for
> > > >> the use of the addressee only, unless otherwise
> indicated. If you
> > are
> > > not
> > > >> the intended recipient, please do not read, copy, use
> or disclose
> > to
> > > others
> > > >> this message or any attachment. Please also notify the
> sender by
> > > replying
> > > >> to this email or by telephone (+44(020 7896 0011) and
> then delete
> > the
> > > email
> > > >> and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to
> > > the
> > > >> official business of this company shall be understood
> as neither
> > given
> > > nor
> > > >> endorsed by it. IG is a trading name of IG Markets
> Limited (a
> > company
> > > >> registered in England and Wales, company number
> 04008957) and IG
> > Index
> > > >> Limited (a company registered in England and Wales,
> company number
> > > >> 01190902). Registered address at Cannon Bridge House,
> 25 Dowgate
> > Hill,
> > > >> London EC4R 2YA. Both IG Markets Limited (register
> number 195355)
> > and IG
> > > >> Index Limited (register number 114059) are authorised
> and
> > regulated by
> > > the
> > > >> Financial Conduct Authority.
> > > >>
> > > >
> > > >
> > >
> > The information contained in this email is strictly
> confidential and
> > for the use of the addressee only, unless otherwise indicated.
> If you are
> > not the intended recipient, please do not read, copy, use or
> disclose to
> > others this message or any attachment. Please also notify the
> sender by
> > replying to this email or by telephone (+44(020 7896 0011) and
> then delete
> > the email and any copies of it. Opinions, conclusion (etc) that
> do not
> > relate to the official business of this company shall be
> understood as
> > neither given nor endorsed by it. IG is a trading name of IG
> Markets
> > Limited (a company registered in England and Wales, company
> number
> > 04008957) and IG Index Limited (a company registered in England
> and Wales,
> > company number 01190902). Registered address at Cannon Bridge
> House, 25
> > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register
> number
> > 195355) and IG Index Limited (register number 114059) are
> authorised and
> > regulated by the Financial Conduct Authority.
> >
> >
> >
> >
>
>
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Re:
“ The point about creation of maps seems orthogonal. We can still represent
the headers as a slice of bytes until the time it is accessed.”
That’s exactly what we’re doing the headers are a slice of bytes, which then gets parsed later if needed, or can be parsed right away, the headers is part of the protocol, so can still be validated if wanted.
If you had a header count then you would have to go through each header key and value length value to work out how much to skip to get to say the value or any future component in the message after the headers. Having it as a byte[] with length value makes this a lot easier to skip.
On 17/02/2017, 20:37, "Michael Pearce" <Mi...@ig.com> wrote:
What’s the issue with exposing a method getHeaders on the producer/consumer record? It doesn’t break anything. We don’t need any special version.
Current batch consumer model and consumer interceptors don’t work where headers need to be acted on at per message level at time of processing, very case is APM (the core one), where the header value is used to continue tracing. JMS/HTTP etc all expose these, without issues. I would NOT want to lock this down to only be usable accessible via interceptors, as we’d fail on one of the main goals.
Regards
Mike
On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io> wrote:
The point about creation of maps seems orthogonal. We can still represent
the headers as a slice of bytes until the time it is accessed.
> Yes exactly we have access to the records thus why the header should be
> accessible via it and not hidden for only interceptors to access.
As explained above, the point is to make the intended usage clear.
Applications should continue to rely on the key/value fields to serialize
their own headers, and it would be more ideal if we can avoid leaking
third-party headers into applications. This is difficult to do with the
current interceptors because they share the record objects with the common
API. What I had in mind is something like an extension of the current
interceptors which exposed a different object (e.g. `RecordAndHeaders`).
The challenge is for MM-like use cases. Let me see if I can come up with a
concrete proposal for that problem.
-Jason
On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <Mi...@ig.com>
wrote:
> I am happy to move the definition of the header into the message body, but
> would cause us not to lazy initialise/parse the headers, as obviously, we
> would have to traverse these reading the message.
>
> This was actually one of Jay’s requests:
>
> “ 2. I think we should think about creating the Map lazily to avoid
> parsing out all the headers into little objects. HashMaps themselves
> are
> kind of expensive and the consumer is very perf sensitive so and making
> gazillions of hashmaps that may or may not get used is probably a bad
> idea.”
>
>
>
>
>
> On 17/02/2017, 19:44, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Yes exactly we have access to the records thus why the header should
> be accessible via it and not hidden for only interceptors to access.
>
> Sent using OWA for iPhone
> ________________________________________
> From: Magnus Edenhill <ma...@edenhill.se>
> Sent: Friday, February 17, 2017 7:34:49 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Big +1 on VarInts.
> CPUs are fast, memory is slow.
>
> I agree with Jason that we'll want to continue verifying messages,
> including their headers, so while I appreciate the idea of the opaque
> header blob it won't be useful in practice.
>
> /Magnus
>
> 2017-02-17 10:41 GMT-08:00 Jason Gustafson <ja...@confluent.io>:
>
> > Sorry, my mistake. The consumer interceptor is per batch, though I'm
> not
> > sure that's an actual limitation since you still have access to the
> > individual records.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> jason@confluent.io>
> > wrote:
> >
> > > Re headers as byte array and future use by broker. This doesn't
> take away
> > >> from that at all. Nor makes it difficult at all in my opinion.
> > >
> > >
> > > Yeah, I didn't say it was difficult, only awkward. You wouldn't
> write the
> > > schema that way if you were planning to use it on the brokers from
> the
> > > beginning. Note also that one of the benefits of letting the broker
> > > understand headers is that it can validate that they are properly
> > > formatted. If cost is the only concern, we should confirm its
> impact
> > > through performance testing.
> > >
> > > One of the key use cases requires access on consume at per
> event/message
> > >> level at the point that message is being processed, as such the
> batch
> > >> interceptors and batch consume api isn't suitable. It needs to be
> at the
> > >> record level.
> > >
> > >
> > > I'm not sure I understand the point about batching. Interceptors
> are
> > > applied per-message, right?
> > >
> > > My intent on interceptors is to keep the usage of headers
> well-defined so
> > > that they don't start leaking unnecessarily into applications. My
> guess
> > is
> > > that it's probably inevitable, but isolating it in the
> interceptors would
> > > at least give people a second thought before deciding to use it.
> The main
> > > challenge in my mind is figuring out how an MM use case would
> work. It
> > > would be more cumbersome to replicate headers through an
> interceptor,
> > > though arguably MM should be working at a lower level anyway.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > >> Re headers available on the record va interceptors only
> > >>
> > >> One of the key use cases requires access on consume at per
> event/message
> > >> level at the point that message is being processed, as such the
> batch
> > >> interceptors and batch consume api isn't suitable. It needs to be
> at the
> > >> record level.
> > >>
> > >> This anyhow is similar to jms/http/amqp where headers are
> available to
> > >> consuming applications.
> > >>
> > >> Re headers as byte array and future use by broker. This doesn't
> take
> > away
> > >> from that at all. Nor makes it difficult at all in my opinion.
> > >>
> > >>
> > >>
> > >> Sent using OWA for iPhone
> > >> ________________________________________
> > >> From: Jason Gustafson <ja...@confluent.io>
> > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >>
> > >> >
> > >> > Would you be proposing in KIP-98 to convert the other message
> int’s
> > (key
> > >> > length, value length) also to varint to keep it uniform.
> > >> > Also I assume there will be a static or helper method made to
> > write/read
> > >> > these in the client and server.
> > >>
> > >>
> > >> Yes, that is what we are proposing, so using varints for headers
> would
> > be
> > >> consistent with the rest of the message. We have used static
> helper
> > >> methods
> > >> in our prototype implementation.
> > >>
> > >> The cost of parsing, we want to parse/interpret the headers
> lazily (this
> > >> is
> > >> > a key point brought up earlier in discussions)
> > >>
> > >>
> > >> I'm a bit skeptical of this. Has anyone done the performance
> testing? I
> > >> can
> > >> probably implement it and test it if no one else has. I was also
> under
> > the
> > >> impression that there may be use cases down the road where the
> broker
> > >> would
> > >> need to interpret headers. That wouldn't be off the table in the
> future
> > if
> > >> it's represented as bytes, but it would be quite a bit more
> awkward,
> > >> right?
> > >>
> > >> By the way, one question I have been wondering about. My
> understanding
> > is
> > >> that headers are primarily for use cases where a third-party
> components
> > >> wants to enrich messages without needing to understand or modify
> the
> > >> schema
> > >> of the message key and value. For the applications which directly
> > produce
> > >> and consume the messages and control the key/value schema
> directly, it
> > >> seems we would rather have them implement headers directly in
> their own
> > >> schema. Supposing for the sake of argument that it was possible,
> my
> > >> question is whether it be sufficient to expose the headers in the
> > >> interceptor API and not in the common API?
> > >>
> > >> -Jason
> > >>
> > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > >> wrote:
> > >>
> > >> > On the point of varInts
> > >> >
> > >> > Would you be proposing in KIP-98 to convert the other message
> int’s
> > (key
> > >> > length, value length) also to varint to keep it uniform.
> > >> > Also I assume there will be a static or helper method made to
> > write/read
> > >> > these in the client and server.
> > >> >
> > >> > Cheers
> > >> > Mike
> > >> >
> > >> >
> > >> >
> > >> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com>
> wrote:
> > >> >
> > >> > On the point re: headers in the message protocol being a
> byte
> > array
> > >> > and not a count of elements followed by the elements. Again
> this was
> > >> > discussed/argued previously.
> > >> >
> > >> > It was agreed on for a few reasons some of which you have
> > obviously
> > >> > picked up on:
> > >> >
> > >> > Broker is able to pass it through opaquely
> > >> > The cost of parsing, we want to parse/interpret the headers
> lazily
> > >> > (this is a key point brought up earlier in discussions)
> > >> > Headers can be copied from consumer record to producer
> record (aka
> > >> > mirror makers etc) without parsing if no changes are being made
> or
> > being
> > >> > looked at.
> > >> > Keeps the broker agnostic to the format
> > >> > You need an int32 either for the byte size of the headers,
> or for
> > >> the
> > >> > count of elements, so overheads are the same, but with going
> with an
> > >> opaque
> > >> > byte array has the above advantages.
> > >> >
> > >> > Cheers
> > >> > Mike
> > >> >
> > >> >
> > >> > On 17/02/2017, 02:50, "Jason Gustafson" <jason@confluent.io
> >
> > wrote:
> > >> >
> > >> > Sorry, should have noted that the performance testing
> was done
> > >> > using the
> > >> > producer performance tool shipped with Kafka.
> > >> >
> > >> > -Jason
> > >> >
> > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> > >> > jason@confluent.io> wrote:
> > >> >
> > >> > > Hey Nacho,
> > >> > >
> > >> > > I've compared performance of our KIP-98
> implementation with
> > >> and
> > >> > without
> > >> > > varints. For messages around 128 bytes, we see an
> increase
> > in
> > >> > throughput of
> > >> > > about 30% using the default configuration settings.
> At 256
> > >> > bytes, the
> > >> > > increase is around 16%. Obviously the performance
> converge
> > as
> > >> > messages get
> > >> > > larger, but it seems well worth the cost. Note that
> we are
> > >> also
> > >> > seeing a
> > >> > > substantial performance increase against trunk
> primarily
> > >> because
> > >> > of the
> > >> > > much more efficient packing that varints provide us.
> > Anything
> > >> > adding to
> > >> > > message overhead, such as record headers, would only
> > increase
> > >> > the relative
> > >> > > difference. (Of course take these numbers with a
> grain of
> > salt
> > >> > since I have
> > >> > > only used the default settings with both the producer
> and
> > >> broker
> > >> > on my
> > >> > > local machine. We intend to provide more extensive
> > performance
> > >> > details as
> > >> > > part of the work for KIP-98.)
> > >> > >
> > >> > > The implementation we are using is from protobuf (
> > >> > > https://developers.google.com/
> > protocol-buffers/docs/encoding
> > >> ),
> > >> > which is
> > >> > > also used in HBase. It is trivial to implement and as
> far
> > as I
> > >> > know doesn't
> > >> > > suffer from the aliasing problem you are describing. I
> > checked
> > >> > with Magnus
> > >> > > (the author of librdkafka) and he agreed that the
> savings
> > >> seemed
> > >> > worth the
> > >> > > cost of implementation.
> > >> > >
> > >> > > -Jason
> > >> > >
> > >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
> > >> isolis@igso.net>
> > >> > wrote:
> > >> > >
> > >> > >> -VarInts
> > >> > >>
> > >> > >> I'm one of the people (if not the most) opposed to
> VarInts.
> > >> > VarInts
> > >> > >> have a place, but this is not it. (We had a large
> > >> discussion
> > >> > about
> > >> > >> them at the beginning of KIP-82 time)
> > >> > >>
> > >> > >> If anybody has real life performance numbers of
> VarInts
> > >> > improving
> > >> > >> things or significantly reducing resources I would
> like to
> > >> know
> > >> > what
> > >> > >> that case may be. Yes, you can save some bytes here
> and
> > >> there,
> > >> > but
> > >> > >> this is probably insignificant to the overall system
> > behavior
> > >> > and
> > >> > >> storage requirements. -- I say this with respect to
> using
> > >> > VarInts in
> > >> > >> the protocol itself, not as part of the data.
> > >> > >>
> > >> > >> VarInts require you to parse the Int before using it
> and
> > >> > depending on
> > >> > >> the encoding they can suffer from aliasing (multiple
> > >> > representations
> > >> > >> for the same value).
> > >> > >>
> > >> > >> Why add complexity?
> > >> > >>
> > >> > >> Nacho
> > >> > >>
> > >> > >>
> > >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> > >> > cmccabe@apache.org>
> > >> > >> wrote:
> > >> > >> > +1 for varints here-- it would save quite a bit of
> space.
> > >> > They are
> > >> > >> > pretty quick to implement as well.
> > >> > >> >
> > >> > >> > I think it makes sense for values to be byte
> arrays.
> > Users
> > >> > might want
> > >> > >> > to attach arbitrary payloads; they shouldn't be
> forced to
> > >> > serialize
> > >> > >> > everything to Java strings.
> > >> > >> >
> > >> > >> > best,
> > >> > >> > Colin
> > >> > >> >
> > >> > >> >
> > >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson
> wrote:
> > >> > >> >> Hey Michael,
> > >> > >> >>
> > >> > >> >> Hmm, I guess the point of representing it as
> bytes is to
> > >> > allow the
> > >> > >> broker
> > >> > >> >> to pass it through opaquely? Is the cost of
> parsing
> > them a
> > >> > concern, or
> > >> > >> >> are
> > >> > >> >> we simply trying to ensure that the broker stays
> > agnostic
> > >> to
> > >> > the
> > >> > >> format?
> > >> > >> >>
> > >> > >> >> On varints, I think adding support for them makes
> less
> > >> sense
> > >> > for an
> > >> > >> >> isolated use case, but as part of a more holistic
> change
> > >> > (such as what
> > >> > >> we
> > >> > >> >> have proposed in KIP-98), I think they are
> justifiable.
> > If
> > >> > we add them,
> > >> > >> >> then the need to use attributes becomes quite a
> bit
> > >> weaker,
> > >> > right? The
> > >> > >> >> other thing I find slightly odd is the fact that
> null
> > >> > headers has no
> > >> > >> >> actual
> > >> > >> >> semantic meaning for the message (unlike null
> keys and
> > >> > values). It is
> > >> > >> >> just
> > >> > >> >> a space optimization. It seems a bit better to
> always
> > use
> > >> > size 0 to
> > >> > >> >> indicate having no headers.
> > >> > >> >>
> > >> > >> >> Overall, the main point is ensuring that the
> message
> > >> schema
> > >> > remains
> > >> > >> >> consistent, either within the larger protocol, or
> at a
> > >> > minimum within
> > >> > >> the
> > >> > >> >> message itself.
> > >> > >> >>
> > >> > >> >> -Jason
> > >> > >> >>
> > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> > >> > Michael.Pearce@ig.com
> > >> > >> >
> > >> > >> >> wrote:
> > >> > >> >>
> > >> > >> >> > Hi Jason,
> > >> > >> >> >
> > >> > >> >> > On point 1) in the message protocol the headers
> are
> > >> simply
> > >> > a byte
> > >> > >> array,
> > >> > >> >> > as like the key or value, this is to clearly
> demarcate
> > >> the
> > >> > header in
> > >> > >> the
> > >> > >> >> > core message. Then the header byte array in the
> core
> > >> > message is an
> > >> > >> array of
> > >> > >> >> > key, value pairs. This is what it is denoting.
> > >> > >> >> >
> > >> > >> >> > Then this would be I guess in the given
> notation:
> > >> > >> >> >
> > >> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> > >> > >> >> > KeyLength => int32 <-----------------NEW
> size of
> > the
> > >> > byte[] of
> > >> > >> the
> > >> > >> >> > serialised key value
> > >> > >> >> > Key => bytes <---------------------- NEW
> > serialised
> > >> > string (UTF8)
> > >> > >> >> > bytes of the header key
> > >> > >> >> > ValueLength => int32 <-------------- NEW
> size of
> > the
> > >> > byte[] of
> > >> > >> the
> > >> > >> >> > serialised header value
> > >> > >> >> > Value => bytes <-------------------- NEW
> > serialised
> > >> > form of the
> > >> > >> header
> > >> > >> >> > value
> > >> > >> >> >
> > >> > >> >> > The key length and value length is matching the
> way
> > the
> > >> > protocol is
> > >> > >> >> > defined in the core message currently.
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On point 2)
> > >> > >> >> > Var sized ints, this was discussed much earlier
> on, in
> > >> > fact I had
> > >> > >> >> > suggested it myself (with Hadoop references),
> the
> > >> > complexity of this
> > >> > >> >> > compared to having a simpler protocol was
> argued and
> > >> > agreed it
> > >> > >> wasn’t worth
> > >> > >> >> > the complexity as all other clients in other
> languages
> > >> > would need to
> > >> > >> ensure
> > >> > >> >> > theyre using the right var size algorithm, as
> there
> > is a
> > >> > few.
> > >> > >> >> >
> > >> > >> >> > On point 3)
> > >> > >> >> > We did the attributes, optional approach as
> originally
> > >> > there was
> > >> > >> marked
> > >> > >> >> > concern that headers would cause a message size
> > overhead
> > >> > for others,
> > >> > >> who
> > >> > >> >> > don’t want them. As such this is the clean
> solution to
> > >> > achieve that.
> > >> > >> If
> > >> > >> >> > that no longer holds, and we don’t care that we
> add
> > >> 4bytes
> > >> > overhead,
> > >> > >> then
> > >> > >> >> > im happy to remove.
> > >> > >> >> >
> > >> > >> >> > I’m personally in favour of keeping the message
> as
> > small
> > >> > as possible
> > >> > >> so
> > >> > >> >> > people don’t get shocks in perf and throughputs
> dues
> > to
> > >> > message size,
> > >> > >> >> > unless they actively use the feature, as such I
> do
> > >> prefer
> > >> > the
> > >> > >> attribute bit
> > >> > >> >> > wise feature flag approach myself.
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> > >> > jason@confluent.io> wrote:
> > >> > >> >> >
> > >> > >> >> > We have proposed a few significant changes
> to the
> > >> > message format
> > >> > >> in
> > >> > >> >> > KIP-98
> > >> > >> >> > which now seems likely to pass (perhaps
> with some
> > >> > iterations on
> > >> > >> >> > implementation details). It would be good
> to try
> > and
> > >> > coordinate
> > >> > >> the
> > >> > >> >> > changes
> > >> > >> >> > in both of the proposals to make sure they
> are
> > >> > consistent and
> > >> > >> >> > compatible.
> > >> > >> >> >
> > >> > >> >> > I think using the attributes to indicate
> null
> > >> headers
> > >> > is a
> > >> > >> reasonable
> > >> > >> >> > approach. We have proposed to do the same
> thing
> > for
> > >> > the message
> > >> > >> key and
> > >> > >> >> > value. That said, I sympathize with Jay's
> > argument.
> > >> > Having
> > >> > >> multiple
> > >> > >> >> > ways to
> > >> > >> >> > specify a null value increases the overall
> > >> complexity
> > >> > of the
> > >> > >> protocol.
> > >> > >> >> > You
> > >> > >> >> > can see this just from the fact that you
> need the
> > >> > extra verbiage
> > >> > >> in the
> > >> > >> >> > protocol specification in this KIP and in
> KIP-98
> > to
> > >> > describe the
> > >> > >> >> > dependence
> > >> > >> >> > between the fields and the attributes. It
> seems
> > >> like a
> > >> > slippery
> > >> > >> slope
> > >> > >> >> > if
> > >> > >> >> > you start allowing different request types
> to
> > >> > implement the
> > >> > >> protocol
> > >> > >> >> > specification differently.
> > >> > >> >> >
> > >> > >> >> > You can also argue that the messages
> already are
> > and
> > >> > are likely
> > >> > >> to
> > >> > >> >> > remain a
> > >> > >> >> > special case. For example, there is
> currently no
> > >> > generality in
> > >> > >> how
> > >> > >> >> > compressed message sets are represented
> that would
> > >> be
> > >> > applicable
> > >> > >> for
> > >> > >> >> > other
> > >> > >> >> > request types. Some might see this
> divergence as
> > an
> > >> > unfortunate
> > >> > >> >> > protocol
> > >> > >> >> > deficiency which should be fixed; others
> might see
> > >> it
> > >> > as sort of
> > >> > >> the
> > >> > >> >> > inevitability of needing to optimize where
> it
> > counts
> > >> > most. I'm
> > >> > >> probably
> > >> > >> >> > somewhere in between, but I think we
> probably all
> > >> > share the
> > >> > >> intuition
> > >> > >> >> > that
> > >> > >> >> > the protocol should be kept as consistent as
> > >> possible.
> > >> > With that
> > >> > >> in
> > >> > >> >> > mind,
> > >> > >> >> > here are a few comments:
> > >> > >> >> >
> > >> > >> >> > 1. One thing I found a little odd when
> reading the
> > >> > current
> > >> > >> proposal is
> > >> > >> >> > that
> > >> > >> >> > the headers are both represented as an
> array of
> > >> bytes
> > >> > and as an
> > >> > >> array
> > >> > >> >> > of
> > >> > >> >> > key/value pairs. I'd probably suggest
> something
> > like
> > >> > this:
> > >> > >> >> >
> > >> > >> >> > Headers => [HeaderKey HeaderValue]
> > >> > >> >> > HeaderKey => String
> > >> > >> >> > HeaderValue => Bytes
> > >> > >> >> >
> > >> > >> >> > An array in the Kafka protocol is
> represented as a
> > >> > 4-byte integer
> > >> > >> >> > indicating the number of elements in the
> array
> > >> > followed by the
> > >> > >> >> > serialization of the elements. Unless I'm
> > >> > misunderstanding, what
> > >> > >> you
> > >> > >> >> > have
> > >> > >> >> > instead is the total size of the headers in
> bytes
> > >> > followed by the
> > >> > >> >> > elements.
> > >> > >> >> > I'm not sure I see any reason for this
> > >> inconsistency.
> > >> > >> >> >
> > >> > >> >> > 2. In KIP-98, we've introduced
> variable-length
> > >> integer
> > >> > fields.
> > >> > >> >> > Effectively,
> > >> > >> >> > we've enriched (or "complicated" as Jay
> might say
> > ;)
> > >> > the protocol
> > >> > >> >> > specification to include the following
> types:
> > >> VarInt,
> > >> > VarLong,
> > >> > >> >> > UnsignedVarInt and UnsignedVarLong.
> > >> > >> >> >
> > >> > >> >> > Along with these primitives, we could
> introduce
> > the
> > >> > following
> > >> > >> types:
> > >> > >> >> >
> > >> > >> >> > VarSizeArray => NumberOfItems Item1 Item2
> .. ItemN
> > >> > >> >> > NumberOfItems => UnsignedVarInt
> > >> > >> >> >
> > >> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull
> Item1
> > >> > Item2 .. ItemN
> > >> > >> >> > NumberOfItemsOrNull => VarInt (-1 means
> null)
> > >> > >> >> >
> > >> > >> >> > And similarly for the `String` and `Bytes`
> types.
> > >> > These types
> > >> > >> can save
> > >> > >> >> > a
> > >> > >> >> > considerable amount of space in this
> proposal
> > >> because
> > >> > they can
> > >> > >> be used
> > >> > >> >> > for
> > >> > >> >> > both the number of headers included in the
> message
> > >> and
> > >> > the
> > >> > >> lengths of
> > >> > >> >> > the
> > >> > >> >> > header keys and values. We could do this
> instead:
> > >> > >> >> >
> > >> > >> >> > Headers => VarSizeArray[HeaderKey
> HeaderValue]
> > >> > >> >> > HeaderKey => VarSizeString
> > >> > >> >> > HeaderValue => VarSizeBytes
> > >> > >> >> >
> > >> > >> >> > Combining the savings from the use of
> variable
> > >> length
> > >> > fields, the
> > >> > >> >> > benefit
> > >> > >> >> > of using the attributes to represent null
> seems
> > >> pretty
> > >> > small.
> > >> > >> >> >
> > >> > >> >> > 3. Whichever way we go (whether we use the
> > >> attributes
> > >> > or not), we
> > >> > >> >> > should at
> > >> > >> >> > least be consistent between this KIP and
> KIP-98.
> > It
> > >> > would be very
> > >> > >> >> > strange
> > >> > >> >> > to have two ways to represent null values
> in the
> > >> same
> > >> > schema.
> > >> > >> Either
> > >> > >> >> > way is
> > >> > >> >> > OK with me. I think some message-level
> > optimizations
> > >> > are
> > >> > >> justifiable,
> > >> > >> >> > but
> > >> > >> >> > the savings here seem minimal (a few bytes
> per
> > >> > message), so
> > >> > >> maybe it's
> > >> > >> >> > not
> > >> > >> >> > worth the cost of letting the message
> diverge even
> > >> > further from
> > >> > >> the
> > >> > >> >> > rest of
> > >> > >> >> > the protocol.
> > >> > >> >> >
> > >> > >> >> > -Jason
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> > >> > >> radai.rosenblatt@gmail.com>
> > >> > >> >> > wrote:
> > >> > >> >> >
> > >> > >> >> > > I've trimmed the inline contents as this
> mail is
> > >> > getting too
> > >> > >> big for
> > >> > >> >> > the
> > >> > >> >> > > apache mailing list software to deliver
> :-(
> > >> > >> >> > >
> > >> > >> >> > > 1. the important thing for
> interoperability is
> > for
> > >> > different
> > >> > >> >> > "interested
> > >> > >> >> > > parties" (plugins, infra layers/wrappers,
> > >> user-code)
> > >> > to be
> > >> > >> able to
> > >> > >> >> > stick
> > >> > >> >> > > pieces of metadata onto msgs without
> getting in
> > >> each
> > >> > other's
> > >> > >> way. a
> > >> > >> >> > common
> > >> > >> >> > > key scheme (Strings, as of the time of
> this
> > >> > writing?) is all
> > >> > >> thats
> > >> > >> >> > required
> > >> > >> >> > > for that. it is assumed that the other end
> > >> > interested in any
> > >> > >> such
> > >> > >> >> > piece of
> > >> > >> >> > > metadata knows the encoding, and byte[]
> provides
> > >> for
> > >> > the most
> > >> > >> >> > flexibility.
> > >> > >> >> > > i believe this is the same logic behind
> core
> > kafka
> > >> > being
> > >> > >> >> > byte[]/byte[] -
> > >> > >> >> > > Strings are more "usable" but bytes are
> flexible
> > >> and
> > >> > so were
> > >> > >> chosen.
> > >> > >> >> > > Also - core kafka doesnt even do that
> good of a
> > >> job
> > >> > on
> > >> > >> usability of
> > >> > >> >> > the
> > >> > >> >> > > payload (example - i have to specify the
> nop
> > >> byte[]
> > >> > "decoders"
> > >> > >> >> > explicitly
> > >> > >> >> > > in conf), and again sacrificies usability
> for
> > the
> > >> > sake of
> > >> > >> >> > performance (no
> > >> > >> >> > > convenient single-record processing as
> poll is a
> > >> > batch, lots of
> > >> > >> >> > obscure
> > >> > >> >> > > little config details exposing internals
> of the
> > >> > batching
> > >> > >> mechanism,
> > >> > >> >> > etc)
> > >> > >> >> > >
> > >> > >> >> > > this is also why i really dislike the
> idea of a
> > >> > "type system"
> > >> > >> for
> > >> > >> >> > header
> > >> > >> >> > > values, it further degrades the
> usability, adds
> > >> > complexity and
> > >> > >> will
> > >> > >> >> > > eventually get in people's way, also, it
> would
> > be
> > >> > the 2nd/3rd
> > >> > >> >> > home-group
> > >> > >> >> > > serialization mechanism in core kafka
> (counting
> > 2
> > >> > iterations
> > >> > >> of the
> > >> > >> >> > "type
> > >> > >> >> > > definition DSL")
> > >> > >> >> > >
> > >> > >> >> > > 2. this is an implementation detail, and
> not
> > even
> > >> a
> > >> > very "user
> > >> > >> >> > facing" one?
> > >> > >> >> > > to the best of my understanding the vote
> process
> > >> is
> > >> > on proposed
> > >> > >> >> > > API/behaviour. also - since we're willing
> to go
> > >> with
> > >> > strings
> > >> > >> just
> > >> > >> >> > serialize
> > >> > >> >> > > a 0-sized header blob and IIUC you dont
> need any
> > >> > optionals
> > >> > >> anymore.
> > >> > >> >> > >
> > >> > >> >> > > 3. yes, we can :-)
> > >> > >> >> > >
> > >> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael
> > Pearce <
> > >> > >> >> > Michael.Pearce@ig.com>
> > >> > >> >> > > wrote:
> > >> > >> >> > >
> > >> > >> >> > > > Hi Jay,
> > >> > >> >> > > >
> > >> > >> >> > > > 1) There was some initial debate on the
> value
> > >> > part, as youll
> > >> > >> note
> > >> > >> >> > String,
> > >> > >> >> > > > String headers were discounted early
> on. The
> > >> > reason for this
> > >> > >> is
> > >> > >> >> > > flexibility
> > >> > >> >> > > > and keeping in line with the
> flexibility of
> > key,
> > >> > value of the
> > >> > >> >> > message
> > >> > >> >> > > > object itself. I don’t think it takes
> away
> > from
> > >> an
> > >> > ecosystem
> > >> > >> as
> > >> > >> >> > each
> > >> > >> >> > > plugin
> > >> > >> >> > > > will care for their own key, this way
> ints,
> > >> > booleans , exotic
> > >> > >> >> > custom
> > >> > >> >> > > binary
> > >> > >> >> > > > can all be catered for=.
> > >> > >> >> > > > a. If you really wanted to push for a
> typed
> > >> value
> > >> > interface,
> > >> > >> I
> > >> > >> >> > wouldn’t
> > >> > >> >> > > > want just String values supported, but
> the the
> > >> > primatives
> > >> > >> plus
> > >> > >> >> > string and
> > >> > >> >> > > > also still keeping the ability to have a
> > binary
> > >> > for custom
> > >> > >> >> > binaries that
> > >> > >> >> > > > some organisations may have.
> > >> > >> >> > > > i. I have written this slight
> alternative
> > here,
> > >> > >> >> > > https://cwiki.apache.org/
> > >> > >> >> > > > confluence/display/KAFKA/KIP-
> > >> > 82+-+Add+Record+Headers+-+Typed
> > >> > >> >> > > > ii. Essentially the value bytes, has a
> leading
> > >> > byte overhead.
> > >> > >> >> > > > 1. This tells you what type the value
> is,
> > >> before
> > >> > reading
> > >> > >> the rest
> > >> > >> >> > of the
> > >> > >> >> > > > bytes, allowing
> serialisation/deserialization
> > to
> > >> > and from the
> > >> > >> >> > primitives,
> > >> > >> >> > > > string and byte[]. This is akin to some
> other
> > >> > messaging
> > >> > >> systems.
> > >> > >> >> > > > 2) We are making it optional, so that
> for
> > those
> > >> > not wanting
> > >> > >> >> > headers have
> > >> > >> >> > > 0
> > >> > >> >> > > > bytes overhead (think of it as a feature
> > flag),
> > >> I
> > >> > don’t
> > >> > >> think this
> > >> > >> >> > is
> > >> > >> >> > > > complex, especially if comparing to
> changes
> > >> > proposed in
> > >> > >> other kips
> > >> > >> >> > like
> > >> > >> >> > > > kip-98.
> > >> > >> >> > > > a. If you really really don’t like
> this, we
> > can
> > >> > drop it, but
> > >> > >> it
> > >> > >> >> > would
> > >> > >> >> > > mean
> > >> > >> >> > > > buying into 4 bytes extra overhead for
> users
> > who
> > >> > do not want
> > >> > >> to use
> > >> > >> >> > > headers.
> > >> > >> >> > > > 3) In the summary yes, it is at a higher
> > level,
> > >> > but I think
> > >> > >> this
> > >> > >> >> > is well
> > >> > >> >> > > > documented in the proposed changes
> section.
> > >> > >> >> > > > a. Added getHeaders method to
> > Producer/Consumer
> > >> > record (that
> > >> > >> is it)
> > >> > >> >> > > > b. We’ve also detailed the new Headers
> class
> > >> that
> > >> > this method
> > >> > >> >> > returns
> > >> > >> >> > > that
> > >> > >> >> > > > encapsulates the headers protocol and
> logic.
> > >> > >> >> > > >
> > >> > >> >> > > > Best,
> > >> > >> >> > > > Mike
> > >> > >> >> > > >
> > >> > >> >> > > > ==Original questions from the vote
> thread from
> > >> > Jay.==
> > >> > >> >> > > >
> > >> > >> >> > > > Couple of things I think we still need
> to work
> > >> out:
> > >> > >> >> > > >
> > >> > >> >> > > > 1. I think we agree about the key,
> but I
> > >> think
> > >> > we haven't
> > >> > >> >> > talked about
> > >> > >> >> > > > the value yet. I think if our goal
> is an
> > open
> > >> > ecosystem
> > >> > >> of these
> > >> > >> >> > > header
> > >> > >> >> > > > spread across many plugins from many
> > systems
> > >> we
> > >> > should
> > >> > >> consider
> > >> > >> >> > making
> > >> > >> >> > > > this
> > >> > >> >> > > > a string as well so it can be
> printed, set
> > >> via
> > >> > a UI, set
> > >> > >> in
> > >> > >> >> > config,
> > >> > >> >> > > etc.
> > >> > >> >> > > > Basically encouraging pluggable
> > serialization
> > >> > formats
> > >> > >> here will
> > >> > >> >> > lead
> > >> > >> >> > > to
> > >> > >> >> > > > a
> > >> > >> >> > > > bit of a tower of babel.
> > >> > >> >> > > > 2. This proposal still includes a
> pretty
> > big
> > >> > change to our
> > >> > >> >> > > serialization
> > >> > >> >> > > > and protocol definition layer.
> Essentially
> > >> it is
> > >> > >> introducing an
> > >> > >> >> > > optional
> > >> > >> >> > > > type, where the format is data
> dependent. I
> > >> > think this is
> > >> > >> >> > actually a
> > >> > >> >> > > big
> > >> > >> >> > > > change though it doesn't seem like
> it. It
> > >> means
> > >> > you can no
> > >> > >> >> > longer
> > >> > >> >> > > > specify
> > >> > >> >> > > > this type with our type definition
> DSL, and
> > >> > likewise it
> > >> > >> requires
> > >> > >> >> > > custom
> > >> > >> >> > > > handling in client libs. This isn't
> a huge
> > >> > thing, since
> > >> > >> the
> > >> > >> >> > Record
> > >> > >> >> > > > definition is custom anyway, but I
> think
> > this
> > >> > kind of
> > >> > >> protocol
> > >> > >> >> > > > inconsistency is very non-desirable
> and
> > ties
> > >> > you to
> > >> > >> hand-coding
> > >> > >> >> > > things.
> > >> > >> >> > > > I
> > >> > >> >> > > > think the type should instead by [Key
> > Value]
> > >> in
> > >> > our BNF,
> > >> > >> where
> > >> > >> >> > key and
> > >> > >> >> > > > value are both short strings as used
> > >> elsewhere.
> > >> > This
> > >> > >> brings it
> > >> > >> >> > in line
> > >> > >> >> > > > with
> > >> > >> >> > > > the rest of the protocol.
> > >> > >> >> > > > 3. Could we get more specific about
> the
> > exact
> > >> > Java API
> > >> > >> change to
> > >> > >> >> > > > ProducerRecord, ConsumerRecord,
> Record,
> > etc?
> > >> > >> >> > > >
> > >> > >> >> > > > -Jay
> > >> > >> >> > > >
> > >> > >> >> > >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > The information contained in this email is
> strictly
> > >> > confidential and
> > >> > >> for
> > >> > >> >> > the use of the addressee only, unless otherwise
> > >> indicated.
> > >> > If you
> > >> > >> are not
> > >> > >> >> > the intended recipient, please do not read,
> copy, use
> > or
> > >> > disclose to
> > >> > >> others
> > >> > >> >> > this message or any attachment. Please also
> notify the
> > >> > sender by
> > >> > >> replying
> > >> > >> >> > to this email or by telephone (+44(020 7896
> 0011) and
> > >> then
> > >> > delete
> > >> > >> the email
> > >> > >> >> > and any copies of it. Opinions, conclusion
> (etc) that
> > do
> > >> > not relate
> > >> > >> to the
> > >> > >> >> > official business of this company shall be
> understood
> > as
> > >> > neither
> > >> > >> given nor
> > >> > >> >> > endorsed by it. IG is a trading name of IG
> Markets
> > >> Limited
> > >> > (a company
> > >> > >> >> > registered in England and Wales, company number
> > >> 04008957)
> > >> > and IG
> > >> > >> Index
> > >> > >> >> > Limited (a company registered in England and
> Wales,
> > >> > company number
> > >> > >> >> > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > >> > Dowgate
> > >> > >> Hill,
> > >> > >> >> > London EC4R 2YA. Both IG Markets Limited
> (register
> > >> number
> > >> > 195355)
> > >> > >> and IG
> > >> > >> >> > Index Limited (register number 114059) are
> authorised
> > >> and
> > >> > regulated
> > >> > >> by the
> > >> > >> >> > Financial Conduct Authority.
> > >> > >> >> >
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Nacho - Ignacio Solis - isolis@igso.net
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > The information contained in this email is strictly
> confidential and
> > for
> > >> > the use of the addressee only, unless otherwise indicated. If
> you are
> > >> not
> > >> > the intended recipient, please do not read, copy, use or
> disclose to
> > >> others
> > >> > this message or any attachment. Please also notify the sender by
> > >> replying
> > >> > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > >> email
> > >> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > >> the
> > >> > official business of this company shall be understood as
> neither given
> > >> nor
> > >> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > >> > registered in England and Wales, company number 04008957) and
> IG Index
> > >> > Limited (a company registered in England and Wales, company
> number
> > >> > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > >> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
> > IG
> > >> > Index Limited (register number 114059) are authorised and
> regulated by
> > >> the
> > >> > Financial Conduct Authority.
> > >> >
> > >> The information contained in this email is strictly confidential
> and for
> > >> the use of the addressee only, unless otherwise indicated. If you
> are
> > not
> > >> the intended recipient, please do not read, copy, use or disclose
> to
> > others
> > >> this message or any attachment. Please also notify the sender by
> > replying
> > >> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > email
> > >> and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > the
> > >> official business of this company shall be understood as neither
> given
> > nor
> > >> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > >> registered in England and Wales, company number 04008957) and IG
> Index
> > >> Limited (a company registered in England and Wales, company number
> > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > >> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > >> Index Limited (register number 114059) are authorised and
> regulated by
> > the
> > >> Financial Conduct Authority.
> > >>
> > >
> > >
> >
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
What’s the issue with exposing a method getHeaders on the producer/consumer record? It doesn’t break anything. We don’t need any special version.
Current batch consumer model and consumer interceptors don’t work where headers need to be acted on at per message level at time of processing, very case is APM (the core one), where the header value is used to continue tracing. JMS/HTTP etc all expose these, without issues. I would NOT want to lock this down to only be usable accessible via interceptors, as we’d fail on one of the main goals.
Regards
Mike
On 17/02/2017, 20:21, "Jason Gustafson" <ja...@confluent.io> wrote:
The point about creation of maps seems orthogonal. We can still represent
the headers as a slice of bytes until the time it is accessed.
> Yes exactly we have access to the records thus why the header should be
> accessible via it and not hidden for only interceptors to access.
As explained above, the point is to make the intended usage clear.
Applications should continue to rely on the key/value fields to serialize
their own headers, and it would be more ideal if we can avoid leaking
third-party headers into applications. This is difficult to do with the
current interceptors because they share the record objects with the common
API. What I had in mind is something like an extension of the current
interceptors which exposed a different object (e.g. `RecordAndHeaders`).
The challenge is for MM-like use cases. Let me see if I can come up with a
concrete proposal for that problem.
-Jason
On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <Mi...@ig.com>
wrote:
> I am happy to move the definition of the header into the message body, but
> would cause us not to lazy initialise/parse the headers, as obviously, we
> would have to traverse these reading the message.
>
> This was actually one of Jay’s requests:
>
> “ 2. I think we should think about creating the Map lazily to avoid
> parsing out all the headers into little objects. HashMaps themselves
> are
> kind of expensive and the consumer is very perf sensitive so and making
> gazillions of hashmaps that may or may not get used is probably a bad
> idea.”
>
>
>
>
>
> On 17/02/2017, 19:44, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Yes exactly we have access to the records thus why the header should
> be accessible via it and not hidden for only interceptors to access.
>
> Sent using OWA for iPhone
> ________________________________________
> From: Magnus Edenhill <ma...@edenhill.se>
> Sent: Friday, February 17, 2017 7:34:49 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Big +1 on VarInts.
> CPUs are fast, memory is slow.
>
> I agree with Jason that we'll want to continue verifying messages,
> including their headers, so while I appreciate the idea of the opaque
> header blob it won't be useful in practice.
>
> /Magnus
>
> 2017-02-17 10:41 GMT-08:00 Jason Gustafson <ja...@confluent.io>:
>
> > Sorry, my mistake. The consumer interceptor is per batch, though I'm
> not
> > sure that's an actual limitation since you still have access to the
> > individual records.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> jason@confluent.io>
> > wrote:
> >
> > > Re headers as byte array and future use by broker. This doesn't
> take away
> > >> from that at all. Nor makes it difficult at all in my opinion.
> > >
> > >
> > > Yeah, I didn't say it was difficult, only awkward. You wouldn't
> write the
> > > schema that way if you were planning to use it on the brokers from
> the
> > > beginning. Note also that one of the benefits of letting the broker
> > > understand headers is that it can validate that they are properly
> > > formatted. If cost is the only concern, we should confirm its
> impact
> > > through performance testing.
> > >
> > > One of the key use cases requires access on consume at per
> event/message
> > >> level at the point that message is being processed, as such the
> batch
> > >> interceptors and batch consume api isn't suitable. It needs to be
> at the
> > >> record level.
> > >
> > >
> > > I'm not sure I understand the point about batching. Interceptors
> are
> > > applied per-message, right?
> > >
> > > My intent on interceptors is to keep the usage of headers
> well-defined so
> > > that they don't start leaking unnecessarily into applications. My
> guess
> > is
> > > that it's probably inevitable, but isolating it in the
> interceptors would
> > > at least give people a second thought before deciding to use it.
> The main
> > > challenge in my mind is figuring out how an MM use case would
> work. It
> > > would be more cumbersome to replicate headers through an
> interceptor,
> > > though arguably MM should be working at a lower level anyway.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > >> Re headers available on the record va interceptors only
> > >>
> > >> One of the key use cases requires access on consume at per
> event/message
> > >> level at the point that message is being processed, as such the
> batch
> > >> interceptors and batch consume api isn't suitable. It needs to be
> at the
> > >> record level.
> > >>
> > >> This anyhow is similar to jms/http/amqp where headers are
> available to
> > >> consuming applications.
> > >>
> > >> Re headers as byte array and future use by broker. This doesn't
> take
> > away
> > >> from that at all. Nor makes it difficult at all in my opinion.
> > >>
> > >>
> > >>
> > >> Sent using OWA for iPhone
> > >> ________________________________________
> > >> From: Jason Gustafson <ja...@confluent.io>
> > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >>
> > >> >
> > >> > Would you be proposing in KIP-98 to convert the other message
> int’s
> > (key
> > >> > length, value length) also to varint to keep it uniform.
> > >> > Also I assume there will be a static or helper method made to
> > write/read
> > >> > these in the client and server.
> > >>
> > >>
> > >> Yes, that is what we are proposing, so using varints for headers
> would
> > be
> > >> consistent with the rest of the message. We have used static
> helper
> > >> methods
> > >> in our prototype implementation.
> > >>
> > >> The cost of parsing, we want to parse/interpret the headers
> lazily (this
> > >> is
> > >> > a key point brought up earlier in discussions)
> > >>
> > >>
> > >> I'm a bit skeptical of this. Has anyone done the performance
> testing? I
> > >> can
> > >> probably implement it and test it if no one else has. I was also
> under
> > the
> > >> impression that there may be use cases down the road where the
> broker
> > >> would
> > >> need to interpret headers. That wouldn't be off the table in the
> future
> > if
> > >> it's represented as bytes, but it would be quite a bit more
> awkward,
> > >> right?
> > >>
> > >> By the way, one question I have been wondering about. My
> understanding
> > is
> > >> that headers are primarily for use cases where a third-party
> components
> > >> wants to enrich messages without needing to understand or modify
> the
> > >> schema
> > >> of the message key and value. For the applications which directly
> > produce
> > >> and consume the messages and control the key/value schema
> directly, it
> > >> seems we would rather have them implement headers directly in
> their own
> > >> schema. Supposing for the sake of argument that it was possible,
> my
> > >> question is whether it be sufficient to expose the headers in the
> > >> interceptor API and not in the common API?
> > >>
> > >> -Jason
> > >>
> > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > >> wrote:
> > >>
> > >> > On the point of varInts
> > >> >
> > >> > Would you be proposing in KIP-98 to convert the other message
> int’s
> > (key
> > >> > length, value length) also to varint to keep it uniform.
> > >> > Also I assume there will be a static or helper method made to
> > write/read
> > >> > these in the client and server.
> > >> >
> > >> > Cheers
> > >> > Mike
> > >> >
> > >> >
> > >> >
> > >> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com>
> wrote:
> > >> >
> > >> > On the point re: headers in the message protocol being a
> byte
> > array
> > >> > and not a count of elements followed by the elements. Again
> this was
> > >> > discussed/argued previously.
> > >> >
> > >> > It was agreed on for a few reasons some of which you have
> > obviously
> > >> > picked up on:
> > >> >
> > >> > Broker is able to pass it through opaquely
> > >> > The cost of parsing, we want to parse/interpret the headers
> lazily
> > >> > (this is a key point brought up earlier in discussions)
> > >> > Headers can be copied from consumer record to producer
> record (aka
> > >> > mirror makers etc) without parsing if no changes are being made
> or
> > being
> > >> > looked at.
> > >> > Keeps the broker agnostic to the format
> > >> > You need an int32 either for the byte size of the headers,
> or for
> > >> the
> > >> > count of elements, so overheads are the same, but with going
> with an
> > >> opaque
> > >> > byte array has the above advantages.
> > >> >
> > >> > Cheers
> > >> > Mike
> > >> >
> > >> >
> > >> > On 17/02/2017, 02:50, "Jason Gustafson" <jason@confluent.io
> >
> > wrote:
> > >> >
> > >> > Sorry, should have noted that the performance testing
> was done
> > >> > using the
> > >> > producer performance tool shipped with Kafka.
> > >> >
> > >> > -Jason
> > >> >
> > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> > >> > jason@confluent.io> wrote:
> > >> >
> > >> > > Hey Nacho,
> > >> > >
> > >> > > I've compared performance of our KIP-98
> implementation with
> > >> and
> > >> > without
> > >> > > varints. For messages around 128 bytes, we see an
> increase
> > in
> > >> > throughput of
> > >> > > about 30% using the default configuration settings.
> At 256
> > >> > bytes, the
> > >> > > increase is around 16%. Obviously the performance
> converge
> > as
> > >> > messages get
> > >> > > larger, but it seems well worth the cost. Note that
> we are
> > >> also
> > >> > seeing a
> > >> > > substantial performance increase against trunk
> primarily
> > >> because
> > >> > of the
> > >> > > much more efficient packing that varints provide us.
> > Anything
> > >> > adding to
> > >> > > message overhead, such as record headers, would only
> > increase
> > >> > the relative
> > >> > > difference. (Of course take these numbers with a
> grain of
> > salt
> > >> > since I have
> > >> > > only used the default settings with both the producer
> and
> > >> broker
> > >> > on my
> > >> > > local machine. We intend to provide more extensive
> > performance
> > >> > details as
> > >> > > part of the work for KIP-98.)
> > >> > >
> > >> > > The implementation we are using is from protobuf (
> > >> > > https://developers.google.com/
> > protocol-buffers/docs/encoding
> > >> ),
> > >> > which is
> > >> > > also used in HBase. It is trivial to implement and as
> far
> > as I
> > >> > know doesn't
> > >> > > suffer from the aliasing problem you are describing. I
> > checked
> > >> > with Magnus
> > >> > > (the author of librdkafka) and he agreed that the
> savings
> > >> seemed
> > >> > worth the
> > >> > > cost of implementation.
> > >> > >
> > >> > > -Jason
> > >> > >
> > >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
> > >> isolis@igso.net>
> > >> > wrote:
> > >> > >
> > >> > >> -VarInts
> > >> > >>
> > >> > >> I'm one of the people (if not the most) opposed to
> VarInts.
> > >> > VarInts
> > >> > >> have a place, but this is not it. (We had a large
> > >> discussion
> > >> > about
> > >> > >> them at the beginning of KIP-82 time)
> > >> > >>
> > >> > >> If anybody has real life performance numbers of
> VarInts
> > >> > improving
> > >> > >> things or significantly reducing resources I would
> like to
> > >> know
> > >> > what
> > >> > >> that case may be. Yes, you can save some bytes here
> and
> > >> there,
> > >> > but
> > >> > >> this is probably insignificant to the overall system
> > behavior
> > >> > and
> > >> > >> storage requirements. -- I say this with respect to
> using
> > >> > VarInts in
> > >> > >> the protocol itself, not as part of the data.
> > >> > >>
> > >> > >> VarInts require you to parse the Int before using it
> and
> > >> > depending on
> > >> > >> the encoding they can suffer from aliasing (multiple
> > >> > representations
> > >> > >> for the same value).
> > >> > >>
> > >> > >> Why add complexity?
> > >> > >>
> > >> > >> Nacho
> > >> > >>
> > >> > >>
> > >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> > >> > cmccabe@apache.org>
> > >> > >> wrote:
> > >> > >> > +1 for varints here-- it would save quite a bit of
> space.
> > >> > They are
> > >> > >> > pretty quick to implement as well.
> > >> > >> >
> > >> > >> > I think it makes sense for values to be byte
> arrays.
> > Users
> > >> > might want
> > >> > >> > to attach arbitrary payloads; they shouldn't be
> forced to
> > >> > serialize
> > >> > >> > everything to Java strings.
> > >> > >> >
> > >> > >> > best,
> > >> > >> > Colin
> > >> > >> >
> > >> > >> >
> > >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson
> wrote:
> > >> > >> >> Hey Michael,
> > >> > >> >>
> > >> > >> >> Hmm, I guess the point of representing it as
> bytes is to
> > >> > allow the
> > >> > >> broker
> > >> > >> >> to pass it through opaquely? Is the cost of
> parsing
> > them a
> > >> > concern, or
> > >> > >> >> are
> > >> > >> >> we simply trying to ensure that the broker stays
> > agnostic
> > >> to
> > >> > the
> > >> > >> format?
> > >> > >> >>
> > >> > >> >> On varints, I think adding support for them makes
> less
> > >> sense
> > >> > for an
> > >> > >> >> isolated use case, but as part of a more holistic
> change
> > >> > (such as what
> > >> > >> we
> > >> > >> >> have proposed in KIP-98), I think they are
> justifiable.
> > If
> > >> > we add them,
> > >> > >> >> then the need to use attributes becomes quite a
> bit
> > >> weaker,
> > >> > right? The
> > >> > >> >> other thing I find slightly odd is the fact that
> null
> > >> > headers has no
> > >> > >> >> actual
> > >> > >> >> semantic meaning for the message (unlike null
> keys and
> > >> > values). It is
> > >> > >> >> just
> > >> > >> >> a space optimization. It seems a bit better to
> always
> > use
> > >> > size 0 to
> > >> > >> >> indicate having no headers.
> > >> > >> >>
> > >> > >> >> Overall, the main point is ensuring that the
> message
> > >> schema
> > >> > remains
> > >> > >> >> consistent, either within the larger protocol, or
> at a
> > >> > minimum within
> > >> > >> the
> > >> > >> >> message itself.
> > >> > >> >>
> > >> > >> >> -Jason
> > >> > >> >>
> > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> > >> > Michael.Pearce@ig.com
> > >> > >> >
> > >> > >> >> wrote:
> > >> > >> >>
> > >> > >> >> > Hi Jason,
> > >> > >> >> >
> > >> > >> >> > On point 1) in the message protocol the headers
> are
> > >> simply
> > >> > a byte
> > >> > >> array,
> > >> > >> >> > as like the key or value, this is to clearly
> demarcate
> > >> the
> > >> > header in
> > >> > >> the
> > >> > >> >> > core message. Then the header byte array in the
> core
> > >> > message is an
> > >> > >> array of
> > >> > >> >> > key, value pairs. This is what it is denoting.
> > >> > >> >> >
> > >> > >> >> > Then this would be I guess in the given
> notation:
> > >> > >> >> >
> > >> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> > >> > >> >> > KeyLength => int32 <-----------------NEW
> size of
> > the
> > >> > byte[] of
> > >> > >> the
> > >> > >> >> > serialised key value
> > >> > >> >> > Key => bytes <---------------------- NEW
> > serialised
> > >> > string (UTF8)
> > >> > >> >> > bytes of the header key
> > >> > >> >> > ValueLength => int32 <-------------- NEW
> size of
> > the
> > >> > byte[] of
> > >> > >> the
> > >> > >> >> > serialised header value
> > >> > >> >> > Value => bytes <-------------------- NEW
> > serialised
> > >> > form of the
> > >> > >> header
> > >> > >> >> > value
> > >> > >> >> >
> > >> > >> >> > The key length and value length is matching the
> way
> > the
> > >> > protocol is
> > >> > >> >> > defined in the core message currently.
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On point 2)
> > >> > >> >> > Var sized ints, this was discussed much earlier
> on, in
> > >> > fact I had
> > >> > >> >> > suggested it myself (with Hadoop references),
> the
> > >> > complexity of this
> > >> > >> >> > compared to having a simpler protocol was
> argued and
> > >> > agreed it
> > >> > >> wasn’t worth
> > >> > >> >> > the complexity as all other clients in other
> languages
> > >> > would need to
> > >> > >> ensure
> > >> > >> >> > theyre using the right var size algorithm, as
> there
> > is a
> > >> > few.
> > >> > >> >> >
> > >> > >> >> > On point 3)
> > >> > >> >> > We did the attributes, optional approach as
> originally
> > >> > there was
> > >> > >> marked
> > >> > >> >> > concern that headers would cause a message size
> > overhead
> > >> > for others,
> > >> > >> who
> > >> > >> >> > don’t want them. As such this is the clean
> solution to
> > >> > achieve that.
> > >> > >> If
> > >> > >> >> > that no longer holds, and we don’t care that we
> add
> > >> 4bytes
> > >> > overhead,
> > >> > >> then
> > >> > >> >> > im happy to remove.
> > >> > >> >> >
> > >> > >> >> > I’m personally in favour of keeping the message
> as
> > small
> > >> > as possible
> > >> > >> so
> > >> > >> >> > people don’t get shocks in perf and throughputs
> dues
> > to
> > >> > message size,
> > >> > >> >> > unless they actively use the feature, as such I
> do
> > >> prefer
> > >> > the
> > >> > >> attribute bit
> > >> > >> >> > wise feature flag approach myself.
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> > >> > jason@confluent.io> wrote:
> > >> > >> >> >
> > >> > >> >> > We have proposed a few significant changes
> to the
> > >> > message format
> > >> > >> in
> > >> > >> >> > KIP-98
> > >> > >> >> > which now seems likely to pass (perhaps
> with some
> > >> > iterations on
> > >> > >> >> > implementation details). It would be good
> to try
> > and
> > >> > coordinate
> > >> > >> the
> > >> > >> >> > changes
> > >> > >> >> > in both of the proposals to make sure they
> are
> > >> > consistent and
> > >> > >> >> > compatible.
> > >> > >> >> >
> > >> > >> >> > I think using the attributes to indicate
> null
> > >> headers
> > >> > is a
> > >> > >> reasonable
> > >> > >> >> > approach. We have proposed to do the same
> thing
> > for
> > >> > the message
> > >> > >> key and
> > >> > >> >> > value. That said, I sympathize with Jay's
> > argument.
> > >> > Having
> > >> > >> multiple
> > >> > >> >> > ways to
> > >> > >> >> > specify a null value increases the overall
> > >> complexity
> > >> > of the
> > >> > >> protocol.
> > >> > >> >> > You
> > >> > >> >> > can see this just from the fact that you
> need the
> > >> > extra verbiage
> > >> > >> in the
> > >> > >> >> > protocol specification in this KIP and in
> KIP-98
> > to
> > >> > describe the
> > >> > >> >> > dependence
> > >> > >> >> > between the fields and the attributes. It
> seems
> > >> like a
> > >> > slippery
> > >> > >> slope
> > >> > >> >> > if
> > >> > >> >> > you start allowing different request types
> to
> > >> > implement the
> > >> > >> protocol
> > >> > >> >> > specification differently.
> > >> > >> >> >
> > >> > >> >> > You can also argue that the messages
> already are
> > and
> > >> > are likely
> > >> > >> to
> > >> > >> >> > remain a
> > >> > >> >> > special case. For example, there is
> currently no
> > >> > generality in
> > >> > >> how
> > >> > >> >> > compressed message sets are represented
> that would
> > >> be
> > >> > applicable
> > >> > >> for
> > >> > >> >> > other
> > >> > >> >> > request types. Some might see this
> divergence as
> > an
> > >> > unfortunate
> > >> > >> >> > protocol
> > >> > >> >> > deficiency which should be fixed; others
> might see
> > >> it
> > >> > as sort of
> > >> > >> the
> > >> > >> >> > inevitability of needing to optimize where
> it
> > counts
> > >> > most. I'm
> > >> > >> probably
> > >> > >> >> > somewhere in between, but I think we
> probably all
> > >> > share the
> > >> > >> intuition
> > >> > >> >> > that
> > >> > >> >> > the protocol should be kept as consistent as
> > >> possible.
> > >> > With that
> > >> > >> in
> > >> > >> >> > mind,
> > >> > >> >> > here are a few comments:
> > >> > >> >> >
> > >> > >> >> > 1. One thing I found a little odd when
> reading the
> > >> > current
> > >> > >> proposal is
> > >> > >> >> > that
> > >> > >> >> > the headers are both represented as an
> array of
> > >> bytes
> > >> > and as an
> > >> > >> array
> > >> > >> >> > of
> > >> > >> >> > key/value pairs. I'd probably suggest
> something
> > like
> > >> > this:
> > >> > >> >> >
> > >> > >> >> > Headers => [HeaderKey HeaderValue]
> > >> > >> >> > HeaderKey => String
> > >> > >> >> > HeaderValue => Bytes
> > >> > >> >> >
> > >> > >> >> > An array in the Kafka protocol is
> represented as a
> > >> > 4-byte integer
> > >> > >> >> > indicating the number of elements in the
> array
> > >> > followed by the
> > >> > >> >> > serialization of the elements. Unless I'm
> > >> > misunderstanding, what
> > >> > >> you
> > >> > >> >> > have
> > >> > >> >> > instead is the total size of the headers in
> bytes
> > >> > followed by the
> > >> > >> >> > elements.
> > >> > >> >> > I'm not sure I see any reason for this
> > >> inconsistency.
> > >> > >> >> >
> > >> > >> >> > 2. In KIP-98, we've introduced
> variable-length
> > >> integer
> > >> > fields.
> > >> > >> >> > Effectively,
> > >> > >> >> > we've enriched (or "complicated" as Jay
> might say
> > ;)
> > >> > the protocol
> > >> > >> >> > specification to include the following
> types:
> > >> VarInt,
> > >> > VarLong,
> > >> > >> >> > UnsignedVarInt and UnsignedVarLong.
> > >> > >> >> >
> > >> > >> >> > Along with these primitives, we could
> introduce
> > the
> > >> > following
> > >> > >> types:
> > >> > >> >> >
> > >> > >> >> > VarSizeArray => NumberOfItems Item1 Item2
> .. ItemN
> > >> > >> >> > NumberOfItems => UnsignedVarInt
> > >> > >> >> >
> > >> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull
> Item1
> > >> > Item2 .. ItemN
> > >> > >> >> > NumberOfItemsOrNull => VarInt (-1 means
> null)
> > >> > >> >> >
> > >> > >> >> > And similarly for the `String` and `Bytes`
> types.
> > >> > These types
> > >> > >> can save
> > >> > >> >> > a
> > >> > >> >> > considerable amount of space in this
> proposal
> > >> because
> > >> > they can
> > >> > >> be used
> > >> > >> >> > for
> > >> > >> >> > both the number of headers included in the
> message
> > >> and
> > >> > the
> > >> > >> lengths of
> > >> > >> >> > the
> > >> > >> >> > header keys and values. We could do this
> instead:
> > >> > >> >> >
> > >> > >> >> > Headers => VarSizeArray[HeaderKey
> HeaderValue]
> > >> > >> >> > HeaderKey => VarSizeString
> > >> > >> >> > HeaderValue => VarSizeBytes
> > >> > >> >> >
> > >> > >> >> > Combining the savings from the use of
> variable
> > >> length
> > >> > fields, the
> > >> > >> >> > benefit
> > >> > >> >> > of using the attributes to represent null
> seems
> > >> pretty
> > >> > small.
> > >> > >> >> >
> > >> > >> >> > 3. Whichever way we go (whether we use the
> > >> attributes
> > >> > or not), we
> > >> > >> >> > should at
> > >> > >> >> > least be consistent between this KIP and
> KIP-98.
> > It
> > >> > would be very
> > >> > >> >> > strange
> > >> > >> >> > to have two ways to represent null values
> in the
> > >> same
> > >> > schema.
> > >> > >> Either
> > >> > >> >> > way is
> > >> > >> >> > OK with me. I think some message-level
> > optimizations
> > >> > are
> > >> > >> justifiable,
> > >> > >> >> > but
> > >> > >> >> > the savings here seem minimal (a few bytes
> per
> > >> > message), so
> > >> > >> maybe it's
> > >> > >> >> > not
> > >> > >> >> > worth the cost of letting the message
> diverge even
> > >> > further from
> > >> > >> the
> > >> > >> >> > rest of
> > >> > >> >> > the protocol.
> > >> > >> >> >
> > >> > >> >> > -Jason
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> > >> > >> radai.rosenblatt@gmail.com>
> > >> > >> >> > wrote:
> > >> > >> >> >
> > >> > >> >> > > I've trimmed the inline contents as this
> mail is
> > >> > getting too
> > >> > >> big for
> > >> > >> >> > the
> > >> > >> >> > > apache mailing list software to deliver
> :-(
> > >> > >> >> > >
> > >> > >> >> > > 1. the important thing for
> interoperability is
> > for
> > >> > different
> > >> > >> >> > "interested
> > >> > >> >> > > parties" (plugins, infra layers/wrappers,
> > >> user-code)
> > >> > to be
> > >> > >> able to
> > >> > >> >> > stick
> > >> > >> >> > > pieces of metadata onto msgs without
> getting in
> > >> each
> > >> > other's
> > >> > >> way. a
> > >> > >> >> > common
> > >> > >> >> > > key scheme (Strings, as of the time of
> this
> > >> > writing?) is all
> > >> > >> thats
> > >> > >> >> > required
> > >> > >> >> > > for that. it is assumed that the other end
> > >> > interested in any
> > >> > >> such
> > >> > >> >> > piece of
> > >> > >> >> > > metadata knows the encoding, and byte[]
> provides
> > >> for
> > >> > the most
> > >> > >> >> > flexibility.
> > >> > >> >> > > i believe this is the same logic behind
> core
> > kafka
> > >> > being
> > >> > >> >> > byte[]/byte[] -
> > >> > >> >> > > Strings are more "usable" but bytes are
> flexible
> > >> and
> > >> > so were
> > >> > >> chosen.
> > >> > >> >> > > Also - core kafka doesnt even do that
> good of a
> > >> job
> > >> > on
> > >> > >> usability of
> > >> > >> >> > the
> > >> > >> >> > > payload (example - i have to specify the
> nop
> > >> byte[]
> > >> > "decoders"
> > >> > >> >> > explicitly
> > >> > >> >> > > in conf), and again sacrificies usability
> for
> > the
> > >> > sake of
> > >> > >> >> > performance (no
> > >> > >> >> > > convenient single-record processing as
> poll is a
> > >> > batch, lots of
> > >> > >> >> > obscure
> > >> > >> >> > > little config details exposing internals
> of the
> > >> > batching
> > >> > >> mechanism,
> > >> > >> >> > etc)
> > >> > >> >> > >
> > >> > >> >> > > this is also why i really dislike the
> idea of a
> > >> > "type system"
> > >> > >> for
> > >> > >> >> > header
> > >> > >> >> > > values, it further degrades the
> usability, adds
> > >> > complexity and
> > >> > >> will
> > >> > >> >> > > eventually get in people's way, also, it
> would
> > be
> > >> > the 2nd/3rd
> > >> > >> >> > home-group
> > >> > >> >> > > serialization mechanism in core kafka
> (counting
> > 2
> > >> > iterations
> > >> > >> of the
> > >> > >> >> > "type
> > >> > >> >> > > definition DSL")
> > >> > >> >> > >
> > >> > >> >> > > 2. this is an implementation detail, and
> not
> > even
> > >> a
> > >> > very "user
> > >> > >> >> > facing" one?
> > >> > >> >> > > to the best of my understanding the vote
> process
> > >> is
> > >> > on proposed
> > >> > >> >> > > API/behaviour. also - since we're willing
> to go
> > >> with
> > >> > strings
> > >> > >> just
> > >> > >> >> > serialize
> > >> > >> >> > > a 0-sized header blob and IIUC you dont
> need any
> > >> > optionals
> > >> > >> anymore.
> > >> > >> >> > >
> > >> > >> >> > > 3. yes, we can :-)
> > >> > >> >> > >
> > >> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael
> > Pearce <
> > >> > >> >> > Michael.Pearce@ig.com>
> > >> > >> >> > > wrote:
> > >> > >> >> > >
> > >> > >> >> > > > Hi Jay,
> > >> > >> >> > > >
> > >> > >> >> > > > 1) There was some initial debate on the
> value
> > >> > part, as youll
> > >> > >> note
> > >> > >> >> > String,
> > >> > >> >> > > > String headers were discounted early
> on. The
> > >> > reason for this
> > >> > >> is
> > >> > >> >> > > flexibility
> > >> > >> >> > > > and keeping in line with the
> flexibility of
> > key,
> > >> > value of the
> > >> > >> >> > message
> > >> > >> >> > > > object itself. I don’t think it takes
> away
> > from
> > >> an
> > >> > ecosystem
> > >> > >> as
> > >> > >> >> > each
> > >> > >> >> > > plugin
> > >> > >> >> > > > will care for their own key, this way
> ints,
> > >> > booleans , exotic
> > >> > >> >> > custom
> > >> > >> >> > > binary
> > >> > >> >> > > > can all be catered for=.
> > >> > >> >> > > > a. If you really wanted to push for a
> typed
> > >> value
> > >> > interface,
> > >> > >> I
> > >> > >> >> > wouldn’t
> > >> > >> >> > > > want just String values supported, but
> the the
> > >> > primatives
> > >> > >> plus
> > >> > >> >> > string and
> > >> > >> >> > > > also still keeping the ability to have a
> > binary
> > >> > for custom
> > >> > >> >> > binaries that
> > >> > >> >> > > > some organisations may have.
> > >> > >> >> > > > i. I have written this slight
> alternative
> > here,
> > >> > >> >> > > https://cwiki.apache.org/
> > >> > >> >> > > > confluence/display/KAFKA/KIP-
> > >> > 82+-+Add+Record+Headers+-+Typed
> > >> > >> >> > > > ii. Essentially the value bytes, has a
> leading
> > >> > byte overhead.
> > >> > >> >> > > > 1. This tells you what type the value
> is,
> > >> before
> > >> > reading
> > >> > >> the rest
> > >> > >> >> > of the
> > >> > >> >> > > > bytes, allowing
> serialisation/deserialization
> > to
> > >> > and from the
> > >> > >> >> > primitives,
> > >> > >> >> > > > string and byte[]. This is akin to some
> other
> > >> > messaging
> > >> > >> systems.
> > >> > >> >> > > > 2) We are making it optional, so that
> for
> > those
> > >> > not wanting
> > >> > >> >> > headers have
> > >> > >> >> > > 0
> > >> > >> >> > > > bytes overhead (think of it as a feature
> > flag),
> > >> I
> > >> > don’t
> > >> > >> think this
> > >> > >> >> > is
> > >> > >> >> > > > complex, especially if comparing to
> changes
> > >> > proposed in
> > >> > >> other kips
> > >> > >> >> > like
> > >> > >> >> > > > kip-98.
> > >> > >> >> > > > a. If you really really don’t like
> this, we
> > can
> > >> > drop it, but
> > >> > >> it
> > >> > >> >> > would
> > >> > >> >> > > mean
> > >> > >> >> > > > buying into 4 bytes extra overhead for
> users
> > who
> > >> > do not want
> > >> > >> to use
> > >> > >> >> > > headers.
> > >> > >> >> > > > 3) In the summary yes, it is at a higher
> > level,
> > >> > but I think
> > >> > >> this
> > >> > >> >> > is well
> > >> > >> >> > > > documented in the proposed changes
> section.
> > >> > >> >> > > > a. Added getHeaders method to
> > Producer/Consumer
> > >> > record (that
> > >> > >> is it)
> > >> > >> >> > > > b. We’ve also detailed the new Headers
> class
> > >> that
> > >> > this method
> > >> > >> >> > returns
> > >> > >> >> > > that
> > >> > >> >> > > > encapsulates the headers protocol and
> logic.
> > >> > >> >> > > >
> > >> > >> >> > > > Best,
> > >> > >> >> > > > Mike
> > >> > >> >> > > >
> > >> > >> >> > > > ==Original questions from the vote
> thread from
> > >> > Jay.==
> > >> > >> >> > > >
> > >> > >> >> > > > Couple of things I think we still need
> to work
> > >> out:
> > >> > >> >> > > >
> > >> > >> >> > > > 1. I think we agree about the key,
> but I
> > >> think
> > >> > we haven't
> > >> > >> >> > talked about
> > >> > >> >> > > > the value yet. I think if our goal
> is an
> > open
> > >> > ecosystem
> > >> > >> of these
> > >> > >> >> > > header
> > >> > >> >> > > > spread across many plugins from many
> > systems
> > >> we
> > >> > should
> > >> > >> consider
> > >> > >> >> > making
> > >> > >> >> > > > this
> > >> > >> >> > > > a string as well so it can be
> printed, set
> > >> via
> > >> > a UI, set
> > >> > >> in
> > >> > >> >> > config,
> > >> > >> >> > > etc.
> > >> > >> >> > > > Basically encouraging pluggable
> > serialization
> > >> > formats
> > >> > >> here will
> > >> > >> >> > lead
> > >> > >> >> > > to
> > >> > >> >> > > > a
> > >> > >> >> > > > bit of a tower of babel.
> > >> > >> >> > > > 2. This proposal still includes a
> pretty
> > big
> > >> > change to our
> > >> > >> >> > > serialization
> > >> > >> >> > > > and protocol definition layer.
> Essentially
> > >> it is
> > >> > >> introducing an
> > >> > >> >> > > optional
> > >> > >> >> > > > type, where the format is data
> dependent. I
> > >> > think this is
> > >> > >> >> > actually a
> > >> > >> >> > > big
> > >> > >> >> > > > change though it doesn't seem like
> it. It
> > >> means
> > >> > you can no
> > >> > >> >> > longer
> > >> > >> >> > > > specify
> > >> > >> >> > > > this type with our type definition
> DSL, and
> > >> > likewise it
> > >> > >> requires
> > >> > >> >> > > custom
> > >> > >> >> > > > handling in client libs. This isn't
> a huge
> > >> > thing, since
> > >> > >> the
> > >> > >> >> > Record
> > >> > >> >> > > > definition is custom anyway, but I
> think
> > this
> > >> > kind of
> > >> > >> protocol
> > >> > >> >> > > > inconsistency is very non-desirable
> and
> > ties
> > >> > you to
> > >> > >> hand-coding
> > >> > >> >> > > things.
> > >> > >> >> > > > I
> > >> > >> >> > > > think the type should instead by [Key
> > Value]
> > >> in
> > >> > our BNF,
> > >> > >> where
> > >> > >> >> > key and
> > >> > >> >> > > > value are both short strings as used
> > >> elsewhere.
> > >> > This
> > >> > >> brings it
> > >> > >> >> > in line
> > >> > >> >> > > > with
> > >> > >> >> > > > the rest of the protocol.
> > >> > >> >> > > > 3. Could we get more specific about
> the
> > exact
> > >> > Java API
> > >> > >> change to
> > >> > >> >> > > > ProducerRecord, ConsumerRecord,
> Record,
> > etc?
> > >> > >> >> > > >
> > >> > >> >> > > > -Jay
> > >> > >> >> > > >
> > >> > >> >> > >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > The information contained in this email is
> strictly
> > >> > confidential and
> > >> > >> for
> > >> > >> >> > the use of the addressee only, unless otherwise
> > >> indicated.
> > >> > If you
> > >> > >> are not
> > >> > >> >> > the intended recipient, please do not read,
> copy, use
> > or
> > >> > disclose to
> > >> > >> others
> > >> > >> >> > this message or any attachment. Please also
> notify the
> > >> > sender by
> > >> > >> replying
> > >> > >> >> > to this email or by telephone (+44(020 7896
> 0011) and
> > >> then
> > >> > delete
> > >> > >> the email
> > >> > >> >> > and any copies of it. Opinions, conclusion
> (etc) that
> > do
> > >> > not relate
> > >> > >> to the
> > >> > >> >> > official business of this company shall be
> understood
> > as
> > >> > neither
> > >> > >> given nor
> > >> > >> >> > endorsed by it. IG is a trading name of IG
> Markets
> > >> Limited
> > >> > (a company
> > >> > >> >> > registered in England and Wales, company number
> > >> 04008957)
> > >> > and IG
> > >> > >> Index
> > >> > >> >> > Limited (a company registered in England and
> Wales,
> > >> > company number
> > >> > >> >> > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > >> > Dowgate
> > >> > >> Hill,
> > >> > >> >> > London EC4R 2YA. Both IG Markets Limited
> (register
> > >> number
> > >> > 195355)
> > >> > >> and IG
> > >> > >> >> > Index Limited (register number 114059) are
> authorised
> > >> and
> > >> > regulated
> > >> > >> by the
> > >> > >> >> > Financial Conduct Authority.
> > >> > >> >> >
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Nacho - Ignacio Solis - isolis@igso.net
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > The information contained in this email is strictly
> confidential and
> > for
> > >> > the use of the addressee only, unless otherwise indicated. If
> you are
> > >> not
> > >> > the intended recipient, please do not read, copy, use or
> disclose to
> > >> others
> > >> > this message or any attachment. Please also notify the sender by
> > >> replying
> > >> > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > >> email
> > >> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > >> the
> > >> > official business of this company shall be understood as
> neither given
> > >> nor
> > >> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > >> > registered in England and Wales, company number 04008957) and
> IG Index
> > >> > Limited (a company registered in England and Wales, company
> number
> > >> > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > >> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
> > IG
> > >> > Index Limited (register number 114059) are authorised and
> regulated by
> > >> the
> > >> > Financial Conduct Authority.
> > >> >
> > >> The information contained in this email is strictly confidential
> and for
> > >> the use of the addressee only, unless otherwise indicated. If you
> are
> > not
> > >> the intended recipient, please do not read, copy, use or disclose
> to
> > others
> > >> this message or any attachment. Please also notify the sender by
> > replying
> > >> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > email
> > >> and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > the
> > >> official business of this company shall be understood as neither
> given
> > nor
> > >> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > >> registered in England and Wales, company number 04008957) and IG
> Index
> > >> Limited (a company registered in England and Wales, company number
> > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > >> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > >> Index Limited (register number 114059) are authorised and
> regulated by
> > the
> > >> Financial Conduct Authority.
> > >>
> > >
> > >
> >
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
The point about creation of maps seems orthogonal. We can still represent
the headers as a slice of bytes until the time it is accessed.
> Yes exactly we have access to the records thus why the header should be
> accessible via it and not hidden for only interceptors to access.
As explained above, the point is to make the intended usage clear.
Applications should continue to rely on the key/value fields to serialize
their own headers, and it would be more ideal if we can avoid leaking
third-party headers into applications. This is difficult to do with the
current interceptors because they share the record objects with the common
API. What I had in mind is something like an extension of the current
interceptors which exposed a different object (e.g. `RecordAndHeaders`).
The challenge is for MM-like use cases. Let me see if I can come up with a
concrete proposal for that problem.
-Jason
On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <Mi...@ig.com>
wrote:
> I am happy to move the definition of the header into the message body, but
> would cause us not to lazy initialise/parse the headers, as obviously, we
> would have to traverse these reading the message.
>
> This was actually one of Jay’s requests:
>
> “ 2. I think we should think about creating the Map lazily to avoid
> parsing out all the headers into little objects. HashMaps themselves
> are
> kind of expensive and the consumer is very perf sensitive so and making
> gazillions of hashmaps that may or may not get used is probably a bad
> idea.”
>
>
>
>
>
> On 17/02/2017, 19:44, "Michael Pearce" <Mi...@ig.com> wrote:
>
> Yes exactly we have access to the records thus why the header should
> be accessible via it and not hidden for only interceptors to access.
>
> Sent using OWA for iPhone
> ________________________________________
> From: Magnus Edenhill <ma...@edenhill.se>
> Sent: Friday, February 17, 2017 7:34:49 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Big +1 on VarInts.
> CPUs are fast, memory is slow.
>
> I agree with Jason that we'll want to continue verifying messages,
> including their headers, so while I appreciate the idea of the opaque
> header blob it won't be useful in practice.
>
> /Magnus
>
> 2017-02-17 10:41 GMT-08:00 Jason Gustafson <ja...@confluent.io>:
>
> > Sorry, my mistake. The consumer interceptor is per batch, though I'm
> not
> > sure that's an actual limitation since you still have access to the
> > individual records.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <
> jason@confluent.io>
> > wrote:
> >
> > > Re headers as byte array and future use by broker. This doesn't
> take away
> > >> from that at all. Nor makes it difficult at all in my opinion.
> > >
> > >
> > > Yeah, I didn't say it was difficult, only awkward. You wouldn't
> write the
> > > schema that way if you were planning to use it on the brokers from
> the
> > > beginning. Note also that one of the benefits of letting the broker
> > > understand headers is that it can validate that they are properly
> > > formatted. If cost is the only concern, we should confirm its
> impact
> > > through performance testing.
> > >
> > > One of the key use cases requires access on consume at per
> event/message
> > >> level at the point that message is being processed, as such the
> batch
> > >> interceptors and batch consume api isn't suitable. It needs to be
> at the
> > >> record level.
> > >
> > >
> > > I'm not sure I understand the point about batching. Interceptors
> are
> > > applied per-message, right?
> > >
> > > My intent on interceptors is to keep the usage of headers
> well-defined so
> > > that they don't start leaking unnecessarily into applications. My
> guess
> > is
> > > that it's probably inevitable, but isolating it in the
> interceptors would
> > > at least give people a second thought before deciding to use it.
> The main
> > > challenge in my mind is figuring out how an MM use case would
> work. It
> > > would be more cumbersome to replicate headers through an
> interceptor,
> > > though arguably MM should be working at a lower level anyway.
> > >
> > > -Jason
> > >
> > > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > > wrote:
> > >
> > >> Re headers available on the record va interceptors only
> > >>
> > >> One of the key use cases requires access on consume at per
> event/message
> > >> level at the point that message is being processed, as such the
> batch
> > >> interceptors and batch consume api isn't suitable. It needs to be
> at the
> > >> record level.
> > >>
> > >> This anyhow is similar to jms/http/amqp where headers are
> available to
> > >> consuming applications.
> > >>
> > >> Re headers as byte array and future use by broker. This doesn't
> take
> > away
> > >> from that at all. Nor makes it difficult at all in my opinion.
> > >>
> > >>
> > >>
> > >> Sent using OWA for iPhone
> > >> ________________________________________
> > >> From: Jason Gustafson <ja...@confluent.io>
> > >> Sent: Friday, February 17, 2017 5:55:42 PM
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >>
> > >> >
> > >> > Would you be proposing in KIP-98 to convert the other message
> int’s
> > (key
> > >> > length, value length) also to varint to keep it uniform.
> > >> > Also I assume there will be a static or helper method made to
> > write/read
> > >> > these in the client and server.
> > >>
> > >>
> > >> Yes, that is what we are proposing, so using varints for headers
> would
> > be
> > >> consistent with the rest of the message. We have used static
> helper
> > >> methods
> > >> in our prototype implementation.
> > >>
> > >> The cost of parsing, we want to parse/interpret the headers
> lazily (this
> > >> is
> > >> > a key point brought up earlier in discussions)
> > >>
> > >>
> > >> I'm a bit skeptical of this. Has anyone done the performance
> testing? I
> > >> can
> > >> probably implement it and test it if no one else has. I was also
> under
> > the
> > >> impression that there may be use cases down the road where the
> broker
> > >> would
> > >> need to interpret headers. That wouldn't be off the table in the
> future
> > if
> > >> it's represented as bytes, but it would be quite a bit more
> awkward,
> > >> right?
> > >>
> > >> By the way, one question I have been wondering about. My
> understanding
> > is
> > >> that headers are primarily for use cases where a third-party
> components
> > >> wants to enrich messages without needing to understand or modify
> the
> > >> schema
> > >> of the message key and value. For the applications which directly
> > produce
> > >> and consume the messages and control the key/value schema
> directly, it
> > >> seems we would rather have them implement headers directly in
> their own
> > >> schema. Supposing for the sake of argument that it was possible,
> my
> > >> question is whether it be sufficient to expose the headers in the
> > >> interceptor API and not in the common API?
> > >>
> > >> -Jason
> > >>
> > >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <
> Michael.Pearce@ig.com>
> > >> wrote:
> > >>
> > >> > On the point of varInts
> > >> >
> > >> > Would you be proposing in KIP-98 to convert the other message
> int’s
> > (key
> > >> > length, value length) also to varint to keep it uniform.
> > >> > Also I assume there will be a static or helper method made to
> > write/read
> > >> > these in the client and server.
> > >> >
> > >> > Cheers
> > >> > Mike
> > >> >
> > >> >
> > >> >
> > >> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com>
> wrote:
> > >> >
> > >> > On the point re: headers in the message protocol being a
> byte
> > array
> > >> > and not a count of elements followed by the elements. Again
> this was
> > >> > discussed/argued previously.
> > >> >
> > >> > It was agreed on for a few reasons some of which you have
> > obviously
> > >> > picked up on:
> > >> >
> > >> > Broker is able to pass it through opaquely
> > >> > The cost of parsing, we want to parse/interpret the headers
> lazily
> > >> > (this is a key point brought up earlier in discussions)
> > >> > Headers can be copied from consumer record to producer
> record (aka
> > >> > mirror makers etc) without parsing if no changes are being made
> or
> > being
> > >> > looked at.
> > >> > Keeps the broker agnostic to the format
> > >> > You need an int32 either for the byte size of the headers,
> or for
> > >> the
> > >> > count of elements, so overheads are the same, but with going
> with an
> > >> opaque
> > >> > byte array has the above advantages.
> > >> >
> > >> > Cheers
> > >> > Mike
> > >> >
> > >> >
> > >> > On 17/02/2017, 02:50, "Jason Gustafson" <jason@confluent.io
> >
> > wrote:
> > >> >
> > >> > Sorry, should have noted that the performance testing
> was done
> > >> > using the
> > >> > producer performance tool shipped with Kafka.
> > >> >
> > >> > -Jason
> > >> >
> > >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> > >> > jason@confluent.io> wrote:
> > >> >
> > >> > > Hey Nacho,
> > >> > >
> > >> > > I've compared performance of our KIP-98
> implementation with
> > >> and
> > >> > without
> > >> > > varints. For messages around 128 bytes, we see an
> increase
> > in
> > >> > throughput of
> > >> > > about 30% using the default configuration settings.
> At 256
> > >> > bytes, the
> > >> > > increase is around 16%. Obviously the performance
> converge
> > as
> > >> > messages get
> > >> > > larger, but it seems well worth the cost. Note that
> we are
> > >> also
> > >> > seeing a
> > >> > > substantial performance increase against trunk
> primarily
> > >> because
> > >> > of the
> > >> > > much more efficient packing that varints provide us.
> > Anything
> > >> > adding to
> > >> > > message overhead, such as record headers, would only
> > increase
> > >> > the relative
> > >> > > difference. (Of course take these numbers with a
> grain of
> > salt
> > >> > since I have
> > >> > > only used the default settings with both the producer
> and
> > >> broker
> > >> > on my
> > >> > > local machine. We intend to provide more extensive
> > performance
> > >> > details as
> > >> > > part of the work for KIP-98.)
> > >> > >
> > >> > > The implementation we are using is from protobuf (
> > >> > > https://developers.google.com/
> > protocol-buffers/docs/encoding
> > >> ),
> > >> > which is
> > >> > > also used in HBase. It is trivial to implement and as
> far
> > as I
> > >> > know doesn't
> > >> > > suffer from the aliasing problem you are describing. I
> > checked
> > >> > with Magnus
> > >> > > (the author of librdkafka) and he agreed that the
> savings
> > >> seemed
> > >> > worth the
> > >> > > cost of implementation.
> > >> > >
> > >> > > -Jason
> > >> > >
> > >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
> > >> isolis@igso.net>
> > >> > wrote:
> > >> > >
> > >> > >> -VarInts
> > >> > >>
> > >> > >> I'm one of the people (if not the most) opposed to
> VarInts.
> > >> > VarInts
> > >> > >> have a place, but this is not it. (We had a large
> > >> discussion
> > >> > about
> > >> > >> them at the beginning of KIP-82 time)
> > >> > >>
> > >> > >> If anybody has real life performance numbers of
> VarInts
> > >> > improving
> > >> > >> things or significantly reducing resources I would
> like to
> > >> know
> > >> > what
> > >> > >> that case may be. Yes, you can save some bytes here
> and
> > >> there,
> > >> > but
> > >> > >> this is probably insignificant to the overall system
> > behavior
> > >> > and
> > >> > >> storage requirements. -- I say this with respect to
> using
> > >> > VarInts in
> > >> > >> the protocol itself, not as part of the data.
> > >> > >>
> > >> > >> VarInts require you to parse the Int before using it
> and
> > >> > depending on
> > >> > >> the encoding they can suffer from aliasing (multiple
> > >> > representations
> > >> > >> for the same value).
> > >> > >>
> > >> > >> Why add complexity?
> > >> > >>
> > >> > >> Nacho
> > >> > >>
> > >> > >>
> > >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> > >> > cmccabe@apache.org>
> > >> > >> wrote:
> > >> > >> > +1 for varints here-- it would save quite a bit of
> space.
> > >> > They are
> > >> > >> > pretty quick to implement as well.
> > >> > >> >
> > >> > >> > I think it makes sense for values to be byte
> arrays.
> > Users
> > >> > might want
> > >> > >> > to attach arbitrary payloads; they shouldn't be
> forced to
> > >> > serialize
> > >> > >> > everything to Java strings.
> > >> > >> >
> > >> > >> > best,
> > >> > >> > Colin
> > >> > >> >
> > >> > >> >
> > >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson
> wrote:
> > >> > >> >> Hey Michael,
> > >> > >> >>
> > >> > >> >> Hmm, I guess the point of representing it as
> bytes is to
> > >> > allow the
> > >> > >> broker
> > >> > >> >> to pass it through opaquely? Is the cost of
> parsing
> > them a
> > >> > concern, or
> > >> > >> >> are
> > >> > >> >> we simply trying to ensure that the broker stays
> > agnostic
> > >> to
> > >> > the
> > >> > >> format?
> > >> > >> >>
> > >> > >> >> On varints, I think adding support for them makes
> less
> > >> sense
> > >> > for an
> > >> > >> >> isolated use case, but as part of a more holistic
> change
> > >> > (such as what
> > >> > >> we
> > >> > >> >> have proposed in KIP-98), I think they are
> justifiable.
> > If
> > >> > we add them,
> > >> > >> >> then the need to use attributes becomes quite a
> bit
> > >> weaker,
> > >> > right? The
> > >> > >> >> other thing I find slightly odd is the fact that
> null
> > >> > headers has no
> > >> > >> >> actual
> > >> > >> >> semantic meaning for the message (unlike null
> keys and
> > >> > values). It is
> > >> > >> >> just
> > >> > >> >> a space optimization. It seems a bit better to
> always
> > use
> > >> > size 0 to
> > >> > >> >> indicate having no headers.
> > >> > >> >>
> > >> > >> >> Overall, the main point is ensuring that the
> message
> > >> schema
> > >> > remains
> > >> > >> >> consistent, either within the larger protocol, or
> at a
> > >> > minimum within
> > >> > >> the
> > >> > >> >> message itself.
> > >> > >> >>
> > >> > >> >> -Jason
> > >> > >> >>
> > >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> > >> > Michael.Pearce@ig.com
> > >> > >> >
> > >> > >> >> wrote:
> > >> > >> >>
> > >> > >> >> > Hi Jason,
> > >> > >> >> >
> > >> > >> >> > On point 1) in the message protocol the headers
> are
> > >> simply
> > >> > a byte
> > >> > >> array,
> > >> > >> >> > as like the key or value, this is to clearly
> demarcate
> > >> the
> > >> > header in
> > >> > >> the
> > >> > >> >> > core message. Then the header byte array in the
> core
> > >> > message is an
> > >> > >> array of
> > >> > >> >> > key, value pairs. This is what it is denoting.
> > >> > >> >> >
> > >> > >> >> > Then this would be I guess in the given
> notation:
> > >> > >> >> >
> > >> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> > >> > >> >> > KeyLength => int32 <-----------------NEW
> size of
> > the
> > >> > byte[] of
> > >> > >> the
> > >> > >> >> > serialised key value
> > >> > >> >> > Key => bytes <---------------------- NEW
> > serialised
> > >> > string (UTF8)
> > >> > >> >> > bytes of the header key
> > >> > >> >> > ValueLength => int32 <-------------- NEW
> size of
> > the
> > >> > byte[] of
> > >> > >> the
> > >> > >> >> > serialised header value
> > >> > >> >> > Value => bytes <-------------------- NEW
> > serialised
> > >> > form of the
> > >> > >> header
> > >> > >> >> > value
> > >> > >> >> >
> > >> > >> >> > The key length and value length is matching the
> way
> > the
> > >> > protocol is
> > >> > >> >> > defined in the core message currently.
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On point 2)
> > >> > >> >> > Var sized ints, this was discussed much earlier
> on, in
> > >> > fact I had
> > >> > >> >> > suggested it myself (with Hadoop references),
> the
> > >> > complexity of this
> > >> > >> >> > compared to having a simpler protocol was
> argued and
> > >> > agreed it
> > >> > >> wasn’t worth
> > >> > >> >> > the complexity as all other clients in other
> languages
> > >> > would need to
> > >> > >> ensure
> > >> > >> >> > theyre using the right var size algorithm, as
> there
> > is a
> > >> > few.
> > >> > >> >> >
> > >> > >> >> > On point 3)
> > >> > >> >> > We did the attributes, optional approach as
> originally
> > >> > there was
> > >> > >> marked
> > >> > >> >> > concern that headers would cause a message size
> > overhead
> > >> > for others,
> > >> > >> who
> > >> > >> >> > don’t want them. As such this is the clean
> solution to
> > >> > achieve that.
> > >> > >> If
> > >> > >> >> > that no longer holds, and we don’t care that we
> add
> > >> 4bytes
> > >> > overhead,
> > >> > >> then
> > >> > >> >> > im happy to remove.
> > >> > >> >> >
> > >> > >> >> > I’m personally in favour of keeping the message
> as
> > small
> > >> > as possible
> > >> > >> so
> > >> > >> >> > people don’t get shocks in perf and throughputs
> dues
> > to
> > >> > message size,
> > >> > >> >> > unless they actively use the feature, as such I
> do
> > >> prefer
> > >> > the
> > >> > >> attribute bit
> > >> > >> >> > wise feature flag approach myself.
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> > >> > jason@confluent.io> wrote:
> > >> > >> >> >
> > >> > >> >> > We have proposed a few significant changes
> to the
> > >> > message format
> > >> > >> in
> > >> > >> >> > KIP-98
> > >> > >> >> > which now seems likely to pass (perhaps
> with some
> > >> > iterations on
> > >> > >> >> > implementation details). It would be good
> to try
> > and
> > >> > coordinate
> > >> > >> the
> > >> > >> >> > changes
> > >> > >> >> > in both of the proposals to make sure they
> are
> > >> > consistent and
> > >> > >> >> > compatible.
> > >> > >> >> >
> > >> > >> >> > I think using the attributes to indicate
> null
> > >> headers
> > >> > is a
> > >> > >> reasonable
> > >> > >> >> > approach. We have proposed to do the same
> thing
> > for
> > >> > the message
> > >> > >> key and
> > >> > >> >> > value. That said, I sympathize with Jay's
> > argument.
> > >> > Having
> > >> > >> multiple
> > >> > >> >> > ways to
> > >> > >> >> > specify a null value increases the overall
> > >> complexity
> > >> > of the
> > >> > >> protocol.
> > >> > >> >> > You
> > >> > >> >> > can see this just from the fact that you
> need the
> > >> > extra verbiage
> > >> > >> in the
> > >> > >> >> > protocol specification in this KIP and in
> KIP-98
> > to
> > >> > describe the
> > >> > >> >> > dependence
> > >> > >> >> > between the fields and the attributes. It
> seems
> > >> like a
> > >> > slippery
> > >> > >> slope
> > >> > >> >> > if
> > >> > >> >> > you start allowing different request types
> to
> > >> > implement the
> > >> > >> protocol
> > >> > >> >> > specification differently.
> > >> > >> >> >
> > >> > >> >> > You can also argue that the messages
> already are
> > and
> > >> > are likely
> > >> > >> to
> > >> > >> >> > remain a
> > >> > >> >> > special case. For example, there is
> currently no
> > >> > generality in
> > >> > >> how
> > >> > >> >> > compressed message sets are represented
> that would
> > >> be
> > >> > applicable
> > >> > >> for
> > >> > >> >> > other
> > >> > >> >> > request types. Some might see this
> divergence as
> > an
> > >> > unfortunate
> > >> > >> >> > protocol
> > >> > >> >> > deficiency which should be fixed; others
> might see
> > >> it
> > >> > as sort of
> > >> > >> the
> > >> > >> >> > inevitability of needing to optimize where
> it
> > counts
> > >> > most. I'm
> > >> > >> probably
> > >> > >> >> > somewhere in between, but I think we
> probably all
> > >> > share the
> > >> > >> intuition
> > >> > >> >> > that
> > >> > >> >> > the protocol should be kept as consistent as
> > >> possible.
> > >> > With that
> > >> > >> in
> > >> > >> >> > mind,
> > >> > >> >> > here are a few comments:
> > >> > >> >> >
> > >> > >> >> > 1. One thing I found a little odd when
> reading the
> > >> > current
> > >> > >> proposal is
> > >> > >> >> > that
> > >> > >> >> > the headers are both represented as an
> array of
> > >> bytes
> > >> > and as an
> > >> > >> array
> > >> > >> >> > of
> > >> > >> >> > key/value pairs. I'd probably suggest
> something
> > like
> > >> > this:
> > >> > >> >> >
> > >> > >> >> > Headers => [HeaderKey HeaderValue]
> > >> > >> >> > HeaderKey => String
> > >> > >> >> > HeaderValue => Bytes
> > >> > >> >> >
> > >> > >> >> > An array in the Kafka protocol is
> represented as a
> > >> > 4-byte integer
> > >> > >> >> > indicating the number of elements in the
> array
> > >> > followed by the
> > >> > >> >> > serialization of the elements. Unless I'm
> > >> > misunderstanding, what
> > >> > >> you
> > >> > >> >> > have
> > >> > >> >> > instead is the total size of the headers in
> bytes
> > >> > followed by the
> > >> > >> >> > elements.
> > >> > >> >> > I'm not sure I see any reason for this
> > >> inconsistency.
> > >> > >> >> >
> > >> > >> >> > 2. In KIP-98, we've introduced
> variable-length
> > >> integer
> > >> > fields.
> > >> > >> >> > Effectively,
> > >> > >> >> > we've enriched (or "complicated" as Jay
> might say
> > ;)
> > >> > the protocol
> > >> > >> >> > specification to include the following
> types:
> > >> VarInt,
> > >> > VarLong,
> > >> > >> >> > UnsignedVarInt and UnsignedVarLong.
> > >> > >> >> >
> > >> > >> >> > Along with these primitives, we could
> introduce
> > the
> > >> > following
> > >> > >> types:
> > >> > >> >> >
> > >> > >> >> > VarSizeArray => NumberOfItems Item1 Item2
> .. ItemN
> > >> > >> >> > NumberOfItems => UnsignedVarInt
> > >> > >> >> >
> > >> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull
> Item1
> > >> > Item2 .. ItemN
> > >> > >> >> > NumberOfItemsOrNull => VarInt (-1 means
> null)
> > >> > >> >> >
> > >> > >> >> > And similarly for the `String` and `Bytes`
> types.
> > >> > These types
> > >> > >> can save
> > >> > >> >> > a
> > >> > >> >> > considerable amount of space in this
> proposal
> > >> because
> > >> > they can
> > >> > >> be used
> > >> > >> >> > for
> > >> > >> >> > both the number of headers included in the
> message
> > >> and
> > >> > the
> > >> > >> lengths of
> > >> > >> >> > the
> > >> > >> >> > header keys and values. We could do this
> instead:
> > >> > >> >> >
> > >> > >> >> > Headers => VarSizeArray[HeaderKey
> HeaderValue]
> > >> > >> >> > HeaderKey => VarSizeString
> > >> > >> >> > HeaderValue => VarSizeBytes
> > >> > >> >> >
> > >> > >> >> > Combining the savings from the use of
> variable
> > >> length
> > >> > fields, the
> > >> > >> >> > benefit
> > >> > >> >> > of using the attributes to represent null
> seems
> > >> pretty
> > >> > small.
> > >> > >> >> >
> > >> > >> >> > 3. Whichever way we go (whether we use the
> > >> attributes
> > >> > or not), we
> > >> > >> >> > should at
> > >> > >> >> > least be consistent between this KIP and
> KIP-98.
> > It
> > >> > would be very
> > >> > >> >> > strange
> > >> > >> >> > to have two ways to represent null values
> in the
> > >> same
> > >> > schema.
> > >> > >> Either
> > >> > >> >> > way is
> > >> > >> >> > OK with me. I think some message-level
> > optimizations
> > >> > are
> > >> > >> justifiable,
> > >> > >> >> > but
> > >> > >> >> > the savings here seem minimal (a few bytes
> per
> > >> > message), so
> > >> > >> maybe it's
> > >> > >> >> > not
> > >> > >> >> > worth the cost of letting the message
> diverge even
> > >> > further from
> > >> > >> the
> > >> > >> >> > rest of
> > >> > >> >> > the protocol.
> > >> > >> >> >
> > >> > >> >> > -Jason
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> > >> > >> radai.rosenblatt@gmail.com>
> > >> > >> >> > wrote:
> > >> > >> >> >
> > >> > >> >> > > I've trimmed the inline contents as this
> mail is
> > >> > getting too
> > >> > >> big for
> > >> > >> >> > the
> > >> > >> >> > > apache mailing list software to deliver
> :-(
> > >> > >> >> > >
> > >> > >> >> > > 1. the important thing for
> interoperability is
> > for
> > >> > different
> > >> > >> >> > "interested
> > >> > >> >> > > parties" (plugins, infra layers/wrappers,
> > >> user-code)
> > >> > to be
> > >> > >> able to
> > >> > >> >> > stick
> > >> > >> >> > > pieces of metadata onto msgs without
> getting in
> > >> each
> > >> > other's
> > >> > >> way. a
> > >> > >> >> > common
> > >> > >> >> > > key scheme (Strings, as of the time of
> this
> > >> > writing?) is all
> > >> > >> thats
> > >> > >> >> > required
> > >> > >> >> > > for that. it is assumed that the other end
> > >> > interested in any
> > >> > >> such
> > >> > >> >> > piece of
> > >> > >> >> > > metadata knows the encoding, and byte[]
> provides
> > >> for
> > >> > the most
> > >> > >> >> > flexibility.
> > >> > >> >> > > i believe this is the same logic behind
> core
> > kafka
> > >> > being
> > >> > >> >> > byte[]/byte[] -
> > >> > >> >> > > Strings are more "usable" but bytes are
> flexible
> > >> and
> > >> > so were
> > >> > >> chosen.
> > >> > >> >> > > Also - core kafka doesnt even do that
> good of a
> > >> job
> > >> > on
> > >> > >> usability of
> > >> > >> >> > the
> > >> > >> >> > > payload (example - i have to specify the
> nop
> > >> byte[]
> > >> > "decoders"
> > >> > >> >> > explicitly
> > >> > >> >> > > in conf), and again sacrificies usability
> for
> > the
> > >> > sake of
> > >> > >> >> > performance (no
> > >> > >> >> > > convenient single-record processing as
> poll is a
> > >> > batch, lots of
> > >> > >> >> > obscure
> > >> > >> >> > > little config details exposing internals
> of the
> > >> > batching
> > >> > >> mechanism,
> > >> > >> >> > etc)
> > >> > >> >> > >
> > >> > >> >> > > this is also why i really dislike the
> idea of a
> > >> > "type system"
> > >> > >> for
> > >> > >> >> > header
> > >> > >> >> > > values, it further degrades the
> usability, adds
> > >> > complexity and
> > >> > >> will
> > >> > >> >> > > eventually get in people's way, also, it
> would
> > be
> > >> > the 2nd/3rd
> > >> > >> >> > home-group
> > >> > >> >> > > serialization mechanism in core kafka
> (counting
> > 2
> > >> > iterations
> > >> > >> of the
> > >> > >> >> > "type
> > >> > >> >> > > definition DSL")
> > >> > >> >> > >
> > >> > >> >> > > 2. this is an implementation detail, and
> not
> > even
> > >> a
> > >> > very "user
> > >> > >> >> > facing" one?
> > >> > >> >> > > to the best of my understanding the vote
> process
> > >> is
> > >> > on proposed
> > >> > >> >> > > API/behaviour. also - since we're willing
> to go
> > >> with
> > >> > strings
> > >> > >> just
> > >> > >> >> > serialize
> > >> > >> >> > > a 0-sized header blob and IIUC you dont
> need any
> > >> > optionals
> > >> > >> anymore.
> > >> > >> >> > >
> > >> > >> >> > > 3. yes, we can :-)
> > >> > >> >> > >
> > >> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael
> > Pearce <
> > >> > >> >> > Michael.Pearce@ig.com>
> > >> > >> >> > > wrote:
> > >> > >> >> > >
> > >> > >> >> > > > Hi Jay,
> > >> > >> >> > > >
> > >> > >> >> > > > 1) There was some initial debate on the
> value
> > >> > part, as youll
> > >> > >> note
> > >> > >> >> > String,
> > >> > >> >> > > > String headers were discounted early
> on. The
> > >> > reason for this
> > >> > >> is
> > >> > >> >> > > flexibility
> > >> > >> >> > > > and keeping in line with the
> flexibility of
> > key,
> > >> > value of the
> > >> > >> >> > message
> > >> > >> >> > > > object itself. I don’t think it takes
> away
> > from
> > >> an
> > >> > ecosystem
> > >> > >> as
> > >> > >> >> > each
> > >> > >> >> > > plugin
> > >> > >> >> > > > will care for their own key, this way
> ints,
> > >> > booleans , exotic
> > >> > >> >> > custom
> > >> > >> >> > > binary
> > >> > >> >> > > > can all be catered for=.
> > >> > >> >> > > > a. If you really wanted to push for a
> typed
> > >> value
> > >> > interface,
> > >> > >> I
> > >> > >> >> > wouldn’t
> > >> > >> >> > > > want just String values supported, but
> the the
> > >> > primatives
> > >> > >> plus
> > >> > >> >> > string and
> > >> > >> >> > > > also still keeping the ability to have a
> > binary
> > >> > for custom
> > >> > >> >> > binaries that
> > >> > >> >> > > > some organisations may have.
> > >> > >> >> > > > i. I have written this slight
> alternative
> > here,
> > >> > >> >> > > https://cwiki.apache.org/
> > >> > >> >> > > > confluence/display/KAFKA/KIP-
> > >> > 82+-+Add+Record+Headers+-+Typed
> > >> > >> >> > > > ii. Essentially the value bytes, has a
> leading
> > >> > byte overhead.
> > >> > >> >> > > > 1. This tells you what type the value
> is,
> > >> before
> > >> > reading
> > >> > >> the rest
> > >> > >> >> > of the
> > >> > >> >> > > > bytes, allowing
> serialisation/deserialization
> > to
> > >> > and from the
> > >> > >> >> > primitives,
> > >> > >> >> > > > string and byte[]. This is akin to some
> other
> > >> > messaging
> > >> > >> systems.
> > >> > >> >> > > > 2) We are making it optional, so that
> for
> > those
> > >> > not wanting
> > >> > >> >> > headers have
> > >> > >> >> > > 0
> > >> > >> >> > > > bytes overhead (think of it as a feature
> > flag),
> > >> I
> > >> > don’t
> > >> > >> think this
> > >> > >> >> > is
> > >> > >> >> > > > complex, especially if comparing to
> changes
> > >> > proposed in
> > >> > >> other kips
> > >> > >> >> > like
> > >> > >> >> > > > kip-98.
> > >> > >> >> > > > a. If you really really don’t like
> this, we
> > can
> > >> > drop it, but
> > >> > >> it
> > >> > >> >> > would
> > >> > >> >> > > mean
> > >> > >> >> > > > buying into 4 bytes extra overhead for
> users
> > who
> > >> > do not want
> > >> > >> to use
> > >> > >> >> > > headers.
> > >> > >> >> > > > 3) In the summary yes, it is at a higher
> > level,
> > >> > but I think
> > >> > >> this
> > >> > >> >> > is well
> > >> > >> >> > > > documented in the proposed changes
> section.
> > >> > >> >> > > > a. Added getHeaders method to
> > Producer/Consumer
> > >> > record (that
> > >> > >> is it)
> > >> > >> >> > > > b. We’ve also detailed the new Headers
> class
> > >> that
> > >> > this method
> > >> > >> >> > returns
> > >> > >> >> > > that
> > >> > >> >> > > > encapsulates the headers protocol and
> logic.
> > >> > >> >> > > >
> > >> > >> >> > > > Best,
> > >> > >> >> > > > Mike
> > >> > >> >> > > >
> > >> > >> >> > > > ==Original questions from the vote
> thread from
> > >> > Jay.==
> > >> > >> >> > > >
> > >> > >> >> > > > Couple of things I think we still need
> to work
> > >> out:
> > >> > >> >> > > >
> > >> > >> >> > > > 1. I think we agree about the key,
> but I
> > >> think
> > >> > we haven't
> > >> > >> >> > talked about
> > >> > >> >> > > > the value yet. I think if our goal
> is an
> > open
> > >> > ecosystem
> > >> > >> of these
> > >> > >> >> > > header
> > >> > >> >> > > > spread across many plugins from many
> > systems
> > >> we
> > >> > should
> > >> > >> consider
> > >> > >> >> > making
> > >> > >> >> > > > this
> > >> > >> >> > > > a string as well so it can be
> printed, set
> > >> via
> > >> > a UI, set
> > >> > >> in
> > >> > >> >> > config,
> > >> > >> >> > > etc.
> > >> > >> >> > > > Basically encouraging pluggable
> > serialization
> > >> > formats
> > >> > >> here will
> > >> > >> >> > lead
> > >> > >> >> > > to
> > >> > >> >> > > > a
> > >> > >> >> > > > bit of a tower of babel.
> > >> > >> >> > > > 2. This proposal still includes a
> pretty
> > big
> > >> > change to our
> > >> > >> >> > > serialization
> > >> > >> >> > > > and protocol definition layer.
> Essentially
> > >> it is
> > >> > >> introducing an
> > >> > >> >> > > optional
> > >> > >> >> > > > type, where the format is data
> dependent. I
> > >> > think this is
> > >> > >> >> > actually a
> > >> > >> >> > > big
> > >> > >> >> > > > change though it doesn't seem like
> it. It
> > >> means
> > >> > you can no
> > >> > >> >> > longer
> > >> > >> >> > > > specify
> > >> > >> >> > > > this type with our type definition
> DSL, and
> > >> > likewise it
> > >> > >> requires
> > >> > >> >> > > custom
> > >> > >> >> > > > handling in client libs. This isn't
> a huge
> > >> > thing, since
> > >> > >> the
> > >> > >> >> > Record
> > >> > >> >> > > > definition is custom anyway, but I
> think
> > this
> > >> > kind of
> > >> > >> protocol
> > >> > >> >> > > > inconsistency is very non-desirable
> and
> > ties
> > >> > you to
> > >> > >> hand-coding
> > >> > >> >> > > things.
> > >> > >> >> > > > I
> > >> > >> >> > > > think the type should instead by [Key
> > Value]
> > >> in
> > >> > our BNF,
> > >> > >> where
> > >> > >> >> > key and
> > >> > >> >> > > > value are both short strings as used
> > >> elsewhere.
> > >> > This
> > >> > >> brings it
> > >> > >> >> > in line
> > >> > >> >> > > > with
> > >> > >> >> > > > the rest of the protocol.
> > >> > >> >> > > > 3. Could we get more specific about
> the
> > exact
> > >> > Java API
> > >> > >> change to
> > >> > >> >> > > > ProducerRecord, ConsumerRecord,
> Record,
> > etc?
> > >> > >> >> > > >
> > >> > >> >> > > > -Jay
> > >> > >> >> > > >
> > >> > >> >> > >
> > >> > >> >> >
> > >> > >> >> >
> > >> > >> >> > The information contained in this email is
> strictly
> > >> > confidential and
> > >> > >> for
> > >> > >> >> > the use of the addressee only, unless otherwise
> > >> indicated.
> > >> > If you
> > >> > >> are not
> > >> > >> >> > the intended recipient, please do not read,
> copy, use
> > or
> > >> > disclose to
> > >> > >> others
> > >> > >> >> > this message or any attachment. Please also
> notify the
> > >> > sender by
> > >> > >> replying
> > >> > >> >> > to this email or by telephone (+44(020 7896
> 0011) and
> > >> then
> > >> > delete
> > >> > >> the email
> > >> > >> >> > and any copies of it. Opinions, conclusion
> (etc) that
> > do
> > >> > not relate
> > >> > >> to the
> > >> > >> >> > official business of this company shall be
> understood
> > as
> > >> > neither
> > >> > >> given nor
> > >> > >> >> > endorsed by it. IG is a trading name of IG
> Markets
> > >> Limited
> > >> > (a company
> > >> > >> >> > registered in England and Wales, company number
> > >> 04008957)
> > >> > and IG
> > >> > >> Index
> > >> > >> >> > Limited (a company registered in England and
> Wales,
> > >> > company number
> > >> > >> >> > 01190902). Registered address at Cannon Bridge
> House,
> > 25
> > >> > Dowgate
> > >> > >> Hill,
> > >> > >> >> > London EC4R 2YA. Both IG Markets Limited
> (register
> > >> number
> > >> > 195355)
> > >> > >> and IG
> > >> > >> >> > Index Limited (register number 114059) are
> authorised
> > >> and
> > >> > regulated
> > >> > >> by the
> > >> > >> >> > Financial Conduct Authority.
> > >> > >> >> >
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Nacho - Ignacio Solis - isolis@igso.net
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > The information contained in this email is strictly
> confidential and
> > for
> > >> > the use of the addressee only, unless otherwise indicated. If
> you are
> > >> not
> > >> > the intended recipient, please do not read, copy, use or
> disclose to
> > >> others
> > >> > this message or any attachment. Please also notify the sender by
> > >> replying
> > >> > to this email or by telephone (+44(020 7896 0011) and then
> delete the
> > >> email
> > >> > and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > >> the
> > >> > official business of this company shall be understood as
> neither given
> > >> nor
> > >> > endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > >> > registered in England and Wales, company number 04008957) and
> IG Index
> > >> > Limited (a company registered in England and Wales, company
> number
> > >> > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill,
> > >> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and
> > IG
> > >> > Index Limited (register number 114059) are authorised and
> regulated by
> > >> the
> > >> > Financial Conduct Authority.
> > >> >
> > >> The information contained in this email is strictly confidential
> and for
> > >> the use of the addressee only, unless otherwise indicated. If you
> are
> > not
> > >> the intended recipient, please do not read, copy, use or disclose
> to
> > others
> > >> this message or any attachment. Please also notify the sender by
> > replying
> > >> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> > email
> > >> and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> > the
> > >> official business of this company shall be understood as neither
> given
> > nor
> > >> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> > >> registered in England and Wales, company number 04008957) and IG
> Index
> > >> Limited (a company registered in England and Wales, company number
> > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> > >> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> > >> Index Limited (register number 114059) are authorised and
> regulated by
> > the
> > >> Financial Conduct Authority.
> > >>
> > >
> > >
> >
> The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
I am happy to move the definition of the header into the message body, but would cause us not to lazy initialise/parse the headers, as obviously, we would have to traverse these reading the message.
This was actually one of Jay’s requests:
“ 2. I think we should think about creating the Map lazily to avoid
parsing out all the headers into little objects. HashMaps themselves are
kind of expensive and the consumer is very perf sensitive so and making
gazillions of hashmaps that may or may not get used is probably a bad idea.”
On 17/02/2017, 19:44, "Michael Pearce" <Mi...@ig.com> wrote:
Yes exactly we have access to the records thus why the header should be accessible via it and not hidden for only interceptors to access.
Sent using OWA for iPhone
________________________________________
From: Magnus Edenhill <ma...@edenhill.se>
Sent: Friday, February 17, 2017 7:34:49 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Big +1 on VarInts.
CPUs are fast, memory is slow.
I agree with Jason that we'll want to continue verifying messages,
including their headers, so while I appreciate the idea of the opaque
header blob it won't be useful in practice.
/Magnus
2017-02-17 10:41 GMT-08:00 Jason Gustafson <ja...@confluent.io>:
> Sorry, my mistake. The consumer interceptor is per batch, though I'm not
> sure that's an actual limitation since you still have access to the
> individual records.
>
> -Jason
>
> On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Re headers as byte array and future use by broker. This doesn't take away
> >> from that at all. Nor makes it difficult at all in my opinion.
> >
> >
> > Yeah, I didn't say it was difficult, only awkward. You wouldn't write the
> > schema that way if you were planning to use it on the brokers from the
> > beginning. Note also that one of the benefits of letting the broker
> > understand headers is that it can validate that they are properly
> > formatted. If cost is the only concern, we should confirm its impact
> > through performance testing.
> >
> > One of the key use cases requires access on consume at per event/message
> >> level at the point that message is being processed, as such the batch
> >> interceptors and batch consume api isn't suitable. It needs to be at the
> >> record level.
> >
> >
> > I'm not sure I understand the point about batching. Interceptors are
> > applied per-message, right?
> >
> > My intent on interceptors is to keep the usage of headers well-defined so
> > that they don't start leaking unnecessarily into applications. My guess
> is
> > that it's probably inevitable, but isolating it in the interceptors would
> > at least give people a second thought before deciding to use it. The main
> > challenge in my mind is figuring out how an MM use case would work. It
> > would be more cumbersome to replicate headers through an interceptor,
> > though arguably MM should be working at a lower level anyway.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> >> Re headers available on the record va interceptors only
> >>
> >> One of the key use cases requires access on consume at per event/message
> >> level at the point that message is being processed, as such the batch
> >> interceptors and batch consume api isn't suitable. It needs to be at the
> >> record level.
> >>
> >> This anyhow is similar to jms/http/amqp where headers are available to
> >> consuming applications.
> >>
> >> Re headers as byte array and future use by broker. This doesn't take
> away
> >> from that at all. Nor makes it difficult at all in my opinion.
> >>
> >>
> >>
> >> Sent using OWA for iPhone
> >> ________________________________________
> >> From: Jason Gustafson <ja...@confluent.io>
> >> Sent: Friday, February 17, 2017 5:55:42 PM
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>
> >> >
> >> > Would you be proposing in KIP-98 to convert the other message int’s
> (key
> >> > length, value length) also to varint to keep it uniform.
> >> > Also I assume there will be a static or helper method made to
> write/read
> >> > these in the client and server.
> >>
> >>
> >> Yes, that is what we are proposing, so using varints for headers would
> be
> >> consistent with the rest of the message. We have used static helper
> >> methods
> >> in our prototype implementation.
> >>
> >> The cost of parsing, we want to parse/interpret the headers lazily (this
> >> is
> >> > a key point brought up earlier in discussions)
> >>
> >>
> >> I'm a bit skeptical of this. Has anyone done the performance testing? I
> >> can
> >> probably implement it and test it if no one else has. I was also under
> the
> >> impression that there may be use cases down the road where the broker
> >> would
> >> need to interpret headers. That wouldn't be off the table in the future
> if
> >> it's represented as bytes, but it would be quite a bit more awkward,
> >> right?
> >>
> >> By the way, one question I have been wondering about. My understanding
> is
> >> that headers are primarily for use cases where a third-party components
> >> wants to enrich messages without needing to understand or modify the
> >> schema
> >> of the message key and value. For the applications which directly
> produce
> >> and consume the messages and control the key/value schema directly, it
> >> seems we would rather have them implement headers directly in their own
> >> schema. Supposing for the sake of argument that it was possible, my
> >> question is whether it be sufficient to expose the headers in the
> >> interceptor API and not in the common API?
> >>
> >> -Jason
> >>
> >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >> > On the point of varInts
> >> >
> >> > Would you be proposing in KIP-98 to convert the other message int’s
> (key
> >> > length, value length) also to varint to keep it uniform.
> >> > Also I assume there will be a static or helper method made to
> write/read
> >> > these in the client and server.
> >> >
> >> > Cheers
> >> > Mike
> >> >
> >> >
> >> >
> >> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >> >
> >> > On the point re: headers in the message protocol being a byte
> array
> >> > and not a count of elements followed by the elements. Again this was
> >> > discussed/argued previously.
> >> >
> >> > It was agreed on for a few reasons some of which you have
> obviously
> >> > picked up on:
> >> >
> >> > Broker is able to pass it through opaquely
> >> > The cost of parsing, we want to parse/interpret the headers lazily
> >> > (this is a key point brought up earlier in discussions)
> >> > Headers can be copied from consumer record to producer record (aka
> >> > mirror makers etc) without parsing if no changes are being made or
> being
> >> > looked at.
> >> > Keeps the broker agnostic to the format
> >> > You need an int32 either for the byte size of the headers, or for
> >> the
> >> > count of elements, so overheads are the same, but with going with an
> >> opaque
> >> > byte array has the above advantages.
> >> >
> >> > Cheers
> >> > Mike
> >> >
> >> >
> >> > On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >> >
> >> > Sorry, should have noted that the performance testing was done
> >> > using the
> >> > producer performance tool shipped with Kafka.
> >> >
> >> > -Jason
> >> >
> >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> >> > jason@confluent.io> wrote:
> >> >
> >> > > Hey Nacho,
> >> > >
> >> > > I've compared performance of our KIP-98 implementation with
> >> and
> >> > without
> >> > > varints. For messages around 128 bytes, we see an increase
> in
> >> > throughput of
> >> > > about 30% using the default configuration settings. At 256
> >> > bytes, the
> >> > > increase is around 16%. Obviously the performance converge
> as
> >> > messages get
> >> > > larger, but it seems well worth the cost. Note that we are
> >> also
> >> > seeing a
> >> > > substantial performance increase against trunk primarily
> >> because
> >> > of the
> >> > > much more efficient packing that varints provide us.
> Anything
> >> > adding to
> >> > > message overhead, such as record headers, would only
> increase
> >> > the relative
> >> > > difference. (Of course take these numbers with a grain of
> salt
> >> > since I have
> >> > > only used the default settings with both the producer and
> >> broker
> >> > on my
> >> > > local machine. We intend to provide more extensive
> performance
> >> > details as
> >> > > part of the work for KIP-98.)
> >> > >
> >> > > The implementation we are using is from protobuf (
> >> > > https://developers.google.com/
> protocol-buffers/docs/encoding
> >> ),
> >> > which is
> >> > > also used in HBase. It is trivial to implement and as far
> as I
> >> > know doesn't
> >> > > suffer from the aliasing problem you are describing. I
> checked
> >> > with Magnus
> >> > > (the author of librdkafka) and he agreed that the savings
> >> seemed
> >> > worth the
> >> > > cost of implementation.
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
> >> isolis@igso.net>
> >> > wrote:
> >> > >
> >> > >> -VarInts
> >> > >>
> >> > >> I'm one of the people (if not the most) opposed to VarInts.
> >> > VarInts
> >> > >> have a place, but this is not it. (We had a large
> >> discussion
> >> > about
> >> > >> them at the beginning of KIP-82 time)
> >> > >>
> >> > >> If anybody has real life performance numbers of VarInts
> >> > improving
> >> > >> things or significantly reducing resources I would like to
> >> know
> >> > what
> >> > >> that case may be. Yes, you can save some bytes here and
> >> there,
> >> > but
> >> > >> this is probably insignificant to the overall system
> behavior
> >> > and
> >> > >> storage requirements. -- I say this with respect to using
> >> > VarInts in
> >> > >> the protocol itself, not as part of the data.
> >> > >>
> >> > >> VarInts require you to parse the Int before using it and
> >> > depending on
> >> > >> the encoding they can suffer from aliasing (multiple
> >> > representations
> >> > >> for the same value).
> >> > >>
> >> > >> Why add complexity?
> >> > >>
> >> > >> Nacho
> >> > >>
> >> > >>
> >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> >> > cmccabe@apache.org>
> >> > >> wrote:
> >> > >> > +1 for varints here-- it would save quite a bit of space.
> >> > They are
> >> > >> > pretty quick to implement as well.
> >> > >> >
> >> > >> > I think it makes sense for values to be byte arrays.
> Users
> >> > might want
> >> > >> > to attach arbitrary payloads; they shouldn't be forced to
> >> > serialize
> >> > >> > everything to Java strings.
> >> > >> >
> >> > >> > best,
> >> > >> > Colin
> >> > >> >
> >> > >> >
> >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> >> > >> >> Hey Michael,
> >> > >> >>
> >> > >> >> Hmm, I guess the point of representing it as bytes is to
> >> > allow the
> >> > >> broker
> >> > >> >> to pass it through opaquely? Is the cost of parsing
> them a
> >> > concern, or
> >> > >> >> are
> >> > >> >> we simply trying to ensure that the broker stays
> agnostic
> >> to
> >> > the
> >> > >> format?
> >> > >> >>
> >> > >> >> On varints, I think adding support for them makes less
> >> sense
> >> > for an
> >> > >> >> isolated use case, but as part of a more holistic change
> >> > (such as what
> >> > >> we
> >> > >> >> have proposed in KIP-98), I think they are justifiable.
> If
> >> > we add them,
> >> > >> >> then the need to use attributes becomes quite a bit
> >> weaker,
> >> > right? The
> >> > >> >> other thing I find slightly odd is the fact that null
> >> > headers has no
> >> > >> >> actual
> >> > >> >> semantic meaning for the message (unlike null keys and
> >> > values). It is
> >> > >> >> just
> >> > >> >> a space optimization. It seems a bit better to always
> use
> >> > size 0 to
> >> > >> >> indicate having no headers.
> >> > >> >>
> >> > >> >> Overall, the main point is ensuring that the message
> >> schema
> >> > remains
> >> > >> >> consistent, either within the larger protocol, or at a
> >> > minimum within
> >> > >> the
> >> > >> >> message itself.
> >> > >> >>
> >> > >> >> -Jason
> >> > >> >>
> >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> >> > Michael.Pearce@ig.com
> >> > >> >
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >> > Hi Jason,
> >> > >> >> >
> >> > >> >> > On point 1) in the message protocol the headers are
> >> simply
> >> > a byte
> >> > >> array,
> >> > >> >> > as like the key or value, this is to clearly demarcate
> >> the
> >> > header in
> >> > >> the
> >> > >> >> > core message. Then the header byte array in the core
> >> > message is an
> >> > >> array of
> >> > >> >> > key, value pairs. This is what it is denoting.
> >> > >> >> >
> >> > >> >> > Then this would be I guess in the given notation:
> >> > >> >> >
> >> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> >> > >> >> > KeyLength => int32 <-----------------NEW size of
> the
> >> > byte[] of
> >> > >> the
> >> > >> >> > serialised key value
> >> > >> >> > Key => bytes <---------------------- NEW
> serialised
> >> > string (UTF8)
> >> > >> >> > bytes of the header key
> >> > >> >> > ValueLength => int32 <-------------- NEW size of
> the
> >> > byte[] of
> >> > >> the
> >> > >> >> > serialised header value
> >> > >> >> > Value => bytes <-------------------- NEW
> serialised
> >> > form of the
> >> > >> header
> >> > >> >> > value
> >> > >> >> >
> >> > >> >> > The key length and value length is matching the way
> the
> >> > protocol is
> >> > >> >> > defined in the core message currently.
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On point 2)
> >> > >> >> > Var sized ints, this was discussed much earlier on, in
> >> > fact I had
> >> > >> >> > suggested it myself (with Hadoop references), the
> >> > complexity of this
> >> > >> >> > compared to having a simpler protocol was argued and
> >> > agreed it
> >> > >> wasn’t worth
> >> > >> >> > the complexity as all other clients in other languages
> >> > would need to
> >> > >> ensure
> >> > >> >> > theyre using the right var size algorithm, as there
> is a
> >> > few.
> >> > >> >> >
> >> > >> >> > On point 3)
> >> > >> >> > We did the attributes, optional approach as originally
> >> > there was
> >> > >> marked
> >> > >> >> > concern that headers would cause a message size
> overhead
> >> > for others,
> >> > >> who
> >> > >> >> > don’t want them. As such this is the clean solution to
> >> > achieve that.
> >> > >> If
> >> > >> >> > that no longer holds, and we don’t care that we add
> >> 4bytes
> >> > overhead,
> >> > >> then
> >> > >> >> > im happy to remove.
> >> > >> >> >
> >> > >> >> > I’m personally in favour of keeping the message as
> small
> >> > as possible
> >> > >> so
> >> > >> >> > people don’t get shocks in perf and throughputs dues
> to
> >> > message size,
> >> > >> >> > unless they actively use the feature, as such I do
> >> prefer
> >> > the
> >> > >> attribute bit
> >> > >> >> > wise feature flag approach myself.
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> >> > jason@confluent.io> wrote:
> >> > >> >> >
> >> > >> >> > We have proposed a few significant changes to the
> >> > message format
> >> > >> in
> >> > >> >> > KIP-98
> >> > >> >> > which now seems likely to pass (perhaps with some
> >> > iterations on
> >> > >> >> > implementation details). It would be good to try
> and
> >> > coordinate
> >> > >> the
> >> > >> >> > changes
> >> > >> >> > in both of the proposals to make sure they are
> >> > consistent and
> >> > >> >> > compatible.
> >> > >> >> >
> >> > >> >> > I think using the attributes to indicate null
> >> headers
> >> > is a
> >> > >> reasonable
> >> > >> >> > approach. We have proposed to do the same thing
> for
> >> > the message
> >> > >> key and
> >> > >> >> > value. That said, I sympathize with Jay's
> argument.
> >> > Having
> >> > >> multiple
> >> > >> >> > ways to
> >> > >> >> > specify a null value increases the overall
> >> complexity
> >> > of the
> >> > >> protocol.
> >> > >> >> > You
> >> > >> >> > can see this just from the fact that you need the
> >> > extra verbiage
> >> > >> in the
> >> > >> >> > protocol specification in this KIP and in KIP-98
> to
> >> > describe the
> >> > >> >> > dependence
> >> > >> >> > between the fields and the attributes. It seems
> >> like a
> >> > slippery
> >> > >> slope
> >> > >> >> > if
> >> > >> >> > you start allowing different request types to
> >> > implement the
> >> > >> protocol
> >> > >> >> > specification differently.
> >> > >> >> >
> >> > >> >> > You can also argue that the messages already are
> and
> >> > are likely
> >> > >> to
> >> > >> >> > remain a
> >> > >> >> > special case. For example, there is currently no
> >> > generality in
> >> > >> how
> >> > >> >> > compressed message sets are represented that would
> >> be
> >> > applicable
> >> > >> for
> >> > >> >> > other
> >> > >> >> > request types. Some might see this divergence as
> an
> >> > unfortunate
> >> > >> >> > protocol
> >> > >> >> > deficiency which should be fixed; others might see
> >> it
> >> > as sort of
> >> > >> the
> >> > >> >> > inevitability of needing to optimize where it
> counts
> >> > most. I'm
> >> > >> probably
> >> > >> >> > somewhere in between, but I think we probably all
> >> > share the
> >> > >> intuition
> >> > >> >> > that
> >> > >> >> > the protocol should be kept as consistent as
> >> possible.
> >> > With that
> >> > >> in
> >> > >> >> > mind,
> >> > >> >> > here are a few comments:
> >> > >> >> >
> >> > >> >> > 1. One thing I found a little odd when reading the
> >> > current
> >> > >> proposal is
> >> > >> >> > that
> >> > >> >> > the headers are both represented as an array of
> >> bytes
> >> > and as an
> >> > >> array
> >> > >> >> > of
> >> > >> >> > key/value pairs. I'd probably suggest something
> like
> >> > this:
> >> > >> >> >
> >> > >> >> > Headers => [HeaderKey HeaderValue]
> >> > >> >> > HeaderKey => String
> >> > >> >> > HeaderValue => Bytes
> >> > >> >> >
> >> > >> >> > An array in the Kafka protocol is represented as a
> >> > 4-byte integer
> >> > >> >> > indicating the number of elements in the array
> >> > followed by the
> >> > >> >> > serialization of the elements. Unless I'm
> >> > misunderstanding, what
> >> > >> you
> >> > >> >> > have
> >> > >> >> > instead is the total size of the headers in bytes
> >> > followed by the
> >> > >> >> > elements.
> >> > >> >> > I'm not sure I see any reason for this
> >> inconsistency.
> >> > >> >> >
> >> > >> >> > 2. In KIP-98, we've introduced variable-length
> >> integer
> >> > fields.
> >> > >> >> > Effectively,
> >> > >> >> > we've enriched (or "complicated" as Jay might say
> ;)
> >> > the protocol
> >> > >> >> > specification to include the following types:
> >> VarInt,
> >> > VarLong,
> >> > >> >> > UnsignedVarInt and UnsignedVarLong.
> >> > >> >> >
> >> > >> >> > Along with these primitives, we could introduce
> the
> >> > following
> >> > >> types:
> >> > >> >> >
> >> > >> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> >> > >> >> > NumberOfItems => UnsignedVarInt
> >> > >> >> >
> >> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1
> >> > Item2 .. ItemN
> >> > >> >> > NumberOfItemsOrNull => VarInt (-1 means null)
> >> > >> >> >
> >> > >> >> > And similarly for the `String` and `Bytes` types.
> >> > These types
> >> > >> can save
> >> > >> >> > a
> >> > >> >> > considerable amount of space in this proposal
> >> because
> >> > they can
> >> > >> be used
> >> > >> >> > for
> >> > >> >> > both the number of headers included in the message
> >> and
> >> > the
> >> > >> lengths of
> >> > >> >> > the
> >> > >> >> > header keys and values. We could do this instead:
> >> > >> >> >
> >> > >> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
> >> > >> >> > HeaderKey => VarSizeString
> >> > >> >> > HeaderValue => VarSizeBytes
> >> > >> >> >
> >> > >> >> > Combining the savings from the use of variable
> >> length
> >> > fields, the
> >> > >> >> > benefit
> >> > >> >> > of using the attributes to represent null seems
> >> pretty
> >> > small.
> >> > >> >> >
> >> > >> >> > 3. Whichever way we go (whether we use the
> >> attributes
> >> > or not), we
> >> > >> >> > should at
> >> > >> >> > least be consistent between this KIP and KIP-98.
> It
> >> > would be very
> >> > >> >> > strange
> >> > >> >> > to have two ways to represent null values in the
> >> same
> >> > schema.
> >> > >> Either
> >> > >> >> > way is
> >> > >> >> > OK with me. I think some message-level
> optimizations
> >> > are
> >> > >> justifiable,
> >> > >> >> > but
> >> > >> >> > the savings here seem minimal (a few bytes per
> >> > message), so
> >> > >> maybe it's
> >> > >> >> > not
> >> > >> >> > worth the cost of letting the message diverge even
> >> > further from
> >> > >> the
> >> > >> >> > rest of
> >> > >> >> > the protocol.
> >> > >> >> >
> >> > >> >> > -Jason
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> >> > >> radai.rosenblatt@gmail.com>
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > I've trimmed the inline contents as this mail is
> >> > getting too
> >> > >> big for
> >> > >> >> > the
> >> > >> >> > > apache mailing list software to deliver :-(
> >> > >> >> > >
> >> > >> >> > > 1. the important thing for interoperability is
> for
> >> > different
> >> > >> >> > "interested
> >> > >> >> > > parties" (plugins, infra layers/wrappers,
> >> user-code)
> >> > to be
> >> > >> able to
> >> > >> >> > stick
> >> > >> >> > > pieces of metadata onto msgs without getting in
> >> each
> >> > other's
> >> > >> way. a
> >> > >> >> > common
> >> > >> >> > > key scheme (Strings, as of the time of this
> >> > writing?) is all
> >> > >> thats
> >> > >> >> > required
> >> > >> >> > > for that. it is assumed that the other end
> >> > interested in any
> >> > >> such
> >> > >> >> > piece of
> >> > >> >> > > metadata knows the encoding, and byte[] provides
> >> for
> >> > the most
> >> > >> >> > flexibility.
> >> > >> >> > > i believe this is the same logic behind core
> kafka
> >> > being
> >> > >> >> > byte[]/byte[] -
> >> > >> >> > > Strings are more "usable" but bytes are flexible
> >> and
> >> > so were
> >> > >> chosen.
> >> > >> >> > > Also - core kafka doesnt even do that good of a
> >> job
> >> > on
> >> > >> usability of
> >> > >> >> > the
> >> > >> >> > > payload (example - i have to specify the nop
> >> byte[]
> >> > "decoders"
> >> > >> >> > explicitly
> >> > >> >> > > in conf), and again sacrificies usability for
> the
> >> > sake of
> >> > >> >> > performance (no
> >> > >> >> > > convenient single-record processing as poll is a
> >> > batch, lots of
> >> > >> >> > obscure
> >> > >> >> > > little config details exposing internals of the
> >> > batching
> >> > >> mechanism,
> >> > >> >> > etc)
> >> > >> >> > >
> >> > >> >> > > this is also why i really dislike the idea of a
> >> > "type system"
> >> > >> for
> >> > >> >> > header
> >> > >> >> > > values, it further degrades the usability, adds
> >> > complexity and
> >> > >> will
> >> > >> >> > > eventually get in people's way, also, it would
> be
> >> > the 2nd/3rd
> >> > >> >> > home-group
> >> > >> >> > > serialization mechanism in core kafka (counting
> 2
> >> > iterations
> >> > >> of the
> >> > >> >> > "type
> >> > >> >> > > definition DSL")
> >> > >> >> > >
> >> > >> >> > > 2. this is an implementation detail, and not
> even
> >> a
> >> > very "user
> >> > >> >> > facing" one?
> >> > >> >> > > to the best of my understanding the vote process
> >> is
> >> > on proposed
> >> > >> >> > > API/behaviour. also - since we're willing to go
> >> with
> >> > strings
> >> > >> just
> >> > >> >> > serialize
> >> > >> >> > > a 0-sized header blob and IIUC you dont need any
> >> > optionals
> >> > >> anymore.
> >> > >> >> > >
> >> > >> >> > > 3. yes, we can :-)
> >> > >> >> > >
> >> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael
> Pearce <
> >> > >> >> > Michael.Pearce@ig.com>
> >> > >> >> > > wrote:
> >> > >> >> > >
> >> > >> >> > > > Hi Jay,
> >> > >> >> > > >
> >> > >> >> > > > 1) There was some initial debate on the value
> >> > part, as youll
> >> > >> note
> >> > >> >> > String,
> >> > >> >> > > > String headers were discounted early on. The
> >> > reason for this
> >> > >> is
> >> > >> >> > > flexibility
> >> > >> >> > > > and keeping in line with the flexibility of
> key,
> >> > value of the
> >> > >> >> > message
> >> > >> >> > > > object itself. I don’t think it takes away
> from
> >> an
> >> > ecosystem
> >> > >> as
> >> > >> >> > each
> >> > >> >> > > plugin
> >> > >> >> > > > will care for their own key, this way ints,
> >> > booleans , exotic
> >> > >> >> > custom
> >> > >> >> > > binary
> >> > >> >> > > > can all be catered for=.
> >> > >> >> > > > a. If you really wanted to push for a typed
> >> value
> >> > interface,
> >> > >> I
> >> > >> >> > wouldn’t
> >> > >> >> > > > want just String values supported, but the the
> >> > primatives
> >> > >> plus
> >> > >> >> > string and
> >> > >> >> > > > also still keeping the ability to have a
> binary
> >> > for custom
> >> > >> >> > binaries that
> >> > >> >> > > > some organisations may have.
> >> > >> >> > > > i. I have written this slight alternative
> here,
> >> > >> >> > > https://cwiki.apache.org/
> >> > >> >> > > > confluence/display/KAFKA/KIP-
> >> > 82+-+Add+Record+Headers+-+Typed
> >> > >> >> > > > ii. Essentially the value bytes, has a leading
> >> > byte overhead.
> >> > >> >> > > > 1. This tells you what type the value is,
> >> before
> >> > reading
> >> > >> the rest
> >> > >> >> > of the
> >> > >> >> > > > bytes, allowing serialisation/deserialization
> to
> >> > and from the
> >> > >> >> > primitives,
> >> > >> >> > > > string and byte[]. This is akin to some other
> >> > messaging
> >> > >> systems.
> >> > >> >> > > > 2) We are making it optional, so that for
> those
> >> > not wanting
> >> > >> >> > headers have
> >> > >> >> > > 0
> >> > >> >> > > > bytes overhead (think of it as a feature
> flag),
> >> I
> >> > don’t
> >> > >> think this
> >> > >> >> > is
> >> > >> >> > > > complex, especially if comparing to changes
> >> > proposed in
> >> > >> other kips
> >> > >> >> > like
> >> > >> >> > > > kip-98.
> >> > >> >> > > > a. If you really really don’t like this, we
> can
> >> > drop it, but
> >> > >> it
> >> > >> >> > would
> >> > >> >> > > mean
> >> > >> >> > > > buying into 4 bytes extra overhead for users
> who
> >> > do not want
> >> > >> to use
> >> > >> >> > > headers.
> >> > >> >> > > > 3) In the summary yes, it is at a higher
> level,
> >> > but I think
> >> > >> this
> >> > >> >> > is well
> >> > >> >> > > > documented in the proposed changes section.
> >> > >> >> > > > a. Added getHeaders method to
> Producer/Consumer
> >> > record (that
> >> > >> is it)
> >> > >> >> > > > b. We’ve also detailed the new Headers class
> >> that
> >> > this method
> >> > >> >> > returns
> >> > >> >> > > that
> >> > >> >> > > > encapsulates the headers protocol and logic.
> >> > >> >> > > >
> >> > >> >> > > > Best,
> >> > >> >> > > > Mike
> >> > >> >> > > >
> >> > >> >> > > > ==Original questions from the vote thread from
> >> > Jay.==
> >> > >> >> > > >
> >> > >> >> > > > Couple of things I think we still need to work
> >> out:
> >> > >> >> > > >
> >> > >> >> > > > 1. I think we agree about the key, but I
> >> think
> >> > we haven't
> >> > >> >> > talked about
> >> > >> >> > > > the value yet. I think if our goal is an
> open
> >> > ecosystem
> >> > >> of these
> >> > >> >> > > header
> >> > >> >> > > > spread across many plugins from many
> systems
> >> we
> >> > should
> >> > >> consider
> >> > >> >> > making
> >> > >> >> > > > this
> >> > >> >> > > > a string as well so it can be printed, set
> >> via
> >> > a UI, set
> >> > >> in
> >> > >> >> > config,
> >> > >> >> > > etc.
> >> > >> >> > > > Basically encouraging pluggable
> serialization
> >> > formats
> >> > >> here will
> >> > >> >> > lead
> >> > >> >> > > to
> >> > >> >> > > > a
> >> > >> >> > > > bit of a tower of babel.
> >> > >> >> > > > 2. This proposal still includes a pretty
> big
> >> > change to our
> >> > >> >> > > serialization
> >> > >> >> > > > and protocol definition layer. Essentially
> >> it is
> >> > >> introducing an
> >> > >> >> > > optional
> >> > >> >> > > > type, where the format is data dependent. I
> >> > think this is
> >> > >> >> > actually a
> >> > >> >> > > big
> >> > >> >> > > > change though it doesn't seem like it. It
> >> means
> >> > you can no
> >> > >> >> > longer
> >> > >> >> > > > specify
> >> > >> >> > > > this type with our type definition DSL, and
> >> > likewise it
> >> > >> requires
> >> > >> >> > > custom
> >> > >> >> > > > handling in client libs. This isn't a huge
> >> > thing, since
> >> > >> the
> >> > >> >> > Record
> >> > >> >> > > > definition is custom anyway, but I think
> this
> >> > kind of
> >> > >> protocol
> >> > >> >> > > > inconsistency is very non-desirable and
> ties
> >> > you to
> >> > >> hand-coding
> >> > >> >> > > things.
> >> > >> >> > > > I
> >> > >> >> > > > think the type should instead by [Key
> Value]
> >> in
> >> > our BNF,
> >> > >> where
> >> > >> >> > key and
> >> > >> >> > > > value are both short strings as used
> >> elsewhere.
> >> > This
> >> > >> brings it
> >> > >> >> > in line
> >> > >> >> > > > with
> >> > >> >> > > > the rest of the protocol.
> >> > >> >> > > > 3. Could we get more specific about the
> exact
> >> > Java API
> >> > >> change to
> >> > >> >> > > > ProducerRecord, ConsumerRecord, Record,
> etc?
> >> > >> >> > > >
> >> > >> >> > > > -Jay
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > The information contained in this email is strictly
> >> > confidential and
> >> > >> for
> >> > >> >> > the use of the addressee only, unless otherwise
> >> indicated.
> >> > If you
> >> > >> are not
> >> > >> >> > the intended recipient, please do not read, copy, use
> or
> >> > disclose to
> >> > >> others
> >> > >> >> > this message or any attachment. Please also notify the
> >> > sender by
> >> > >> replying
> >> > >> >> > to this email or by telephone (+44(020 7896 0011) and
> >> then
> >> > delete
> >> > >> the email
> >> > >> >> > and any copies of it. Opinions, conclusion (etc) that
> do
> >> > not relate
> >> > >> to the
> >> > >> >> > official business of this company shall be understood
> as
> >> > neither
> >> > >> given nor
> >> > >> >> > endorsed by it. IG is a trading name of IG Markets
> >> Limited
> >> > (a company
> >> > >> >> > registered in England and Wales, company number
> >> 04008957)
> >> > and IG
> >> > >> Index
> >> > >> >> > Limited (a company registered in England and Wales,
> >> > company number
> >> > >> >> > 01190902). Registered address at Cannon Bridge House,
> 25
> >> > Dowgate
> >> > >> Hill,
> >> > >> >> > London EC4R 2YA. Both IG Markets Limited (register
> >> number
> >> > 195355)
> >> > >> and IG
> >> > >> >> > Index Limited (register number 114059) are authorised
> >> and
> >> > regulated
> >> > >> by the
> >> > >> >> > Financial Conduct Authority.
> >> > >> >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Nacho - Ignacio Solis - isolis@igso.net
> >> > >>
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >> > The information contained in this email is strictly confidential and
> for
> >> > the use of the addressee only, unless otherwise indicated. If you are
> >> not
> >> > the intended recipient, please do not read, copy, use or disclose to
> >> others
> >> > this message or any attachment. Please also notify the sender by
> >> replying
> >> > to this email or by telephone (+44(020 7896 0011) and then delete the
> >> email
> >> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> >> the
> >> > official business of this company shall be understood as neither given
> >> nor
> >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> > registered in England and Wales, company number 04008957) and IG Index
> >> > Limited (a company registered in England and Wales, company number
> >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >> > Index Limited (register number 114059) are authorised and regulated by
> >> the
> >> > Financial Conduct Authority.
> >> >
> >> The information contained in this email is strictly confidential and for
> >> the use of the addressee only, unless otherwise indicated. If you are
> not
> >> the intended recipient, please do not read, copy, use or disclose to
> others
> >> this message or any attachment. Please also notify the sender by
> replying
> >> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >> official business of this company shall be understood as neither given
> nor
> >> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> registered in England and Wales, company number 04008957) and IG Index
> >> Limited (a company registered in England and Wales, company number
> >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> >> Index Limited (register number 114059) are authorised and regulated by
> the
> >> Financial Conduct Authority.
> >>
> >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Yes exactly we have access to the records thus why the header should be accessible via it and not hidden for only interceptors to access.
Sent using OWA for iPhone
________________________________________
From: Magnus Edenhill <ma...@edenhill.se>
Sent: Friday, February 17, 2017 7:34:49 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
Big +1 on VarInts.
CPUs are fast, memory is slow.
I agree with Jason that we'll want to continue verifying messages,
including their headers, so while I appreciate the idea of the opaque
header blob it won't be useful in practice.
/Magnus
2017-02-17 10:41 GMT-08:00 Jason Gustafson <ja...@confluent.io>:
> Sorry, my mistake. The consumer interceptor is per batch, though I'm not
> sure that's an actual limitation since you still have access to the
> individual records.
>
> -Jason
>
> On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Re headers as byte array and future use by broker. This doesn't take away
> >> from that at all. Nor makes it difficult at all in my opinion.
> >
> >
> > Yeah, I didn't say it was difficult, only awkward. You wouldn't write the
> > schema that way if you were planning to use it on the brokers from the
> > beginning. Note also that one of the benefits of letting the broker
> > understand headers is that it can validate that they are properly
> > formatted. If cost is the only concern, we should confirm its impact
> > through performance testing.
> >
> > One of the key use cases requires access on consume at per event/message
> >> level at the point that message is being processed, as such the batch
> >> interceptors and batch consume api isn't suitable. It needs to be at the
> >> record level.
> >
> >
> > I'm not sure I understand the point about batching. Interceptors are
> > applied per-message, right?
> >
> > My intent on interceptors is to keep the usage of headers well-defined so
> > that they don't start leaking unnecessarily into applications. My guess
> is
> > that it's probably inevitable, but isolating it in the interceptors would
> > at least give people a second thought before deciding to use it. The main
> > challenge in my mind is figuring out how an MM use case would work. It
> > would be more cumbersome to replicate headers through an interceptor,
> > though arguably MM should be working at a lower level anyway.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> >> Re headers available on the record va interceptors only
> >>
> >> One of the key use cases requires access on consume at per event/message
> >> level at the point that message is being processed, as such the batch
> >> interceptors and batch consume api isn't suitable. It needs to be at the
> >> record level.
> >>
> >> This anyhow is similar to jms/http/amqp where headers are available to
> >> consuming applications.
> >>
> >> Re headers as byte array and future use by broker. This doesn't take
> away
> >> from that at all. Nor makes it difficult at all in my opinion.
> >>
> >>
> >>
> >> Sent using OWA for iPhone
> >> ________________________________________
> >> From: Jason Gustafson <ja...@confluent.io>
> >> Sent: Friday, February 17, 2017 5:55:42 PM
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>
> >> >
> >> > Would you be proposing in KIP-98 to convert the other message int’s
> (key
> >> > length, value length) also to varint to keep it uniform.
> >> > Also I assume there will be a static or helper method made to
> write/read
> >> > these in the client and server.
> >>
> >>
> >> Yes, that is what we are proposing, so using varints for headers would
> be
> >> consistent with the rest of the message. We have used static helper
> >> methods
> >> in our prototype implementation.
> >>
> >> The cost of parsing, we want to parse/interpret the headers lazily (this
> >> is
> >> > a key point brought up earlier in discussions)
> >>
> >>
> >> I'm a bit skeptical of this. Has anyone done the performance testing? I
> >> can
> >> probably implement it and test it if no one else has. I was also under
> the
> >> impression that there may be use cases down the road where the broker
> >> would
> >> need to interpret headers. That wouldn't be off the table in the future
> if
> >> it's represented as bytes, but it would be quite a bit more awkward,
> >> right?
> >>
> >> By the way, one question I have been wondering about. My understanding
> is
> >> that headers are primarily for use cases where a third-party components
> >> wants to enrich messages without needing to understand or modify the
> >> schema
> >> of the message key and value. For the applications which directly
> produce
> >> and consume the messages and control the key/value schema directly, it
> >> seems we would rather have them implement headers directly in their own
> >> schema. Supposing for the sake of argument that it was possible, my
> >> question is whether it be sufficient to expose the headers in the
> >> interceptor API and not in the common API?
> >>
> >> -Jason
> >>
> >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >> > On the point of varInts
> >> >
> >> > Would you be proposing in KIP-98 to convert the other message int’s
> (key
> >> > length, value length) also to varint to keep it uniform.
> >> > Also I assume there will be a static or helper method made to
> write/read
> >> > these in the client and server.
> >> >
> >> > Cheers
> >> > Mike
> >> >
> >> >
> >> >
> >> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >> >
> >> > On the point re: headers in the message protocol being a byte
> array
> >> > and not a count of elements followed by the elements. Again this was
> >> > discussed/argued previously.
> >> >
> >> > It was agreed on for a few reasons some of which you have
> obviously
> >> > picked up on:
> >> >
> >> > Broker is able to pass it through opaquely
> >> > The cost of parsing, we want to parse/interpret the headers lazily
> >> > (this is a key point brought up earlier in discussions)
> >> > Headers can be copied from consumer record to producer record (aka
> >> > mirror makers etc) without parsing if no changes are being made or
> being
> >> > looked at.
> >> > Keeps the broker agnostic to the format
> >> > You need an int32 either for the byte size of the headers, or for
> >> the
> >> > count of elements, so overheads are the same, but with going with an
> >> opaque
> >> > byte array has the above advantages.
> >> >
> >> > Cheers
> >> > Mike
> >> >
> >> >
> >> > On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >> >
> >> > Sorry, should have noted that the performance testing was done
> >> > using the
> >> > producer performance tool shipped with Kafka.
> >> >
> >> > -Jason
> >> >
> >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> >> > jason@confluent.io> wrote:
> >> >
> >> > > Hey Nacho,
> >> > >
> >> > > I've compared performance of our KIP-98 implementation with
> >> and
> >> > without
> >> > > varints. For messages around 128 bytes, we see an increase
> in
> >> > throughput of
> >> > > about 30% using the default configuration settings. At 256
> >> > bytes, the
> >> > > increase is around 16%. Obviously the performance converge
> as
> >> > messages get
> >> > > larger, but it seems well worth the cost. Note that we are
> >> also
> >> > seeing a
> >> > > substantial performance increase against trunk primarily
> >> because
> >> > of the
> >> > > much more efficient packing that varints provide us.
> Anything
> >> > adding to
> >> > > message overhead, such as record headers, would only
> increase
> >> > the relative
> >> > > difference. (Of course take these numbers with a grain of
> salt
> >> > since I have
> >> > > only used the default settings with both the producer and
> >> broker
> >> > on my
> >> > > local machine. We intend to provide more extensive
> performance
> >> > details as
> >> > > part of the work for KIP-98.)
> >> > >
> >> > > The implementation we are using is from protobuf (
> >> > > https://developers.google.com/
> protocol-buffers/docs/encoding
> >> ),
> >> > which is
> >> > > also used in HBase. It is trivial to implement and as far
> as I
> >> > know doesn't
> >> > > suffer from the aliasing problem you are describing. I
> checked
> >> > with Magnus
> >> > > (the author of librdkafka) and he agreed that the savings
> >> seemed
> >> > worth the
> >> > > cost of implementation.
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
> >> isolis@igso.net>
> >> > wrote:
> >> > >
> >> > >> -VarInts
> >> > >>
> >> > >> I'm one of the people (if not the most) opposed to VarInts.
> >> > VarInts
> >> > >> have a place, but this is not it. (We had a large
> >> discussion
> >> > about
> >> > >> them at the beginning of KIP-82 time)
> >> > >>
> >> > >> If anybody has real life performance numbers of VarInts
> >> > improving
> >> > >> things or significantly reducing resources I would like to
> >> know
> >> > what
> >> > >> that case may be. Yes, you can save some bytes here and
> >> there,
> >> > but
> >> > >> this is probably insignificant to the overall system
> behavior
> >> > and
> >> > >> storage requirements. -- I say this with respect to using
> >> > VarInts in
> >> > >> the protocol itself, not as part of the data.
> >> > >>
> >> > >> VarInts require you to parse the Int before using it and
> >> > depending on
> >> > >> the encoding they can suffer from aliasing (multiple
> >> > representations
> >> > >> for the same value).
> >> > >>
> >> > >> Why add complexity?
> >> > >>
> >> > >> Nacho
> >> > >>
> >> > >>
> >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> >> > cmccabe@apache.org>
> >> > >> wrote:
> >> > >> > +1 for varints here-- it would save quite a bit of space.
> >> > They are
> >> > >> > pretty quick to implement as well.
> >> > >> >
> >> > >> > I think it makes sense for values to be byte arrays.
> Users
> >> > might want
> >> > >> > to attach arbitrary payloads; they shouldn't be forced to
> >> > serialize
> >> > >> > everything to Java strings.
> >> > >> >
> >> > >> > best,
> >> > >> > Colin
> >> > >> >
> >> > >> >
> >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> >> > >> >> Hey Michael,
> >> > >> >>
> >> > >> >> Hmm, I guess the point of representing it as bytes is to
> >> > allow the
> >> > >> broker
> >> > >> >> to pass it through opaquely? Is the cost of parsing
> them a
> >> > concern, or
> >> > >> >> are
> >> > >> >> we simply trying to ensure that the broker stays
> agnostic
> >> to
> >> > the
> >> > >> format?
> >> > >> >>
> >> > >> >> On varints, I think adding support for them makes less
> >> sense
> >> > for an
> >> > >> >> isolated use case, but as part of a more holistic change
> >> > (such as what
> >> > >> we
> >> > >> >> have proposed in KIP-98), I think they are justifiable.
> If
> >> > we add them,
> >> > >> >> then the need to use attributes becomes quite a bit
> >> weaker,
> >> > right? The
> >> > >> >> other thing I find slightly odd is the fact that null
> >> > headers has no
> >> > >> >> actual
> >> > >> >> semantic meaning for the message (unlike null keys and
> >> > values). It is
> >> > >> >> just
> >> > >> >> a space optimization. It seems a bit better to always
> use
> >> > size 0 to
> >> > >> >> indicate having no headers.
> >> > >> >>
> >> > >> >> Overall, the main point is ensuring that the message
> >> schema
> >> > remains
> >> > >> >> consistent, either within the larger protocol, or at a
> >> > minimum within
> >> > >> the
> >> > >> >> message itself.
> >> > >> >>
> >> > >> >> -Jason
> >> > >> >>
> >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> >> > Michael.Pearce@ig.com
> >> > >> >
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >> > Hi Jason,
> >> > >> >> >
> >> > >> >> > On point 1) in the message protocol the headers are
> >> simply
> >> > a byte
> >> > >> array,
> >> > >> >> > as like the key or value, this is to clearly demarcate
> >> the
> >> > header in
> >> > >> the
> >> > >> >> > core message. Then the header byte array in the core
> >> > message is an
> >> > >> array of
> >> > >> >> > key, value pairs. This is what it is denoting.
> >> > >> >> >
> >> > >> >> > Then this would be I guess in the given notation:
> >> > >> >> >
> >> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> >> > >> >> > KeyLength => int32 <-----------------NEW size of
> the
> >> > byte[] of
> >> > >> the
> >> > >> >> > serialised key value
> >> > >> >> > Key => bytes <---------------------- NEW
> serialised
> >> > string (UTF8)
> >> > >> >> > bytes of the header key
> >> > >> >> > ValueLength => int32 <-------------- NEW size of
> the
> >> > byte[] of
> >> > >> the
> >> > >> >> > serialised header value
> >> > >> >> > Value => bytes <-------------------- NEW
> serialised
> >> > form of the
> >> > >> header
> >> > >> >> > value
> >> > >> >> >
> >> > >> >> > The key length and value length is matching the way
> the
> >> > protocol is
> >> > >> >> > defined in the core message currently.
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On point 2)
> >> > >> >> > Var sized ints, this was discussed much earlier on, in
> >> > fact I had
> >> > >> >> > suggested it myself (with Hadoop references), the
> >> > complexity of this
> >> > >> >> > compared to having a simpler protocol was argued and
> >> > agreed it
> >> > >> wasn’t worth
> >> > >> >> > the complexity as all other clients in other languages
> >> > would need to
> >> > >> ensure
> >> > >> >> > theyre using the right var size algorithm, as there
> is a
> >> > few.
> >> > >> >> >
> >> > >> >> > On point 3)
> >> > >> >> > We did the attributes, optional approach as originally
> >> > there was
> >> > >> marked
> >> > >> >> > concern that headers would cause a message size
> overhead
> >> > for others,
> >> > >> who
> >> > >> >> > don’t want them. As such this is the clean solution to
> >> > achieve that.
> >> > >> If
> >> > >> >> > that no longer holds, and we don’t care that we add
> >> 4bytes
> >> > overhead,
> >> > >> then
> >> > >> >> > im happy to remove.
> >> > >> >> >
> >> > >> >> > I’m personally in favour of keeping the message as
> small
> >> > as possible
> >> > >> so
> >> > >> >> > people don’t get shocks in perf and throughputs dues
> to
> >> > message size,
> >> > >> >> > unless they actively use the feature, as such I do
> >> prefer
> >> > the
> >> > >> attribute bit
> >> > >> >> > wise feature flag approach myself.
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> >> > jason@confluent.io> wrote:
> >> > >> >> >
> >> > >> >> > We have proposed a few significant changes to the
> >> > message format
> >> > >> in
> >> > >> >> > KIP-98
> >> > >> >> > which now seems likely to pass (perhaps with some
> >> > iterations on
> >> > >> >> > implementation details). It would be good to try
> and
> >> > coordinate
> >> > >> the
> >> > >> >> > changes
> >> > >> >> > in both of the proposals to make sure they are
> >> > consistent and
> >> > >> >> > compatible.
> >> > >> >> >
> >> > >> >> > I think using the attributes to indicate null
> >> headers
> >> > is a
> >> > >> reasonable
> >> > >> >> > approach. We have proposed to do the same thing
> for
> >> > the message
> >> > >> key and
> >> > >> >> > value. That said, I sympathize with Jay's
> argument.
> >> > Having
> >> > >> multiple
> >> > >> >> > ways to
> >> > >> >> > specify a null value increases the overall
> >> complexity
> >> > of the
> >> > >> protocol.
> >> > >> >> > You
> >> > >> >> > can see this just from the fact that you need the
> >> > extra verbiage
> >> > >> in the
> >> > >> >> > protocol specification in this KIP and in KIP-98
> to
> >> > describe the
> >> > >> >> > dependence
> >> > >> >> > between the fields and the attributes. It seems
> >> like a
> >> > slippery
> >> > >> slope
> >> > >> >> > if
> >> > >> >> > you start allowing different request types to
> >> > implement the
> >> > >> protocol
> >> > >> >> > specification differently.
> >> > >> >> >
> >> > >> >> > You can also argue that the messages already are
> and
> >> > are likely
> >> > >> to
> >> > >> >> > remain a
> >> > >> >> > special case. For example, there is currently no
> >> > generality in
> >> > >> how
> >> > >> >> > compressed message sets are represented that would
> >> be
> >> > applicable
> >> > >> for
> >> > >> >> > other
> >> > >> >> > request types. Some might see this divergence as
> an
> >> > unfortunate
> >> > >> >> > protocol
> >> > >> >> > deficiency which should be fixed; others might see
> >> it
> >> > as sort of
> >> > >> the
> >> > >> >> > inevitability of needing to optimize where it
> counts
> >> > most. I'm
> >> > >> probably
> >> > >> >> > somewhere in between, but I think we probably all
> >> > share the
> >> > >> intuition
> >> > >> >> > that
> >> > >> >> > the protocol should be kept as consistent as
> >> possible.
> >> > With that
> >> > >> in
> >> > >> >> > mind,
> >> > >> >> > here are a few comments:
> >> > >> >> >
> >> > >> >> > 1. One thing I found a little odd when reading the
> >> > current
> >> > >> proposal is
> >> > >> >> > that
> >> > >> >> > the headers are both represented as an array of
> >> bytes
> >> > and as an
> >> > >> array
> >> > >> >> > of
> >> > >> >> > key/value pairs. I'd probably suggest something
> like
> >> > this:
> >> > >> >> >
> >> > >> >> > Headers => [HeaderKey HeaderValue]
> >> > >> >> > HeaderKey => String
> >> > >> >> > HeaderValue => Bytes
> >> > >> >> >
> >> > >> >> > An array in the Kafka protocol is represented as a
> >> > 4-byte integer
> >> > >> >> > indicating the number of elements in the array
> >> > followed by the
> >> > >> >> > serialization of the elements. Unless I'm
> >> > misunderstanding, what
> >> > >> you
> >> > >> >> > have
> >> > >> >> > instead is the total size of the headers in bytes
> >> > followed by the
> >> > >> >> > elements.
> >> > >> >> > I'm not sure I see any reason for this
> >> inconsistency.
> >> > >> >> >
> >> > >> >> > 2. In KIP-98, we've introduced variable-length
> >> integer
> >> > fields.
> >> > >> >> > Effectively,
> >> > >> >> > we've enriched (or "complicated" as Jay might say
> ;)
> >> > the protocol
> >> > >> >> > specification to include the following types:
> >> VarInt,
> >> > VarLong,
> >> > >> >> > UnsignedVarInt and UnsignedVarLong.
> >> > >> >> >
> >> > >> >> > Along with these primitives, we could introduce
> the
> >> > following
> >> > >> types:
> >> > >> >> >
> >> > >> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> >> > >> >> > NumberOfItems => UnsignedVarInt
> >> > >> >> >
> >> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1
> >> > Item2 .. ItemN
> >> > >> >> > NumberOfItemsOrNull => VarInt (-1 means null)
> >> > >> >> >
> >> > >> >> > And similarly for the `String` and `Bytes` types.
> >> > These types
> >> > >> can save
> >> > >> >> > a
> >> > >> >> > considerable amount of space in this proposal
> >> because
> >> > they can
> >> > >> be used
> >> > >> >> > for
> >> > >> >> > both the number of headers included in the message
> >> and
> >> > the
> >> > >> lengths of
> >> > >> >> > the
> >> > >> >> > header keys and values. We could do this instead:
> >> > >> >> >
> >> > >> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
> >> > >> >> > HeaderKey => VarSizeString
> >> > >> >> > HeaderValue => VarSizeBytes
> >> > >> >> >
> >> > >> >> > Combining the savings from the use of variable
> >> length
> >> > fields, the
> >> > >> >> > benefit
> >> > >> >> > of using the attributes to represent null seems
> >> pretty
> >> > small.
> >> > >> >> >
> >> > >> >> > 3. Whichever way we go (whether we use the
> >> attributes
> >> > or not), we
> >> > >> >> > should at
> >> > >> >> > least be consistent between this KIP and KIP-98.
> It
> >> > would be very
> >> > >> >> > strange
> >> > >> >> > to have two ways to represent null values in the
> >> same
> >> > schema.
> >> > >> Either
> >> > >> >> > way is
> >> > >> >> > OK with me. I think some message-level
> optimizations
> >> > are
> >> > >> justifiable,
> >> > >> >> > but
> >> > >> >> > the savings here seem minimal (a few bytes per
> >> > message), so
> >> > >> maybe it's
> >> > >> >> > not
> >> > >> >> > worth the cost of letting the message diverge even
> >> > further from
> >> > >> the
> >> > >> >> > rest of
> >> > >> >> > the protocol.
> >> > >> >> >
> >> > >> >> > -Jason
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> >> > >> radai.rosenblatt@gmail.com>
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > I've trimmed the inline contents as this mail is
> >> > getting too
> >> > >> big for
> >> > >> >> > the
> >> > >> >> > > apache mailing list software to deliver :-(
> >> > >> >> > >
> >> > >> >> > > 1. the important thing for interoperability is
> for
> >> > different
> >> > >> >> > "interested
> >> > >> >> > > parties" (plugins, infra layers/wrappers,
> >> user-code)
> >> > to be
> >> > >> able to
> >> > >> >> > stick
> >> > >> >> > > pieces of metadata onto msgs without getting in
> >> each
> >> > other's
> >> > >> way. a
> >> > >> >> > common
> >> > >> >> > > key scheme (Strings, as of the time of this
> >> > writing?) is all
> >> > >> thats
> >> > >> >> > required
> >> > >> >> > > for that. it is assumed that the other end
> >> > interested in any
> >> > >> such
> >> > >> >> > piece of
> >> > >> >> > > metadata knows the encoding, and byte[] provides
> >> for
> >> > the most
> >> > >> >> > flexibility.
> >> > >> >> > > i believe this is the same logic behind core
> kafka
> >> > being
> >> > >> >> > byte[]/byte[] -
> >> > >> >> > > Strings are more "usable" but bytes are flexible
> >> and
> >> > so were
> >> > >> chosen.
> >> > >> >> > > Also - core kafka doesnt even do that good of a
> >> job
> >> > on
> >> > >> usability of
> >> > >> >> > the
> >> > >> >> > > payload (example - i have to specify the nop
> >> byte[]
> >> > "decoders"
> >> > >> >> > explicitly
> >> > >> >> > > in conf), and again sacrificies usability for
> the
> >> > sake of
> >> > >> >> > performance (no
> >> > >> >> > > convenient single-record processing as poll is a
> >> > batch, lots of
> >> > >> >> > obscure
> >> > >> >> > > little config details exposing internals of the
> >> > batching
> >> > >> mechanism,
> >> > >> >> > etc)
> >> > >> >> > >
> >> > >> >> > > this is also why i really dislike the idea of a
> >> > "type system"
> >> > >> for
> >> > >> >> > header
> >> > >> >> > > values, it further degrades the usability, adds
> >> > complexity and
> >> > >> will
> >> > >> >> > > eventually get in people's way, also, it would
> be
> >> > the 2nd/3rd
> >> > >> >> > home-group
> >> > >> >> > > serialization mechanism in core kafka (counting
> 2
> >> > iterations
> >> > >> of the
> >> > >> >> > "type
> >> > >> >> > > definition DSL")
> >> > >> >> > >
> >> > >> >> > > 2. this is an implementation detail, and not
> even
> >> a
> >> > very "user
> >> > >> >> > facing" one?
> >> > >> >> > > to the best of my understanding the vote process
> >> is
> >> > on proposed
> >> > >> >> > > API/behaviour. also - since we're willing to go
> >> with
> >> > strings
> >> > >> just
> >> > >> >> > serialize
> >> > >> >> > > a 0-sized header blob and IIUC you dont need any
> >> > optionals
> >> > >> anymore.
> >> > >> >> > >
> >> > >> >> > > 3. yes, we can :-)
> >> > >> >> > >
> >> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael
> Pearce <
> >> > >> >> > Michael.Pearce@ig.com>
> >> > >> >> > > wrote:
> >> > >> >> > >
> >> > >> >> > > > Hi Jay,
> >> > >> >> > > >
> >> > >> >> > > > 1) There was some initial debate on the value
> >> > part, as youll
> >> > >> note
> >> > >> >> > String,
> >> > >> >> > > > String headers were discounted early on. The
> >> > reason for this
> >> > >> is
> >> > >> >> > > flexibility
> >> > >> >> > > > and keeping in line with the flexibility of
> key,
> >> > value of the
> >> > >> >> > message
> >> > >> >> > > > object itself. I don’t think it takes away
> from
> >> an
> >> > ecosystem
> >> > >> as
> >> > >> >> > each
> >> > >> >> > > plugin
> >> > >> >> > > > will care for their own key, this way ints,
> >> > booleans , exotic
> >> > >> >> > custom
> >> > >> >> > > binary
> >> > >> >> > > > can all be catered for=.
> >> > >> >> > > > a. If you really wanted to push for a typed
> >> value
> >> > interface,
> >> > >> I
> >> > >> >> > wouldn’t
> >> > >> >> > > > want just String values supported, but the the
> >> > primatives
> >> > >> plus
> >> > >> >> > string and
> >> > >> >> > > > also still keeping the ability to have a
> binary
> >> > for custom
> >> > >> >> > binaries that
> >> > >> >> > > > some organisations may have.
> >> > >> >> > > > i. I have written this slight alternative
> here,
> >> > >> >> > > https://cwiki.apache.org/
> >> > >> >> > > > confluence/display/KAFKA/KIP-
> >> > 82+-+Add+Record+Headers+-+Typed
> >> > >> >> > > > ii. Essentially the value bytes, has a leading
> >> > byte overhead.
> >> > >> >> > > > 1. This tells you what type the value is,
> >> before
> >> > reading
> >> > >> the rest
> >> > >> >> > of the
> >> > >> >> > > > bytes, allowing serialisation/deserialization
> to
> >> > and from the
> >> > >> >> > primitives,
> >> > >> >> > > > string and byte[]. This is akin to some other
> >> > messaging
> >> > >> systems.
> >> > >> >> > > > 2) We are making it optional, so that for
> those
> >> > not wanting
> >> > >> >> > headers have
> >> > >> >> > > 0
> >> > >> >> > > > bytes overhead (think of it as a feature
> flag),
> >> I
> >> > don’t
> >> > >> think this
> >> > >> >> > is
> >> > >> >> > > > complex, especially if comparing to changes
> >> > proposed in
> >> > >> other kips
> >> > >> >> > like
> >> > >> >> > > > kip-98.
> >> > >> >> > > > a. If you really really don’t like this, we
> can
> >> > drop it, but
> >> > >> it
> >> > >> >> > would
> >> > >> >> > > mean
> >> > >> >> > > > buying into 4 bytes extra overhead for users
> who
> >> > do not want
> >> > >> to use
> >> > >> >> > > headers.
> >> > >> >> > > > 3) In the summary yes, it is at a higher
> level,
> >> > but I think
> >> > >> this
> >> > >> >> > is well
> >> > >> >> > > > documented in the proposed changes section.
> >> > >> >> > > > a. Added getHeaders method to
> Producer/Consumer
> >> > record (that
> >> > >> is it)
> >> > >> >> > > > b. We’ve also detailed the new Headers class
> >> that
> >> > this method
> >> > >> >> > returns
> >> > >> >> > > that
> >> > >> >> > > > encapsulates the headers protocol and logic.
> >> > >> >> > > >
> >> > >> >> > > > Best,
> >> > >> >> > > > Mike
> >> > >> >> > > >
> >> > >> >> > > > ==Original questions from the vote thread from
> >> > Jay.==
> >> > >> >> > > >
> >> > >> >> > > > Couple of things I think we still need to work
> >> out:
> >> > >> >> > > >
> >> > >> >> > > > 1. I think we agree about the key, but I
> >> think
> >> > we haven't
> >> > >> >> > talked about
> >> > >> >> > > > the value yet. I think if our goal is an
> open
> >> > ecosystem
> >> > >> of these
> >> > >> >> > > header
> >> > >> >> > > > spread across many plugins from many
> systems
> >> we
> >> > should
> >> > >> consider
> >> > >> >> > making
> >> > >> >> > > > this
> >> > >> >> > > > a string as well so it can be printed, set
> >> via
> >> > a UI, set
> >> > >> in
> >> > >> >> > config,
> >> > >> >> > > etc.
> >> > >> >> > > > Basically encouraging pluggable
> serialization
> >> > formats
> >> > >> here will
> >> > >> >> > lead
> >> > >> >> > > to
> >> > >> >> > > > a
> >> > >> >> > > > bit of a tower of babel.
> >> > >> >> > > > 2. This proposal still includes a pretty
> big
> >> > change to our
> >> > >> >> > > serialization
> >> > >> >> > > > and protocol definition layer. Essentially
> >> it is
> >> > >> introducing an
> >> > >> >> > > optional
> >> > >> >> > > > type, where the format is data dependent. I
> >> > think this is
> >> > >> >> > actually a
> >> > >> >> > > big
> >> > >> >> > > > change though it doesn't seem like it. It
> >> means
> >> > you can no
> >> > >> >> > longer
> >> > >> >> > > > specify
> >> > >> >> > > > this type with our type definition DSL, and
> >> > likewise it
> >> > >> requires
> >> > >> >> > > custom
> >> > >> >> > > > handling in client libs. This isn't a huge
> >> > thing, since
> >> > >> the
> >> > >> >> > Record
> >> > >> >> > > > definition is custom anyway, but I think
> this
> >> > kind of
> >> > >> protocol
> >> > >> >> > > > inconsistency is very non-desirable and
> ties
> >> > you to
> >> > >> hand-coding
> >> > >> >> > > things.
> >> > >> >> > > > I
> >> > >> >> > > > think the type should instead by [Key
> Value]
> >> in
> >> > our BNF,
> >> > >> where
> >> > >> >> > key and
> >> > >> >> > > > value are both short strings as used
> >> elsewhere.
> >> > This
> >> > >> brings it
> >> > >> >> > in line
> >> > >> >> > > > with
> >> > >> >> > > > the rest of the protocol.
> >> > >> >> > > > 3. Could we get more specific about the
> exact
> >> > Java API
> >> > >> change to
> >> > >> >> > > > ProducerRecord, ConsumerRecord, Record,
> etc?
> >> > >> >> > > >
> >> > >> >> > > > -Jay
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > The information contained in this email is strictly
> >> > confidential and
> >> > >> for
> >> > >> >> > the use of the addressee only, unless otherwise
> >> indicated.
> >> > If you
> >> > >> are not
> >> > >> >> > the intended recipient, please do not read, copy, use
> or
> >> > disclose to
> >> > >> others
> >> > >> >> > this message or any attachment. Please also notify the
> >> > sender by
> >> > >> replying
> >> > >> >> > to this email or by telephone (+44(020 7896 0011) and
> >> then
> >> > delete
> >> > >> the email
> >> > >> >> > and any copies of it. Opinions, conclusion (etc) that
> do
> >> > not relate
> >> > >> to the
> >> > >> >> > official business of this company shall be understood
> as
> >> > neither
> >> > >> given nor
> >> > >> >> > endorsed by it. IG is a trading name of IG Markets
> >> Limited
> >> > (a company
> >> > >> >> > registered in England and Wales, company number
> >> 04008957)
> >> > and IG
> >> > >> Index
> >> > >> >> > Limited (a company registered in England and Wales,
> >> > company number
> >> > >> >> > 01190902). Registered address at Cannon Bridge House,
> 25
> >> > Dowgate
> >> > >> Hill,
> >> > >> >> > London EC4R 2YA. Both IG Markets Limited (register
> >> number
> >> > 195355)
> >> > >> and IG
> >> > >> >> > Index Limited (register number 114059) are authorised
> >> and
> >> > regulated
> >> > >> by the
> >> > >> >> > Financial Conduct Authority.
> >> > >> >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Nacho - Ignacio Solis - isolis@igso.net
> >> > >>
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >> > The information contained in this email is strictly confidential and
> for
> >> > the use of the addressee only, unless otherwise indicated. If you are
> >> not
> >> > the intended recipient, please do not read, copy, use or disclose to
> >> others
> >> > this message or any attachment. Please also notify the sender by
> >> replying
> >> > to this email or by telephone (+44(020 7896 0011) and then delete the
> >> email
> >> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> >> the
> >> > official business of this company shall be understood as neither given
> >> nor
> >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> > registered in England and Wales, company number 04008957) and IG Index
> >> > Limited (a company registered in England and Wales, company number
> >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >> > Index Limited (register number 114059) are authorised and regulated by
> >> the
> >> > Financial Conduct Authority.
> >> >
> >> The information contained in this email is strictly confidential and for
> >> the use of the addressee only, unless otherwise indicated. If you are
> not
> >> the intended recipient, please do not read, copy, use or disclose to
> others
> >> this message or any attachment. Please also notify the sender by
> replying
> >> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >> official business of this company shall be understood as neither given
> nor
> >> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> registered in England and Wales, company number 04008957) and IG Index
> >> Limited (a company registered in England and Wales, company number
> >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> >> Index Limited (register number 114059) are authorised and regulated by
> the
> >> Financial Conduct Authority.
> >>
> >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Magnus Edenhill <ma...@edenhill.se>.
Big +1 on VarInts.
CPUs are fast, memory is slow.
I agree with Jason that we'll want to continue verifying messages,
including their headers, so while I appreciate the idea of the opaque
header blob it won't be useful in practice.
/Magnus
2017-02-17 10:41 GMT-08:00 Jason Gustafson <ja...@confluent.io>:
> Sorry, my mistake. The consumer interceptor is per batch, though I'm not
> sure that's an actual limitation since you still have access to the
> individual records.
>
> -Jason
>
> On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Re headers as byte array and future use by broker. This doesn't take away
> >> from that at all. Nor makes it difficult at all in my opinion.
> >
> >
> > Yeah, I didn't say it was difficult, only awkward. You wouldn't write the
> > schema that way if you were planning to use it on the brokers from the
> > beginning. Note also that one of the benefits of letting the broker
> > understand headers is that it can validate that they are properly
> > formatted. If cost is the only concern, we should confirm its impact
> > through performance testing.
> >
> > One of the key use cases requires access on consume at per event/message
> >> level at the point that message is being processed, as such the batch
> >> interceptors and batch consume api isn't suitable. It needs to be at the
> >> record level.
> >
> >
> > I'm not sure I understand the point about batching. Interceptors are
> > applied per-message, right?
> >
> > My intent on interceptors is to keep the usage of headers well-defined so
> > that they don't start leaking unnecessarily into applications. My guess
> is
> > that it's probably inevitable, but isolating it in the interceptors would
> > at least give people a second thought before deciding to use it. The main
> > challenge in my mind is figuring out how an MM use case would work. It
> > would be more cumbersome to replicate headers through an interceptor,
> > though arguably MM should be working at a lower level anyway.
> >
> > -Jason
> >
> > On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> >> Re headers available on the record va interceptors only
> >>
> >> One of the key use cases requires access on consume at per event/message
> >> level at the point that message is being processed, as such the batch
> >> interceptors and batch consume api isn't suitable. It needs to be at the
> >> record level.
> >>
> >> This anyhow is similar to jms/http/amqp where headers are available to
> >> consuming applications.
> >>
> >> Re headers as byte array and future use by broker. This doesn't take
> away
> >> from that at all. Nor makes it difficult at all in my opinion.
> >>
> >>
> >>
> >> Sent using OWA for iPhone
> >> ________________________________________
> >> From: Jason Gustafson <ja...@confluent.io>
> >> Sent: Friday, February 17, 2017 5:55:42 PM
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>
> >> >
> >> > Would you be proposing in KIP-98 to convert the other message int’s
> (key
> >> > length, value length) also to varint to keep it uniform.
> >> > Also I assume there will be a static or helper method made to
> write/read
> >> > these in the client and server.
> >>
> >>
> >> Yes, that is what we are proposing, so using varints for headers would
> be
> >> consistent with the rest of the message. We have used static helper
> >> methods
> >> in our prototype implementation.
> >>
> >> The cost of parsing, we want to parse/interpret the headers lazily (this
> >> is
> >> > a key point brought up earlier in discussions)
> >>
> >>
> >> I'm a bit skeptical of this. Has anyone done the performance testing? I
> >> can
> >> probably implement it and test it if no one else has. I was also under
> the
> >> impression that there may be use cases down the road where the broker
> >> would
> >> need to interpret headers. That wouldn't be off the table in the future
> if
> >> it's represented as bytes, but it would be quite a bit more awkward,
> >> right?
> >>
> >> By the way, one question I have been wondering about. My understanding
> is
> >> that headers are primarily for use cases where a third-party components
> >> wants to enrich messages without needing to understand or modify the
> >> schema
> >> of the message key and value. For the applications which directly
> produce
> >> and consume the messages and control the key/value schema directly, it
> >> seems we would rather have them implement headers directly in their own
> >> schema. Supposing for the sake of argument that it was possible, my
> >> question is whether it be sufficient to expose the headers in the
> >> interceptor API and not in the common API?
> >>
> >> -Jason
> >>
> >> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >> > On the point of varInts
> >> >
> >> > Would you be proposing in KIP-98 to convert the other message int’s
> (key
> >> > length, value length) also to varint to keep it uniform.
> >> > Also I assume there will be a static or helper method made to
> write/read
> >> > these in the client and server.
> >> >
> >> > Cheers
> >> > Mike
> >> >
> >> >
> >> >
> >> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >> >
> >> > On the point re: headers in the message protocol being a byte
> array
> >> > and not a count of elements followed by the elements. Again this was
> >> > discussed/argued previously.
> >> >
> >> > It was agreed on for a few reasons some of which you have
> obviously
> >> > picked up on:
> >> >
> >> > Broker is able to pass it through opaquely
> >> > The cost of parsing, we want to parse/interpret the headers lazily
> >> > (this is a key point brought up earlier in discussions)
> >> > Headers can be copied from consumer record to producer record (aka
> >> > mirror makers etc) without parsing if no changes are being made or
> being
> >> > looked at.
> >> > Keeps the broker agnostic to the format
> >> > You need an int32 either for the byte size of the headers, or for
> >> the
> >> > count of elements, so overheads are the same, but with going with an
> >> opaque
> >> > byte array has the above advantages.
> >> >
> >> > Cheers
> >> > Mike
> >> >
> >> >
> >> > On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io>
> wrote:
> >> >
> >> > Sorry, should have noted that the performance testing was done
> >> > using the
> >> > producer performance tool shipped with Kafka.
> >> >
> >> > -Jason
> >> >
> >> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> >> > jason@confluent.io> wrote:
> >> >
> >> > > Hey Nacho,
> >> > >
> >> > > I've compared performance of our KIP-98 implementation with
> >> and
> >> > without
> >> > > varints. For messages around 128 bytes, we see an increase
> in
> >> > throughput of
> >> > > about 30% using the default configuration settings. At 256
> >> > bytes, the
> >> > > increase is around 16%. Obviously the performance converge
> as
> >> > messages get
> >> > > larger, but it seems well worth the cost. Note that we are
> >> also
> >> > seeing a
> >> > > substantial performance increase against trunk primarily
> >> because
> >> > of the
> >> > > much more efficient packing that varints provide us.
> Anything
> >> > adding to
> >> > > message overhead, such as record headers, would only
> increase
> >> > the relative
> >> > > difference. (Of course take these numbers with a grain of
> salt
> >> > since I have
> >> > > only used the default settings with both the producer and
> >> broker
> >> > on my
> >> > > local machine. We intend to provide more extensive
> performance
> >> > details as
> >> > > part of the work for KIP-98.)
> >> > >
> >> > > The implementation we are using is from protobuf (
> >> > > https://developers.google.com/
> protocol-buffers/docs/encoding
> >> ),
> >> > which is
> >> > > also used in HBase. It is trivial to implement and as far
> as I
> >> > know doesn't
> >> > > suffer from the aliasing problem you are describing. I
> checked
> >> > with Magnus
> >> > > (the author of librdkafka) and he agreed that the savings
> >> seemed
> >> > worth the
> >> > > cost of implementation.
> >> > >
> >> > > -Jason
> >> > >
> >> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
> >> isolis@igso.net>
> >> > wrote:
> >> > >
> >> > >> -VarInts
> >> > >>
> >> > >> I'm one of the people (if not the most) opposed to VarInts.
> >> > VarInts
> >> > >> have a place, but this is not it. (We had a large
> >> discussion
> >> > about
> >> > >> them at the beginning of KIP-82 time)
> >> > >>
> >> > >> If anybody has real life performance numbers of VarInts
> >> > improving
> >> > >> things or significantly reducing resources I would like to
> >> know
> >> > what
> >> > >> that case may be. Yes, you can save some bytes here and
> >> there,
> >> > but
> >> > >> this is probably insignificant to the overall system
> behavior
> >> > and
> >> > >> storage requirements. -- I say this with respect to using
> >> > VarInts in
> >> > >> the protocol itself, not as part of the data.
> >> > >>
> >> > >> VarInts require you to parse the Int before using it and
> >> > depending on
> >> > >> the encoding they can suffer from aliasing (multiple
> >> > representations
> >> > >> for the same value).
> >> > >>
> >> > >> Why add complexity?
> >> > >>
> >> > >> Nacho
> >> > >>
> >> > >>
> >> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> >> > cmccabe@apache.org>
> >> > >> wrote:
> >> > >> > +1 for varints here-- it would save quite a bit of space.
> >> > They are
> >> > >> > pretty quick to implement as well.
> >> > >> >
> >> > >> > I think it makes sense for values to be byte arrays.
> Users
> >> > might want
> >> > >> > to attach arbitrary payloads; they shouldn't be forced to
> >> > serialize
> >> > >> > everything to Java strings.
> >> > >> >
> >> > >> > best,
> >> > >> > Colin
> >> > >> >
> >> > >> >
> >> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> >> > >> >> Hey Michael,
> >> > >> >>
> >> > >> >> Hmm, I guess the point of representing it as bytes is to
> >> > allow the
> >> > >> broker
> >> > >> >> to pass it through opaquely? Is the cost of parsing
> them a
> >> > concern, or
> >> > >> >> are
> >> > >> >> we simply trying to ensure that the broker stays
> agnostic
> >> to
> >> > the
> >> > >> format?
> >> > >> >>
> >> > >> >> On varints, I think adding support for them makes less
> >> sense
> >> > for an
> >> > >> >> isolated use case, but as part of a more holistic change
> >> > (such as what
> >> > >> we
> >> > >> >> have proposed in KIP-98), I think they are justifiable.
> If
> >> > we add them,
> >> > >> >> then the need to use attributes becomes quite a bit
> >> weaker,
> >> > right? The
> >> > >> >> other thing I find slightly odd is the fact that null
> >> > headers has no
> >> > >> >> actual
> >> > >> >> semantic meaning for the message (unlike null keys and
> >> > values). It is
> >> > >> >> just
> >> > >> >> a space optimization. It seems a bit better to always
> use
> >> > size 0 to
> >> > >> >> indicate having no headers.
> >> > >> >>
> >> > >> >> Overall, the main point is ensuring that the message
> >> schema
> >> > remains
> >> > >> >> consistent, either within the larger protocol, or at a
> >> > minimum within
> >> > >> the
> >> > >> >> message itself.
> >> > >> >>
> >> > >> >> -Jason
> >> > >> >>
> >> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> >> > Michael.Pearce@ig.com
> >> > >> >
> >> > >> >> wrote:
> >> > >> >>
> >> > >> >> > Hi Jason,
> >> > >> >> >
> >> > >> >> > On point 1) in the message protocol the headers are
> >> simply
> >> > a byte
> >> > >> array,
> >> > >> >> > as like the key or value, this is to clearly demarcate
> >> the
> >> > header in
> >> > >> the
> >> > >> >> > core message. Then the header byte array in the core
> >> > message is an
> >> > >> array of
> >> > >> >> > key, value pairs. This is what it is denoting.
> >> > >> >> >
> >> > >> >> > Then this would be I guess in the given notation:
> >> > >> >> >
> >> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> >> > >> >> > KeyLength => int32 <-----------------NEW size of
> the
> >> > byte[] of
> >> > >> the
> >> > >> >> > serialised key value
> >> > >> >> > Key => bytes <---------------------- NEW
> serialised
> >> > string (UTF8)
> >> > >> >> > bytes of the header key
> >> > >> >> > ValueLength => int32 <-------------- NEW size of
> the
> >> > byte[] of
> >> > >> the
> >> > >> >> > serialised header value
> >> > >> >> > Value => bytes <-------------------- NEW
> serialised
> >> > form of the
> >> > >> header
> >> > >> >> > value
> >> > >> >> >
> >> > >> >> > The key length and value length is matching the way
> the
> >> > protocol is
> >> > >> >> > defined in the core message currently.
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On point 2)
> >> > >> >> > Var sized ints, this was discussed much earlier on, in
> >> > fact I had
> >> > >> >> > suggested it myself (with Hadoop references), the
> >> > complexity of this
> >> > >> >> > compared to having a simpler protocol was argued and
> >> > agreed it
> >> > >> wasn’t worth
> >> > >> >> > the complexity as all other clients in other languages
> >> > would need to
> >> > >> ensure
> >> > >> >> > theyre using the right var size algorithm, as there
> is a
> >> > few.
> >> > >> >> >
> >> > >> >> > On point 3)
> >> > >> >> > We did the attributes, optional approach as originally
> >> > there was
> >> > >> marked
> >> > >> >> > concern that headers would cause a message size
> overhead
> >> > for others,
> >> > >> who
> >> > >> >> > don’t want them. As such this is the clean solution to
> >> > achieve that.
> >> > >> If
> >> > >> >> > that no longer holds, and we don’t care that we add
> >> 4bytes
> >> > overhead,
> >> > >> then
> >> > >> >> > im happy to remove.
> >> > >> >> >
> >> > >> >> > I’m personally in favour of keeping the message as
> small
> >> > as possible
> >> > >> so
> >> > >> >> > people don’t get shocks in perf and throughputs dues
> to
> >> > message size,
> >> > >> >> > unless they actively use the feature, as such I do
> >> prefer
> >> > the
> >> > >> attribute bit
> >> > >> >> > wise feature flag approach myself.
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> >> > jason@confluent.io> wrote:
> >> > >> >> >
> >> > >> >> > We have proposed a few significant changes to the
> >> > message format
> >> > >> in
> >> > >> >> > KIP-98
> >> > >> >> > which now seems likely to pass (perhaps with some
> >> > iterations on
> >> > >> >> > implementation details). It would be good to try
> and
> >> > coordinate
> >> > >> the
> >> > >> >> > changes
> >> > >> >> > in both of the proposals to make sure they are
> >> > consistent and
> >> > >> >> > compatible.
> >> > >> >> >
> >> > >> >> > I think using the attributes to indicate null
> >> headers
> >> > is a
> >> > >> reasonable
> >> > >> >> > approach. We have proposed to do the same thing
> for
> >> > the message
> >> > >> key and
> >> > >> >> > value. That said, I sympathize with Jay's
> argument.
> >> > Having
> >> > >> multiple
> >> > >> >> > ways to
> >> > >> >> > specify a null value increases the overall
> >> complexity
> >> > of the
> >> > >> protocol.
> >> > >> >> > You
> >> > >> >> > can see this just from the fact that you need the
> >> > extra verbiage
> >> > >> in the
> >> > >> >> > protocol specification in this KIP and in KIP-98
> to
> >> > describe the
> >> > >> >> > dependence
> >> > >> >> > between the fields and the attributes. It seems
> >> like a
> >> > slippery
> >> > >> slope
> >> > >> >> > if
> >> > >> >> > you start allowing different request types to
> >> > implement the
> >> > >> protocol
> >> > >> >> > specification differently.
> >> > >> >> >
> >> > >> >> > You can also argue that the messages already are
> and
> >> > are likely
> >> > >> to
> >> > >> >> > remain a
> >> > >> >> > special case. For example, there is currently no
> >> > generality in
> >> > >> how
> >> > >> >> > compressed message sets are represented that would
> >> be
> >> > applicable
> >> > >> for
> >> > >> >> > other
> >> > >> >> > request types. Some might see this divergence as
> an
> >> > unfortunate
> >> > >> >> > protocol
> >> > >> >> > deficiency which should be fixed; others might see
> >> it
> >> > as sort of
> >> > >> the
> >> > >> >> > inevitability of needing to optimize where it
> counts
> >> > most. I'm
> >> > >> probably
> >> > >> >> > somewhere in between, but I think we probably all
> >> > share the
> >> > >> intuition
> >> > >> >> > that
> >> > >> >> > the protocol should be kept as consistent as
> >> possible.
> >> > With that
> >> > >> in
> >> > >> >> > mind,
> >> > >> >> > here are a few comments:
> >> > >> >> >
> >> > >> >> > 1. One thing I found a little odd when reading the
> >> > current
> >> > >> proposal is
> >> > >> >> > that
> >> > >> >> > the headers are both represented as an array of
> >> bytes
> >> > and as an
> >> > >> array
> >> > >> >> > of
> >> > >> >> > key/value pairs. I'd probably suggest something
> like
> >> > this:
> >> > >> >> >
> >> > >> >> > Headers => [HeaderKey HeaderValue]
> >> > >> >> > HeaderKey => String
> >> > >> >> > HeaderValue => Bytes
> >> > >> >> >
> >> > >> >> > An array in the Kafka protocol is represented as a
> >> > 4-byte integer
> >> > >> >> > indicating the number of elements in the array
> >> > followed by the
> >> > >> >> > serialization of the elements. Unless I'm
> >> > misunderstanding, what
> >> > >> you
> >> > >> >> > have
> >> > >> >> > instead is the total size of the headers in bytes
> >> > followed by the
> >> > >> >> > elements.
> >> > >> >> > I'm not sure I see any reason for this
> >> inconsistency.
> >> > >> >> >
> >> > >> >> > 2. In KIP-98, we've introduced variable-length
> >> integer
> >> > fields.
> >> > >> >> > Effectively,
> >> > >> >> > we've enriched (or "complicated" as Jay might say
> ;)
> >> > the protocol
> >> > >> >> > specification to include the following types:
> >> VarInt,
> >> > VarLong,
> >> > >> >> > UnsignedVarInt and UnsignedVarLong.
> >> > >> >> >
> >> > >> >> > Along with these primitives, we could introduce
> the
> >> > following
> >> > >> types:
> >> > >> >> >
> >> > >> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> >> > >> >> > NumberOfItems => UnsignedVarInt
> >> > >> >> >
> >> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1
> >> > Item2 .. ItemN
> >> > >> >> > NumberOfItemsOrNull => VarInt (-1 means null)
> >> > >> >> >
> >> > >> >> > And similarly for the `String` and `Bytes` types.
> >> > These types
> >> > >> can save
> >> > >> >> > a
> >> > >> >> > considerable amount of space in this proposal
> >> because
> >> > they can
> >> > >> be used
> >> > >> >> > for
> >> > >> >> > both the number of headers included in the message
> >> and
> >> > the
> >> > >> lengths of
> >> > >> >> > the
> >> > >> >> > header keys and values. We could do this instead:
> >> > >> >> >
> >> > >> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
> >> > >> >> > HeaderKey => VarSizeString
> >> > >> >> > HeaderValue => VarSizeBytes
> >> > >> >> >
> >> > >> >> > Combining the savings from the use of variable
> >> length
> >> > fields, the
> >> > >> >> > benefit
> >> > >> >> > of using the attributes to represent null seems
> >> pretty
> >> > small.
> >> > >> >> >
> >> > >> >> > 3. Whichever way we go (whether we use the
> >> attributes
> >> > or not), we
> >> > >> >> > should at
> >> > >> >> > least be consistent between this KIP and KIP-98.
> It
> >> > would be very
> >> > >> >> > strange
> >> > >> >> > to have two ways to represent null values in the
> >> same
> >> > schema.
> >> > >> Either
> >> > >> >> > way is
> >> > >> >> > OK with me. I think some message-level
> optimizations
> >> > are
> >> > >> justifiable,
> >> > >> >> > but
> >> > >> >> > the savings here seem minimal (a few bytes per
> >> > message), so
> >> > >> maybe it's
> >> > >> >> > not
> >> > >> >> > worth the cost of letting the message diverge even
> >> > further from
> >> > >> the
> >> > >> >> > rest of
> >> > >> >> > the protocol.
> >> > >> >> >
> >> > >> >> > -Jason
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> >> > >> radai.rosenblatt@gmail.com>
> >> > >> >> > wrote:
> >> > >> >> >
> >> > >> >> > > I've trimmed the inline contents as this mail is
> >> > getting too
> >> > >> big for
> >> > >> >> > the
> >> > >> >> > > apache mailing list software to deliver :-(
> >> > >> >> > >
> >> > >> >> > > 1. the important thing for interoperability is
> for
> >> > different
> >> > >> >> > "interested
> >> > >> >> > > parties" (plugins, infra layers/wrappers,
> >> user-code)
> >> > to be
> >> > >> able to
> >> > >> >> > stick
> >> > >> >> > > pieces of metadata onto msgs without getting in
> >> each
> >> > other's
> >> > >> way. a
> >> > >> >> > common
> >> > >> >> > > key scheme (Strings, as of the time of this
> >> > writing?) is all
> >> > >> thats
> >> > >> >> > required
> >> > >> >> > > for that. it is assumed that the other end
> >> > interested in any
> >> > >> such
> >> > >> >> > piece of
> >> > >> >> > > metadata knows the encoding, and byte[] provides
> >> for
> >> > the most
> >> > >> >> > flexibility.
> >> > >> >> > > i believe this is the same logic behind core
> kafka
> >> > being
> >> > >> >> > byte[]/byte[] -
> >> > >> >> > > Strings are more "usable" but bytes are flexible
> >> and
> >> > so were
> >> > >> chosen.
> >> > >> >> > > Also - core kafka doesnt even do that good of a
> >> job
> >> > on
> >> > >> usability of
> >> > >> >> > the
> >> > >> >> > > payload (example - i have to specify the nop
> >> byte[]
> >> > "decoders"
> >> > >> >> > explicitly
> >> > >> >> > > in conf), and again sacrificies usability for
> the
> >> > sake of
> >> > >> >> > performance (no
> >> > >> >> > > convenient single-record processing as poll is a
> >> > batch, lots of
> >> > >> >> > obscure
> >> > >> >> > > little config details exposing internals of the
> >> > batching
> >> > >> mechanism,
> >> > >> >> > etc)
> >> > >> >> > >
> >> > >> >> > > this is also why i really dislike the idea of a
> >> > "type system"
> >> > >> for
> >> > >> >> > header
> >> > >> >> > > values, it further degrades the usability, adds
> >> > complexity and
> >> > >> will
> >> > >> >> > > eventually get in people's way, also, it would
> be
> >> > the 2nd/3rd
> >> > >> >> > home-group
> >> > >> >> > > serialization mechanism in core kafka (counting
> 2
> >> > iterations
> >> > >> of the
> >> > >> >> > "type
> >> > >> >> > > definition DSL")
> >> > >> >> > >
> >> > >> >> > > 2. this is an implementation detail, and not
> even
> >> a
> >> > very "user
> >> > >> >> > facing" one?
> >> > >> >> > > to the best of my understanding the vote process
> >> is
> >> > on proposed
> >> > >> >> > > API/behaviour. also - since we're willing to go
> >> with
> >> > strings
> >> > >> just
> >> > >> >> > serialize
> >> > >> >> > > a 0-sized header blob and IIUC you dont need any
> >> > optionals
> >> > >> anymore.
> >> > >> >> > >
> >> > >> >> > > 3. yes, we can :-)
> >> > >> >> > >
> >> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael
> Pearce <
> >> > >> >> > Michael.Pearce@ig.com>
> >> > >> >> > > wrote:
> >> > >> >> > >
> >> > >> >> > > > Hi Jay,
> >> > >> >> > > >
> >> > >> >> > > > 1) There was some initial debate on the value
> >> > part, as youll
> >> > >> note
> >> > >> >> > String,
> >> > >> >> > > > String headers were discounted early on. The
> >> > reason for this
> >> > >> is
> >> > >> >> > > flexibility
> >> > >> >> > > > and keeping in line with the flexibility of
> key,
> >> > value of the
> >> > >> >> > message
> >> > >> >> > > > object itself. I don’t think it takes away
> from
> >> an
> >> > ecosystem
> >> > >> as
> >> > >> >> > each
> >> > >> >> > > plugin
> >> > >> >> > > > will care for their own key, this way ints,
> >> > booleans , exotic
> >> > >> >> > custom
> >> > >> >> > > binary
> >> > >> >> > > > can all be catered for=.
> >> > >> >> > > > a. If you really wanted to push for a typed
> >> value
> >> > interface,
> >> > >> I
> >> > >> >> > wouldn’t
> >> > >> >> > > > want just String values supported, but the the
> >> > primatives
> >> > >> plus
> >> > >> >> > string and
> >> > >> >> > > > also still keeping the ability to have a
> binary
> >> > for custom
> >> > >> >> > binaries that
> >> > >> >> > > > some organisations may have.
> >> > >> >> > > > i. I have written this slight alternative
> here,
> >> > >> >> > > https://cwiki.apache.org/
> >> > >> >> > > > confluence/display/KAFKA/KIP-
> >> > 82+-+Add+Record+Headers+-+Typed
> >> > >> >> > > > ii. Essentially the value bytes, has a leading
> >> > byte overhead.
> >> > >> >> > > > 1. This tells you what type the value is,
> >> before
> >> > reading
> >> > >> the rest
> >> > >> >> > of the
> >> > >> >> > > > bytes, allowing serialisation/deserialization
> to
> >> > and from the
> >> > >> >> > primitives,
> >> > >> >> > > > string and byte[]. This is akin to some other
> >> > messaging
> >> > >> systems.
> >> > >> >> > > > 2) We are making it optional, so that for
> those
> >> > not wanting
> >> > >> >> > headers have
> >> > >> >> > > 0
> >> > >> >> > > > bytes overhead (think of it as a feature
> flag),
> >> I
> >> > don’t
> >> > >> think this
> >> > >> >> > is
> >> > >> >> > > > complex, especially if comparing to changes
> >> > proposed in
> >> > >> other kips
> >> > >> >> > like
> >> > >> >> > > > kip-98.
> >> > >> >> > > > a. If you really really don’t like this, we
> can
> >> > drop it, but
> >> > >> it
> >> > >> >> > would
> >> > >> >> > > mean
> >> > >> >> > > > buying into 4 bytes extra overhead for users
> who
> >> > do not want
> >> > >> to use
> >> > >> >> > > headers.
> >> > >> >> > > > 3) In the summary yes, it is at a higher
> level,
> >> > but I think
> >> > >> this
> >> > >> >> > is well
> >> > >> >> > > > documented in the proposed changes section.
> >> > >> >> > > > a. Added getHeaders method to
> Producer/Consumer
> >> > record (that
> >> > >> is it)
> >> > >> >> > > > b. We’ve also detailed the new Headers class
> >> that
> >> > this method
> >> > >> >> > returns
> >> > >> >> > > that
> >> > >> >> > > > encapsulates the headers protocol and logic.
> >> > >> >> > > >
> >> > >> >> > > > Best,
> >> > >> >> > > > Mike
> >> > >> >> > > >
> >> > >> >> > > > ==Original questions from the vote thread from
> >> > Jay.==
> >> > >> >> > > >
> >> > >> >> > > > Couple of things I think we still need to work
> >> out:
> >> > >> >> > > >
> >> > >> >> > > > 1. I think we agree about the key, but I
> >> think
> >> > we haven't
> >> > >> >> > talked about
> >> > >> >> > > > the value yet. I think if our goal is an
> open
> >> > ecosystem
> >> > >> of these
> >> > >> >> > > header
> >> > >> >> > > > spread across many plugins from many
> systems
> >> we
> >> > should
> >> > >> consider
> >> > >> >> > making
> >> > >> >> > > > this
> >> > >> >> > > > a string as well so it can be printed, set
> >> via
> >> > a UI, set
> >> > >> in
> >> > >> >> > config,
> >> > >> >> > > etc.
> >> > >> >> > > > Basically encouraging pluggable
> serialization
> >> > formats
> >> > >> here will
> >> > >> >> > lead
> >> > >> >> > > to
> >> > >> >> > > > a
> >> > >> >> > > > bit of a tower of babel.
> >> > >> >> > > > 2. This proposal still includes a pretty
> big
> >> > change to our
> >> > >> >> > > serialization
> >> > >> >> > > > and protocol definition layer. Essentially
> >> it is
> >> > >> introducing an
> >> > >> >> > > optional
> >> > >> >> > > > type, where the format is data dependent. I
> >> > think this is
> >> > >> >> > actually a
> >> > >> >> > > big
> >> > >> >> > > > change though it doesn't seem like it. It
> >> means
> >> > you can no
> >> > >> >> > longer
> >> > >> >> > > > specify
> >> > >> >> > > > this type with our type definition DSL, and
> >> > likewise it
> >> > >> requires
> >> > >> >> > > custom
> >> > >> >> > > > handling in client libs. This isn't a huge
> >> > thing, since
> >> > >> the
> >> > >> >> > Record
> >> > >> >> > > > definition is custom anyway, but I think
> this
> >> > kind of
> >> > >> protocol
> >> > >> >> > > > inconsistency is very non-desirable and
> ties
> >> > you to
> >> > >> hand-coding
> >> > >> >> > > things.
> >> > >> >> > > > I
> >> > >> >> > > > think the type should instead by [Key
> Value]
> >> in
> >> > our BNF,
> >> > >> where
> >> > >> >> > key and
> >> > >> >> > > > value are both short strings as used
> >> elsewhere.
> >> > This
> >> > >> brings it
> >> > >> >> > in line
> >> > >> >> > > > with
> >> > >> >> > > > the rest of the protocol.
> >> > >> >> > > > 3. Could we get more specific about the
> exact
> >> > Java API
> >> > >> change to
> >> > >> >> > > > ProducerRecord, ConsumerRecord, Record,
> etc?
> >> > >> >> > > >
> >> > >> >> > > > -Jay
> >> > >> >> > > >
> >> > >> >> > >
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > The information contained in this email is strictly
> >> > confidential and
> >> > >> for
> >> > >> >> > the use of the addressee only, unless otherwise
> >> indicated.
> >> > If you
> >> > >> are not
> >> > >> >> > the intended recipient, please do not read, copy, use
> or
> >> > disclose to
> >> > >> others
> >> > >> >> > this message or any attachment. Please also notify the
> >> > sender by
> >> > >> replying
> >> > >> >> > to this email or by telephone (+44(020 7896 0011) and
> >> then
> >> > delete
> >> > >> the email
> >> > >> >> > and any copies of it. Opinions, conclusion (etc) that
> do
> >> > not relate
> >> > >> to the
> >> > >> >> > official business of this company shall be understood
> as
> >> > neither
> >> > >> given nor
> >> > >> >> > endorsed by it. IG is a trading name of IG Markets
> >> Limited
> >> > (a company
> >> > >> >> > registered in England and Wales, company number
> >> 04008957)
> >> > and IG
> >> > >> Index
> >> > >> >> > Limited (a company registered in England and Wales,
> >> > company number
> >> > >> >> > 01190902). Registered address at Cannon Bridge House,
> 25
> >> > Dowgate
> >> > >> Hill,
> >> > >> >> > London EC4R 2YA. Both IG Markets Limited (register
> >> number
> >> > 195355)
> >> > >> and IG
> >> > >> >> > Index Limited (register number 114059) are authorised
> >> and
> >> > regulated
> >> > >> by the
> >> > >> >> > Financial Conduct Authority.
> >> > >> >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Nacho - Ignacio Solis - isolis@igso.net
> >> > >>
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >> > The information contained in this email is strictly confidential and
> for
> >> > the use of the addressee only, unless otherwise indicated. If you are
> >> not
> >> > the intended recipient, please do not read, copy, use or disclose to
> >> others
> >> > this message or any attachment. Please also notify the sender by
> >> replying
> >> > to this email or by telephone (+44(020 7896 0011) and then delete the
> >> email
> >> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> >> the
> >> > official business of this company shall be understood as neither given
> >> nor
> >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> > registered in England and Wales, company number 04008957) and IG Index
> >> > Limited (a company registered in England and Wales, company number
> >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >> > Index Limited (register number 114059) are authorised and regulated by
> >> the
> >> > Financial Conduct Authority.
> >> >
> >> The information contained in this email is strictly confidential and for
> >> the use of the addressee only, unless otherwise indicated. If you are
> not
> >> the intended recipient, please do not read, copy, use or disclose to
> others
> >> this message or any attachment. Please also notify the sender by
> replying
> >> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >> official business of this company shall be understood as neither given
> nor
> >> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> registered in England and Wales, company number 04008957) and IG Index
> >> Limited (a company registered in England and Wales, company number
> >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> >> Index Limited (register number 114059) are authorised and regulated by
> the
> >> Financial Conduct Authority.
> >>
> >
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Sorry, my mistake. The consumer interceptor is per batch, though I'm not
sure that's an actual limitation since you still have access to the
individual records.
-Jason
On Fri, Feb 17, 2017 at 10:39 AM, Jason Gustafson <ja...@confluent.io>
wrote:
> Re headers as byte array and future use by broker. This doesn't take away
>> from that at all. Nor makes it difficult at all in my opinion.
>
>
> Yeah, I didn't say it was difficult, only awkward. You wouldn't write the
> schema that way if you were planning to use it on the brokers from the
> beginning. Note also that one of the benefits of letting the broker
> understand headers is that it can validate that they are properly
> formatted. If cost is the only concern, we should confirm its impact
> through performance testing.
>
> One of the key use cases requires access on consume at per event/message
>> level at the point that message is being processed, as such the batch
>> interceptors and batch consume api isn't suitable. It needs to be at the
>> record level.
>
>
> I'm not sure I understand the point about batching. Interceptors are
> applied per-message, right?
>
> My intent on interceptors is to keep the usage of headers well-defined so
> that they don't start leaking unnecessarily into applications. My guess is
> that it's probably inevitable, but isolating it in the interceptors would
> at least give people a second thought before deciding to use it. The main
> challenge in my mind is figuring out how an MM use case would work. It
> would be more cumbersome to replicate headers through an interceptor,
> though arguably MM should be working at a lower level anyway.
>
> -Jason
>
> On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> Re headers available on the record va interceptors only
>>
>> One of the key use cases requires access on consume at per event/message
>> level at the point that message is being processed, as such the batch
>> interceptors and batch consume api isn't suitable. It needs to be at the
>> record level.
>>
>> This anyhow is similar to jms/http/amqp where headers are available to
>> consuming applications.
>>
>> Re headers as byte array and future use by broker. This doesn't take away
>> from that at all. Nor makes it difficult at all in my opinion.
>>
>>
>>
>> Sent using OWA for iPhone
>> ________________________________________
>> From: Jason Gustafson <ja...@confluent.io>
>> Sent: Friday, February 17, 2017 5:55:42 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>> >
>> > Would you be proposing in KIP-98 to convert the other message int’s (key
>> > length, value length) also to varint to keep it uniform.
>> > Also I assume there will be a static or helper method made to write/read
>> > these in the client and server.
>>
>>
>> Yes, that is what we are proposing, so using varints for headers would be
>> consistent with the rest of the message. We have used static helper
>> methods
>> in our prototype implementation.
>>
>> The cost of parsing, we want to parse/interpret the headers lazily (this
>> is
>> > a key point brought up earlier in discussions)
>>
>>
>> I'm a bit skeptical of this. Has anyone done the performance testing? I
>> can
>> probably implement it and test it if no one else has. I was also under the
>> impression that there may be use cases down the road where the broker
>> would
>> need to interpret headers. That wouldn't be off the table in the future if
>> it's represented as bytes, but it would be quite a bit more awkward,
>> right?
>>
>> By the way, one question I have been wondering about. My understanding is
>> that headers are primarily for use cases where a third-party components
>> wants to enrich messages without needing to understand or modify the
>> schema
>> of the message key and value. For the applications which directly produce
>> and consume the messages and control the key/value schema directly, it
>> seems we would rather have them implement headers directly in their own
>> schema. Supposing for the sake of argument that it was possible, my
>> question is whether it be sufficient to expose the headers in the
>> interceptor API and not in the common API?
>>
>> -Jason
>>
>> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <Mi...@ig.com>
>> wrote:
>>
>> > On the point of varInts
>> >
>> > Would you be proposing in KIP-98 to convert the other message int’s (key
>> > length, value length) also to varint to keep it uniform.
>> > Also I assume there will be a static or helper method made to write/read
>> > these in the client and server.
>> >
>> > Cheers
>> > Mike
>> >
>> >
>> >
>> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
>> >
>> > On the point re: headers in the message protocol being a byte array
>> > and not a count of elements followed by the elements. Again this was
>> > discussed/argued previously.
>> >
>> > It was agreed on for a few reasons some of which you have obviously
>> > picked up on:
>> >
>> > Broker is able to pass it through opaquely
>> > The cost of parsing, we want to parse/interpret the headers lazily
>> > (this is a key point brought up earlier in discussions)
>> > Headers can be copied from consumer record to producer record (aka
>> > mirror makers etc) without parsing if no changes are being made or being
>> > looked at.
>> > Keeps the broker agnostic to the format
>> > You need an int32 either for the byte size of the headers, or for
>> the
>> > count of elements, so overheads are the same, but with going with an
>> opaque
>> > byte array has the above advantages.
>> >
>> > Cheers
>> > Mike
>> >
>> >
>> > On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io> wrote:
>> >
>> > Sorry, should have noted that the performance testing was done
>> > using the
>> > producer performance tool shipped with Kafka.
>> >
>> > -Jason
>> >
>> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
>> > jason@confluent.io> wrote:
>> >
>> > > Hey Nacho,
>> > >
>> > > I've compared performance of our KIP-98 implementation with
>> and
>> > without
>> > > varints. For messages around 128 bytes, we see an increase in
>> > throughput of
>> > > about 30% using the default configuration settings. At 256
>> > bytes, the
>> > > increase is around 16%. Obviously the performance converge as
>> > messages get
>> > > larger, but it seems well worth the cost. Note that we are
>> also
>> > seeing a
>> > > substantial performance increase against trunk primarily
>> because
>> > of the
>> > > much more efficient packing that varints provide us. Anything
>> > adding to
>> > > message overhead, such as record headers, would only increase
>> > the relative
>> > > difference. (Of course take these numbers with a grain of salt
>> > since I have
>> > > only used the default settings with both the producer and
>> broker
>> > on my
>> > > local machine. We intend to provide more extensive performance
>> > details as
>> > > part of the work for KIP-98.)
>> > >
>> > > The implementation we are using is from protobuf (
>> > > https://developers.google.com/protocol-buffers/docs/encoding
>> ),
>> > which is
>> > > also used in HBase. It is trivial to implement and as far as I
>> > know doesn't
>> > > suffer from the aliasing problem you are describing. I checked
>> > with Magnus
>> > > (the author of librdkafka) and he agreed that the savings
>> seemed
>> > worth the
>> > > cost of implementation.
>> > >
>> > > -Jason
>> > >
>> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
>> isolis@igso.net>
>> > wrote:
>> > >
>> > >> -VarInts
>> > >>
>> > >> I'm one of the people (if not the most) opposed to VarInts.
>> > VarInts
>> > >> have a place, but this is not it. (We had a large
>> discussion
>> > about
>> > >> them at the beginning of KIP-82 time)
>> > >>
>> > >> If anybody has real life performance numbers of VarInts
>> > improving
>> > >> things or significantly reducing resources I would like to
>> know
>> > what
>> > >> that case may be. Yes, you can save some bytes here and
>> there,
>> > but
>> > >> this is probably insignificant to the overall system behavior
>> > and
>> > >> storage requirements. -- I say this with respect to using
>> > VarInts in
>> > >> the protocol itself, not as part of the data.
>> > >>
>> > >> VarInts require you to parse the Int before using it and
>> > depending on
>> > >> the encoding they can suffer from aliasing (multiple
>> > representations
>> > >> for the same value).
>> > >>
>> > >> Why add complexity?
>> > >>
>> > >> Nacho
>> > >>
>> > >>
>> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
>> > cmccabe@apache.org>
>> > >> wrote:
>> > >> > +1 for varints here-- it would save quite a bit of space.
>> > They are
>> > >> > pretty quick to implement as well.
>> > >> >
>> > >> > I think it makes sense for values to be byte arrays. Users
>> > might want
>> > >> > to attach arbitrary payloads; they shouldn't be forced to
>> > serialize
>> > >> > everything to Java strings.
>> > >> >
>> > >> > best,
>> > >> > Colin
>> > >> >
>> > >> >
>> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
>> > >> >> Hey Michael,
>> > >> >>
>> > >> >> Hmm, I guess the point of representing it as bytes is to
>> > allow the
>> > >> broker
>> > >> >> to pass it through opaquely? Is the cost of parsing them a
>> > concern, or
>> > >> >> are
>> > >> >> we simply trying to ensure that the broker stays agnostic
>> to
>> > the
>> > >> format?
>> > >> >>
>> > >> >> On varints, I think adding support for them makes less
>> sense
>> > for an
>> > >> >> isolated use case, but as part of a more holistic change
>> > (such as what
>> > >> we
>> > >> >> have proposed in KIP-98), I think they are justifiable. If
>> > we add them,
>> > >> >> then the need to use attributes becomes quite a bit
>> weaker,
>> > right? The
>> > >> >> other thing I find slightly odd is the fact that null
>> > headers has no
>> > >> >> actual
>> > >> >> semantic meaning for the message (unlike null keys and
>> > values). It is
>> > >> >> just
>> > >> >> a space optimization. It seems a bit better to always use
>> > size 0 to
>> > >> >> indicate having no headers.
>> > >> >>
>> > >> >> Overall, the main point is ensuring that the message
>> schema
>> > remains
>> > >> >> consistent, either within the larger protocol, or at a
>> > minimum within
>> > >> the
>> > >> >> message itself.
>> > >> >>
>> > >> >> -Jason
>> > >> >>
>> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
>> > Michael.Pearce@ig.com
>> > >> >
>> > >> >> wrote:
>> > >> >>
>> > >> >> > Hi Jason,
>> > >> >> >
>> > >> >> > On point 1) in the message protocol the headers are
>> simply
>> > a byte
>> > >> array,
>> > >> >> > as like the key or value, this is to clearly demarcate
>> the
>> > header in
>> > >> the
>> > >> >> > core message. Then the header byte array in the core
>> > message is an
>> > >> array of
>> > >> >> > key, value pairs. This is what it is denoting.
>> > >> >> >
>> > >> >> > Then this would be I guess in the given notation:
>> > >> >> >
>> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
>> > >> >> > KeyLength => int32 <-----------------NEW size of the
>> > byte[] of
>> > >> the
>> > >> >> > serialised key value
>> > >> >> > Key => bytes <---------------------- NEW serialised
>> > string (UTF8)
>> > >> >> > bytes of the header key
>> > >> >> > ValueLength => int32 <-------------- NEW size of the
>> > byte[] of
>> > >> the
>> > >> >> > serialised header value
>> > >> >> > Value => bytes <-------------------- NEW serialised
>> > form of the
>> > >> header
>> > >> >> > value
>> > >> >> >
>> > >> >> > The key length and value length is matching the way the
>> > protocol is
>> > >> >> > defined in the core message currently.
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> > On point 2)
>> > >> >> > Var sized ints, this was discussed much earlier on, in
>> > fact I had
>> > >> >> > suggested it myself (with Hadoop references), the
>> > complexity of this
>> > >> >> > compared to having a simpler protocol was argued and
>> > agreed it
>> > >> wasn’t worth
>> > >> >> > the complexity as all other clients in other languages
>> > would need to
>> > >> ensure
>> > >> >> > theyre using the right var size algorithm, as there is a
>> > few.
>> > >> >> >
>> > >> >> > On point 3)
>> > >> >> > We did the attributes, optional approach as originally
>> > there was
>> > >> marked
>> > >> >> > concern that headers would cause a message size overhead
>> > for others,
>> > >> who
>> > >> >> > don’t want them. As such this is the clean solution to
>> > achieve that.
>> > >> If
>> > >> >> > that no longer holds, and we don’t care that we add
>> 4bytes
>> > overhead,
>> > >> then
>> > >> >> > im happy to remove.
>> > >> >> >
>> > >> >> > I’m personally in favour of keeping the message as small
>> > as possible
>> > >> so
>> > >> >> > people don’t get shocks in perf and throughputs dues to
>> > message size,
>> > >> >> > unless they actively use the feature, as such I do
>> prefer
>> > the
>> > >> attribute bit
>> > >> >> > wise feature flag approach myself.
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> >
>> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
>> > jason@confluent.io> wrote:
>> > >> >> >
>> > >> >> > We have proposed a few significant changes to the
>> > message format
>> > >> in
>> > >> >> > KIP-98
>> > >> >> > which now seems likely to pass (perhaps with some
>> > iterations on
>> > >> >> > implementation details). It would be good to try and
>> > coordinate
>> > >> the
>> > >> >> > changes
>> > >> >> > in both of the proposals to make sure they are
>> > consistent and
>> > >> >> > compatible.
>> > >> >> >
>> > >> >> > I think using the attributes to indicate null
>> headers
>> > is a
>> > >> reasonable
>> > >> >> > approach. We have proposed to do the same thing for
>> > the message
>> > >> key and
>> > >> >> > value. That said, I sympathize with Jay's argument.
>> > Having
>> > >> multiple
>> > >> >> > ways to
>> > >> >> > specify a null value increases the overall
>> complexity
>> > of the
>> > >> protocol.
>> > >> >> > You
>> > >> >> > can see this just from the fact that you need the
>> > extra verbiage
>> > >> in the
>> > >> >> > protocol specification in this KIP and in KIP-98 to
>> > describe the
>> > >> >> > dependence
>> > >> >> > between the fields and the attributes. It seems
>> like a
>> > slippery
>> > >> slope
>> > >> >> > if
>> > >> >> > you start allowing different request types to
>> > implement the
>> > >> protocol
>> > >> >> > specification differently.
>> > >> >> >
>> > >> >> > You can also argue that the messages already are and
>> > are likely
>> > >> to
>> > >> >> > remain a
>> > >> >> > special case. For example, there is currently no
>> > generality in
>> > >> how
>> > >> >> > compressed message sets are represented that would
>> be
>> > applicable
>> > >> for
>> > >> >> > other
>> > >> >> > request types. Some might see this divergence as an
>> > unfortunate
>> > >> >> > protocol
>> > >> >> > deficiency which should be fixed; others might see
>> it
>> > as sort of
>> > >> the
>> > >> >> > inevitability of needing to optimize where it counts
>> > most. I'm
>> > >> probably
>> > >> >> > somewhere in between, but I think we probably all
>> > share the
>> > >> intuition
>> > >> >> > that
>> > >> >> > the protocol should be kept as consistent as
>> possible.
>> > With that
>> > >> in
>> > >> >> > mind,
>> > >> >> > here are a few comments:
>> > >> >> >
>> > >> >> > 1. One thing I found a little odd when reading the
>> > current
>> > >> proposal is
>> > >> >> > that
>> > >> >> > the headers are both represented as an array of
>> bytes
>> > and as an
>> > >> array
>> > >> >> > of
>> > >> >> > key/value pairs. I'd probably suggest something like
>> > this:
>> > >> >> >
>> > >> >> > Headers => [HeaderKey HeaderValue]
>> > >> >> > HeaderKey => String
>> > >> >> > HeaderValue => Bytes
>> > >> >> >
>> > >> >> > An array in the Kafka protocol is represented as a
>> > 4-byte integer
>> > >> >> > indicating the number of elements in the array
>> > followed by the
>> > >> >> > serialization of the elements. Unless I'm
>> > misunderstanding, what
>> > >> you
>> > >> >> > have
>> > >> >> > instead is the total size of the headers in bytes
>> > followed by the
>> > >> >> > elements.
>> > >> >> > I'm not sure I see any reason for this
>> inconsistency.
>> > >> >> >
>> > >> >> > 2. In KIP-98, we've introduced variable-length
>> integer
>> > fields.
>> > >> >> > Effectively,
>> > >> >> > we've enriched (or "complicated" as Jay might say ;)
>> > the protocol
>> > >> >> > specification to include the following types:
>> VarInt,
>> > VarLong,
>> > >> >> > UnsignedVarInt and UnsignedVarLong.
>> > >> >> >
>> > >> >> > Along with these primitives, we could introduce the
>> > following
>> > >> types:
>> > >> >> >
>> > >> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
>> > >> >> > NumberOfItems => UnsignedVarInt
>> > >> >> >
>> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1
>> > Item2 .. ItemN
>> > >> >> > NumberOfItemsOrNull => VarInt (-1 means null)
>> > >> >> >
>> > >> >> > And similarly for the `String` and `Bytes` types.
>> > These types
>> > >> can save
>> > >> >> > a
>> > >> >> > considerable amount of space in this proposal
>> because
>> > they can
>> > >> be used
>> > >> >> > for
>> > >> >> > both the number of headers included in the message
>> and
>> > the
>> > >> lengths of
>> > >> >> > the
>> > >> >> > header keys and values. We could do this instead:
>> > >> >> >
>> > >> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
>> > >> >> > HeaderKey => VarSizeString
>> > >> >> > HeaderValue => VarSizeBytes
>> > >> >> >
>> > >> >> > Combining the savings from the use of variable
>> length
>> > fields, the
>> > >> >> > benefit
>> > >> >> > of using the attributes to represent null seems
>> pretty
>> > small.
>> > >> >> >
>> > >> >> > 3. Whichever way we go (whether we use the
>> attributes
>> > or not), we
>> > >> >> > should at
>> > >> >> > least be consistent between this KIP and KIP-98. It
>> > would be very
>> > >> >> > strange
>> > >> >> > to have two ways to represent null values in the
>> same
>> > schema.
>> > >> Either
>> > >> >> > way is
>> > >> >> > OK with me. I think some message-level optimizations
>> > are
>> > >> justifiable,
>> > >> >> > but
>> > >> >> > the savings here seem minimal (a few bytes per
>> > message), so
>> > >> maybe it's
>> > >> >> > not
>> > >> >> > worth the cost of letting the message diverge even
>> > further from
>> > >> the
>> > >> >> > rest of
>> > >> >> > the protocol.
>> > >> >> >
>> > >> >> > -Jason
>> > >> >> >
>> > >> >> >
>> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
>> > >> radai.rosenblatt@gmail.com>
>> > >> >> > wrote:
>> > >> >> >
>> > >> >> > > I've trimmed the inline contents as this mail is
>> > getting too
>> > >> big for
>> > >> >> > the
>> > >> >> > > apache mailing list software to deliver :-(
>> > >> >> > >
>> > >> >> > > 1. the important thing for interoperability is for
>> > different
>> > >> >> > "interested
>> > >> >> > > parties" (plugins, infra layers/wrappers,
>> user-code)
>> > to be
>> > >> able to
>> > >> >> > stick
>> > >> >> > > pieces of metadata onto msgs without getting in
>> each
>> > other's
>> > >> way. a
>> > >> >> > common
>> > >> >> > > key scheme (Strings, as of the time of this
>> > writing?) is all
>> > >> thats
>> > >> >> > required
>> > >> >> > > for that. it is assumed that the other end
>> > interested in any
>> > >> such
>> > >> >> > piece of
>> > >> >> > > metadata knows the encoding, and byte[] provides
>> for
>> > the most
>> > >> >> > flexibility.
>> > >> >> > > i believe this is the same logic behind core kafka
>> > being
>> > >> >> > byte[]/byte[] -
>> > >> >> > > Strings are more "usable" but bytes are flexible
>> and
>> > so were
>> > >> chosen.
>> > >> >> > > Also - core kafka doesnt even do that good of a
>> job
>> > on
>> > >> usability of
>> > >> >> > the
>> > >> >> > > payload (example - i have to specify the nop
>> byte[]
>> > "decoders"
>> > >> >> > explicitly
>> > >> >> > > in conf), and again sacrificies usability for the
>> > sake of
>> > >> >> > performance (no
>> > >> >> > > convenient single-record processing as poll is a
>> > batch, lots of
>> > >> >> > obscure
>> > >> >> > > little config details exposing internals of the
>> > batching
>> > >> mechanism,
>> > >> >> > etc)
>> > >> >> > >
>> > >> >> > > this is also why i really dislike the idea of a
>> > "type system"
>> > >> for
>> > >> >> > header
>> > >> >> > > values, it further degrades the usability, adds
>> > complexity and
>> > >> will
>> > >> >> > > eventually get in people's way, also, it would be
>> > the 2nd/3rd
>> > >> >> > home-group
>> > >> >> > > serialization mechanism in core kafka (counting 2
>> > iterations
>> > >> of the
>> > >> >> > "type
>> > >> >> > > definition DSL")
>> > >> >> > >
>> > >> >> > > 2. this is an implementation detail, and not even
>> a
>> > very "user
>> > >> >> > facing" one?
>> > >> >> > > to the best of my understanding the vote process
>> is
>> > on proposed
>> > >> >> > > API/behaviour. also - since we're willing to go
>> with
>> > strings
>> > >> just
>> > >> >> > serialize
>> > >> >> > > a 0-sized header blob and IIUC you dont need any
>> > optionals
>> > >> anymore.
>> > >> >> > >
>> > >> >> > > 3. yes, we can :-)
>> > >> >> > >
>> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
>> > >> >> > Michael.Pearce@ig.com>
>> > >> >> > > wrote:
>> > >> >> > >
>> > >> >> > > > Hi Jay,
>> > >> >> > > >
>> > >> >> > > > 1) There was some initial debate on the value
>> > part, as youll
>> > >> note
>> > >> >> > String,
>> > >> >> > > > String headers were discounted early on. The
>> > reason for this
>> > >> is
>> > >> >> > > flexibility
>> > >> >> > > > and keeping in line with the flexibility of key,
>> > value of the
>> > >> >> > message
>> > >> >> > > > object itself. I don’t think it takes away from
>> an
>> > ecosystem
>> > >> as
>> > >> >> > each
>> > >> >> > > plugin
>> > >> >> > > > will care for their own key, this way ints,
>> > booleans , exotic
>> > >> >> > custom
>> > >> >> > > binary
>> > >> >> > > > can all be catered for=.
>> > >> >> > > > a. If you really wanted to push for a typed
>> value
>> > interface,
>> > >> I
>> > >> >> > wouldn’t
>> > >> >> > > > want just String values supported, but the the
>> > primatives
>> > >> plus
>> > >> >> > string and
>> > >> >> > > > also still keeping the ability to have a binary
>> > for custom
>> > >> >> > binaries that
>> > >> >> > > > some organisations may have.
>> > >> >> > > > i. I have written this slight alternative here,
>> > >> >> > > https://cwiki.apache.org/
>> > >> >> > > > confluence/display/KAFKA/KIP-
>> > 82+-+Add+Record+Headers+-+Typed
>> > >> >> > > > ii. Essentially the value bytes, has a leading
>> > byte overhead.
>> > >> >> > > > 1. This tells you what type the value is,
>> before
>> > reading
>> > >> the rest
>> > >> >> > of the
>> > >> >> > > > bytes, allowing serialisation/deserialization to
>> > and from the
>> > >> >> > primitives,
>> > >> >> > > > string and byte[]. This is akin to some other
>> > messaging
>> > >> systems.
>> > >> >> > > > 2) We are making it optional, so that for those
>> > not wanting
>> > >> >> > headers have
>> > >> >> > > 0
>> > >> >> > > > bytes overhead (think of it as a feature flag),
>> I
>> > don’t
>> > >> think this
>> > >> >> > is
>> > >> >> > > > complex, especially if comparing to changes
>> > proposed in
>> > >> other kips
>> > >> >> > like
>> > >> >> > > > kip-98.
>> > >> >> > > > a. If you really really don’t like this, we can
>> > drop it, but
>> > >> it
>> > >> >> > would
>> > >> >> > > mean
>> > >> >> > > > buying into 4 bytes extra overhead for users who
>> > do not want
>> > >> to use
>> > >> >> > > headers.
>> > >> >> > > > 3) In the summary yes, it is at a higher level,
>> > but I think
>> > >> this
>> > >> >> > is well
>> > >> >> > > > documented in the proposed changes section.
>> > >> >> > > > a. Added getHeaders method to Producer/Consumer
>> > record (that
>> > >> is it)
>> > >> >> > > > b. We’ve also detailed the new Headers class
>> that
>> > this method
>> > >> >> > returns
>> > >> >> > > that
>> > >> >> > > > encapsulates the headers protocol and logic.
>> > >> >> > > >
>> > >> >> > > > Best,
>> > >> >> > > > Mike
>> > >> >> > > >
>> > >> >> > > > ==Original questions from the vote thread from
>> > Jay.==
>> > >> >> > > >
>> > >> >> > > > Couple of things I think we still need to work
>> out:
>> > >> >> > > >
>> > >> >> > > > 1. I think we agree about the key, but I
>> think
>> > we haven't
>> > >> >> > talked about
>> > >> >> > > > the value yet. I think if our goal is an open
>> > ecosystem
>> > >> of these
>> > >> >> > > header
>> > >> >> > > > spread across many plugins from many systems
>> we
>> > should
>> > >> consider
>> > >> >> > making
>> > >> >> > > > this
>> > >> >> > > > a string as well so it can be printed, set
>> via
>> > a UI, set
>> > >> in
>> > >> >> > config,
>> > >> >> > > etc.
>> > >> >> > > > Basically encouraging pluggable serialization
>> > formats
>> > >> here will
>> > >> >> > lead
>> > >> >> > > to
>> > >> >> > > > a
>> > >> >> > > > bit of a tower of babel.
>> > >> >> > > > 2. This proposal still includes a pretty big
>> > change to our
>> > >> >> > > serialization
>> > >> >> > > > and protocol definition layer. Essentially
>> it is
>> > >> introducing an
>> > >> >> > > optional
>> > >> >> > > > type, where the format is data dependent. I
>> > think this is
>> > >> >> > actually a
>> > >> >> > > big
>> > >> >> > > > change though it doesn't seem like it. It
>> means
>> > you can no
>> > >> >> > longer
>> > >> >> > > > specify
>> > >> >> > > > this type with our type definition DSL, and
>> > likewise it
>> > >> requires
>> > >> >> > > custom
>> > >> >> > > > handling in client libs. This isn't a huge
>> > thing, since
>> > >> the
>> > >> >> > Record
>> > >> >> > > > definition is custom anyway, but I think this
>> > kind of
>> > >> protocol
>> > >> >> > > > inconsistency is very non-desirable and ties
>> > you to
>> > >> hand-coding
>> > >> >> > > things.
>> > >> >> > > > I
>> > >> >> > > > think the type should instead by [Key Value]
>> in
>> > our BNF,
>> > >> where
>> > >> >> > key and
>> > >> >> > > > value are both short strings as used
>> elsewhere.
>> > This
>> > >> brings it
>> > >> >> > in line
>> > >> >> > > > with
>> > >> >> > > > the rest of the protocol.
>> > >> >> > > > 3. Could we get more specific about the exact
>> > Java API
>> > >> change to
>> > >> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
>> > >> >> > > >
>> > >> >> > > > -Jay
>> > >> >> > > >
>> > >> >> > >
>> > >> >> >
>> > >> >> >
>> > >> >> > The information contained in this email is strictly
>> > confidential and
>> > >> for
>> > >> >> > the use of the addressee only, unless otherwise
>> indicated.
>> > If you
>> > >> are not
>> > >> >> > the intended recipient, please do not read, copy, use or
>> > disclose to
>> > >> others
>> > >> >> > this message or any attachment. Please also notify the
>> > sender by
>> > >> replying
>> > >> >> > to this email or by telephone (+44(020 7896 0011) and
>> then
>> > delete
>> > >> the email
>> > >> >> > and any copies of it. Opinions, conclusion (etc) that do
>> > not relate
>> > >> to the
>> > >> >> > official business of this company shall be understood as
>> > neither
>> > >> given nor
>> > >> >> > endorsed by it. IG is a trading name of IG Markets
>> Limited
>> > (a company
>> > >> >> > registered in England and Wales, company number
>> 04008957)
>> > and IG
>> > >> Index
>> > >> >> > Limited (a company registered in England and Wales,
>> > company number
>> > >> >> > 01190902). Registered address at Cannon Bridge House, 25
>> > Dowgate
>> > >> Hill,
>> > >> >> > London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > 195355)
>> > >> and IG
>> > >> >> > Index Limited (register number 114059) are authorised
>> and
>> > regulated
>> > >> by the
>> > >> >> > Financial Conduct Authority.
>> > >> >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Nacho - Ignacio Solis - isolis@igso.net
>> > >>
>> > >
>> > >
>> >
>> >
>> >
>> >
>> > The information contained in this email is strictly confidential and for
>> > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > the intended recipient, please do not read, copy, use or disclose to
>> others
>> > this message or any attachment. Please also notify the sender by
>> replying
>> > to this email or by telephone (+44(020 7896 0011) and then delete the
>> email
>> > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> the
>> > official business of this company shall be understood as neither given
>> nor
>> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > registered in England and Wales, company number 04008957) and IG Index
>> > Limited (a company registered in England and Wales, company number
>> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> > Index Limited (register number 114059) are authorised and regulated by
>> the
>> > Financial Conduct Authority.
>> >
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
>
> Re headers as byte array and future use by broker. This doesn't take away
> from that at all. Nor makes it difficult at all in my opinion.
Yeah, I didn't say it was difficult, only awkward. You wouldn't write the
schema that way if you were planning to use it on the brokers from the
beginning. Note also that one of the benefits of letting the broker
understand headers is that it can validate that they are properly
formatted. If cost is the only concern, we should confirm its impact
through performance testing.
One of the key use cases requires access on consume at per event/message
> level at the point that message is being processed, as such the batch
> interceptors and batch consume api isn't suitable. It needs to be at the
> record level.
I'm not sure I understand the point about batching. Interceptors are
applied per-message, right?
My intent on interceptors is to keep the usage of headers well-defined so
that they don't start leaking unnecessarily into applications. My guess is
that it's probably inevitable, but isolating it in the interceptors would
at least give people a second thought before deciding to use it. The main
challenge in my mind is figuring out how an MM use case would work. It
would be more cumbersome to replicate headers through an interceptor,
though arguably MM should be working at a lower level anyway.
-Jason
On Fri, Feb 17, 2017 at 10:16 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Re headers available on the record va interceptors only
>
> One of the key use cases requires access on consume at per event/message
> level at the point that message is being processed, as such the batch
> interceptors and batch consume api isn't suitable. It needs to be at the
> record level.
>
> This anyhow is similar to jms/http/amqp where headers are available to
> consuming applications.
>
> Re headers as byte array and future use by broker. This doesn't take away
> from that at all. Nor makes it difficult at all in my opinion.
>
>
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jason Gustafson <ja...@confluent.io>
> Sent: Friday, February 17, 2017 5:55:42 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> >
> > Would you be proposing in KIP-98 to convert the other message int’s (key
> > length, value length) also to varint to keep it uniform.
> > Also I assume there will be a static or helper method made to write/read
> > these in the client and server.
>
>
> Yes, that is what we are proposing, so using varints for headers would be
> consistent with the rest of the message. We have used static helper methods
> in our prototype implementation.
>
> The cost of parsing, we want to parse/interpret the headers lazily (this is
> > a key point brought up earlier in discussions)
>
>
> I'm a bit skeptical of this. Has anyone done the performance testing? I can
> probably implement it and test it if no one else has. I was also under the
> impression that there may be use cases down the road where the broker would
> need to interpret headers. That wouldn't be off the table in the future if
> it's represented as bytes, but it would be quite a bit more awkward, right?
>
> By the way, one question I have been wondering about. My understanding is
> that headers are primarily for use cases where a third-party components
> wants to enrich messages without needing to understand or modify the schema
> of the message key and value. For the applications which directly produce
> and consume the messages and control the key/value schema directly, it
> seems we would rather have them implement headers directly in their own
> schema. Supposing for the sake of argument that it was possible, my
> question is whether it be sufficient to expose the headers in the
> interceptor API and not in the common API?
>
> -Jason
>
> On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > On the point of varInts
> >
> > Would you be proposing in KIP-98 to convert the other message int’s (key
> > length, value length) also to varint to keep it uniform.
> > Also I assume there will be a static or helper method made to write/read
> > these in the client and server.
> >
> > Cheers
> > Mike
> >
> >
> >
> > On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
> >
> > On the point re: headers in the message protocol being a byte array
> > and not a count of elements followed by the elements. Again this was
> > discussed/argued previously.
> >
> > It was agreed on for a few reasons some of which you have obviously
> > picked up on:
> >
> > Broker is able to pass it through opaquely
> > The cost of parsing, we want to parse/interpret the headers lazily
> > (this is a key point brought up earlier in discussions)
> > Headers can be copied from consumer record to producer record (aka
> > mirror makers etc) without parsing if no changes are being made or being
> > looked at.
> > Keeps the broker agnostic to the format
> > You need an int32 either for the byte size of the headers, or for the
> > count of elements, so overheads are the same, but with going with an
> opaque
> > byte array has the above advantages.
> >
> > Cheers
> > Mike
> >
> >
> > On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > Sorry, should have noted that the performance testing was done
> > using the
> > producer performance tool shipped with Kafka.
> >
> > -Jason
> >
> > On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> > jason@confluent.io> wrote:
> >
> > > Hey Nacho,
> > >
> > > I've compared performance of our KIP-98 implementation with and
> > without
> > > varints. For messages around 128 bytes, we see an increase in
> > throughput of
> > > about 30% using the default configuration settings. At 256
> > bytes, the
> > > increase is around 16%. Obviously the performance converge as
> > messages get
> > > larger, but it seems well worth the cost. Note that we are also
> > seeing a
> > > substantial performance increase against trunk primarily
> because
> > of the
> > > much more efficient packing that varints provide us. Anything
> > adding to
> > > message overhead, such as record headers, would only increase
> > the relative
> > > difference. (Of course take these numbers with a grain of salt
> > since I have
> > > only used the default settings with both the producer and
> broker
> > on my
> > > local machine. We intend to provide more extensive performance
> > details as
> > > part of the work for KIP-98.)
> > >
> > > The implementation we are using is from protobuf (
> > > https://developers.google.com/protocol-buffers/docs/encoding),
> > which is
> > > also used in HBase. It is trivial to implement and as far as I
> > know doesn't
> > > suffer from the aliasing problem you are describing. I checked
> > with Magnus
> > > (the author of librdkafka) and he agreed that the savings
> seemed
> > worth the
> > > cost of implementation.
> > >
> > > -Jason
> > >
> > > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <
> isolis@igso.net>
> > wrote:
> > >
> > >> -VarInts
> > >>
> > >> I'm one of the people (if not the most) opposed to VarInts.
> > VarInts
> > >> have a place, but this is not it. (We had a large discussion
> > about
> > >> them at the beginning of KIP-82 time)
> > >>
> > >> If anybody has real life performance numbers of VarInts
> > improving
> > >> things or significantly reducing resources I would like to
> know
> > what
> > >> that case may be. Yes, you can save some bytes here and there,
> > but
> > >> this is probably insignificant to the overall system behavior
> > and
> > >> storage requirements. -- I say this with respect to using
> > VarInts in
> > >> the protocol itself, not as part of the data.
> > >>
> > >> VarInts require you to parse the Int before using it and
> > depending on
> > >> the encoding they can suffer from aliasing (multiple
> > representations
> > >> for the same value).
> > >>
> > >> Why add complexity?
> > >>
> > >> Nacho
> > >>
> > >>
> > >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> > cmccabe@apache.org>
> > >> wrote:
> > >> > +1 for varints here-- it would save quite a bit of space.
> > They are
> > >> > pretty quick to implement as well.
> > >> >
> > >> > I think it makes sense for values to be byte arrays. Users
> > might want
> > >> > to attach arbitrary payloads; they shouldn't be forced to
> > serialize
> > >> > everything to Java strings.
> > >> >
> > >> > best,
> > >> > Colin
> > >> >
> > >> >
> > >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> > >> >> Hey Michael,
> > >> >>
> > >> >> Hmm, I guess the point of representing it as bytes is to
> > allow the
> > >> broker
> > >> >> to pass it through opaquely? Is the cost of parsing them a
> > concern, or
> > >> >> are
> > >> >> we simply trying to ensure that the broker stays agnostic
> to
> > the
> > >> format?
> > >> >>
> > >> >> On varints, I think adding support for them makes less
> sense
> > for an
> > >> >> isolated use case, but as part of a more holistic change
> > (such as what
> > >> we
> > >> >> have proposed in KIP-98), I think they are justifiable. If
> > we add them,
> > >> >> then the need to use attributes becomes quite a bit weaker,
> > right? The
> > >> >> other thing I find slightly odd is the fact that null
> > headers has no
> > >> >> actual
> > >> >> semantic meaning for the message (unlike null keys and
> > values). It is
> > >> >> just
> > >> >> a space optimization. It seems a bit better to always use
> > size 0 to
> > >> >> indicate having no headers.
> > >> >>
> > >> >> Overall, the main point is ensuring that the message schema
> > remains
> > >> >> consistent, either within the larger protocol, or at a
> > minimum within
> > >> the
> > >> >> message itself.
> > >> >>
> > >> >> -Jason
> > >> >>
> > >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> > Michael.Pearce@ig.com
> > >> >
> > >> >> wrote:
> > >> >>
> > >> >> > Hi Jason,
> > >> >> >
> > >> >> > On point 1) in the message protocol the headers are
> simply
> > a byte
> > >> array,
> > >> >> > as like the key or value, this is to clearly demarcate
> the
> > header in
> > >> the
> > >> >> > core message. Then the header byte array in the core
> > message is an
> > >> array of
> > >> >> > key, value pairs. This is what it is denoting.
> > >> >> >
> > >> >> > Then this would be I guess in the given notation:
> > >> >> >
> > >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> > >> >> > KeyLength => int32 <-----------------NEW size of the
> > byte[] of
> > >> the
> > >> >> > serialised key value
> > >> >> > Key => bytes <---------------------- NEW serialised
> > string (UTF8)
> > >> >> > bytes of the header key
> > >> >> > ValueLength => int32 <-------------- NEW size of the
> > byte[] of
> > >> the
> > >> >> > serialised header value
> > >> >> > Value => bytes <-------------------- NEW serialised
> > form of the
> > >> header
> > >> >> > value
> > >> >> >
> > >> >> > The key length and value length is matching the way the
> > protocol is
> > >> >> > defined in the core message currently.
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On point 2)
> > >> >> > Var sized ints, this was discussed much earlier on, in
> > fact I had
> > >> >> > suggested it myself (with Hadoop references), the
> > complexity of this
> > >> >> > compared to having a simpler protocol was argued and
> > agreed it
> > >> wasn’t worth
> > >> >> > the complexity as all other clients in other languages
> > would need to
> > >> ensure
> > >> >> > theyre using the right var size algorithm, as there is a
> > few.
> > >> >> >
> > >> >> > On point 3)
> > >> >> > We did the attributes, optional approach as originally
> > there was
> > >> marked
> > >> >> > concern that headers would cause a message size overhead
> > for others,
> > >> who
> > >> >> > don’t want them. As such this is the clean solution to
> > achieve that.
> > >> If
> > >> >> > that no longer holds, and we don’t care that we add
> 4bytes
> > overhead,
> > >> then
> > >> >> > im happy to remove.
> > >> >> >
> > >> >> > I’m personally in favour of keeping the message as small
> > as possible
> > >> so
> > >> >> > people don’t get shocks in perf and throughputs dues to
> > message size,
> > >> >> > unless they actively use the feature, as such I do prefer
> > the
> > >> attribute bit
> > >> >> > wise feature flag approach myself.
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> > jason@confluent.io> wrote:
> > >> >> >
> > >> >> > We have proposed a few significant changes to the
> > message format
> > >> in
> > >> >> > KIP-98
> > >> >> > which now seems likely to pass (perhaps with some
> > iterations on
> > >> >> > implementation details). It would be good to try and
> > coordinate
> > >> the
> > >> >> > changes
> > >> >> > in both of the proposals to make sure they are
> > consistent and
> > >> >> > compatible.
> > >> >> >
> > >> >> > I think using the attributes to indicate null headers
> > is a
> > >> reasonable
> > >> >> > approach. We have proposed to do the same thing for
> > the message
> > >> key and
> > >> >> > value. That said, I sympathize with Jay's argument.
> > Having
> > >> multiple
> > >> >> > ways to
> > >> >> > specify a null value increases the overall complexity
> > of the
> > >> protocol.
> > >> >> > You
> > >> >> > can see this just from the fact that you need the
> > extra verbiage
> > >> in the
> > >> >> > protocol specification in this KIP and in KIP-98 to
> > describe the
> > >> >> > dependence
> > >> >> > between the fields and the attributes. It seems like
> a
> > slippery
> > >> slope
> > >> >> > if
> > >> >> > you start allowing different request types to
> > implement the
> > >> protocol
> > >> >> > specification differently.
> > >> >> >
> > >> >> > You can also argue that the messages already are and
> > are likely
> > >> to
> > >> >> > remain a
> > >> >> > special case. For example, there is currently no
> > generality in
> > >> how
> > >> >> > compressed message sets are represented that would be
> > applicable
> > >> for
> > >> >> > other
> > >> >> > request types. Some might see this divergence as an
> > unfortunate
> > >> >> > protocol
> > >> >> > deficiency which should be fixed; others might see it
> > as sort of
> > >> the
> > >> >> > inevitability of needing to optimize where it counts
> > most. I'm
> > >> probably
> > >> >> > somewhere in between, but I think we probably all
> > share the
> > >> intuition
> > >> >> > that
> > >> >> > the protocol should be kept as consistent as
> possible.
> > With that
> > >> in
> > >> >> > mind,
> > >> >> > here are a few comments:
> > >> >> >
> > >> >> > 1. One thing I found a little odd when reading the
> > current
> > >> proposal is
> > >> >> > that
> > >> >> > the headers are both represented as an array of bytes
> > and as an
> > >> array
> > >> >> > of
> > >> >> > key/value pairs. I'd probably suggest something like
> > this:
> > >> >> >
> > >> >> > Headers => [HeaderKey HeaderValue]
> > >> >> > HeaderKey => String
> > >> >> > HeaderValue => Bytes
> > >> >> >
> > >> >> > An array in the Kafka protocol is represented as a
> > 4-byte integer
> > >> >> > indicating the number of elements in the array
> > followed by the
> > >> >> > serialization of the elements. Unless I'm
> > misunderstanding, what
> > >> you
> > >> >> > have
> > >> >> > instead is the total size of the headers in bytes
> > followed by the
> > >> >> > elements.
> > >> >> > I'm not sure I see any reason for this inconsistency.
> > >> >> >
> > >> >> > 2. In KIP-98, we've introduced variable-length
> integer
> > fields.
> > >> >> > Effectively,
> > >> >> > we've enriched (or "complicated" as Jay might say ;)
> > the protocol
> > >> >> > specification to include the following types: VarInt,
> > VarLong,
> > >> >> > UnsignedVarInt and UnsignedVarLong.
> > >> >> >
> > >> >> > Along with these primitives, we could introduce the
> > following
> > >> types:
> > >> >> >
> > >> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> > >> >> > NumberOfItems => UnsignedVarInt
> > >> >> >
> > >> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1
> > Item2 .. ItemN
> > >> >> > NumberOfItemsOrNull => VarInt (-1 means null)
> > >> >> >
> > >> >> > And similarly for the `String` and `Bytes` types.
> > These types
> > >> can save
> > >> >> > a
> > >> >> > considerable amount of space in this proposal because
> > they can
> > >> be used
> > >> >> > for
> > >> >> > both the number of headers included in the message
> and
> > the
> > >> lengths of
> > >> >> > the
> > >> >> > header keys and values. We could do this instead:
> > >> >> >
> > >> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
> > >> >> > HeaderKey => VarSizeString
> > >> >> > HeaderValue => VarSizeBytes
> > >> >> >
> > >> >> > Combining the savings from the use of variable length
> > fields, the
> > >> >> > benefit
> > >> >> > of using the attributes to represent null seems
> pretty
> > small.
> > >> >> >
> > >> >> > 3. Whichever way we go (whether we use the attributes
> > or not), we
> > >> >> > should at
> > >> >> > least be consistent between this KIP and KIP-98. It
> > would be very
> > >> >> > strange
> > >> >> > to have two ways to represent null values in the same
> > schema.
> > >> Either
> > >> >> > way is
> > >> >> > OK with me. I think some message-level optimizations
> > are
> > >> justifiable,
> > >> >> > but
> > >> >> > the savings here seem minimal (a few bytes per
> > message), so
> > >> maybe it's
> > >> >> > not
> > >> >> > worth the cost of letting the message diverge even
> > further from
> > >> the
> > >> >> > rest of
> > >> >> > the protocol.
> > >> >> >
> > >> >> > -Jason
> > >> >> >
> > >> >> >
> > >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> > >> radai.rosenblatt@gmail.com>
> > >> >> > wrote:
> > >> >> >
> > >> >> > > I've trimmed the inline contents as this mail is
> > getting too
> > >> big for
> > >> >> > the
> > >> >> > > apache mailing list software to deliver :-(
> > >> >> > >
> > >> >> > > 1. the important thing for interoperability is for
> > different
> > >> >> > "interested
> > >> >> > > parties" (plugins, infra layers/wrappers,
> user-code)
> > to be
> > >> able to
> > >> >> > stick
> > >> >> > > pieces of metadata onto msgs without getting in
> each
> > other's
> > >> way. a
> > >> >> > common
> > >> >> > > key scheme (Strings, as of the time of this
> > writing?) is all
> > >> thats
> > >> >> > required
> > >> >> > > for that. it is assumed that the other end
> > interested in any
> > >> such
> > >> >> > piece of
> > >> >> > > metadata knows the encoding, and byte[] provides
> for
> > the most
> > >> >> > flexibility.
> > >> >> > > i believe this is the same logic behind core kafka
> > being
> > >> >> > byte[]/byte[] -
> > >> >> > > Strings are more "usable" but bytes are flexible
> and
> > so were
> > >> chosen.
> > >> >> > > Also - core kafka doesnt even do that good of a job
> > on
> > >> usability of
> > >> >> > the
> > >> >> > > payload (example - i have to specify the nop byte[]
> > "decoders"
> > >> >> > explicitly
> > >> >> > > in conf), and again sacrificies usability for the
> > sake of
> > >> >> > performance (no
> > >> >> > > convenient single-record processing as poll is a
> > batch, lots of
> > >> >> > obscure
> > >> >> > > little config details exposing internals of the
> > batching
> > >> mechanism,
> > >> >> > etc)
> > >> >> > >
> > >> >> > > this is also why i really dislike the idea of a
> > "type system"
> > >> for
> > >> >> > header
> > >> >> > > values, it further degrades the usability, adds
> > complexity and
> > >> will
> > >> >> > > eventually get in people's way, also, it would be
> > the 2nd/3rd
> > >> >> > home-group
> > >> >> > > serialization mechanism in core kafka (counting 2
> > iterations
> > >> of the
> > >> >> > "type
> > >> >> > > definition DSL")
> > >> >> > >
> > >> >> > > 2. this is an implementation detail, and not even a
> > very "user
> > >> >> > facing" one?
> > >> >> > > to the best of my understanding the vote process is
> > on proposed
> > >> >> > > API/behaviour. also - since we're willing to go
> with
> > strings
> > >> just
> > >> >> > serialize
> > >> >> > > a 0-sized header blob and IIUC you dont need any
> > optionals
> > >> anymore.
> > >> >> > >
> > >> >> > > 3. yes, we can :-)
> > >> >> > >
> > >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
> > >> >> > Michael.Pearce@ig.com>
> > >> >> > > wrote:
> > >> >> > >
> > >> >> > > > Hi Jay,
> > >> >> > > >
> > >> >> > > > 1) There was some initial debate on the value
> > part, as youll
> > >> note
> > >> >> > String,
> > >> >> > > > String headers were discounted early on. The
> > reason for this
> > >> is
> > >> >> > > flexibility
> > >> >> > > > and keeping in line with the flexibility of key,
> > value of the
> > >> >> > message
> > >> >> > > > object itself. I don’t think it takes away from
> an
> > ecosystem
> > >> as
> > >> >> > each
> > >> >> > > plugin
> > >> >> > > > will care for their own key, this way ints,
> > booleans , exotic
> > >> >> > custom
> > >> >> > > binary
> > >> >> > > > can all be catered for=.
> > >> >> > > > a. If you really wanted to push for a typed value
> > interface,
> > >> I
> > >> >> > wouldn’t
> > >> >> > > > want just String values supported, but the the
> > primatives
> > >> plus
> > >> >> > string and
> > >> >> > > > also still keeping the ability to have a binary
> > for custom
> > >> >> > binaries that
> > >> >> > > > some organisations may have.
> > >> >> > > > i. I have written this slight alternative here,
> > >> >> > > https://cwiki.apache.org/
> > >> >> > > > confluence/display/KAFKA/KIP-
> > 82+-+Add+Record+Headers+-+Typed
> > >> >> > > > ii. Essentially the value bytes, has a leading
> > byte overhead.
> > >> >> > > > 1. This tells you what type the value is, before
> > reading
> > >> the rest
> > >> >> > of the
> > >> >> > > > bytes, allowing serialisation/deserialization to
> > and from the
> > >> >> > primitives,
> > >> >> > > > string and byte[]. This is akin to some other
> > messaging
> > >> systems.
> > >> >> > > > 2) We are making it optional, so that for those
> > not wanting
> > >> >> > headers have
> > >> >> > > 0
> > >> >> > > > bytes overhead (think of it as a feature flag), I
> > don’t
> > >> think this
> > >> >> > is
> > >> >> > > > complex, especially if comparing to changes
> > proposed in
> > >> other kips
> > >> >> > like
> > >> >> > > > kip-98.
> > >> >> > > > a. If you really really don’t like this, we can
> > drop it, but
> > >> it
> > >> >> > would
> > >> >> > > mean
> > >> >> > > > buying into 4 bytes extra overhead for users who
> > do not want
> > >> to use
> > >> >> > > headers.
> > >> >> > > > 3) In the summary yes, it is at a higher level,
> > but I think
> > >> this
> > >> >> > is well
> > >> >> > > > documented in the proposed changes section.
> > >> >> > > > a. Added getHeaders method to Producer/Consumer
> > record (that
> > >> is it)
> > >> >> > > > b. We’ve also detailed the new Headers class that
> > this method
> > >> >> > returns
> > >> >> > > that
> > >> >> > > > encapsulates the headers protocol and logic.
> > >> >> > > >
> > >> >> > > > Best,
> > >> >> > > > Mike
> > >> >> > > >
> > >> >> > > > ==Original questions from the vote thread from
> > Jay.==
> > >> >> > > >
> > >> >> > > > Couple of things I think we still need to work
> out:
> > >> >> > > >
> > >> >> > > > 1. I think we agree about the key, but I think
> > we haven't
> > >> >> > talked about
> > >> >> > > > the value yet. I think if our goal is an open
> > ecosystem
> > >> of these
> > >> >> > > header
> > >> >> > > > spread across many plugins from many systems
> we
> > should
> > >> consider
> > >> >> > making
> > >> >> > > > this
> > >> >> > > > a string as well so it can be printed, set via
> > a UI, set
> > >> in
> > >> >> > config,
> > >> >> > > etc.
> > >> >> > > > Basically encouraging pluggable serialization
> > formats
> > >> here will
> > >> >> > lead
> > >> >> > > to
> > >> >> > > > a
> > >> >> > > > bit of a tower of babel.
> > >> >> > > > 2. This proposal still includes a pretty big
> > change to our
> > >> >> > > serialization
> > >> >> > > > and protocol definition layer. Essentially it
> is
> > >> introducing an
> > >> >> > > optional
> > >> >> > > > type, where the format is data dependent. I
> > think this is
> > >> >> > actually a
> > >> >> > > big
> > >> >> > > > change though it doesn't seem like it. It
> means
> > you can no
> > >> >> > longer
> > >> >> > > > specify
> > >> >> > > > this type with our type definition DSL, and
> > likewise it
> > >> requires
> > >> >> > > custom
> > >> >> > > > handling in client libs. This isn't a huge
> > thing, since
> > >> the
> > >> >> > Record
> > >> >> > > > definition is custom anyway, but I think this
> > kind of
> > >> protocol
> > >> >> > > > inconsistency is very non-desirable and ties
> > you to
> > >> hand-coding
> > >> >> > > things.
> > >> >> > > > I
> > >> >> > > > think the type should instead by [Key Value]
> in
> > our BNF,
> > >> where
> > >> >> > key and
> > >> >> > > > value are both short strings as used
> elsewhere.
> > This
> > >> brings it
> > >> >> > in line
> > >> >> > > > with
> > >> >> > > > the rest of the protocol.
> > >> >> > > > 3. Could we get more specific about the exact
> > Java API
> > >> change to
> > >> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
> > >> >> > > >
> > >> >> > > > -Jay
> > >> >> > > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >> > The information contained in this email is strictly
> > confidential and
> > >> for
> > >> >> > the use of the addressee only, unless otherwise
> indicated.
> > If you
> > >> are not
> > >> >> > the intended recipient, please do not read, copy, use or
> > disclose to
> > >> others
> > >> >> > this message or any attachment. Please also notify the
> > sender by
> > >> replying
> > >> >> > to this email or by telephone (+44(020 7896 0011) and
> then
> > delete
> > >> the email
> > >> >> > and any copies of it. Opinions, conclusion (etc) that do
> > not relate
> > >> to the
> > >> >> > official business of this company shall be understood as
> > neither
> > >> given nor
> > >> >> > endorsed by it. IG is a trading name of IG Markets
> Limited
> > (a company
> > >> >> > registered in England and Wales, company number 04008957)
> > and IG
> > >> Index
> > >> >> > Limited (a company registered in England and Wales,
> > company number
> > >> >> > 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > >> Hill,
> > >> >> > London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > >> and IG
> > >> >> > Index Limited (register number 114059) are authorised and
> > regulated
> > >> by the
> > >> >> > Financial Conduct Authority.
> > >> >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Nacho - Ignacio Solis - isolis@igso.net
> > >>
> > >
> > >
> >
> >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Re headers available on the record va interceptors only
One of the key use cases requires access on consume at per event/message level at the point that message is being processed, as such the batch interceptors and batch consume api isn't suitable. It needs to be at the record level.
This anyhow is similar to jms/http/amqp where headers are available to consuming applications.
Re headers as byte array and future use by broker. This doesn't take away from that at all. Nor makes it difficult at all in my opinion.
Sent using OWA for iPhone
________________________________________
From: Jason Gustafson <ja...@confluent.io>
Sent: Friday, February 17, 2017 5:55:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> Would you be proposing in KIP-98 to convert the other message int’s (key
> length, value length) also to varint to keep it uniform.
> Also I assume there will be a static or helper method made to write/read
> these in the client and server.
Yes, that is what we are proposing, so using varints for headers would be
consistent with the rest of the message. We have used static helper methods
in our prototype implementation.
The cost of parsing, we want to parse/interpret the headers lazily (this is
> a key point brought up earlier in discussions)
I'm a bit skeptical of this. Has anyone done the performance testing? I can
probably implement it and test it if no one else has. I was also under the
impression that there may be use cases down the road where the broker would
need to interpret headers. That wouldn't be off the table in the future if
it's represented as bytes, but it would be quite a bit more awkward, right?
By the way, one question I have been wondering about. My understanding is
that headers are primarily for use cases where a third-party components
wants to enrich messages without needing to understand or modify the schema
of the message key and value. For the applications which directly produce
and consume the messages and control the key/value schema directly, it
seems we would rather have them implement headers directly in their own
schema. Supposing for the sake of argument that it was possible, my
question is whether it be sufficient to expose the headers in the
interceptor API and not in the common API?
-Jason
On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <Mi...@ig.com>
wrote:
> On the point of varInts
>
> Would you be proposing in KIP-98 to convert the other message int’s (key
> length, value length) also to varint to keep it uniform.
> Also I assume there will be a static or helper method made to write/read
> these in the client and server.
>
> Cheers
> Mike
>
>
>
> On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
>
> On the point re: headers in the message protocol being a byte array
> and not a count of elements followed by the elements. Again this was
> discussed/argued previously.
>
> It was agreed on for a few reasons some of which you have obviously
> picked up on:
>
> Broker is able to pass it through opaquely
> The cost of parsing, we want to parse/interpret the headers lazily
> (this is a key point brought up earlier in discussions)
> Headers can be copied from consumer record to producer record (aka
> mirror makers etc) without parsing if no changes are being made or being
> looked at.
> Keeps the broker agnostic to the format
> You need an int32 either for the byte size of the headers, or for the
> count of elements, so overheads are the same, but with going with an opaque
> byte array has the above advantages.
>
> Cheers
> Mike
>
>
> On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Sorry, should have noted that the performance testing was done
> using the
> producer performance tool shipped with Kafka.
>
> -Jason
>
> On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> jason@confluent.io> wrote:
>
> > Hey Nacho,
> >
> > I've compared performance of our KIP-98 implementation with and
> without
> > varints. For messages around 128 bytes, we see an increase in
> throughput of
> > about 30% using the default configuration settings. At 256
> bytes, the
> > increase is around 16%. Obviously the performance converge as
> messages get
> > larger, but it seems well worth the cost. Note that we are also
> seeing a
> > substantial performance increase against trunk primarily because
> of the
> > much more efficient packing that varints provide us. Anything
> adding to
> > message overhead, such as record headers, would only increase
> the relative
> > difference. (Of course take these numbers with a grain of salt
> since I have
> > only used the default settings with both the producer and broker
> on my
> > local machine. We intend to provide more extensive performance
> details as
> > part of the work for KIP-98.)
> >
> > The implementation we are using is from protobuf (
> > https://developers.google.com/protocol-buffers/docs/encoding),
> which is
> > also used in HBase. It is trivial to implement and as far as I
> know doesn't
> > suffer from the aliasing problem you are describing. I checked
> with Magnus
> > (the author of librdkafka) and he agreed that the savings seemed
> worth the
> > cost of implementation.
> >
> > -Jason
> >
> > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <is...@igso.net>
> wrote:
> >
> >> -VarInts
> >>
> >> I'm one of the people (if not the most) opposed to VarInts.
> VarInts
> >> have a place, but this is not it. (We had a large discussion
> about
> >> them at the beginning of KIP-82 time)
> >>
> >> If anybody has real life performance numbers of VarInts
> improving
> >> things or significantly reducing resources I would like to know
> what
> >> that case may be. Yes, you can save some bytes here and there,
> but
> >> this is probably insignificant to the overall system behavior
> and
> >> storage requirements. -- I say this with respect to using
> VarInts in
> >> the protocol itself, not as part of the data.
> >>
> >> VarInts require you to parse the Int before using it and
> depending on
> >> the encoding they can suffer from aliasing (multiple
> representations
> >> for the same value).
> >>
> >> Why add complexity?
> >>
> >> Nacho
> >>
> >>
> >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> cmccabe@apache.org>
> >> wrote:
> >> > +1 for varints here-- it would save quite a bit of space.
> They are
> >> > pretty quick to implement as well.
> >> >
> >> > I think it makes sense for values to be byte arrays. Users
> might want
> >> > to attach arbitrary payloads; they shouldn't be forced to
> serialize
> >> > everything to Java strings.
> >> >
> >> > best,
> >> > Colin
> >> >
> >> >
> >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> >> >> Hey Michael,
> >> >>
> >> >> Hmm, I guess the point of representing it as bytes is to
> allow the
> >> broker
> >> >> to pass it through opaquely? Is the cost of parsing them a
> concern, or
> >> >> are
> >> >> we simply trying to ensure that the broker stays agnostic to
> the
> >> format?
> >> >>
> >> >> On varints, I think adding support for them makes less sense
> for an
> >> >> isolated use case, but as part of a more holistic change
> (such as what
> >> we
> >> >> have proposed in KIP-98), I think they are justifiable. If
> we add them,
> >> >> then the need to use attributes becomes quite a bit weaker,
> right? The
> >> >> other thing I find slightly odd is the fact that null
> headers has no
> >> >> actual
> >> >> semantic meaning for the message (unlike null keys and
> values). It is
> >> >> just
> >> >> a space optimization. It seems a bit better to always use
> size 0 to
> >> >> indicate having no headers.
> >> >>
> >> >> Overall, the main point is ensuring that the message schema
> remains
> >> >> consistent, either within the larger protocol, or at a
> minimum within
> >> the
> >> >> message itself.
> >> >>
> >> >> -Jason
> >> >>
> >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> Michael.Pearce@ig.com
> >> >
> >> >> wrote:
> >> >>
> >> >> > Hi Jason,
> >> >> >
> >> >> > On point 1) in the message protocol the headers are simply
> a byte
> >> array,
> >> >> > as like the key or value, this is to clearly demarcate the
> header in
> >> the
> >> >> > core message. Then the header byte array in the core
> message is an
> >> array of
> >> >> > key, value pairs. This is what it is denoting.
> >> >> >
> >> >> > Then this would be I guess in the given notation:
> >> >> >
> >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> >> >> > KeyLength => int32 <-----------------NEW size of the
> byte[] of
> >> the
> >> >> > serialised key value
> >> >> > Key => bytes <---------------------- NEW serialised
> string (UTF8)
> >> >> > bytes of the header key
> >> >> > ValueLength => int32 <-------------- NEW size of the
> byte[] of
> >> the
> >> >> > serialised header value
> >> >> > Value => bytes <-------------------- NEW serialised
> form of the
> >> header
> >> >> > value
> >> >> >
> >> >> > The key length and value length is matching the way the
> protocol is
> >> >> > defined in the core message currently.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On point 2)
> >> >> > Var sized ints, this was discussed much earlier on, in
> fact I had
> >> >> > suggested it myself (with Hadoop references), the
> complexity of this
> >> >> > compared to having a simpler protocol was argued and
> agreed it
> >> wasn’t worth
> >> >> > the complexity as all other clients in other languages
> would need to
> >> ensure
> >> >> > theyre using the right var size algorithm, as there is a
> few.
> >> >> >
> >> >> > On point 3)
> >> >> > We did the attributes, optional approach as originally
> there was
> >> marked
> >> >> > concern that headers would cause a message size overhead
> for others,
> >> who
> >> >> > don’t want them. As such this is the clean solution to
> achieve that.
> >> If
> >> >> > that no longer holds, and we don’t care that we add 4bytes
> overhead,
> >> then
> >> >> > im happy to remove.
> >> >> >
> >> >> > I’m personally in favour of keeping the message as small
> as possible
> >> so
> >> >> > people don’t get shocks in perf and throughputs dues to
> message size,
> >> >> > unless they actively use the feature, as such I do prefer
> the
> >> attribute bit
> >> >> > wise feature flag approach myself.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> jason@confluent.io> wrote:
> >> >> >
> >> >> > We have proposed a few significant changes to the
> message format
> >> in
> >> >> > KIP-98
> >> >> > which now seems likely to pass (perhaps with some
> iterations on
> >> >> > implementation details). It would be good to try and
> coordinate
> >> the
> >> >> > changes
> >> >> > in both of the proposals to make sure they are
> consistent and
> >> >> > compatible.
> >> >> >
> >> >> > I think using the attributes to indicate null headers
> is a
> >> reasonable
> >> >> > approach. We have proposed to do the same thing for
> the message
> >> key and
> >> >> > value. That said, I sympathize with Jay's argument.
> Having
> >> multiple
> >> >> > ways to
> >> >> > specify a null value increases the overall complexity
> of the
> >> protocol.
> >> >> > You
> >> >> > can see this just from the fact that you need the
> extra verbiage
> >> in the
> >> >> > protocol specification in this KIP and in KIP-98 to
> describe the
> >> >> > dependence
> >> >> > between the fields and the attributes. It seems like a
> slippery
> >> slope
> >> >> > if
> >> >> > you start allowing different request types to
> implement the
> >> protocol
> >> >> > specification differently.
> >> >> >
> >> >> > You can also argue that the messages already are and
> are likely
> >> to
> >> >> > remain a
> >> >> > special case. For example, there is currently no
> generality in
> >> how
> >> >> > compressed message sets are represented that would be
> applicable
> >> for
> >> >> > other
> >> >> > request types. Some might see this divergence as an
> unfortunate
> >> >> > protocol
> >> >> > deficiency which should be fixed; others might see it
> as sort of
> >> the
> >> >> > inevitability of needing to optimize where it counts
> most. I'm
> >> probably
> >> >> > somewhere in between, but I think we probably all
> share the
> >> intuition
> >> >> > that
> >> >> > the protocol should be kept as consistent as possible.
> With that
> >> in
> >> >> > mind,
> >> >> > here are a few comments:
> >> >> >
> >> >> > 1. One thing I found a little odd when reading the
> current
> >> proposal is
> >> >> > that
> >> >> > the headers are both represented as an array of bytes
> and as an
> >> array
> >> >> > of
> >> >> > key/value pairs. I'd probably suggest something like
> this:
> >> >> >
> >> >> > Headers => [HeaderKey HeaderValue]
> >> >> > HeaderKey => String
> >> >> > HeaderValue => Bytes
> >> >> >
> >> >> > An array in the Kafka protocol is represented as a
> 4-byte integer
> >> >> > indicating the number of elements in the array
> followed by the
> >> >> > serialization of the elements. Unless I'm
> misunderstanding, what
> >> you
> >> >> > have
> >> >> > instead is the total size of the headers in bytes
> followed by the
> >> >> > elements.
> >> >> > I'm not sure I see any reason for this inconsistency.
> >> >> >
> >> >> > 2. In KIP-98, we've introduced variable-length integer
> fields.
> >> >> > Effectively,
> >> >> > we've enriched (or "complicated" as Jay might say ;)
> the protocol
> >> >> > specification to include the following types: VarInt,
> VarLong,
> >> >> > UnsignedVarInt and UnsignedVarLong.
> >> >> >
> >> >> > Along with these primitives, we could introduce the
> following
> >> types:
> >> >> >
> >> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> >> >> > NumberOfItems => UnsignedVarInt
> >> >> >
> >> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1
> Item2 .. ItemN
> >> >> > NumberOfItemsOrNull => VarInt (-1 means null)
> >> >> >
> >> >> > And similarly for the `String` and `Bytes` types.
> These types
> >> can save
> >> >> > a
> >> >> > considerable amount of space in this proposal because
> they can
> >> be used
> >> >> > for
> >> >> > both the number of headers included in the message and
> the
> >> lengths of
> >> >> > the
> >> >> > header keys and values. We could do this instead:
> >> >> >
> >> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
> >> >> > HeaderKey => VarSizeString
> >> >> > HeaderValue => VarSizeBytes
> >> >> >
> >> >> > Combining the savings from the use of variable length
> fields, the
> >> >> > benefit
> >> >> > of using the attributes to represent null seems pretty
> small.
> >> >> >
> >> >> > 3. Whichever way we go (whether we use the attributes
> or not), we
> >> >> > should at
> >> >> > least be consistent between this KIP and KIP-98. It
> would be very
> >> >> > strange
> >> >> > to have two ways to represent null values in the same
> schema.
> >> Either
> >> >> > way is
> >> >> > OK with me. I think some message-level optimizations
> are
> >> justifiable,
> >> >> > but
> >> >> > the savings here seem minimal (a few bytes per
> message), so
> >> maybe it's
> >> >> > not
> >> >> > worth the cost of letting the message diverge even
> further from
> >> the
> >> >> > rest of
> >> >> > the protocol.
> >> >> >
> >> >> > -Jason
> >> >> >
> >> >> >
> >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> >> radai.rosenblatt@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > I've trimmed the inline contents as this mail is
> getting too
> >> big for
> >> >> > the
> >> >> > > apache mailing list software to deliver :-(
> >> >> > >
> >> >> > > 1. the important thing for interoperability is for
> different
> >> >> > "interested
> >> >> > > parties" (plugins, infra layers/wrappers, user-code)
> to be
> >> able to
> >> >> > stick
> >> >> > > pieces of metadata onto msgs without getting in each
> other's
> >> way. a
> >> >> > common
> >> >> > > key scheme (Strings, as of the time of this
> writing?) is all
> >> thats
> >> >> > required
> >> >> > > for that. it is assumed that the other end
> interested in any
> >> such
> >> >> > piece of
> >> >> > > metadata knows the encoding, and byte[] provides for
> the most
> >> >> > flexibility.
> >> >> > > i believe this is the same logic behind core kafka
> being
> >> >> > byte[]/byte[] -
> >> >> > > Strings are more "usable" but bytes are flexible and
> so were
> >> chosen.
> >> >> > > Also - core kafka doesnt even do that good of a job
> on
> >> usability of
> >> >> > the
> >> >> > > payload (example - i have to specify the nop byte[]
> "decoders"
> >> >> > explicitly
> >> >> > > in conf), and again sacrificies usability for the
> sake of
> >> >> > performance (no
> >> >> > > convenient single-record processing as poll is a
> batch, lots of
> >> >> > obscure
> >> >> > > little config details exposing internals of the
> batching
> >> mechanism,
> >> >> > etc)
> >> >> > >
> >> >> > > this is also why i really dislike the idea of a
> "type system"
> >> for
> >> >> > header
> >> >> > > values, it further degrades the usability, adds
> complexity and
> >> will
> >> >> > > eventually get in people's way, also, it would be
> the 2nd/3rd
> >> >> > home-group
> >> >> > > serialization mechanism in core kafka (counting 2
> iterations
> >> of the
> >> >> > "type
> >> >> > > definition DSL")
> >> >> > >
> >> >> > > 2. this is an implementation detail, and not even a
> very "user
> >> >> > facing" one?
> >> >> > > to the best of my understanding the vote process is
> on proposed
> >> >> > > API/behaviour. also - since we're willing to go with
> strings
> >> just
> >> >> > serialize
> >> >> > > a 0-sized header blob and IIUC you dont need any
> optionals
> >> anymore.
> >> >> > >
> >> >> > > 3. yes, we can :-)
> >> >> > >
> >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
> >> >> > Michael.Pearce@ig.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > Hi Jay,
> >> >> > > >
> >> >> > > > 1) There was some initial debate on the value
> part, as youll
> >> note
> >> >> > String,
> >> >> > > > String headers were discounted early on. The
> reason for this
> >> is
> >> >> > > flexibility
> >> >> > > > and keeping in line with the flexibility of key,
> value of the
> >> >> > message
> >> >> > > > object itself. I don’t think it takes away from an
> ecosystem
> >> as
> >> >> > each
> >> >> > > plugin
> >> >> > > > will care for their own key, this way ints,
> booleans , exotic
> >> >> > custom
> >> >> > > binary
> >> >> > > > can all be catered for=.
> >> >> > > > a. If you really wanted to push for a typed value
> interface,
> >> I
> >> >> > wouldn’t
> >> >> > > > want just String values supported, but the the
> primatives
> >> plus
> >> >> > string and
> >> >> > > > also still keeping the ability to have a binary
> for custom
> >> >> > binaries that
> >> >> > > > some organisations may have.
> >> >> > > > i. I have written this slight alternative here,
> >> >> > > https://cwiki.apache.org/
> >> >> > > > confluence/display/KAFKA/KIP-
> 82+-+Add+Record+Headers+-+Typed
> >> >> > > > ii. Essentially the value bytes, has a leading
> byte overhead.
> >> >> > > > 1. This tells you what type the value is, before
> reading
> >> the rest
> >> >> > of the
> >> >> > > > bytes, allowing serialisation/deserialization to
> and from the
> >> >> > primitives,
> >> >> > > > string and byte[]. This is akin to some other
> messaging
> >> systems.
> >> >> > > > 2) We are making it optional, so that for those
> not wanting
> >> >> > headers have
> >> >> > > 0
> >> >> > > > bytes overhead (think of it as a feature flag), I
> don’t
> >> think this
> >> >> > is
> >> >> > > > complex, especially if comparing to changes
> proposed in
> >> other kips
> >> >> > like
> >> >> > > > kip-98.
> >> >> > > > a. If you really really don’t like this, we can
> drop it, but
> >> it
> >> >> > would
> >> >> > > mean
> >> >> > > > buying into 4 bytes extra overhead for users who
> do not want
> >> to use
> >> >> > > headers.
> >> >> > > > 3) In the summary yes, it is at a higher level,
> but I think
> >> this
> >> >> > is well
> >> >> > > > documented in the proposed changes section.
> >> >> > > > a. Added getHeaders method to Producer/Consumer
> record (that
> >> is it)
> >> >> > > > b. We’ve also detailed the new Headers class that
> this method
> >> >> > returns
> >> >> > > that
> >> >> > > > encapsulates the headers protocol and logic.
> >> >> > > >
> >> >> > > > Best,
> >> >> > > > Mike
> >> >> > > >
> >> >> > > > ==Original questions from the vote thread from
> Jay.==
> >> >> > > >
> >> >> > > > Couple of things I think we still need to work out:
> >> >> > > >
> >> >> > > > 1. I think we agree about the key, but I think
> we haven't
> >> >> > talked about
> >> >> > > > the value yet. I think if our goal is an open
> ecosystem
> >> of these
> >> >> > > header
> >> >> > > > spread across many plugins from many systems we
> should
> >> consider
> >> >> > making
> >> >> > > > this
> >> >> > > > a string as well so it can be printed, set via
> a UI, set
> >> in
> >> >> > config,
> >> >> > > etc.
> >> >> > > > Basically encouraging pluggable serialization
> formats
> >> here will
> >> >> > lead
> >> >> > > to
> >> >> > > > a
> >> >> > > > bit of a tower of babel.
> >> >> > > > 2. This proposal still includes a pretty big
> change to our
> >> >> > > serialization
> >> >> > > > and protocol definition layer. Essentially it is
> >> introducing an
> >> >> > > optional
> >> >> > > > type, where the format is data dependent. I
> think this is
> >> >> > actually a
> >> >> > > big
> >> >> > > > change though it doesn't seem like it. It means
> you can no
> >> >> > longer
> >> >> > > > specify
> >> >> > > > this type with our type definition DSL, and
> likewise it
> >> requires
> >> >> > > custom
> >> >> > > > handling in client libs. This isn't a huge
> thing, since
> >> the
> >> >> > Record
> >> >> > > > definition is custom anyway, but I think this
> kind of
> >> protocol
> >> >> > > > inconsistency is very non-desirable and ties
> you to
> >> hand-coding
> >> >> > > things.
> >> >> > > > I
> >> >> > > > think the type should instead by [Key Value] in
> our BNF,
> >> where
> >> >> > key and
> >> >> > > > value are both short strings as used elsewhere.
> This
> >> brings it
> >> >> > in line
> >> >> > > > with
> >> >> > > > the rest of the protocol.
> >> >> > > > 3. Could we get more specific about the exact
> Java API
> >> change to
> >> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
> >> >> > > >
> >> >> > > > -Jay
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> > The information contained in this email is strictly
> confidential and
> >> for
> >> >> > the use of the addressee only, unless otherwise indicated.
> If you
> >> are not
> >> >> > the intended recipient, please do not read, copy, use or
> disclose to
> >> others
> >> >> > this message or any attachment. Please also notify the
> sender by
> >> replying
> >> >> > to this email or by telephone (+44(020 7896 0011) and then
> delete
> >> the email
> >> >> > and any copies of it. Opinions, conclusion (etc) that do
> not relate
> >> to the
> >> >> > official business of this company shall be understood as
> neither
> >> given nor
> >> >> > endorsed by it. IG is a trading name of IG Markets Limited
> (a company
> >> >> > registered in England and Wales, company number 04008957)
> and IG
> >> Index
> >> >> > Limited (a company registered in England and Wales,
> company number
> >> >> > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >> Hill,
> >> >> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> >> and IG
> >> >> > Index Limited (register number 114059) are authorised and
> regulated
> >> by the
> >> >> > Financial Conduct Authority.
> >> >> >
> >>
> >>
> >>
> >> --
> >> Nacho - Ignacio Solis - isolis@igso.net
> >>
> >
> >
>
>
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
>
> Would you be proposing in KIP-98 to convert the other message int’s (key
> length, value length) also to varint to keep it uniform.
> Also I assume there will be a static or helper method made to write/read
> these in the client and server.
Yes, that is what we are proposing, so using varints for headers would be
consistent with the rest of the message. We have used static helper methods
in our prototype implementation.
The cost of parsing, we want to parse/interpret the headers lazily (this is
> a key point brought up earlier in discussions)
I'm a bit skeptical of this. Has anyone done the performance testing? I can
probably implement it and test it if no one else has. I was also under the
impression that there may be use cases down the road where the broker would
need to interpret headers. That wouldn't be off the table in the future if
it's represented as bytes, but it would be quite a bit more awkward, right?
By the way, one question I have been wondering about. My understanding is
that headers are primarily for use cases where a third-party components
wants to enrich messages without needing to understand or modify the schema
of the message key and value. For the applications which directly produce
and consume the messages and control the key/value schema directly, it
seems we would rather have them implement headers directly in their own
schema. Supposing for the sake of argument that it was possible, my
question is whether it be sufficient to expose the headers in the
interceptor API and not in the common API?
-Jason
On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <Mi...@ig.com>
wrote:
> On the point of varInts
>
> Would you be proposing in KIP-98 to convert the other message int’s (key
> length, value length) also to varint to keep it uniform.
> Also I assume there will be a static or helper method made to write/read
> these in the client and server.
>
> Cheers
> Mike
>
>
>
> On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
>
> On the point re: headers in the message protocol being a byte array
> and not a count of elements followed by the elements. Again this was
> discussed/argued previously.
>
> It was agreed on for a few reasons some of which you have obviously
> picked up on:
>
> Broker is able to pass it through opaquely
> The cost of parsing, we want to parse/interpret the headers lazily
> (this is a key point brought up earlier in discussions)
> Headers can be copied from consumer record to producer record (aka
> mirror makers etc) without parsing if no changes are being made or being
> looked at.
> Keeps the broker agnostic to the format
> You need an int32 either for the byte size of the headers, or for the
> count of elements, so overheads are the same, but with going with an opaque
> byte array has the above advantages.
>
> Cheers
> Mike
>
>
> On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> Sorry, should have noted that the performance testing was done
> using the
> producer performance tool shipped with Kafka.
>
> -Jason
>
> On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <
> jason@confluent.io> wrote:
>
> > Hey Nacho,
> >
> > I've compared performance of our KIP-98 implementation with and
> without
> > varints. For messages around 128 bytes, we see an increase in
> throughput of
> > about 30% using the default configuration settings. At 256
> bytes, the
> > increase is around 16%. Obviously the performance converge as
> messages get
> > larger, but it seems well worth the cost. Note that we are also
> seeing a
> > substantial performance increase against trunk primarily because
> of the
> > much more efficient packing that varints provide us. Anything
> adding to
> > message overhead, such as record headers, would only increase
> the relative
> > difference. (Of course take these numbers with a grain of salt
> since I have
> > only used the default settings with both the producer and broker
> on my
> > local machine. We intend to provide more extensive performance
> details as
> > part of the work for KIP-98.)
> >
> > The implementation we are using is from protobuf (
> > https://developers.google.com/protocol-buffers/docs/encoding),
> which is
> > also used in HBase. It is trivial to implement and as far as I
> know doesn't
> > suffer from the aliasing problem you are describing. I checked
> with Magnus
> > (the author of librdkafka) and he agreed that the savings seemed
> worth the
> > cost of implementation.
> >
> > -Jason
> >
> > On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <is...@igso.net>
> wrote:
> >
> >> -VarInts
> >>
> >> I'm one of the people (if not the most) opposed to VarInts.
> VarInts
> >> have a place, but this is not it. (We had a large discussion
> about
> >> them at the beginning of KIP-82 time)
> >>
> >> If anybody has real life performance numbers of VarInts
> improving
> >> things or significantly reducing resources I would like to know
> what
> >> that case may be. Yes, you can save some bytes here and there,
> but
> >> this is probably insignificant to the overall system behavior
> and
> >> storage requirements. -- I say this with respect to using
> VarInts in
> >> the protocol itself, not as part of the data.
> >>
> >> VarInts require you to parse the Int before using it and
> depending on
> >> the encoding they can suffer from aliasing (multiple
> representations
> >> for the same value).
> >>
> >> Why add complexity?
> >>
> >> Nacho
> >>
> >>
> >> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <
> cmccabe@apache.org>
> >> wrote:
> >> > +1 for varints here-- it would save quite a bit of space.
> They are
> >> > pretty quick to implement as well.
> >> >
> >> > I think it makes sense for values to be byte arrays. Users
> might want
> >> > to attach arbitrary payloads; they shouldn't be forced to
> serialize
> >> > everything to Java strings.
> >> >
> >> > best,
> >> > Colin
> >> >
> >> >
> >> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> >> >> Hey Michael,
> >> >>
> >> >> Hmm, I guess the point of representing it as bytes is to
> allow the
> >> broker
> >> >> to pass it through opaquely? Is the cost of parsing them a
> concern, or
> >> >> are
> >> >> we simply trying to ensure that the broker stays agnostic to
> the
> >> format?
> >> >>
> >> >> On varints, I think adding support for them makes less sense
> for an
> >> >> isolated use case, but as part of a more holistic change
> (such as what
> >> we
> >> >> have proposed in KIP-98), I think they are justifiable. If
> we add them,
> >> >> then the need to use attributes becomes quite a bit weaker,
> right? The
> >> >> other thing I find slightly odd is the fact that null
> headers has no
> >> >> actual
> >> >> semantic meaning for the message (unlike null keys and
> values). It is
> >> >> just
> >> >> a space optimization. It seems a bit better to always use
> size 0 to
> >> >> indicate having no headers.
> >> >>
> >> >> Overall, the main point is ensuring that the message schema
> remains
> >> >> consistent, either within the larger protocol, or at a
> minimum within
> >> the
> >> >> message itself.
> >> >>
> >> >> -Jason
> >> >>
> >> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <
> Michael.Pearce@ig.com
> >> >
> >> >> wrote:
> >> >>
> >> >> > Hi Jason,
> >> >> >
> >> >> > On point 1) in the message protocol the headers are simply
> a byte
> >> array,
> >> >> > as like the key or value, this is to clearly demarcate the
> header in
> >> the
> >> >> > core message. Then the header byte array in the core
> message is an
> >> array of
> >> >> > key, value pairs. This is what it is denoting.
> >> >> >
> >> >> > Then this would be I guess in the given notation:
> >> >> >
> >> >> > Headers => [KeyLength, Key, ValueLength, Value]
> >> >> > KeyLength => int32 <-----------------NEW size of the
> byte[] of
> >> the
> >> >> > serialised key value
> >> >> > Key => bytes <---------------------- NEW serialised
> string (UTF8)
> >> >> > bytes of the header key
> >> >> > ValueLength => int32 <-------------- NEW size of the
> byte[] of
> >> the
> >> >> > serialised header value
> >> >> > Value => bytes <-------------------- NEW serialised
> form of the
> >> header
> >> >> > value
> >> >> >
> >> >> > The key length and value length is matching the way the
> protocol is
> >> >> > defined in the core message currently.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On point 2)
> >> >> > Var sized ints, this was discussed much earlier on, in
> fact I had
> >> >> > suggested it myself (with Hadoop references), the
> complexity of this
> >> >> > compared to having a simpler protocol was argued and
> agreed it
> >> wasn’t worth
> >> >> > the complexity as all other clients in other languages
> would need to
> >> ensure
> >> >> > theyre using the right var size algorithm, as there is a
> few.
> >> >> >
> >> >> > On point 3)
> >> >> > We did the attributes, optional approach as originally
> there was
> >> marked
> >> >> > concern that headers would cause a message size overhead
> for others,
> >> who
> >> >> > don’t want them. As such this is the clean solution to
> achieve that.
> >> If
> >> >> > that no longer holds, and we don’t care that we add 4bytes
> overhead,
> >> then
> >> >> > im happy to remove.
> >> >> >
> >> >> > I’m personally in favour of keeping the message as small
> as possible
> >> so
> >> >> > people don’t get shocks in perf and throughputs dues to
> message size,
> >> >> > unless they actively use the feature, as such I do prefer
> the
> >> attribute bit
> >> >> > wise feature flag approach myself.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On 16/02/2017, 05:40, "Jason Gustafson" <
> jason@confluent.io> wrote:
> >> >> >
> >> >> > We have proposed a few significant changes to the
> message format
> >> in
> >> >> > KIP-98
> >> >> > which now seems likely to pass (perhaps with some
> iterations on
> >> >> > implementation details). It would be good to try and
> coordinate
> >> the
> >> >> > changes
> >> >> > in both of the proposals to make sure they are
> consistent and
> >> >> > compatible.
> >> >> >
> >> >> > I think using the attributes to indicate null headers
> is a
> >> reasonable
> >> >> > approach. We have proposed to do the same thing for
> the message
> >> key and
> >> >> > value. That said, I sympathize with Jay's argument.
> Having
> >> multiple
> >> >> > ways to
> >> >> > specify a null value increases the overall complexity
> of the
> >> protocol.
> >> >> > You
> >> >> > can see this just from the fact that you need the
> extra verbiage
> >> in the
> >> >> > protocol specification in this KIP and in KIP-98 to
> describe the
> >> >> > dependence
> >> >> > between the fields and the attributes. It seems like a
> slippery
> >> slope
> >> >> > if
> >> >> > you start allowing different request types to
> implement the
> >> protocol
> >> >> > specification differently.
> >> >> >
> >> >> > You can also argue that the messages already are and
> are likely
> >> to
> >> >> > remain a
> >> >> > special case. For example, there is currently no
> generality in
> >> how
> >> >> > compressed message sets are represented that would be
> applicable
> >> for
> >> >> > other
> >> >> > request types. Some might see this divergence as an
> unfortunate
> >> >> > protocol
> >> >> > deficiency which should be fixed; others might see it
> as sort of
> >> the
> >> >> > inevitability of needing to optimize where it counts
> most. I'm
> >> probably
> >> >> > somewhere in between, but I think we probably all
> share the
> >> intuition
> >> >> > that
> >> >> > the protocol should be kept as consistent as possible.
> With that
> >> in
> >> >> > mind,
> >> >> > here are a few comments:
> >> >> >
> >> >> > 1. One thing I found a little odd when reading the
> current
> >> proposal is
> >> >> > that
> >> >> > the headers are both represented as an array of bytes
> and as an
> >> array
> >> >> > of
> >> >> > key/value pairs. I'd probably suggest something like
> this:
> >> >> >
> >> >> > Headers => [HeaderKey HeaderValue]
> >> >> > HeaderKey => String
> >> >> > HeaderValue => Bytes
> >> >> >
> >> >> > An array in the Kafka protocol is represented as a
> 4-byte integer
> >> >> > indicating the number of elements in the array
> followed by the
> >> >> > serialization of the elements. Unless I'm
> misunderstanding, what
> >> you
> >> >> > have
> >> >> > instead is the total size of the headers in bytes
> followed by the
> >> >> > elements.
> >> >> > I'm not sure I see any reason for this inconsistency.
> >> >> >
> >> >> > 2. In KIP-98, we've introduced variable-length integer
> fields.
> >> >> > Effectively,
> >> >> > we've enriched (or "complicated" as Jay might say ;)
> the protocol
> >> >> > specification to include the following types: VarInt,
> VarLong,
> >> >> > UnsignedVarInt and UnsignedVarLong.
> >> >> >
> >> >> > Along with these primitives, we could introduce the
> following
> >> types:
> >> >> >
> >> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> >> >> > NumberOfItems => UnsignedVarInt
> >> >> >
> >> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1
> Item2 .. ItemN
> >> >> > NumberOfItemsOrNull => VarInt (-1 means null)
> >> >> >
> >> >> > And similarly for the `String` and `Bytes` types.
> These types
> >> can save
> >> >> > a
> >> >> > considerable amount of space in this proposal because
> they can
> >> be used
> >> >> > for
> >> >> > both the number of headers included in the message and
> the
> >> lengths of
> >> >> > the
> >> >> > header keys and values. We could do this instead:
> >> >> >
> >> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
> >> >> > HeaderKey => VarSizeString
> >> >> > HeaderValue => VarSizeBytes
> >> >> >
> >> >> > Combining the savings from the use of variable length
> fields, the
> >> >> > benefit
> >> >> > of using the attributes to represent null seems pretty
> small.
> >> >> >
> >> >> > 3. Whichever way we go (whether we use the attributes
> or not), we
> >> >> > should at
> >> >> > least be consistent between this KIP and KIP-98. It
> would be very
> >> >> > strange
> >> >> > to have two ways to represent null values in the same
> schema.
> >> Either
> >> >> > way is
> >> >> > OK with me. I think some message-level optimizations
> are
> >> justifiable,
> >> >> > but
> >> >> > the savings here seem minimal (a few bytes per
> message), so
> >> maybe it's
> >> >> > not
> >> >> > worth the cost of letting the message diverge even
> further from
> >> the
> >> >> > rest of
> >> >> > the protocol.
> >> >> >
> >> >> > -Jason
> >> >> >
> >> >> >
> >> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> >> radai.rosenblatt@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > I've trimmed the inline contents as this mail is
> getting too
> >> big for
> >> >> > the
> >> >> > > apache mailing list software to deliver :-(
> >> >> > >
> >> >> > > 1. the important thing for interoperability is for
> different
> >> >> > "interested
> >> >> > > parties" (plugins, infra layers/wrappers, user-code)
> to be
> >> able to
> >> >> > stick
> >> >> > > pieces of metadata onto msgs without getting in each
> other's
> >> way. a
> >> >> > common
> >> >> > > key scheme (Strings, as of the time of this
> writing?) is all
> >> thats
> >> >> > required
> >> >> > > for that. it is assumed that the other end
> interested in any
> >> such
> >> >> > piece of
> >> >> > > metadata knows the encoding, and byte[] provides for
> the most
> >> >> > flexibility.
> >> >> > > i believe this is the same logic behind core kafka
> being
> >> >> > byte[]/byte[] -
> >> >> > > Strings are more "usable" but bytes are flexible and
> so were
> >> chosen.
> >> >> > > Also - core kafka doesnt even do that good of a job
> on
> >> usability of
> >> >> > the
> >> >> > > payload (example - i have to specify the nop byte[]
> "decoders"
> >> >> > explicitly
> >> >> > > in conf), and again sacrificies usability for the
> sake of
> >> >> > performance (no
> >> >> > > convenient single-record processing as poll is a
> batch, lots of
> >> >> > obscure
> >> >> > > little config details exposing internals of the
> batching
> >> mechanism,
> >> >> > etc)
> >> >> > >
> >> >> > > this is also why i really dislike the idea of a
> "type system"
> >> for
> >> >> > header
> >> >> > > values, it further degrades the usability, adds
> complexity and
> >> will
> >> >> > > eventually get in people's way, also, it would be
> the 2nd/3rd
> >> >> > home-group
> >> >> > > serialization mechanism in core kafka (counting 2
> iterations
> >> of the
> >> >> > "type
> >> >> > > definition DSL")
> >> >> > >
> >> >> > > 2. this is an implementation detail, and not even a
> very "user
> >> >> > facing" one?
> >> >> > > to the best of my understanding the vote process is
> on proposed
> >> >> > > API/behaviour. also - since we're willing to go with
> strings
> >> just
> >> >> > serialize
> >> >> > > a 0-sized header blob and IIUC you dont need any
> optionals
> >> anymore.
> >> >> > >
> >> >> > > 3. yes, we can :-)
> >> >> > >
> >> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
> >> >> > Michael.Pearce@ig.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > > > Hi Jay,
> >> >> > > >
> >> >> > > > 1) There was some initial debate on the value
> part, as youll
> >> note
> >> >> > String,
> >> >> > > > String headers were discounted early on. The
> reason for this
> >> is
> >> >> > > flexibility
> >> >> > > > and keeping in line with the flexibility of key,
> value of the
> >> >> > message
> >> >> > > > object itself. I don’t think it takes away from an
> ecosystem
> >> as
> >> >> > each
> >> >> > > plugin
> >> >> > > > will care for their own key, this way ints,
> booleans , exotic
> >> >> > custom
> >> >> > > binary
> >> >> > > > can all be catered for=.
> >> >> > > > a. If you really wanted to push for a typed value
> interface,
> >> I
> >> >> > wouldn’t
> >> >> > > > want just String values supported, but the the
> primatives
> >> plus
> >> >> > string and
> >> >> > > > also still keeping the ability to have a binary
> for custom
> >> >> > binaries that
> >> >> > > > some organisations may have.
> >> >> > > > i. I have written this slight alternative here,
> >> >> > > https://cwiki.apache.org/
> >> >> > > > confluence/display/KAFKA/KIP-
> 82+-+Add+Record+Headers+-+Typed
> >> >> > > > ii. Essentially the value bytes, has a leading
> byte overhead.
> >> >> > > > 1. This tells you what type the value is, before
> reading
> >> the rest
> >> >> > of the
> >> >> > > > bytes, allowing serialisation/deserialization to
> and from the
> >> >> > primitives,
> >> >> > > > string and byte[]. This is akin to some other
> messaging
> >> systems.
> >> >> > > > 2) We are making it optional, so that for those
> not wanting
> >> >> > headers have
> >> >> > > 0
> >> >> > > > bytes overhead (think of it as a feature flag), I
> don’t
> >> think this
> >> >> > is
> >> >> > > > complex, especially if comparing to changes
> proposed in
> >> other kips
> >> >> > like
> >> >> > > > kip-98.
> >> >> > > > a. If you really really don’t like this, we can
> drop it, but
> >> it
> >> >> > would
> >> >> > > mean
> >> >> > > > buying into 4 bytes extra overhead for users who
> do not want
> >> to use
> >> >> > > headers.
> >> >> > > > 3) In the summary yes, it is at a higher level,
> but I think
> >> this
> >> >> > is well
> >> >> > > > documented in the proposed changes section.
> >> >> > > > a. Added getHeaders method to Producer/Consumer
> record (that
> >> is it)
> >> >> > > > b. We’ve also detailed the new Headers class that
> this method
> >> >> > returns
> >> >> > > that
> >> >> > > > encapsulates the headers protocol and logic.
> >> >> > > >
> >> >> > > > Best,
> >> >> > > > Mike
> >> >> > > >
> >> >> > > > ==Original questions from the vote thread from
> Jay.==
> >> >> > > >
> >> >> > > > Couple of things I think we still need to work out:
> >> >> > > >
> >> >> > > > 1. I think we agree about the key, but I think
> we haven't
> >> >> > talked about
> >> >> > > > the value yet. I think if our goal is an open
> ecosystem
> >> of these
> >> >> > > header
> >> >> > > > spread across many plugins from many systems we
> should
> >> consider
> >> >> > making
> >> >> > > > this
> >> >> > > > a string as well so it can be printed, set via
> a UI, set
> >> in
> >> >> > config,
> >> >> > > etc.
> >> >> > > > Basically encouraging pluggable serialization
> formats
> >> here will
> >> >> > lead
> >> >> > > to
> >> >> > > > a
> >> >> > > > bit of a tower of babel.
> >> >> > > > 2. This proposal still includes a pretty big
> change to our
> >> >> > > serialization
> >> >> > > > and protocol definition layer. Essentially it is
> >> introducing an
> >> >> > > optional
> >> >> > > > type, where the format is data dependent. I
> think this is
> >> >> > actually a
> >> >> > > big
> >> >> > > > change though it doesn't seem like it. It means
> you can no
> >> >> > longer
> >> >> > > > specify
> >> >> > > > this type with our type definition DSL, and
> likewise it
> >> requires
> >> >> > > custom
> >> >> > > > handling in client libs. This isn't a huge
> thing, since
> >> the
> >> >> > Record
> >> >> > > > definition is custom anyway, but I think this
> kind of
> >> protocol
> >> >> > > > inconsistency is very non-desirable and ties
> you to
> >> hand-coding
> >> >> > > things.
> >> >> > > > I
> >> >> > > > think the type should instead by [Key Value] in
> our BNF,
> >> where
> >> >> > key and
> >> >> > > > value are both short strings as used elsewhere.
> This
> >> brings it
> >> >> > in line
> >> >> > > > with
> >> >> > > > the rest of the protocol.
> >> >> > > > 3. Could we get more specific about the exact
> Java API
> >> change to
> >> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
> >> >> > > >
> >> >> > > > -Jay
> >> >> > > >
> >> >> > >
> >> >> >
> >> >> >
> >> >> > The information contained in this email is strictly
> confidential and
> >> for
> >> >> > the use of the addressee only, unless otherwise indicated.
> If you
> >> are not
> >> >> > the intended recipient, please do not read, copy, use or
> disclose to
> >> others
> >> >> > this message or any attachment. Please also notify the
> sender by
> >> replying
> >> >> > to this email or by telephone (+44(020 7896 0011) and then
> delete
> >> the email
> >> >> > and any copies of it. Opinions, conclusion (etc) that do
> not relate
> >> to the
> >> >> > official business of this company shall be understood as
> neither
> >> given nor
> >> >> > endorsed by it. IG is a trading name of IG Markets Limited
> (a company
> >> >> > registered in England and Wales, company number 04008957)
> and IG
> >> Index
> >> >> > Limited (a company registered in England and Wales,
> company number
> >> >> > 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >> Hill,
> >> >> > London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> >> and IG
> >> >> > Index Limited (register number 114059) are authorised and
> regulated
> >> by the
> >> >> > Financial Conduct Authority.
> >> >> >
> >>
> >>
> >>
> >> --
> >> Nacho - Ignacio Solis - isolis@igso.net
> >>
> >
> >
>
>
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
On the point of varInts
Would you be proposing in KIP-98 to convert the other message int’s (key length, value length) also to varint to keep it uniform.
Also I assume there will be a static or helper method made to write/read these in the client and server.
Cheers
Mike
On 17/02/2017, 11:22, "Michael Pearce" <Mi...@ig.com> wrote:
On the point re: headers in the message protocol being a byte array and not a count of elements followed by the elements. Again this was discussed/argued previously.
It was agreed on for a few reasons some of which you have obviously picked up on:
Broker is able to pass it through opaquely
The cost of parsing, we want to parse/interpret the headers lazily (this is a key point brought up earlier in discussions)
Headers can be copied from consumer record to producer record (aka mirror makers etc) without parsing if no changes are being made or being looked at.
Keeps the broker agnostic to the format
You need an int32 either for the byte size of the headers, or for the count of elements, so overheads are the same, but with going with an opaque byte array has the above advantages.
Cheers
Mike
On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io> wrote:
Sorry, should have noted that the performance testing was done using the
producer performance tool shipped with Kafka.
-Jason
On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Nacho,
>
> I've compared performance of our KIP-98 implementation with and without
> varints. For messages around 128 bytes, we see an increase in throughput of
> about 30% using the default configuration settings. At 256 bytes, the
> increase is around 16%. Obviously the performance converge as messages get
> larger, but it seems well worth the cost. Note that we are also seeing a
> substantial performance increase against trunk primarily because of the
> much more efficient packing that varints provide us. Anything adding to
> message overhead, such as record headers, would only increase the relative
> difference. (Of course take these numbers with a grain of salt since I have
> only used the default settings with both the producer and broker on my
> local machine. We intend to provide more extensive performance details as
> part of the work for KIP-98.)
>
> The implementation we are using is from protobuf (
> https://developers.google.com/protocol-buffers/docs/encoding), which is
> also used in HBase. It is trivial to implement and as far as I know doesn't
> suffer from the aliasing problem you are describing. I checked with Magnus
> (the author of librdkafka) and he agreed that the savings seemed worth the
> cost of implementation.
>
> -Jason
>
> On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <is...@igso.net> wrote:
>
>> -VarInts
>>
>> I'm one of the people (if not the most) opposed to VarInts. VarInts
>> have a place, but this is not it. (We had a large discussion about
>> them at the beginning of KIP-82 time)
>>
>> If anybody has real life performance numbers of VarInts improving
>> things or significantly reducing resources I would like to know what
>> that case may be. Yes, you can save some bytes here and there, but
>> this is probably insignificant to the overall system behavior and
>> storage requirements. -- I say this with respect to using VarInts in
>> the protocol itself, not as part of the data.
>>
>> VarInts require you to parse the Int before using it and depending on
>> the encoding they can suffer from aliasing (multiple representations
>> for the same value).
>>
>> Why add complexity?
>>
>> Nacho
>>
>>
>> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <cm...@apache.org>
>> wrote:
>> > +1 for varints here-- it would save quite a bit of space. They are
>> > pretty quick to implement as well.
>> >
>> > I think it makes sense for values to be byte arrays. Users might want
>> > to attach arbitrary payloads; they shouldn't be forced to serialize
>> > everything to Java strings.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
>> >> Hey Michael,
>> >>
>> >> Hmm, I guess the point of representing it as bytes is to allow the
>> broker
>> >> to pass it through opaquely? Is the cost of parsing them a concern, or
>> >> are
>> >> we simply trying to ensure that the broker stays agnostic to the
>> format?
>> >>
>> >> On varints, I think adding support for them makes less sense for an
>> >> isolated use case, but as part of a more holistic change (such as what
>> we
>> >> have proposed in KIP-98), I think they are justifiable. If we add them,
>> >> then the need to use attributes becomes quite a bit weaker, right? The
>> >> other thing I find slightly odd is the fact that null headers has no
>> >> actual
>> >> semantic meaning for the message (unlike null keys and values). It is
>> >> just
>> >> a space optimization. It seems a bit better to always use size 0 to
>> >> indicate having no headers.
>> >>
>> >> Overall, the main point is ensuring that the message schema remains
>> >> consistent, either within the larger protocol, or at a minimum within
>> the
>> >> message itself.
>> >>
>> >> -Jason
>> >>
>> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <Michael.Pearce@ig.com
>> >
>> >> wrote:
>> >>
>> >> > Hi Jason,
>> >> >
>> >> > On point 1) in the message protocol the headers are simply a byte
>> array,
>> >> > as like the key or value, this is to clearly demarcate the header in
>> the
>> >> > core message. Then the header byte array in the core message is an
>> array of
>> >> > key, value pairs. This is what it is denoting.
>> >> >
>> >> > Then this would be I guess in the given notation:
>> >> >
>> >> > Headers => [KeyLength, Key, ValueLength, Value]
>> >> > KeyLength => int32 <-----------------NEW size of the byte[] of
>> the
>> >> > serialised key value
>> >> > Key => bytes <---------------------- NEW serialised string (UTF8)
>> >> > bytes of the header key
>> >> > ValueLength => int32 <-------------- NEW size of the byte[] of
>> the
>> >> > serialised header value
>> >> > Value => bytes <-------------------- NEW serialised form of the
>> header
>> >> > value
>> >> >
>> >> > The key length and value length is matching the way the protocol is
>> >> > defined in the core message currently.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On point 2)
>> >> > Var sized ints, this was discussed much earlier on, in fact I had
>> >> > suggested it myself (with Hadoop references), the complexity of this
>> >> > compared to having a simpler protocol was argued and agreed it
>> wasn’t worth
>> >> > the complexity as all other clients in other languages would need to
>> ensure
>> >> > theyre using the right var size algorithm, as there is a few.
>> >> >
>> >> > On point 3)
>> >> > We did the attributes, optional approach as originally there was
>> marked
>> >> > concern that headers would cause a message size overhead for others,
>> who
>> >> > don’t want them. As such this is the clean solution to achieve that.
>> If
>> >> > that no longer holds, and we don’t care that we add 4bytes overhead,
>> then
>> >> > im happy to remove.
>> >> >
>> >> > I’m personally in favour of keeping the message as small as possible
>> so
>> >> > people don’t get shocks in perf and throughputs dues to message size,
>> >> > unless they actively use the feature, as such I do prefer the
>> attribute bit
>> >> > wise feature flag approach myself.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
>> >> >
>> >> > We have proposed a few significant changes to the message format
>> in
>> >> > KIP-98
>> >> > which now seems likely to pass (perhaps with some iterations on
>> >> > implementation details). It would be good to try and coordinate
>> the
>> >> > changes
>> >> > in both of the proposals to make sure they are consistent and
>> >> > compatible.
>> >> >
>> >> > I think using the attributes to indicate null headers is a
>> reasonable
>> >> > approach. We have proposed to do the same thing for the message
>> key and
>> >> > value. That said, I sympathize with Jay's argument. Having
>> multiple
>> >> > ways to
>> >> > specify a null value increases the overall complexity of the
>> protocol.
>> >> > You
>> >> > can see this just from the fact that you need the extra verbiage
>> in the
>> >> > protocol specification in this KIP and in KIP-98 to describe the
>> >> > dependence
>> >> > between the fields and the attributes. It seems like a slippery
>> slope
>> >> > if
>> >> > you start allowing different request types to implement the
>> protocol
>> >> > specification differently.
>> >> >
>> >> > You can also argue that the messages already are and are likely
>> to
>> >> > remain a
>> >> > special case. For example, there is currently no generality in
>> how
>> >> > compressed message sets are represented that would be applicable
>> for
>> >> > other
>> >> > request types. Some might see this divergence as an unfortunate
>> >> > protocol
>> >> > deficiency which should be fixed; others might see it as sort of
>> the
>> >> > inevitability of needing to optimize where it counts most. I'm
>> probably
>> >> > somewhere in between, but I think we probably all share the
>> intuition
>> >> > that
>> >> > the protocol should be kept as consistent as possible. With that
>> in
>> >> > mind,
>> >> > here are a few comments:
>> >> >
>> >> > 1. One thing I found a little odd when reading the current
>> proposal is
>> >> > that
>> >> > the headers are both represented as an array of bytes and as an
>> array
>> >> > of
>> >> > key/value pairs. I'd probably suggest something like this:
>> >> >
>> >> > Headers => [HeaderKey HeaderValue]
>> >> > HeaderKey => String
>> >> > HeaderValue => Bytes
>> >> >
>> >> > An array in the Kafka protocol is represented as a 4-byte integer
>> >> > indicating the number of elements in the array followed by the
>> >> > serialization of the elements. Unless I'm misunderstanding, what
>> you
>> >> > have
>> >> > instead is the total size of the headers in bytes followed by the
>> >> > elements.
>> >> > I'm not sure I see any reason for this inconsistency.
>> >> >
>> >> > 2. In KIP-98, we've introduced variable-length integer fields.
>> >> > Effectively,
>> >> > we've enriched (or "complicated" as Jay might say ;) the protocol
>> >> > specification to include the following types: VarInt, VarLong,
>> >> > UnsignedVarInt and UnsignedVarLong.
>> >> >
>> >> > Along with these primitives, we could introduce the following
>> types:
>> >> >
>> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
>> >> > NumberOfItems => UnsignedVarInt
>> >> >
>> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
>> >> > NumberOfItemsOrNull => VarInt (-1 means null)
>> >> >
>> >> > And similarly for the `String` and `Bytes` types. These types
>> can save
>> >> > a
>> >> > considerable amount of space in this proposal because they can
>> be used
>> >> > for
>> >> > both the number of headers included in the message and the
>> lengths of
>> >> > the
>> >> > header keys and values. We could do this instead:
>> >> >
>> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
>> >> > HeaderKey => VarSizeString
>> >> > HeaderValue => VarSizeBytes
>> >> >
>> >> > Combining the savings from the use of variable length fields, the
>> >> > benefit
>> >> > of using the attributes to represent null seems pretty small.
>> >> >
>> >> > 3. Whichever way we go (whether we use the attributes or not), we
>> >> > should at
>> >> > least be consistent between this KIP and KIP-98. It would be very
>> >> > strange
>> >> > to have two ways to represent null values in the same schema.
>> Either
>> >> > way is
>> >> > OK with me. I think some message-level optimizations are
>> justifiable,
>> >> > but
>> >> > the savings here seem minimal (a few bytes per message), so
>> maybe it's
>> >> > not
>> >> > worth the cost of letting the message diverge even further from
>> the
>> >> > rest of
>> >> > the protocol.
>> >> >
>> >> > -Jason
>> >> >
>> >> >
>> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
>> radai.rosenblatt@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > I've trimmed the inline contents as this mail is getting too
>> big for
>> >> > the
>> >> > > apache mailing list software to deliver :-(
>> >> > >
>> >> > > 1. the important thing for interoperability is for different
>> >> > "interested
>> >> > > parties" (plugins, infra layers/wrappers, user-code) to be
>> able to
>> >> > stick
>> >> > > pieces of metadata onto msgs without getting in each other's
>> way. a
>> >> > common
>> >> > > key scheme (Strings, as of the time of this writing?) is all
>> thats
>> >> > required
>> >> > > for that. it is assumed that the other end interested in any
>> such
>> >> > piece of
>> >> > > metadata knows the encoding, and byte[] provides for the most
>> >> > flexibility.
>> >> > > i believe this is the same logic behind core kafka being
>> >> > byte[]/byte[] -
>> >> > > Strings are more "usable" but bytes are flexible and so were
>> chosen.
>> >> > > Also - core kafka doesnt even do that good of a job on
>> usability of
>> >> > the
>> >> > > payload (example - i have to specify the nop byte[] "decoders"
>> >> > explicitly
>> >> > > in conf), and again sacrificies usability for the sake of
>> >> > performance (no
>> >> > > convenient single-record processing as poll is a batch, lots of
>> >> > obscure
>> >> > > little config details exposing internals of the batching
>> mechanism,
>> >> > etc)
>> >> > >
>> >> > > this is also why i really dislike the idea of a "type system"
>> for
>> >> > header
>> >> > > values, it further degrades the usability, adds complexity and
>> will
>> >> > > eventually get in people's way, also, it would be the 2nd/3rd
>> >> > home-group
>> >> > > serialization mechanism in core kafka (counting 2 iterations
>> of the
>> >> > "type
>> >> > > definition DSL")
>> >> > >
>> >> > > 2. this is an implementation detail, and not even a very "user
>> >> > facing" one?
>> >> > > to the best of my understanding the vote process is on proposed
>> >> > > API/behaviour. also - since we're willing to go with strings
>> just
>> >> > serialize
>> >> > > a 0-sized header blob and IIUC you dont need any optionals
>> anymore.
>> >> > >
>> >> > > 3. yes, we can :-)
>> >> > >
>> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
>> >> > Michael.Pearce@ig.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Jay,
>> >> > > >
>> >> > > > 1) There was some initial debate on the value part, as youll
>> note
>> >> > String,
>> >> > > > String headers were discounted early on. The reason for this
>> is
>> >> > > flexibility
>> >> > > > and keeping in line with the flexibility of key, value of the
>> >> > message
>> >> > > > object itself. I don’t think it takes away from an ecosystem
>> as
>> >> > each
>> >> > > plugin
>> >> > > > will care for their own key, this way ints, booleans , exotic
>> >> > custom
>> >> > > binary
>> >> > > > can all be catered for=.
>> >> > > > a. If you really wanted to push for a typed value interface,
>> I
>> >> > wouldn’t
>> >> > > > want just String values supported, but the the primatives
>> plus
>> >> > string and
>> >> > > > also still keeping the ability to have a binary for custom
>> >> > binaries that
>> >> > > > some organisations may have.
>> >> > > > i. I have written this slight alternative here,
>> >> > > https://cwiki.apache.org/
>> >> > > > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
>> >> > > > ii. Essentially the value bytes, has a leading byte overhead.
>> >> > > > 1. This tells you what type the value is, before reading
>> the rest
>> >> > of the
>> >> > > > bytes, allowing serialisation/deserialization to and from the
>> >> > primitives,
>> >> > > > string and byte[]. This is akin to some other messaging
>> systems.
>> >> > > > 2) We are making it optional, so that for those not wanting
>> >> > headers have
>> >> > > 0
>> >> > > > bytes overhead (think of it as a feature flag), I don’t
>> think this
>> >> > is
>> >> > > > complex, especially if comparing to changes proposed in
>> other kips
>> >> > like
>> >> > > > kip-98.
>> >> > > > a. If you really really don’t like this, we can drop it, but
>> it
>> >> > would
>> >> > > mean
>> >> > > > buying into 4 bytes extra overhead for users who do not want
>> to use
>> >> > > headers.
>> >> > > > 3) In the summary yes, it is at a higher level, but I think
>> this
>> >> > is well
>> >> > > > documented in the proposed changes section.
>> >> > > > a. Added getHeaders method to Producer/Consumer record (that
>> is it)
>> >> > > > b. We’ve also detailed the new Headers class that this method
>> >> > returns
>> >> > > that
>> >> > > > encapsulates the headers protocol and logic.
>> >> > > >
>> >> > > > Best,
>> >> > > > Mike
>> >> > > >
>> >> > > > ==Original questions from the vote thread from Jay.==
>> >> > > >
>> >> > > > Couple of things I think we still need to work out:
>> >> > > >
>> >> > > > 1. I think we agree about the key, but I think we haven't
>> >> > talked about
>> >> > > > the value yet. I think if our goal is an open ecosystem
>> of these
>> >> > > header
>> >> > > > spread across many plugins from many systems we should
>> consider
>> >> > making
>> >> > > > this
>> >> > > > a string as well so it can be printed, set via a UI, set
>> in
>> >> > config,
>> >> > > etc.
>> >> > > > Basically encouraging pluggable serialization formats
>> here will
>> >> > lead
>> >> > > to
>> >> > > > a
>> >> > > > bit of a tower of babel.
>> >> > > > 2. This proposal still includes a pretty big change to our
>> >> > > serialization
>> >> > > > and protocol definition layer. Essentially it is
>> introducing an
>> >> > > optional
>> >> > > > type, where the format is data dependent. I think this is
>> >> > actually a
>> >> > > big
>> >> > > > change though it doesn't seem like it. It means you can no
>> >> > longer
>> >> > > > specify
>> >> > > > this type with our type definition DSL, and likewise it
>> requires
>> >> > > custom
>> >> > > > handling in client libs. This isn't a huge thing, since
>> the
>> >> > Record
>> >> > > > definition is custom anyway, but I think this kind of
>> protocol
>> >> > > > inconsistency is very non-desirable and ties you to
>> hand-coding
>> >> > > things.
>> >> > > > I
>> >> > > > think the type should instead by [Key Value] in our BNF,
>> where
>> >> > key and
>> >> > > > value are both short strings as used elsewhere. This
>> brings it
>> >> > in line
>> >> > > > with
>> >> > > > the rest of the protocol.
>> >> > > > 3. Could we get more specific about the exact Java API
>> change to
>> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
>> >> > > >
>> >> > > > -Jay
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> > The information contained in this email is strictly confidential and
>> for
>> >> > the use of the addressee only, unless otherwise indicated. If you
>> are not
>> >> > the intended recipient, please do not read, copy, use or disclose to
>> others
>> >> > this message or any attachment. Please also notify the sender by
>> replying
>> >> > to this email or by telephone (+44(020 7896 0011) and then delete
>> the email
>> >> > and any copies of it. Opinions, conclusion (etc) that do not relate
>> to the
>> >> > official business of this company shall be understood as neither
>> given nor
>> >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> >> > registered in England and Wales, company number 04008957) and IG
>> Index
>> >> > Limited (a company registered in England and Wales, company number
>> >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> >> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> and IG
>> >> > Index Limited (register number 114059) are authorised and regulated
>> by the
>> >> > Financial Conduct Authority.
>> >> >
>>
>>
>>
>> --
>> Nacho - Ignacio Solis - isolis@igso.net
>>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
On the point re: headers in the message protocol being a byte array and not a count of elements followed by the elements. Again this was discussed/argued previously.
It was agreed on for a few reasons some of which you have obviously picked up on:
Broker is able to pass it through opaquely
The cost of parsing, we want to parse/interpret the headers lazily (this is a key point brought up earlier in discussions)
Headers can be copied from consumer record to producer record (aka mirror makers etc) without parsing if no changes are being made or being looked at.
Keeps the broker agnostic to the format
You need an int32 either for the byte size of the headers, or for the count of elements, so overheads are the same, but with going with an opaque byte array has the above advantages.
Cheers
Mike
On 17/02/2017, 02:50, "Jason Gustafson" <ja...@confluent.io> wrote:
Sorry, should have noted that the performance testing was done using the
producer performance tool shipped with Kafka.
-Jason
On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Nacho,
>
> I've compared performance of our KIP-98 implementation with and without
> varints. For messages around 128 bytes, we see an increase in throughput of
> about 30% using the default configuration settings. At 256 bytes, the
> increase is around 16%. Obviously the performance converge as messages get
> larger, but it seems well worth the cost. Note that we are also seeing a
> substantial performance increase against trunk primarily because of the
> much more efficient packing that varints provide us. Anything adding to
> message overhead, such as record headers, would only increase the relative
> difference. (Of course take these numbers with a grain of salt since I have
> only used the default settings with both the producer and broker on my
> local machine. We intend to provide more extensive performance details as
> part of the work for KIP-98.)
>
> The implementation we are using is from protobuf (
> https://developers.google.com/protocol-buffers/docs/encoding), which is
> also used in HBase. It is trivial to implement and as far as I know doesn't
> suffer from the aliasing problem you are describing. I checked with Magnus
> (the author of librdkafka) and he agreed that the savings seemed worth the
> cost of implementation.
>
> -Jason
>
> On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <is...@igso.net> wrote:
>
>> -VarInts
>>
>> I'm one of the people (if not the most) opposed to VarInts. VarInts
>> have a place, but this is not it. (We had a large discussion about
>> them at the beginning of KIP-82 time)
>>
>> If anybody has real life performance numbers of VarInts improving
>> things or significantly reducing resources I would like to know what
>> that case may be. Yes, you can save some bytes here and there, but
>> this is probably insignificant to the overall system behavior and
>> storage requirements. -- I say this with respect to using VarInts in
>> the protocol itself, not as part of the data.
>>
>> VarInts require you to parse the Int before using it and depending on
>> the encoding they can suffer from aliasing (multiple representations
>> for the same value).
>>
>> Why add complexity?
>>
>> Nacho
>>
>>
>> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <cm...@apache.org>
>> wrote:
>> > +1 for varints here-- it would save quite a bit of space. They are
>> > pretty quick to implement as well.
>> >
>> > I think it makes sense for values to be byte arrays. Users might want
>> > to attach arbitrary payloads; they shouldn't be forced to serialize
>> > everything to Java strings.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
>> >> Hey Michael,
>> >>
>> >> Hmm, I guess the point of representing it as bytes is to allow the
>> broker
>> >> to pass it through opaquely? Is the cost of parsing them a concern, or
>> >> are
>> >> we simply trying to ensure that the broker stays agnostic to the
>> format?
>> >>
>> >> On varints, I think adding support for them makes less sense for an
>> >> isolated use case, but as part of a more holistic change (such as what
>> we
>> >> have proposed in KIP-98), I think they are justifiable. If we add them,
>> >> then the need to use attributes becomes quite a bit weaker, right? The
>> >> other thing I find slightly odd is the fact that null headers has no
>> >> actual
>> >> semantic meaning for the message (unlike null keys and values). It is
>> >> just
>> >> a space optimization. It seems a bit better to always use size 0 to
>> >> indicate having no headers.
>> >>
>> >> Overall, the main point is ensuring that the message schema remains
>> >> consistent, either within the larger protocol, or at a minimum within
>> the
>> >> message itself.
>> >>
>> >> -Jason
>> >>
>> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <Michael.Pearce@ig.com
>> >
>> >> wrote:
>> >>
>> >> > Hi Jason,
>> >> >
>> >> > On point 1) in the message protocol the headers are simply a byte
>> array,
>> >> > as like the key or value, this is to clearly demarcate the header in
>> the
>> >> > core message. Then the header byte array in the core message is an
>> array of
>> >> > key, value pairs. This is what it is denoting.
>> >> >
>> >> > Then this would be I guess in the given notation:
>> >> >
>> >> > Headers => [KeyLength, Key, ValueLength, Value]
>> >> > KeyLength => int32 <-----------------NEW size of the byte[] of
>> the
>> >> > serialised key value
>> >> > Key => bytes <---------------------- NEW serialised string (UTF8)
>> >> > bytes of the header key
>> >> > ValueLength => int32 <-------------- NEW size of the byte[] of
>> the
>> >> > serialised header value
>> >> > Value => bytes <-------------------- NEW serialised form of the
>> header
>> >> > value
>> >> >
>> >> > The key length and value length is matching the way the protocol is
>> >> > defined in the core message currently.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On point 2)
>> >> > Var sized ints, this was discussed much earlier on, in fact I had
>> >> > suggested it myself (with Hadoop references), the complexity of this
>> >> > compared to having a simpler protocol was argued and agreed it
>> wasn’t worth
>> >> > the complexity as all other clients in other languages would need to
>> ensure
>> >> > theyre using the right var size algorithm, as there is a few.
>> >> >
>> >> > On point 3)
>> >> > We did the attributes, optional approach as originally there was
>> marked
>> >> > concern that headers would cause a message size overhead for others,
>> who
>> >> > don’t want them. As such this is the clean solution to achieve that.
>> If
>> >> > that no longer holds, and we don’t care that we add 4bytes overhead,
>> then
>> >> > im happy to remove.
>> >> >
>> >> > I’m personally in favour of keeping the message as small as possible
>> so
>> >> > people don’t get shocks in perf and throughputs dues to message size,
>> >> > unless they actively use the feature, as such I do prefer the
>> attribute bit
>> >> > wise feature flag approach myself.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
>> >> >
>> >> > We have proposed a few significant changes to the message format
>> in
>> >> > KIP-98
>> >> > which now seems likely to pass (perhaps with some iterations on
>> >> > implementation details). It would be good to try and coordinate
>> the
>> >> > changes
>> >> > in both of the proposals to make sure they are consistent and
>> >> > compatible.
>> >> >
>> >> > I think using the attributes to indicate null headers is a
>> reasonable
>> >> > approach. We have proposed to do the same thing for the message
>> key and
>> >> > value. That said, I sympathize with Jay's argument. Having
>> multiple
>> >> > ways to
>> >> > specify a null value increases the overall complexity of the
>> protocol.
>> >> > You
>> >> > can see this just from the fact that you need the extra verbiage
>> in the
>> >> > protocol specification in this KIP and in KIP-98 to describe the
>> >> > dependence
>> >> > between the fields and the attributes. It seems like a slippery
>> slope
>> >> > if
>> >> > you start allowing different request types to implement the
>> protocol
>> >> > specification differently.
>> >> >
>> >> > You can also argue that the messages already are and are likely
>> to
>> >> > remain a
>> >> > special case. For example, there is currently no generality in
>> how
>> >> > compressed message sets are represented that would be applicable
>> for
>> >> > other
>> >> > request types. Some might see this divergence as an unfortunate
>> >> > protocol
>> >> > deficiency which should be fixed; others might see it as sort of
>> the
>> >> > inevitability of needing to optimize where it counts most. I'm
>> probably
>> >> > somewhere in between, but I think we probably all share the
>> intuition
>> >> > that
>> >> > the protocol should be kept as consistent as possible. With that
>> in
>> >> > mind,
>> >> > here are a few comments:
>> >> >
>> >> > 1. One thing I found a little odd when reading the current
>> proposal is
>> >> > that
>> >> > the headers are both represented as an array of bytes and as an
>> array
>> >> > of
>> >> > key/value pairs. I'd probably suggest something like this:
>> >> >
>> >> > Headers => [HeaderKey HeaderValue]
>> >> > HeaderKey => String
>> >> > HeaderValue => Bytes
>> >> >
>> >> > An array in the Kafka protocol is represented as a 4-byte integer
>> >> > indicating the number of elements in the array followed by the
>> >> > serialization of the elements. Unless I'm misunderstanding, what
>> you
>> >> > have
>> >> > instead is the total size of the headers in bytes followed by the
>> >> > elements.
>> >> > I'm not sure I see any reason for this inconsistency.
>> >> >
>> >> > 2. In KIP-98, we've introduced variable-length integer fields.
>> >> > Effectively,
>> >> > we've enriched (or "complicated" as Jay might say ;) the protocol
>> >> > specification to include the following types: VarInt, VarLong,
>> >> > UnsignedVarInt and UnsignedVarLong.
>> >> >
>> >> > Along with these primitives, we could introduce the following
>> types:
>> >> >
>> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
>> >> > NumberOfItems => UnsignedVarInt
>> >> >
>> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
>> >> > NumberOfItemsOrNull => VarInt (-1 means null)
>> >> >
>> >> > And similarly for the `String` and `Bytes` types. These types
>> can save
>> >> > a
>> >> > considerable amount of space in this proposal because they can
>> be used
>> >> > for
>> >> > both the number of headers included in the message and the
>> lengths of
>> >> > the
>> >> > header keys and values. We could do this instead:
>> >> >
>> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
>> >> > HeaderKey => VarSizeString
>> >> > HeaderValue => VarSizeBytes
>> >> >
>> >> > Combining the savings from the use of variable length fields, the
>> >> > benefit
>> >> > of using the attributes to represent null seems pretty small.
>> >> >
>> >> > 3. Whichever way we go (whether we use the attributes or not), we
>> >> > should at
>> >> > least be consistent between this KIP and KIP-98. It would be very
>> >> > strange
>> >> > to have two ways to represent null values in the same schema.
>> Either
>> >> > way is
>> >> > OK with me. I think some message-level optimizations are
>> justifiable,
>> >> > but
>> >> > the savings here seem minimal (a few bytes per message), so
>> maybe it's
>> >> > not
>> >> > worth the cost of letting the message diverge even further from
>> the
>> >> > rest of
>> >> > the protocol.
>> >> >
>> >> > -Jason
>> >> >
>> >> >
>> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
>> radai.rosenblatt@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > I've trimmed the inline contents as this mail is getting too
>> big for
>> >> > the
>> >> > > apache mailing list software to deliver :-(
>> >> > >
>> >> > > 1. the important thing for interoperability is for different
>> >> > "interested
>> >> > > parties" (plugins, infra layers/wrappers, user-code) to be
>> able to
>> >> > stick
>> >> > > pieces of metadata onto msgs without getting in each other's
>> way. a
>> >> > common
>> >> > > key scheme (Strings, as of the time of this writing?) is all
>> thats
>> >> > required
>> >> > > for that. it is assumed that the other end interested in any
>> such
>> >> > piece of
>> >> > > metadata knows the encoding, and byte[] provides for the most
>> >> > flexibility.
>> >> > > i believe this is the same logic behind core kafka being
>> >> > byte[]/byte[] -
>> >> > > Strings are more "usable" but bytes are flexible and so were
>> chosen.
>> >> > > Also - core kafka doesnt even do that good of a job on
>> usability of
>> >> > the
>> >> > > payload (example - i have to specify the nop byte[] "decoders"
>> >> > explicitly
>> >> > > in conf), and again sacrificies usability for the sake of
>> >> > performance (no
>> >> > > convenient single-record processing as poll is a batch, lots of
>> >> > obscure
>> >> > > little config details exposing internals of the batching
>> mechanism,
>> >> > etc)
>> >> > >
>> >> > > this is also why i really dislike the idea of a "type system"
>> for
>> >> > header
>> >> > > values, it further degrades the usability, adds complexity and
>> will
>> >> > > eventually get in people's way, also, it would be the 2nd/3rd
>> >> > home-group
>> >> > > serialization mechanism in core kafka (counting 2 iterations
>> of the
>> >> > "type
>> >> > > definition DSL")
>> >> > >
>> >> > > 2. this is an implementation detail, and not even a very "user
>> >> > facing" one?
>> >> > > to the best of my understanding the vote process is on proposed
>> >> > > API/behaviour. also - since we're willing to go with strings
>> just
>> >> > serialize
>> >> > > a 0-sized header blob and IIUC you dont need any optionals
>> anymore.
>> >> > >
>> >> > > 3. yes, we can :-)
>> >> > >
>> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
>> >> > Michael.Pearce@ig.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Jay,
>> >> > > >
>> >> > > > 1) There was some initial debate on the value part, as youll
>> note
>> >> > String,
>> >> > > > String headers were discounted early on. The reason for this
>> is
>> >> > > flexibility
>> >> > > > and keeping in line with the flexibility of key, value of the
>> >> > message
>> >> > > > object itself. I don’t think it takes away from an ecosystem
>> as
>> >> > each
>> >> > > plugin
>> >> > > > will care for their own key, this way ints, booleans , exotic
>> >> > custom
>> >> > > binary
>> >> > > > can all be catered for=.
>> >> > > > a. If you really wanted to push for a typed value interface,
>> I
>> >> > wouldn’t
>> >> > > > want just String values supported, but the the primatives
>> plus
>> >> > string and
>> >> > > > also still keeping the ability to have a binary for custom
>> >> > binaries that
>> >> > > > some organisations may have.
>> >> > > > i. I have written this slight alternative here,
>> >> > > https://cwiki.apache.org/
>> >> > > > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
>> >> > > > ii. Essentially the value bytes, has a leading byte overhead.
>> >> > > > 1. This tells you what type the value is, before reading
>> the rest
>> >> > of the
>> >> > > > bytes, allowing serialisation/deserialization to and from the
>> >> > primitives,
>> >> > > > string and byte[]. This is akin to some other messaging
>> systems.
>> >> > > > 2) We are making it optional, so that for those not wanting
>> >> > headers have
>> >> > > 0
>> >> > > > bytes overhead (think of it as a feature flag), I don’t
>> think this
>> >> > is
>> >> > > > complex, especially if comparing to changes proposed in
>> other kips
>> >> > like
>> >> > > > kip-98.
>> >> > > > a. If you really really don’t like this, we can drop it, but
>> it
>> >> > would
>> >> > > mean
>> >> > > > buying into 4 bytes extra overhead for users who do not want
>> to use
>> >> > > headers.
>> >> > > > 3) In the summary yes, it is at a higher level, but I think
>> this
>> >> > is well
>> >> > > > documented in the proposed changes section.
>> >> > > > a. Added getHeaders method to Producer/Consumer record (that
>> is it)
>> >> > > > b. We’ve also detailed the new Headers class that this method
>> >> > returns
>> >> > > that
>> >> > > > encapsulates the headers protocol and logic.
>> >> > > >
>> >> > > > Best,
>> >> > > > Mike
>> >> > > >
>> >> > > > ==Original questions from the vote thread from Jay.==
>> >> > > >
>> >> > > > Couple of things I think we still need to work out:
>> >> > > >
>> >> > > > 1. I think we agree about the key, but I think we haven't
>> >> > talked about
>> >> > > > the value yet. I think if our goal is an open ecosystem
>> of these
>> >> > > header
>> >> > > > spread across many plugins from many systems we should
>> consider
>> >> > making
>> >> > > > this
>> >> > > > a string as well so it can be printed, set via a UI, set
>> in
>> >> > config,
>> >> > > etc.
>> >> > > > Basically encouraging pluggable serialization formats
>> here will
>> >> > lead
>> >> > > to
>> >> > > > a
>> >> > > > bit of a tower of babel.
>> >> > > > 2. This proposal still includes a pretty big change to our
>> >> > > serialization
>> >> > > > and protocol definition layer. Essentially it is
>> introducing an
>> >> > > optional
>> >> > > > type, where the format is data dependent. I think this is
>> >> > actually a
>> >> > > big
>> >> > > > change though it doesn't seem like it. It means you can no
>> >> > longer
>> >> > > > specify
>> >> > > > this type with our type definition DSL, and likewise it
>> requires
>> >> > > custom
>> >> > > > handling in client libs. This isn't a huge thing, since
>> the
>> >> > Record
>> >> > > > definition is custom anyway, but I think this kind of
>> protocol
>> >> > > > inconsistency is very non-desirable and ties you to
>> hand-coding
>> >> > > things.
>> >> > > > I
>> >> > > > think the type should instead by [Key Value] in our BNF,
>> where
>> >> > key and
>> >> > > > value are both short strings as used elsewhere. This
>> brings it
>> >> > in line
>> >> > > > with
>> >> > > > the rest of the protocol.
>> >> > > > 3. Could we get more specific about the exact Java API
>> change to
>> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
>> >> > > >
>> >> > > > -Jay
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> > The information contained in this email is strictly confidential and
>> for
>> >> > the use of the addressee only, unless otherwise indicated. If you
>> are not
>> >> > the intended recipient, please do not read, copy, use or disclose to
>> others
>> >> > this message or any attachment. Please also notify the sender by
>> replying
>> >> > to this email or by telephone (+44(020 7896 0011) and then delete
>> the email
>> >> > and any copies of it. Opinions, conclusion (etc) that do not relate
>> to the
>> >> > official business of this company shall be understood as neither
>> given nor
>> >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> >> > registered in England and Wales, company number 04008957) and IG
>> Index
>> >> > Limited (a company registered in England and Wales, company number
>> >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> >> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> and IG
>> >> > Index Limited (register number 114059) are authorised and regulated
>> by the
>> >> > Financial Conduct Authority.
>> >> >
>>
>>
>>
>> --
>> Nacho - Ignacio Solis - isolis@igso.net
>>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Sorry, should have noted that the performance testing was done using the
producer performance tool shipped with Kafka.
-Jason
On Thu, Feb 16, 2017 at 6:44 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hey Nacho,
>
> I've compared performance of our KIP-98 implementation with and without
> varints. For messages around 128 bytes, we see an increase in throughput of
> about 30% using the default configuration settings. At 256 bytes, the
> increase is around 16%. Obviously the performance converge as messages get
> larger, but it seems well worth the cost. Note that we are also seeing a
> substantial performance increase against trunk primarily because of the
> much more efficient packing that varints provide us. Anything adding to
> message overhead, such as record headers, would only increase the relative
> difference. (Of course take these numbers with a grain of salt since I have
> only used the default settings with both the producer and broker on my
> local machine. We intend to provide more extensive performance details as
> part of the work for KIP-98.)
>
> The implementation we are using is from protobuf (
> https://developers.google.com/protocol-buffers/docs/encoding), which is
> also used in HBase. It is trivial to implement and as far as I know doesn't
> suffer from the aliasing problem you are describing. I checked with Magnus
> (the author of librdkafka) and he agreed that the savings seemed worth the
> cost of implementation.
>
> -Jason
>
> On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <is...@igso.net> wrote:
>
>> -VarInts
>>
>> I'm one of the people (if not the most) opposed to VarInts. VarInts
>> have a place, but this is not it. (We had a large discussion about
>> them at the beginning of KIP-82 time)
>>
>> If anybody has real life performance numbers of VarInts improving
>> things or significantly reducing resources I would like to know what
>> that case may be. Yes, you can save some bytes here and there, but
>> this is probably insignificant to the overall system behavior and
>> storage requirements. -- I say this with respect to using VarInts in
>> the protocol itself, not as part of the data.
>>
>> VarInts require you to parse the Int before using it and depending on
>> the encoding they can suffer from aliasing (multiple representations
>> for the same value).
>>
>> Why add complexity?
>>
>> Nacho
>>
>>
>> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <cm...@apache.org>
>> wrote:
>> > +1 for varints here-- it would save quite a bit of space. They are
>> > pretty quick to implement as well.
>> >
>> > I think it makes sense for values to be byte arrays. Users might want
>> > to attach arbitrary payloads; they shouldn't be forced to serialize
>> > everything to Java strings.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
>> >> Hey Michael,
>> >>
>> >> Hmm, I guess the point of representing it as bytes is to allow the
>> broker
>> >> to pass it through opaquely? Is the cost of parsing them a concern, or
>> >> are
>> >> we simply trying to ensure that the broker stays agnostic to the
>> format?
>> >>
>> >> On varints, I think adding support for them makes less sense for an
>> >> isolated use case, but as part of a more holistic change (such as what
>> we
>> >> have proposed in KIP-98), I think they are justifiable. If we add them,
>> >> then the need to use attributes becomes quite a bit weaker, right? The
>> >> other thing I find slightly odd is the fact that null headers has no
>> >> actual
>> >> semantic meaning for the message (unlike null keys and values). It is
>> >> just
>> >> a space optimization. It seems a bit better to always use size 0 to
>> >> indicate having no headers.
>> >>
>> >> Overall, the main point is ensuring that the message schema remains
>> >> consistent, either within the larger protocol, or at a minimum within
>> the
>> >> message itself.
>> >>
>> >> -Jason
>> >>
>> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <Michael.Pearce@ig.com
>> >
>> >> wrote:
>> >>
>> >> > Hi Jason,
>> >> >
>> >> > On point 1) in the message protocol the headers are simply a byte
>> array,
>> >> > as like the key or value, this is to clearly demarcate the header in
>> the
>> >> > core message. Then the header byte array in the core message is an
>> array of
>> >> > key, value pairs. This is what it is denoting.
>> >> >
>> >> > Then this would be I guess in the given notation:
>> >> >
>> >> > Headers => [KeyLength, Key, ValueLength, Value]
>> >> > KeyLength => int32 <-----------------NEW size of the byte[] of
>> the
>> >> > serialised key value
>> >> > Key => bytes <---------------------- NEW serialised string (UTF8)
>> >> > bytes of the header key
>> >> > ValueLength => int32 <-------------- NEW size of the byte[] of
>> the
>> >> > serialised header value
>> >> > Value => bytes <-------------------- NEW serialised form of the
>> header
>> >> > value
>> >> >
>> >> > The key length and value length is matching the way the protocol is
>> >> > defined in the core message currently.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On point 2)
>> >> > Var sized ints, this was discussed much earlier on, in fact I had
>> >> > suggested it myself (with Hadoop references), the complexity of this
>> >> > compared to having a simpler protocol was argued and agreed it
>> wasn’t worth
>> >> > the complexity as all other clients in other languages would need to
>> ensure
>> >> > theyre using the right var size algorithm, as there is a few.
>> >> >
>> >> > On point 3)
>> >> > We did the attributes, optional approach as originally there was
>> marked
>> >> > concern that headers would cause a message size overhead for others,
>> who
>> >> > don’t want them. As such this is the clean solution to achieve that.
>> If
>> >> > that no longer holds, and we don’t care that we add 4bytes overhead,
>> then
>> >> > im happy to remove.
>> >> >
>> >> > I’m personally in favour of keeping the message as small as possible
>> so
>> >> > people don’t get shocks in perf and throughputs dues to message size,
>> >> > unless they actively use the feature, as such I do prefer the
>> attribute bit
>> >> > wise feature flag approach myself.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
>> >> >
>> >> > We have proposed a few significant changes to the message format
>> in
>> >> > KIP-98
>> >> > which now seems likely to pass (perhaps with some iterations on
>> >> > implementation details). It would be good to try and coordinate
>> the
>> >> > changes
>> >> > in both of the proposals to make sure they are consistent and
>> >> > compatible.
>> >> >
>> >> > I think using the attributes to indicate null headers is a
>> reasonable
>> >> > approach. We have proposed to do the same thing for the message
>> key and
>> >> > value. That said, I sympathize with Jay's argument. Having
>> multiple
>> >> > ways to
>> >> > specify a null value increases the overall complexity of the
>> protocol.
>> >> > You
>> >> > can see this just from the fact that you need the extra verbiage
>> in the
>> >> > protocol specification in this KIP and in KIP-98 to describe the
>> >> > dependence
>> >> > between the fields and the attributes. It seems like a slippery
>> slope
>> >> > if
>> >> > you start allowing different request types to implement the
>> protocol
>> >> > specification differently.
>> >> >
>> >> > You can also argue that the messages already are and are likely
>> to
>> >> > remain a
>> >> > special case. For example, there is currently no generality in
>> how
>> >> > compressed message sets are represented that would be applicable
>> for
>> >> > other
>> >> > request types. Some might see this divergence as an unfortunate
>> >> > protocol
>> >> > deficiency which should be fixed; others might see it as sort of
>> the
>> >> > inevitability of needing to optimize where it counts most. I'm
>> probably
>> >> > somewhere in between, but I think we probably all share the
>> intuition
>> >> > that
>> >> > the protocol should be kept as consistent as possible. With that
>> in
>> >> > mind,
>> >> > here are a few comments:
>> >> >
>> >> > 1. One thing I found a little odd when reading the current
>> proposal is
>> >> > that
>> >> > the headers are both represented as an array of bytes and as an
>> array
>> >> > of
>> >> > key/value pairs. I'd probably suggest something like this:
>> >> >
>> >> > Headers => [HeaderKey HeaderValue]
>> >> > HeaderKey => String
>> >> > HeaderValue => Bytes
>> >> >
>> >> > An array in the Kafka protocol is represented as a 4-byte integer
>> >> > indicating the number of elements in the array followed by the
>> >> > serialization of the elements. Unless I'm misunderstanding, what
>> you
>> >> > have
>> >> > instead is the total size of the headers in bytes followed by the
>> >> > elements.
>> >> > I'm not sure I see any reason for this inconsistency.
>> >> >
>> >> > 2. In KIP-98, we've introduced variable-length integer fields.
>> >> > Effectively,
>> >> > we've enriched (or "complicated" as Jay might say ;) the protocol
>> >> > specification to include the following types: VarInt, VarLong,
>> >> > UnsignedVarInt and UnsignedVarLong.
>> >> >
>> >> > Along with these primitives, we could introduce the following
>> types:
>> >> >
>> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
>> >> > NumberOfItems => UnsignedVarInt
>> >> >
>> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
>> >> > NumberOfItemsOrNull => VarInt (-1 means null)
>> >> >
>> >> > And similarly for the `String` and `Bytes` types. These types
>> can save
>> >> > a
>> >> > considerable amount of space in this proposal because they can
>> be used
>> >> > for
>> >> > both the number of headers included in the message and the
>> lengths of
>> >> > the
>> >> > header keys and values. We could do this instead:
>> >> >
>> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
>> >> > HeaderKey => VarSizeString
>> >> > HeaderValue => VarSizeBytes
>> >> >
>> >> > Combining the savings from the use of variable length fields, the
>> >> > benefit
>> >> > of using the attributes to represent null seems pretty small.
>> >> >
>> >> > 3. Whichever way we go (whether we use the attributes or not), we
>> >> > should at
>> >> > least be consistent between this KIP and KIP-98. It would be very
>> >> > strange
>> >> > to have two ways to represent null values in the same schema.
>> Either
>> >> > way is
>> >> > OK with me. I think some message-level optimizations are
>> justifiable,
>> >> > but
>> >> > the savings here seem minimal (a few bytes per message), so
>> maybe it's
>> >> > not
>> >> > worth the cost of letting the message diverge even further from
>> the
>> >> > rest of
>> >> > the protocol.
>> >> >
>> >> > -Jason
>> >> >
>> >> >
>> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
>> radai.rosenblatt@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > I've trimmed the inline contents as this mail is getting too
>> big for
>> >> > the
>> >> > > apache mailing list software to deliver :-(
>> >> > >
>> >> > > 1. the important thing for interoperability is for different
>> >> > "interested
>> >> > > parties" (plugins, infra layers/wrappers, user-code) to be
>> able to
>> >> > stick
>> >> > > pieces of metadata onto msgs without getting in each other's
>> way. a
>> >> > common
>> >> > > key scheme (Strings, as of the time of this writing?) is all
>> thats
>> >> > required
>> >> > > for that. it is assumed that the other end interested in any
>> such
>> >> > piece of
>> >> > > metadata knows the encoding, and byte[] provides for the most
>> >> > flexibility.
>> >> > > i believe this is the same logic behind core kafka being
>> >> > byte[]/byte[] -
>> >> > > Strings are more "usable" but bytes are flexible and so were
>> chosen.
>> >> > > Also - core kafka doesnt even do that good of a job on
>> usability of
>> >> > the
>> >> > > payload (example - i have to specify the nop byte[] "decoders"
>> >> > explicitly
>> >> > > in conf), and again sacrificies usability for the sake of
>> >> > performance (no
>> >> > > convenient single-record processing as poll is a batch, lots of
>> >> > obscure
>> >> > > little config details exposing internals of the batching
>> mechanism,
>> >> > etc)
>> >> > >
>> >> > > this is also why i really dislike the idea of a "type system"
>> for
>> >> > header
>> >> > > values, it further degrades the usability, adds complexity and
>> will
>> >> > > eventually get in people's way, also, it would be the 2nd/3rd
>> >> > home-group
>> >> > > serialization mechanism in core kafka (counting 2 iterations
>> of the
>> >> > "type
>> >> > > definition DSL")
>> >> > >
>> >> > > 2. this is an implementation detail, and not even a very "user
>> >> > facing" one?
>> >> > > to the best of my understanding the vote process is on proposed
>> >> > > API/behaviour. also - since we're willing to go with strings
>> just
>> >> > serialize
>> >> > > a 0-sized header blob and IIUC you dont need any optionals
>> anymore.
>> >> > >
>> >> > > 3. yes, we can :-)
>> >> > >
>> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
>> >> > Michael.Pearce@ig.com>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Jay,
>> >> > > >
>> >> > > > 1) There was some initial debate on the value part, as youll
>> note
>> >> > String,
>> >> > > > String headers were discounted early on. The reason for this
>> is
>> >> > > flexibility
>> >> > > > and keeping in line with the flexibility of key, value of the
>> >> > message
>> >> > > > object itself. I don’t think it takes away from an ecosystem
>> as
>> >> > each
>> >> > > plugin
>> >> > > > will care for their own key, this way ints, booleans , exotic
>> >> > custom
>> >> > > binary
>> >> > > > can all be catered for=.
>> >> > > > a. If you really wanted to push for a typed value interface,
>> I
>> >> > wouldn’t
>> >> > > > want just String values supported, but the the primatives
>> plus
>> >> > string and
>> >> > > > also still keeping the ability to have a binary for custom
>> >> > binaries that
>> >> > > > some organisations may have.
>> >> > > > i. I have written this slight alternative here,
>> >> > > https://cwiki.apache.org/
>> >> > > > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
>> >> > > > ii. Essentially the value bytes, has a leading byte overhead.
>> >> > > > 1. This tells you what type the value is, before reading
>> the rest
>> >> > of the
>> >> > > > bytes, allowing serialisation/deserialization to and from the
>> >> > primitives,
>> >> > > > string and byte[]. This is akin to some other messaging
>> systems.
>> >> > > > 2) We are making it optional, so that for those not wanting
>> >> > headers have
>> >> > > 0
>> >> > > > bytes overhead (think of it as a feature flag), I don’t
>> think this
>> >> > is
>> >> > > > complex, especially if comparing to changes proposed in
>> other kips
>> >> > like
>> >> > > > kip-98.
>> >> > > > a. If you really really don’t like this, we can drop it, but
>> it
>> >> > would
>> >> > > mean
>> >> > > > buying into 4 bytes extra overhead for users who do not want
>> to use
>> >> > > headers.
>> >> > > > 3) In the summary yes, it is at a higher level, but I think
>> this
>> >> > is well
>> >> > > > documented in the proposed changes section.
>> >> > > > a. Added getHeaders method to Producer/Consumer record (that
>> is it)
>> >> > > > b. We’ve also detailed the new Headers class that this method
>> >> > returns
>> >> > > that
>> >> > > > encapsulates the headers protocol and logic.
>> >> > > >
>> >> > > > Best,
>> >> > > > Mike
>> >> > > >
>> >> > > > ==Original questions from the vote thread from Jay.==
>> >> > > >
>> >> > > > Couple of things I think we still need to work out:
>> >> > > >
>> >> > > > 1. I think we agree about the key, but I think we haven't
>> >> > talked about
>> >> > > > the value yet. I think if our goal is an open ecosystem
>> of these
>> >> > > header
>> >> > > > spread across many plugins from many systems we should
>> consider
>> >> > making
>> >> > > > this
>> >> > > > a string as well so it can be printed, set via a UI, set
>> in
>> >> > config,
>> >> > > etc.
>> >> > > > Basically encouraging pluggable serialization formats
>> here will
>> >> > lead
>> >> > > to
>> >> > > > a
>> >> > > > bit of a tower of babel.
>> >> > > > 2. This proposal still includes a pretty big change to our
>> >> > > serialization
>> >> > > > and protocol definition layer. Essentially it is
>> introducing an
>> >> > > optional
>> >> > > > type, where the format is data dependent. I think this is
>> >> > actually a
>> >> > > big
>> >> > > > change though it doesn't seem like it. It means you can no
>> >> > longer
>> >> > > > specify
>> >> > > > this type with our type definition DSL, and likewise it
>> requires
>> >> > > custom
>> >> > > > handling in client libs. This isn't a huge thing, since
>> the
>> >> > Record
>> >> > > > definition is custom anyway, but I think this kind of
>> protocol
>> >> > > > inconsistency is very non-desirable and ties you to
>> hand-coding
>> >> > > things.
>> >> > > > I
>> >> > > > think the type should instead by [Key Value] in our BNF,
>> where
>> >> > key and
>> >> > > > value are both short strings as used elsewhere. This
>> brings it
>> >> > in line
>> >> > > > with
>> >> > > > the rest of the protocol.
>> >> > > > 3. Could we get more specific about the exact Java API
>> change to
>> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
>> >> > > >
>> >> > > > -Jay
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> > The information contained in this email is strictly confidential and
>> for
>> >> > the use of the addressee only, unless otherwise indicated. If you
>> are not
>> >> > the intended recipient, please do not read, copy, use or disclose to
>> others
>> >> > this message or any attachment. Please also notify the sender by
>> replying
>> >> > to this email or by telephone (+44(020 7896 0011) and then delete
>> the email
>> >> > and any copies of it. Opinions, conclusion (etc) that do not relate
>> to the
>> >> > official business of this company shall be understood as neither
>> given nor
>> >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> >> > registered in England and Wales, company number 04008957) and IG
>> Index
>> >> > Limited (a company registered in England and Wales, company number
>> >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> >> > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> and IG
>> >> > Index Limited (register number 114059) are authorised and regulated
>> by the
>> >> > Financial Conduct Authority.
>> >> >
>>
>>
>>
>> --
>> Nacho - Ignacio Solis - isolis@igso.net
>>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Nacho,
I've compared performance of our KIP-98 implementation with and without
varints. For messages around 128 bytes, we see an increase in throughput of
about 30% using the default configuration settings. At 256 bytes, the
increase is around 16%. Obviously the performance converge as messages get
larger, but it seems well worth the cost. Note that we are also seeing a
substantial performance increase against trunk primarily because of the
much more efficient packing that varints provide us. Anything adding to
message overhead, such as record headers, would only increase the relative
difference. (Of course take these numbers with a grain of salt since I have
only used the default settings with both the producer and broker on my
local machine. We intend to provide more extensive performance details as
part of the work for KIP-98.)
The implementation we are using is from protobuf (
https://developers.google.com/protocol-buffers/docs/encoding), which is
also used in HBase. It is trivial to implement and as far as I know doesn't
suffer from the aliasing problem you are describing. I checked with Magnus
(the author of librdkafka) and he agreed that the savings seemed worth the
cost of implementation.
-Jason
On Thu, Feb 16, 2017 at 4:32 PM, Ignacio Solis <is...@igso.net> wrote:
> -VarInts
>
> I'm one of the people (if not the most) opposed to VarInts. VarInts
> have a place, but this is not it. (We had a large discussion about
> them at the beginning of KIP-82 time)
>
> If anybody has real life performance numbers of VarInts improving
> things or significantly reducing resources I would like to know what
> that case may be. Yes, you can save some bytes here and there, but
> this is probably insignificant to the overall system behavior and
> storage requirements. -- I say this with respect to using VarInts in
> the protocol itself, not as part of the data.
>
> VarInts require you to parse the Int before using it and depending on
> the encoding they can suffer from aliasing (multiple representations
> for the same value).
>
> Why add complexity?
>
> Nacho
>
>
> On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <cm...@apache.org> wrote:
> > +1 for varints here-- it would save quite a bit of space. They are
> > pretty quick to implement as well.
> >
> > I think it makes sense for values to be byte arrays. Users might want
> > to attach arbitrary payloads; they shouldn't be forced to serialize
> > everything to Java strings.
> >
> > best,
> > Colin
> >
> >
> > On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> >> Hey Michael,
> >>
> >> Hmm, I guess the point of representing it as bytes is to allow the
> broker
> >> to pass it through opaquely? Is the cost of parsing them a concern, or
> >> are
> >> we simply trying to ensure that the broker stays agnostic to the format?
> >>
> >> On varints, I think adding support for them makes less sense for an
> >> isolated use case, but as part of a more holistic change (such as what
> we
> >> have proposed in KIP-98), I think they are justifiable. If we add them,
> >> then the need to use attributes becomes quite a bit weaker, right? The
> >> other thing I find slightly odd is the fact that null headers has no
> >> actual
> >> semantic meaning for the message (unlike null keys and values). It is
> >> just
> >> a space optimization. It seems a bit better to always use size 0 to
> >> indicate having no headers.
> >>
> >> Overall, the main point is ensuring that the message schema remains
> >> consistent, either within the larger protocol, or at a minimum within
> the
> >> message itself.
> >>
> >> -Jason
> >>
> >> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >> > Hi Jason,
> >> >
> >> > On point 1) in the message protocol the headers are simply a byte
> array,
> >> > as like the key or value, this is to clearly demarcate the header in
> the
> >> > core message. Then the header byte array in the core message is an
> array of
> >> > key, value pairs. This is what it is denoting.
> >> >
> >> > Then this would be I guess in the given notation:
> >> >
> >> > Headers => [KeyLength, Key, ValueLength, Value]
> >> > KeyLength => int32 <-----------------NEW size of the byte[] of the
> >> > serialised key value
> >> > Key => bytes <---------------------- NEW serialised string (UTF8)
> >> > bytes of the header key
> >> > ValueLength => int32 <-------------- NEW size of the byte[] of the
> >> > serialised header value
> >> > Value => bytes <-------------------- NEW serialised form of the
> header
> >> > value
> >> >
> >> > The key length and value length is matching the way the protocol is
> >> > defined in the core message currently.
> >> >
> >> >
> >> >
> >> >
> >> > On point 2)
> >> > Var sized ints, this was discussed much earlier on, in fact I had
> >> > suggested it myself (with Hadoop references), the complexity of this
> >> > compared to having a simpler protocol was argued and agreed it wasn’t
> worth
> >> > the complexity as all other clients in other languages would need to
> ensure
> >> > theyre using the right var size algorithm, as there is a few.
> >> >
> >> > On point 3)
> >> > We did the attributes, optional approach as originally there was
> marked
> >> > concern that headers would cause a message size overhead for others,
> who
> >> > don’t want them. As such this is the clean solution to achieve that.
> If
> >> > that no longer holds, and we don’t care that we add 4bytes overhead,
> then
> >> > im happy to remove.
> >> >
> >> > I’m personally in favour of keeping the message as small as possible
> so
> >> > people don’t get shocks in perf and throughputs dues to message size,
> >> > unless they actively use the feature, as such I do prefer the
> attribute bit
> >> > wise feature flag approach myself.
> >> >
> >> >
> >> >
> >> >
> >> > On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
> >> >
> >> > We have proposed a few significant changes to the message format
> in
> >> > KIP-98
> >> > which now seems likely to pass (perhaps with some iterations on
> >> > implementation details). It would be good to try and coordinate
> the
> >> > changes
> >> > in both of the proposals to make sure they are consistent and
> >> > compatible.
> >> >
> >> > I think using the attributes to indicate null headers is a
> reasonable
> >> > approach. We have proposed to do the same thing for the message
> key and
> >> > value. That said, I sympathize with Jay's argument. Having
> multiple
> >> > ways to
> >> > specify a null value increases the overall complexity of the
> protocol.
> >> > You
> >> > can see this just from the fact that you need the extra verbiage
> in the
> >> > protocol specification in this KIP and in KIP-98 to describe the
> >> > dependence
> >> > between the fields and the attributes. It seems like a slippery
> slope
> >> > if
> >> > you start allowing different request types to implement the
> protocol
> >> > specification differently.
> >> >
> >> > You can also argue that the messages already are and are likely to
> >> > remain a
> >> > special case. For example, there is currently no generality in how
> >> > compressed message sets are represented that would be applicable
> for
> >> > other
> >> > request types. Some might see this divergence as an unfortunate
> >> > protocol
> >> > deficiency which should be fixed; others might see it as sort of
> the
> >> > inevitability of needing to optimize where it counts most. I'm
> probably
> >> > somewhere in between, but I think we probably all share the
> intuition
> >> > that
> >> > the protocol should be kept as consistent as possible. With that
> in
> >> > mind,
> >> > here are a few comments:
> >> >
> >> > 1. One thing I found a little odd when reading the current
> proposal is
> >> > that
> >> > the headers are both represented as an array of bytes and as an
> array
> >> > of
> >> > key/value pairs. I'd probably suggest something like this:
> >> >
> >> > Headers => [HeaderKey HeaderValue]
> >> > HeaderKey => String
> >> > HeaderValue => Bytes
> >> >
> >> > An array in the Kafka protocol is represented as a 4-byte integer
> >> > indicating the number of elements in the array followed by the
> >> > serialization of the elements. Unless I'm misunderstanding, what
> you
> >> > have
> >> > instead is the total size of the headers in bytes followed by the
> >> > elements.
> >> > I'm not sure I see any reason for this inconsistency.
> >> >
> >> > 2. In KIP-98, we've introduced variable-length integer fields.
> >> > Effectively,
> >> > we've enriched (or "complicated" as Jay might say ;) the protocol
> >> > specification to include the following types: VarInt, VarLong,
> >> > UnsignedVarInt and UnsignedVarLong.
> >> >
> >> > Along with these primitives, we could introduce the following
> types:
> >> >
> >> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> >> > NumberOfItems => UnsignedVarInt
> >> >
> >> > VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
> >> > NumberOfItemsOrNull => VarInt (-1 means null)
> >> >
> >> > And similarly for the `String` and `Bytes` types. These types can
> save
> >> > a
> >> > considerable amount of space in this proposal because they can be
> used
> >> > for
> >> > both the number of headers included in the message and the
> lengths of
> >> > the
> >> > header keys and values. We could do this instead:
> >> >
> >> > Headers => VarSizeArray[HeaderKey HeaderValue]
> >> > HeaderKey => VarSizeString
> >> > HeaderValue => VarSizeBytes
> >> >
> >> > Combining the savings from the use of variable length fields, the
> >> > benefit
> >> > of using the attributes to represent null seems pretty small.
> >> >
> >> > 3. Whichever way we go (whether we use the attributes or not), we
> >> > should at
> >> > least be consistent between this KIP and KIP-98. It would be very
> >> > strange
> >> > to have two ways to represent null values in the same schema.
> Either
> >> > way is
> >> > OK with me. I think some message-level optimizations are
> justifiable,
> >> > but
> >> > the savings here seem minimal (a few bytes per message), so maybe
> it's
> >> > not
> >> > worth the cost of letting the message diverge even further from
> the
> >> > rest of
> >> > the protocol.
> >> >
> >> > -Jason
> >> >
> >> >
> >> > On Wed, Feb 15, 2017 at 8:52 AM, radai <
> radai.rosenblatt@gmail.com>
> >> > wrote:
> >> >
> >> > > I've trimmed the inline contents as this mail is getting too
> big for
> >> > the
> >> > > apache mailing list software to deliver :-(
> >> > >
> >> > > 1. the important thing for interoperability is for different
> >> > "interested
> >> > > parties" (plugins, infra layers/wrappers, user-code) to be able
> to
> >> > stick
> >> > > pieces of metadata onto msgs without getting in each other's
> way. a
> >> > common
> >> > > key scheme (Strings, as of the time of this writing?) is all
> thats
> >> > required
> >> > > for that. it is assumed that the other end interested in any
> such
> >> > piece of
> >> > > metadata knows the encoding, and byte[] provides for the most
> >> > flexibility.
> >> > > i believe this is the same logic behind core kafka being
> >> > byte[]/byte[] -
> >> > > Strings are more "usable" but bytes are flexible and so were
> chosen.
> >> > > Also - core kafka doesnt even do that good of a job on
> usability of
> >> > the
> >> > > payload (example - i have to specify the nop byte[] "decoders"
> >> > explicitly
> >> > > in conf), and again sacrificies usability for the sake of
> >> > performance (no
> >> > > convenient single-record processing as poll is a batch, lots of
> >> > obscure
> >> > > little config details exposing internals of the batching
> mechanism,
> >> > etc)
> >> > >
> >> > > this is also why i really dislike the idea of a "type system"
> for
> >> > header
> >> > > values, it further degrades the usability, adds complexity and
> will
> >> > > eventually get in people's way, also, it would be the 2nd/3rd
> >> > home-group
> >> > > serialization mechanism in core kafka (counting 2 iterations of
> the
> >> > "type
> >> > > definition DSL")
> >> > >
> >> > > 2. this is an implementation detail, and not even a very "user
> >> > facing" one?
> >> > > to the best of my understanding the vote process is on proposed
> >> > > API/behaviour. also - since we're willing to go with strings
> just
> >> > serialize
> >> > > a 0-sized header blob and IIUC you dont need any optionals
> anymore.
> >> > >
> >> > > 3. yes, we can :-)
> >> > >
> >> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
> >> > Michael.Pearce@ig.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Jay,
> >> > > >
> >> > > > 1) There was some initial debate on the value part, as youll
> note
> >> > String,
> >> > > > String headers were discounted early on. The reason for this
> is
> >> > > flexibility
> >> > > > and keeping in line with the flexibility of key, value of the
> >> > message
> >> > > > object itself. I don’t think it takes away from an ecosystem
> as
> >> > each
> >> > > plugin
> >> > > > will care for their own key, this way ints, booleans , exotic
> >> > custom
> >> > > binary
> >> > > > can all be catered for=.
> >> > > > a. If you really wanted to push for a typed value interface, I
> >> > wouldn’t
> >> > > > want just String values supported, but the the primatives plus
> >> > string and
> >> > > > also still keeping the ability to have a binary for custom
> >> > binaries that
> >> > > > some organisations may have.
> >> > > > i. I have written this slight alternative here,
> >> > > https://cwiki.apache.org/
> >> > > > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
> >> > > > ii. Essentially the value bytes, has a leading byte overhead.
> >> > > > 1. This tells you what type the value is, before reading the
> rest
> >> > of the
> >> > > > bytes, allowing serialisation/deserialization to and from the
> >> > primitives,
> >> > > > string and byte[]. This is akin to some other messaging
> systems.
> >> > > > 2) We are making it optional, so that for those not wanting
> >> > headers have
> >> > > 0
> >> > > > bytes overhead (think of it as a feature flag), I don’t think
> this
> >> > is
> >> > > > complex, especially if comparing to changes proposed in other
> kips
> >> > like
> >> > > > kip-98.
> >> > > > a. If you really really don’t like this, we can drop it, but
> it
> >> > would
> >> > > mean
> >> > > > buying into 4 bytes extra overhead for users who do not want
> to use
> >> > > headers.
> >> > > > 3) In the summary yes, it is at a higher level, but I think
> this
> >> > is well
> >> > > > documented in the proposed changes section.
> >> > > > a. Added getHeaders method to Producer/Consumer record (that
> is it)
> >> > > > b. We’ve also detailed the new Headers class that this method
> >> > returns
> >> > > that
> >> > > > encapsulates the headers protocol and logic.
> >> > > >
> >> > > > Best,
> >> > > > Mike
> >> > > >
> >> > > > ==Original questions from the vote thread from Jay.==
> >> > > >
> >> > > > Couple of things I think we still need to work out:
> >> > > >
> >> > > > 1. I think we agree about the key, but I think we haven't
> >> > talked about
> >> > > > the value yet. I think if our goal is an open ecosystem of
> these
> >> > > header
> >> > > > spread across many plugins from many systems we should
> consider
> >> > making
> >> > > > this
> >> > > > a string as well so it can be printed, set via a UI, set in
> >> > config,
> >> > > etc.
> >> > > > Basically encouraging pluggable serialization formats here
> will
> >> > lead
> >> > > to
> >> > > > a
> >> > > > bit of a tower of babel.
> >> > > > 2. This proposal still includes a pretty big change to our
> >> > > serialization
> >> > > > and protocol definition layer. Essentially it is
> introducing an
> >> > > optional
> >> > > > type, where the format is data dependent. I think this is
> >> > actually a
> >> > > big
> >> > > > change though it doesn't seem like it. It means you can no
> >> > longer
> >> > > > specify
> >> > > > this type with our type definition DSL, and likewise it
> requires
> >> > > custom
> >> > > > handling in client libs. This isn't a huge thing, since the
> >> > Record
> >> > > > definition is custom anyway, but I think this kind of
> protocol
> >> > > > inconsistency is very non-desirable and ties you to
> hand-coding
> >> > > things.
> >> > > > I
> >> > > > think the type should instead by [Key Value] in our BNF,
> where
> >> > key and
> >> > > > value are both short strings as used elsewhere. This
> brings it
> >> > in line
> >> > > > with
> >> > > > the rest of the protocol.
> >> > > > 3. Could we get more specific about the exact Java API
> change to
> >> > > > ProducerRecord, ConsumerRecord, Record, etc?
> >> > > >
> >> > > > -Jay
> >> > > >
> >> > >
> >> >
> >> >
> >> > The information contained in this email is strictly confidential and
> for
> >> > the use of the addressee only, unless otherwise indicated. If you are
> not
> >> > the intended recipient, please do not read, copy, use or disclose to
> others
> >> > this message or any attachment. Please also notify the sender by
> replying
> >> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >> > and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> >> > official business of this company shall be understood as neither
> given nor
> >> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> >> > registered in England and Wales, company number 04008957) and IG Index
> >> > Limited (a company registered in England and Wales, company number
> >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >> > Index Limited (register number 114059) are authorised and regulated
> by the
> >> > Financial Conduct Authority.
> >> >
>
>
>
> --
> Nacho - Ignacio Solis - isolis@igso.net
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ignacio Solis <is...@igso.net>.
-VarInts
I'm one of the people (if not the most) opposed to VarInts. VarInts
have a place, but this is not it. (We had a large discussion about
them at the beginning of KIP-82 time)
If anybody has real life performance numbers of VarInts improving
things or significantly reducing resources I would like to know what
that case may be. Yes, you can save some bytes here and there, but
this is probably insignificant to the overall system behavior and
storage requirements. -- I say this with respect to using VarInts in
the protocol itself, not as part of the data.
VarInts require you to parse the Int before using it and depending on
the encoding they can suffer from aliasing (multiple representations
for the same value).
Why add complexity?
Nacho
On Thu, Feb 16, 2017 at 10:29 AM, Colin McCabe <cm...@apache.org> wrote:
> +1 for varints here-- it would save quite a bit of space. They are
> pretty quick to implement as well.
>
> I think it makes sense for values to be byte arrays. Users might want
> to attach arbitrary payloads; they shouldn't be forced to serialize
> everything to Java strings.
>
> best,
> Colin
>
>
> On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
>> Hey Michael,
>>
>> Hmm, I guess the point of representing it as bytes is to allow the broker
>> to pass it through opaquely? Is the cost of parsing them a concern, or
>> are
>> we simply trying to ensure that the broker stays agnostic to the format?
>>
>> On varints, I think adding support for them makes less sense for an
>> isolated use case, but as part of a more holistic change (such as what we
>> have proposed in KIP-98), I think they are justifiable. If we add them,
>> then the need to use attributes becomes quite a bit weaker, right? The
>> other thing I find slightly odd is the fact that null headers has no
>> actual
>> semantic meaning for the message (unlike null keys and values). It is
>> just
>> a space optimization. It seems a bit better to always use size 0 to
>> indicate having no headers.
>>
>> Overall, the main point is ensuring that the message schema remains
>> consistent, either within the larger protocol, or at a minimum within the
>> message itself.
>>
>> -Jason
>>
>> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <Mi...@ig.com>
>> wrote:
>>
>> > Hi Jason,
>> >
>> > On point 1) in the message protocol the headers are simply a byte array,
>> > as like the key or value, this is to clearly demarcate the header in the
>> > core message. Then the header byte array in the core message is an array of
>> > key, value pairs. This is what it is denoting.
>> >
>> > Then this would be I guess in the given notation:
>> >
>> > Headers => [KeyLength, Key, ValueLength, Value]
>> > KeyLength => int32 <-----------------NEW size of the byte[] of the
>> > serialised key value
>> > Key => bytes <---------------------- NEW serialised string (UTF8)
>> > bytes of the header key
>> > ValueLength => int32 <-------------- NEW size of the byte[] of the
>> > serialised header value
>> > Value => bytes <-------------------- NEW serialised form of the header
>> > value
>> >
>> > The key length and value length is matching the way the protocol is
>> > defined in the core message currently.
>> >
>> >
>> >
>> >
>> > On point 2)
>> > Var sized ints, this was discussed much earlier on, in fact I had
>> > suggested it myself (with Hadoop references), the complexity of this
>> > compared to having a simpler protocol was argued and agreed it wasn’t worth
>> > the complexity as all other clients in other languages would need to ensure
>> > theyre using the right var size algorithm, as there is a few.
>> >
>> > On point 3)
>> > We did the attributes, optional approach as originally there was marked
>> > concern that headers would cause a message size overhead for others, who
>> > don’t want them. As such this is the clean solution to achieve that. If
>> > that no longer holds, and we don’t care that we add 4bytes overhead, then
>> > im happy to remove.
>> >
>> > I’m personally in favour of keeping the message as small as possible so
>> > people don’t get shocks in perf and throughputs dues to message size,
>> > unless they actively use the feature, as such I do prefer the attribute bit
>> > wise feature flag approach myself.
>> >
>> >
>> >
>> >
>> > On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
>> >
>> > We have proposed a few significant changes to the message format in
>> > KIP-98
>> > which now seems likely to pass (perhaps with some iterations on
>> > implementation details). It would be good to try and coordinate the
>> > changes
>> > in both of the proposals to make sure they are consistent and
>> > compatible.
>> >
>> > I think using the attributes to indicate null headers is a reasonable
>> > approach. We have proposed to do the same thing for the message key and
>> > value. That said, I sympathize with Jay's argument. Having multiple
>> > ways to
>> > specify a null value increases the overall complexity of the protocol.
>> > You
>> > can see this just from the fact that you need the extra verbiage in the
>> > protocol specification in this KIP and in KIP-98 to describe the
>> > dependence
>> > between the fields and the attributes. It seems like a slippery slope
>> > if
>> > you start allowing different request types to implement the protocol
>> > specification differently.
>> >
>> > You can also argue that the messages already are and are likely to
>> > remain a
>> > special case. For example, there is currently no generality in how
>> > compressed message sets are represented that would be applicable for
>> > other
>> > request types. Some might see this divergence as an unfortunate
>> > protocol
>> > deficiency which should be fixed; others might see it as sort of the
>> > inevitability of needing to optimize where it counts most. I'm probably
>> > somewhere in between, but I think we probably all share the intuition
>> > that
>> > the protocol should be kept as consistent as possible. With that in
>> > mind,
>> > here are a few comments:
>> >
>> > 1. One thing I found a little odd when reading the current proposal is
>> > that
>> > the headers are both represented as an array of bytes and as an array
>> > of
>> > key/value pairs. I'd probably suggest something like this:
>> >
>> > Headers => [HeaderKey HeaderValue]
>> > HeaderKey => String
>> > HeaderValue => Bytes
>> >
>> > An array in the Kafka protocol is represented as a 4-byte integer
>> > indicating the number of elements in the array followed by the
>> > serialization of the elements. Unless I'm misunderstanding, what you
>> > have
>> > instead is the total size of the headers in bytes followed by the
>> > elements.
>> > I'm not sure I see any reason for this inconsistency.
>> >
>> > 2. In KIP-98, we've introduced variable-length integer fields.
>> > Effectively,
>> > we've enriched (or "complicated" as Jay might say ;) the protocol
>> > specification to include the following types: VarInt, VarLong,
>> > UnsignedVarInt and UnsignedVarLong.
>> >
>> > Along with these primitives, we could introduce the following types:
>> >
>> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
>> > NumberOfItems => UnsignedVarInt
>> >
>> > VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
>> > NumberOfItemsOrNull => VarInt (-1 means null)
>> >
>> > And similarly for the `String` and `Bytes` types. These types can save
>> > a
>> > considerable amount of space in this proposal because they can be used
>> > for
>> > both the number of headers included in the message and the lengths of
>> > the
>> > header keys and values. We could do this instead:
>> >
>> > Headers => VarSizeArray[HeaderKey HeaderValue]
>> > HeaderKey => VarSizeString
>> > HeaderValue => VarSizeBytes
>> >
>> > Combining the savings from the use of variable length fields, the
>> > benefit
>> > of using the attributes to represent null seems pretty small.
>> >
>> > 3. Whichever way we go (whether we use the attributes or not), we
>> > should at
>> > least be consistent between this KIP and KIP-98. It would be very
>> > strange
>> > to have two ways to represent null values in the same schema. Either
>> > way is
>> > OK with me. I think some message-level optimizations are justifiable,
>> > but
>> > the savings here seem minimal (a few bytes per message), so maybe it's
>> > not
>> > worth the cost of letting the message diverge even further from the
>> > rest of
>> > the protocol.
>> >
>> > -Jason
>> >
>> >
>> > On Wed, Feb 15, 2017 at 8:52 AM, radai <ra...@gmail.com>
>> > wrote:
>> >
>> > > I've trimmed the inline contents as this mail is getting too big for
>> > the
>> > > apache mailing list software to deliver :-(
>> > >
>> > > 1. the important thing for interoperability is for different
>> > "interested
>> > > parties" (plugins, infra layers/wrappers, user-code) to be able to
>> > stick
>> > > pieces of metadata onto msgs without getting in each other's way. a
>> > common
>> > > key scheme (Strings, as of the time of this writing?) is all thats
>> > required
>> > > for that. it is assumed that the other end interested in any such
>> > piece of
>> > > metadata knows the encoding, and byte[] provides for the most
>> > flexibility.
>> > > i believe this is the same logic behind core kafka being
>> > byte[]/byte[] -
>> > > Strings are more "usable" but bytes are flexible and so were chosen.
>> > > Also - core kafka doesnt even do that good of a job on usability of
>> > the
>> > > payload (example - i have to specify the nop byte[] "decoders"
>> > explicitly
>> > > in conf), and again sacrificies usability for the sake of
>> > performance (no
>> > > convenient single-record processing as poll is a batch, lots of
>> > obscure
>> > > little config details exposing internals of the batching mechanism,
>> > etc)
>> > >
>> > > this is also why i really dislike the idea of a "type system" for
>> > header
>> > > values, it further degrades the usability, adds complexity and will
>> > > eventually get in people's way, also, it would be the 2nd/3rd
>> > home-group
>> > > serialization mechanism in core kafka (counting 2 iterations of the
>> > "type
>> > > definition DSL")
>> > >
>> > > 2. this is an implementation detail, and not even a very "user
>> > facing" one?
>> > > to the best of my understanding the vote process is on proposed
>> > > API/behaviour. also - since we're willing to go with strings just
>> > serialize
>> > > a 0-sized header blob and IIUC you dont need any optionals anymore.
>> > >
>> > > 3. yes, we can :-)
>> > >
>> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
>> > Michael.Pearce@ig.com>
>> > > wrote:
>> > >
>> > > > Hi Jay,
>> > > >
>> > > > 1) There was some initial debate on the value part, as youll note
>> > String,
>> > > > String headers were discounted early on. The reason for this is
>> > > flexibility
>> > > > and keeping in line with the flexibility of key, value of the
>> > message
>> > > > object itself. I don’t think it takes away from an ecosystem as
>> > each
>> > > plugin
>> > > > will care for their own key, this way ints, booleans , exotic
>> > custom
>> > > binary
>> > > > can all be catered for=.
>> > > > a. If you really wanted to push for a typed value interface, I
>> > wouldn’t
>> > > > want just String values supported, but the the primatives plus
>> > string and
>> > > > also still keeping the ability to have a binary for custom
>> > binaries that
>> > > > some organisations may have.
>> > > > i. I have written this slight alternative here,
>> > > https://cwiki.apache.org/
>> > > > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
>> > > > ii. Essentially the value bytes, has a leading byte overhead.
>> > > > 1. This tells you what type the value is, before reading the rest
>> > of the
>> > > > bytes, allowing serialisation/deserialization to and from the
>> > primitives,
>> > > > string and byte[]. This is akin to some other messaging systems.
>> > > > 2) We are making it optional, so that for those not wanting
>> > headers have
>> > > 0
>> > > > bytes overhead (think of it as a feature flag), I don’t think this
>> > is
>> > > > complex, especially if comparing to changes proposed in other kips
>> > like
>> > > > kip-98.
>> > > > a. If you really really don’t like this, we can drop it, but it
>> > would
>> > > mean
>> > > > buying into 4 bytes extra overhead for users who do not want to use
>> > > headers.
>> > > > 3) In the summary yes, it is at a higher level, but I think this
>> > is well
>> > > > documented in the proposed changes section.
>> > > > a. Added getHeaders method to Producer/Consumer record (that is it)
>> > > > b. We’ve also detailed the new Headers class that this method
>> > returns
>> > > that
>> > > > encapsulates the headers protocol and logic.
>> > > >
>> > > > Best,
>> > > > Mike
>> > > >
>> > > > ==Original questions from the vote thread from Jay.==
>> > > >
>> > > > Couple of things I think we still need to work out:
>> > > >
>> > > > 1. I think we agree about the key, but I think we haven't
>> > talked about
>> > > > the value yet. I think if our goal is an open ecosystem of these
>> > > header
>> > > > spread across many plugins from many systems we should consider
>> > making
>> > > > this
>> > > > a string as well so it can be printed, set via a UI, set in
>> > config,
>> > > etc.
>> > > > Basically encouraging pluggable serialization formats here will
>> > lead
>> > > to
>> > > > a
>> > > > bit of a tower of babel.
>> > > > 2. This proposal still includes a pretty big change to our
>> > > serialization
>> > > > and protocol definition layer. Essentially it is introducing an
>> > > optional
>> > > > type, where the format is data dependent. I think this is
>> > actually a
>> > > big
>> > > > change though it doesn't seem like it. It means you can no
>> > longer
>> > > > specify
>> > > > this type with our type definition DSL, and likewise it requires
>> > > custom
>> > > > handling in client libs. This isn't a huge thing, since the
>> > Record
>> > > > definition is custom anyway, but I think this kind of protocol
>> > > > inconsistency is very non-desirable and ties you to hand-coding
>> > > things.
>> > > > I
>> > > > think the type should instead by [Key Value] in our BNF, where
>> > key and
>> > > > value are both short strings as used elsewhere. This brings it
>> > in line
>> > > > with
>> > > > the rest of the protocol.
>> > > > 3. Could we get more specific about the exact Java API change to
>> > > > ProducerRecord, ConsumerRecord, Record, etc?
>> > > >
>> > > > -Jay
>> > > >
>> > >
>> >
>> >
>> > The information contained in this email is strictly confidential and for
>> > the use of the addressee only, unless otherwise indicated. If you are not
>> > the intended recipient, please do not read, copy, use or disclose to others
>> > this message or any attachment. Please also notify the sender by replying
>> > to this email or by telephone (+44(020 7896 0011) and then delete the email
>> > and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> > official business of this company shall be understood as neither given nor
>> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > registered in England and Wales, company number 04008957) and IG Index
>> > Limited (a company registered in England and Wales, company number
>> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> > Index Limited (register number 114059) are authorised and regulated by the
>> > Financial Conduct Authority.
>> >
--
Nacho - Ignacio Solis - isolis@igso.net
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Colin McCabe <cm...@apache.org>.
+1 for varints here-- it would save quite a bit of space. They are
pretty quick to implement as well.
I think it makes sense for values to be byte arrays. Users might want
to attach arbitrary payloads; they shouldn't be forced to serialize
everything to Java strings.
best,
Colin
On Thu, Feb 16, 2017, at 09:52, Jason Gustafson wrote:
> Hey Michael,
>
> Hmm, I guess the point of representing it as bytes is to allow the broker
> to pass it through opaquely? Is the cost of parsing them a concern, or
> are
> we simply trying to ensure that the broker stays agnostic to the format?
>
> On varints, I think adding support for them makes less sense for an
> isolated use case, but as part of a more holistic change (such as what we
> have proposed in KIP-98), I think they are justifiable. If we add them,
> then the need to use attributes becomes quite a bit weaker, right? The
> other thing I find slightly odd is the fact that null headers has no
> actual
> semantic meaning for the message (unlike null keys and values). It is
> just
> a space optimization. It seems a bit better to always use size 0 to
> indicate having no headers.
>
> Overall, the main point is ensuring that the message schema remains
> consistent, either within the larger protocol, or at a minimum within the
> message itself.
>
> -Jason
>
> On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jason,
> >
> > On point 1) in the message protocol the headers are simply a byte array,
> > as like the key or value, this is to clearly demarcate the header in the
> > core message. Then the header byte array in the core message is an array of
> > key, value pairs. This is what it is denoting.
> >
> > Then this would be I guess in the given notation:
> >
> > Headers => [KeyLength, Key, ValueLength, Value]
> > KeyLength => int32 <-----------------NEW size of the byte[] of the
> > serialised key value
> > Key => bytes <---------------------- NEW serialised string (UTF8)
> > bytes of the header key
> > ValueLength => int32 <-------------- NEW size of the byte[] of the
> > serialised header value
> > Value => bytes <-------------------- NEW serialised form of the header
> > value
> >
> > The key length and value length is matching the way the protocol is
> > defined in the core message currently.
> >
> >
> >
> >
> > On point 2)
> > Var sized ints, this was discussed much earlier on, in fact I had
> > suggested it myself (with Hadoop references), the complexity of this
> > compared to having a simpler protocol was argued and agreed it wasn’t worth
> > the complexity as all other clients in other languages would need to ensure
> > theyre using the right var size algorithm, as there is a few.
> >
> > On point 3)
> > We did the attributes, optional approach as originally there was marked
> > concern that headers would cause a message size overhead for others, who
> > don’t want them. As such this is the clean solution to achieve that. If
> > that no longer holds, and we don’t care that we add 4bytes overhead, then
> > im happy to remove.
> >
> > I’m personally in favour of keeping the message as small as possible so
> > people don’t get shocks in perf and throughputs dues to message size,
> > unless they actively use the feature, as such I do prefer the attribute bit
> > wise feature flag approach myself.
> >
> >
> >
> >
> > On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
> >
> > We have proposed a few significant changes to the message format in
> > KIP-98
> > which now seems likely to pass (perhaps with some iterations on
> > implementation details). It would be good to try and coordinate the
> > changes
> > in both of the proposals to make sure they are consistent and
> > compatible.
> >
> > I think using the attributes to indicate null headers is a reasonable
> > approach. We have proposed to do the same thing for the message key and
> > value. That said, I sympathize with Jay's argument. Having multiple
> > ways to
> > specify a null value increases the overall complexity of the protocol.
> > You
> > can see this just from the fact that you need the extra verbiage in the
> > protocol specification in this KIP and in KIP-98 to describe the
> > dependence
> > between the fields and the attributes. It seems like a slippery slope
> > if
> > you start allowing different request types to implement the protocol
> > specification differently.
> >
> > You can also argue that the messages already are and are likely to
> > remain a
> > special case. For example, there is currently no generality in how
> > compressed message sets are represented that would be applicable for
> > other
> > request types. Some might see this divergence as an unfortunate
> > protocol
> > deficiency which should be fixed; others might see it as sort of the
> > inevitability of needing to optimize where it counts most. I'm probably
> > somewhere in between, but I think we probably all share the intuition
> > that
> > the protocol should be kept as consistent as possible. With that in
> > mind,
> > here are a few comments:
> >
> > 1. One thing I found a little odd when reading the current proposal is
> > that
> > the headers are both represented as an array of bytes and as an array
> > of
> > key/value pairs. I'd probably suggest something like this:
> >
> > Headers => [HeaderKey HeaderValue]
> > HeaderKey => String
> > HeaderValue => Bytes
> >
> > An array in the Kafka protocol is represented as a 4-byte integer
> > indicating the number of elements in the array followed by the
> > serialization of the elements. Unless I'm misunderstanding, what you
> > have
> > instead is the total size of the headers in bytes followed by the
> > elements.
> > I'm not sure I see any reason for this inconsistency.
> >
> > 2. In KIP-98, we've introduced variable-length integer fields.
> > Effectively,
> > we've enriched (or "complicated" as Jay might say ;) the protocol
> > specification to include the following types: VarInt, VarLong,
> > UnsignedVarInt and UnsignedVarLong.
> >
> > Along with these primitives, we could introduce the following types:
> >
> > VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> > NumberOfItems => UnsignedVarInt
> >
> > VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
> > NumberOfItemsOrNull => VarInt (-1 means null)
> >
> > And similarly for the `String` and `Bytes` types. These types can save
> > a
> > considerable amount of space in this proposal because they can be used
> > for
> > both the number of headers included in the message and the lengths of
> > the
> > header keys and values. We could do this instead:
> >
> > Headers => VarSizeArray[HeaderKey HeaderValue]
> > HeaderKey => VarSizeString
> > HeaderValue => VarSizeBytes
> >
> > Combining the savings from the use of variable length fields, the
> > benefit
> > of using the attributes to represent null seems pretty small.
> >
> > 3. Whichever way we go (whether we use the attributes or not), we
> > should at
> > least be consistent between this KIP and KIP-98. It would be very
> > strange
> > to have two ways to represent null values in the same schema. Either
> > way is
> > OK with me. I think some message-level optimizations are justifiable,
> > but
> > the savings here seem minimal (a few bytes per message), so maybe it's
> > not
> > worth the cost of letting the message diverge even further from the
> > rest of
> > the protocol.
> >
> > -Jason
> >
> >
> > On Wed, Feb 15, 2017 at 8:52 AM, radai <ra...@gmail.com>
> > wrote:
> >
> > > I've trimmed the inline contents as this mail is getting too big for
> > the
> > > apache mailing list software to deliver :-(
> > >
> > > 1. the important thing for interoperability is for different
> > "interested
> > > parties" (plugins, infra layers/wrappers, user-code) to be able to
> > stick
> > > pieces of metadata onto msgs without getting in each other's way. a
> > common
> > > key scheme (Strings, as of the time of this writing?) is all thats
> > required
> > > for that. it is assumed that the other end interested in any such
> > piece of
> > > metadata knows the encoding, and byte[] provides for the most
> > flexibility.
> > > i believe this is the same logic behind core kafka being
> > byte[]/byte[] -
> > > Strings are more "usable" but bytes are flexible and so were chosen.
> > > Also - core kafka doesnt even do that good of a job on usability of
> > the
> > > payload (example - i have to specify the nop byte[] "decoders"
> > explicitly
> > > in conf), and again sacrificies usability for the sake of
> > performance (no
> > > convenient single-record processing as poll is a batch, lots of
> > obscure
> > > little config details exposing internals of the batching mechanism,
> > etc)
> > >
> > > this is also why i really dislike the idea of a "type system" for
> > header
> > > values, it further degrades the usability, adds complexity and will
> > > eventually get in people's way, also, it would be the 2nd/3rd
> > home-group
> > > serialization mechanism in core kafka (counting 2 iterations of the
> > "type
> > > definition DSL")
> > >
> > > 2. this is an implementation detail, and not even a very "user
> > facing" one?
> > > to the best of my understanding the vote process is on proposed
> > > API/behaviour. also - since we're willing to go with strings just
> > serialize
> > > a 0-sized header blob and IIUC you dont need any optionals anymore.
> > >
> > > 3. yes, we can :-)
> > >
> > > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > > > Hi Jay,
> > > >
> > > > 1) There was some initial debate on the value part, as youll note
> > String,
> > > > String headers were discounted early on. The reason for this is
> > > flexibility
> > > > and keeping in line with the flexibility of key, value of the
> > message
> > > > object itself. I don’t think it takes away from an ecosystem as
> > each
> > > plugin
> > > > will care for their own key, this way ints, booleans , exotic
> > custom
> > > binary
> > > > can all be catered for=.
> > > > a. If you really wanted to push for a typed value interface, I
> > wouldn’t
> > > > want just String values supported, but the the primatives plus
> > string and
> > > > also still keeping the ability to have a binary for custom
> > binaries that
> > > > some organisations may have.
> > > > i. I have written this slight alternative here,
> > > https://cwiki.apache.org/
> > > > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
> > > > ii. Essentially the value bytes, has a leading byte overhead.
> > > > 1. This tells you what type the value is, before reading the rest
> > of the
> > > > bytes, allowing serialisation/deserialization to and from the
> > primitives,
> > > > string and byte[]. This is akin to some other messaging systems.
> > > > 2) We are making it optional, so that for those not wanting
> > headers have
> > > 0
> > > > bytes overhead (think of it as a feature flag), I don’t think this
> > is
> > > > complex, especially if comparing to changes proposed in other kips
> > like
> > > > kip-98.
> > > > a. If you really really don’t like this, we can drop it, but it
> > would
> > > mean
> > > > buying into 4 bytes extra overhead for users who do not want to use
> > > headers.
> > > > 3) In the summary yes, it is at a higher level, but I think this
> > is well
> > > > documented in the proposed changes section.
> > > > a. Added getHeaders method to Producer/Consumer record (that is it)
> > > > b. We’ve also detailed the new Headers class that this method
> > returns
> > > that
> > > > encapsulates the headers protocol and logic.
> > > >
> > > > Best,
> > > > Mike
> > > >
> > > > ==Original questions from the vote thread from Jay.==
> > > >
> > > > Couple of things I think we still need to work out:
> > > >
> > > > 1. I think we agree about the key, but I think we haven't
> > talked about
> > > > the value yet. I think if our goal is an open ecosystem of these
> > > header
> > > > spread across many plugins from many systems we should consider
> > making
> > > > this
> > > > a string as well so it can be printed, set via a UI, set in
> > config,
> > > etc.
> > > > Basically encouraging pluggable serialization formats here will
> > lead
> > > to
> > > > a
> > > > bit of a tower of babel.
> > > > 2. This proposal still includes a pretty big change to our
> > > serialization
> > > > and protocol definition layer. Essentially it is introducing an
> > > optional
> > > > type, where the format is data dependent. I think this is
> > actually a
> > > big
> > > > change though it doesn't seem like it. It means you can no
> > longer
> > > > specify
> > > > this type with our type definition DSL, and likewise it requires
> > > custom
> > > > handling in client libs. This isn't a huge thing, since the
> > Record
> > > > definition is custom anyway, but I think this kind of protocol
> > > > inconsistency is very non-desirable and ties you to hand-coding
> > > things.
> > > > I
> > > > think the type should instead by [Key Value] in our BNF, where
> > key and
> > > > value are both short strings as used elsewhere. This brings it
> > in line
> > > > with
> > > > the rest of the protocol.
> > > > 3. Could we get more specific about the exact Java API change to
> > > > ProducerRecord, ConsumerRecord, Record, etc?
> > > >
> > > > -Jay
> > > >
> > >
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to the
> > official business of this company shall be understood as neither given nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by the
> > Financial Conduct Authority.
> >
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
Hey Michael,
Hmm, I guess the point of representing it as bytes is to allow the broker
to pass it through opaquely? Is the cost of parsing them a concern, or are
we simply trying to ensure that the broker stays agnostic to the format?
On varints, I think adding support for them makes less sense for an
isolated use case, but as part of a more holistic change (such as what we
have proposed in KIP-98), I think they are justifiable. If we add them,
then the need to use attributes becomes quite a bit weaker, right? The
other thing I find slightly odd is the fact that null headers has no actual
semantic meaning for the message (unlike null keys and values). It is just
a space optimization. It seems a bit better to always use size 0 to
indicate having no headers.
Overall, the main point is ensuring that the message schema remains
consistent, either within the larger protocol, or at a minimum within the
message itself.
-Jason
On Thu, Feb 16, 2017 at 6:39 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jason,
>
> On point 1) in the message protocol the headers are simply a byte array,
> as like the key or value, this is to clearly demarcate the header in the
> core message. Then the header byte array in the core message is an array of
> key, value pairs. This is what it is denoting.
>
> Then this would be I guess in the given notation:
>
> Headers => [KeyLength, Key, ValueLength, Value]
> KeyLength => int32 <-----------------NEW size of the byte[] of the
> serialised key value
> Key => bytes <---------------------- NEW serialised string (UTF8)
> bytes of the header key
> ValueLength => int32 <-------------- NEW size of the byte[] of the
> serialised header value
> Value => bytes <-------------------- NEW serialised form of the header
> value
>
> The key length and value length is matching the way the protocol is
> defined in the core message currently.
>
>
>
>
> On point 2)
> Var sized ints, this was discussed much earlier on, in fact I had
> suggested it myself (with Hadoop references), the complexity of this
> compared to having a simpler protocol was argued and agreed it wasn’t worth
> the complexity as all other clients in other languages would need to ensure
> theyre using the right var size algorithm, as there is a few.
>
> On point 3)
> We did the attributes, optional approach as originally there was marked
> concern that headers would cause a message size overhead for others, who
> don’t want them. As such this is the clean solution to achieve that. If
> that no longer holds, and we don’t care that we add 4bytes overhead, then
> im happy to remove.
>
> I’m personally in favour of keeping the message as small as possible so
> people don’t get shocks in perf and throughputs dues to message size,
> unless they actively use the feature, as such I do prefer the attribute bit
> wise feature flag approach myself.
>
>
>
>
> On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
>
> We have proposed a few significant changes to the message format in
> KIP-98
> which now seems likely to pass (perhaps with some iterations on
> implementation details). It would be good to try and coordinate the
> changes
> in both of the proposals to make sure they are consistent and
> compatible.
>
> I think using the attributes to indicate null headers is a reasonable
> approach. We have proposed to do the same thing for the message key and
> value. That said, I sympathize with Jay's argument. Having multiple
> ways to
> specify a null value increases the overall complexity of the protocol.
> You
> can see this just from the fact that you need the extra verbiage in the
> protocol specification in this KIP and in KIP-98 to describe the
> dependence
> between the fields and the attributes. It seems like a slippery slope
> if
> you start allowing different request types to implement the protocol
> specification differently.
>
> You can also argue that the messages already are and are likely to
> remain a
> special case. For example, there is currently no generality in how
> compressed message sets are represented that would be applicable for
> other
> request types. Some might see this divergence as an unfortunate
> protocol
> deficiency which should be fixed; others might see it as sort of the
> inevitability of needing to optimize where it counts most. I'm probably
> somewhere in between, but I think we probably all share the intuition
> that
> the protocol should be kept as consistent as possible. With that in
> mind,
> here are a few comments:
>
> 1. One thing I found a little odd when reading the current proposal is
> that
> the headers are both represented as an array of bytes and as an array
> of
> key/value pairs. I'd probably suggest something like this:
>
> Headers => [HeaderKey HeaderValue]
> HeaderKey => String
> HeaderValue => Bytes
>
> An array in the Kafka protocol is represented as a 4-byte integer
> indicating the number of elements in the array followed by the
> serialization of the elements. Unless I'm misunderstanding, what you
> have
> instead is the total size of the headers in bytes followed by the
> elements.
> I'm not sure I see any reason for this inconsistency.
>
> 2. In KIP-98, we've introduced variable-length integer fields.
> Effectively,
> we've enriched (or "complicated" as Jay might say ;) the protocol
> specification to include the following types: VarInt, VarLong,
> UnsignedVarInt and UnsignedVarLong.
>
> Along with these primitives, we could introduce the following types:
>
> VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
> NumberOfItems => UnsignedVarInt
>
> VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
> NumberOfItemsOrNull => VarInt (-1 means null)
>
> And similarly for the `String` and `Bytes` types. These types can save
> a
> considerable amount of space in this proposal because they can be used
> for
> both the number of headers included in the message and the lengths of
> the
> header keys and values. We could do this instead:
>
> Headers => VarSizeArray[HeaderKey HeaderValue]
> HeaderKey => VarSizeString
> HeaderValue => VarSizeBytes
>
> Combining the savings from the use of variable length fields, the
> benefit
> of using the attributes to represent null seems pretty small.
>
> 3. Whichever way we go (whether we use the attributes or not), we
> should at
> least be consistent between this KIP and KIP-98. It would be very
> strange
> to have two ways to represent null values in the same schema. Either
> way is
> OK with me. I think some message-level optimizations are justifiable,
> but
> the savings here seem minimal (a few bytes per message), so maybe it's
> not
> worth the cost of letting the message diverge even further from the
> rest of
> the protocol.
>
> -Jason
>
>
> On Wed, Feb 15, 2017 at 8:52 AM, radai <ra...@gmail.com>
> wrote:
>
> > I've trimmed the inline contents as this mail is getting too big for
> the
> > apache mailing list software to deliver :-(
> >
> > 1. the important thing for interoperability is for different
> "interested
> > parties" (plugins, infra layers/wrappers, user-code) to be able to
> stick
> > pieces of metadata onto msgs without getting in each other's way. a
> common
> > key scheme (Strings, as of the time of this writing?) is all thats
> required
> > for that. it is assumed that the other end interested in any such
> piece of
> > metadata knows the encoding, and byte[] provides for the most
> flexibility.
> > i believe this is the same logic behind core kafka being
> byte[]/byte[] -
> > Strings are more "usable" but bytes are flexible and so were chosen.
> > Also - core kafka doesnt even do that good of a job on usability of
> the
> > payload (example - i have to specify the nop byte[] "decoders"
> explicitly
> > in conf), and again sacrificies usability for the sake of
> performance (no
> > convenient single-record processing as poll is a batch, lots of
> obscure
> > little config details exposing internals of the batching mechanism,
> etc)
> >
> > this is also why i really dislike the idea of a "type system" for
> header
> > values, it further degrades the usability, adds complexity and will
> > eventually get in people's way, also, it would be the 2nd/3rd
> home-group
> > serialization mechanism in core kafka (counting 2 iterations of the
> "type
> > definition DSL")
> >
> > 2. this is an implementation detail, and not even a very "user
> facing" one?
> > to the best of my understanding the vote process is on proposed
> > API/behaviour. also - since we're willing to go with strings just
> serialize
> > a 0-sized header blob and IIUC you dont need any optionals anymore.
> >
> > 3. yes, we can :-)
> >
> > On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > wrote:
> >
> > > Hi Jay,
> > >
> > > 1) There was some initial debate on the value part, as youll note
> String,
> > > String headers were discounted early on. The reason for this is
> > flexibility
> > > and keeping in line with the flexibility of key, value of the
> message
> > > object itself. I don’t think it takes away from an ecosystem as
> each
> > plugin
> > > will care for their own key, this way ints, booleans , exotic
> custom
> > binary
> > > can all be catered for=.
> > > a. If you really wanted to push for a typed value interface, I
> wouldn’t
> > > want just String values supported, but the the primatives plus
> string and
> > > also still keeping the ability to have a binary for custom
> binaries that
> > > some organisations may have.
> > > i. I have written this slight alternative here,
> > https://cwiki.apache.org/
> > > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
> > > ii. Essentially the value bytes, has a leading byte overhead.
> > > 1. This tells you what type the value is, before reading the rest
> of the
> > > bytes, allowing serialisation/deserialization to and from the
> primitives,
> > > string and byte[]. This is akin to some other messaging systems.
> > > 2) We are making it optional, so that for those not wanting
> headers have
> > 0
> > > bytes overhead (think of it as a feature flag), I don’t think this
> is
> > > complex, especially if comparing to changes proposed in other kips
> like
> > > kip-98.
> > > a. If you really really don’t like this, we can drop it, but it
> would
> > mean
> > > buying into 4 bytes extra overhead for users who do not want to use
> > headers.
> > > 3) In the summary yes, it is at a higher level, but I think this
> is well
> > > documented in the proposed changes section.
> > > a. Added getHeaders method to Producer/Consumer record (that is it)
> > > b. We’ve also detailed the new Headers class that this method
> returns
> > that
> > > encapsulates the headers protocol and logic.
> > >
> > > Best,
> > > Mike
> > >
> > > ==Original questions from the vote thread from Jay.==
> > >
> > > Couple of things I think we still need to work out:
> > >
> > > 1. I think we agree about the key, but I think we haven't
> talked about
> > > the value yet. I think if our goal is an open ecosystem of these
> > header
> > > spread across many plugins from many systems we should consider
> making
> > > this
> > > a string as well so it can be printed, set via a UI, set in
> config,
> > etc.
> > > Basically encouraging pluggable serialization formats here will
> lead
> > to
> > > a
> > > bit of a tower of babel.
> > > 2. This proposal still includes a pretty big change to our
> > serialization
> > > and protocol definition layer. Essentially it is introducing an
> > optional
> > > type, where the format is data dependent. I think this is
> actually a
> > big
> > > change though it doesn't seem like it. It means you can no
> longer
> > > specify
> > > this type with our type definition DSL, and likewise it requires
> > custom
> > > handling in client libs. This isn't a huge thing, since the
> Record
> > > definition is custom anyway, but I think this kind of protocol
> > > inconsistency is very non-desirable and ties you to hand-coding
> > things.
> > > I
> > > think the type should instead by [Key Value] in our BNF, where
> key and
> > > value are both short strings as used elsewhere. This brings it
> in line
> > > with
> > > the rest of the protocol.
> > > 3. Could we get more specific about the exact Java API change to
> > > ProducerRecord, ConsumerRecord, Record, etc?
> > >
> > > -Jay
> > >
> >
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Jason,
On point 1) in the message protocol the headers are simply a byte array, as like the key or value, this is to clearly demarcate the header in the core message. Then the header byte array in the core message is an array of key, value pairs. This is what it is denoting.
Then this would be I guess in the given notation:
Headers => [KeyLength, Key, ValueLength, Value]
KeyLength => int32 <-----------------NEW size of the byte[] of the serialised key value
Key => bytes <---------------------- NEW serialised string (UTF8) bytes of the header key
ValueLength => int32 <-------------- NEW size of the byte[] of the serialised header value
Value => bytes <-------------------- NEW serialised form of the header value
The key length and value length is matching the way the protocol is defined in the core message currently.
On point 2)
Var sized ints, this was discussed much earlier on, in fact I had suggested it myself (with Hadoop references), the complexity of this compared to having a simpler protocol was argued and agreed it wasn’t worth the complexity as all other clients in other languages would need to ensure theyre using the right var size algorithm, as there is a few.
On point 3)
We did the attributes, optional approach as originally there was marked concern that headers would cause a message size overhead for others, who don’t want them. As such this is the clean solution to achieve that. If that no longer holds, and we don’t care that we add 4bytes overhead, then im happy to remove.
I’m personally in favour of keeping the message as small as possible so people don’t get shocks in perf and throughputs dues to message size, unless they actively use the feature, as such I do prefer the attribute bit wise feature flag approach myself.
On 16/02/2017, 05:40, "Jason Gustafson" <ja...@confluent.io> wrote:
We have proposed a few significant changes to the message format in KIP-98
which now seems likely to pass (perhaps with some iterations on
implementation details). It would be good to try and coordinate the changes
in both of the proposals to make sure they are consistent and compatible.
I think using the attributes to indicate null headers is a reasonable
approach. We have proposed to do the same thing for the message key and
value. That said, I sympathize with Jay's argument. Having multiple ways to
specify a null value increases the overall complexity of the protocol. You
can see this just from the fact that you need the extra verbiage in the
protocol specification in this KIP and in KIP-98 to describe the dependence
between the fields and the attributes. It seems like a slippery slope if
you start allowing different request types to implement the protocol
specification differently.
You can also argue that the messages already are and are likely to remain a
special case. For example, there is currently no generality in how
compressed message sets are represented that would be applicable for other
request types. Some might see this divergence as an unfortunate protocol
deficiency which should be fixed; others might see it as sort of the
inevitability of needing to optimize where it counts most. I'm probably
somewhere in between, but I think we probably all share the intuition that
the protocol should be kept as consistent as possible. With that in mind,
here are a few comments:
1. One thing I found a little odd when reading the current proposal is that
the headers are both represented as an array of bytes and as an array of
key/value pairs. I'd probably suggest something like this:
Headers => [HeaderKey HeaderValue]
HeaderKey => String
HeaderValue => Bytes
An array in the Kafka protocol is represented as a 4-byte integer
indicating the number of elements in the array followed by the
serialization of the elements. Unless I'm misunderstanding, what you have
instead is the total size of the headers in bytes followed by the elements.
I'm not sure I see any reason for this inconsistency.
2. In KIP-98, we've introduced variable-length integer fields. Effectively,
we've enriched (or "complicated" as Jay might say ;) the protocol
specification to include the following types: VarInt, VarLong,
UnsignedVarInt and UnsignedVarLong.
Along with these primitives, we could introduce the following types:
VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
NumberOfItems => UnsignedVarInt
VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
NumberOfItemsOrNull => VarInt (-1 means null)
And similarly for the `String` and `Bytes` types. These types can save a
considerable amount of space in this proposal because they can be used for
both the number of headers included in the message and the lengths of the
header keys and values. We could do this instead:
Headers => VarSizeArray[HeaderKey HeaderValue]
HeaderKey => VarSizeString
HeaderValue => VarSizeBytes
Combining the savings from the use of variable length fields, the benefit
of using the attributes to represent null seems pretty small.
3. Whichever way we go (whether we use the attributes or not), we should at
least be consistent between this KIP and KIP-98. It would be very strange
to have two ways to represent null values in the same schema. Either way is
OK with me. I think some message-level optimizations are justifiable, but
the savings here seem minimal (a few bytes per message), so maybe it's not
worth the cost of letting the message diverge even further from the rest of
the protocol.
-Jason
On Wed, Feb 15, 2017 at 8:52 AM, radai <ra...@gmail.com> wrote:
> I've trimmed the inline contents as this mail is getting too big for the
> apache mailing list software to deliver :-(
>
> 1. the important thing for interoperability is for different "interested
> parties" (plugins, infra layers/wrappers, user-code) to be able to stick
> pieces of metadata onto msgs without getting in each other's way. a common
> key scheme (Strings, as of the time of this writing?) is all thats required
> for that. it is assumed that the other end interested in any such piece of
> metadata knows the encoding, and byte[] provides for the most flexibility.
> i believe this is the same logic behind core kafka being byte[]/byte[] -
> Strings are more "usable" but bytes are flexible and so were chosen.
> Also - core kafka doesnt even do that good of a job on usability of the
> payload (example - i have to specify the nop byte[] "decoders" explicitly
> in conf), and again sacrificies usability for the sake of performance (no
> convenient single-record processing as poll is a batch, lots of obscure
> little config details exposing internals of the batching mechanism, etc)
>
> this is also why i really dislike the idea of a "type system" for header
> values, it further degrades the usability, adds complexity and will
> eventually get in people's way, also, it would be the 2nd/3rd home-group
> serialization mechanism in core kafka (counting 2 iterations of the "type
> definition DSL")
>
> 2. this is an implementation detail, and not even a very "user facing" one?
> to the best of my understanding the vote process is on proposed
> API/behaviour. also - since we're willing to go with strings just serialize
> a 0-sized header blob and IIUC you dont need any optionals anymore.
>
> 3. yes, we can :-)
>
> On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay,
> >
> > 1) There was some initial debate on the value part, as youll note String,
> > String headers were discounted early on. The reason for this is
> flexibility
> > and keeping in line with the flexibility of key, value of the message
> > object itself. I don’t think it takes away from an ecosystem as each
> plugin
> > will care for their own key, this way ints, booleans , exotic custom
> binary
> > can all be catered for=.
> > a. If you really wanted to push for a typed value interface, I wouldn’t
> > want just String values supported, but the the primatives plus string and
> > also still keeping the ability to have a binary for custom binaries that
> > some organisations may have.
> > i. I have written this slight alternative here,
> https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
> > ii. Essentially the value bytes, has a leading byte overhead.
> > 1. This tells you what type the value is, before reading the rest of the
> > bytes, allowing serialisation/deserialization to and from the primitives,
> > string and byte[]. This is akin to some other messaging systems.
> > 2) We are making it optional, so that for those not wanting headers have
> 0
> > bytes overhead (think of it as a feature flag), I don’t think this is
> > complex, especially if comparing to changes proposed in other kips like
> > kip-98.
> > a. If you really really don’t like this, we can drop it, but it would
> mean
> > buying into 4 bytes extra overhead for users who do not want to use
> headers.
> > 3) In the summary yes, it is at a higher level, but I think this is well
> > documented in the proposed changes section.
> > a. Added getHeaders method to Producer/Consumer record (that is it)
> > b. We’ve also detailed the new Headers class that this method returns
> that
> > encapsulates the headers protocol and logic.
> >
> > Best,
> > Mike
> >
> > ==Original questions from the vote thread from Jay.==
> >
> > Couple of things I think we still need to work out:
> >
> > 1. I think we agree about the key, but I think we haven't talked about
> > the value yet. I think if our goal is an open ecosystem of these
> header
> > spread across many plugins from many systems we should consider making
> > this
> > a string as well so it can be printed, set via a UI, set in config,
> etc.
> > Basically encouraging pluggable serialization formats here will lead
> to
> > a
> > bit of a tower of babel.
> > 2. This proposal still includes a pretty big change to our
> serialization
> > and protocol definition layer. Essentially it is introducing an
> optional
> > type, where the format is data dependent. I think this is actually a
> big
> > change though it doesn't seem like it. It means you can no longer
> > specify
> > this type with our type definition DSL, and likewise it requires
> custom
> > handling in client libs. This isn't a huge thing, since the Record
> > definition is custom anyway, but I think this kind of protocol
> > inconsistency is very non-desirable and ties you to hand-coding
> things.
> > I
> > think the type should instead by [Key Value] in our BNF, where key and
> > value are both short strings as used elsewhere. This brings it in line
> > with
> > the rest of the protocol.
> > 3. Could we get more specific about the exact Java API change to
> > ProducerRecord, ConsumerRecord, Record, etc?
> >
> > -Jay
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jason Gustafson <ja...@confluent.io>.
We have proposed a few significant changes to the message format in KIP-98
which now seems likely to pass (perhaps with some iterations on
implementation details). It would be good to try and coordinate the changes
in both of the proposals to make sure they are consistent and compatible.
I think using the attributes to indicate null headers is a reasonable
approach. We have proposed to do the same thing for the message key and
value. That said, I sympathize with Jay's argument. Having multiple ways to
specify a null value increases the overall complexity of the protocol. You
can see this just from the fact that you need the extra verbiage in the
protocol specification in this KIP and in KIP-98 to describe the dependence
between the fields and the attributes. It seems like a slippery slope if
you start allowing different request types to implement the protocol
specification differently.
You can also argue that the messages already are and are likely to remain a
special case. For example, there is currently no generality in how
compressed message sets are represented that would be applicable for other
request types. Some might see this divergence as an unfortunate protocol
deficiency which should be fixed; others might see it as sort of the
inevitability of needing to optimize where it counts most. I'm probably
somewhere in between, but I think we probably all share the intuition that
the protocol should be kept as consistent as possible. With that in mind,
here are a few comments:
1. One thing I found a little odd when reading the current proposal is that
the headers are both represented as an array of bytes and as an array of
key/value pairs. I'd probably suggest something like this:
Headers => [HeaderKey HeaderValue]
HeaderKey => String
HeaderValue => Bytes
An array in the Kafka protocol is represented as a 4-byte integer
indicating the number of elements in the array followed by the
serialization of the elements. Unless I'm misunderstanding, what you have
instead is the total size of the headers in bytes followed by the elements.
I'm not sure I see any reason for this inconsistency.
2. In KIP-98, we've introduced variable-length integer fields. Effectively,
we've enriched (or "complicated" as Jay might say ;) the protocol
specification to include the following types: VarInt, VarLong,
UnsignedVarInt and UnsignedVarLong.
Along with these primitives, we could introduce the following types:
VarSizeArray => NumberOfItems Item1 Item2 .. ItemN
NumberOfItems => UnsignedVarInt
VarSizeNullableArray => NumberOfItemsOrNull Item1 Item2 .. ItemN
NumberOfItemsOrNull => VarInt (-1 means null)
And similarly for the `String` and `Bytes` types. These types can save a
considerable amount of space in this proposal because they can be used for
both the number of headers included in the message and the lengths of the
header keys and values. We could do this instead:
Headers => VarSizeArray[HeaderKey HeaderValue]
HeaderKey => VarSizeString
HeaderValue => VarSizeBytes
Combining the savings from the use of variable length fields, the benefit
of using the attributes to represent null seems pretty small.
3. Whichever way we go (whether we use the attributes or not), we should at
least be consistent between this KIP and KIP-98. It would be very strange
to have two ways to represent null values in the same schema. Either way is
OK with me. I think some message-level optimizations are justifiable, but
the savings here seem minimal (a few bytes per message), so maybe it's not
worth the cost of letting the message diverge even further from the rest of
the protocol.
-Jason
On Wed, Feb 15, 2017 at 8:52 AM, radai <ra...@gmail.com> wrote:
> I've trimmed the inline contents as this mail is getting too big for the
> apache mailing list software to deliver :-(
>
> 1. the important thing for interoperability is for different "interested
> parties" (plugins, infra layers/wrappers, user-code) to be able to stick
> pieces of metadata onto msgs without getting in each other's way. a common
> key scheme (Strings, as of the time of this writing?) is all thats required
> for that. it is assumed that the other end interested in any such piece of
> metadata knows the encoding, and byte[] provides for the most flexibility.
> i believe this is the same logic behind core kafka being byte[]/byte[] -
> Strings are more "usable" but bytes are flexible and so were chosen.
> Also - core kafka doesnt even do that good of a job on usability of the
> payload (example - i have to specify the nop byte[] "decoders" explicitly
> in conf), and again sacrificies usability for the sake of performance (no
> convenient single-record processing as poll is a batch, lots of obscure
> little config details exposing internals of the batching mechanism, etc)
>
> this is also why i really dislike the idea of a "type system" for header
> values, it further degrades the usability, adds complexity and will
> eventually get in people's way, also, it would be the 2nd/3rd home-group
> serialization mechanism in core kafka (counting 2 iterations of the "type
> definition DSL")
>
> 2. this is an implementation detail, and not even a very "user facing" one?
> to the best of my understanding the vote process is on proposed
> API/behaviour. also - since we're willing to go with strings just serialize
> a 0-sized header blob and IIUC you dont need any optionals anymore.
>
> 3. yes, we can :-)
>
> On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay,
> >
> > 1) There was some initial debate on the value part, as youll note String,
> > String headers were discounted early on. The reason for this is
> flexibility
> > and keeping in line with the flexibility of key, value of the message
> > object itself. I don’t think it takes away from an ecosystem as each
> plugin
> > will care for their own key, this way ints, booleans , exotic custom
> binary
> > can all be catered for=.
> > a. If you really wanted to push for a typed value interface, I wouldn’t
> > want just String values supported, but the the primatives plus string and
> > also still keeping the ability to have a binary for custom binaries that
> > some organisations may have.
> > i. I have written this slight alternative here,
> https://cwiki.apache.org/
> > confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
> > ii. Essentially the value bytes, has a leading byte overhead.
> > 1. This tells you what type the value is, before reading the rest of the
> > bytes, allowing serialisation/deserialization to and from the primitives,
> > string and byte[]. This is akin to some other messaging systems.
> > 2) We are making it optional, so that for those not wanting headers have
> 0
> > bytes overhead (think of it as a feature flag), I don’t think this is
> > complex, especially if comparing to changes proposed in other kips like
> > kip-98.
> > a. If you really really don’t like this, we can drop it, but it would
> mean
> > buying into 4 bytes extra overhead for users who do not want to use
> headers.
> > 3) In the summary yes, it is at a higher level, but I think this is well
> > documented in the proposed changes section.
> > a. Added getHeaders method to Producer/Consumer record (that is it)
> > b. We’ve also detailed the new Headers class that this method returns
> that
> > encapsulates the headers protocol and logic.
> >
> > Best,
> > Mike
> >
> > ==Original questions from the vote thread from Jay.==
> >
> > Couple of things I think we still need to work out:
> >
> > 1. I think we agree about the key, but I think we haven't talked about
> > the value yet. I think if our goal is an open ecosystem of these
> header
> > spread across many plugins from many systems we should consider making
> > this
> > a string as well so it can be printed, set via a UI, set in config,
> etc.
> > Basically encouraging pluggable serialization formats here will lead
> to
> > a
> > bit of a tower of babel.
> > 2. This proposal still includes a pretty big change to our
> serialization
> > and protocol definition layer. Essentially it is introducing an
> optional
> > type, where the format is data dependent. I think this is actually a
> big
> > change though it doesn't seem like it. It means you can no longer
> > specify
> > this type with our type definition DSL, and likewise it requires
> custom
> > handling in client libs. This isn't a huge thing, since the Record
> > definition is custom anyway, but I think this kind of protocol
> > inconsistency is very non-desirable and ties you to hand-coding
> things.
> > I
> > think the type should instead by [Key Value] in our BNF, where key and
> > value are both short strings as used elsewhere. This brings it in line
> > with
> > the rest of the protocol.
> > 3. Could we get more specific about the exact Java API change to
> > ProducerRecord, ConsumerRecord, Record, etc?
> >
> > -Jay
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
I've trimmed the inline contents as this mail is getting too big for the
apache mailing list software to deliver :-(
1. the important thing for interoperability is for different "interested
parties" (plugins, infra layers/wrappers, user-code) to be able to stick
pieces of metadata onto msgs without getting in each other's way. a common
key scheme (Strings, as of the time of this writing?) is all thats required
for that. it is assumed that the other end interested in any such piece of
metadata knows the encoding, and byte[] provides for the most flexibility.
i believe this is the same logic behind core kafka being byte[]/byte[] -
Strings are more "usable" but bytes are flexible and so were chosen.
Also - core kafka doesnt even do that good of a job on usability of the
payload (example - i have to specify the nop byte[] "decoders" explicitly
in conf), and again sacrificies usability for the sake of performance (no
convenient single-record processing as poll is a batch, lots of obscure
little config details exposing internals of the batching mechanism, etc)
this is also why i really dislike the idea of a "type system" for header
values, it further degrades the usability, adds complexity and will
eventually get in people's way, also, it would be the 2nd/3rd home-group
serialization mechanism in core kafka (counting 2 iterations of the "type
definition DSL")
2. this is an implementation detail, and not even a very "user facing" one?
to the best of my understanding the vote process is on proposed
API/behaviour. also - since we're willing to go with strings just serialize
a 0-sized header blob and IIUC you dont need any optionals anymore.
3. yes, we can :-)
On Tue, Feb 14, 2017 at 11:56 PM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jay,
>
> 1) There was some initial debate on the value part, as youll note String,
> String headers were discounted early on. The reason for this is flexibility
> and keeping in line with the flexibility of key, value of the message
> object itself. I don’t think it takes away from an ecosystem as each plugin
> will care for their own key, this way ints, booleans , exotic custom binary
> can all be catered for=.
> a. If you really wanted to push for a typed value interface, I wouldn’t
> want just String values supported, but the the primatives plus string and
> also still keeping the ability to have a binary for custom binaries that
> some organisations may have.
> i. I have written this slight alternative here, https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
> ii. Essentially the value bytes, has a leading byte overhead.
> 1. This tells you what type the value is, before reading the rest of the
> bytes, allowing serialisation/deserialization to and from the primitives,
> string and byte[]. This is akin to some other messaging systems.
> 2) We are making it optional, so that for those not wanting headers have 0
> bytes overhead (think of it as a feature flag), I don’t think this is
> complex, especially if comparing to changes proposed in other kips like
> kip-98.
> a. If you really really don’t like this, we can drop it, but it would mean
> buying into 4 bytes extra overhead for users who do not want to use headers.
> 3) In the summary yes, it is at a higher level, but I think this is well
> documented in the proposed changes section.
> a. Added getHeaders method to Producer/Consumer record (that is it)
> b. We’ve also detailed the new Headers class that this method returns that
> encapsulates the headers protocol and logic.
>
> Best,
> Mike
>
> ==Original questions from the vote thread from Jay.==
>
> Couple of things I think we still need to work out:
>
> 1. I think we agree about the key, but I think we haven't talked about
> the value yet. I think if our goal is an open ecosystem of these header
> spread across many plugins from many systems we should consider making
> this
> a string as well so it can be printed, set via a UI, set in config, etc.
> Basically encouraging pluggable serialization formats here will lead to
> a
> bit of a tower of babel.
> 2. This proposal still includes a pretty big change to our serialization
> and protocol definition layer. Essentially it is introducing an optional
> type, where the format is data dependent. I think this is actually a big
> change though it doesn't seem like it. It means you can no longer
> specify
> this type with our type definition DSL, and likewise it requires custom
> handling in client libs. This isn't a huge thing, since the Record
> definition is custom anyway, but I think this kind of protocol
> inconsistency is very non-desirable and ties you to hand-coding things.
> I
> think the type should instead by [Key Value] in our BNF, where key and
> value are both short strings as used elsewhere. This brings it in line
> with
> the rest of the protocol.
> 3. Could we get more specific about the exact Java API change to
> ProducerRecord, ConsumerRecord, Record, etc?
>
> -Jay
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Jay,
1) There was some initial debate on the value part, as youll note String, String headers were discounted early on. The reason for this is flexibility and keeping in line with the flexibility of key, value of the message object itself. I don’t think it takes away from an ecosystem as each plugin will care for their own key, this way ints, booleans , exotic custom binary can all be catered for=.
a. If you really wanted to push for a typed value interface, I wouldn’t want just String values supported, but the the primatives plus string and also still keeping the ability to have a binary for custom binaries that some organisations may have.
i. I have written this slight alternative here, https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers+-+Typed
ii. Essentially the value bytes, has a leading byte overhead.
1. This tells you what type the value is, before reading the rest of the bytes, allowing serialisation/deserialization to and from the primitives, string and byte[]. This is akin to some other messaging systems.
2) We are making it optional, so that for those not wanting headers have 0 bytes overhead (think of it as a feature flag), I don’t think this is complex, especially if comparing to changes proposed in other kips like kip-98.
a. If you really really don’t like this, we can drop it, but it would mean buying into 4 bytes extra overhead for users who do not want to use headers.
3) In the summary yes, it is at a higher level, but I think this is well documented in the proposed changes section.
a. Added getHeaders method to Producer/Consumer record (that is it)
b. We’ve also detailed the new Headers class that this method returns that encapsulates the headers protocol and logic.
Best,
Mike
==Original questions from the vote thread from Jay.==
Couple of things I think we still need to work out:
1. I think we agree about the key, but I think we haven't talked about
the value yet. I think if our goal is an open ecosystem of these header
spread across many plugins from many systems we should consider making this
a string as well so it can be printed, set via a UI, set in config, etc.
Basically encouraging pluggable serialization formats here will lead to a
bit of a tower of babel.
2. This proposal still includes a pretty big change to our serialization
and protocol definition layer. Essentially it is introducing an optional
type, where the format is data dependent. I think this is actually a big
change though it doesn't seem like it. It means you can no longer specify
this type with our type definition DSL, and likewise it requires custom
handling in client libs. This isn't a huge thing, since the Record
definition is custom anyway, but I think this kind of protocol
inconsistency is very non-desirable and ties you to hand-coding things. I
think the type should instead by [Key Value] in our BNF, where key and
value are both short strings as used elsewhere. This brings it in line with
the rest of the protocol.
3. Could we get more specific about the exact Java API change to
ProducerRecord, ConsumerRecord, Record, etc?
-Jay
On 13/02/2017, 17:44, "Michael Pearce" <Mi...@ig.com> wrote:
Fantastic. Thanks all for compromises ☺
Based on getting what seems some consensus to move this forward, ive updated:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers
• Changed int keys to String keys
• Made it clear a key can occur twice, aka multimap.
Is there any other blocking issues. Or do we all think we’re good to go to a vote?
Many thanks
Mike
On 10/02/2017, 22:57, "radai" <ra...@gmail.com> wrote:
yeah I'm ok with both strings and multi-maps
On Fri, Feb 10, 2017 at 2:37 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Can we also compromise on a multi-map?
>
> One of my use-cases is tagging on multi-DC setup. It will be nice if
> each "hop" can add "Tag" header with its own DC-name without having to
> edit a single string that someone else wrote. For some financial
> use-cases it will be a requirement.
>
> Gwen
>
> On Fri, Feb 10, 2017 at 12:19 PM, Ignacio Solis <is...@igso.net> wrote:
> > I would be willing to compromise on Strings as keys if that moves all
> > of us forward.
> >
> > Right now headers are blocking a few of the things we want to do.
> >
> > Nacho
> >
> > On Thu, Feb 9, 2017 at 2:14 PM, Jay Kreps <ja...@confluent.io> wrote:
> >> I'm neutral positive on adding headers conceptually. I'm -1 on a
> proposal
> >> using ints for header names. I just don't think a global namespace with
> >> integer names is an acceptable end-user interface given that our goal
> is to
> >> encourage a federated ecosystem of header users. Not trying to be a
> pain in
> >> the ass, I just think it would be a bit of a wart.
> >>
> >> -Jay
> >>
> >> On Thu, Feb 9, 2017 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >>> As we'd like to start work on this to make it into the next release in
> a
> >>> few months.
> >>>
> >>> If we were to put this to the vote today who would actually give this
> a -1
> >>> vote?
> >>>
> >>> Based on the number of identified usecases (some stronger than others)
> >>>
> >>> Also if we were to keep with int keys based on all the discussion
> thread
> >>> for the pros vs cons who would give a -1 as it stands? Eg anyone
> believe it
> >>> simply wouldn't work.
> >>>
> >>> If not so we make progress would it be sensible to put this to vote?
> So we
> >>> can start work?
> >>>
> >>> Cheers
> >>> Mike
> >>>
> >>>
> >>> Sent using OWA for iPhone
> >>> ________________________________________
> >>> From: Jeroen van Disseldorp <je...@axual.io>
> >>> Sent: Wednesday, February 1, 2017 2:43:45 PM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>>
> >>> I'm lagging a bit here too, but want to respond to the usefulness of
> >>> this KIP for two use cases: bidirectional mirroring and smart cluster
> >>> switches.
> >>>
> >>> Bidirectional mirroring
> >>>
> >>> For a Dutch bank we have currently implemented bidirectional mirroring
> >>> using our own headers, which we encode in the Value using a custom
> >>> serializer. It encodes just one byte extra, containing a
> "copy/original"
> >>> bit. When applications serialize an avro object, the bit is unset by
> >>> default. Our mirroring mechanism subscribes to all topics, checks the
> >>> bit, and when unset the bit is set and the message is forwarded to the
> >>> other DC.
> >>>
> >>> The overhead of this is pretty low (1 byte) and it's cheap, since the
> >>> mirroring mechanism does not need to fully deserialize the value, but
> >>> just check+flip one bit in the byte[] value. On the receiving end, the
> >>> mirroring mechanism sees the message for the second time, but notices
> >>> the message is a copy already and ignores it.
> >>>
> >>> The mechanism has been in production since last year and it works
> really
> >>> well. Its downside however is the incompatibility with avro serialized
> >>> values, as the whole organization is mandated to use our own custom
> >>> serde and we were forced to create a wrapper client library around the
> >>> default kafka client. The callback we offer to applications is very
> >>> similar to that proposed in the KIP, as every message our users get
> >>> consists of <Metadata, K, V>.
> >>>
> >>> Additionally our header contains a message id and serialization
> >>> timestamp (invented before 0.10.0.0). These fields could also be done
> >>> away with when this KIP is implemented.
> >>>
> >>> Smart cluster switches
> >>>
> >>> Another case we are considering to use headers for is enabling "smart
> >>> cluster switches" for consumers and producers in a way that would avoid
> >>> duplicates and/or missed messages in most cases. This involves encoding
> >>> a producer id (app name or id), producer instance id (uuid generated
> >>> upon producer's start) and sequence number in every message. We expect
> >>> to use one sequence per partition.
> >>>
> >>> These fields enable our consumers to keep track of which messages were
> >>> seen without relying on offsets, which is a cluster-bound value.
> >>> Consumers keep track of seen messages by keeping a small administration
> >>> of ranges of sequence numbers per producer instance and partition.
> Since
> >>> sequence numbers are always increasing per producer and partition, the
> >>> memory consumption of such administration is pretty low, while being
> >>> meaningful across clusters. After every poll() we can commit the
> >>> administration to a topic in much the same way as we do with offsets.
> >>> The difference is that this topic can be mirrored to the other cluster
> >>> and upon a switch of consumer, all instances can use the topic to
> >>> initialize themselves and skip all messages already processed.
> >>>
> >>> Hope these views help in the upcoming decision on this KIP.
> >>>
> >>> Regards
> >>> Jeroen
> >>>
> >>>
> >>>
> >>> On 09/01/2017 20:40, Gwen Shapira wrote:
> >>> > Mirroring + ClusterIDs:
> >>> >
> >>> > Kafka clusters already have an automatically generated ID that is
> >>> > exposed to clients:
> >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 78%3A+Cluster+Id
> >>> >
> >>> > It sounds trivial to get MirrorMaker's consumer to grab this ID
> (which
> >>> > always exists) from the source cluster and add it to the message
> >>> > headers.
> >>> >
> >>> > Obviously adding stuff to the message makes it larger so this should
> >>> > be a feature you should be able to turn on/off depending on the
> >>> > replication requirements. If you need the feature - you pay the
> price.
> >>> > I don't think there is an immediate need to put an effort into
> >>> > shrinking the Cluster ID, but others can comment on that.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> >>> >> Hey Michael,
> >>> >>
> >>> >> Sorry for lagging, I've been out for the holiday break. What would
> help
> >>> me
> >>> >> would be to get to a consolidated list of use cases we've worked
> out and
> >>> >> think add real value. I think getting down to the details really
> matters
> >>> >> for these.
> >>> >>
> >>> >> The two lists I'm aware of are the one in the KIP and the one you
> gave
> >>> in
> >>> >> this email thread. Both seem to be more at the sketch level. Let me
> >>> review
> >>> >> the use cases and see if I have it right. If so let's get it in the
> >>> design
> >>> >> proposal.
> >>> >>
> >>> >> Your use cases:
> >>> >>
> >>> >> 1. Tracing IDs -- I get this use case and think I see how it
> would
> >>> work.
> >>> >> I think because the tracing is from a third party system there
> is
> >>> some
> >>> >> value to having a header versus a custom format to make this
> work
> >>> fully
> >>> >> transparently.
> >>> >> 2. Mirroring -- I'm a bit unsure how this will work, I'll
> explain my
> >>> >> thinking below
> >>> >> 3. End-to-end encryption -- I think what you are saying is that
> if
> >>> you
> >>> >> want "routing' fields that are accessible in transit without
> >>> decrypting you
> >>> >> need a scheme to leave this decrypted.I think the idea is to
> avoid
> >>> >> decrypting to get a header, but I'm not sure if I get how this
> is
> >>> better
> >>> >> than a custom message format that doesn't encrypt the things you
> >>> don't want
> >>> >> encrypted since you have to be aware of the scheme anyway.
> >>> >> 4. EOS -- I'm not convinced this is a good approach for EOS.
> >>> >>
> >>> >> Linked from KIP:
> >>> >>
> >>> >> 1. Schema-Registry-aware avro serialization -- not clear how
> this is
> >>> >> better than keeping the schema id in the value
> >>> >> 2. Polyglot kafka topics -- marginally better than a container
> >>> format,
> >>> >> but of course you still have to all agree on the the header and
> >>> >> header=>serialization mapping so not sure if it's really better
> >>> than a
> >>> >> container format. Also not wild about polyglot topics :-)
> >>> >> 3. End-to-End routing trail including mirroring --- I'd put this
> >>> under
> >>> >> tracing ids
> >>> >> 4. Cross-service process tracing -- ditto
> >>> >> 5. Signing -- I like this use case but it seems to depend on a
> >>> feature
> >>> >> we don't have and aren't proposing adding, right?
> >>> >> 6. Per-record custom compression -- why do you want this?
> >>> compressing
> >>> >> per message won't work well compared to batch compression,
> right?
> >>> >> 7. Large Message Support -- I think we can agree that although
> this
> >>> >> might work it's clearly not the right way to support large
> messages
> >>> in Kafka
> >>> >>
> >>> >> Matthias
> >>> >>
> >>> >> 1. Control messages --- Matthias, not sure if I get the
> motivation
> >>> in
> >>> >> streams for this. If we did want to do it, wouldn't it be
> easier to
> >>> >> implement and by allowing users to send control messages (in the
> >>> EOS sense)
> >>> >> and adding a consumer config to allow either passing these on or
> >>> >> suppressing them?
> >>> >>
> >>> >> So I if I were to summarize I'd list the use cases as follows:
> >>> >>
> >>> >> - Much better with headers: tracing IDs
> >>> >> - Somewhat better: end-to-end encryption, polyglot topics
> >>> >> - Don't buy: Mirroring (though perhaps could be made to work
> with
> >>> >> changes to the proposal), EOS, large messages, compression
> >>> >> - Not sure: Control messages
> >>> >>
> >>> >> Does this match where other people are at? From my point of view for
> >>> >> something like this where we are changing the core data model we
> should
> >>> >> ideally get a handful of use cases to the point where they are
> really
> >>> well
> >>> >> thought out and we know they can use the feature. I think we have
> one
> >>> solid
> >>> >> one with tracing IDs, and I think maybe mirroring could be another
> with
> >>> >> some work. I'm less sure about the others.
> >>> >>
> >>> >> Bi-directional Mirroring
> >>> >> I think bidirectional mirroring could potentially become a
> compelling
> >>> use
> >>> >> case if we an find a way to make it work. If I understand it, the
> >>> proposal
> >>> >> is to add a header like "origin-cluster" with the cluster id of the
> >>> Kafka
> >>> >> cluster. Then modify the mirroring tool to disallow mirroring a
> message
> >>> >> back to it's own origin cluster. This basic idea is definitely
> workable
> >>> and
> >>> >> useful, but can we really get to a good implementation using
> headers?
> >>> >> I imagine the approach would be to add this header in the producer
> if
> >>> some
> >>> >> config is set. There are two problems:
> >>> >>
> >>> >> - You add a pretty big honking cluster ID to each message. For
> small
> >>> >> messages this maybe larger than the contents itself.
> >>> >> - What happens if someone somewhere forgets to set that config
> on
> >>> some
> >>> >> client somewhere?
> >>> >>
> >>> >> If someone forgets to set the config in some client somewhere I
> guess
> >>> you
> >>> >> have various options in mirror maker:
> >>> >>
> >>> >> 1. drop the data (pretty bad, now your data is out of sync)
> >>> >> 2. crash (pretty bad--how do you recover)
> >>> >> 3. insert some default (pretty bad since would also lead to data
> >>> loss if
> >>> >> that's not right)
> >>> >>
> >>> >> Basically although I think a cluster tag could be a good way to do
> >>> >> bi-directional mirroring, I'm not sure this naive approach I
> described
> >>> >> would get us to a usable implementation. This could be fixable. I
> think
> >>> if
> >>> >> there was some scheme to check required headers on the server that
> could
> >>> >> alleviate the missing header problem. Not really sure what to do
> about
> >>> the
> >>> >> message bloat problem, but maybe there could be some way to shrink
> the
> >>> >> cluster id?
> >>> >>
> >>> >> Cheers,
> >>> >>
> >>> >> -Jay
> >>> >>
> >>> >>
> >>> >> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> Michael.Pearce@ig.com
> >>> >
> >>> >> wrote:
> >>> >>
> >>> >>> @Jay, Ismael and a few others.
> >>> >>>
> >>> >>> Based on the discussion on kip-87, and the additional use case by
> >>> Matthias.
> >>> >>>
> >>> >>> Just for clarity do we now have consensus on the earlier points (1)
> >>> >>> headers are useful (2) headers should be native in Kafka?
> >>> >>>
> >>> >>> Cheers
> >>> >>> Mike
> >>> >>> ________________________________________
> >>> >>> From: radai <ra...@gmail.com>
> >>> >>> Sent: Monday, December 19, 2016 5:01 PM
> >>> >>> To: dev@kafka.apache.org
> >>> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>> >>>
> >>> >>> having said that im perfectly fine with something like a "no
> payload"
> >>> flag
> >>> >>> (as opposed to null payload), which would mean such a msg is not
> >>> returned
> >>> >>> from poll() unless the user code specifically "opts in" and would
> serve
> >>> >>> only to carry headers (making it a control message)
> >>> >>>
> >>> >>> On Mon, Dec 19, 2016 at 8:58 AM, radai <radai.rosenblatt@gmail.com
> >
> >>> wrote:
> >>> >>>
> >>> >>>> if all you want is to tag "end of stream" wouldnt a header on the
> last
> >>> >>> msg
> >>> >>>> in the stream be enough?
> >>> >>>>
> >>> >>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> >>> matthias@confluent.io>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> The details about headers for control messages are still to
> define.
> >>> But
> >>> >>>>> yes, the idea is to have some common default behavior that
> clients
> >>> would
> >>> >>>>> need to implement.
> >>> >>>>>
> >>> >>>>> The point is, that "regular headers" add meta data to regular
> >>> messages.
> >>> >>>>> Thus, those messages will be returned to the user via .poll().
> And
> >>> after
> >>> >>>>> the message is received the user can check if meta data is
> present
> >>> and
> >>> >>>>> read it.
> >>> >>>>>
> >>> >>>>> For control messages, we do not want those to pop up via .poll()
> as
> >>> >>>>> those are no regular messages. A client would need to opt-in to
> see
> >>> >>>>> those messages (either via poll() or maybe a callback). Thus, we
> need
> >>> >>>>> some special (standardized) header IDs that indicate control
> messages
> >>> >>>>> that should not be returned to the user via poll() by default.
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> -Matthias
> >>> >>>>>
> >>> >>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
> >>> >>>>>> Matthias,
> >>> >>>>>>
> >>> >>>>>> Thanks for your input. I'm +1 on control messages as they seem
> to
> >>> be
> >>> >>>>> the
> >>> >>>>>> simplest way to implement watermarks (
> >>> >>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-
> streaming-102
> >>> ),
> >>> >>> a
> >>> >>>>>> feature that would add a lot of value to Kafka Streams IMHO.
> >>> >>>>>>
> >>> >>>>>> Your argument that the control-message indicator belongs in the
> >>> >>>>> client-only
> >>> >>>>>> section of the record format make sense. Just to make sure I
> >>> >>>>> understand,
> >>> >>>>>> are you suggesting that control messages would be indicated by a
> >>> >>>>> standard
> >>> >>>>>> reserved header? By standard, I mean that ALL Kafka consumers
> would
> >>> >>>>> know
> >>> >>>>>> to handle these messages differently (possibly just ignoring
> them).
> >>> >>>>> This
> >>> >>>>>> would need to be added to the specification of the consumer
> protocol
> >>> >>> so
> >>> >>>>>> that all Kafka clients implement it, right? I think it's a good
> >>> idea
> >>> >>>>> but
> >>> >>>>>> just checking.
> >>> >>>>>>
> >>> >>>>>> Cheers,
> >>> >>>>>>
> >>> >>>>>> Roger
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> >>> >>> matthias@confluent.io
> >>> >>>>>> wrote:
> >>> >>>>>>
> >>> >>>>>>> Yes and no. I did overload the term "control message".
> >>> >>>>>>>
> >>> >>>>>>> EOS control messages are for client-broker communication and
> thus
> >>> >>> never
> >>> >>>>>>> exposed to any application. And I think this is a good design
> >>> because
> >>> >>>>>>> broker needs to understand those control messages. Thus, this
> >>> should
> >>> >>> be
> >>> >>>>>>> a protocol change.
> >>> >>>>>>>
> >>> >>>>>>> The type of control messages I have in mind are for
> client-client
> >>> >>>>>>> (application-application) communication and the broker is
> agnostic
> >>> to
> >>> >>>>>>> them. Thus, it should not be a protocol change.
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> -Matthias
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> On 12/14/16 9:42 AM, radai wrote:
> >>> >>>>>>>> arent control messages getting pushed as their own top level
> >>> >>> protocol
> >>> >>>>>>>> change (and a fairly massive one) for the transactions KIP ?
> >>> >>>>>>>>
> >>> >>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> >>> >>>>> matthias@confluent.io>
> >>> >>>>>>>> wrote:
> >>> >>>>>>>>
> >>> >>>>>>>>> Hi,
> >>> >>>>>>>>>
> >>> >>>>>>>>> I want to add a completely new angle to this discussion. For
> >>> this,
> >>> >>> I
> >>> >>>>>>>>> want to propose an extension for the headers feature that
> enables
> >>> >>> new
> >>> >>>>>>>>> uses cases -- and those new use cases might convince people
> to
> >>> >>>>> support
> >>> >>>>>>>>> headers (of course including the larger scoped proposal).
> >>> >>>>>>>>>
> >>> >>>>>>>>> Extended Proposal:
> >>> >>>>>>>>>
> >>> >>>>>>>>> Allow messages with a certain header key to be special
> "control
> >>> >>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an
> >>> application
> >>> >>>>> via
> >>> >>>>>>>>> .poll().
> >>> >>>>>>>>>
> >>> >>>>>>>>> Thus, a consumer client would automatically skip over those
> >>> >>>>> messages. If
> >>> >>>>>>>>> an application knows about embedded control messages, it can
> >>> "sing
> >>> >>>>> up"
> >>> >>>>>>>>> to those messages by the consumer client and either get a
> >>> callback
> >>> >>> or
> >>> >>>>>>>>> the consumer auto-drop for this messages gets disabled
> (allowing
> >>> to
> >>> >>>>>>>>> consumer those messages via poll()).
> >>> >>>>>>>>>
> >>> >>>>>>>>> (The details need further considerations/discussion. I just
> want
> >>> to
> >>> >>>>>>>>> sketch the main idea.)
> >>> >>>>>>>>>
> >>> >>>>>>>>> Usage:
> >>> >>>>>>>>>
> >>> >>>>>>>>> There is a shared topic (ie, used by multiple applications)
> and a
> >>> >>>>>>>>> producer application wants to embed a special message in the
> >>> topic
> >>> >>>>> for a
> >>> >>>>>>>>> dedicated consumer application. Because only one application
> will
> >>> >>>>>>>>> understand this message, it cannot be a regular message as
> this
> >>> >>> would
> >>> >>>>>>>>> break all applications that do not understand this message.
> The
> >>> >>>>> producer
> >>> >>>>>>>>> application would set a special metadata key and no consumer
> >>> >>>>> application
> >>> >>>>>>>>> would see this control message by default because they did
> not
> >>> >>> enable
> >>> >>>>>>>>> their consumer client to return this message in poll() (and
> the
> >>> >>>>> client
> >>> >>>>>>>>> would just drop this message with special metadata key).
> Only the
> >>> >>>>> single
> >>> >>>>>>>>> application that should receive this message, will subscribe
> to
> >>> >>> this
> >>> >>>>>>>>> message on its consumer client and process it.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Concrete Use Case: Kafka Streams
> >>> >>>>>>>>>
> >>> >>>>>>>>> In Kafka Streams, we would like to propagate "control
> messages"
> >>> >>> from
> >>> >>>>>>>>> subtopology to subtopology. There are multiple scenarios for
> >>> which
> >>> >>>>> this
> >>> >>>>>>>>> would be useful. For example, currently we do not guarantee a
> >>> >>>>>>>>> "consistent shutdown" of an application. By this, I mean that
> >>> input
> >>> >>>>>>>>> records might not be completely processed by the whole
> topology
> >>> >>>>> because
> >>> >>>>>>>>> the application shutdown happens "in between" and an
> intermediate
> >>> >>>>> result
> >>> >>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user
> would
> >>> see
> >>> >>>>> an
> >>> >>>>>>>>> committed offset of the source topic of the application, but
> no
> >>> >>>>>>>>> corresponding result record in the output topic.
> >>> >>>>>>>>>
> >>> >>>>>>>>> Having "shutdown markers" would allow us, to first stop the
> >>> >>> upstream
> >>> >>>>>>>>> subtopology and write this marker into the intermediate
> topic and
> >>> >>> the
> >>> >>>>>>>>> downstream subtopology would only shut down itself after is
> sees
> >>> >>> the
> >>> >>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that
> no
> >>> >>>>>>>>> "in-flight" messages got stuck in intermediate topics.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch
> >>> Processing).
> >>> >>>>>>>>> There was a discussion about the proposed metadata topic,
> and we
> >>> >>>>> could
> >>> >>>>>>>>> avoid this metadata topic if we would have "control
> messages".
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Right now, we cannot insert an "application control message"
> >>> >>> because
> >>> >>>>>>>>> Kafka Streams does not own all topics it read/writes and thus
> >>> might
> >>> >>>>>>>>> break other consumer application (as described above) if we
> >>> inject
> >>> >>>>>>>>> random messages that are not understood by other apps.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Of course, one can work around "embedded control messaged" by
> >>> using
> >>> >>>>> an
> >>> >>>>>>>>> additional topic to propagate control messaged between
> >>> application
> >>> >>>>> (as
> >>> >>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka
> Streams). But
> >>> >>>>> there
> >>> >>>>>>>>> are major concerns about adding this metadata topic in the
> KIP
> >>> and
> >>> >>>>> this
> >>> >>>>>>>>> shows that other application that need a similar pattern
> might
> >>> >>> profit
> >>> >>>>>>>>> from topic embedded "control messages", too.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> One last important consideration: those "control messages"
> are
> >>> used
> >>> >>>>> for
> >>> >>>>>>>>> client to client communication and are not understood by the
> >>> >>> broker.
> >>> >>>>>>>>> Thus, those messages should not be enabled within the message
> >>> >>> format
> >>> >>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> record
> >>> >>>>> headers
> >>> >>>>>>>>> would be a nice way to implement them. Because KIP-82 did
> >>> consider
> >>> >>>>> key
> >>> >>>>>>>>> namespaces for metatdata keys, this extension should not be
> an
> >>> own
> >>> >>>>> KIP
> >>> >>>>>>>>> but should be included in KIP-82 to reserve a namespace for
> >>> >>> "control
> >>> >>>>>>>>> message" in the first place.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Sorry for the long email... Looking forward to your feedback.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> -Matthias
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> >>> >>>>>>>>>> Hi Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
> >>> >>>>> (HTTP/JMS/
> >>> >>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
> >>> >>> (though I
> >>> >>>>>>>>> believe it contains the end2end uuid embedded in this id), on
> >>> >>>>> receiving
> >>> >>>>>>> the
> >>> >>>>>>>>> message at the receiving JVM the apm code takes this out, and
> >>> >>>>> continues
> >>> >>>>>>> its
> >>> >>>>>>>>> tracing on the that new thread. Both JVM's (and other
> languages
> >>> the
> >>> >>>>> APM
> >>> >>>>>>>>> tool supports) send this data async back to the central
> >>> controllers
> >>> >>>>>>> where
> >>> >>>>>>>>> the stiching togeather occurs. For this they need some header
> >>> space
> >>> >>>>> for
> >>> >>>>>>>>> them to put this id.
> >>> >>>>>>>>>> 101) Yes indeed we have a business transaction Id in the
> >>> payload.
> >>> >>>>>>> Though
> >>> >>>>>>>>> this is a system level tracing, that we need to have marry
> up.
> >>> Also
> >>> >>>>> as
> >>> >>>>>>> per
> >>> >>>>>>>>> note on end2end encryption we'd be unable to prove the flow
> if
> >>> the
> >>> >>>>>>> payload
> >>> >>>>>>>>> is encrypted as we'd not have access to this at certain
> points of
> >>> >>> the
> >>> >>>>>>> flow
> >>> >>>>>>>>> through the infrastructure/platform.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 103) As said we use this mechanism in IG very successfully,
> as
> >>> >>>>> stated
> >>> >>>>>>>>> per key we guarantee the transaction producing app to handle
> the
> >>> >>>>>>>>> transaction of a key at one DC unless at point of critical
> >>> failure
> >>> >>>>>>> where we
> >>> >>>>>>>>> have to flip processing to another. We care about key
> ordering.
> >>> >>>>>>>>>> I disagree on the offset comment for the partition solution
> >>> unless
> >>> >>>>> you
> >>> >>>>>>>>> do full ISR, or expensive full XA transactions even with
> >>> partitions
> >>> >>>>> you
> >>> >>>>>>>>> cannot fully guarantee offsets would match.
> >>> >>>>>>>>>> 105) Very much so, I need to have access at the platform
> level
> >>> to
> >>> >>>>> the
> >>> >>>>>>>>> other meta data all mentioned, without having to need to have
> >>> >>> access
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> encryption keys of the payload.
> >>> >>>>>>>>>> 106)
> >>> >>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to
> >>> have
> >>> >>> a
> >>> >>>>>>>>> global producerId register which would be very hard to
> >>> >>>>> enforce/ensure is
> >>> >>>>>>>>> current and correct, just to understand the message origins
> of
> >>> its
> >>> >>>>>>>>> region/az/cluster for routing.
> >>> >>>>>>>>>> The client wrapper version, producerId can be the same, as
> >>> >>> obviously
> >>> >>>>>>> the
> >>> >>>>>>>>> producer could upgrade its wrapper, as such we need to know
> what
> >>> >>>>> wrapper
> >>> >>>>>>>>> version the message is created with.
> >>> >>>>>>>>>> Likewise the IP address, as stated we can have our producer
> >>> move,
> >>> >>>>> where
> >>> >>>>>>>>> its IP would change.
> >>> >>>>>>>>>> 107)
> >>> >>>>>>>>>> UUID is set on the message by interceptors before actual
> >>> producer
> >>> >>>>>>>>> transport send. This is for platform level message dedupe
> >>> >>> guarantee,
> >>> >>>>> the
> >>> >>>>>>>>> business payload should be agnostic to this. Please see
> >>> >>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
> >>> >>>>> ection.html
> >>> >>>>>>>>> note this is not touching business payloads.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Hi, Michael,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Thanks for the reply. I find it very helpful.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Data lineage:
> >>> >>>>>>>>>> 100. I'd like to understand the APM use case a bit
> more. It
> >>> >>>>> sounds
> >>> >>>>>>>>> like
> >>> >>>>>>>>>> that those APM plugins can generate a transaction id
> that
> >>> we
> >>> >>>>> could
> >>> >>>>>>>>>> potentially put in the header of every message. How
> would
> >>> you
> >>> >>>>>>>>> typically
> >>> >>>>>>>>>> make use of such transaction ids? Are there other
> metadata
> >>> >>>>>>>>> associated with
> >>> >>>>>>>>>> the transaction id and if so, how are they propagated
> >>> >>>>> downstream?
> >>> >>>>>>>>>> 101. For the finance use case, if the concept of
> >>> transaction
> >>> >>> is
> >>> >>>>>>>>> important,
> >>> >>>>>>>>>> wouldn't it be typically included in the message
> payload
> >>> >>>>> instead of
> >>> >>>>>>>>> as an
> >>> >>>>>>>>>> optional header field?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 102. The data lineage that Altas and Navigator support
> >>> seems
> >>> >>> to
> >>> >>>>> be
> >>> >>>>>>>>> at the
> >>> >>>>>>>>>> dataset level, not per record level? So, not sure if
> per
> >>> >>> message
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> are relevant there.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Mirroring:
> >>> >>>>>>>>>> 103. The benefit of using separate partitions is that
> it
> >>> >>>>>>> potentially
> >>> >>>>>>>>> makes
> >>> >>>>>>>>>> it easy to preserve offsets during mirroring. This will
> >>> make
> >>> >>> it
> >>> >>>>>>>>> easier for
> >>> >>>>>>>>>> consumer to switch clusters. Currently, the consumers
> can
> >>> >>> switch
> >>> >>>>>>>>> clusters
> >>> >>>>>>>>>> by using the timestampToOffset() api, but it has to
> deal
> >>> with
> >>> >>>>>>>>> duplicates.
> >>> >>>>>>>>>> Good point on the issue with log compact and I am not
> sure
> >>> how
> >>> >>>>> to
> >>> >>>>>>>>> address
> >>> >>>>>>>>>> this. However, even if we mirror into the existing
> >>> partitions,
> >>> >>>>> the
> >>> >>>>>>>>> ordering
> >>> >>>>>>>>>> for messages generated from different clusters seems
> >>> >>>>>>>>> non-deterministic
> >>> >>>>>>>>>> anyway. So, it seems that the consumers already have to
> >>> deal
> >>> >>>>> with
> >>> >>>>>>>>> that? If
> >>> >>>>>>>>>> a topic is compacted, does that mean which messages are
> >>> >>>>> preserved
> >>> >>>>>>> is
> >>> >>>>>>>>> also
> >>> >>>>>>>>>> non-deterministic across clusters?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 104. Good point on partition key.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> End-to-end encryption:
> >>> >>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are
> >>> headers
> >>> >>>>>>>>> useful there?
> >>> >>>>>>>>>> Auditing:
> >>> >>>>>>>>>> 106. It seems other than the UUID, all other metadata
> are
> >>> per
> >>> >>>>>>>>> producer?
> >>> >>>>>>>>>> EOS:
> >>> >>>>>>>>>> 107. How are those UUIDs generated? I am not sure if
> they
> >>> can
> >>> >>> be
> >>> >>>>>>>>> generated
> >>> >>>>>>>>>> in the producer library. An application may send
> messages
> >>> >>>>> through a
> >>> >>>>>>>>> load
> >>> >>>>>>>>>> balancer and on retry, the same message could be
> routed to
> >>> a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> producer instance. So, it seems that the application
> has to
> >>> >>>>>>> generate
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> UUIDs. In that case, shouldn't the application just
> put the
> >>> >>>>> UUID in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> payload?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Thanks,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> >>> >>>>>>>>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> > Hi Jun.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Per Transaction Tracing / Data Lineage.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As Stated in the KIP this has the first use case of
> how
> >>> many
> >>> >>>>> APM
> >>> >>>>>>>>> tools now
> >>> >>>>>>>>>> > work.
> >>> >>>>>>>>>> > I would find it impossible for any one to argue this
> is
> >>> not
> >>> >>>>>>>>> important or a
> >>> >>>>>>>>>> > niche market as it has its own gartner report for
> this
> >>> >>> space.
> >>> >>>>>>> Such
> >>> >>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace,
> Hawqular
> >>> are
> >>> >>>>> but a
> >>> >>>>>>>>> few.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise these APM tools can help very rapidly track
> down
> >>> >>>>> issues
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > automatically capture metrics, perform actions based
> on
> >>> >>>>>>> unexpected
> >>> >>>>>>>>> behavior
> >>> >>>>>>>>>> > to auto recover services.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Before mentioning looking at aggregated stats, in
> these
> >>> >>> cases
> >>> >>>>>>> where
> >>> >>>>>>>>>> > actually on critical flows we cannot afford to have
> >>> >>> aggregated
> >>> >>>>>>>>> rolled up
> >>> >>>>>>>>>> > stats only.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > With the APM tool we use its actually able to detect
> a
> >>> >>> single
> >>> >>>>>>>>> transaction
> >>> >>>>>>>>>> > failure and capture the thread traces in the JVM
> where it
> >>> >>>>> failed
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > everything for us, to the point it sends us alerts
> where
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > giving the line number of the code that caused it,
> the
> >>> >>>>>>> transaction
> >>> >>>>>>>>> trace
> >>> >>>>>>>>>> > through all the services and endpoints (supported)
> upto
> >>> the
> >>> >>>>> point
> >>> >>>>>>>>> of
> >>> >>>>>>>>>> > failure, it can also capture the data in and out (so
> we
> >>> can
> >>> >>>>>>>>> replay).
> >>> >>>>>>>>>> > Because atm Kafka doesn't support us being able to
> stich
> >>> in
> >>> >>>>> these
> >>> >>>>>>>>> tracing
> >>> >>>>>>>>>> > transaction ids natively, we cannot get these
> benefits as
> >>> >>>>> such is
> >>> >>>>>>>>> limiting
> >>> >>>>>>>>>> > our ability support apps and monitor them to the same
> >>> >>>>> standards
> >>> >>>>>>> we
> >>> >>>>>>>>> come to
> >>> >>>>>>>>>> > expect when on a kafka flow.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > This actually ties in with Data Lineage, as the same
> >>> tracing
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>> > to back stich this. Essentially many times due to the
> >>> sums
> >>> >>> of
> >>> >>>>>>> money
> >>> >>>>>>>>>> > involved there are disputes, and typically as a
> financial
> >>> >>>>>>>>> institute the
> >>> >>>>>>>>>> > easiest and cleanest way to prove when disputes
> arise is
> >>> to
> >>> >>>>>>>>> present the
> >>> >>>>>>>>>> > actual flow and processes involved in a transaction.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise as Hadoop matures its evident this case is
> >>> >>>>> important, as
> >>> >>>>>>>>> tools
> >>> >>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator
> (cloudera
> >>> led)
> >>> >>>>> are
> >>> >>>>>>>>> evident
> >>> >>>>>>>>>> > also I believe the importance here is very much NOT
> just
> >>> a
> >>> >>>>>>>>> financial issue.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > From a MDM point of view any company wanting to care
> >>> about
> >>> >>>>> Data
> >>> >>>>>>>>> Quality
> >>> >>>>>>>>>> > and Data Governance - Data Lineage is a key piece in
> this
> >>> >>>>> puzzle.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Mirroring,
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As per the KIP in-fact this is exactly what we do re
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> to mirror
> >>> >>>>>>>>>> > a network of clusters between AZ's / Regions. We
> know a
> >>> >>>>>>>>> transaction for a
> >>> >>>>>>>>>> > key will be done within a AZ/Region, as such we
> know the
> >>> >>>>> write
> >>> >>>>>>> to
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>> > would be ordered per key. But we need eventual view
> of
> >>> that
> >>> >>>>>>> across
> >>> >>>>>>>>> in our
> >>> >>>>>>>>>> > other regions/az's. When we have complete AZ or
> Region
> >>> >>>>> failure we
> >>> >>>>>>>>> know
> >>> >>>>>>>>>> > there will be a brief interruption whilst those
> >>> transactions
> >>> >>>>> are
> >>> >>>>>>>>> moved to
> >>> >>>>>>>>>> > another region but we expect after it to continue.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As mentioned having separate Partions to do this
> starts
> >>> to
> >>> >>> get
> >>> >>>>>>>>>> > ugly/complicated for us:
> >>> >>>>>>>>>> > how would I do compaction where a key is in two
> >>> partitions?
> >>> >>>>>>>>>> > How do we balance consumers so where multiple
> partitions
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>> same key
> >>> >>>>>>>>>> > goto the same consumer
> >>> >>>>>>>>>> > What do you do if cluster 1 has 5 partitions but
> cluster
> >>> 20
> >>> >>>>> has
> >>> >>>>>>> 10
> >>> >>>>>>>>> because
> >>> >>>>>>>>>> > its larger kit in our more core DC's, as such key to
> >>> >>> partition
> >>> >>>>>>>>> mappings for
> >>> >>>>>>>>>> > consumers get even more complicated.
> >>> >>>>>>>>>> > What do you do if we add or remove a complete region
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Where as simple mirror will work we just need to
> ensure
> >>> we
> >>> >>>>> don't
> >>> >>>>>>>>> have a
> >>> >>>>>>>>>> > cycle which we can do with clusterId.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > We even have started to look at shortest path mirror
> >>> routing
> >>> >>>>>>> based
> >>> >>>>>>>>> on
> >>> >>>>>>>>>> > clusterId, if we also had the region and az info on
> the
> >>> >>>>>>> originating
> >>> >>>>>>>>>> > message, this we have not implemented but some ideas
> come
> >>> >>> from
> >>> >>>>>>>>> network
> >>> >>>>>>>>>> > routing, and also the dispatcher router in apache
> qpid.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Also we need to have data perimeters e.g. certain
> data
> >>> >>> cannot
> >>> >>>>>>> leave
> >>> >>>>>>>>>> > certain countries borders. We want this all
> automated so
> >>> >>> that
> >>> >>>>> at
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > platform level without having to touch or look at the
> >>> >>> business
> >>> >>>>>>>>> data inside
> >>> >>>>>>>>>> > we can have headers we can put tags into so that we
> can
> >>> >>> ensure
> >>> >>>>>>>>> this doesn't
> >>> >>>>>>>>>> > occur when we mirror. (actually links in to data
> lineage
> >>> /
> >>> >>>>>>> tracing
> >>> >>>>>>>>> as again
> >>> >>>>>>>>>> > we need to tag messages at a platform level) Examples
> >>> are we
> >>> >>>>> are
> >>> >>>>>>>>> not
> >>> >>>>>>>>>> > allowed Private customer details to leave
> Switzerland,
> >>> yet
> >>> >>> we
> >>> >>>>>>> need
> >>> >>>>>>>>> those
> >>> >>>>>>>>>> > systems integrated.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Lastly around mirroring we have a partionKey field,
> as
> >>> the
> >>> >>> key
> >>> >>>>>>>>> used for
> >>> >>>>>>>>>> > portioning logic != compaction key all the time but
> we
> >>> want
> >>> >>> to
> >>> >>>>>>>>> preserve it
> >>> >>>>>>>>>> > for when we mirror so that if source cluster
> partition
> >>> count
> >>> >>>>> !=
> >>> >>>>>>>>> destination
> >>> >>>>>>>>>> > cluster partition count we can honour the same
> >>> partitioning
> >>> >>>>>>> logic.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE End 2 End encryption
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As I believe mentioned just before, the solution you
> >>> mention
> >>> >>>>> just
> >>> >>>>>>>>> doesn't
> >>> >>>>>>>>>> > cut the mustard these days with many regulators. An
> >>> >>> operations
> >>> >>>>>>>>> person with
> >>> >>>>>>>>>> > access to the box should not be able to have access
> to
> >>> the
> >>> >>>>> data.
> >>> >>>>>>>>> Many now
> >>> >>>>>>>>>> > actually impose quite literally the implementation
> >>> expected
> >>> >>>>> being
> >>> >>>>>>>>> end2end
> >>> >>>>>>>>>> > encryption for certain data (Singapore for us is one
> >>> that I
> >>> >>> am
> >>> >>>>>>>>> most aware
> >>> >>>>>>>>>> > of). In fact we're even now needing encrypt the data
> and
> >>> >>> store
> >>> >>>>>>> the
> >>> >>>>>>>>> keys in
> >>> >>>>>>>>>> > HSM modules.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise the performance penalty on encrypting
> >>> decrypting as
> >>> >>>>> you
> >>> >>>>>>>>> produce
> >>> >>>>>>>>>> > over wire, then again encrypt decrypt as the data is
> >>> stored
> >>> >>> on
> >>> >>>>>>> the
> >>> >>>>>>>>> brokers
> >>> >>>>>>>>>> > disks and back again, then again encrypted and
> decrypted
> >>> >>> back
> >>> >>>>>>> over
> >>> >>>>>>>>> the wire
> >>> >>>>>>>>>> > each time for each consumer all adds up, ignoring
> this
> >>> >>>>> doubling
> >>> >>>>>>>>> with mirror
> >>> >>>>>>>>>> > makers etc. simply encrypting the value once on
> write by
> >>> the
> >>> >>>>>>>>> client and
> >>> >>>>>>>>>> > again decrypting on consume by the consumer is far
> more
> >>> >>>>>>>>> performant, but
> >>> >>>>>>>>>> > then the routing and platform meta data needs to be
> >>> separate
> >>> >>>>>>> (thus
> >>> >>>>>>>>> headers)
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Auditing:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Our Auditing needs are:
> >>> >>>>>>>>>> > Producer Id,
> >>> >>>>>>>>>> > Origin Cluster Id that message first produced into
> >>> >>>>>>>>>> > Origin AZ - agreed we can derive this if we have
> cluster
> >>> id,
> >>> >>>>> but
> >>> >>>>>>>>> it makes
> >>> >>>>>>>>>> > resolving this for audit reporting a lot easier.
> >>> >>>>>>>>>> > Origin Region - agreed we can derive this if we have
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> but it
> >>> >>>>>>>>>> > makes resolving this for audit reporting a lot
> easier.
> >>> >>>>>>>>>> > Unique Message Identification (this is not the same
> as
> >>> >>>>>>> transaction
> >>> >>>>>>>>>> > tracing) - note offset and partition are not the
> same, as
> >>> >>>>> when we
> >>> >>>>>>>>> mirror or
> >>> >>>>>>>>>> > have for what ever system failure duplicate send,
> >>> >>>>>>>>>> > Custom Client wrapper version (where organizations
> have
> >>> to
> >>> >>>>> wrap
> >>> >>>>>>>>> the kafka
> >>> >>>>>>>>>> > client for added features) so we know what version
> of the
> >>> >>>>> wrapper
> >>> >>>>>>>>> is used
> >>> >>>>>>>>>> > Producer IP address (in case of clients being in our
> >>> vm/open
> >>> >>>>>>> stack
> >>> >>>>>>>>> infra
> >>> >>>>>>>>>> > where they can move around, producer id will stay the
> >>> same
> >>> >>> but
> >>> >>>>>>>>> this would
> >>> >>>>>>>>>> > change)
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Once and only once delivery case
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Using the same Message UUID for auditing we can
> achieve
> >>> this
> >>> >>>>>>> quite
> >>> >>>>>>>>> simply.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As per how some other brokers do this (cough qpid,
> >>> artemis)
> >>> >>>>>>>>> message uuid
> >>> >>>>>>>>>> > are used to dedupe where message is sent and
> produced but
> >>> >>> the
> >>> >>>>>>>>> client didn't
> >>> >>>>>>>>>> > receive the ack, and there for replays the send, by
> >>> having a
> >>> >>>>>>>>> unique message
> >>> >>>>>>>>>> > id per message, this can be filtered out, on
> consumers
> >>> where
> >>> >>>>>>>>> message
> >>> >>>>>>>>>> > delivery may occur twice for what ever reasons a
> message
> >>> >>> uuid
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>> > to remove duplicates being deliverd , like wise we
> can do
> >>> >>>>> this in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > mirrormakers so if we detect a dupe message we can
> avoid
> >>> >>>>>>>>> replicating it.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Cheers
> >>> >>>>>>>>>> > Mike
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
> >>> wrote:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Since this KIP affects message format, wire
> protocol,
> >>> >>>>> apis, I
> >>> >>>>>>>>> think
> >>> >>>>>>>>>> > it's
> >>> >>>>>>>>>> > worth spending a bit more time to nail down the
> >>> concrete
> >>> >>>>> use
> >>> >>>>>>>>> cases. It
> >>> >>>>>>>>>> > would be bad if we add this feature, but when
> start
> >>> >>>>>>>>> implementing it
> >>> >>>>>>>>>> > for say
> >>> >>>>>>>>>> > mirroring, we then realize that header is not the
> >>> best
> >>> >>>>>>>>> approach.
> >>> >>>>>>>>>> > Initially,
> >>> >>>>>>>>>> > I thought I was convinced of the use cases of
> headers
> >>> >>> and
> >>> >>>>> was
> >>> >>>>>>>>> trying to
> >>> >>>>>>>>>> > write down a few use cases to convince others.
> That's
> >>> >>>>> when I
> >>> >>>>>>>>> became
> >>> >>>>>>>>>> > less
> >>> >>>>>>>>>> > certain. For me to be convinced, I just want to
> see
> >>> two
> >>> >>>>>>> strong
> >>> >>>>>>>>> use
> >>> >>>>>>>>>> > cases
> >>> >>>>>>>>>> > (instead of 10 maybe use cases) in the
> third-party
> >>> >>> space.
> >>> >>>>> The
> >>> >>>>>>>>> reason is
> >>> >>>>>>>>>> > that when we discussed the use cases within a
> >>> company,
> >>> >>>>> often
> >>> >>>>>>>>> it ends
> >>> >>>>>>>>>> > with
> >>> >>>>>>>>>> > "we can't force everyone to use this standard
> since
> >>> we
> >>> >>> may
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>> > integrate with third-party tools".
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > At present, I am not sure why headers are useful
> for
> >>> >>>>> things
> >>> >>>>>>>>> like
> >>> >>>>>>>>>> > schemaId
> >>> >>>>>>>>>> > or encryption. In order to do anything useful to
> the
> >>> >>>>> value,
> >>> >>>>>>>>> one needs
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > know the schemaId or how data is encrypted, but
> >>> header
> >>> >>> is
> >>> >>>>>>>>> optional.
> >>> >>>>>>>>>> > But, I
> >>> >>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
> >>> >>> provides
> >>> >>>>>>> more
> >>> >>>>>>>>> details
> >>> >>>>>>>>>> > on
> >>> >>>>>>>>>> > the argument.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I am not very sure header is the best approach
> for
> >>> >>>>> mirroring
> >>> >>>>>>>>> either. If
> >>> >>>>>>>>>> > someone has thought about this more, I'd be
> happy to
> >>> >>> hear.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I can see the data lineage use case. I am just
> not
> >>> sure
> >>> >>>>> how
> >>> >>>>>>>>> widely
> >>> >>>>>>>>>> > applicable this is. If someone familiar with this
> >>> space
> >>> >>>>> can
> >>> >>>>>>>>> justify
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > a significant use case, say in the finance
> industry,
> >>> >>> this
> >>> >>>>>>>>> would be a
> >>> >>>>>>>>>> > strong
> >>> >>>>>>>>>> > use case.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I can see the auditing use case. I am just not
> sure
> >>> if a
> >>> >>>>>>> native
> >>> >>>>>>>>>> > producer id
> >>> >>>>>>>>>> > solves that problem. If there are additional
> metadata
> >>> >>>>> that's
> >>> >>>>>>>>> worth
> >>> >>>>>>>>>> > collecting but not covered by the producer id,
> that
> >>> >>> would
> >>> >>>>>>> make
> >>> >>>>>>>>> this a
> >>> >>>>>>>>>> > strong use case.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Thanks,
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Jun
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > > this KIP is about enabling headers, nothing
> more
> >>> >>> nothing
> >>> >>>>>>>>> less - so
> >>> >>>>>>>>>> > no,
> >>> >>>>>>>>>> > > broker-side use of headers is not in the KIP
> scope.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > obviously though, once you have headers
> potential
> >>> use
> >>> >>>>> cases
> >>> >>>>>>>>> could
> >>> >>>>>>>>>> > include
> >>> >>>>>>>>>> > > broker-side header-aware interceptors (which
> would
> >>> be
> >>> >>>>> the
> >>> >>>>>>>>> topic of
> >>> >>>>>>>>>> > other
> >>> >>>>>>>>>> > > future KIPs).
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > a trivially clear use case (to me) would be
> using
> >>> such
> >>> >>>>>>>>> broker-side
> >>> >>>>>>>>>> > > interceptors to enforce compliance with
> >>> organizational
> >>> >>>>>>>>> policies - it
> >>> >>>>>>>>>> > would
> >>> >>>>>>>>>> > > make our SREs lives much easier if instead of
> >>> >>>>> retroactively
> >>> >>>>>>>>>> > discovering
> >>> >>>>>>>>>> > > "rogue" topics/users those messages would have
> been
> >>> >>>>>>> rejected
> >>> >>>>>>>>>> > up-front.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > the kafka broker code is lacking any such
> >>> >>> extensibility
> >>> >>>>>>>>> support
> >>> >>>>>>>>>> > (beyond
> >>> >>>>>>>>>> > > maybe authorizer) which is why these use cases
> were
> >>> >>> left
> >>> >>>>>>> out
> >>> >>>>>>>>> of the
> >>> >>>>>>>>>> > "case
> >>> >>>>>>>>>> > > for headers" doc - broker extensibility is a
> >>> separate
> >>> >>>>>>>>> discussion.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
> >>> >>>>>>>>> gwen@confluent.io>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > > Woah, I wasn't aware this is something we'll
> do.
> >>> It
> >>> >>>>>>> wasn't
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > KIP,
> >>> >>>>>>>>>> > > > right?
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > I guess we could do it the same way ACLs
> >>> currently
> >>> >>>>> work.
> >>> >>>>>>>>>> > > > I had in mind something that will allow
> admins to
> >>> >>>>> apply
> >>> >>>>>>>>> rules to
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > new create/delete/config topic APIs. So Todd
> can
> >>> >>>>> decide
> >>> >>>>>>> to
> >>> >>>>>>>>> reject
> >>> >>>>>>>>>> > > > "create topic" requests that ask for more
> than 40
> >>> >>>>>>>>> partitions, or
> >>> >>>>>>>>>> > > > require exactly 3 replicas, or no more than
> 50GB
> >>> >>>>>>> partition
> >>> >>>>>>>>> size,
> >>> >>>>>>>>>> > etc.
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are
> planning
> >>> to
> >>> >>>>> apply
> >>> >>>>>>>>> more
> >>> >>>>>>>>>> > rules
> >>> >>>>>>>>>> > > > to requests (and I think we should), we may
> want
> >>> a
> >>> >>> bit
> >>> >>>>>>>>> more generic
> >>> >>>>>>>>>> > > > design around that.
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > wrote:
> >>> >>>>>>>>>> > > > > "wouldn't you be in the business of making
> sure
> >>> >>>>>>> everyone
> >>> >>>>>>>>> uses
> >>> >>>>>>>>>> > them
> >>> >>>>>>>>>> > > > > properly?"
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > > thats where a broker-side plugin would come
> >>> handy
> >>> >>> -
> >>> >>>>> any
> >>> >>>>>>>>> incoming
> >>> >>>>>>>>>> > > message
> >>> >>>>>>>>>> > > > > that does not conform to org policy (read -
> >>> does
> >>> >>> not
> >>> >>>>>>>>> have the
> >>> >>>>>>>>>> > proper
> >>> >>>>>>>>>> > > > > headers) gets thrown out (with an error
> >>> returned
> >>> >>> to
> >>> >>>>>>> user)
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> Palino <
> >>> >>>>>>>>> tpalino@gmail.com>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > >> Come on, I've done at least 2 talks on
> this
> >>> one
> >>> >>> :)
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> Producing counts to a topic is part of
> it, but
> >>> >>>>> that's
> >>> >>>>>>>>> only
> >>> >>>>>>>>>> > part. So
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> count you have 100 messages in topic A.
> When
> >>> you
> >>> >>>>>>> mirror
> >>> >>>>>>>>> topic A
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > > another
> >>> >>>>>>>>>> > > > >> cluster, you have 99 messages. Where was
> your
> >>> >>>>> problem?
> >>> >>>>>>>>> Or
> >>> >>>>>>>>>> > worse, you
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> 100 messages, but one producer duplicated
> >>> >>> messages
> >>> >>>>> and
> >>> >>>>>>>>> another
> >>> >>>>>>>>>> > one
> >>> >>>>>>>>>> > > lost
> >>> >>>>>>>>>> > > > >> messages. You need details about where the
> >>> >>> message
> >>> >>>>>>> came
> >>> >>>>>>>>> from in
> >>> >>>>>>>>>> > order
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> pinpoint problems when they happen. Source
> >>> >>> producer
> >>> >>>>>>>>> info, where
> >>> >>>>>>>>>> > it was
> >>> >>>>>>>>>> > > > >> produced into your infrastructure, and
> when it
> >>> >>> was
> >>> >>>>>>>>> produced.
> >>> >>>>>>>>>> > This
> >>> >>>>>>>>>> > > > requires
> >>> >>>>>>>>>> > > > >> you to add the information to the message.
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> And yes, you still need to maintain your
> >>> clients.
> >>> >>>>> So
> >>> >>>>>>>>> maybe my
> >>> >>>>>>>>>> > original
> >>> >>>>>>>>>> > > > >> example was not the best. My thoughts on
> not
> >>> >>>>> wanting
> >>> >>>>>>> to
> >>> >>>>>>>>> be
> >>> >>>>>>>>>> > responsible
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> message formats stands, because that's
> very
> >>> much
> >>> >>>>>>>>> separate from
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > client.
> >>> >>>>>>>>>> > > > >> As you know, we have our own internal
> client
> >>> >>>>> library
> >>> >>>>>>>>> that can
> >>> >>>>>>>>>> > insert
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> right headers, and right now inserts the
> right
> >>> >>>>> audit
> >>> >>>>>>>>>> > information into
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> message fields. If they exist, and
> assuming
> >>> the
> >>> >>>>>>> message
> >>> >>>>>>>>> is Avro
> >>> >>>>>>>>>> > > encoded.
> >>> >>>>>>>>>> > > > >> What if someone wants to use JSON instead
> for
> >>> a
> >>> >>>>> good
> >>> >>>>>>>>> reason?
> >>> >>>>>>>>>> > What if
> >>> >>>>>>>>>> > > > user X
> >>> >>>>>>>>>> > > > >> wants to encrypt messages, but user Y does
> >>> not?
> >>> >>>>>>>>> Maintaining the
> >>> >>>>>>>>>> > client
> >>> >>>>>>>>>> > > > >> library is still much easier than
> maintaining
> >>> the
> >>> >>>>>>>>> message
> >>> >>>>>>>>>> > formats.
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> -Todd
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> Shapira <
> >>> >>>>>>>>> gwen@confluent.io
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > wrote:
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> > Based on your last sentence, consider me
> >>> >>>>> convinced
> >>> >>>>>>> :)
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > I get why headers are critical for
> Mirroring
> >>> >>> (you
> >>> >>>>>>>>> need tags to
> >>> >>>>>>>>>> > > prevent
> >>> >>>>>>>>>> > > > >> > loops and sometimes to route messages
> to the
> >>> >>>>> correct
> >>> >>>>>>>>>> > destination).
> >>> >>>>>>>>>> > > > >> > But why do you need headers to audit?
> We are
> >>> >>>>>>> auditing
> >>> >>>>>>>>> by
> >>> >>>>>>>>>> > producing
> >>> >>>>>>>>>> > > > >> > counts to a side topic (and I was under
> the
> >>> >>>>>>>>> impression you do
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > >> > same), so we never need to modify the
> >>> message.
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > Another thing - after we added headers,
> >>> >>> wouldn't
> >>> >>>>> you
> >>> >>>>>>>>> be in the
> >>> >>>>>>>>>> > > > >> > business of making sure everyone uses
> them
> >>> >>>>> properly?
> >>> >>>>>>>>> Making
> >>> >>>>>>>>>> > sure
> >>> >>>>>>>>>> > > > >> > everyone includes the right headers you
> >>> need,
> >>> >>> not
> >>> >>>>>>>>> using the
> >>> >>>>>>>>>> > header
> >>> >>>>>>>>>> > > > >> > names you intend to use, etc. I don't
> think
> >>> the
> >>> >>>>>>>>> "policing"
> >>> >>>>>>>>>> > business
> >>> >>>>>>>>>> > > > >> > will ever go away.
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> Palino
> >>> <
> >>> >>>>>>>>>> > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > wrote:
> >>> >>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very
> happy
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>>> > workaround.
> >>> >>>>>>>>>> > > Avro
> >>> >>>>>>>>>> > > > >> means
> >>> >>>>>>>>>> > > > >> > > that I have to be concerned with the
> >>> format
> >>> >>> of
> >>> >>>>> the
> >>> >>>>>>>>> messages
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > order to
> >>> >>>>>>>>>> > > > >> > run
> >>> >>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring,
> >>> etc.).
> >>> >>>>> That
> >>> >>>>>>>>> means
> >>> >>>>>>>>>> > that I
> >>> >>>>>>>>>> > > have
> >>> >>>>>>>>>> > > > to
> >>> >>>>>>>>>> > > > >> > > handle the schemas, and I have to
> enforce
> >>> >>> rules
> >>> >>>>>>>>> about good
> >>> >>>>>>>>>> > > formats.
> >>> >>>>>>>>>> > > > >> This
> >>> >>>>>>>>>> > > > >> > is
> >>> >>>>>>>>>> > > > >> > > not something I want to be in the
> business
> >>> >>> of,
> >>> >>>>>>>>> because I
> >>> >>>>>>>>>> > should be
> >>> >>>>>>>>>> > > > able
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > > run a service infrastructure without
> >>> needing
> >>> >>>>> to be
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > weeds of
> >>> >>>>>>>>>> > > > >> > dealing
> >>> >>>>>>>>>> > > > >> > > with customer data formats.
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > Trust me, a sizable portion of my
> support
> >>> >>> time
> >>> >>>>> is
> >>> >>>>>>>>> spent
> >>> >>>>>>>>>> > dealing
> >>> >>>>>>>>>> > > with
> >>> >>>>>>>>>> > > > >> > schema
> >>> >>>>>>>>>> > > > >> > > issues. I really would like to get
> away
> >>> from
> >>> >>>>> that.
> >>> >>>>>>>>> Maybe
> >>> >>>>>>>>>> > I'd have
> >>> >>>>>>>>>> > > > more
> >>> >>>>>>>>>> > > > >> > time
> >>> >>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > -Todd
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
> >>> Shapira <
> >>> >>>>>>>>>> > gwen@confluent.io>
> >>> >>>>>>>>>> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
> >>> >>>>> workarounds
> >>> >>>>>>>>> (Avro
> >>> >>>>>>>>>> > container
> >>> >>>>>>>>>> > > > >> > >> format), so I'm not too excited
> about the
> >>> >>>>> extra
> >>> >>>>>>>>> work
> >>> >>>>>>>>>> > required to
> >>> >>>>>>>>>> > > do
> >>> >>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't
> >>> mind it
> >>> >>>>> if
> >>> >>>>>>>>> you do
> >>> >>>>>>>>>> > it...
> >>> >>>>>>>>>> > > > >> > >> I think the Apache convention for
> "good
> >>> >>> idea,
> >>> >>>>> but
> >>> >>>>>>>>> not
> >>> >>>>>>>>>> > willing to
> >>> >>>>>>>>>> > > > put
> >>> >>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway,
> >>> that's
> >>> >>>>> what
> >>> >>>>>>> I
> >>> >>>>>>>>> was
> >>> >>>>>>>>>> > trying to
> >>> >>>>>>>>>> > > > >> > >> convey :)
> >>> >>>>>>>>>> > > > >> > >>
> >>> >>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
> >>> Palino
> >>> >>> <
> >>> >>>>>>>>>> > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> > Well I guess my question for you,
> >>> then, is
> >>> >>>>> what
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > holding you
> >>> >>>>>>>>>> > > > back
> >>> >>>>>>>>>> > > > >> > from
> >>> >>>>>>>>>> > > > >> > >> > full support for headers? What's
> the
> >>> bit
> >>> >>>>> that
> >>> >>>>>>>>> you're
> >>> >>>>>>>>>> > missing
> >>> >>>>>>>>>> > > that
> >>> >>>>>>>>>> > > > >> has
> >>> >>>>>>>>>> > > > >> > you
> >>> >>>>>>>>>> > > > >> > >> > under a full +1?
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> > -Todd
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> Gwen
> >>> >>>>> Shapira <
> >>> >>>>>>>>>> > > gwen@confluent.io>
> >>> >>>>>>>>>> > > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> I know why people who support
> headers
> >>> >>>>> support
> >>> >>>>>>>>> them, and
> >>> >>>>>>>>>> > I've
> >>> >>>>>>>>>> > > > seen
> >>> >>>>>>>>>> > > > >> > what
> >>> >>>>>>>>>> > > > >> > >> >> the discussion is like.
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> This is why I'm asking people who
> are
> >>> >>>>> against
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > > > (especially
> >>> >>>>>>>>>> > > > >> > >> >> committers) what will make them
> change
> >>> >>>>> their
> >>> >>>>>>>>> mind - so
> >>> >>>>>>>>>> > we can
> >>> >>>>>>>>>> > > > get
> >>> >>>>>>>>>> > > > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> part over one way or another.
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at
> >>> Radai,
> >>> >>>>> Jun
> >>> >>>>>>>>> or you
> >>> >>>>>>>>>> > (Todd)...
> >>> >>>>>>>>>> > > > I am
> >>> >>>>>>>>>> > > > >> > >> >> just looking for something
> concrete we
> >>> >>> can
> >>> >>>>> do
> >>> >>>>>>>>> to move
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > >> discussion
> >>> >>>>>>>>>> > > > >> > >> >> along to the yummy design details
> >>> (which
> >>> >>> is
> >>> >>>>>>> the
> >>> >>>>>>>>>> > argument I
> >>> >>>>>>>>>> > > > really
> >>> >>>>>>>>>> > > > >> am
> >>> >>>>>>>>>> > > > >> > >> >> looking forward to).
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM,
> Todd
> >>> >>>>> Palino <
> >>> >>>>>>>>>> > > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even
> >>> though
> >>> >>>>> I'm
> >>> >>>>>>>>> not a
> >>> >>>>>>>>>> > > > committer)...
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > I have always been a strong
> >>> supporter
> >>> >>> of
> >>> >>>>>>>>> introducing
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > concept
> >>> >>>>>>>>>> > > > >> > of an
> >>> >>>>>>>>>> > > > >> > >> >> > envelope to messages, which
> headers
> >>> >>>>>>>>> accomplishes. The
> >>> >>>>>>>>>> > > message
> >>> >>>>>>>>>> > > > key
> >>> >>>>>>>>>> > > > >> > is
> >>> >>>>>>>>>> > > > >> > >> >> > already an example of a piece of
> >>> >>> envelope
> >>> >>>>>>>>>> > information. By
> >>> >>>>>>>>>> > > > >> > providing a
> >>> >>>>>>>>>> > > > >> > >> >> means
> >>> >>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself,
> and
> >>> not
> >>> >>>>>>>>> relying on
> >>> >>>>>>>>>> > use-case
> >>> >>>>>>>>>> > > > >> > specific
> >>> >>>>>>>>>> > > > >> > >> >> > implementations, you make it
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>>> > components to
> >>> >>>>>>>>>> > > > >> > >> interoperate.
> >>> >>>>>>>>>> > > > >> > >> >> It
> >>> >>>>>>>>>> > > > >> > >> >> > simplifies development of all
> these
> >>> >>>>> things
> >>> >>>>>>>>> (message
> >>> >>>>>>>>>> > routing,
> >>> >>>>>>>>>> > > > >> > auditing,
> >>> >>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each
> one
> >>> does
> >>> >>>>> not
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>> > reinvent
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> wheel.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > It also makes it much easier
> from a
> >>> >>>>> client
> >>> >>>>>>>>> point of
> >>> >>>>>>>>>> > view if
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> headers
> >>> >>>>>>>>>> > > > >> > >> >> are
> >>> >>>>>>>>>> > > > >> > >> >> > defined as part of the protocol
> >>> and/or
> >>> >>>>>>>>> message format
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > general
> >>> >>>>>>>>>> > > > >> > >> because
> >>> >>>>>>>>>> > > > >> > >> >> > you can easily produce and
> consume
> >>> >>>>> messages
> >>> >>>>>>>>> without
> >>> >>>>>>>>>> > having
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> take
> >>> >>>>>>>>>> > > > >> > >> into
> >>> >>>>>>>>>> > > > >> > >> >> > account specific cases. For
> >>> example, I
> >>> >>>>> want
> >>> >>>>>>>>> to route
> >>> >>>>>>>>>> > > messages,
> >>> >>>>>>>>>> > > > >> but
> >>> >>>>>>>>>> > > > >> > >> >> client A
> >>> >>>>>>>>>> > > > >> > >> >> > doesn't support the way audit
> >>> >>> implemented
> >>> >>>>>>>>> headers, and
> >>> >>>>>>>>>> > > client
> >>> >>>>>>>>>> > > > B
> >>> >>>>>>>>>> > > > >> > >> doesn't
> >>> >>>>>>>>>> > > > >> > >> >> > support the way encryption or
> >>> routing
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>> > headers,
> >>> >>>>>>>>>> > > so
> >>> >>>>>>>>>> > > > now
> >>> >>>>>>>>>> > > > >> > my
> >>> >>>>>>>>>> > > > >> > >> >> > application has to create some
> >>> really
> >>> >>>>>>> fragile
> >>> >>>>>>>>> (my
> >>> >>>>>>>>>> > > autocorrect
> >>> >>>>>>>>>> > > > >> just
> >>> >>>>>>>>>> > > > >> > >> tried
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> > make that "tragic", which is
> >>> probably
> >>> >>>>>>>>> appropriate
> >>> >>>>>>>>>> > too) code
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> > strip
> >>> >>>>>>>>>> > > > >> > >> >> > everything off, rather than just
> >>> >>>>> consuming
> >>> >>>>>>> the
> >>> >>>>>>>>>> > messages,
> >>> >>>>>>>>>> > > > picking
> >>> >>>>>>>>>> > > > >> > out
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> 1
> >>> >>>>>>>>>> > > > >> > >> >> > or 2 headers it's interested
> in, and
> >>> >>>>>>>>> performing its
> >>> >>>>>>>>>> > > function.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has
> been
> >>> >>> going
> >>> >>>>> on
> >>> >>>>>>>>> for a
> >>> >>>>>>>>>> > long time,
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > it's
> >>> >>>>>>>>>> > > > >> > >> >> > always "Oh, you came up with 2
> use
> >>> >>> cases,
> >>> >>>>>>> and
> >>> >>>>>>>>> yeah,
> >>> >>>>>>>>>> > those
> >>> >>>>>>>>>> > > use
> >>> >>>>>>>>>> > > > >> cases
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> > real things that someone would
> want
> >>> to
> >>> >>>>> do.
> >>> >>>>>>>>> Here's an
> >>> >>>>>>>>>> > > alternate
> >>> >>>>>>>>>> > > > >> way
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> >> > implement them so let's not do
> >>> >>> headers."
> >>> >>>>> If
> >>> >>>>>>>>> we have a
> >>> >>>>>>>>>> > few
> >>> >>>>>>>>>> > > use
> >>> >>>>>>>>>> > > > >> cases
> >>> >>>>>>>>>> > > > >> > >> that
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> > actually came up with, you can
> be
> >>> sure
> >>> >>>>> that
> >>> >>>>>>>>> over the
> >>> >>>>>>>>>> > next
> >>> >>>>>>>>>> > > year
> >>> >>>>>>>>>> > > > >> > >> there's a
> >>> >>>>>>>>>> > > > >> > >> >> > dozen others that we didn't
> think of
> >>> >>> that
> >>> >>>>>>>>> someone
> >>> >>>>>>>>>> > would like
> >>> >>>>>>>>>> > > > to
> >>> >>>>>>>>>> > > > >> > do. I
> >>> >>>>>>>>>> > > > >> > >> >> > really think it's time to stop
> >>> >>> rehashing
> >>> >>>>>>> this
> >>> >>>>>>>>>> > discussion and
> >>> >>>>>>>>>> > > > >> > instead
> >>> >>>>>>>>>> > > > >> > >> >> focus
> >>> >>>>>>>>>> > > > >> > >> >> > on a workable standard that we
> can
> >>> >>> adopt.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > -Todd
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM,
> Todd
> >>> >>>>> Palino
> >>> >>>>>>> <
> >>> >>>>>>>>>> > > > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >> C. per message encryption
> >>> >>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach
> is
> >>> that
> >>> >>>>> this
> >>> >>>>>>>>>> > significantly
> >>> >>>>>>>>>> > > > reduce
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression,
> >>> which
> >>> >>>>>>> happens
> >>> >>>>>>>>> on a
> >>> >>>>>>>>>> > set of
> >>> >>>>>>>>>> > > > >> > serialized
> >>> >>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to
> >>> enable
> >>> >>>>> SSL
> >>> >>>>>>>>> for wire
> >>> >>>>>>>>>> > > > encryption
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> rely
> >>> >>>>>>>>>> > > > >> > >> >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS)
> >>> for at
> >>> >>>>> rest
> >>> >>>>>>>>>> > encryption.
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient.
> While
> >>> >>> this
> >>> >>>>>>> does
> >>> >>>>>>>>> cover
> >>> >>>>>>>>>> > the case
> >>> >>>>>>>>>> > > > of
> >>> >>>>>>>>>> > > > >> > >> removing
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >> drive from the system, it will
> not
> >>> >>>>> satisfy
> >>> >>>>>>>>> most
> >>> >>>>>>>>>> > compliance
> >>> >>>>>>>>>> > > > >> > >> requirements
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever
> has
> >>> >>>>> access to
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > broker
> >>> >>>>>>>>>> > > itself
> >>> >>>>>>>>>> > > > >> > still
> >>> >>>>>>>>>> > > > >> > >> has
> >>> >>>>>>>>>> > > > >> > >> >> >> access to the unencrypted
> data. For
> >>> >>>>>>>>> end-to-end
> >>> >>>>>>>>>> > encryption
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> > need to
> >>> >>>>>>>>>> > > > >> > >> >> >> encrypt at the producer,
> before it
> >>> >>>>> enters
> >>> >>>>>>> the
> >>> >>>>>>>>>> > system, and
> >>> >>>>>>>>>> > > > >> decrypt
> >>> >>>>>>>>>> > > > >> > at
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the
> >>> system.
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> -Todd
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM,
> >>> radai
> >>> >>> <
> >>> >>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> another big plus of headers
> in the
> >>> >>>>>>> protocol
> >>> >>>>>>>>> is that
> >>> >>>>>>>>>> > it
> >>> >>>>>>>>>> > > would
> >>> >>>>>>>>>> > > > >> > enable
> >>> >>>>>>>>>> > > > >> > >> >> rapid
> >>> >>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of
> core
> >>> >>>>> kafka
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>> > reduce
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> number of
> >>> >>>>>>>>>> > > > >> > >> >> >>> future wire format changes
> >>> required.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a
> KIP
> >>> >>>>>>> represents
> >>> >>>>>>>>> use
> >>> >>>>>>>>>> > cases that
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> > not
> >>> >>>>>>>>>> > > > >> > >> 100%
> >>> >>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and
> some of
> >>> >>> them
> >>> >>>>>>>>> require
> >>> >>>>>>>>>> > rather
> >>> >>>>>>>>>> > > > invasive
> >>> >>>>>>>>>> > > > >> > wire
> >>> >>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a
> good
> >>> >>> recent
> >>> >>>>>>>>> example of
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > > > >> > kip-98.
> >>> >>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is
> expected
> >>> to
> >>> >>> be
> >>> >>>>> a
> >>> >>>>>>>>> very small
> >>> >>>>>>>>>> > > > fraction of
> >>> >>>>>>>>>> > > > >> > >> total
> >>> >>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes
> are
> >>> >>>>> invasive.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> every such wire format change
> >>> >>>>> translates
> >>> >>>>>>>>> into
> >>> >>>>>>>>>> > painful and
> >>> >>>>>>>>>> > > > slow
> >>> >>>>>>>>>> > > > >> > >> >> adoption of
> >>> >>>>>>>>>> > > > >> > >> >> >>> new versions.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
> >>> >>>>> currently
> >>> >>>>>>> in
> >>> >>>>>>>>> KIPs
> >>> >>>>>>>>>> > could be
> >>> >>>>>>>>>> > > > "spun
> >>> >>>>>>>>>> > > > >> > out"
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
> >>> >>>>> transmitting
> >>> >>>>>>>>> data over
> >>> >>>>>>>>>> > > > headers.
> >>> >>>>>>>>>> > > > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> keep the core wire format
> >>> stable(r),
> >>> >>>>> core
> >>> >>>>>>>>> codebase
> >>> >>>>>>>>>> > > smaller,
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > >> avoid
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats
> sometimes
> >>> >>>>> required
> >>> >>>>>>>>> to prove
> >>> >>>>>>>>>> > a
> >>> >>>>>>>>>> > > > certain
> >>> >>>>>>>>>> > > > >> > >> feature
> >>> >>>>>>>>>> > > > >> > >> >> is
> >>> >>>>>>>>>> > > > >> > >> >> >>> useful enough for a
> wide-enough
> >>> >>>>> audience
> >>> >>>>>>> to
> >>> >>>>>>>>> warrant
> >>> >>>>>>>>>> > a wire
> >>> >>>>>>>>>> > > > >> format
> >>> >>>>>>>>>> > > > >> > >> >> change
> >>> >>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
> beyond
> >>> >>>>> "mere"
> >>> >>>>>>>>> wire format
> >>> >>>>>>>>>> > > changes
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > im
> >>> >>>>>>>>>> > > > >> > >> not
> >>> >>>>>>>>>> > > > >> > >> >> >>> saying it could have been
> >>> completely
> >>> >>>>> done
> >>> >>>>>>>>> with
> >>> >>>>>>>>>> > headers,
> >>> >>>>>>>>>> > > but
> >>> >>>>>>>>>> > > > >> > >> >> exactly-once
> >>> >>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20
> AM,
> >>> Gwen
> >>> >>>>>>>>> Shapira <
> >>> >>>>>>>>>> > > > >> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> 10:24 AM,
> >>> >>>>> radai <
> >>> >>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
> >>> >>>>> organization,
> >>> >>>>>>>>> one could
> >>> >>>>>>>>>> > > always
> >>> >>>>>>>>>> > > > use
> >>> >>>>>>>>>> > > > >> > >> other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers"
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
> >>> >>>>>>> traditionally
> >>> >>>>>>>>> done
> >>> >>>>>>>>>> > but there
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> > now
> >>> >>>>>>>>>> > > > >> > >> >> cases
> >>> >>>>>>>>>> > > > >> > >> >> >>> > (read
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is
> not
> >>> >>>>>>> acceptable.
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > makes
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > >> useful
> >>> >>>>>>>>>> > > > >> > >> >> even
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for
> cases
> >>> >>> where
> >>> >>>>>>>>>> > > > one-container-fits-all
> >>> >>>>>>>>>> > > > >> > cannot
> >>> >>>>>>>>>> > > > >> > >> >> >>> apply.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use
> >>> cases
> >>> >>>>>>> listed,
> >>> >>>>>>>>> i dont
> >>> >>>>>>>>>> > want
> >>> >>>>>>>>>> > > > this to
> >>> >>>>>>>>>> > > > >> > >> devolve
> >>> >>>>>>>>>> > > > >> > >> >> >>> to a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular
> use
> >>> >>> cases
> >>> >>>>> - i
> >>> >>>>>>>>> think its
> >>> >>>>>>>>>> > > enough
> >>> >>>>>>>>>> > > > >> that
> >>> >>>>>>>>>> > > > >> > >> some
> >>> >>>>>>>>>> > > > >> > >> >> of
> >>> >>>>>>>>>> > > > >> > >> >> >>> them
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I think a main point of
> >>> contention
> >>> >>> is
> >>> >>>>>>>>> that: We
> >>> >>>>>>>>>> > > identified
> >>> >>>>>>>>>> > > > few
> >>> >>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are
> >>> useful,
> >>> >>>>> do
> >>> >>>>>>> we
> >>> >>>>>>>>> want
> >>> >>>>>>>>>> > Kafka to
> >>> >>>>>>>>>> > > > be a
> >>> >>>>>>>>>> > > > >> > >> system
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that supports those
> use-cases?
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
> >>> >>>>> record-level
> >>> >>>>>>>>> lineage
> >>> >>>>>>>>>> > is
> >>> >>>>>>>>>> > > though
> >>> >>>>>>>>>> > > > >> > since
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > overhead could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > be significant."
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record
> >>> level
> >>> >>>>>>>>> lineage. I
> >>> >>>>>>>>>> > don't
> >>> >>>>>>>>>> > > think
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > was
> >>> >>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think
> it
> >>> is
> >>> >>>>> safe
> >>> >>>>>>> to
> >>> >>>>>>>>> assume
> >>> >>>>>>>>>> > that
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > NSA
> >>> >>>>>>>>>> > > > >> > >> needed
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also
> know
> >>> >>> that
> >>> >>>>>>>>> certain
> >>> >>>>>>>>>> > financial
> >>> >>>>>>>>>> > > > >> > institutes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
> >>> >>> records
> >>> >>>>> at
> >>> >>>>>>> a
> >>> >>>>>>>>> record
> >>> >>>>>>>>>> > level
> >>> >>>>>>>>>> > > and
> >>> >>>>>>>>>> > > > >> > there
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > federal regulations that
> >>> absolutely
> >>> >>>>>>>>> require
> >>> >>>>>>>>>> > this. They
> >>> >>>>>>>>>> > > > also
> >>> >>>>>>>>>> > > > >> > need
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
> >>> >>>>> "touches"
> >>> >>>>>>> the
> >>> >>>>>>>>>> > messages and
> >>> >>>>>>>>>> > > > >> either
> >>> >>>>>>>>>> > > > >> > >> reads
> >>> >>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't
> have
> >>> >>>>>>> possibly
> >>> >>>>>>>>>> > modified the
> >>> >>>>>>>>>> > > > >> payload
> >>> >>>>>>>>>> > > > >> > >> >> itself.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > They use record level
> >>> encryption to
> >>> >>>>> do
> >>> >>>>>>>>> that -
> >>> >>>>>>>>>> > apps can
> >>> >>>>>>>>>> > > > read
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers
> but
> >>> >>> can't
> >>> >>>>>>>>> touch the
> >>> >>>>>>>>>> > payload.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those
> are
> >>> >>> corner
> >>> >>>>>>>>> cases and
> >>> >>>>>>>>>> > not worth
> >>> >>>>>>>>>> > > > >> adding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they
> >>> should
> >>> >>>>> use a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> > > pubsub
> >>> >>>>>>>>>> > > > >> > message
> >>> >>>>>>>>>> > > > >> > >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the
> other
> >>> 1000
> >>> >>>>> that
> >>> >>>>>>>>> cater
> >>> >>>>>>>>>> > > > specifically
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > financial industry).
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a
> catch
> >>> 22:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific
> >>> use-case,
> >>> >>>>>>>>> someone can
> >>> >>>>>>>>>> > always
> >>> >>>>>>>>>> > > say
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > isn't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > interesting enough for
> Kafka.
> >>> If we
> >>> >>>>>>>>> discuss more
> >>> >>>>>>>>>> > general
> >>> >>>>>>>>>> > > > >> > trends,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we
> are not
> >>> >>> sure
> >>> >>>>>>> any
> >>> >>>>>>>>> of them
> >>> >>>>>>>>>> > really
> >>> >>>>>>>>>> > > > >> needs
> >>> >>>>>>>>>> > > > >> > >> >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just
> hand
> >>> >>>>> waving
> >>> >>>>>>>>> and not
> >>> >>>>>>>>>> > > > interesting.".
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I think discussing
> use-cases in
> >>> >>>>>>> specifics
> >>> >>>>>>>>> is super
> >>> >>>>>>>>>> > > > important
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> decide
> >>> >>>>>>>>>> > > > >> > >> >> >>> > implementation details for
> >>> headers
> >>> >>>>> (my
> >>> >>>>>>>>> use-cases
> >>> >>>>>>>>>> > lean
> >>> >>>>>>>>>> > > > toward
> >>> >>>>>>>>>> > > > >> > >> >> numerical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and
> object
> >>> >>>>> values,
> >>> >>>>>>>>> others
> >>> >>>>>>>>>> > differ),
> >>> >>>>>>>>>> > > > but I
> >>> >>>>>>>>>> > > > >> > >> think
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > need to answer the general
> "Are
> >>> we
> >>> >>>>> going
> >>> >>>>>>>>> to have
> >>> >>>>>>>>>> > > headers"
> >>> >>>>>>>>>> > > > >> > question
> >>> >>>>>>>>>> > > > >> > >> >> >>> > first.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the
> other
> >>> >>>>>>>>> committers in the
> >>> >>>>>>>>>> > > > discussion:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > What would it take to
> convince
> >>> you
> >>> >>>>> that
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > in Kafka
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> a
> >>> >>>>>>>>>> > > > >> > >> good
> >>> >>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can
> move
> >>> >>> ahead
> >>> >>>>>>> and
> >>> >>>>>>>>> try to
> >>> >>>>>>>>>> > agree
> >>> >>>>>>>>>> > > on
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> details?
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving
> the
> >>> goal
> >>> >>>>>>> posts
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > > > truly
> >>> >>>>>>>>>> > > > >> > >> >> exhausting.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly
> support
> >>> >>>>> adding
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > to Kafka
> >>> >>>>>>>>>> > > > >> > (+0.5?).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > The community can continue
> to
> >>> find
> >>> >>>>>>>>> workarounds to
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > issue
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > are some benefits to
> keeping the
> >>> >>>>> message
> >>> >>>>>>>>> format
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > > clients
> >>> >>>>>>>>>> > > > >> > >> simpler.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of
> >>> headers
> >>> >>>>> to
> >>> >>>>>>>>> many
> >>> >>>>>>>>>> > use-cases
> >>> >>>>>>>>>> > > and
> >>> >>>>>>>>>> > > > if
> >>> >>>>>>>>>> > > > >> we
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> > find a good and generally
> useful
> >>> >>> way
> >>> >>>>> to
> >>> >>>>>>>>> add it to
> >>> >>>>>>>>>> > Kafka,
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > will
> >>> >>>>>>>>>> > > > >> > >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for
> many -
> >>> >>> worthy
> >>> >>>>>>>>> goal in my
> >>> >>>>>>>>>> > eyes.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible,
> but:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> polyglot
> >>> >>>>>>>>>> > topics.
> >>> >>>>>>>>>> > > > >> > >> especially if
> >>> >>>>>>>>>> > > > >> > >> >> >>> kafka
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
> >>> >>> something
> >>> >>>>>>> else.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would
> make
> >>> it
> >>> >>>>>>> harder
> >>> >>>>>>>>> to write
> >>> >>>>>>>>>> > > > portable
> >>> >>>>>>>>>> > > > >> > >> consumer
> >>> >>>>>>>>>> > > > >> > >> >> >>> code.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would
> mess
> >>> >>> with
> >>> >>>>>>>>> locality of
> >>> >>>>>>>>>> > > > consumption
> >>> >>>>>>>>>> > > > >> > >> >> >>> guarantees.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
> >>> >>>>>>>>> lineage/metadata is
> >>> >>>>>>>>>> > > > >> > >> billing/chargeback.
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough
> to
> >>> >>> simply
> >>> >>>>>>>>> record the
> >>> >>>>>>>>>> > point
> >>> >>>>>>>>>> > > of
> >>> >>>>>>>>>> > > > >> > origin,
> >>> >>>>>>>>>> > > > >> > >> but
> >>> >>>>>>>>>> > > > >> > >> >> >>> every
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think
> mirror
> >>> >>>>> maker)
> >>> >>>>>>>>> must also
> >>> >>>>>>>>>> > add a
> >>> >>>>>>>>>> > > > >> record
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> >> form a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing
> on
> >>> top
> >>> >>> of
> >>> >>>>>>>>> kafka - i
> >>> >>>>>>>>>> > know
> >>> >>>>>>>>>> > > samza
> >>> >>>>>>>>>> > > > >> has a
> >>> >>>>>>>>>> > > > >> > >> >> metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > map
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
> >>> >>>>> addition to
> >>> >>>>>>>>> user
> >>> >>>>>>>>>> > values.
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > perfect
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at
> 6:50
> >>> PM,
> >>> >>>>> Jun
> >>> >>>>>>>>> Rao <
> >>> >>>>>>>>>> > > > jun@confluent.io
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the
> first
> >>> two
> >>> >>>>>>>>> questions, it
> >>> >>>>>>>>>> > would
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > >> > helpful
> >>> >>>>>>>>>> > > > >> > >> >> if we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong
> use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > space
> >>> >>>>>>>>>> > > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases
> >>> within an
> >>> >>>>>>>>> organization,
> >>> >>>>>>>>>> > one
> >>> >>>>>>>>>> > > > could
> >>> >>>>>>>>>> > > > >> > always
> >>> >>>>>>>>>> > > > >> > >> >> use
> >>> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers to
> >>> >>>>>>>>>> > get
> >>> >>>>>>>>>> > > > around
> >>> >>>>>>>>>> > > > >> w/o
> >>> >>>>>>>>>> > > > >> > >> >> >>> headers. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> went through the use
> cases in
> >>> >>> the
> >>> >>>>> KIP
> >>> >>>>>>>>> and in
> >>> >>>>>>>>>> > Radai's
> >>> >>>>>>>>>> > > > wiki
> >>> >>>>>>>>>> > > > >> (
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> https://cwiki.apache.org/confl
> >>> >>>>>>>>>> > uence/display/KAFKA/A+
> >>> >>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> ).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> The following are the
> ones
> >>> that
> >>> >>>>> that
> >>> >>>>>>> I
> >>> >>>>>>>>>> > understand and
> >>> >>>>>>>>>> > > > >> could
> >>> >>>>>>>>>> > > > >> > be
> >>> >>>>>>>>>> > > > >> > >> in
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case
> >>> category.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
> >>> >>>>>>> content-type
> >>> >>>>>>>>> should
> >>> >>>>>>>>>> > be set
> >>> >>>>>>>>>> > > at
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> topic
> >>> >>>>>>>>>> > > > >> > >> >> >>> level.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing
> messages
> >>> with
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> > content
> >>> >>>>>>>>>> > > > types
> >>> >>>>>>>>>> > > > >> > >> should be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
> >>> >>> useless
> >>> >>>>>>>>> without
> >>> >>>>>>>>>> > schema id,
> >>> >>>>>>>>>> > > it
> >>> >>>>>>>>>> > > > >> > seems
> >>> >>>>>>>>>> > > > >> > >> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > storing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together
> with
> >>> >>>>>>> serialized
> >>> >>>>>>>>> bytes
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > value
> >>> >>>>>>>>>> > > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> better?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this
> >>> approach is
> >>> >>>>> that
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > > > significantly
> >>> >>>>>>>>>> > > > >> > reduce
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of
> compression,
> >>> >>>>> which
> >>> >>>>>>>>> happens on
> >>> >>>>>>>>>> > a set
> >>> >>>>>>>>>> > > of
> >>> >>>>>>>>>> > > > >> > >> serialized
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative
> is
> >>> to
> >>> >>>>> enable
> >>> >>>>>>>>> SSL for
> >>> >>>>>>>>>> > wire
> >>> >>>>>>>>>> > > > >> > encryption
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> rely
> >>> >>>>>>>>>> > > > >> > >> >> >>> > on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g.
> >>> LUKS)
> >>> >>>>> for at
> >>> >>>>>>>>> rest
> >>> >>>>>>>>>> > > encryption.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for
> mirroring
> >>> >>> across
> >>> >>>>>>>>> Kafka
> >>> >>>>>>>>>> > clusters
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> This is actually
> interesting.
> >>> >>>>> Today,
> >>> >>>>>>>>> to avoid
> >>> >>>>>>>>>> > > > introducing
> >>> >>>>>>>>>> > > > >> > >> cycles
> >>> >>>>>>>>>> > > > >> > >> >> when
> >>> >>>>>>>>>> > > > >> > >> >> >>> > doing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data
> >>> centers,
> >>> >>> one
> >>> >>>>>>>>> would
> >>> >>>>>>>>>> > either have
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> set
> >>> >>>>>>>>>> > > > >> > up
> >>> >>>>>>>>>> > > > >> > >> two
> >>> >>>>>>>>>> > > > >> > >> >> >>> Kafka
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
> >>> >>>>> aggregate)
> >>> >>>>>>>>> per data
> >>> >>>>>>>>>> > center
> >>> >>>>>>>>>> > > or
> >>> >>>>>>>>>> > > > >> > rename
> >>> >>>>>>>>>> > > > >> > >> >> topics.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
> >>> headers,
> >>> >>>>> the
> >>> >>>>>>>>> producer
> >>> >>>>>>>>>> > could
> >>> >>>>>>>>>> > > tag
> >>> >>>>>>>>>> > > > >> each
> >>> >>>>>>>>>> > > > >> > >> >> message
> >>> >>>>>>>>>> > > > >> > >> >> >>> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID
> in
> >>> the
> >>> >>>>>>> header.
> >>> >>>>>>>>>> > MirrorMaker
> >>> >>>>>>>>>> > > > could
> >>> >>>>>>>>>> > > > >> > then
> >>> >>>>>>>>>> > > > >> > >> >> avoid
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a
> >>> cluster
> >>> >>> if
> >>> >>>>>>>>> they are
> >>> >>>>>>>>>> > tagged
> >>> >>>>>>>>>> > > with
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> same
> >>> >>>>>>>>>> > > > >> > >> >> >>> cluster
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> id.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative
> >>> approach
> >>> >>>>> is
> >>> >>>>>>> to
> >>> >>>>>>>>>> > introduce sth
> >>> >>>>>>>>>> > > > like
> >>> >>>>>>>>>> > > > >> > >> >> >>> hierarchical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages
> from
> >>> >>>>>>> different
> >>> >>>>>>>>>> > clusters in
> >>> >>>>>>>>>> > > > >> > different
> >>> >>>>>>>>>> > > > >> > >> >> >>> partitions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic.
> This
> >>> >>>>> approach
> >>> >>>>>>>>> avoids
> >>> >>>>>>>>>> > filtering
> >>> >>>>>>>>>> > > > out
> >>> >>>>>>>>>> > > > >> > >> unneeded
> >>> >>>>>>>>>> > > > >> > >> >> >>> data
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving
> >>> easier
> >>> >>> to
> >>> >>>>>>>>> support. It
> >>> >>>>>>>>>> > may
> >>> >>>>>>>>>> > > make
> >>> >>>>>>>>>> > > > >> > >> compaction
> >>> >>>>>>>>>> > > > >> > >> >> >>> > trickier
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> though since the same
> key may
> >>> >>>>> show up
> >>> >>>>>>>>> in
> >>> >>>>>>>>>> > different
> >>> >>>>>>>>>> > > > >> > partitions.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source
> >>> connector
> >>> >>>>> could
> >>> >>>>>>>>> store in
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > message
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
> >>> >>> record.
> >>> >>>>>>>>> Similarly,
> >>> >>>>>>>>>> > if a
> >>> >>>>>>>>>> > > > stream
> >>> >>>>>>>>>> > > > >> job
> >>> >>>>>>>>>> > > > >> > >> >> >>> transforms
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to
> >>> topic
> >>> >>> B,
> >>> >>>>> the
> >>> >>>>>>>>> library
> >>> >>>>>>>>>> > could
> >>> >>>>>>>>>> > > > >> include
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> source
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each
> of the
> >>> >>>>>>>>> transformed
> >>> >>>>>>>>>> > message in
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> header.
> >>> >>>>>>>>>> > > > >> > >> >> Not
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sure
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful
> >>> record-level
> >>> >>>>>>> lineage
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > though
> >>> >>>>>>>>>> > > since
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> overhead
> >>> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
> >>> >>>>>>>>> clientId/host/user in
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > header
> >>> >>>>>>>>>> > > > >> in
> >>> >>>>>>>>>> > > > >> > >> each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These
> metadata
> >>> are
> >>> >>>>>>>>> really at the
> >>> >>>>>>>>>> > > producer
> >>> >>>>>>>>>> > > > >> > level
> >>> >>>>>>>>>> > > > >> > >> >> though.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > So, a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to
> only
> >>> >>>>>>> include a
> >>> >>>>>>>>>> > "producerId"
> >>> >>>>>>>>>> > > > per
> >>> >>>>>>>>>> > > > >> > >> message
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > send
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producerId ->
> metadata
> >>> >>> mapping
> >>> >>>>>>>>>> > independently.
> >>> >>>>>>>>>> > > > KIP-98
> >>> >>>>>>>>>> > > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> actually
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such
> a
> >>> >>>>> producerId
> >>> >>>>>>>>> natively
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > >> > message.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure
> that
> >>> I
> >>> >>> am
> >>> >>>>>>> fully
> >>> >>>>>>>>>> > convinced of
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > strong
> >>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
> >>> >>> Perhaps
> >>> >>>>> we
> >>> >>>>>>>>> could
> >>> >>>>>>>>>> > discuss a
> >>> >>>>>>>>>> > > > bit
> >>> >>>>>>>>>> > > > >> > more
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > one
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing
> use
> >>> >>>>> cases.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal
> question
> >>> is
> >>> >>>>>>>>> whether header
> >>> >>>>>>>>>> > should
> >>> >>>>>>>>>> > > > be
> >>> >>>>>>>>>> > > > >> > >> exposed
> >>> >>>>>>>>>> > > > >> > >> >> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > stream
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such
> Kafka
> >>> >>>>> stream,
> >>> >>>>>>>>> Samza,
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > Spark
> >>> >>>>>>>>>> > > > >> > >> streaming.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems
> just
> >>> >>> deal
> >>> >>>>>>> with
> >>> >>>>>>>>>> > key/value
> >>> >>>>>>>>>> > > > pairs.
> >>> >>>>>>>>>> > > > >> > >> Should we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > expose a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there
> too
> >>> or
> >>> >>>>>>>>> somehow map
> >>> >>>>>>>>>> > header to
> >>> >>>>>>>>>> > > > key
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> >> value?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Jun
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at
> 3:35
> >>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>> > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a
> >>> period
> >>> >>>>> of a
> >>> >>>>>>>>> week,
> >>> >>>>>>>>>> > that there
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> no
> >>> >>>>>>>>>> > > > >> > >> >> concerns
> >>> >>>>>>>>>> > > > >> > >> >> >>> now
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2
> and
> >>> now
> >>> >>> we
> >>> >>>>>>> have
> >>> >>>>>>>>>> > agreement that
> >>> >>>>>>>>>> > > > >> > headers
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> >>> useful
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
> such if
> >>> >>> put
> >>> >>>>> to
> >>> >>>>>>> a
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>> > vote, this
> >>> >>>>>>>>>> > > > >> > wouldn't
> >>> >>>>>>>>>> > > > >> > >> be
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > reason
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > @
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of
> >>> getting
> >>> >>>>> this
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>> > moving past
> >>> >>>>>>>>>> > > > >> this,
> >>> >>>>>>>>>> > > > >> > we
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> state
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes
> space
> >>> >>> that
> >>> >>>>>>> can
> >>> >>>>>>>>> will be
> >>> >>>>>>>>>> > > > naturally
> >>> >>>>>>>>>> > > > >> > >> >> interpreted
> >>> >>>>>>>>>> > > > >> > >> >> >>> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing
> is
> >>> later
> >>> >>>>>>>>> wanted you can
> >>> >>>>>>>>>> > > easily
> >>> >>>>>>>>>> > > > >> split
> >>> >>>>>>>>>> > > > >> > >> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> into
> >>> >>>>>>>>>> > > > >> > >> >> >>> > two
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the
> >>> wire
> >>> >>>>>>> protocol
> >>> >>>>>>>>>> > > implementation
> >>> >>>>>>>>>> > > > >> this
> >>> >>>>>>>>>> > > > >> > >> makes
> >>> >>>>>>>>>> > > > >> > >> >> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don't
> >>> believe. Is
> >>> >>>>> this
> >>> >>>>>>>>>> > reasonable to
> >>> >>>>>>>>>> > > > all?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
> >>> therefor
> >>> >>>>> happy
> >>> >>>>>>>>> we keep
> >>> >>>>>>>>>> > with 32
> >>> >>>>>>>>>> > > > >> bits.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
> >>> >>>>>>>>>> > ignacio.solis@gmail.com on
> >>> >>>>>>>>>> > > > >> behalf
> >>> >>>>>>>>>> > > > >> > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> Ignacio
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
> >>> >>> ignacio.solis@gmail.com
> >>> >>>>> on
> >>> >>>>>>>>> behalf of
> >>> >>>>>>>>>> > > > >> > isolis@igso.net
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
> value
> >>> as
> >>> >>>>> byte[]
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String -
> Reluctant
> >>> >>> maybe
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the
> header
> >>> >>>>> system
> >>> >>>>>>>>> should
> >>> >>>>>>>>>> > take a
> >>> >>>>>>>>>> > > > single
> >>> >>>>>>>>>> > > > >> > >> int. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you
> >>> want
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>> > this as
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> 16bit
> >>> >>>>>>>>>> > > > >> > >> >> numbers
> >>> >>>>>>>>>> > > > >> > >> >> >>> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > above go right
> ahead.
> >>> If
> >>> >>>>>>>>> somebody wants
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > argue
> >>> >>>>>>>>>> > > > >> for
> >>> >>>>>>>>>> > > > >> > 16
> >>> >>>>>>>>>> > > > >> > >> >> bits
> >>> >>>>>>>>>> > > > >> > >> >> >>> or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 64
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I
> >>> would
> >>> >>>>>>> listen.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key
> space
> >>> >>> into
> >>> >>>>>>>>> sub_key_1 and
> >>> >>>>>>>>>> > > > sub_key_2
> >>> >>>>>>>>>> > > > >> > >> makes no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sense to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > me at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we
> >>> going
> >>> >>> to
> >>> >>>>>>>>> start
> >>> >>>>>>>>>> > providing
> >>> >>>>>>>>>> > > > APIs to
> >>> >>>>>>>>>> > > > >> > get
> >>> >>>>>>>>>> > > > >> > >> all
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all
> the
> >>> >>>>>>>>> sub_key_2s? If
> >>> >>>>>>>>>> > there is
> >>> >>>>>>>>>> > > > no
> >>> >>>>>>>>>> > > > >> > >> >> >>> distinguishing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > functions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to
> >>> each
> >>> >>> one
> >>> >>>>>>>>> then they
> >>> >>>>>>>>>> > should
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > a
> >>> >>>>>>>>>> > > > >> > single
> >>> >>>>>>>>>> > > > >> > >> >> >>> value.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > At
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're
> doing
> >>> is
> >>> >>>>>>>>> equality.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer
> >>> wants
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>> > this as
> >>> >>>>>>>>>> > > 2,
> >>> >>>>>>>>>> > > > 3
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> more
> >>> >>>>>>>>>> > > > >> > >> >> >>> values
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > different
> question. I
> >>> >>>>>>>>> personally think
> >>> >>>>>>>>>> > it's
> >>> >>>>>>>>>> > > all
> >>> >>>>>>>>>> > > > one
> >>> >>>>>>>>>> > > > >> > >> >> keyspace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned
> using
> >>> >>> some
> >>> >>>>>>>>> structure,
> >>> >>>>>>>>>> > but if
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> > want to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sub-assign
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > parts
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's
> fine.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
> >>> >>> applies
> >>> >>>>> to
> >>> >>>>>>>>> strings.
> >>> >>>>>>>>>> > If
> >>> >>>>>>>>>> > > > somebody
> >>> >>>>>>>>>> > > > >> > >> argued
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would we be
> arguing to
> >>> >>>>> divide
> >>> >>>>>>> the
> >>> >>>>>>>>>> > strings with
> >>> >>>>>>>>>> > > > dots
> >>> >>>>>>>>>> > > > >> > ('.')
> >>> >>>>>>>>>> > > > >> > >> >> as a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them
> to
> >>> give
> >>> >>>>> us
> >>> >>>>>>> the
> >>> >>>>>>>>>> > different
> >>> >>>>>>>>>> > > name
> >>> >>>>>>>>>> > > > >> > segments
> >>> >>>>>>>>>> > > > >> > >> >> >>> > separately?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be
> performing
> >>> any
> >>> >>>>>>>>> actions on
> >>> >>>>>>>>>> > this key
> >>> >>>>>>>>>> > > > other
> >>> >>>>>>>>>> > > > >> > than
> >>> >>>>>>>>>> > > > >> > >> >> >>> > matching?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
> 2016 at
> >>> >>> 9:30
> >>> >>>>>>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>> > > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> >>> concerns
> >>> >>> on
> >>> >>>>> 1
> >>> >>>>>>>>> and 2
> >>> >>>>>>>>>> > still?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this
> moving
> >>> >>> along a
> >>> >>>>>>> bit
> >>> >>>>>>>>> more
> >>> >>>>>>>>>> > I'd also
> >>> >>>>>>>>>> > > > like
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> ask
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> get
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last
> >>> points:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe
> we're
> >>> all
> >>> >>>>>>> roughly
> >>> >>>>>>>>> happy
> >>> >>>>>>>>>> > with the
> >>> >>>>>>>>>> > > > >> header
> >>> >>>>>>>>>> > > > >> > >> value
> >>> >>>>>>>>>> > > > >> > >> >> >>> > being a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> >>> consensus
> >>> >>> has
> >>> >>>>>>>>> been for an
> >>> >>>>>>>>>> > > > namespace
> >>> >>>>>>>>>> > > > >> > based
> >>> >>>>>>>>>> > > > >> > >> int
> >>> >>>>>>>>>> > > > >> > >> >> >>> > approach
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the
> >>> key.
> >>> >>> Any
> >>> >>>>>>>>> objections
> >>> >>>>>>>>>> > if this
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> > what
> >>> >>>>>>>>>> > > > >> > >> we
> >>> >>>>>>>>>> > > > >> > >> >> go
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
> >>> >>>>> assumption
> >>> >>>>>>> in
> >>> >>>>>>>>> (4) is
> >>> >>>>>>>>>> > > correct,
> >>> >>>>>>>>>> > > > >> > >> {int,int}
> >>> >>>>>>>>>> > > > >> > >> >> >>> keys.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both
> int's be
> >>> >>>>> int16 or
> >>> >>>>>>>>> int32?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them
> being
> >>> >>> int16(2
> >>> >>>>>>>>> bytes) as
> >>> >>>>>>>>>> > combined
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> > space
> >>> >>>>>>>>>> > > > >> > >> of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > per
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > original and
> gives
> >>> >>> plenty
> >>> >>>>> of
> >>> >>>>>>>>>> > combinations for
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> foreseeable,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead
> small.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any
> >>> benefit in
> >>> >>>>>>>>> another kip
> >>> >>>>>>>>>> > call to
> >>> >>>>>>>>>> > > > >> discuss
> >>> >>>>>>>>>> > > > >> > >> >> these at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > all?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>> ______________________________
> >>> >>>>>>>>>> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
> >>> >>>>>>>>> k.burstev@yandex.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> >>> November
> >>> >>> 18,
> >>> >>>>>>> 2016
> >>> >>>>>>>>>> > 7:07:07 AM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
> >>> >>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> >>> [DISCUSS]
> >>> >>>>> KIP-82
> >>> >>>>>>>>> - Add
> >>> >>>>>>>>>> > Record
> >>> >>>>>>>>>> > > > Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is
> worth
> >>> >>> also
> >>> >>>>> i
> >>> >>>>>>>>> agree. As
> >>> >>>>>>>>>> > a user:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
> Headers are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers
> >>> should
> >>> >>>>> be a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>> > > option
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
> 21:15,
> >>> >>>>> "Ignacio
> >>> >>>>>>>>> Solis" <
> >>> >>>>>>>>>> > > > >> isolis@igso.net
> >>> >>>>>>>>>> > > > >> > >:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
> Headers
> >>> are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
> Headers
> >>> >>> should
> >>> >>>>> be
> >>> >>>>>>> a
> >>> >>>>>>>>> top
> >>> >>>>>>>>>> > level
> >>> >>>>>>>>>> > > option
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14,
> >>> 2016
> >>> >>> at
> >>> >>>>>>> 9:16
> >>> >>>>>>>>> AM,
> >>> >>>>>>>>>> > Michael
> >>> >>>>>>>>>> > > > Pearce
> >>> >>>>>>>>>> > > > >> <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
> >>> >>>>> details/examples
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > original
> >>> >>>>>>>>>> > > > proposal
> >>> >>>>>>>>>> > > > >> > for
> >>> >>>>>>>>>> > > > >> > >> key
> >>> >>>>>>>>>> > > > >> > >> >> >>> > spacing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> ,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > not
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new
> mentioned as
> >>> per
> >>> >>>>>>>>> discussion
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > >> idea.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need
> to
> >>> >>> update
> >>> >>>>>>> the
> >>> >>>>>>>>> kip,
> >>> >>>>>>>>>> > when we
> >>> >>>>>>>>>> > > get
> >>> >>>>>>>>>> > > > >> > >> agreement
> >>> >>>>>>>>>> > > > >> > >> >> >>> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > is a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > better
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach
> (which
> >>> >>> seems
> >>> >>>>> to
> >>> >>>>>>>>> be the
> >>> >>>>>>>>>> > case if I
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> > >> >> understood
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > general
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in
> the
> >>> >>>>>>>>> conversation)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the
> variable
> >>> >>> ints,
> >>> >>>>> at
> >>> >>>>>>>>> very
> >>> >>>>>>>>>> > early stage
> >>> >>>>>>>>>> > > > we
> >>> >>>>>>>>>> > > > >> did
> >>> >>>>>>>>>> > > > >> > >> think
> >>> >>>>>>>>>> > > > >> > >> >> >>> about
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added
> >>> complexity
> >>> >>>>> for
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > saving isn't
> >>> >>>>>>>>>> > > > >> worth
> >>> >>>>>>>>>> > > > >> > it.
> >>> >>>>>>>>>> > > > >> > >> >> I'd
> >>> >>>>>>>>>> > > > >> > >> >> >>> > rather
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> go
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to
> reduce
> >>> >>>>>>>>> overheads and
> >>> >>>>>>>>>> > size
> >>> >>>>>>>>>> > > int16
> >>> >>>>>>>>>> > > > >> > (2bytes)
> >>> >>>>>>>>>> > > > >> > >> >> keys
> >>> >>>>>>>>>> > > > >> > >> >> >>> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note
> of no
> >>> >>>>>>> headers,
> >>> >>>>>>>>> there
> >>> >>>>>>>>>> > is as
> >>> >>>>>>>>>> > > per
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > kip
> >>> >>>>>>>>>> > > > >> > >> as
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > use an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to
> denote if
> >>> >>>>> headers
> >>> >>>>>>>>> are
> >>> >>>>>>>>>> > present or
> >>> >>>>>>>>>> > > > not as
> >>> >>>>>>>>>> > > > >> > such
> >>> >>>>>>>>>> > > > >> > >> >> >>> > provides a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > zero
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead
> >>> currently
> >>> >>> if
> >>> >>>>>>>>> headers are
> >>> >>>>>>>>>> > not
> >>> >>>>>>>>>> > > used.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as
> radai
> >>> >>>>> mentions
> >>> >>>>>>>>> would be
> >>> >>>>>>>>>> > good
> >>> >>>>>>>>>> > > > first
> >>> >>>>>>>>>> > > > >> > if we
> >>> >>>>>>>>>> > > > >> > >> >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> get
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have
> >>> general
> >>> >>>>>>>>> consensus that
> >>> >>>>>>>>>> > (1)
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we
> want
> >>> it
> >>> >>> as
> >>> >>>>> a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>> > entity.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to
> state the
> >>> >>>>>>> obvious i
> >>> >>>>>>>>>> > believe (1)
> >>> >>>>>>>>>> > > > >> headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a
> top
> >>> level
> >>> >>>>>>>>> entity.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>> ______________________________
> >>> >>>>>>>>>> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
> >>> Hoover <
> >>> >>>>>>>>>> > > roger.hoover@gmail.com
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent:
> Wednesday,
> >>> >>>>> November
> >>> >>>>>>>>> 9, 2016
> >>> >>>>>>>>>> > 9:10:47
> >>> >>>>>>>>>> > > > PM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
> >>> >>>>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
> >>> >>> [DISCUSS]
> >>> >>>>>>>>> KIP-82 - Add
> >>> >>>>>>>>>> > > Record
> >>> >>>>>>>>>> > > > >> > Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
> going a
> >>> >>>>> little
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > weeds but
> >>> >>>>>>>>>> > > > >> thanks
> >>> >>>>>>>>>> > > > >> > >> for
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> replies
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a
> >>> prefix
> >>> >>>>> and
> >>> >>>>>>>>> {int,
> >>> >>>>>>>>>> > int} can
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> same.
> >>> >>>>>>>>>> > > > >> > >> >> It
> >>> >>>>>>>>>> > > > >> > >> >> >>> > doesn't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > look
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > like
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what
> the
> >>> KIP
> >>> >>> is
> >>> >>>>>>>>> saying the
> >>> >>>>>>>>>> > "Open"
> >>> >>>>>>>>>> > > > >> > section.
> >>> >>>>>>>>>> > > > >> > >> The
> >>> >>>>>>>>>> > > > >> > >> >> >>> > example
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shows
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New
> Relic and
> >>> >>>>> 210002
> >>> >>>>>>>>> for App
> >>> >>>>>>>>>> > Dynamics
> >>> >>>>>>>>>> > > > >> > implying
> >>> >>>>>>>>>> > > > >> > >> >> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > New
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization
> will
> >>> >>> have
> >>> >>>>>>>>> only a
> >>> >>>>>>>>>> > single
> >>> >>>>>>>>>> > > > header id
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> work
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with. Or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The
> >>> main
> >>> >>>>> point
> >>> >>>>>>>>> of a
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > or
> >>> >>>>>>>>>> > > > >> > prefix
> >>> >>>>>>>>>> > > > >> > >> is
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > reduce
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of
> >>> config
> >>> >>>>>>> mapping
> >>> >>>>>>>>> or
> >>> >>>>>>>>>> > > registration
> >>> >>>>>>>>>> > > > >> > >> depending
> >>> >>>>>>>>>> > > > >> > >> >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> how
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> namespaces/prefixes
> >>> >>>>> are
> >>> >>>>>>>>> managed.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to
> >>> hear
> >>> >>>>> more
> >>> >>>>>>>>> feedback
> >>> >>>>>>>>>> > on the
> >>> >>>>>>>>>> > > > >> > >> higher-level
> >>> >>>>>>>>>> > > > >> > >> >> >>> > questions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov
> 9,
> >>> 2016
> >>> >>> at
> >>> >>>>>>>>> 11:38 AM,
> >>> >>>>>>>>>> > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think
> this
> >>> >>>>> discussion
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > getting a
> >>> >>>>>>>>>> > > bit
> >>> >>>>>>>>>> > > > >> into
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> weeds on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > technical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> implementation
> >>> >>>>> details.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to
> >>> step
> >>> >>>>> back a
> >>> >>>>>>>>> minute
> >>> >>>>>>>>>> > and try
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > >> establish
> >>> >>>>>>>>>> > > > >> > >> >> >>> > where we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > are in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger
> picture:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
> >>> >>> nacho's
> >>> >>>>>>> last
> >>> >>>>>>>>>> > paragraph)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we
> all
> >>> in
> >>> >>>>>>>>> agreement that
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > useful
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to
> >>> have?
> >>> >>>>> this
> >>> >>>>>>>>> was
> >>> >>>>>>>>>> > contested
> >>> >>>>>>>>>> > > > early
> >>> >>>>>>>>>> > > > >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we
> all
> >>> in
> >>> >>>>>>>>> agreement on
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > as
> >>> >>>>>>>>>> > > > top
> >>> >>>>>>>>>> > > > >> > >> level
> >>> >>>>>>>>>> > > > >> > >> >> >>> entity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > vs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> squirreled-away in
> >>> >>>>> V?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there
> are
> >>> still
> >>> >>>>>>>>> concerns
> >>> >>>>>>>>>> > around
> >>> >>>>>>>>>> > > these
> >>> >>>>>>>>>> > > > #2
> >>> >>>>>>>>>> > > > >> > >> points
> >>> >>>>>>>>>> > > > >> > >> >> >>> (#jay?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now
> back
> >>> to
> >>> >>> our
> >>> >>>>>>>>> normal
> >>> >>>>>>>>>> > programming
> >>> >>>>>>>>>> > > > ...)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are
> >>> nice.
> >>> >>>>>>> having
> >>> >>>>>>>>> said
> >>> >>>>>>>>>> > that, its
> >>> >>>>>>>>>> > > > >> adding
> >>> >>>>>>>>>> > > > >> > >> >> >>> complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (see
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>> https://github.com/addthis/
> >>> >>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>> main/java/com/clearspring/
> >>> >>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
> google
> >>> >>>>> result)
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>> > require
> >>> >>>>>>>>>> > > > >> anyone
> >>> >>>>>>>>>> > > > >> > >> >> writing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clients
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go?
> >>> Bash?
> >>> >>>>> ;-) )
> >>> >>>>>>>>> to
> >>> >>>>>>>>>> > > get/implement
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> same,
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain
> >>> (int
> >>> >>> vs
> >>> >>>>>>>>> string is
> >>> >>>>>>>>>> > order of
> >>> >>>>>>>>>> > > > >> > magnitude,
> >>> >>>>>>>>>> > > > >> > >> >> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > isnt).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int
> >>> namespacing vs
> >>> >>>>>>> {int,
> >>> >>>>>>>>> int}
> >>> >>>>>>>>>> > > namespacing
> >>> >>>>>>>>>> > > > >> are
> >>> >>>>>>>>>> > > > >> > >> >> basically
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > same
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
> >>> >>>>> namespacing
> >>> >>>>>>>>> an int64
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > > giving
> >>> >>>>>>>>>> > > > >> > people
> >>> >>>>>>>>>> > > > >> > >> >> while
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 2^32
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the
> part
> >>> i
> >>> >>>>> like
> >>> >>>>>>>>> about this
> >>> >>>>>>>>>> > is
> >>> >>>>>>>>>> > > > letting
> >>> >>>>>>>>>> > > > >> > >> people
> >>> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> large
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers
> with
> >>> one
> >>> >>>>>>>>> registration so
> >>> >>>>>>>>>> > they
> >>> >>>>>>>>>> > > > dont
> >>> >>>>>>>>>> > > > >> > have
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> come
> >>> >>>>>>>>>> > > > >> > >> >> >>> > back
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > every
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
> >>> >>> plugin/header
> >>> >>>>>>>>> they want to
> >>> >>>>>>>>>> > > > "reserve".
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
> Nov 9,
> >>> >>> 2016
> >>> >>>>> at
> >>> >>>>>>>>> 11:01 AM,
> >>> >>>>>>>>>> > Roger
> >>> >>>>>>>>>> > > > >> Hoover
> >>> >>>>>>>>>> > > > >> > <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> roger.hoover@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since
> some of
> >>> >>> the
> >>> >>>>>>>>> debate has
> >>> >>>>>>>>>> > been
> >>> >>>>>>>>>> > > about
> >>> >>>>>>>>>> > > > >> > >> overhead +
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> wondering if
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>>> > considered a
> >>> >>>>>>>>>> > > > varint
> >>> >>>>>>>>>> > > > >> > >> encoding
> >>> >>>>>>>>>> > > > >> > >> >> (
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>> https://developers.google.com/
> >>> >>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the
> header
> >>> >>> length
> >>> >>>>>>>>> field (int32
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > >> > proposal)
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
> >>> >>> headers,
> >>> >>>>>>> the
> >>> >>>>>>>>>> > overhead would
> >>> >>>>>>>>>> > > > be a
> >>> >>>>>>>>>> > > > >> > >> single
> >>> >>>>>>>>>> > > > >> > >> >> >>> byte
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128
> >>> would
> >>> >>>>> also
> >>> >>>>>>>>> need only a
> >>> >>>>>>>>>> > > single
> >>> >>>>>>>>>> > > > >> byte?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed,
> Nov
> >>> 9,
> >>> >>>>> 2016
> >>> >>>>>>> at
> >>> >>>>>>>>> 6:43 AM,
> >>> >>>>>>>>>> > > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> @magnus -
> >>> and
> >>> >>>>> very
> >>> >>>>>>>>> dangerous
> >>> >>>>>>>>>> > (youre
> >>> >>>>>>>>>> > > > >> > >> essentially
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> arbitrary
> >>> code
> >>> >>>>> off
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > internet on
> >>> >>>>>>>>>> > > > your
> >>> >>>>>>>>>> > > > >> > >> servers
> >>> >>>>>>>>>> > > > >> > >> >> ...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > bad
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > idea
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > without
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> sandbox,
> >>> even
> >>> >>>>> with)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for
> it
> >>> >>> being
> >>> >>>>> a
> >>> >>>>>>>>> purely
> >>> >>>>>>>>>> > > > administrative
> >>> >>>>>>>>>> > > > >> > task
> >>> >>>>>>>>>> > > > >> > >> - i
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish
> it
> >>> >>> would,
> >>> >>>>>>>>> really,
> >>> >>>>>>>>>> > because
> >>> >>>>>>>>>> > > > then my
> >>> >>>>>>>>>> > > > >> > >> earlier
> >>> >>>>>>>>>> > > > >> > >> >> >>> > point on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >>> remapping
> >>> >>>>>>>>> process would
> >>> >>>>>>>>>> > be
> >>> >>>>>>>>>> > > > invalid,
> >>> >>>>>>>>>> > > > >> > but
> >>> >>>>>>>>>> > > > >> > >> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > linkedin,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the
> team
> >>> im
> >>> >>> in)
> >>> >>>>>>> run
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>> > as a
> >>> >>>>>>>>>> > > > service.
> >>> >>>>>>>>>> > > > >> > we
> >>> >>>>>>>>>> > > > >> > >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > really
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > know
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> (developing
> >>> >>>>>>>>> applications
> >>> >>>>>>>>>> > that use
> >>> >>>>>>>>>> > > > kafka)
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> up
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > any
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > given
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
> >>> >>> possible
> >>> >>>>>>>>> (given the
> >>> >>>>>>>>>> > > > existance of
> >>> >>>>>>>>>> > > > >> > >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> and a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> ecosystem)
> >>> for
> >>> >>>>> some
> >>> >>>>>>>>>> > application to
> >>> >>>>>>>>>> > > > >> "equip"
> >>> >>>>>>>>>> > > > >> > >> their
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> producers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with
> the
> >>> >>>>> required
> >>> >>>>>>>>> plugin
> >>> >>>>>>>>>> > without us
> >>> >>>>>>>>>> > > > >> > knowing.
> >>> >>>>>>>>>> > > > >> > >> i
> >>> >>>>>>>>>> > > > >> > >> >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > mean
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i
> just
> >>> >>>>> want to
> >>> >>>>>>>>> make the
> >>> >>>>>>>>>> > point
> >>> >>>>>>>>>> > > > that
> >>> >>>>>>>>>> > > > >> > its
> >>> >>>>>>>>>> > > > >> > >> not
> >>> >>>>>>>>>> > > > >> > >> >> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > simple
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
> >>> across a
> >>> >>>>>>>>> large-enough
> >>> >>>>>>>>>> > > > organization.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed,
> >>> Nov 9,
> >>> >>>>> 2016
> >>> >>>>>>>>> at 6:17
> >>> >>>>>>>>>> > AM,
> >>> >>>>>>>>>> > > > Magnus
> >>> >>>>>>>>>> > > > >> > >> Edenhill
> >>> >>>>>>>>>> > > > >> > >> >> <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> magnus@edenhill.se>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
> think
> >>> >>> there
> >>> >>>>> is
> >>> >>>>>>> a
> >>> >>>>>>>>> piece
> >>> >>>>>>>>>> > missing
> >>> >>>>>>>>>> > > in
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> >> Strings
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > where
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>> pro-Stringers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> reason
> >>> that
> >>> >>> by
> >>> >>>>>>>>> providing
> >>> >>>>>>>>>> > unique
> >>> >>>>>>>>>> > > > string
> >>> >>>>>>>>>> > > > >> > >> >> >>> identifiers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> everything
> >>> >>>>> will
> >>> >>>>>>>>> just
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> magically
> >>> >>> work
> >>> >>>>>>> for
> >>> >>>>>>>>> all
> >>> >>>>>>>>>> > parts of
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> > stream
> >>> >>>>>>>>>> > > > >> > >> >> >>> pipeline.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But
> the
> >>> >>>>> strings
> >>> >>>>>>>>> dont mean
> >>> >>>>>>>>>> > > anything
> >>> >>>>>>>>>> > > > by
> >>> >>>>>>>>>> > > > >> > >> >> themselves,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > while we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> probably
> >>> >>>>> envision
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> auto
> >>> >>>>> plugin
> >>> >>>>>>>>> loader
> >>> >>>>>>>>>> > that
> >>> >>>>>>>>>> > > > >> downloads,
> >>> >>>>>>>>>> > > > >> > >> >> compiles,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > links
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> on-demand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as
> soon
> >>> as
> >>> >>>>>>> they're
> >>> >>>>>>>>> seen by
> >>> >>>>>>>>>> > a
> >>> >>>>>>>>>> > > > >> consumer, I
> >>> >>>>>>>>>> > > > >> > >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> really
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> see
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> something
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
> >>> dynamic
> >>> >>>>> (and
> >>> >>>>>>>>> fragile) in
> >>> >>>>>>>>>> > > > practice.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In
> the
> >>> real
> >>> >>>>> world
> >>> >>>>>>>>> an
> >>> >>>>>>>>>> > application
> >>> >>>>>>>>>> > > > will
> >>> >>>>>>>>>> > > > >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> configured
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a set
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to
> either
> >>> >>> add
> >>> >>>>>>>>> (producer)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or
> read
> >>> >>>>>>> (consumer)
> >>> >>>>>>>>> headers.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This
> is
> >>> an
> >>> >>>>>>>>> administrative
> >>> >>>>>>>>>> > task
> >>> >>>>>>>>>> > > > based
> >>> >>>>>>>>>> > > > >> on
> >>> >>>>>>>>>> > > > >> > >> what
> >>> >>>>>>>>>> > > > >> > >> >> >>> > features a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > client
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>> needs/provides
> >>> >>>>>>> and
> >>> >>>>>>>>> results
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> >>> sort of
> >>> >>>>>>>>> configuration
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > > enable
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> configure
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > desired
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since
> >>> this
> >>> >>>>> needs
> >>> >>>>>>>>> to be kept
> >>> >>>>>>>>>> > > > somewhat
> >>> >>>>>>>>>> > > > >> in
> >>> >>>>>>>>>> > > > >> > >> sync
> >>> >>>>>>>>>> > > > >> > >> >> >>> across
> >>> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no
> >>> point
> >>> >>> in
> >>> >>>>>>>>> having
> >>> >>>>>>>>>> > producers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
> >>> headers
> >>> >>> no
> >>> >>>>>>>>> consumers
> >>> >>>>>>>>>> > will
> >>> >>>>>>>>>> > > read,
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> vice
> >>> >>>>>>>>>> > > > >> > >> >> >>> versa),
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > added
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
> >>> assigning
> >>> >>>>> an
> >>> >>>>>>> id
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for
> each
> >>> >>>>> plugin
> >>> >>>>>>> as
> >>> >>>>>>>>> it is
> >>> >>>>>>>>>> > being
> >>> >>>>>>>>>> > > > >> > configured
> >>> >>>>>>>>>> > > > >> > >> >> should
> >>> >>>>>>>>>> > > > >> > >> >> >>> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> /Magnus
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> 2016-11-09
> >>> >>>>> 13:06
> >>> >>>>>>>>> GMT+01:00
> >>> >>>>>>>>>> > > Michael
> >>> >>>>>>>>>> > > > >> > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> Michael.Pearce@ig.com>:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Just
> >>> >>>>>>>>> following/catching
> >>> >>>>>>>>>> > up on
> >>> >>>>>>>>>> > > > what
> >>> >>>>>>>>>> > > > >> > seems
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > active
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> @Radai
> >>> >>>>> sorry if
> >>> >>>>>>>>> it may
> >>> >>>>>>>>>> > seem
> >>> >>>>>>>>>> > > > obvious
> >>> >>>>>>>>>> > > > >> > but
> >>> >>>>>>>>>> > > > >> > >> what
> >>> >>>>>>>>>> > > > >> > >> >> >>> does
> >>> >>>>>>>>>> > > > >> > >> >> >>> > MD
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > stand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My
> >>> take on
> >>> >>>>>>>>> String vs Int:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> will
> >>> >>> state
> >>> >>>>>>>>> first I am
> >>> >>>>>>>>>> > pro Int
> >>> >>>>>>>>>> > > > (16
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> 32).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> do
> >>> >>> though
> >>> >>>>>>>>> playing
> >>> >>>>>>>>>> > devils
> >>> >>>>>>>>>> > > > advocate
> >>> >>>>>>>>>> > > > >> > see a
> >>> >>>>>>>>>> > > > >> > >> >> big
> >>> >>>>>>>>>> > > > >> > >> >> >>> plus
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> String
> >>> >>> keys,
> >>> >>>>>>>>> this is
> >>> >>>>>>>>>> > around
> >>> >>>>>>>>>> > > > >> > integrating
> >>> >>>>>>>>>> > > > >> > >> >> into an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > existing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As
> many
> >>> >>>>> other
> >>> >>>>>>>>> systems use
> >>> >>>>>>>>>> > > String
> >>> >>>>>>>>>> > > > >> based
> >>> >>>>>>>>>> > > > >> > >> >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>> these to
> >>> >>>>>>>>>> > be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > into.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How
> >>> with
> >>> >>> Int
> >>> >>>>>>>>> based
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > could
> >>> >>>>>>>>>> > > > we
> >>> >>>>>>>>>> > > > >> > >> provide
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>> integration
> >>> >>>>>>>>> simple /
> >>> >>>>>>>>>> > easy with
> >>> >>>>>>>>>> > > > >> > transition
> >>> >>>>>>>>>> > > > >> > >> >> flows
> >>> >>>>>>>>>> > > > >> > >> >> >>> > over
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> tough
> >>> >>> luck
> >>> >>>>>>>>> buddy
> >>> >>>>>>>>>> > you're on
> >>> >>>>>>>>>> > > your
> >>> >>>>>>>>>> > > > >> own
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> >>> simply
> >>> >>>>> hash
> >>> >>>>>>>>> the string
> >>> >>>>>>>>>> > into
> >>> >>>>>>>>>> > > int
> >>> >>>>>>>>>> > > > >> code
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> hope
> >>> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> convert
> >>> >>> back
> >>> >>>>>>>>> though?)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> http2
> >>> >>>>> style
> >>> >>>>>>> as
> >>> >>>>>>>>>> > mentioned by
> >>> >>>>>>>>>> > > > nacho.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> cheers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > ______________________________
> >>> >>>>>>>>>> > > > >> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> From:
> >>> >>> radai
> >>> >>>>> <
> >>> >>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Sent:
> >>> >>>>>>> Wednesday,
> >>> >>>>>>>>>> > November 9,
> >>> >>>>>>>>>> > > 2016
> >>> >>>>>>>>>> > > > >> > 8:12 AM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
> >>> >>>>>>>>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> Subject:
> >>> >>> Re:
> >>> >>>>>>>>> [DISCUSS]
> >>> >>>>>>>>>> > KIP-82 -
> >>> >>>>>>>>>> > > > Add
> >>> >>>>>>>>>> > > > >> > >> Record
> >>> >>>>>>>>>> > > > >> > >> >> >>> Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> thinking
> >>> >>>>> about
> >>> >>>>>>>>> it some
> >>> >>>>>>>>>> > more,
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> best
> >>> >>>>>>>>>> > > > >> > >> way to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > transmit
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> remapping
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> data to
> >>> >>>>>>>>> consumers would
> >>> >>>>>>>>>> > be to
> >>> >>>>>>>>>> > > > put it
> >>> >>>>>>>>>> > > > >> > in
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> MD
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> response
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it
> >>> should
> >>> >>> be
> >>> >>>>>>>>> discussed
> >>> >>>>>>>>>> > now.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On
> Wed,
> >>> >>> Nov
> >>> >>>>> 9,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>> > 12:09
> >>> >>>>>>>>>> > > AM,
> >>> >>>>>>>>>> > > > >> > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> im
> >>> not
> >>> >>>>>>> opposed
> >>> >>>>>>>>> to the
> >>> >>>>>>>>>> > idea of
> >>> >>>>>>>>>> > > > >> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> mapping.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > im
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> not
> >>> part
> >>> >>>>> of
> >>> >>>>>>>>> the "mvp"
> >>> >>>>>>>>>> > and,
> >>> >>>>>>>>>> > > > since
> >>> >>>>>>>>>> > > > >> it
> >>> >>>>>>>>>> > > > >> > >> >> requires
> >>> >>>>>>>>>> > > > >> > >> >> >>> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> wire
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > format
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> always
> >>> >>> be
> >>> >>>>>>>>> added later.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also,
> >>> >>> its
> >>> >>>>> not
> >>> >>>>>>>>> as
> >>> >>>>>>>>>> > simple as
> >>> >>>>>>>>>> > > just
> >>> >>>>>>>>>> > > > >> > >> >> configuring
> >>> >>>>>>>>>> > > > >> > >> >> >>> MM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> say
> >>> i've
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>> > large
> >>> >>>>>>>>>> > > > message
> >>> >>>>>>>>>> > > > >> > >> >> support as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> mirror
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> target
> >>> >>>>>>> cluster
> >>> >>>>>>>>> its been
> >>> >>>>>>>>>> > > > remapped
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> {999,1}.
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also
> >>> >>> need
> >>> >>>>> to
> >>> >>>>>>>>> be told
> >>> >>>>>>>>>> > to look
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> large
> >>> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> under
> >>> >>>>>>> {999,1}.
> >>> >>>>>>>>> doable,
> >>> >>>>>>>>>> > but
> >>> >>>>>>>>>> > > > tricky.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>> 8,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>> > 10:29
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> While
> >>> >>> you
> >>> >>>>>>> can
> >>> >>>>>>>>> do
> >>> >>>>>>>>>> > whatever
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> want
> >>> >>>>>>>>>> > > > >> > >> with a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > your
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> what
> >>> >>> I'd
> >>> >>>>>>>>> expect is
> >>> >>>>>>>>>> > for each
> >>> >>>>>>>>>> > > > app
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> namespaces
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> So
> >>> if I
> >>> >>>>>>>>> accidentally
> >>> >>>>>>>>>> > used
> >>> >>>>>>>>>> > > 666
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> > my
> >>> >>>>>>>>>> > > > >> > >> HR
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> department,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > still
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> run
> >>> >>>>>>> RadaiApp,
> >>> >>>>>>>>> I can
> >>> >>>>>>>>>> > config
> >>> >>>>>>>>>> > > > >> > >> "namespace=42"
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> everything
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> will
> >>> >>> look
> >>> >>>>>>>>> normal.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> This
> >>> >>>>> means
> >>> >>>>>>>>> you only
> >>> >>>>>>>>>> > need to
> >>> >>>>>>>>>> > > > sync
> >>> >>>>>>>>>> > > > >> > usage
> >>> >>>>>>>>>> > > > >> > >> >> >>> inside
> >>> >>>>>>>>>> > > > >> > >> >> >>> > your
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > own
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> organization.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> Still
> >>> >>>>> hard,
> >>> >>>>>>>>> but
> >>> >>>>>>>>>> > somewhat
> >>> >>>>>>>>>> > > > easier
> >>> >>>>>>>>>> > > > >> > than
> >>> >>>>>>>>>> > > > >> > >> >> syncing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>> > at 10:07
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > >> radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> and
> >>> >>> we
> >>> >>>>> can
> >>> >>>>>>>>> start
> >>> >>>>>>>>>> > with
> >>> >>>>>>>>>> > > > >> {namespace,
> >>> >>>>>>>>>> > > > >> > >> id}
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> always
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> add
> >>> >>> it
> >>> >>>>>>>>> later on
> >>> >>>>>>>>>> > if/when
> >>> >>>>>>>>>> > > > >> > collisions
> >>> >>>>>>>>>> > > > >> > >> >> >>> actually
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> they'd
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > a
> >>> >>>>>>> problem).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> every
> >>> >>>>>>>>> interested
> >>> >>>>>>>>>> > party (so
> >>> >>>>>>>>>> > > > orgs
> >>> >>>>>>>>>> > > > >> > or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > individuals)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> register
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>> prefix
> >>> >>>>> (0
> >>> >>>>>>> =
> >>> >>>>>>>>>> > reserved, 1 =
> >>> >>>>>>>>>> > > > >> > confluent
> >>> >>>>>>>>>> > > > >> > >> ...
> >>> >>>>>>>>>> > > > >> > >> >> >>> 666
> >>> >>>>>>>>>> > > > >> > >> >> >>> > = me
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> whatever
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> the
> >>> >>>>> 2nd ID
> >>> >>>>>>>>> - so once
> >>> >>>>>>>>>> > > > linkedin
> >>> >>>>>>>>>> > > > >> > >> >> registers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> say
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 3,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> free
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > to
> >>> >>> use
> >>> >>>>> {3,
> >>> >>>>>>>>> *} with a
> >>> >>>>>>>>>> > > > reasonable
> >>> >>>>>>>>>> > > > >> > >> >> >>> expectation
> >>> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> anything
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> else.
> >>> >>>>>>>>> further
> >>> >>>>>>>>>> > partitioning
> >>> >>>>>>>>>> > > > of
> >>> >>>>>>>>>> > > > >> > that *
> >>> >>>>>>>>>> > > > >> > >> >> >>> becomes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>> "upstream
> >>> >>>>>>>>>> > registration"
> >>> >>>>>>>>>> > > of a
> >>> >>>>>>>>>> > > > >> > >> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> only
> >>> >>>>>>>>>> > > > >> > >> >> >>> > has
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > On
> >>> >>> Tue,
> >>> >>>>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>> > at
> >>> >>>>>>>>>> > > 9:03
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > >> James
> >>> >>>>>>>>>> > > > >> > >> >> >>> Cheng <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>> wushujames@gmail.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > On
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016,
> >>> >>>>>>>>>> > at 5:54
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> Thank
> >>> >>>>>>>>> you so
> >>> >>>>>>>>>> > much for
> >>> >>>>>>>>>> > > > this
> >>> >>>>>>>>>> > > > >> > clear
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> fair
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> arguments.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> I'm
> >>> >>>>> in
> >>> >>>>>>>>> favor of
> >>> >>>>>>>>>> > ints.
> >>> >>>>>>>>>> > > > Not a
> >>> >>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > but
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> Even
> >>> >>>>>>>>> more in
> >>> >>>>>>>>>> > favor of
> >>> >>>>>>>>>> > > > >> Magnus's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > decentralized
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Roger's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> tweak:
> >>> >>>>>>>>> add a
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> > >> headers.
> >>> >>>>>>>>>> > > > >> > >> >> >>> This
> >>> >>>>>>>>>> > > > >> > >> >> >>> > will
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > allow
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> just
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> use
> >>> >>>>>>>>> whatever IDs
> >>> >>>>>>>>>> > it
> >>> >>>>>>>>>> > > wants
> >>> >>>>>>>>>> > > > >> > >> >> internally,
> >>> >>>>>>>>>> > > > >> > >> >> >>> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > let
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>> deploying
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> the
> >>> >>>>> app
> >>> >>>>>>>>> figure
> >>> >>>>>>>>>> > out an
> >>> >>>>>>>>>> > > > >> > available
> >>> >>>>>>>>>> > > > >> > >> >> >>> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > ID
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > So
> >>> >>>>>>>>>> > > > >> > io.confluent.schema-registry
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>> deployment
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> and
> >>> >>>>>>> 0x57
> >>> >>>>>>>>> on
> >>> >>>>>>>>>> > yours, and
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > poor
> >>> >>>>>>>>>> > > > >> > >> guys
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> worry
> >>> >>>>>>>>> about that.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> Gwen,
> >>> >>>>> if
> >>> >>>>>>> I
> >>> >>>>>>>>>> > understand
> >>> >>>>>>>>>> > > your
> >>> >>>>>>>>>> > > > >> > example
> >>> >>>>>>>>>> > > > >> > >> >> >>> right, an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > application
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> might
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> decide to
> >>> >>>>>>>>> use 0x01
> >>> >>>>>>>>>> > in one
> >>> >>>>>>>>>> > > > >> > >> deployment,
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > means
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> is
> >>> >>>>>>> written
> >>> >>>>>>>>> into the
> >>> >>>>>>>>>> > > > broker, it
> >>> >>>>>>>>>> > > > >> > >> will be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > saved on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> specific
> >>> >>>>>>>>> namespace
> >>> >>>>>>>>>> > > (0x01).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> If
> >>> >>> you
> >>> >>>>>>>>> were to
> >>> >>>>>>>>>> > mirror
> >>> >>>>>>>>>> > > that
> >>> >>>>>>>>>> > > > >> > message
> >>> >>>>>>>>>> > > > >> > >> >> into
> >>> >>>>>>>>>> > > > >> > >> >> >>> > another
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> accompany
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > message,
> >>> >>>>>>>>>> > > > right?
> >>> >>>>>>>>>> > > > >> > What
> >>> >>>>>>>>>> > > > >> > >> if
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> other
> >>> >>>>>>>>> cluster uses
> >>> >>>>>>>>>> > 0x57?
> >>> >>>>>>>>>> > > > They
> >>> >>>>>>>>>> > > > >> > won't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > understand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> I'm
> >>> >>>>> not
> >>> >>>>>>>>> sure
> >>> >>>>>>>>>> > that's an
> >>> >>>>>>>>>> > > > >> avoidable
> >>> >>>>>>>>>> > > > >> > >> >> >>> problem. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > think it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> order
> >>> >>>>> to
> >>> >>>>>>>>> share
> >>> >>>>>>>>>> > data, you
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> also
> >>> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shared
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>> understanding of
> >>> >>>>>>>>>> > what the
> >>> >>>>>>>>>> > > > >> > >> namespaces
> >>> >>>>>>>>>> > > > >> > >> >> >>> mean.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Which
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> sense,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> because
> >>> >>>>>>> the
> >>> >>>>>>>>>> > alternate
> >>> >>>>>>>>>> > > > (sharing
> >>> >>>>>>>>>> > > > >> > >> >> *nothing*
> >>> >>>>>>>>>> > > > >> > >> >> >>> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> all)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> would
> >>> >>>>> be
> >>> >>>>>>>>> no way to
> >>> >>>>>>>>>> > > > understand
> >>> >>>>>>>>>> > > > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> other.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> -James
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > On
> >>> >>>>> Tue,
> >>> >>>>>>>>> Nov 8,
> >>> >>>>>>>>>> > 2016 at
> >>> >>>>>>>>>> > > > 4:23
> >>> >>>>>>>>>> > > > >> > PM,
> >>> >>>>>>>>>> > > > >> > >> >> radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>> +1
> >>> >>>>> for
> >>> >>>>>>>>> sean's
> >>> >>>>>>>>>> > > document.
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > >> covers
> >>> >>>>>>>>>> > > > >> > >> >> >>> pretty
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> much
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> trade-offs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>>>>>> provides
> >>> >>>>>>>>>> > concrete
> >>> >>>>>>>>>> > > > figures
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> argue
> >>> >>>>>>>>>> > > > >> > >> >> >>> about
> >>> >>>>>>>>>> > > > >> > >> >> >>> > :-)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>>>>>>>> (nit-picking -
> >>> >>>>>>>>>> > used
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> same
> >>> >>>>>>>>>> > > > >> > >> xkcd
> >>> >>>>>>>>>> > > > >> > >> >> >>> twice,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> also
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > trove
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > has
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>> superceded
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > --
> >>> >>>>>>>>>> > > > >> > Gwen Shapira
> >>> >>>>>>>>>> > > > >> > Product Manager | Confluent
> >>> >>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
> @gwenshap
> >>> >>>>>>>>>> > > > >> > Follow us: Twitter | blog
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> --
> >>> >>>>>>>>>> > > > >> *Todd Palino*
> >>> >>>>>>>>>> > > > >> Staff Site Reliability Engineer
> >>> >>>>>>>>>> > > > >> Data Infrastructure Streaming
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> linkedin.com/in/toddpalino
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > --
> >>> >>>>>>>>>> > > > Gwen Shapira
> >>> >>>>>>>>>> > > > Product Manager | Confluent
> >>> >>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >>> >>>>>>>>>> > > > Follow us: Twitter | blog
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > The information contained in this email is strictly
> >>> >>>>> confidential
> >>> >>>>>>>>> and for
> >>> >>>>>>>>>> > the use of the addressee only, unless otherwise
> >>> indicated.
> >>> >>> If
> >>> >>>>> you
> >>> >>>>>>>>> are not
> >>> >>>>>>>>>> > the intended recipient, please do not read, copy,
> use or
> >>> >>>>> disclose
> >>> >>>>>>>>> to others
> >>> >>>>>>>>>> > this message or any attachment. Please also notify
> the
> >>> >>> sender
> >>> >>>>> by
> >>> >>>>>>>>> replying
> >>> >>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and
> >>> then
> >>> >>>>> delete
> >>> >>>>>>>>> the email
> >>> >>>>>>>>>> > and any copies of it. Opinions, conclusion (etc)
> that do
> >>> not
> >>> >>>>>>>>> relate to the
> >>> >>>>>>>>>> > official business of this company shall be
> understood as
> >>> >>>>> neither
> >>> >>>>>>>>> given nor
> >>> >>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets
> >>> Limited
> >>> >>> (a
> >>> >>>>>>>>> company
> >>> >>>>>>>>>> > registered in England and Wales, company number
> 04008957)
> >>> >>> and
> >>> >>>>> IG
> >>> >>>>>>>>> Index
> >>> >>>>>>>>>> > Limited (a company registered in England and Wales,
> >>> company
> >>> >>>>>>> number
> >>> >>>>>>>>>> > 01190902). Registered address at Cannon Bridge
> House, 25
> >>> >>>>> Dowgate
> >>> >>>>>>>>> Hill,
> >>> >>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register
> number
> >>> >>>>> 195355)
> >>> >>>>>>>>> and IG
> >>> >>>>>>>>>> > Index Limited (register number 114059) are
> authorised and
> >>> >>>>>>>>> regulated by the
> >>> >>>>>>>>>> > Financial Conduct Authority.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> The information contained in this email is strictly
> confidential
> >>> >>> and
> >>> >>>>>>> for
> >>> >>>>>>>>> the use of the addressee only, unless otherwise indicated.
> If you
> >>> >>> are
> >>> >>>>>>> not
> >>> >>>>>>>>> the intended recipient, please do not read, copy, use or
> disclose
> >>> >>> to
> >>> >>>>>>> others
> >>> >>>>>>>>> this message or any attachment. Please also notify the
> sender by
> >>> >>>>>>> replying
> >>> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then
> delete
> >>> >>> the
> >>> >>>>>>> email
> >>> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
> >>> relate
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> official business of this company shall be understood as
> neither
> >>> >>>>> given
> >>> >>>>>>> nor
> >>> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> >>> >>> company
> >>> >>>>>>>>> registered in England and Wales, company number 04008957)
> and IG
> >>> >>>>> Index
> >>> >>>>>>>>> Limited (a company registered in England and Wales, company
> >>> number
> >>> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >>> >>>>> Hill,
> >>> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> >>> >>>>> and IG
> >>> >>>>>>>>> Index Limited (register number 114059) are authorised and
> >>> regulated
> >>> >>>>> by
> >>> >>>>>>> the
> >>> >>>>>>>>> Financial Conduct Authority.
> >>> >>>>>>>>>
> >>> >>>>>>>
> >>> >>>>>
> >>> >>> The information contained in this email is strictly confidential
> and
> >>> for
> >>> >>> the use of the addressee only, unless otherwise indicated. If you
> are
> >>> not
> >>> >>> the intended recipient, please do not read, copy, use or disclose
> to
> >>> others
> >>> >>> this message or any attachment. Please also notify the sender by
> >>> replying
> >>> >>> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> >>> email
> >>> >>> and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> >>> the
> >>> >>> official business of this company shall be understood as neither
> given
> >>> nor
> >>> >>> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >>> >>> registered in England and Wales, company number 04008957) and IG
> Index
> >>> >>> Limited (a company registered in England and Wales, company number
> >>> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> >>> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> >>> IG
> >>> >>> Index Limited (register number 114059) are authorised and
> regulated by
> >>> the
> >>> >>> Financial Conduct Authority.
> >>> >>>
> >>> >
> >>> >
> >>>
> >>> The information contained in this email is strictly confidential and
> for
> >>> the use of the addressee only, unless otherwise indicated. If you are
> not
> >>> the intended recipient, please do not read, copy, use or disclose to
> others
> >>> this message or any attachment. Please also notify the sender by
> replying
> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >>> official business of this company shall be understood as neither given
> nor
> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >>> registered in England and Wales, company number 04008957) and IG Index
> >>> Limited (a company registered in England and Wales, company number
> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >>> Index Limited (register number 114059) are authorised and regulated by
> the
> >>> Financial Conduct Authority.
> >>>
> >
> >
> >
> > --
> > Nacho - Ignacio Solis - isolis@igso.net
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Fantastic. Thanks all for compromises ☺
Based on getting what seems some consensus to move this forward, ive updated:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers
• Changed int keys to String keys
• Made it clear a key can occur twice, aka multimap.
Is there any other blocking issues. Or do we all think we’re good to go to a vote?
Many thanks
Mike
On 10/02/2017, 22:57, "radai" <ra...@gmail.com> wrote:
yeah I'm ok with both strings and multi-maps
On Fri, Feb 10, 2017 at 2:37 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Can we also compromise on a multi-map?
>
> One of my use-cases is tagging on multi-DC setup. It will be nice if
> each "hop" can add "Tag" header with its own DC-name without having to
> edit a single string that someone else wrote. For some financial
> use-cases it will be a requirement.
>
> Gwen
>
> On Fri, Feb 10, 2017 at 12:19 PM, Ignacio Solis <is...@igso.net> wrote:
> > I would be willing to compromise on Strings as keys if that moves all
> > of us forward.
> >
> > Right now headers are blocking a few of the things we want to do.
> >
> > Nacho
> >
> > On Thu, Feb 9, 2017 at 2:14 PM, Jay Kreps <ja...@confluent.io> wrote:
> >> I'm neutral positive on adding headers conceptually. I'm -1 on a
> proposal
> >> using ints for header names. I just don't think a global namespace with
> >> integer names is an acceptable end-user interface given that our goal
> is to
> >> encourage a federated ecosystem of header users. Not trying to be a
> pain in
> >> the ass, I just think it would be a bit of a wart.
> >>
> >> -Jay
> >>
> >> On Thu, Feb 9, 2017 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >>> As we'd like to start work on this to make it into the next release in
> a
> >>> few months.
> >>>
> >>> If we were to put this to the vote today who would actually give this
> a -1
> >>> vote?
> >>>
> >>> Based on the number of identified usecases (some stronger than others)
> >>>
> >>> Also if we were to keep with int keys based on all the discussion
> thread
> >>> for the pros vs cons who would give a -1 as it stands? Eg anyone
> believe it
> >>> simply wouldn't work.
> >>>
> >>> If not so we make progress would it be sensible to put this to vote?
> So we
> >>> can start work?
> >>>
> >>> Cheers
> >>> Mike
> >>>
> >>>
> >>> Sent using OWA for iPhone
> >>> ________________________________________
> >>> From: Jeroen van Disseldorp <je...@axual.io>
> >>> Sent: Wednesday, February 1, 2017 2:43:45 PM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>>
> >>> I'm lagging a bit here too, but want to respond to the usefulness of
> >>> this KIP for two use cases: bidirectional mirroring and smart cluster
> >>> switches.
> >>>
> >>> Bidirectional mirroring
> >>>
> >>> For a Dutch bank we have currently implemented bidirectional mirroring
> >>> using our own headers, which we encode in the Value using a custom
> >>> serializer. It encodes just one byte extra, containing a
> "copy/original"
> >>> bit. When applications serialize an avro object, the bit is unset by
> >>> default. Our mirroring mechanism subscribes to all topics, checks the
> >>> bit, and when unset the bit is set and the message is forwarded to the
> >>> other DC.
> >>>
> >>> The overhead of this is pretty low (1 byte) and it's cheap, since the
> >>> mirroring mechanism does not need to fully deserialize the value, but
> >>> just check+flip one bit in the byte[] value. On the receiving end, the
> >>> mirroring mechanism sees the message for the second time, but notices
> >>> the message is a copy already and ignores it.
> >>>
> >>> The mechanism has been in production since last year and it works
> really
> >>> well. Its downside however is the incompatibility with avro serialized
> >>> values, as the whole organization is mandated to use our own custom
> >>> serde and we were forced to create a wrapper client library around the
> >>> default kafka client. The callback we offer to applications is very
> >>> similar to that proposed in the KIP, as every message our users get
> >>> consists of <Metadata, K, V>.
> >>>
> >>> Additionally our header contains a message id and serialization
> >>> timestamp (invented before 0.10.0.0). These fields could also be done
> >>> away with when this KIP is implemented.
> >>>
> >>> Smart cluster switches
> >>>
> >>> Another case we are considering to use headers for is enabling "smart
> >>> cluster switches" for consumers and producers in a way that would avoid
> >>> duplicates and/or missed messages in most cases. This involves encoding
> >>> a producer id (app name or id), producer instance id (uuid generated
> >>> upon producer's start) and sequence number in every message. We expect
> >>> to use one sequence per partition.
> >>>
> >>> These fields enable our consumers to keep track of which messages were
> >>> seen without relying on offsets, which is a cluster-bound value.
> >>> Consumers keep track of seen messages by keeping a small administration
> >>> of ranges of sequence numbers per producer instance and partition.
> Since
> >>> sequence numbers are always increasing per producer and partition, the
> >>> memory consumption of such administration is pretty low, while being
> >>> meaningful across clusters. After every poll() we can commit the
> >>> administration to a topic in much the same way as we do with offsets.
> >>> The difference is that this topic can be mirrored to the other cluster
> >>> and upon a switch of consumer, all instances can use the topic to
> >>> initialize themselves and skip all messages already processed.
> >>>
> >>> Hope these views help in the upcoming decision on this KIP.
> >>>
> >>> Regards
> >>> Jeroen
> >>>
> >>>
> >>>
> >>> On 09/01/2017 20:40, Gwen Shapira wrote:
> >>> > Mirroring + ClusterIDs:
> >>> >
> >>> > Kafka clusters already have an automatically generated ID that is
> >>> > exposed to clients:
> >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 78%3A+Cluster+Id
> >>> >
> >>> > It sounds trivial to get MirrorMaker's consumer to grab this ID
> (which
> >>> > always exists) from the source cluster and add it to the message
> >>> > headers.
> >>> >
> >>> > Obviously adding stuff to the message makes it larger so this should
> >>> > be a feature you should be able to turn on/off depending on the
> >>> > replication requirements. If you need the feature - you pay the
> price.
> >>> > I don't think there is an immediate need to put an effort into
> >>> > shrinking the Cluster ID, but others can comment on that.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> >>> >> Hey Michael,
> >>> >>
> >>> >> Sorry for lagging, I've been out for the holiday break. What would
> help
> >>> me
> >>> >> would be to get to a consolidated list of use cases we've worked
> out and
> >>> >> think add real value. I think getting down to the details really
> matters
> >>> >> for these.
> >>> >>
> >>> >> The two lists I'm aware of are the one in the KIP and the one you
> gave
> >>> in
> >>> >> this email thread. Both seem to be more at the sketch level. Let me
> >>> review
> >>> >> the use cases and see if I have it right. If so let's get it in the
> >>> design
> >>> >> proposal.
> >>> >>
> >>> >> Your use cases:
> >>> >>
> >>> >> 1. Tracing IDs -- I get this use case and think I see how it
> would
> >>> work.
> >>> >> I think because the tracing is from a third party system there
> is
> >>> some
> >>> >> value to having a header versus a custom format to make this
> work
> >>> fully
> >>> >> transparently.
> >>> >> 2. Mirroring -- I'm a bit unsure how this will work, I'll
> explain my
> >>> >> thinking below
> >>> >> 3. End-to-end encryption -- I think what you are saying is that
> if
> >>> you
> >>> >> want "routing' fields that are accessible in transit without
> >>> decrypting you
> >>> >> need a scheme to leave this decrypted.I think the idea is to
> avoid
> >>> >> decrypting to get a header, but I'm not sure if I get how this
> is
> >>> better
> >>> >> than a custom message format that doesn't encrypt the things you
> >>> don't want
> >>> >> encrypted since you have to be aware of the scheme anyway.
> >>> >> 4. EOS -- I'm not convinced this is a good approach for EOS.
> >>> >>
> >>> >> Linked from KIP:
> >>> >>
> >>> >> 1. Schema-Registry-aware avro serialization -- not clear how
> this is
> >>> >> better than keeping the schema id in the value
> >>> >> 2. Polyglot kafka topics -- marginally better than a container
> >>> format,
> >>> >> but of course you still have to all agree on the the header and
> >>> >> header=>serialization mapping so not sure if it's really better
> >>> than a
> >>> >> container format. Also not wild about polyglot topics :-)
> >>> >> 3. End-to-End routing trail including mirroring --- I'd put this
> >>> under
> >>> >> tracing ids
> >>> >> 4. Cross-service process tracing -- ditto
> >>> >> 5. Signing -- I like this use case but it seems to depend on a
> >>> feature
> >>> >> we don't have and aren't proposing adding, right?
> >>> >> 6. Per-record custom compression -- why do you want this?
> >>> compressing
> >>> >> per message won't work well compared to batch compression,
> right?
> >>> >> 7. Large Message Support -- I think we can agree that although
> this
> >>> >> might work it's clearly not the right way to support large
> messages
> >>> in Kafka
> >>> >>
> >>> >> Matthias
> >>> >>
> >>> >> 1. Control messages --- Matthias, not sure if I get the
> motivation
> >>> in
> >>> >> streams for this. If we did want to do it, wouldn't it be
> easier to
> >>> >> implement and by allowing users to send control messages (in the
> >>> EOS sense)
> >>> >> and adding a consumer config to allow either passing these on or
> >>> >> suppressing them?
> >>> >>
> >>> >> So I if I were to summarize I'd list the use cases as follows:
> >>> >>
> >>> >> - Much better with headers: tracing IDs
> >>> >> - Somewhat better: end-to-end encryption, polyglot topics
> >>> >> - Don't buy: Mirroring (though perhaps could be made to work
> with
> >>> >> changes to the proposal), EOS, large messages, compression
> >>> >> - Not sure: Control messages
> >>> >>
> >>> >> Does this match where other people are at? From my point of view for
> >>> >> something like this where we are changing the core data model we
> should
> >>> >> ideally get a handful of use cases to the point where they are
> really
> >>> well
> >>> >> thought out and we know they can use the feature. I think we have
> one
> >>> solid
> >>> >> one with tracing IDs, and I think maybe mirroring could be another
> with
> >>> >> some work. I'm less sure about the others.
> >>> >>
> >>> >> Bi-directional Mirroring
> >>> >> I think bidirectional mirroring could potentially become a
> compelling
> >>> use
> >>> >> case if we an find a way to make it work. If I understand it, the
> >>> proposal
> >>> >> is to add a header like "origin-cluster" with the cluster id of the
> >>> Kafka
> >>> >> cluster. Then modify the mirroring tool to disallow mirroring a
> message
> >>> >> back to it's own origin cluster. This basic idea is definitely
> workable
> >>> and
> >>> >> useful, but can we really get to a good implementation using
> headers?
> >>> >> I imagine the approach would be to add this header in the producer
> if
> >>> some
> >>> >> config is set. There are two problems:
> >>> >>
> >>> >> - You add a pretty big honking cluster ID to each message. For
> small
> >>> >> messages this maybe larger than the contents itself.
> >>> >> - What happens if someone somewhere forgets to set that config
> on
> >>> some
> >>> >> client somewhere?
> >>> >>
> >>> >> If someone forgets to set the config in some client somewhere I
> guess
> >>> you
> >>> >> have various options in mirror maker:
> >>> >>
> >>> >> 1. drop the data (pretty bad, now your data is out of sync)
> >>> >> 2. crash (pretty bad--how do you recover)
> >>> >> 3. insert some default (pretty bad since would also lead to data
> >>> loss if
> >>> >> that's not right)
> >>> >>
> >>> >> Basically although I think a cluster tag could be a good way to do
> >>> >> bi-directional mirroring, I'm not sure this naive approach I
> described
> >>> >> would get us to a usable implementation. This could be fixable. I
> think
> >>> if
> >>> >> there was some scheme to check required headers on the server that
> could
> >>> >> alleviate the missing header problem. Not really sure what to do
> about
> >>> the
> >>> >> message bloat problem, but maybe there could be some way to shrink
> the
> >>> >> cluster id?
> >>> >>
> >>> >> Cheers,
> >>> >>
> >>> >> -Jay
> >>> >>
> >>> >>
> >>> >> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> Michael.Pearce@ig.com
> >>> >
> >>> >> wrote:
> >>> >>
> >>> >>> @Jay, Ismael and a few others.
> >>> >>>
> >>> >>> Based on the discussion on kip-87, and the additional use case by
> >>> Matthias.
> >>> >>>
> >>> >>> Just for clarity do we now have consensus on the earlier points (1)
> >>> >>> headers are useful (2) headers should be native in Kafka?
> >>> >>>
> >>> >>> Cheers
> >>> >>> Mike
> >>> >>> ________________________________________
> >>> >>> From: radai <ra...@gmail.com>
> >>> >>> Sent: Monday, December 19, 2016 5:01 PM
> >>> >>> To: dev@kafka.apache.org
> >>> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>> >>>
> >>> >>> having said that im perfectly fine with something like a "no
> payload"
> >>> flag
> >>> >>> (as opposed to null payload), which would mean such a msg is not
> >>> returned
> >>> >>> from poll() unless the user code specifically "opts in" and would
> serve
> >>> >>> only to carry headers (making it a control message)
> >>> >>>
> >>> >>> On Mon, Dec 19, 2016 at 8:58 AM, radai <radai.rosenblatt@gmail.com
> >
> >>> wrote:
> >>> >>>
> >>> >>>> if all you want is to tag "end of stream" wouldnt a header on the
> last
> >>> >>> msg
> >>> >>>> in the stream be enough?
> >>> >>>>
> >>> >>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> >>> matthias@confluent.io>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> The details about headers for control messages are still to
> define.
> >>> But
> >>> >>>>> yes, the idea is to have some common default behavior that
> clients
> >>> would
> >>> >>>>> need to implement.
> >>> >>>>>
> >>> >>>>> The point is, that "regular headers" add meta data to regular
> >>> messages.
> >>> >>>>> Thus, those messages will be returned to the user via .poll().
> And
> >>> after
> >>> >>>>> the message is received the user can check if meta data is
> present
> >>> and
> >>> >>>>> read it.
> >>> >>>>>
> >>> >>>>> For control messages, we do not want those to pop up via .poll()
> as
> >>> >>>>> those are no regular messages. A client would need to opt-in to
> see
> >>> >>>>> those messages (either via poll() or maybe a callback). Thus, we
> need
> >>> >>>>> some special (standardized) header IDs that indicate control
> messages
> >>> >>>>> that should not be returned to the user via poll() by default.
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> -Matthias
> >>> >>>>>
> >>> >>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
> >>> >>>>>> Matthias,
> >>> >>>>>>
> >>> >>>>>> Thanks for your input. I'm +1 on control messages as they seem
> to
> >>> be
> >>> >>>>> the
> >>> >>>>>> simplest way to implement watermarks (
> >>> >>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-
> streaming-102
> >>> ),
> >>> >>> a
> >>> >>>>>> feature that would add a lot of value to Kafka Streams IMHO.
> >>> >>>>>>
> >>> >>>>>> Your argument that the control-message indicator belongs in the
> >>> >>>>> client-only
> >>> >>>>>> section of the record format make sense. Just to make sure I
> >>> >>>>> understand,
> >>> >>>>>> are you suggesting that control messages would be indicated by a
> >>> >>>>> standard
> >>> >>>>>> reserved header? By standard, I mean that ALL Kafka consumers
> would
> >>> >>>>> know
> >>> >>>>>> to handle these messages differently (possibly just ignoring
> them).
> >>> >>>>> This
> >>> >>>>>> would need to be added to the specification of the consumer
> protocol
> >>> >>> so
> >>> >>>>>> that all Kafka clients implement it, right? I think it's a good
> >>> idea
> >>> >>>>> but
> >>> >>>>>> just checking.
> >>> >>>>>>
> >>> >>>>>> Cheers,
> >>> >>>>>>
> >>> >>>>>> Roger
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> >>> >>> matthias@confluent.io
> >>> >>>>>> wrote:
> >>> >>>>>>
> >>> >>>>>>> Yes and no. I did overload the term "control message".
> >>> >>>>>>>
> >>> >>>>>>> EOS control messages are for client-broker communication and
> thus
> >>> >>> never
> >>> >>>>>>> exposed to any application. And I think this is a good design
> >>> because
> >>> >>>>>>> broker needs to understand those control messages. Thus, this
> >>> should
> >>> >>> be
> >>> >>>>>>> a protocol change.
> >>> >>>>>>>
> >>> >>>>>>> The type of control messages I have in mind are for
> client-client
> >>> >>>>>>> (application-application) communication and the broker is
> agnostic
> >>> to
> >>> >>>>>>> them. Thus, it should not be a protocol change.
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> -Matthias
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> On 12/14/16 9:42 AM, radai wrote:
> >>> >>>>>>>> arent control messages getting pushed as their own top level
> >>> >>> protocol
> >>> >>>>>>>> change (and a fairly massive one) for the transactions KIP ?
> >>> >>>>>>>>
> >>> >>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> >>> >>>>> matthias@confluent.io>
> >>> >>>>>>>> wrote:
> >>> >>>>>>>>
> >>> >>>>>>>>> Hi,
> >>> >>>>>>>>>
> >>> >>>>>>>>> I want to add a completely new angle to this discussion. For
> >>> this,
> >>> >>> I
> >>> >>>>>>>>> want to propose an extension for the headers feature that
> enables
> >>> >>> new
> >>> >>>>>>>>> uses cases -- and those new use cases might convince people
> to
> >>> >>>>> support
> >>> >>>>>>>>> headers (of course including the larger scoped proposal).
> >>> >>>>>>>>>
> >>> >>>>>>>>> Extended Proposal:
> >>> >>>>>>>>>
> >>> >>>>>>>>> Allow messages with a certain header key to be special
> "control
> >>> >>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an
> >>> application
> >>> >>>>> via
> >>> >>>>>>>>> .poll().
> >>> >>>>>>>>>
> >>> >>>>>>>>> Thus, a consumer client would automatically skip over those
> >>> >>>>> messages. If
> >>> >>>>>>>>> an application knows about embedded control messages, it can
> >>> "sing
> >>> >>>>> up"
> >>> >>>>>>>>> to those messages by the consumer client and either get a
> >>> callback
> >>> >>> or
> >>> >>>>>>>>> the consumer auto-drop for this messages gets disabled
> (allowing
> >>> to
> >>> >>>>>>>>> consumer those messages via poll()).
> >>> >>>>>>>>>
> >>> >>>>>>>>> (The details need further considerations/discussion. I just
> want
> >>> to
> >>> >>>>>>>>> sketch the main idea.)
> >>> >>>>>>>>>
> >>> >>>>>>>>> Usage:
> >>> >>>>>>>>>
> >>> >>>>>>>>> There is a shared topic (ie, used by multiple applications)
> and a
> >>> >>>>>>>>> producer application wants to embed a special message in the
> >>> topic
> >>> >>>>> for a
> >>> >>>>>>>>> dedicated consumer application. Because only one application
> will
> >>> >>>>>>>>> understand this message, it cannot be a regular message as
> this
> >>> >>> would
> >>> >>>>>>>>> break all applications that do not understand this message.
> The
> >>> >>>>> producer
> >>> >>>>>>>>> application would set a special metadata key and no consumer
> >>> >>>>> application
> >>> >>>>>>>>> would see this control message by default because they did
> not
> >>> >>> enable
> >>> >>>>>>>>> their consumer client to return this message in poll() (and
> the
> >>> >>>>> client
> >>> >>>>>>>>> would just drop this message with special metadata key).
> Only the
> >>> >>>>> single
> >>> >>>>>>>>> application that should receive this message, will subscribe
> to
> >>> >>> this
> >>> >>>>>>>>> message on its consumer client and process it.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Concrete Use Case: Kafka Streams
> >>> >>>>>>>>>
> >>> >>>>>>>>> In Kafka Streams, we would like to propagate "control
> messages"
> >>> >>> from
> >>> >>>>>>>>> subtopology to subtopology. There are multiple scenarios for
> >>> which
> >>> >>>>> this
> >>> >>>>>>>>> would be useful. For example, currently we do not guarantee a
> >>> >>>>>>>>> "consistent shutdown" of an application. By this, I mean that
> >>> input
> >>> >>>>>>>>> records might not be completely processed by the whole
> topology
> >>> >>>>> because
> >>> >>>>>>>>> the application shutdown happens "in between" and an
> intermediate
> >>> >>>>> result
> >>> >>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user
> would
> >>> see
> >>> >>>>> an
> >>> >>>>>>>>> committed offset of the source topic of the application, but
> no
> >>> >>>>>>>>> corresponding result record in the output topic.
> >>> >>>>>>>>>
> >>> >>>>>>>>> Having "shutdown markers" would allow us, to first stop the
> >>> >>> upstream
> >>> >>>>>>>>> subtopology and write this marker into the intermediate
> topic and
> >>> >>> the
> >>> >>>>>>>>> downstream subtopology would only shut down itself after is
> sees
> >>> >>> the
> >>> >>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that
> no
> >>> >>>>>>>>> "in-flight" messages got stuck in intermediate topics.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch
> >>> Processing).
> >>> >>>>>>>>> There was a discussion about the proposed metadata topic,
> and we
> >>> >>>>> could
> >>> >>>>>>>>> avoid this metadata topic if we would have "control
> messages".
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Right now, we cannot insert an "application control message"
> >>> >>> because
> >>> >>>>>>>>> Kafka Streams does not own all topics it read/writes and thus
> >>> might
> >>> >>>>>>>>> break other consumer application (as described above) if we
> >>> inject
> >>> >>>>>>>>> random messages that are not understood by other apps.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Of course, one can work around "embedded control messaged" by
> >>> using
> >>> >>>>> an
> >>> >>>>>>>>> additional topic to propagate control messaged between
> >>> application
> >>> >>>>> (as
> >>> >>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka
> Streams). But
> >>> >>>>> there
> >>> >>>>>>>>> are major concerns about adding this metadata topic in the
> KIP
> >>> and
> >>> >>>>> this
> >>> >>>>>>>>> shows that other application that need a similar pattern
> might
> >>> >>> profit
> >>> >>>>>>>>> from topic embedded "control messages", too.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> One last important consideration: those "control messages"
> are
> >>> used
> >>> >>>>> for
> >>> >>>>>>>>> client to client communication and are not understood by the
> >>> >>> broker.
> >>> >>>>>>>>> Thus, those messages should not be enabled within the message
> >>> >>> format
> >>> >>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> record
> >>> >>>>> headers
> >>> >>>>>>>>> would be a nice way to implement them. Because KIP-82 did
> >>> consider
> >>> >>>>> key
> >>> >>>>>>>>> namespaces for metatdata keys, this extension should not be
> an
> >>> own
> >>> >>>>> KIP
> >>> >>>>>>>>> but should be included in KIP-82 to reserve a namespace for
> >>> >>> "control
> >>> >>>>>>>>> message" in the first place.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Sorry for the long email... Looking forward to your feedback.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> -Matthias
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> >>> >>>>>>>>>> Hi Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
> >>> >>>>> (HTTP/JMS/
> >>> >>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
> >>> >>> (though I
> >>> >>>>>>>>> believe it contains the end2end uuid embedded in this id), on
> >>> >>>>> receiving
> >>> >>>>>>> the
> >>> >>>>>>>>> message at the receiving JVM the apm code takes this out, and
> >>> >>>>> continues
> >>> >>>>>>> its
> >>> >>>>>>>>> tracing on the that new thread. Both JVM's (and other
> languages
> >>> the
> >>> >>>>> APM
> >>> >>>>>>>>> tool supports) send this data async back to the central
> >>> controllers
> >>> >>>>>>> where
> >>> >>>>>>>>> the stiching togeather occurs. For this they need some header
> >>> space
> >>> >>>>> for
> >>> >>>>>>>>> them to put this id.
> >>> >>>>>>>>>> 101) Yes indeed we have a business transaction Id in the
> >>> payload.
> >>> >>>>>>> Though
> >>> >>>>>>>>> this is a system level tracing, that we need to have marry
> up.
> >>> Also
> >>> >>>>> as
> >>> >>>>>>> per
> >>> >>>>>>>>> note on end2end encryption we'd be unable to prove the flow
> if
> >>> the
> >>> >>>>>>> payload
> >>> >>>>>>>>> is encrypted as we'd not have access to this at certain
> points of
> >>> >>> the
> >>> >>>>>>> flow
> >>> >>>>>>>>> through the infrastructure/platform.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 103) As said we use this mechanism in IG very successfully,
> as
> >>> >>>>> stated
> >>> >>>>>>>>> per key we guarantee the transaction producing app to handle
> the
> >>> >>>>>>>>> transaction of a key at one DC unless at point of critical
> >>> failure
> >>> >>>>>>> where we
> >>> >>>>>>>>> have to flip processing to another. We care about key
> ordering.
> >>> >>>>>>>>>> I disagree on the offset comment for the partition solution
> >>> unless
> >>> >>>>> you
> >>> >>>>>>>>> do full ISR, or expensive full XA transactions even with
> >>> partitions
> >>> >>>>> you
> >>> >>>>>>>>> cannot fully guarantee offsets would match.
> >>> >>>>>>>>>> 105) Very much so, I need to have access at the platform
> level
> >>> to
> >>> >>>>> the
> >>> >>>>>>>>> other meta data all mentioned, without having to need to have
> >>> >>> access
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> encryption keys of the payload.
> >>> >>>>>>>>>> 106)
> >>> >>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to
> >>> have
> >>> >>> a
> >>> >>>>>>>>> global producerId register which would be very hard to
> >>> >>>>> enforce/ensure is
> >>> >>>>>>>>> current and correct, just to understand the message origins
> of
> >>> its
> >>> >>>>>>>>> region/az/cluster for routing.
> >>> >>>>>>>>>> The client wrapper version, producerId can be the same, as
> >>> >>> obviously
> >>> >>>>>>> the
> >>> >>>>>>>>> producer could upgrade its wrapper, as such we need to know
> what
> >>> >>>>> wrapper
> >>> >>>>>>>>> version the message is created with.
> >>> >>>>>>>>>> Likewise the IP address, as stated we can have our producer
> >>> move,
> >>> >>>>> where
> >>> >>>>>>>>> its IP would change.
> >>> >>>>>>>>>> 107)
> >>> >>>>>>>>>> UUID is set on the message by interceptors before actual
> >>> producer
> >>> >>>>>>>>> transport send. This is for platform level message dedupe
> >>> >>> guarantee,
> >>> >>>>> the
> >>> >>>>>>>>> business payload should be agnostic to this. Please see
> >>> >>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
> >>> >>>>> ection.html
> >>> >>>>>>>>> note this is not touching business payloads.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Hi, Michael,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Thanks for the reply. I find it very helpful.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Data lineage:
> >>> >>>>>>>>>> 100. I'd like to understand the APM use case a bit
> more. It
> >>> >>>>> sounds
> >>> >>>>>>>>> like
> >>> >>>>>>>>>> that those APM plugins can generate a transaction id
> that
> >>> we
> >>> >>>>> could
> >>> >>>>>>>>>> potentially put in the header of every message. How
> would
> >>> you
> >>> >>>>>>>>> typically
> >>> >>>>>>>>>> make use of such transaction ids? Are there other
> metadata
> >>> >>>>>>>>> associated with
> >>> >>>>>>>>>> the transaction id and if so, how are they propagated
> >>> >>>>> downstream?
> >>> >>>>>>>>>> 101. For the finance use case, if the concept of
> >>> transaction
> >>> >>> is
> >>> >>>>>>>>> important,
> >>> >>>>>>>>>> wouldn't it be typically included in the message
> payload
> >>> >>>>> instead of
> >>> >>>>>>>>> as an
> >>> >>>>>>>>>> optional header field?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 102. The data lineage that Altas and Navigator support
> >>> seems
> >>> >>> to
> >>> >>>>> be
> >>> >>>>>>>>> at the
> >>> >>>>>>>>>> dataset level, not per record level? So, not sure if
> per
> >>> >>> message
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> are relevant there.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Mirroring:
> >>> >>>>>>>>>> 103. The benefit of using separate partitions is that
> it
> >>> >>>>>>> potentially
> >>> >>>>>>>>> makes
> >>> >>>>>>>>>> it easy to preserve offsets during mirroring. This will
> >>> make
> >>> >>> it
> >>> >>>>>>>>> easier for
> >>> >>>>>>>>>> consumer to switch clusters. Currently, the consumers
> can
> >>> >>> switch
> >>> >>>>>>>>> clusters
> >>> >>>>>>>>>> by using the timestampToOffset() api, but it has to
> deal
> >>> with
> >>> >>>>>>>>> duplicates.
> >>> >>>>>>>>>> Good point on the issue with log compact and I am not
> sure
> >>> how
> >>> >>>>> to
> >>> >>>>>>>>> address
> >>> >>>>>>>>>> this. However, even if we mirror into the existing
> >>> partitions,
> >>> >>>>> the
> >>> >>>>>>>>> ordering
> >>> >>>>>>>>>> for messages generated from different clusters seems
> >>> >>>>>>>>> non-deterministic
> >>> >>>>>>>>>> anyway. So, it seems that the consumers already have to
> >>> deal
> >>> >>>>> with
> >>> >>>>>>>>> that? If
> >>> >>>>>>>>>> a topic is compacted, does that mean which messages are
> >>> >>>>> preserved
> >>> >>>>>>> is
> >>> >>>>>>>>> also
> >>> >>>>>>>>>> non-deterministic across clusters?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 104. Good point on partition key.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> End-to-end encryption:
> >>> >>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are
> >>> headers
> >>> >>>>>>>>> useful there?
> >>> >>>>>>>>>> Auditing:
> >>> >>>>>>>>>> 106. It seems other than the UUID, all other metadata
> are
> >>> per
> >>> >>>>>>>>> producer?
> >>> >>>>>>>>>> EOS:
> >>> >>>>>>>>>> 107. How are those UUIDs generated? I am not sure if
> they
> >>> can
> >>> >>> be
> >>> >>>>>>>>> generated
> >>> >>>>>>>>>> in the producer library. An application may send
> messages
> >>> >>>>> through a
> >>> >>>>>>>>> load
> >>> >>>>>>>>>> balancer and on retry, the same message could be
> routed to
> >>> a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> producer instance. So, it seems that the application
> has to
> >>> >>>>>>> generate
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> UUIDs. In that case, shouldn't the application just
> put the
> >>> >>>>> UUID in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> payload?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Thanks,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> >>> >>>>>>>>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> > Hi Jun.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Per Transaction Tracing / Data Lineage.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As Stated in the KIP this has the first use case of
> how
> >>> many
> >>> >>>>> APM
> >>> >>>>>>>>> tools now
> >>> >>>>>>>>>> > work.
> >>> >>>>>>>>>> > I would find it impossible for any one to argue this
> is
> >>> not
> >>> >>>>>>>>> important or a
> >>> >>>>>>>>>> > niche market as it has its own gartner report for
> this
> >>> >>> space.
> >>> >>>>>>> Such
> >>> >>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace,
> Hawqular
> >>> are
> >>> >>>>> but a
> >>> >>>>>>>>> few.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise these APM tools can help very rapidly track
> down
> >>> >>>>> issues
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > automatically capture metrics, perform actions based
> on
> >>> >>>>>>> unexpected
> >>> >>>>>>>>> behavior
> >>> >>>>>>>>>> > to auto recover services.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Before mentioning looking at aggregated stats, in
> these
> >>> >>> cases
> >>> >>>>>>> where
> >>> >>>>>>>>>> > actually on critical flows we cannot afford to have
> >>> >>> aggregated
> >>> >>>>>>>>> rolled up
> >>> >>>>>>>>>> > stats only.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > With the APM tool we use its actually able to detect
> a
> >>> >>> single
> >>> >>>>>>>>> transaction
> >>> >>>>>>>>>> > failure and capture the thread traces in the JVM
> where it
> >>> >>>>> failed
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > everything for us, to the point it sends us alerts
> where
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > giving the line number of the code that caused it,
> the
> >>> >>>>>>> transaction
> >>> >>>>>>>>> trace
> >>> >>>>>>>>>> > through all the services and endpoints (supported)
> upto
> >>> the
> >>> >>>>> point
> >>> >>>>>>>>> of
> >>> >>>>>>>>>> > failure, it can also capture the data in and out (so
> we
> >>> can
> >>> >>>>>>>>> replay).
> >>> >>>>>>>>>> > Because atm Kafka doesn't support us being able to
> stich
> >>> in
> >>> >>>>> these
> >>> >>>>>>>>> tracing
> >>> >>>>>>>>>> > transaction ids natively, we cannot get these
> benefits as
> >>> >>>>> such is
> >>> >>>>>>>>> limiting
> >>> >>>>>>>>>> > our ability support apps and monitor them to the same
> >>> >>>>> standards
> >>> >>>>>>> we
> >>> >>>>>>>>> come to
> >>> >>>>>>>>>> > expect when on a kafka flow.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > This actually ties in with Data Lineage, as the same
> >>> tracing
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>> > to back stich this. Essentially many times due to the
> >>> sums
> >>> >>> of
> >>> >>>>>>> money
> >>> >>>>>>>>>> > involved there are disputes, and typically as a
> financial
> >>> >>>>>>>>> institute the
> >>> >>>>>>>>>> > easiest and cleanest way to prove when disputes
> arise is
> >>> to
> >>> >>>>>>>>> present the
> >>> >>>>>>>>>> > actual flow and processes involved in a transaction.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise as Hadoop matures its evident this case is
> >>> >>>>> important, as
> >>> >>>>>>>>> tools
> >>> >>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator
> (cloudera
> >>> led)
> >>> >>>>> are
> >>> >>>>>>>>> evident
> >>> >>>>>>>>>> > also I believe the importance here is very much NOT
> just
> >>> a
> >>> >>>>>>>>> financial issue.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > From a MDM point of view any company wanting to care
> >>> about
> >>> >>>>> Data
> >>> >>>>>>>>> Quality
> >>> >>>>>>>>>> > and Data Governance - Data Lineage is a key piece in
> this
> >>> >>>>> puzzle.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Mirroring,
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As per the KIP in-fact this is exactly what we do re
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> to mirror
> >>> >>>>>>>>>> > a network of clusters between AZ's / Regions. We
> know a
> >>> >>>>>>>>> transaction for a
> >>> >>>>>>>>>> > key will be done within a AZ/Region, as such we
> know the
> >>> >>>>> write
> >>> >>>>>>> to
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>> > would be ordered per key. But we need eventual view
> of
> >>> that
> >>> >>>>>>> across
> >>> >>>>>>>>> in our
> >>> >>>>>>>>>> > other regions/az's. When we have complete AZ or
> Region
> >>> >>>>> failure we
> >>> >>>>>>>>> know
> >>> >>>>>>>>>> > there will be a brief interruption whilst those
> >>> transactions
> >>> >>>>> are
> >>> >>>>>>>>> moved to
> >>> >>>>>>>>>> > another region but we expect after it to continue.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As mentioned having separate Partions to do this
> starts
> >>> to
> >>> >>> get
> >>> >>>>>>>>>> > ugly/complicated for us:
> >>> >>>>>>>>>> > how would I do compaction where a key is in two
> >>> partitions?
> >>> >>>>>>>>>> > How do we balance consumers so where multiple
> partitions
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>> same key
> >>> >>>>>>>>>> > goto the same consumer
> >>> >>>>>>>>>> > What do you do if cluster 1 has 5 partitions but
> cluster
> >>> 20
> >>> >>>>> has
> >>> >>>>>>> 10
> >>> >>>>>>>>> because
> >>> >>>>>>>>>> > its larger kit in our more core DC's, as such key to
> >>> >>> partition
> >>> >>>>>>>>> mappings for
> >>> >>>>>>>>>> > consumers get even more complicated.
> >>> >>>>>>>>>> > What do you do if we add or remove a complete region
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Where as simple mirror will work we just need to
> ensure
> >>> we
> >>> >>>>> don't
> >>> >>>>>>>>> have a
> >>> >>>>>>>>>> > cycle which we can do with clusterId.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > We even have started to look at shortest path mirror
> >>> routing
> >>> >>>>>>> based
> >>> >>>>>>>>> on
> >>> >>>>>>>>>> > clusterId, if we also had the region and az info on
> the
> >>> >>>>>>> originating
> >>> >>>>>>>>>> > message, this we have not implemented but some ideas
> come
> >>> >>> from
> >>> >>>>>>>>> network
> >>> >>>>>>>>>> > routing, and also the dispatcher router in apache
> qpid.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Also we need to have data perimeters e.g. certain
> data
> >>> >>> cannot
> >>> >>>>>>> leave
> >>> >>>>>>>>>> > certain countries borders. We want this all
> automated so
> >>> >>> that
> >>> >>>>> at
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > platform level without having to touch or look at the
> >>> >>> business
> >>> >>>>>>>>> data inside
> >>> >>>>>>>>>> > we can have headers we can put tags into so that we
> can
> >>> >>> ensure
> >>> >>>>>>>>> this doesn't
> >>> >>>>>>>>>> > occur when we mirror. (actually links in to data
> lineage
> >>> /
> >>> >>>>>>> tracing
> >>> >>>>>>>>> as again
> >>> >>>>>>>>>> > we need to tag messages at a platform level) Examples
> >>> are we
> >>> >>>>> are
> >>> >>>>>>>>> not
> >>> >>>>>>>>>> > allowed Private customer details to leave
> Switzerland,
> >>> yet
> >>> >>> we
> >>> >>>>>>> need
> >>> >>>>>>>>> those
> >>> >>>>>>>>>> > systems integrated.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Lastly around mirroring we have a partionKey field,
> as
> >>> the
> >>> >>> key
> >>> >>>>>>>>> used for
> >>> >>>>>>>>>> > portioning logic != compaction key all the time but
> we
> >>> want
> >>> >>> to
> >>> >>>>>>>>> preserve it
> >>> >>>>>>>>>> > for when we mirror so that if source cluster
> partition
> >>> count
> >>> >>>>> !=
> >>> >>>>>>>>> destination
> >>> >>>>>>>>>> > cluster partition count we can honour the same
> >>> partitioning
> >>> >>>>>>> logic.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE End 2 End encryption
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As I believe mentioned just before, the solution you
> >>> mention
> >>> >>>>> just
> >>> >>>>>>>>> doesn't
> >>> >>>>>>>>>> > cut the mustard these days with many regulators. An
> >>> >>> operations
> >>> >>>>>>>>> person with
> >>> >>>>>>>>>> > access to the box should not be able to have access
> to
> >>> the
> >>> >>>>> data.
> >>> >>>>>>>>> Many now
> >>> >>>>>>>>>> > actually impose quite literally the implementation
> >>> expected
> >>> >>>>> being
> >>> >>>>>>>>> end2end
> >>> >>>>>>>>>> > encryption for certain data (Singapore for us is one
> >>> that I
> >>> >>> am
> >>> >>>>>>>>> most aware
> >>> >>>>>>>>>> > of). In fact we're even now needing encrypt the data
> and
> >>> >>> store
> >>> >>>>>>> the
> >>> >>>>>>>>> keys in
> >>> >>>>>>>>>> > HSM modules.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise the performance penalty on encrypting
> >>> decrypting as
> >>> >>>>> you
> >>> >>>>>>>>> produce
> >>> >>>>>>>>>> > over wire, then again encrypt decrypt as the data is
> >>> stored
> >>> >>> on
> >>> >>>>>>> the
> >>> >>>>>>>>> brokers
> >>> >>>>>>>>>> > disks and back again, then again encrypted and
> decrypted
> >>> >>> back
> >>> >>>>>>> over
> >>> >>>>>>>>> the wire
> >>> >>>>>>>>>> > each time for each consumer all adds up, ignoring
> this
> >>> >>>>> doubling
> >>> >>>>>>>>> with mirror
> >>> >>>>>>>>>> > makers etc. simply encrypting the value once on
> write by
> >>> the
> >>> >>>>>>>>> client and
> >>> >>>>>>>>>> > again decrypting on consume by the consumer is far
> more
> >>> >>>>>>>>> performant, but
> >>> >>>>>>>>>> > then the routing and platform meta data needs to be
> >>> separate
> >>> >>>>>>> (thus
> >>> >>>>>>>>> headers)
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Auditing:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Our Auditing needs are:
> >>> >>>>>>>>>> > Producer Id,
> >>> >>>>>>>>>> > Origin Cluster Id that message first produced into
> >>> >>>>>>>>>> > Origin AZ - agreed we can derive this if we have
> cluster
> >>> id,
> >>> >>>>> but
> >>> >>>>>>>>> it makes
> >>> >>>>>>>>>> > resolving this for audit reporting a lot easier.
> >>> >>>>>>>>>> > Origin Region - agreed we can derive this if we have
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> but it
> >>> >>>>>>>>>> > makes resolving this for audit reporting a lot
> easier.
> >>> >>>>>>>>>> > Unique Message Identification (this is not the same
> as
> >>> >>>>>>> transaction
> >>> >>>>>>>>>> > tracing) - note offset and partition are not the
> same, as
> >>> >>>>> when we
> >>> >>>>>>>>> mirror or
> >>> >>>>>>>>>> > have for what ever system failure duplicate send,
> >>> >>>>>>>>>> > Custom Client wrapper version (where organizations
> have
> >>> to
> >>> >>>>> wrap
> >>> >>>>>>>>> the kafka
> >>> >>>>>>>>>> > client for added features) so we know what version
> of the
> >>> >>>>> wrapper
> >>> >>>>>>>>> is used
> >>> >>>>>>>>>> > Producer IP address (in case of clients being in our
> >>> vm/open
> >>> >>>>>>> stack
> >>> >>>>>>>>> infra
> >>> >>>>>>>>>> > where they can move around, producer id will stay the
> >>> same
> >>> >>> but
> >>> >>>>>>>>> this would
> >>> >>>>>>>>>> > change)
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Once and only once delivery case
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Using the same Message UUID for auditing we can
> achieve
> >>> this
> >>> >>>>>>> quite
> >>> >>>>>>>>> simply.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As per how some other brokers do this (cough qpid,
> >>> artemis)
> >>> >>>>>>>>> message uuid
> >>> >>>>>>>>>> > are used to dedupe where message is sent and
> produced but
> >>> >>> the
> >>> >>>>>>>>> client didn't
> >>> >>>>>>>>>> > receive the ack, and there for replays the send, by
> >>> having a
> >>> >>>>>>>>> unique message
> >>> >>>>>>>>>> > id per message, this can be filtered out, on
> consumers
> >>> where
> >>> >>>>>>>>> message
> >>> >>>>>>>>>> > delivery may occur twice for what ever reasons a
> message
> >>> >>> uuid
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>> > to remove duplicates being deliverd , like wise we
> can do
> >>> >>>>> this in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > mirrormakers so if we detect a dupe message we can
> avoid
> >>> >>>>>>>>> replicating it.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Cheers
> >>> >>>>>>>>>> > Mike
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
> >>> wrote:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Since this KIP affects message format, wire
> protocol,
> >>> >>>>> apis, I
> >>> >>>>>>>>> think
> >>> >>>>>>>>>> > it's
> >>> >>>>>>>>>> > worth spending a bit more time to nail down the
> >>> concrete
> >>> >>>>> use
> >>> >>>>>>>>> cases. It
> >>> >>>>>>>>>> > would be bad if we add this feature, but when
> start
> >>> >>>>>>>>> implementing it
> >>> >>>>>>>>>> > for say
> >>> >>>>>>>>>> > mirroring, we then realize that header is not the
> >>> best
> >>> >>>>>>>>> approach.
> >>> >>>>>>>>>> > Initially,
> >>> >>>>>>>>>> > I thought I was convinced of the use cases of
> headers
> >>> >>> and
> >>> >>>>> was
> >>> >>>>>>>>> trying to
> >>> >>>>>>>>>> > write down a few use cases to convince others.
> That's
> >>> >>>>> when I
> >>> >>>>>>>>> became
> >>> >>>>>>>>>> > less
> >>> >>>>>>>>>> > certain. For me to be convinced, I just want to
> see
> >>> two
> >>> >>>>>>> strong
> >>> >>>>>>>>> use
> >>> >>>>>>>>>> > cases
> >>> >>>>>>>>>> > (instead of 10 maybe use cases) in the
> third-party
> >>> >>> space.
> >>> >>>>> The
> >>> >>>>>>>>> reason is
> >>> >>>>>>>>>> > that when we discussed the use cases within a
> >>> company,
> >>> >>>>> often
> >>> >>>>>>>>> it ends
> >>> >>>>>>>>>> > with
> >>> >>>>>>>>>> > "we can't force everyone to use this standard
> since
> >>> we
> >>> >>> may
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>> > integrate with third-party tools".
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > At present, I am not sure why headers are useful
> for
> >>> >>>>> things
> >>> >>>>>>>>> like
> >>> >>>>>>>>>> > schemaId
> >>> >>>>>>>>>> > or encryption. In order to do anything useful to
> the
> >>> >>>>> value,
> >>> >>>>>>>>> one needs
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > know the schemaId or how data is encrypted, but
> >>> header
> >>> >>> is
> >>> >>>>>>>>> optional.
> >>> >>>>>>>>>> > But, I
> >>> >>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
> >>> >>> provides
> >>> >>>>>>> more
> >>> >>>>>>>>> details
> >>> >>>>>>>>>> > on
> >>> >>>>>>>>>> > the argument.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I am not very sure header is the best approach
> for
> >>> >>>>> mirroring
> >>> >>>>>>>>> either. If
> >>> >>>>>>>>>> > someone has thought about this more, I'd be
> happy to
> >>> >>> hear.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I can see the data lineage use case. I am just
> not
> >>> sure
> >>> >>>>> how
> >>> >>>>>>>>> widely
> >>> >>>>>>>>>> > applicable this is. If someone familiar with this
> >>> space
> >>> >>>>> can
> >>> >>>>>>>>> justify
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > a significant use case, say in the finance
> industry,
> >>> >>> this
> >>> >>>>>>>>> would be a
> >>> >>>>>>>>>> > strong
> >>> >>>>>>>>>> > use case.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I can see the auditing use case. I am just not
> sure
> >>> if a
> >>> >>>>>>> native
> >>> >>>>>>>>>> > producer id
> >>> >>>>>>>>>> > solves that problem. If there are additional
> metadata
> >>> >>>>> that's
> >>> >>>>>>>>> worth
> >>> >>>>>>>>>> > collecting but not covered by the producer id,
> that
> >>> >>> would
> >>> >>>>>>> make
> >>> >>>>>>>>> this a
> >>> >>>>>>>>>> > strong use case.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Thanks,
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Jun
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > > this KIP is about enabling headers, nothing
> more
> >>> >>> nothing
> >>> >>>>>>>>> less - so
> >>> >>>>>>>>>> > no,
> >>> >>>>>>>>>> > > broker-side use of headers is not in the KIP
> scope.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > obviously though, once you have headers
> potential
> >>> use
> >>> >>>>> cases
> >>> >>>>>>>>> could
> >>> >>>>>>>>>> > include
> >>> >>>>>>>>>> > > broker-side header-aware interceptors (which
> would
> >>> be
> >>> >>>>> the
> >>> >>>>>>>>> topic of
> >>> >>>>>>>>>> > other
> >>> >>>>>>>>>> > > future KIPs).
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > a trivially clear use case (to me) would be
> using
> >>> such
> >>> >>>>>>>>> broker-side
> >>> >>>>>>>>>> > > interceptors to enforce compliance with
> >>> organizational
> >>> >>>>>>>>> policies - it
> >>> >>>>>>>>>> > would
> >>> >>>>>>>>>> > > make our SREs lives much easier if instead of
> >>> >>>>> retroactively
> >>> >>>>>>>>>> > discovering
> >>> >>>>>>>>>> > > "rogue" topics/users those messages would have
> been
> >>> >>>>>>> rejected
> >>> >>>>>>>>>> > up-front.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > the kafka broker code is lacking any such
> >>> >>> extensibility
> >>> >>>>>>>>> support
> >>> >>>>>>>>>> > (beyond
> >>> >>>>>>>>>> > > maybe authorizer) which is why these use cases
> were
> >>> >>> left
> >>> >>>>>>> out
> >>> >>>>>>>>> of the
> >>> >>>>>>>>>> > "case
> >>> >>>>>>>>>> > > for headers" doc - broker extensibility is a
> >>> separate
> >>> >>>>>>>>> discussion.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
> >>> >>>>>>>>> gwen@confluent.io>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > > Woah, I wasn't aware this is something we'll
> do.
> >>> It
> >>> >>>>>>> wasn't
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > KIP,
> >>> >>>>>>>>>> > > > right?
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > I guess we could do it the same way ACLs
> >>> currently
> >>> >>>>> work.
> >>> >>>>>>>>>> > > > I had in mind something that will allow
> admins to
> >>> >>>>> apply
> >>> >>>>>>>>> rules to
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > new create/delete/config topic APIs. So Todd
> can
> >>> >>>>> decide
> >>> >>>>>>> to
> >>> >>>>>>>>> reject
> >>> >>>>>>>>>> > > > "create topic" requests that ask for more
> than 40
> >>> >>>>>>>>> partitions, or
> >>> >>>>>>>>>> > > > require exactly 3 replicas, or no more than
> 50GB
> >>> >>>>>>> partition
> >>> >>>>>>>>> size,
> >>> >>>>>>>>>> > etc.
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are
> planning
> >>> to
> >>> >>>>> apply
> >>> >>>>>>>>> more
> >>> >>>>>>>>>> > rules
> >>> >>>>>>>>>> > > > to requests (and I think we should), we may
> want
> >>> a
> >>> >>> bit
> >>> >>>>>>>>> more generic
> >>> >>>>>>>>>> > > > design around that.
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > wrote:
> >>> >>>>>>>>>> > > > > "wouldn't you be in the business of making
> sure
> >>> >>>>>>> everyone
> >>> >>>>>>>>> uses
> >>> >>>>>>>>>> > them
> >>> >>>>>>>>>> > > > > properly?"
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > > thats where a broker-side plugin would come
> >>> handy
> >>> >>> -
> >>> >>>>> any
> >>> >>>>>>>>> incoming
> >>> >>>>>>>>>> > > message
> >>> >>>>>>>>>> > > > > that does not conform to org policy (read -
> >>> does
> >>> >>> not
> >>> >>>>>>>>> have the
> >>> >>>>>>>>>> > proper
> >>> >>>>>>>>>> > > > > headers) gets thrown out (with an error
> >>> returned
> >>> >>> to
> >>> >>>>>>> user)
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> Palino <
> >>> >>>>>>>>> tpalino@gmail.com>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > >> Come on, I've done at least 2 talks on
> this
> >>> one
> >>> >>> :)
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> Producing counts to a topic is part of
> it, but
> >>> >>>>> that's
> >>> >>>>>>>>> only
> >>> >>>>>>>>>> > part. So
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> count you have 100 messages in topic A.
> When
> >>> you
> >>> >>>>>>> mirror
> >>> >>>>>>>>> topic A
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > > another
> >>> >>>>>>>>>> > > > >> cluster, you have 99 messages. Where was
> your
> >>> >>>>> problem?
> >>> >>>>>>>>> Or
> >>> >>>>>>>>>> > worse, you
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> 100 messages, but one producer duplicated
> >>> >>> messages
> >>> >>>>> and
> >>> >>>>>>>>> another
> >>> >>>>>>>>>> > one
> >>> >>>>>>>>>> > > lost
> >>> >>>>>>>>>> > > > >> messages. You need details about where the
> >>> >>> message
> >>> >>>>>>> came
> >>> >>>>>>>>> from in
> >>> >>>>>>>>>> > order
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> pinpoint problems when they happen. Source
> >>> >>> producer
> >>> >>>>>>>>> info, where
> >>> >>>>>>>>>> > it was
> >>> >>>>>>>>>> > > > >> produced into your infrastructure, and
> when it
> >>> >>> was
> >>> >>>>>>>>> produced.
> >>> >>>>>>>>>> > This
> >>> >>>>>>>>>> > > > requires
> >>> >>>>>>>>>> > > > >> you to add the information to the message.
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> And yes, you still need to maintain your
> >>> clients.
> >>> >>>>> So
> >>> >>>>>>>>> maybe my
> >>> >>>>>>>>>> > original
> >>> >>>>>>>>>> > > > >> example was not the best. My thoughts on
> not
> >>> >>>>> wanting
> >>> >>>>>>> to
> >>> >>>>>>>>> be
> >>> >>>>>>>>>> > responsible
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> message formats stands, because that's
> very
> >>> much
> >>> >>>>>>>>> separate from
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > client.
> >>> >>>>>>>>>> > > > >> As you know, we have our own internal
> client
> >>> >>>>> library
> >>> >>>>>>>>> that can
> >>> >>>>>>>>>> > insert
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> right headers, and right now inserts the
> right
> >>> >>>>> audit
> >>> >>>>>>>>>> > information into
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> message fields. If they exist, and
> assuming
> >>> the
> >>> >>>>>>> message
> >>> >>>>>>>>> is Avro
> >>> >>>>>>>>>> > > encoded.
> >>> >>>>>>>>>> > > > >> What if someone wants to use JSON instead
> for
> >>> a
> >>> >>>>> good
> >>> >>>>>>>>> reason?
> >>> >>>>>>>>>> > What if
> >>> >>>>>>>>>> > > > user X
> >>> >>>>>>>>>> > > > >> wants to encrypt messages, but user Y does
> >>> not?
> >>> >>>>>>>>> Maintaining the
> >>> >>>>>>>>>> > client
> >>> >>>>>>>>>> > > > >> library is still much easier than
> maintaining
> >>> the
> >>> >>>>>>>>> message
> >>> >>>>>>>>>> > formats.
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> -Todd
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> Shapira <
> >>> >>>>>>>>> gwen@confluent.io
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > wrote:
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> > Based on your last sentence, consider me
> >>> >>>>> convinced
> >>> >>>>>>> :)
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > I get why headers are critical for
> Mirroring
> >>> >>> (you
> >>> >>>>>>>>> need tags to
> >>> >>>>>>>>>> > > prevent
> >>> >>>>>>>>>> > > > >> > loops and sometimes to route messages
> to the
> >>> >>>>> correct
> >>> >>>>>>>>>> > destination).
> >>> >>>>>>>>>> > > > >> > But why do you need headers to audit?
> We are
> >>> >>>>>>> auditing
> >>> >>>>>>>>> by
> >>> >>>>>>>>>> > producing
> >>> >>>>>>>>>> > > > >> > counts to a side topic (and I was under
> the
> >>> >>>>>>>>> impression you do
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > >> > same), so we never need to modify the
> >>> message.
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > Another thing - after we added headers,
> >>> >>> wouldn't
> >>> >>>>> you
> >>> >>>>>>>>> be in the
> >>> >>>>>>>>>> > > > >> > business of making sure everyone uses
> them
> >>> >>>>> properly?
> >>> >>>>>>>>> Making
> >>> >>>>>>>>>> > sure
> >>> >>>>>>>>>> > > > >> > everyone includes the right headers you
> >>> need,
> >>> >>> not
> >>> >>>>>>>>> using the
> >>> >>>>>>>>>> > header
> >>> >>>>>>>>>> > > > >> > names you intend to use, etc. I don't
> think
> >>> the
> >>> >>>>>>>>> "policing"
> >>> >>>>>>>>>> > business
> >>> >>>>>>>>>> > > > >> > will ever go away.
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> Palino
> >>> <
> >>> >>>>>>>>>> > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > wrote:
> >>> >>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very
> happy
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>>> > workaround.
> >>> >>>>>>>>>> > > Avro
> >>> >>>>>>>>>> > > > >> means
> >>> >>>>>>>>>> > > > >> > > that I have to be concerned with the
> >>> format
> >>> >>> of
> >>> >>>>> the
> >>> >>>>>>>>> messages
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > order to
> >>> >>>>>>>>>> > > > >> > run
> >>> >>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring,
> >>> etc.).
> >>> >>>>> That
> >>> >>>>>>>>> means
> >>> >>>>>>>>>> > that I
> >>> >>>>>>>>>> > > have
> >>> >>>>>>>>>> > > > to
> >>> >>>>>>>>>> > > > >> > > handle the schemas, and I have to
> enforce
> >>> >>> rules
> >>> >>>>>>>>> about good
> >>> >>>>>>>>>> > > formats.
> >>> >>>>>>>>>> > > > >> This
> >>> >>>>>>>>>> > > > >> > is
> >>> >>>>>>>>>> > > > >> > > not something I want to be in the
> business
> >>> >>> of,
> >>> >>>>>>>>> because I
> >>> >>>>>>>>>> > should be
> >>> >>>>>>>>>> > > > able
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > > run a service infrastructure without
> >>> needing
> >>> >>>>> to be
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > weeds of
> >>> >>>>>>>>>> > > > >> > dealing
> >>> >>>>>>>>>> > > > >> > > with customer data formats.
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > Trust me, a sizable portion of my
> support
> >>> >>> time
> >>> >>>>> is
> >>> >>>>>>>>> spent
> >>> >>>>>>>>>> > dealing
> >>> >>>>>>>>>> > > with
> >>> >>>>>>>>>> > > > >> > schema
> >>> >>>>>>>>>> > > > >> > > issues. I really would like to get
> away
> >>> from
> >>> >>>>> that.
> >>> >>>>>>>>> Maybe
> >>> >>>>>>>>>> > I'd have
> >>> >>>>>>>>>> > > > more
> >>> >>>>>>>>>> > > > >> > time
> >>> >>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > -Todd
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
> >>> Shapira <
> >>> >>>>>>>>>> > gwen@confluent.io>
> >>> >>>>>>>>>> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
> >>> >>>>> workarounds
> >>> >>>>>>>>> (Avro
> >>> >>>>>>>>>> > container
> >>> >>>>>>>>>> > > > >> > >> format), so I'm not too excited
> about the
> >>> >>>>> extra
> >>> >>>>>>>>> work
> >>> >>>>>>>>>> > required to
> >>> >>>>>>>>>> > > do
> >>> >>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't
> >>> mind it
> >>> >>>>> if
> >>> >>>>>>>>> you do
> >>> >>>>>>>>>> > it...
> >>> >>>>>>>>>> > > > >> > >> I think the Apache convention for
> "good
> >>> >>> idea,
> >>> >>>>> but
> >>> >>>>>>>>> not
> >>> >>>>>>>>>> > willing to
> >>> >>>>>>>>>> > > > put
> >>> >>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway,
> >>> that's
> >>> >>>>> what
> >>> >>>>>>> I
> >>> >>>>>>>>> was
> >>> >>>>>>>>>> > trying to
> >>> >>>>>>>>>> > > > >> > >> convey :)
> >>> >>>>>>>>>> > > > >> > >>
> >>> >>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
> >>> Palino
> >>> >>> <
> >>> >>>>>>>>>> > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> > Well I guess my question for you,
> >>> then, is
> >>> >>>>> what
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > holding you
> >>> >>>>>>>>>> > > > back
> >>> >>>>>>>>>> > > > >> > from
> >>> >>>>>>>>>> > > > >> > >> > full support for headers? What's
> the
> >>> bit
> >>> >>>>> that
> >>> >>>>>>>>> you're
> >>> >>>>>>>>>> > missing
> >>> >>>>>>>>>> > > that
> >>> >>>>>>>>>> > > > >> has
> >>> >>>>>>>>>> > > > >> > you
> >>> >>>>>>>>>> > > > >> > >> > under a full +1?
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> > -Todd
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> Gwen
> >>> >>>>> Shapira <
> >>> >>>>>>>>>> > > gwen@confluent.io>
> >>> >>>>>>>>>> > > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> I know why people who support
> headers
> >>> >>>>> support
> >>> >>>>>>>>> them, and
> >>> >>>>>>>>>> > I've
> >>> >>>>>>>>>> > > > seen
> >>> >>>>>>>>>> > > > >> > what
> >>> >>>>>>>>>> > > > >> > >> >> the discussion is like.
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> This is why I'm asking people who
> are
> >>> >>>>> against
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > > > (especially
> >>> >>>>>>>>>> > > > >> > >> >> committers) what will make them
> change
> >>> >>>>> their
> >>> >>>>>>>>> mind - so
> >>> >>>>>>>>>> > we can
> >>> >>>>>>>>>> > > > get
> >>> >>>>>>>>>> > > > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> part over one way or another.
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at
> >>> Radai,
> >>> >>>>> Jun
> >>> >>>>>>>>> or you
> >>> >>>>>>>>>> > (Todd)...
> >>> >>>>>>>>>> > > > I am
> >>> >>>>>>>>>> > > > >> > >> >> just looking for something
> concrete we
> >>> >>> can
> >>> >>>>> do
> >>> >>>>>>>>> to move
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > >> discussion
> >>> >>>>>>>>>> > > > >> > >> >> along to the yummy design details
> >>> (which
> >>> >>> is
> >>> >>>>>>> the
> >>> >>>>>>>>>> > argument I
> >>> >>>>>>>>>> > > > really
> >>> >>>>>>>>>> > > > >> am
> >>> >>>>>>>>>> > > > >> > >> >> looking forward to).
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM,
> Todd
> >>> >>>>> Palino <
> >>> >>>>>>>>>> > > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even
> >>> though
> >>> >>>>> I'm
> >>> >>>>>>>>> not a
> >>> >>>>>>>>>> > > > committer)...
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > I have always been a strong
> >>> supporter
> >>> >>> of
> >>> >>>>>>>>> introducing
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > concept
> >>> >>>>>>>>>> > > > >> > of an
> >>> >>>>>>>>>> > > > >> > >> >> > envelope to messages, which
> headers
> >>> >>>>>>>>> accomplishes. The
> >>> >>>>>>>>>> > > message
> >>> >>>>>>>>>> > > > key
> >>> >>>>>>>>>> > > > >> > is
> >>> >>>>>>>>>> > > > >> > >> >> > already an example of a piece of
> >>> >>> envelope
> >>> >>>>>>>>>> > information. By
> >>> >>>>>>>>>> > > > >> > providing a
> >>> >>>>>>>>>> > > > >> > >> >> means
> >>> >>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself,
> and
> >>> not
> >>> >>>>>>>>> relying on
> >>> >>>>>>>>>> > use-case
> >>> >>>>>>>>>> > > > >> > specific
> >>> >>>>>>>>>> > > > >> > >> >> > implementations, you make it
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>>> > components to
> >>> >>>>>>>>>> > > > >> > >> interoperate.
> >>> >>>>>>>>>> > > > >> > >> >> It
> >>> >>>>>>>>>> > > > >> > >> >> > simplifies development of all
> these
> >>> >>>>> things
> >>> >>>>>>>>> (message
> >>> >>>>>>>>>> > routing,
> >>> >>>>>>>>>> > > > >> > auditing,
> >>> >>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each
> one
> >>> does
> >>> >>>>> not
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>> > reinvent
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> wheel.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > It also makes it much easier
> from a
> >>> >>>>> client
> >>> >>>>>>>>> point of
> >>> >>>>>>>>>> > view if
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> headers
> >>> >>>>>>>>>> > > > >> > >> >> are
> >>> >>>>>>>>>> > > > >> > >> >> > defined as part of the protocol
> >>> and/or
> >>> >>>>>>>>> message format
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > general
> >>> >>>>>>>>>> > > > >> > >> because
> >>> >>>>>>>>>> > > > >> > >> >> > you can easily produce and
> consume
> >>> >>>>> messages
> >>> >>>>>>>>> without
> >>> >>>>>>>>>> > having
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> take
> >>> >>>>>>>>>> > > > >> > >> into
> >>> >>>>>>>>>> > > > >> > >> >> > account specific cases. For
> >>> example, I
> >>> >>>>> want
> >>> >>>>>>>>> to route
> >>> >>>>>>>>>> > > messages,
> >>> >>>>>>>>>> > > > >> but
> >>> >>>>>>>>>> > > > >> > >> >> client A
> >>> >>>>>>>>>> > > > >> > >> >> > doesn't support the way audit
> >>> >>> implemented
> >>> >>>>>>>>> headers, and
> >>> >>>>>>>>>> > > client
> >>> >>>>>>>>>> > > > B
> >>> >>>>>>>>>> > > > >> > >> doesn't
> >>> >>>>>>>>>> > > > >> > >> >> > support the way encryption or
> >>> routing
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>> > headers,
> >>> >>>>>>>>>> > > so
> >>> >>>>>>>>>> > > > now
> >>> >>>>>>>>>> > > > >> > my
> >>> >>>>>>>>>> > > > >> > >> >> > application has to create some
> >>> really
> >>> >>>>>>> fragile
> >>> >>>>>>>>> (my
> >>> >>>>>>>>>> > > autocorrect
> >>> >>>>>>>>>> > > > >> just
> >>> >>>>>>>>>> > > > >> > >> tried
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> > make that "tragic", which is
> >>> probably
> >>> >>>>>>>>> appropriate
> >>> >>>>>>>>>> > too) code
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> > strip
> >>> >>>>>>>>>> > > > >> > >> >> > everything off, rather than just
> >>> >>>>> consuming
> >>> >>>>>>> the
> >>> >>>>>>>>>> > messages,
> >>> >>>>>>>>>> > > > picking
> >>> >>>>>>>>>> > > > >> > out
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> 1
> >>> >>>>>>>>>> > > > >> > >> >> > or 2 headers it's interested
> in, and
> >>> >>>>>>>>> performing its
> >>> >>>>>>>>>> > > function.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has
> been
> >>> >>> going
> >>> >>>>> on
> >>> >>>>>>>>> for a
> >>> >>>>>>>>>> > long time,
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > it's
> >>> >>>>>>>>>> > > > >> > >> >> > always "Oh, you came up with 2
> use
> >>> >>> cases,
> >>> >>>>>>> and
> >>> >>>>>>>>> yeah,
> >>> >>>>>>>>>> > those
> >>> >>>>>>>>>> > > use
> >>> >>>>>>>>>> > > > >> cases
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> > real things that someone would
> want
> >>> to
> >>> >>>>> do.
> >>> >>>>>>>>> Here's an
> >>> >>>>>>>>>> > > alternate
> >>> >>>>>>>>>> > > > >> way
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> >> > implement them so let's not do
> >>> >>> headers."
> >>> >>>>> If
> >>> >>>>>>>>> we have a
> >>> >>>>>>>>>> > few
> >>> >>>>>>>>>> > > use
> >>> >>>>>>>>>> > > > >> cases
> >>> >>>>>>>>>> > > > >> > >> that
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> > actually came up with, you can
> be
> >>> sure
> >>> >>>>> that
> >>> >>>>>>>>> over the
> >>> >>>>>>>>>> > next
> >>> >>>>>>>>>> > > year
> >>> >>>>>>>>>> > > > >> > >> there's a
> >>> >>>>>>>>>> > > > >> > >> >> > dozen others that we didn't
> think of
> >>> >>> that
> >>> >>>>>>>>> someone
> >>> >>>>>>>>>> > would like
> >>> >>>>>>>>>> > > > to
> >>> >>>>>>>>>> > > > >> > do. I
> >>> >>>>>>>>>> > > > >> > >> >> > really think it's time to stop
> >>> >>> rehashing
> >>> >>>>>>> this
> >>> >>>>>>>>>> > discussion and
> >>> >>>>>>>>>> > > > >> > instead
> >>> >>>>>>>>>> > > > >> > >> >> focus
> >>> >>>>>>>>>> > > > >> > >> >> > on a workable standard that we
> can
> >>> >>> adopt.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > -Todd
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM,
> Todd
> >>> >>>>> Palino
> >>> >>>>>>> <
> >>> >>>>>>>>>> > > > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >> C. per message encryption
> >>> >>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach
> is
> >>> that
> >>> >>>>> this
> >>> >>>>>>>>>> > significantly
> >>> >>>>>>>>>> > > > reduce
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression,
> >>> which
> >>> >>>>>>> happens
> >>> >>>>>>>>> on a
> >>> >>>>>>>>>> > set of
> >>> >>>>>>>>>> > > > >> > serialized
> >>> >>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to
> >>> enable
> >>> >>>>> SSL
> >>> >>>>>>>>> for wire
> >>> >>>>>>>>>> > > > encryption
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> rely
> >>> >>>>>>>>>> > > > >> > >> >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS)
> >>> for at
> >>> >>>>> rest
> >>> >>>>>>>>>> > encryption.
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient.
> While
> >>> >>> this
> >>> >>>>>>> does
> >>> >>>>>>>>> cover
> >>> >>>>>>>>>> > the case
> >>> >>>>>>>>>> > > > of
> >>> >>>>>>>>>> > > > >> > >> removing
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >> drive from the system, it will
> not
> >>> >>>>> satisfy
> >>> >>>>>>>>> most
> >>> >>>>>>>>>> > compliance
> >>> >>>>>>>>>> > > > >> > >> requirements
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever
> has
> >>> >>>>> access to
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > broker
> >>> >>>>>>>>>> > > itself
> >>> >>>>>>>>>> > > > >> > still
> >>> >>>>>>>>>> > > > >> > >> has
> >>> >>>>>>>>>> > > > >> > >> >> >> access to the unencrypted
> data. For
> >>> >>>>>>>>> end-to-end
> >>> >>>>>>>>>> > encryption
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> > need to
> >>> >>>>>>>>>> > > > >> > >> >> >> encrypt at the producer,
> before it
> >>> >>>>> enters
> >>> >>>>>>> the
> >>> >>>>>>>>>> > system, and
> >>> >>>>>>>>>> > > > >> decrypt
> >>> >>>>>>>>>> > > > >> > at
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the
> >>> system.
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> -Todd
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM,
> >>> radai
> >>> >>> <
> >>> >>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> another big plus of headers
> in the
> >>> >>>>>>> protocol
> >>> >>>>>>>>> is that
> >>> >>>>>>>>>> > it
> >>> >>>>>>>>>> > > would
> >>> >>>>>>>>>> > > > >> > enable
> >>> >>>>>>>>>> > > > >> > >> >> rapid
> >>> >>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of
> core
> >>> >>>>> kafka
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>> > reduce
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> number of
> >>> >>>>>>>>>> > > > >> > >> >> >>> future wire format changes
> >>> required.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a
> KIP
> >>> >>>>>>> represents
> >>> >>>>>>>>> use
> >>> >>>>>>>>>> > cases that
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> > not
> >>> >>>>>>>>>> > > > >> > >> 100%
> >>> >>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and
> some of
> >>> >>> them
> >>> >>>>>>>>> require
> >>> >>>>>>>>>> > rather
> >>> >>>>>>>>>> > > > invasive
> >>> >>>>>>>>>> > > > >> > wire
> >>> >>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a
> good
> >>> >>> recent
> >>> >>>>>>>>> example of
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > > > >> > kip-98.
> >>> >>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is
> expected
> >>> to
> >>> >>> be
> >>> >>>>> a
> >>> >>>>>>>>> very small
> >>> >>>>>>>>>> > > > fraction of
> >>> >>>>>>>>>> > > > >> > >> total
> >>> >>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes
> are
> >>> >>>>> invasive.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> every such wire format change
> >>> >>>>> translates
> >>> >>>>>>>>> into
> >>> >>>>>>>>>> > painful and
> >>> >>>>>>>>>> > > > slow
> >>> >>>>>>>>>> > > > >> > >> >> adoption of
> >>> >>>>>>>>>> > > > >> > >> >> >>> new versions.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
> >>> >>>>> currently
> >>> >>>>>>> in
> >>> >>>>>>>>> KIPs
> >>> >>>>>>>>>> > could be
> >>> >>>>>>>>>> > > > "spun
> >>> >>>>>>>>>> > > > >> > out"
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
> >>> >>>>> transmitting
> >>> >>>>>>>>> data over
> >>> >>>>>>>>>> > > > headers.
> >>> >>>>>>>>>> > > > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> keep the core wire format
> >>> stable(r),
> >>> >>>>> core
> >>> >>>>>>>>> codebase
> >>> >>>>>>>>>> > > smaller,
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > >> avoid
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats
> sometimes
> >>> >>>>> required
> >>> >>>>>>>>> to prove
> >>> >>>>>>>>>> > a
> >>> >>>>>>>>>> > > > certain
> >>> >>>>>>>>>> > > > >> > >> feature
> >>> >>>>>>>>>> > > > >> > >> >> is
> >>> >>>>>>>>>> > > > >> > >> >> >>> useful enough for a
> wide-enough
> >>> >>>>> audience
> >>> >>>>>>> to
> >>> >>>>>>>>> warrant
> >>> >>>>>>>>>> > a wire
> >>> >>>>>>>>>> > > > >> format
> >>> >>>>>>>>>> > > > >> > >> >> change
> >>> >>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
> beyond
> >>> >>>>> "mere"
> >>> >>>>>>>>> wire format
> >>> >>>>>>>>>> > > changes
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > im
> >>> >>>>>>>>>> > > > >> > >> not
> >>> >>>>>>>>>> > > > >> > >> >> >>> saying it could have been
> >>> completely
> >>> >>>>> done
> >>> >>>>>>>>> with
> >>> >>>>>>>>>> > headers,
> >>> >>>>>>>>>> > > but
> >>> >>>>>>>>>> > > > >> > >> >> exactly-once
> >>> >>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20
> AM,
> >>> Gwen
> >>> >>>>>>>>> Shapira <
> >>> >>>>>>>>>> > > > >> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> 10:24 AM,
> >>> >>>>> radai <
> >>> >>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
> >>> >>>>> organization,
> >>> >>>>>>>>> one could
> >>> >>>>>>>>>> > > always
> >>> >>>>>>>>>> > > > use
> >>> >>>>>>>>>> > > > >> > >> other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers"
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
> >>> >>>>>>> traditionally
> >>> >>>>>>>>> done
> >>> >>>>>>>>>> > but there
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> > now
> >>> >>>>>>>>>> > > > >> > >> >> cases
> >>> >>>>>>>>>> > > > >> > >> >> >>> > (read
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is
> not
> >>> >>>>>>> acceptable.
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > makes
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > >> useful
> >>> >>>>>>>>>> > > > >> > >> >> even
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for
> cases
> >>> >>> where
> >>> >>>>>>>>>> > > > one-container-fits-all
> >>> >>>>>>>>>> > > > >> > cannot
> >>> >>>>>>>>>> > > > >> > >> >> >>> apply.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use
> >>> cases
> >>> >>>>>>> listed,
> >>> >>>>>>>>> i dont
> >>> >>>>>>>>>> > want
> >>> >>>>>>>>>> > > > this to
> >>> >>>>>>>>>> > > > >> > >> devolve
> >>> >>>>>>>>>> > > > >> > >> >> >>> to a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular
> use
> >>> >>> cases
> >>> >>>>> - i
> >>> >>>>>>>>> think its
> >>> >>>>>>>>>> > > enough
> >>> >>>>>>>>>> > > > >> that
> >>> >>>>>>>>>> > > > >> > >> some
> >>> >>>>>>>>>> > > > >> > >> >> of
> >>> >>>>>>>>>> > > > >> > >> >> >>> them
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I think a main point of
> >>> contention
> >>> >>> is
> >>> >>>>>>>>> that: We
> >>> >>>>>>>>>> > > identified
> >>> >>>>>>>>>> > > > few
> >>> >>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are
> >>> useful,
> >>> >>>>> do
> >>> >>>>>>> we
> >>> >>>>>>>>> want
> >>> >>>>>>>>>> > Kafka to
> >>> >>>>>>>>>> > > > be a
> >>> >>>>>>>>>> > > > >> > >> system
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that supports those
> use-cases?
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
> >>> >>>>> record-level
> >>> >>>>>>>>> lineage
> >>> >>>>>>>>>> > is
> >>> >>>>>>>>>> > > though
> >>> >>>>>>>>>> > > > >> > since
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > overhead could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > be significant."
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record
> >>> level
> >>> >>>>>>>>> lineage. I
> >>> >>>>>>>>>> > don't
> >>> >>>>>>>>>> > > think
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > was
> >>> >>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think
> it
> >>> is
> >>> >>>>> safe
> >>> >>>>>>> to
> >>> >>>>>>>>> assume
> >>> >>>>>>>>>> > that
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > NSA
> >>> >>>>>>>>>> > > > >> > >> needed
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also
> know
> >>> >>> that
> >>> >>>>>>>>> certain
> >>> >>>>>>>>>> > financial
> >>> >>>>>>>>>> > > > >> > institutes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
> >>> >>> records
> >>> >>>>> at
> >>> >>>>>>> a
> >>> >>>>>>>>> record
> >>> >>>>>>>>>> > level
> >>> >>>>>>>>>> > > and
> >>> >>>>>>>>>> > > > >> > there
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > federal regulations that
> >>> absolutely
> >>> >>>>>>>>> require
> >>> >>>>>>>>>> > this. They
> >>> >>>>>>>>>> > > > also
> >>> >>>>>>>>>> > > > >> > need
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
> >>> >>>>> "touches"
> >>> >>>>>>> the
> >>> >>>>>>>>>> > messages and
> >>> >>>>>>>>>> > > > >> either
> >>> >>>>>>>>>> > > > >> > >> reads
> >>> >>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't
> have
> >>> >>>>>>> possibly
> >>> >>>>>>>>>> > modified the
> >>> >>>>>>>>>> > > > >> payload
> >>> >>>>>>>>>> > > > >> > >> >> itself.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > They use record level
> >>> encryption to
> >>> >>>>> do
> >>> >>>>>>>>> that -
> >>> >>>>>>>>>> > apps can
> >>> >>>>>>>>>> > > > read
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers
> but
> >>> >>> can't
> >>> >>>>>>>>> touch the
> >>> >>>>>>>>>> > payload.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those
> are
> >>> >>> corner
> >>> >>>>>>>>> cases and
> >>> >>>>>>>>>> > not worth
> >>> >>>>>>>>>> > > > >> adding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they
> >>> should
> >>> >>>>> use a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> > > pubsub
> >>> >>>>>>>>>> > > > >> > message
> >>> >>>>>>>>>> > > > >> > >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the
> other
> >>> 1000
> >>> >>>>> that
> >>> >>>>>>>>> cater
> >>> >>>>>>>>>> > > > specifically
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > financial industry).
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a
> catch
> >>> 22:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific
> >>> use-case,
> >>> >>>>>>>>> someone can
> >>> >>>>>>>>>> > always
> >>> >>>>>>>>>> > > say
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > isn't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > interesting enough for
> Kafka.
> >>> If we
> >>> >>>>>>>>> discuss more
> >>> >>>>>>>>>> > general
> >>> >>>>>>>>>> > > > >> > trends,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we
> are not
> >>> >>> sure
> >>> >>>>>>> any
> >>> >>>>>>>>> of them
> >>> >>>>>>>>>> > really
> >>> >>>>>>>>>> > > > >> needs
> >>> >>>>>>>>>> > > > >> > >> >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just
> hand
> >>> >>>>> waving
> >>> >>>>>>>>> and not
> >>> >>>>>>>>>> > > > interesting.".
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I think discussing
> use-cases in
> >>> >>>>>>> specifics
> >>> >>>>>>>>> is super
> >>> >>>>>>>>>> > > > important
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> decide
> >>> >>>>>>>>>> > > > >> > >> >> >>> > implementation details for
> >>> headers
> >>> >>>>> (my
> >>> >>>>>>>>> use-cases
> >>> >>>>>>>>>> > lean
> >>> >>>>>>>>>> > > > toward
> >>> >>>>>>>>>> > > > >> > >> >> numerical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and
> object
> >>> >>>>> values,
> >>> >>>>>>>>> others
> >>> >>>>>>>>>> > differ),
> >>> >>>>>>>>>> > > > but I
> >>> >>>>>>>>>> > > > >> > >> think
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > need to answer the general
> "Are
> >>> we
> >>> >>>>> going
> >>> >>>>>>>>> to have
> >>> >>>>>>>>>> > > headers"
> >>> >>>>>>>>>> > > > >> > question
> >>> >>>>>>>>>> > > > >> > >> >> >>> > first.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the
> other
> >>> >>>>>>>>> committers in the
> >>> >>>>>>>>>> > > > discussion:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > What would it take to
> convince
> >>> you
> >>> >>>>> that
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > in Kafka
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> a
> >>> >>>>>>>>>> > > > >> > >> good
> >>> >>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can
> move
> >>> >>> ahead
> >>> >>>>>>> and
> >>> >>>>>>>>> try to
> >>> >>>>>>>>>> > agree
> >>> >>>>>>>>>> > > on
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> details?
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving
> the
> >>> goal
> >>> >>>>>>> posts
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > > > truly
> >>> >>>>>>>>>> > > > >> > >> >> exhausting.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly
> support
> >>> >>>>> adding
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > to Kafka
> >>> >>>>>>>>>> > > > >> > (+0.5?).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > The community can continue
> to
> >>> find
> >>> >>>>>>>>> workarounds to
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > issue
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > are some benefits to
> keeping the
> >>> >>>>> message
> >>> >>>>>>>>> format
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > > clients
> >>> >>>>>>>>>> > > > >> > >> simpler.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of
> >>> headers
> >>> >>>>> to
> >>> >>>>>>>>> many
> >>> >>>>>>>>>> > use-cases
> >>> >>>>>>>>>> > > and
> >>> >>>>>>>>>> > > > if
> >>> >>>>>>>>>> > > > >> we
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> > find a good and generally
> useful
> >>> >>> way
> >>> >>>>> to
> >>> >>>>>>>>> add it to
> >>> >>>>>>>>>> > Kafka,
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > will
> >>> >>>>>>>>>> > > > >> > >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for
> many -
> >>> >>> worthy
> >>> >>>>>>>>> goal in my
> >>> >>>>>>>>>> > eyes.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible,
> but:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> polyglot
> >>> >>>>>>>>>> > topics.
> >>> >>>>>>>>>> > > > >> > >> especially if
> >>> >>>>>>>>>> > > > >> > >> >> >>> kafka
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
> >>> >>> something
> >>> >>>>>>> else.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would
> make
> >>> it
> >>> >>>>>>> harder
> >>> >>>>>>>>> to write
> >>> >>>>>>>>>> > > > portable
> >>> >>>>>>>>>> > > > >> > >> consumer
> >>> >>>>>>>>>> > > > >> > >> >> >>> code.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would
> mess
> >>> >>> with
> >>> >>>>>>>>> locality of
> >>> >>>>>>>>>> > > > consumption
> >>> >>>>>>>>>> > > > >> > >> >> >>> guarantees.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
> >>> >>>>>>>>> lineage/metadata is
> >>> >>>>>>>>>> > > > >> > >> billing/chargeback.
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough
> to
> >>> >>> simply
> >>> >>>>>>>>> record the
> >>> >>>>>>>>>> > point
> >>> >>>>>>>>>> > > of
> >>> >>>>>>>>>> > > > >> > origin,
> >>> >>>>>>>>>> > > > >> > >> but
> >>> >>>>>>>>>> > > > >> > >> >> >>> every
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think
> mirror
> >>> >>>>> maker)
> >>> >>>>>>>>> must also
> >>> >>>>>>>>>> > add a
> >>> >>>>>>>>>> > > > >> record
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> >> form a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing
> on
> >>> top
> >>> >>> of
> >>> >>>>>>>>> kafka - i
> >>> >>>>>>>>>> > know
> >>> >>>>>>>>>> > > samza
> >>> >>>>>>>>>> > > > >> has a
> >>> >>>>>>>>>> > > > >> > >> >> metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > map
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
> >>> >>>>> addition to
> >>> >>>>>>>>> user
> >>> >>>>>>>>>> > values.
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > perfect
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at
> 6:50
> >>> PM,
> >>> >>>>> Jun
> >>> >>>>>>>>> Rao <
> >>> >>>>>>>>>> > > > jun@confluent.io
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the
> first
> >>> two
> >>> >>>>>>>>> questions, it
> >>> >>>>>>>>>> > would
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > >> > helpful
> >>> >>>>>>>>>> > > > >> > >> >> if we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong
> use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > space
> >>> >>>>>>>>>> > > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases
> >>> within an
> >>> >>>>>>>>> organization,
> >>> >>>>>>>>>> > one
> >>> >>>>>>>>>> > > > could
> >>> >>>>>>>>>> > > > >> > always
> >>> >>>>>>>>>> > > > >> > >> >> use
> >>> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers to
> >>> >>>>>>>>>> > get
> >>> >>>>>>>>>> > > > around
> >>> >>>>>>>>>> > > > >> w/o
> >>> >>>>>>>>>> > > > >> > >> >> >>> headers. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> went through the use
> cases in
> >>> >>> the
> >>> >>>>> KIP
> >>> >>>>>>>>> and in
> >>> >>>>>>>>>> > Radai's
> >>> >>>>>>>>>> > > > wiki
> >>> >>>>>>>>>> > > > >> (
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> https://cwiki.apache.org/confl
> >>> >>>>>>>>>> > uence/display/KAFKA/A+
> >>> >>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> ).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> The following are the
> ones
> >>> that
> >>> >>>>> that
> >>> >>>>>>> I
> >>> >>>>>>>>>> > understand and
> >>> >>>>>>>>>> > > > >> could
> >>> >>>>>>>>>> > > > >> > be
> >>> >>>>>>>>>> > > > >> > >> in
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case
> >>> category.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
> >>> >>>>>>> content-type
> >>> >>>>>>>>> should
> >>> >>>>>>>>>> > be set
> >>> >>>>>>>>>> > > at
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> topic
> >>> >>>>>>>>>> > > > >> > >> >> >>> level.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing
> messages
> >>> with
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> > content
> >>> >>>>>>>>>> > > > types
> >>> >>>>>>>>>> > > > >> > >> should be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
> >>> >>> useless
> >>> >>>>>>>>> without
> >>> >>>>>>>>>> > schema id,
> >>> >>>>>>>>>> > > it
> >>> >>>>>>>>>> > > > >> > seems
> >>> >>>>>>>>>> > > > >> > >> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > storing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together
> with
> >>> >>>>>>> serialized
> >>> >>>>>>>>> bytes
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > value
> >>> >>>>>>>>>> > > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> better?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this
> >>> approach is
> >>> >>>>> that
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > > > significantly
> >>> >>>>>>>>>> > > > >> > reduce
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of
> compression,
> >>> >>>>> which
> >>> >>>>>>>>> happens on
> >>> >>>>>>>>>> > a set
> >>> >>>>>>>>>> > > of
> >>> >>>>>>>>>> > > > >> > >> serialized
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative
> is
> >>> to
> >>> >>>>> enable
> >>> >>>>>>>>> SSL for
> >>> >>>>>>>>>> > wire
> >>> >>>>>>>>>> > > > >> > encryption
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> rely
> >>> >>>>>>>>>> > > > >> > >> >> >>> > on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g.
> >>> LUKS)
> >>> >>>>> for at
> >>> >>>>>>>>> rest
> >>> >>>>>>>>>> > > encryption.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for
> mirroring
> >>> >>> across
> >>> >>>>>>>>> Kafka
> >>> >>>>>>>>>> > clusters
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> This is actually
> interesting.
> >>> >>>>> Today,
> >>> >>>>>>>>> to avoid
> >>> >>>>>>>>>> > > > introducing
> >>> >>>>>>>>>> > > > >> > >> cycles
> >>> >>>>>>>>>> > > > >> > >> >> when
> >>> >>>>>>>>>> > > > >> > >> >> >>> > doing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data
> >>> centers,
> >>> >>> one
> >>> >>>>>>>>> would
> >>> >>>>>>>>>> > either have
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> set
> >>> >>>>>>>>>> > > > >> > up
> >>> >>>>>>>>>> > > > >> > >> two
> >>> >>>>>>>>>> > > > >> > >> >> >>> Kafka
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
> >>> >>>>> aggregate)
> >>> >>>>>>>>> per data
> >>> >>>>>>>>>> > center
> >>> >>>>>>>>>> > > or
> >>> >>>>>>>>>> > > > >> > rename
> >>> >>>>>>>>>> > > > >> > >> >> topics.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
> >>> headers,
> >>> >>>>> the
> >>> >>>>>>>>> producer
> >>> >>>>>>>>>> > could
> >>> >>>>>>>>>> > > tag
> >>> >>>>>>>>>> > > > >> each
> >>> >>>>>>>>>> > > > >> > >> >> message
> >>> >>>>>>>>>> > > > >> > >> >> >>> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID
> in
> >>> the
> >>> >>>>>>> header.
> >>> >>>>>>>>>> > MirrorMaker
> >>> >>>>>>>>>> > > > could
> >>> >>>>>>>>>> > > > >> > then
> >>> >>>>>>>>>> > > > >> > >> >> avoid
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a
> >>> cluster
> >>> >>> if
> >>> >>>>>>>>> they are
> >>> >>>>>>>>>> > tagged
> >>> >>>>>>>>>> > > with
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> same
> >>> >>>>>>>>>> > > > >> > >> >> >>> cluster
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> id.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative
> >>> approach
> >>> >>>>> is
> >>> >>>>>>> to
> >>> >>>>>>>>>> > introduce sth
> >>> >>>>>>>>>> > > > like
> >>> >>>>>>>>>> > > > >> > >> >> >>> hierarchical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages
> from
> >>> >>>>>>> different
> >>> >>>>>>>>>> > clusters in
> >>> >>>>>>>>>> > > > >> > different
> >>> >>>>>>>>>> > > > >> > >> >> >>> partitions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic.
> This
> >>> >>>>> approach
> >>> >>>>>>>>> avoids
> >>> >>>>>>>>>> > filtering
> >>> >>>>>>>>>> > > > out
> >>> >>>>>>>>>> > > > >> > >> unneeded
> >>> >>>>>>>>>> > > > >> > >> >> >>> data
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving
> >>> easier
> >>> >>> to
> >>> >>>>>>>>> support. It
> >>> >>>>>>>>>> > may
> >>> >>>>>>>>>> > > make
> >>> >>>>>>>>>> > > > >> > >> compaction
> >>> >>>>>>>>>> > > > >> > >> >> >>> > trickier
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> though since the same
> key may
> >>> >>>>> show up
> >>> >>>>>>>>> in
> >>> >>>>>>>>>> > different
> >>> >>>>>>>>>> > > > >> > partitions.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source
> >>> connector
> >>> >>>>> could
> >>> >>>>>>>>> store in
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > message
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
> >>> >>> record.
> >>> >>>>>>>>> Similarly,
> >>> >>>>>>>>>> > if a
> >>> >>>>>>>>>> > > > stream
> >>> >>>>>>>>>> > > > >> job
> >>> >>>>>>>>>> > > > >> > >> >> >>> transforms
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to
> >>> topic
> >>> >>> B,
> >>> >>>>> the
> >>> >>>>>>>>> library
> >>> >>>>>>>>>> > could
> >>> >>>>>>>>>> > > > >> include
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> source
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each
> of the
> >>> >>>>>>>>> transformed
> >>> >>>>>>>>>> > message in
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> header.
> >>> >>>>>>>>>> > > > >> > >> >> Not
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sure
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful
> >>> record-level
> >>> >>>>>>> lineage
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > though
> >>> >>>>>>>>>> > > since
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> overhead
> >>> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
> >>> >>>>>>>>> clientId/host/user in
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > header
> >>> >>>>>>>>>> > > > >> in
> >>> >>>>>>>>>> > > > >> > >> each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These
> metadata
> >>> are
> >>> >>>>>>>>> really at the
> >>> >>>>>>>>>> > > producer
> >>> >>>>>>>>>> > > > >> > level
> >>> >>>>>>>>>> > > > >> > >> >> though.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > So, a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to
> only
> >>> >>>>>>> include a
> >>> >>>>>>>>>> > "producerId"
> >>> >>>>>>>>>> > > > per
> >>> >>>>>>>>>> > > > >> > >> message
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > send
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producerId ->
> metadata
> >>> >>> mapping
> >>> >>>>>>>>>> > independently.
> >>> >>>>>>>>>> > > > KIP-98
> >>> >>>>>>>>>> > > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> actually
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such
> a
> >>> >>>>> producerId
> >>> >>>>>>>>> natively
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > >> > message.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure
> that
> >>> I
> >>> >>> am
> >>> >>>>>>> fully
> >>> >>>>>>>>>> > convinced of
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > strong
> >>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
> >>> >>> Perhaps
> >>> >>>>> we
> >>> >>>>>>>>> could
> >>> >>>>>>>>>> > discuss a
> >>> >>>>>>>>>> > > > bit
> >>> >>>>>>>>>> > > > >> > more
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > one
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing
> use
> >>> >>>>> cases.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal
> question
> >>> is
> >>> >>>>>>>>> whether header
> >>> >>>>>>>>>> > should
> >>> >>>>>>>>>> > > > be
> >>> >>>>>>>>>> > > > >> > >> exposed
> >>> >>>>>>>>>> > > > >> > >> >> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > stream
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such
> Kafka
> >>> >>>>> stream,
> >>> >>>>>>>>> Samza,
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > Spark
> >>> >>>>>>>>>> > > > >> > >> streaming.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems
> just
> >>> >>> deal
> >>> >>>>>>> with
> >>> >>>>>>>>>> > key/value
> >>> >>>>>>>>>> > > > pairs.
> >>> >>>>>>>>>> > > > >> > >> Should we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > expose a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there
> too
> >>> or
> >>> >>>>>>>>> somehow map
> >>> >>>>>>>>>> > header to
> >>> >>>>>>>>>> > > > key
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> >> value?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Jun
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at
> 3:35
> >>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>> > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a
> >>> period
> >>> >>>>> of a
> >>> >>>>>>>>> week,
> >>> >>>>>>>>>> > that there
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> no
> >>> >>>>>>>>>> > > > >> > >> >> concerns
> >>> >>>>>>>>>> > > > >> > >> >> >>> now
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2
> and
> >>> now
> >>> >>> we
> >>> >>>>>>> have
> >>> >>>>>>>>>> > agreement that
> >>> >>>>>>>>>> > > > >> > headers
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> >>> useful
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
> such if
> >>> >>> put
> >>> >>>>> to
> >>> >>>>>>> a
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>> > vote, this
> >>> >>>>>>>>>> > > > >> > wouldn't
> >>> >>>>>>>>>> > > > >> > >> be
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > reason
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > @
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of
> >>> getting
> >>> >>>>> this
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>> > moving past
> >>> >>>>>>>>>> > > > >> this,
> >>> >>>>>>>>>> > > > >> > we
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> state
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes
> space
> >>> >>> that
> >>> >>>>>>> can
> >>> >>>>>>>>> will be
> >>> >>>>>>>>>> > > > naturally
> >>> >>>>>>>>>> > > > >> > >> >> interpreted
> >>> >>>>>>>>>> > > > >> > >> >> >>> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing
> is
> >>> later
> >>> >>>>>>>>> wanted you can
> >>> >>>>>>>>>> > > easily
> >>> >>>>>>>>>> > > > >> split
> >>> >>>>>>>>>> > > > >> > >> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> into
> >>> >>>>>>>>>> > > > >> > >> >> >>> > two
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the
> >>> wire
> >>> >>>>>>> protocol
> >>> >>>>>>>>>> > > implementation
> >>> >>>>>>>>>> > > > >> this
> >>> >>>>>>>>>> > > > >> > >> makes
> >>> >>>>>>>>>> > > > >> > >> >> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don't
> >>> believe. Is
> >>> >>>>> this
> >>> >>>>>>>>>> > reasonable to
> >>> >>>>>>>>>> > > > all?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
> >>> therefor
> >>> >>>>> happy
> >>> >>>>>>>>> we keep
> >>> >>>>>>>>>> > with 32
> >>> >>>>>>>>>> > > > >> bits.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
> >>> >>>>>>>>>> > ignacio.solis@gmail.com on
> >>> >>>>>>>>>> > > > >> behalf
> >>> >>>>>>>>>> > > > >> > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> Ignacio
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
> >>> >>> ignacio.solis@gmail.com
> >>> >>>>> on
> >>> >>>>>>>>> behalf of
> >>> >>>>>>>>>> > > > >> > isolis@igso.net
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
> value
> >>> as
> >>> >>>>> byte[]
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String -
> Reluctant
> >>> >>> maybe
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the
> header
> >>> >>>>> system
> >>> >>>>>>>>> should
> >>> >>>>>>>>>> > take a
> >>> >>>>>>>>>> > > > single
> >>> >>>>>>>>>> > > > >> > >> int. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you
> >>> want
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>> > this as
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> 16bit
> >>> >>>>>>>>>> > > > >> > >> >> numbers
> >>> >>>>>>>>>> > > > >> > >> >> >>> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > above go right
> ahead.
> >>> If
> >>> >>>>>>>>> somebody wants
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > argue
> >>> >>>>>>>>>> > > > >> for
> >>> >>>>>>>>>> > > > >> > 16
> >>> >>>>>>>>>> > > > >> > >> >> bits
> >>> >>>>>>>>>> > > > >> > >> >> >>> or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 64
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I
> >>> would
> >>> >>>>>>> listen.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key
> space
> >>> >>> into
> >>> >>>>>>>>> sub_key_1 and
> >>> >>>>>>>>>> > > > sub_key_2
> >>> >>>>>>>>>> > > > >> > >> makes no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sense to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > me at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we
> >>> going
> >>> >>> to
> >>> >>>>>>>>> start
> >>> >>>>>>>>>> > providing
> >>> >>>>>>>>>> > > > APIs to
> >>> >>>>>>>>>> > > > >> > get
> >>> >>>>>>>>>> > > > >> > >> all
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all
> the
> >>> >>>>>>>>> sub_key_2s? If
> >>> >>>>>>>>>> > there is
> >>> >>>>>>>>>> > > > no
> >>> >>>>>>>>>> > > > >> > >> >> >>> distinguishing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > functions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to
> >>> each
> >>> >>> one
> >>> >>>>>>>>> then they
> >>> >>>>>>>>>> > should
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > a
> >>> >>>>>>>>>> > > > >> > single
> >>> >>>>>>>>>> > > > >> > >> >> >>> value.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > At
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're
> doing
> >>> is
> >>> >>>>>>>>> equality.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer
> >>> wants
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>> > this as
> >>> >>>>>>>>>> > > 2,
> >>> >>>>>>>>>> > > > 3
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> more
> >>> >>>>>>>>>> > > > >> > >> >> >>> values
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > different
> question. I
> >>> >>>>>>>>> personally think
> >>> >>>>>>>>>> > it's
> >>> >>>>>>>>>> > > all
> >>> >>>>>>>>>> > > > one
> >>> >>>>>>>>>> > > > >> > >> >> keyspace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned
> using
> >>> >>> some
> >>> >>>>>>>>> structure,
> >>> >>>>>>>>>> > but if
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> > want to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sub-assign
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > parts
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's
> fine.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
> >>> >>> applies
> >>> >>>>> to
> >>> >>>>>>>>> strings.
> >>> >>>>>>>>>> > If
> >>> >>>>>>>>>> > > > somebody
> >>> >>>>>>>>>> > > > >> > >> argued
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would we be
> arguing to
> >>> >>>>> divide
> >>> >>>>>>> the
> >>> >>>>>>>>>> > strings with
> >>> >>>>>>>>>> > > > dots
> >>> >>>>>>>>>> > > > >> > ('.')
> >>> >>>>>>>>>> > > > >> > >> >> as a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them
> to
> >>> give
> >>> >>>>> us
> >>> >>>>>>> the
> >>> >>>>>>>>>> > different
> >>> >>>>>>>>>> > > name
> >>> >>>>>>>>>> > > > >> > segments
> >>> >>>>>>>>>> > > > >> > >> >> >>> > separately?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be
> performing
> >>> any
> >>> >>>>>>>>> actions on
> >>> >>>>>>>>>> > this key
> >>> >>>>>>>>>> > > > other
> >>> >>>>>>>>>> > > > >> > than
> >>> >>>>>>>>>> > > > >> > >> >> >>> > matching?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
> 2016 at
> >>> >>> 9:30
> >>> >>>>>>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>> > > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> >>> concerns
> >>> >>> on
> >>> >>>>> 1
> >>> >>>>>>>>> and 2
> >>> >>>>>>>>>> > still?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this
> moving
> >>> >>> along a
> >>> >>>>>>> bit
> >>> >>>>>>>>> more
> >>> >>>>>>>>>> > I'd also
> >>> >>>>>>>>>> > > > like
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> ask
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> get
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last
> >>> points:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe
> we're
> >>> all
> >>> >>>>>>> roughly
> >>> >>>>>>>>> happy
> >>> >>>>>>>>>> > with the
> >>> >>>>>>>>>> > > > >> header
> >>> >>>>>>>>>> > > > >> > >> value
> >>> >>>>>>>>>> > > > >> > >> >> >>> > being a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> >>> consensus
> >>> >>> has
> >>> >>>>>>>>> been for an
> >>> >>>>>>>>>> > > > namespace
> >>> >>>>>>>>>> > > > >> > based
> >>> >>>>>>>>>> > > > >> > >> int
> >>> >>>>>>>>>> > > > >> > >> >> >>> > approach
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the
> >>> key.
> >>> >>> Any
> >>> >>>>>>>>> objections
> >>> >>>>>>>>>> > if this
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> > what
> >>> >>>>>>>>>> > > > >> > >> we
> >>> >>>>>>>>>> > > > >> > >> >> go
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
> >>> >>>>> assumption
> >>> >>>>>>> in
> >>> >>>>>>>>> (4) is
> >>> >>>>>>>>>> > > correct,
> >>> >>>>>>>>>> > > > >> > >> {int,int}
> >>> >>>>>>>>>> > > > >> > >> >> >>> keys.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both
> int's be
> >>> >>>>> int16 or
> >>> >>>>>>>>> int32?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them
> being
> >>> >>> int16(2
> >>> >>>>>>>>> bytes) as
> >>> >>>>>>>>>> > combined
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> > space
> >>> >>>>>>>>>> > > > >> > >> of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > per
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > original and
> gives
> >>> >>> plenty
> >>> >>>>> of
> >>> >>>>>>>>>> > combinations for
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> foreseeable,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead
> small.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any
> >>> benefit in
> >>> >>>>>>>>> another kip
> >>> >>>>>>>>>> > call to
> >>> >>>>>>>>>> > > > >> discuss
> >>> >>>>>>>>>> > > > >> > >> >> these at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > all?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>> ______________________________
> >>> >>>>>>>>>> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
> >>> >>>>>>>>> k.burstev@yandex.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> >>> November
> >>> >>> 18,
> >>> >>>>>>> 2016
> >>> >>>>>>>>>> > 7:07:07 AM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
> >>> >>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> >>> [DISCUSS]
> >>> >>>>> KIP-82
> >>> >>>>>>>>> - Add
> >>> >>>>>>>>>> > Record
> >>> >>>>>>>>>> > > > Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is
> worth
> >>> >>> also
> >>> >>>>> i
> >>> >>>>>>>>> agree. As
> >>> >>>>>>>>>> > a user:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
> Headers are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers
> >>> should
> >>> >>>>> be a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>> > > option
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
> 21:15,
> >>> >>>>> "Ignacio
> >>> >>>>>>>>> Solis" <
> >>> >>>>>>>>>> > > > >> isolis@igso.net
> >>> >>>>>>>>>> > > > >> > >:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
> Headers
> >>> are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
> Headers
> >>> >>> should
> >>> >>>>> be
> >>> >>>>>>> a
> >>> >>>>>>>>> top
> >>> >>>>>>>>>> > level
> >>> >>>>>>>>>> > > option
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14,
> >>> 2016
> >>> >>> at
> >>> >>>>>>> 9:16
> >>> >>>>>>>>> AM,
> >>> >>>>>>>>>> > Michael
> >>> >>>>>>>>>> > > > Pearce
> >>> >>>>>>>>>> > > > >> <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
> >>> >>>>> details/examples
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > original
> >>> >>>>>>>>>> > > > proposal
> >>> >>>>>>>>>> > > > >> > for
> >>> >>>>>>>>>> > > > >> > >> key
> >>> >>>>>>>>>> > > > >> > >> >> >>> > spacing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> ,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > not
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new
> mentioned as
> >>> per
> >>> >>>>>>>>> discussion
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > >> idea.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need
> to
> >>> >>> update
> >>> >>>>>>> the
> >>> >>>>>>>>> kip,
> >>> >>>>>>>>>> > when we
> >>> >>>>>>>>>> > > get
> >>> >>>>>>>>>> > > > >> > >> agreement
> >>> >>>>>>>>>> > > > >> > >> >> >>> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > is a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > better
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach
> (which
> >>> >>> seems
> >>> >>>>> to
> >>> >>>>>>>>> be the
> >>> >>>>>>>>>> > case if I
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> > >> >> understood
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > general
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in
> the
> >>> >>>>>>>>> conversation)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the
> variable
> >>> >>> ints,
> >>> >>>>> at
> >>> >>>>>>>>> very
> >>> >>>>>>>>>> > early stage
> >>> >>>>>>>>>> > > > we
> >>> >>>>>>>>>> > > > >> did
> >>> >>>>>>>>>> > > > >> > >> think
> >>> >>>>>>>>>> > > > >> > >> >> >>> about
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added
> >>> complexity
> >>> >>>>> for
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > saving isn't
> >>> >>>>>>>>>> > > > >> worth
> >>> >>>>>>>>>> > > > >> > it.
> >>> >>>>>>>>>> > > > >> > >> >> I'd
> >>> >>>>>>>>>> > > > >> > >> >> >>> > rather
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> go
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to
> reduce
> >>> >>>>>>>>> overheads and
> >>> >>>>>>>>>> > size
> >>> >>>>>>>>>> > > int16
> >>> >>>>>>>>>> > > > >> > (2bytes)
> >>> >>>>>>>>>> > > > >> > >> >> keys
> >>> >>>>>>>>>> > > > >> > >> >> >>> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note
> of no
> >>> >>>>>>> headers,
> >>> >>>>>>>>> there
> >>> >>>>>>>>>> > is as
> >>> >>>>>>>>>> > > per
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > kip
> >>> >>>>>>>>>> > > > >> > >> as
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > use an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to
> denote if
> >>> >>>>> headers
> >>> >>>>>>>>> are
> >>> >>>>>>>>>> > present or
> >>> >>>>>>>>>> > > > not as
> >>> >>>>>>>>>> > > > >> > such
> >>> >>>>>>>>>> > > > >> > >> >> >>> > provides a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > zero
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead
> >>> currently
> >>> >>> if
> >>> >>>>>>>>> headers are
> >>> >>>>>>>>>> > not
> >>> >>>>>>>>>> > > used.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as
> radai
> >>> >>>>> mentions
> >>> >>>>>>>>> would be
> >>> >>>>>>>>>> > good
> >>> >>>>>>>>>> > > > first
> >>> >>>>>>>>>> > > > >> > if we
> >>> >>>>>>>>>> > > > >> > >> >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> get
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have
> >>> general
> >>> >>>>>>>>> consensus that
> >>> >>>>>>>>>> > (1)
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we
> want
> >>> it
> >>> >>> as
> >>> >>>>> a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>> > entity.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to
> state the
> >>> >>>>>>> obvious i
> >>> >>>>>>>>>> > believe (1)
> >>> >>>>>>>>>> > > > >> headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a
> top
> >>> level
> >>> >>>>>>>>> entity.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>> ______________________________
> >>> >>>>>>>>>> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
> >>> Hoover <
> >>> >>>>>>>>>> > > roger.hoover@gmail.com
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent:
> Wednesday,
> >>> >>>>> November
> >>> >>>>>>>>> 9, 2016
> >>> >>>>>>>>>> > 9:10:47
> >>> >>>>>>>>>> > > > PM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
> >>> >>>>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
> >>> >>> [DISCUSS]
> >>> >>>>>>>>> KIP-82 - Add
> >>> >>>>>>>>>> > > Record
> >>> >>>>>>>>>> > > > >> > Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
> going a
> >>> >>>>> little
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > weeds but
> >>> >>>>>>>>>> > > > >> thanks
> >>> >>>>>>>>>> > > > >> > >> for
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> replies
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a
> >>> prefix
> >>> >>>>> and
> >>> >>>>>>>>> {int,
> >>> >>>>>>>>>> > int} can
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> same.
> >>> >>>>>>>>>> > > > >> > >> >> It
> >>> >>>>>>>>>> > > > >> > >> >> >>> > doesn't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > look
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > like
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what
> the
> >>> KIP
> >>> >>> is
> >>> >>>>>>>>> saying the
> >>> >>>>>>>>>> > "Open"
> >>> >>>>>>>>>> > > > >> > section.
> >>> >>>>>>>>>> > > > >> > >> The
> >>> >>>>>>>>>> > > > >> > >> >> >>> > example
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shows
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New
> Relic and
> >>> >>>>> 210002
> >>> >>>>>>>>> for App
> >>> >>>>>>>>>> > Dynamics
> >>> >>>>>>>>>> > > > >> > implying
> >>> >>>>>>>>>> > > > >> > >> >> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > New
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization
> will
> >>> >>> have
> >>> >>>>>>>>> only a
> >>> >>>>>>>>>> > single
> >>> >>>>>>>>>> > > > header id
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> work
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with. Or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The
> >>> main
> >>> >>>>> point
> >>> >>>>>>>>> of a
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > or
> >>> >>>>>>>>>> > > > >> > prefix
> >>> >>>>>>>>>> > > > >> > >> is
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > reduce
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of
> >>> config
> >>> >>>>>>> mapping
> >>> >>>>>>>>> or
> >>> >>>>>>>>>> > > registration
> >>> >>>>>>>>>> > > > >> > >> depending
> >>> >>>>>>>>>> > > > >> > >> >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> how
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> namespaces/prefixes
> >>> >>>>> are
> >>> >>>>>>>>> managed.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to
> >>> hear
> >>> >>>>> more
> >>> >>>>>>>>> feedback
> >>> >>>>>>>>>> > on the
> >>> >>>>>>>>>> > > > >> > >> higher-level
> >>> >>>>>>>>>> > > > >> > >> >> >>> > questions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov
> 9,
> >>> 2016
> >>> >>> at
> >>> >>>>>>>>> 11:38 AM,
> >>> >>>>>>>>>> > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think
> this
> >>> >>>>> discussion
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > getting a
> >>> >>>>>>>>>> > > bit
> >>> >>>>>>>>>> > > > >> into
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> weeds on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > technical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> implementation
> >>> >>>>> details.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to
> >>> step
> >>> >>>>> back a
> >>> >>>>>>>>> minute
> >>> >>>>>>>>>> > and try
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > >> establish
> >>> >>>>>>>>>> > > > >> > >> >> >>> > where we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > are in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger
> picture:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
> >>> >>> nacho's
> >>> >>>>>>> last
> >>> >>>>>>>>>> > paragraph)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we
> all
> >>> in
> >>> >>>>>>>>> agreement that
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > useful
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to
> >>> have?
> >>> >>>>> this
> >>> >>>>>>>>> was
> >>> >>>>>>>>>> > contested
> >>> >>>>>>>>>> > > > early
> >>> >>>>>>>>>> > > > >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we
> all
> >>> in
> >>> >>>>>>>>> agreement on
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > as
> >>> >>>>>>>>>> > > > top
> >>> >>>>>>>>>> > > > >> > >> level
> >>> >>>>>>>>>> > > > >> > >> >> >>> entity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > vs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> squirreled-away in
> >>> >>>>> V?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there
> are
> >>> still
> >>> >>>>>>>>> concerns
> >>> >>>>>>>>>> > around
> >>> >>>>>>>>>> > > these
> >>> >>>>>>>>>> > > > #2
> >>> >>>>>>>>>> > > > >> > >> points
> >>> >>>>>>>>>> > > > >> > >> >> >>> (#jay?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now
> back
> >>> to
> >>> >>> our
> >>> >>>>>>>>> normal
> >>> >>>>>>>>>> > programming
> >>> >>>>>>>>>> > > > ...)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are
> >>> nice.
> >>> >>>>>>> having
> >>> >>>>>>>>> said
> >>> >>>>>>>>>> > that, its
> >>> >>>>>>>>>> > > > >> adding
> >>> >>>>>>>>>> > > > >> > >> >> >>> complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (see
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>> https://github.com/addthis/
> >>> >>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>> main/java/com/clearspring/
> >>> >>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
> google
> >>> >>>>> result)
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>> > require
> >>> >>>>>>>>>> > > > >> anyone
> >>> >>>>>>>>>> > > > >> > >> >> writing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clients
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go?
> >>> Bash?
> >>> >>>>> ;-) )
> >>> >>>>>>>>> to
> >>> >>>>>>>>>> > > get/implement
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> same,
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain
> >>> (int
> >>> >>> vs
> >>> >>>>>>>>> string is
> >>> >>>>>>>>>> > order of
> >>> >>>>>>>>>> > > > >> > magnitude,
> >>> >>>>>>>>>> > > > >> > >> >> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > isnt).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int
> >>> namespacing vs
> >>> >>>>>>> {int,
> >>> >>>>>>>>> int}
> >>> >>>>>>>>>> > > namespacing
> >>> >>>>>>>>>> > > > >> are
> >>> >>>>>>>>>> > > > >> > >> >> basically
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > same
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
> >>> >>>>> namespacing
> >>> >>>>>>>>> an int64
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > > giving
> >>> >>>>>>>>>> > > > >> > people
> >>> >>>>>>>>>> > > > >> > >> >> while
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 2^32
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the
> part
> >>> i
> >>> >>>>> like
> >>> >>>>>>>>> about this
> >>> >>>>>>>>>> > is
> >>> >>>>>>>>>> > > > letting
> >>> >>>>>>>>>> > > > >> > >> people
> >>> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> large
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers
> with
> >>> one
> >>> >>>>>>>>> registration so
> >>> >>>>>>>>>> > they
> >>> >>>>>>>>>> > > > dont
> >>> >>>>>>>>>> > > > >> > have
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> come
> >>> >>>>>>>>>> > > > >> > >> >> >>> > back
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > every
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
> >>> >>> plugin/header
> >>> >>>>>>>>> they want to
> >>> >>>>>>>>>> > > > "reserve".
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
> Nov 9,
> >>> >>> 2016
> >>> >>>>> at
> >>> >>>>>>>>> 11:01 AM,
> >>> >>>>>>>>>> > Roger
> >>> >>>>>>>>>> > > > >> Hoover
> >>> >>>>>>>>>> > > > >> > <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> roger.hoover@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since
> some of
> >>> >>> the
> >>> >>>>>>>>> debate has
> >>> >>>>>>>>>> > been
> >>> >>>>>>>>>> > > about
> >>> >>>>>>>>>> > > > >> > >> overhead +
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> wondering if
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>>> > considered a
> >>> >>>>>>>>>> > > > varint
> >>> >>>>>>>>>> > > > >> > >> encoding
> >>> >>>>>>>>>> > > > >> > >> >> (
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>> https://developers.google.com/
> >>> >>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the
> header
> >>> >>> length
> >>> >>>>>>>>> field (int32
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > >> > proposal)
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
> >>> >>> headers,
> >>> >>>>>>> the
> >>> >>>>>>>>>> > overhead would
> >>> >>>>>>>>>> > > > be a
> >>> >>>>>>>>>> > > > >> > >> single
> >>> >>>>>>>>>> > > > >> > >> >> >>> byte
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128
> >>> would
> >>> >>>>> also
> >>> >>>>>>>>> need only a
> >>> >>>>>>>>>> > > single
> >>> >>>>>>>>>> > > > >> byte?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed,
> Nov
> >>> 9,
> >>> >>>>> 2016
> >>> >>>>>>> at
> >>> >>>>>>>>> 6:43 AM,
> >>> >>>>>>>>>> > > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> @magnus -
> >>> and
> >>> >>>>> very
> >>> >>>>>>>>> dangerous
> >>> >>>>>>>>>> > (youre
> >>> >>>>>>>>>> > > > >> > >> essentially
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> arbitrary
> >>> code
> >>> >>>>> off
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > internet on
> >>> >>>>>>>>>> > > > your
> >>> >>>>>>>>>> > > > >> > >> servers
> >>> >>>>>>>>>> > > > >> > >> >> ...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > bad
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > idea
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > without
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> sandbox,
> >>> even
> >>> >>>>> with)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for
> it
> >>> >>> being
> >>> >>>>> a
> >>> >>>>>>>>> purely
> >>> >>>>>>>>>> > > > administrative
> >>> >>>>>>>>>> > > > >> > task
> >>> >>>>>>>>>> > > > >> > >> - i
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish
> it
> >>> >>> would,
> >>> >>>>>>>>> really,
> >>> >>>>>>>>>> > because
> >>> >>>>>>>>>> > > > then my
> >>> >>>>>>>>>> > > > >> > >> earlier
> >>> >>>>>>>>>> > > > >> > >> >> >>> > point on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >>> remapping
> >>> >>>>>>>>> process would
> >>> >>>>>>>>>> > be
> >>> >>>>>>>>>> > > > invalid,
> >>> >>>>>>>>>> > > > >> > but
> >>> >>>>>>>>>> > > > >> > >> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > linkedin,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the
> team
> >>> im
> >>> >>> in)
> >>> >>>>>>> run
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>> > as a
> >>> >>>>>>>>>> > > > service.
> >>> >>>>>>>>>> > > > >> > we
> >>> >>>>>>>>>> > > > >> > >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > really
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > know
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> (developing
> >>> >>>>>>>>> applications
> >>> >>>>>>>>>> > that use
> >>> >>>>>>>>>> > > > kafka)
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> up
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > any
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > given
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
> >>> >>> possible
> >>> >>>>>>>>> (given the
> >>> >>>>>>>>>> > > > existance of
> >>> >>>>>>>>>> > > > >> > >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> and a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> ecosystem)
> >>> for
> >>> >>>>> some
> >>> >>>>>>>>>> > application to
> >>> >>>>>>>>>> > > > >> "equip"
> >>> >>>>>>>>>> > > > >> > >> their
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> producers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with
> the
> >>> >>>>> required
> >>> >>>>>>>>> plugin
> >>> >>>>>>>>>> > without us
> >>> >>>>>>>>>> > > > >> > knowing.
> >>> >>>>>>>>>> > > > >> > >> i
> >>> >>>>>>>>>> > > > >> > >> >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > mean
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i
> just
> >>> >>>>> want to
> >>> >>>>>>>>> make the
> >>> >>>>>>>>>> > point
> >>> >>>>>>>>>> > > > that
> >>> >>>>>>>>>> > > > >> > its
> >>> >>>>>>>>>> > > > >> > >> not
> >>> >>>>>>>>>> > > > >> > >> >> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > simple
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
> >>> across a
> >>> >>>>>>>>> large-enough
> >>> >>>>>>>>>> > > > organization.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed,
> >>> Nov 9,
> >>> >>>>> 2016
> >>> >>>>>>>>> at 6:17
> >>> >>>>>>>>>> > AM,
> >>> >>>>>>>>>> > > > Magnus
> >>> >>>>>>>>>> > > > >> > >> Edenhill
> >>> >>>>>>>>>> > > > >> > >> >> <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> magnus@edenhill.se>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
> think
> >>> >>> there
> >>> >>>>> is
> >>> >>>>>>> a
> >>> >>>>>>>>> piece
> >>> >>>>>>>>>> > missing
> >>> >>>>>>>>>> > > in
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> >> Strings
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > where
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>> pro-Stringers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> reason
> >>> that
> >>> >>> by
> >>> >>>>>>>>> providing
> >>> >>>>>>>>>> > unique
> >>> >>>>>>>>>> > > > string
> >>> >>>>>>>>>> > > > >> > >> >> >>> identifiers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> everything
> >>> >>>>> will
> >>> >>>>>>>>> just
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> magically
> >>> >>> work
> >>> >>>>>>> for
> >>> >>>>>>>>> all
> >>> >>>>>>>>>> > parts of
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> > stream
> >>> >>>>>>>>>> > > > >> > >> >> >>> pipeline.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But
> the
> >>> >>>>> strings
> >>> >>>>>>>>> dont mean
> >>> >>>>>>>>>> > > anything
> >>> >>>>>>>>>> > > > by
> >>> >>>>>>>>>> > > > >> > >> >> themselves,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > while we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> probably
> >>> >>>>> envision
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> auto
> >>> >>>>> plugin
> >>> >>>>>>>>> loader
> >>> >>>>>>>>>> > that
> >>> >>>>>>>>>> > > > >> downloads,
> >>> >>>>>>>>>> > > > >> > >> >> compiles,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > links
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> on-demand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as
> soon
> >>> as
> >>> >>>>>>> they're
> >>> >>>>>>>>> seen by
> >>> >>>>>>>>>> > a
> >>> >>>>>>>>>> > > > >> consumer, I
> >>> >>>>>>>>>> > > > >> > >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> really
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> see
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> something
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
> >>> dynamic
> >>> >>>>> (and
> >>> >>>>>>>>> fragile) in
> >>> >>>>>>>>>> > > > practice.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In
> the
> >>> real
> >>> >>>>> world
> >>> >>>>>>>>> an
> >>> >>>>>>>>>> > application
> >>> >>>>>>>>>> > > > will
> >>> >>>>>>>>>> > > > >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> configured
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a set
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to
> either
> >>> >>> add
> >>> >>>>>>>>> (producer)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or
> read
> >>> >>>>>>> (consumer)
> >>> >>>>>>>>> headers.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This
> is
> >>> an
> >>> >>>>>>>>> administrative
> >>> >>>>>>>>>> > task
> >>> >>>>>>>>>> > > > based
> >>> >>>>>>>>>> > > > >> on
> >>> >>>>>>>>>> > > > >> > >> what
> >>> >>>>>>>>>> > > > >> > >> >> >>> > features a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > client
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>> needs/provides
> >>> >>>>>>> and
> >>> >>>>>>>>> results
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> >>> sort of
> >>> >>>>>>>>> configuration
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > > enable
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> configure
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > desired
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since
> >>> this
> >>> >>>>> needs
> >>> >>>>>>>>> to be kept
> >>> >>>>>>>>>> > > > somewhat
> >>> >>>>>>>>>> > > > >> in
> >>> >>>>>>>>>> > > > >> > >> sync
> >>> >>>>>>>>>> > > > >> > >> >> >>> across
> >>> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no
> >>> point
> >>> >>> in
> >>> >>>>>>>>> having
> >>> >>>>>>>>>> > producers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
> >>> headers
> >>> >>> no
> >>> >>>>>>>>> consumers
> >>> >>>>>>>>>> > will
> >>> >>>>>>>>>> > > read,
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> vice
> >>> >>>>>>>>>> > > > >> > >> >> >>> versa),
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > added
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
> >>> assigning
> >>> >>>>> an
> >>> >>>>>>> id
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for
> each
> >>> >>>>> plugin
> >>> >>>>>>> as
> >>> >>>>>>>>> it is
> >>> >>>>>>>>>> > being
> >>> >>>>>>>>>> > > > >> > configured
> >>> >>>>>>>>>> > > > >> > >> >> should
> >>> >>>>>>>>>> > > > >> > >> >> >>> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> /Magnus
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> 2016-11-09
> >>> >>>>> 13:06
> >>> >>>>>>>>> GMT+01:00
> >>> >>>>>>>>>> > > Michael
> >>> >>>>>>>>>> > > > >> > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> Michael.Pearce@ig.com>:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Just
> >>> >>>>>>>>> following/catching
> >>> >>>>>>>>>> > up on
> >>> >>>>>>>>>> > > > what
> >>> >>>>>>>>>> > > > >> > seems
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > active
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> @Radai
> >>> >>>>> sorry if
> >>> >>>>>>>>> it may
> >>> >>>>>>>>>> > seem
> >>> >>>>>>>>>> > > > obvious
> >>> >>>>>>>>>> > > > >> > but
> >>> >>>>>>>>>> > > > >> > >> what
> >>> >>>>>>>>>> > > > >> > >> >> >>> does
> >>> >>>>>>>>>> > > > >> > >> >> >>> > MD
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > stand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My
> >>> take on
> >>> >>>>>>>>> String vs Int:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> will
> >>> >>> state
> >>> >>>>>>>>> first I am
> >>> >>>>>>>>>> > pro Int
> >>> >>>>>>>>>> > > > (16
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> 32).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> do
> >>> >>> though
> >>> >>>>>>>>> playing
> >>> >>>>>>>>>> > devils
> >>> >>>>>>>>>> > > > advocate
> >>> >>>>>>>>>> > > > >> > see a
> >>> >>>>>>>>>> > > > >> > >> >> big
> >>> >>>>>>>>>> > > > >> > >> >> >>> plus
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> String
> >>> >>> keys,
> >>> >>>>>>>>> this is
> >>> >>>>>>>>>> > around
> >>> >>>>>>>>>> > > > >> > integrating
> >>> >>>>>>>>>> > > > >> > >> >> into an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > existing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As
> many
> >>> >>>>> other
> >>> >>>>>>>>> systems use
> >>> >>>>>>>>>> > > String
> >>> >>>>>>>>>> > > > >> based
> >>> >>>>>>>>>> > > > >> > >> >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>> these to
> >>> >>>>>>>>>> > be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > into.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How
> >>> with
> >>> >>> Int
> >>> >>>>>>>>> based
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > could
> >>> >>>>>>>>>> > > > we
> >>> >>>>>>>>>> > > > >> > >> provide
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>> integration
> >>> >>>>>>>>> simple /
> >>> >>>>>>>>>> > easy with
> >>> >>>>>>>>>> > > > >> > transition
> >>> >>>>>>>>>> > > > >> > >> >> flows
> >>> >>>>>>>>>> > > > >> > >> >> >>> > over
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> tough
> >>> >>> luck
> >>> >>>>>>>>> buddy
> >>> >>>>>>>>>> > you're on
> >>> >>>>>>>>>> > > your
> >>> >>>>>>>>>> > > > >> own
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> >>> simply
> >>> >>>>> hash
> >>> >>>>>>>>> the string
> >>> >>>>>>>>>> > into
> >>> >>>>>>>>>> > > int
> >>> >>>>>>>>>> > > > >> code
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> hope
> >>> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> convert
> >>> >>> back
> >>> >>>>>>>>> though?)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> http2
> >>> >>>>> style
> >>> >>>>>>> as
> >>> >>>>>>>>>> > mentioned by
> >>> >>>>>>>>>> > > > nacho.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> cheers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > ______________________________
> >>> >>>>>>>>>> > > > >> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> From:
> >>> >>> radai
> >>> >>>>> <
> >>> >>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Sent:
> >>> >>>>>>> Wednesday,
> >>> >>>>>>>>>> > November 9,
> >>> >>>>>>>>>> > > 2016
> >>> >>>>>>>>>> > > > >> > 8:12 AM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
> >>> >>>>>>>>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> Subject:
> >>> >>> Re:
> >>> >>>>>>>>> [DISCUSS]
> >>> >>>>>>>>>> > KIP-82 -
> >>> >>>>>>>>>> > > > Add
> >>> >>>>>>>>>> > > > >> > >> Record
> >>> >>>>>>>>>> > > > >> > >> >> >>> Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> thinking
> >>> >>>>> about
> >>> >>>>>>>>> it some
> >>> >>>>>>>>>> > more,
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> best
> >>> >>>>>>>>>> > > > >> > >> way to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > transmit
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> remapping
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> data to
> >>> >>>>>>>>> consumers would
> >>> >>>>>>>>>> > be to
> >>> >>>>>>>>>> > > > put it
> >>> >>>>>>>>>> > > > >> > in
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> MD
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> response
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it
> >>> should
> >>> >>> be
> >>> >>>>>>>>> discussed
> >>> >>>>>>>>>> > now.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On
> Wed,
> >>> >>> Nov
> >>> >>>>> 9,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>> > 12:09
> >>> >>>>>>>>>> > > AM,
> >>> >>>>>>>>>> > > > >> > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> im
> >>> not
> >>> >>>>>>> opposed
> >>> >>>>>>>>> to the
> >>> >>>>>>>>>> > idea of
> >>> >>>>>>>>>> > > > >> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> mapping.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > im
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> not
> >>> part
> >>> >>>>> of
> >>> >>>>>>>>> the "mvp"
> >>> >>>>>>>>>> > and,
> >>> >>>>>>>>>> > > > since
> >>> >>>>>>>>>> > > > >> it
> >>> >>>>>>>>>> > > > >> > >> >> requires
> >>> >>>>>>>>>> > > > >> > >> >> >>> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> wire
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > format
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> always
> >>> >>> be
> >>> >>>>>>>>> added later.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also,
> >>> >>> its
> >>> >>>>> not
> >>> >>>>>>>>> as
> >>> >>>>>>>>>> > simple as
> >>> >>>>>>>>>> > > just
> >>> >>>>>>>>>> > > > >> > >> >> configuring
> >>> >>>>>>>>>> > > > >> > >> >> >>> MM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> say
> >>> i've
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>> > large
> >>> >>>>>>>>>> > > > message
> >>> >>>>>>>>>> > > > >> > >> >> support as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> mirror
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> target
> >>> >>>>>>> cluster
> >>> >>>>>>>>> its been
> >>> >>>>>>>>>> > > > remapped
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> {999,1}.
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also
> >>> >>> need
> >>> >>>>> to
> >>> >>>>>>>>> be told
> >>> >>>>>>>>>> > to look
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> large
> >>> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> under
> >>> >>>>>>> {999,1}.
> >>> >>>>>>>>> doable,
> >>> >>>>>>>>>> > but
> >>> >>>>>>>>>> > > > tricky.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>> 8,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>> > 10:29
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> While
> >>> >>> you
> >>> >>>>>>> can
> >>> >>>>>>>>> do
> >>> >>>>>>>>>> > whatever
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> want
> >>> >>>>>>>>>> > > > >> > >> with a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > your
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> what
> >>> >>> I'd
> >>> >>>>>>>>> expect is
> >>> >>>>>>>>>> > for each
> >>> >>>>>>>>>> > > > app
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> namespaces
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> So
> >>> if I
> >>> >>>>>>>>> accidentally
> >>> >>>>>>>>>> > used
> >>> >>>>>>>>>> > > 666
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> > my
> >>> >>>>>>>>>> > > > >> > >> HR
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> department,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > still
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> run
> >>> >>>>>>> RadaiApp,
> >>> >>>>>>>>> I can
> >>> >>>>>>>>>> > config
> >>> >>>>>>>>>> > > > >> > >> "namespace=42"
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> everything
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> will
> >>> >>> look
> >>> >>>>>>>>> normal.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> This
> >>> >>>>> means
> >>> >>>>>>>>> you only
> >>> >>>>>>>>>> > need to
> >>> >>>>>>>>>> > > > sync
> >>> >>>>>>>>>> > > > >> > usage
> >>> >>>>>>>>>> > > > >> > >> >> >>> inside
> >>> >>>>>>>>>> > > > >> > >> >> >>> > your
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > own
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> organization.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> Still
> >>> >>>>> hard,
> >>> >>>>>>>>> but
> >>> >>>>>>>>>> > somewhat
> >>> >>>>>>>>>> > > > easier
> >>> >>>>>>>>>> > > > >> > than
> >>> >>>>>>>>>> > > > >> > >> >> syncing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>> > at 10:07
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > >> radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> and
> >>> >>> we
> >>> >>>>> can
> >>> >>>>>>>>> start
> >>> >>>>>>>>>> > with
> >>> >>>>>>>>>> > > > >> {namespace,
> >>> >>>>>>>>>> > > > >> > >> id}
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> always
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> add
> >>> >>> it
> >>> >>>>>>>>> later on
> >>> >>>>>>>>>> > if/when
> >>> >>>>>>>>>> > > > >> > collisions
> >>> >>>>>>>>>> > > > >> > >> >> >>> actually
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> they'd
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > a
> >>> >>>>>>> problem).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> every
> >>> >>>>>>>>> interested
> >>> >>>>>>>>>> > party (so
> >>> >>>>>>>>>> > > > orgs
> >>> >>>>>>>>>> > > > >> > or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > individuals)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> register
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>> prefix
> >>> >>>>> (0
> >>> >>>>>>> =
> >>> >>>>>>>>>> > reserved, 1 =
> >>> >>>>>>>>>> > > > >> > confluent
> >>> >>>>>>>>>> > > > >> > >> ...
> >>> >>>>>>>>>> > > > >> > >> >> >>> 666
> >>> >>>>>>>>>> > > > >> > >> >> >>> > = me
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> whatever
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> the
> >>> >>>>> 2nd ID
> >>> >>>>>>>>> - so once
> >>> >>>>>>>>>> > > > linkedin
> >>> >>>>>>>>>> > > > >> > >> >> registers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> say
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 3,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> free
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > to
> >>> >>> use
> >>> >>>>> {3,
> >>> >>>>>>>>> *} with a
> >>> >>>>>>>>>> > > > reasonable
> >>> >>>>>>>>>> > > > >> > >> >> >>> expectation
> >>> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> anything
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> else.
> >>> >>>>>>>>> further
> >>> >>>>>>>>>> > partitioning
> >>> >>>>>>>>>> > > > of
> >>> >>>>>>>>>> > > > >> > that *
> >>> >>>>>>>>>> > > > >> > >> >> >>> becomes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>> "upstream
> >>> >>>>>>>>>> > registration"
> >>> >>>>>>>>>> > > of a
> >>> >>>>>>>>>> > > > >> > >> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> only
> >>> >>>>>>>>>> > > > >> > >> >> >>> > has
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > On
> >>> >>> Tue,
> >>> >>>>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>> > at
> >>> >>>>>>>>>> > > 9:03
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > >> James
> >>> >>>>>>>>>> > > > >> > >> >> >>> Cheng <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>> wushujames@gmail.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > On
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016,
> >>> >>>>>>>>>> > at 5:54
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> Thank
> >>> >>>>>>>>> you so
> >>> >>>>>>>>>> > much for
> >>> >>>>>>>>>> > > > this
> >>> >>>>>>>>>> > > > >> > clear
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> fair
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> arguments.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> I'm
> >>> >>>>> in
> >>> >>>>>>>>> favor of
> >>> >>>>>>>>>> > ints.
> >>> >>>>>>>>>> > > > Not a
> >>> >>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > but
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> Even
> >>> >>>>>>>>> more in
> >>> >>>>>>>>>> > favor of
> >>> >>>>>>>>>> > > > >> Magnus's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > decentralized
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Roger's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> tweak:
> >>> >>>>>>>>> add a
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> > >> headers.
> >>> >>>>>>>>>> > > > >> > >> >> >>> This
> >>> >>>>>>>>>> > > > >> > >> >> >>> > will
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > allow
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> just
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> use
> >>> >>>>>>>>> whatever IDs
> >>> >>>>>>>>>> > it
> >>> >>>>>>>>>> > > wants
> >>> >>>>>>>>>> > > > >> > >> >> internally,
> >>> >>>>>>>>>> > > > >> > >> >> >>> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > let
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>> deploying
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> the
> >>> >>>>> app
> >>> >>>>>>>>> figure
> >>> >>>>>>>>>> > out an
> >>> >>>>>>>>>> > > > >> > available
> >>> >>>>>>>>>> > > > >> > >> >> >>> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > ID
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > So
> >>> >>>>>>>>>> > > > >> > io.confluent.schema-registry
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>> deployment
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> and
> >>> >>>>>>> 0x57
> >>> >>>>>>>>> on
> >>> >>>>>>>>>> > yours, and
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > poor
> >>> >>>>>>>>>> > > > >> > >> guys
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> worry
> >>> >>>>>>>>> about that.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> Gwen,
> >>> >>>>> if
> >>> >>>>>>> I
> >>> >>>>>>>>>> > understand
> >>> >>>>>>>>>> > > your
> >>> >>>>>>>>>> > > > >> > example
> >>> >>>>>>>>>> > > > >> > >> >> >>> right, an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > application
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> might
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> decide to
> >>> >>>>>>>>> use 0x01
> >>> >>>>>>>>>> > in one
> >>> >>>>>>>>>> > > > >> > >> deployment,
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > means
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> is
> >>> >>>>>>> written
> >>> >>>>>>>>> into the
> >>> >>>>>>>>>> > > > broker, it
> >>> >>>>>>>>>> > > > >> > >> will be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > saved on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> specific
> >>> >>>>>>>>> namespace
> >>> >>>>>>>>>> > > (0x01).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> If
> >>> >>> you
> >>> >>>>>>>>> were to
> >>> >>>>>>>>>> > mirror
> >>> >>>>>>>>>> > > that
> >>> >>>>>>>>>> > > > >> > message
> >>> >>>>>>>>>> > > > >> > >> >> into
> >>> >>>>>>>>>> > > > >> > >> >> >>> > another
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> accompany
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > message,
> >>> >>>>>>>>>> > > > right?
> >>> >>>>>>>>>> > > > >> > What
> >>> >>>>>>>>>> > > > >> > >> if
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> other
> >>> >>>>>>>>> cluster uses
> >>> >>>>>>>>>> > 0x57?
> >>> >>>>>>>>>> > > > They
> >>> >>>>>>>>>> > > > >> > won't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > understand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> I'm
> >>> >>>>> not
> >>> >>>>>>>>> sure
> >>> >>>>>>>>>> > that's an
> >>> >>>>>>>>>> > > > >> avoidable
> >>> >>>>>>>>>> > > > >> > >> >> >>> problem. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > think it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> order
> >>> >>>>> to
> >>> >>>>>>>>> share
> >>> >>>>>>>>>> > data, you
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> also
> >>> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shared
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>> understanding of
> >>> >>>>>>>>>> > what the
> >>> >>>>>>>>>> > > > >> > >> namespaces
> >>> >>>>>>>>>> > > > >> > >> >> >>> mean.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Which
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> sense,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> because
> >>> >>>>>>> the
> >>> >>>>>>>>>> > alternate
> >>> >>>>>>>>>> > > > (sharing
> >>> >>>>>>>>>> > > > >> > >> >> *nothing*
> >>> >>>>>>>>>> > > > >> > >> >> >>> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> all)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> would
> >>> >>>>> be
> >>> >>>>>>>>> no way to
> >>> >>>>>>>>>> > > > understand
> >>> >>>>>>>>>> > > > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> other.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> -James
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > On
> >>> >>>>> Tue,
> >>> >>>>>>>>> Nov 8,
> >>> >>>>>>>>>> > 2016 at
> >>> >>>>>>>>>> > > > 4:23
> >>> >>>>>>>>>> > > > >> > PM,
> >>> >>>>>>>>>> > > > >> > >> >> radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>> +1
> >>> >>>>> for
> >>> >>>>>>>>> sean's
> >>> >>>>>>>>>> > > document.
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > >> covers
> >>> >>>>>>>>>> > > > >> > >> >> >>> pretty
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> much
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> trade-offs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>>>>>> provides
> >>> >>>>>>>>>> > concrete
> >>> >>>>>>>>>> > > > figures
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> argue
> >>> >>>>>>>>>> > > > >> > >> >> >>> about
> >>> >>>>>>>>>> > > > >> > >> >> >>> > :-)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>>>>>>>> (nit-picking -
> >>> >>>>>>>>>> > used
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> same
> >>> >>>>>>>>>> > > > >> > >> xkcd
> >>> >>>>>>>>>> > > > >> > >> >> >>> twice,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> also
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > trove
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > has
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>> superceded
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > --
> >>> >>>>>>>>>> > > > >> > Gwen Shapira
> >>> >>>>>>>>>> > > > >> > Product Manager | Confluent
> >>> >>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
> @gwenshap
> >>> >>>>>>>>>> > > > >> > Follow us: Twitter | blog
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> --
> >>> >>>>>>>>>> > > > >> *Todd Palino*
> >>> >>>>>>>>>> > > > >> Staff Site Reliability Engineer
> >>> >>>>>>>>>> > > > >> Data Infrastructure Streaming
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> linkedin.com/in/toddpalino
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > --
> >>> >>>>>>>>>> > > > Gwen Shapira
> >>> >>>>>>>>>> > > > Product Manager | Confluent
> >>> >>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >>> >>>>>>>>>> > > > Follow us: Twitter | blog
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > The information contained in this email is strictly
> >>> >>>>> confidential
> >>> >>>>>>>>> and for
> >>> >>>>>>>>>> > the use of the addressee only, unless otherwise
> >>> indicated.
> >>> >>> If
> >>> >>>>> you
> >>> >>>>>>>>> are not
> >>> >>>>>>>>>> > the intended recipient, please do not read, copy,
> use or
> >>> >>>>> disclose
> >>> >>>>>>>>> to others
> >>> >>>>>>>>>> > this message or any attachment. Please also notify
> the
> >>> >>> sender
> >>> >>>>> by
> >>> >>>>>>>>> replying
> >>> >>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and
> >>> then
> >>> >>>>> delete
> >>> >>>>>>>>> the email
> >>> >>>>>>>>>> > and any copies of it. Opinions, conclusion (etc)
> that do
> >>> not
> >>> >>>>>>>>> relate to the
> >>> >>>>>>>>>> > official business of this company shall be
> understood as
> >>> >>>>> neither
> >>> >>>>>>>>> given nor
> >>> >>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets
> >>> Limited
> >>> >>> (a
> >>> >>>>>>>>> company
> >>> >>>>>>>>>> > registered in England and Wales, company number
> 04008957)
> >>> >>> and
> >>> >>>>> IG
> >>> >>>>>>>>> Index
> >>> >>>>>>>>>> > Limited (a company registered in England and Wales,
> >>> company
> >>> >>>>>>> number
> >>> >>>>>>>>>> > 01190902). Registered address at Cannon Bridge
> House, 25
> >>> >>>>> Dowgate
> >>> >>>>>>>>> Hill,
> >>> >>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register
> number
> >>> >>>>> 195355)
> >>> >>>>>>>>> and IG
> >>> >>>>>>>>>> > Index Limited (register number 114059) are
> authorised and
> >>> >>>>>>>>> regulated by the
> >>> >>>>>>>>>> > Financial Conduct Authority.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> The information contained in this email is strictly
> confidential
> >>> >>> and
> >>> >>>>>>> for
> >>> >>>>>>>>> the use of the addressee only, unless otherwise indicated.
> If you
> >>> >>> are
> >>> >>>>>>> not
> >>> >>>>>>>>> the intended recipient, please do not read, copy, use or
> disclose
> >>> >>> to
> >>> >>>>>>> others
> >>> >>>>>>>>> this message or any attachment. Please also notify the
> sender by
> >>> >>>>>>> replying
> >>> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then
> delete
> >>> >>> the
> >>> >>>>>>> email
> >>> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
> >>> relate
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> official business of this company shall be understood as
> neither
> >>> >>>>> given
> >>> >>>>>>> nor
> >>> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> >>> >>> company
> >>> >>>>>>>>> registered in England and Wales, company number 04008957)
> and IG
> >>> >>>>> Index
> >>> >>>>>>>>> Limited (a company registered in England and Wales, company
> >>> number
> >>> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >>> >>>>> Hill,
> >>> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> >>> >>>>> and IG
> >>> >>>>>>>>> Index Limited (register number 114059) are authorised and
> >>> regulated
> >>> >>>>> by
> >>> >>>>>>> the
> >>> >>>>>>>>> Financial Conduct Authority.
> >>> >>>>>>>>>
> >>> >>>>>>>
> >>> >>>>>
> >>> >>> The information contained in this email is strictly confidential
> and
> >>> for
> >>> >>> the use of the addressee only, unless otherwise indicated. If you
> are
> >>> not
> >>> >>> the intended recipient, please do not read, copy, use or disclose
> to
> >>> others
> >>> >>> this message or any attachment. Please also notify the sender by
> >>> replying
> >>> >>> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> >>> email
> >>> >>> and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> >>> the
> >>> >>> official business of this company shall be understood as neither
> given
> >>> nor
> >>> >>> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >>> >>> registered in England and Wales, company number 04008957) and IG
> Index
> >>> >>> Limited (a company registered in England and Wales, company number
> >>> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> >>> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> >>> IG
> >>> >>> Index Limited (register number 114059) are authorised and
> regulated by
> >>> the
> >>> >>> Financial Conduct Authority.
> >>> >>>
> >>> >
> >>> >
> >>>
> >>> The information contained in this email is strictly confidential and
> for
> >>> the use of the addressee only, unless otherwise indicated. If you are
> not
> >>> the intended recipient, please do not read, copy, use or disclose to
> others
> >>> this message or any attachment. Please also notify the sender by
> replying
> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >>> official business of this company shall be understood as neither given
> nor
> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >>> registered in England and Wales, company number 04008957) and IG Index
> >>> Limited (a company registered in England and Wales, company number
> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >>> Index Limited (register number 114059) are authorised and regulated by
> the
> >>> Financial Conduct Authority.
> >>>
> >
> >
> >
> > --
> > Nacho - Ignacio Solis - isolis@igso.net
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
yeah I'm ok with both strings and multi-maps
On Fri, Feb 10, 2017 at 2:37 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Can we also compromise on a multi-map?
>
> One of my use-cases is tagging on multi-DC setup. It will be nice if
> each "hop" can add "Tag" header with its own DC-name without having to
> edit a single string that someone else wrote. For some financial
> use-cases it will be a requirement.
>
> Gwen
>
> On Fri, Feb 10, 2017 at 12:19 PM, Ignacio Solis <is...@igso.net> wrote:
> > I would be willing to compromise on Strings as keys if that moves all
> > of us forward.
> >
> > Right now headers are blocking a few of the things we want to do.
> >
> > Nacho
> >
> > On Thu, Feb 9, 2017 at 2:14 PM, Jay Kreps <ja...@confluent.io> wrote:
> >> I'm neutral positive on adding headers conceptually. I'm -1 on a
> proposal
> >> using ints for header names. I just don't think a global namespace with
> >> integer names is an acceptable end-user interface given that our goal
> is to
> >> encourage a federated ecosystem of header users. Not trying to be a
> pain in
> >> the ass, I just think it would be a bit of a wart.
> >>
> >> -Jay
> >>
> >> On Thu, Feb 9, 2017 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> >> wrote:
> >>
> >>> As we'd like to start work on this to make it into the next release in
> a
> >>> few months.
> >>>
> >>> If we were to put this to the vote today who would actually give this
> a -1
> >>> vote?
> >>>
> >>> Based on the number of identified usecases (some stronger than others)
> >>>
> >>> Also if we were to keep with int keys based on all the discussion
> thread
> >>> for the pros vs cons who would give a -1 as it stands? Eg anyone
> believe it
> >>> simply wouldn't work.
> >>>
> >>> If not so we make progress would it be sensible to put this to vote?
> So we
> >>> can start work?
> >>>
> >>> Cheers
> >>> Mike
> >>>
> >>>
> >>> Sent using OWA for iPhone
> >>> ________________________________________
> >>> From: Jeroen van Disseldorp <je...@axual.io>
> >>> Sent: Wednesday, February 1, 2017 2:43:45 PM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>>
> >>> I'm lagging a bit here too, but want to respond to the usefulness of
> >>> this KIP for two use cases: bidirectional mirroring and smart cluster
> >>> switches.
> >>>
> >>> Bidirectional mirroring
> >>>
> >>> For a Dutch bank we have currently implemented bidirectional mirroring
> >>> using our own headers, which we encode in the Value using a custom
> >>> serializer. It encodes just one byte extra, containing a
> "copy/original"
> >>> bit. When applications serialize an avro object, the bit is unset by
> >>> default. Our mirroring mechanism subscribes to all topics, checks the
> >>> bit, and when unset the bit is set and the message is forwarded to the
> >>> other DC.
> >>>
> >>> The overhead of this is pretty low (1 byte) and it's cheap, since the
> >>> mirroring mechanism does not need to fully deserialize the value, but
> >>> just check+flip one bit in the byte[] value. On the receiving end, the
> >>> mirroring mechanism sees the message for the second time, but notices
> >>> the message is a copy already and ignores it.
> >>>
> >>> The mechanism has been in production since last year and it works
> really
> >>> well. Its downside however is the incompatibility with avro serialized
> >>> values, as the whole organization is mandated to use our own custom
> >>> serde and we were forced to create a wrapper client library around the
> >>> default kafka client. The callback we offer to applications is very
> >>> similar to that proposed in the KIP, as every message our users get
> >>> consists of <Metadata, K, V>.
> >>>
> >>> Additionally our header contains a message id and serialization
> >>> timestamp (invented before 0.10.0.0). These fields could also be done
> >>> away with when this KIP is implemented.
> >>>
> >>> Smart cluster switches
> >>>
> >>> Another case we are considering to use headers for is enabling "smart
> >>> cluster switches" for consumers and producers in a way that would avoid
> >>> duplicates and/or missed messages in most cases. This involves encoding
> >>> a producer id (app name or id), producer instance id (uuid generated
> >>> upon producer's start) and sequence number in every message. We expect
> >>> to use one sequence per partition.
> >>>
> >>> These fields enable our consumers to keep track of which messages were
> >>> seen without relying on offsets, which is a cluster-bound value.
> >>> Consumers keep track of seen messages by keeping a small administration
> >>> of ranges of sequence numbers per producer instance and partition.
> Since
> >>> sequence numbers are always increasing per producer and partition, the
> >>> memory consumption of such administration is pretty low, while being
> >>> meaningful across clusters. After every poll() we can commit the
> >>> administration to a topic in much the same way as we do with offsets.
> >>> The difference is that this topic can be mirrored to the other cluster
> >>> and upon a switch of consumer, all instances can use the topic to
> >>> initialize themselves and skip all messages already processed.
> >>>
> >>> Hope these views help in the upcoming decision on this KIP.
> >>>
> >>> Regards
> >>> Jeroen
> >>>
> >>>
> >>>
> >>> On 09/01/2017 20:40, Gwen Shapira wrote:
> >>> > Mirroring + ClusterIDs:
> >>> >
> >>> > Kafka clusters already have an automatically generated ID that is
> >>> > exposed to clients:
> >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 78%3A+Cluster+Id
> >>> >
> >>> > It sounds trivial to get MirrorMaker's consumer to grab this ID
> (which
> >>> > always exists) from the source cluster and add it to the message
> >>> > headers.
> >>> >
> >>> > Obviously adding stuff to the message makes it larger so this should
> >>> > be a feature you should be able to turn on/off depending on the
> >>> > replication requirements. If you need the feature - you pay the
> price.
> >>> > I don't think there is an immediate need to put an effort into
> >>> > shrinking the Cluster ID, but others can comment on that.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> >>> >> Hey Michael,
> >>> >>
> >>> >> Sorry for lagging, I've been out for the holiday break. What would
> help
> >>> me
> >>> >> would be to get to a consolidated list of use cases we've worked
> out and
> >>> >> think add real value. I think getting down to the details really
> matters
> >>> >> for these.
> >>> >>
> >>> >> The two lists I'm aware of are the one in the KIP and the one you
> gave
> >>> in
> >>> >> this email thread. Both seem to be more at the sketch level. Let me
> >>> review
> >>> >> the use cases and see if I have it right. If so let's get it in the
> >>> design
> >>> >> proposal.
> >>> >>
> >>> >> Your use cases:
> >>> >>
> >>> >> 1. Tracing IDs -- I get this use case and think I see how it
> would
> >>> work.
> >>> >> I think because the tracing is from a third party system there
> is
> >>> some
> >>> >> value to having a header versus a custom format to make this
> work
> >>> fully
> >>> >> transparently.
> >>> >> 2. Mirroring -- I'm a bit unsure how this will work, I'll
> explain my
> >>> >> thinking below
> >>> >> 3. End-to-end encryption -- I think what you are saying is that
> if
> >>> you
> >>> >> want "routing' fields that are accessible in transit without
> >>> decrypting you
> >>> >> need a scheme to leave this decrypted.I think the idea is to
> avoid
> >>> >> decrypting to get a header, but I'm not sure if I get how this
> is
> >>> better
> >>> >> than a custom message format that doesn't encrypt the things you
> >>> don't want
> >>> >> encrypted since you have to be aware of the scheme anyway.
> >>> >> 4. EOS -- I'm not convinced this is a good approach for EOS.
> >>> >>
> >>> >> Linked from KIP:
> >>> >>
> >>> >> 1. Schema-Registry-aware avro serialization -- not clear how
> this is
> >>> >> better than keeping the schema id in the value
> >>> >> 2. Polyglot kafka topics -- marginally better than a container
> >>> format,
> >>> >> but of course you still have to all agree on the the header and
> >>> >> header=>serialization mapping so not sure if it's really better
> >>> than a
> >>> >> container format. Also not wild about polyglot topics :-)
> >>> >> 3. End-to-End routing trail including mirroring --- I'd put this
> >>> under
> >>> >> tracing ids
> >>> >> 4. Cross-service process tracing -- ditto
> >>> >> 5. Signing -- I like this use case but it seems to depend on a
> >>> feature
> >>> >> we don't have and aren't proposing adding, right?
> >>> >> 6. Per-record custom compression -- why do you want this?
> >>> compressing
> >>> >> per message won't work well compared to batch compression,
> right?
> >>> >> 7. Large Message Support -- I think we can agree that although
> this
> >>> >> might work it's clearly not the right way to support large
> messages
> >>> in Kafka
> >>> >>
> >>> >> Matthias
> >>> >>
> >>> >> 1. Control messages --- Matthias, not sure if I get the
> motivation
> >>> in
> >>> >> streams for this. If we did want to do it, wouldn't it be
> easier to
> >>> >> implement and by allowing users to send control messages (in the
> >>> EOS sense)
> >>> >> and adding a consumer config to allow either passing these on or
> >>> >> suppressing them?
> >>> >>
> >>> >> So I if I were to summarize I'd list the use cases as follows:
> >>> >>
> >>> >> - Much better with headers: tracing IDs
> >>> >> - Somewhat better: end-to-end encryption, polyglot topics
> >>> >> - Don't buy: Mirroring (though perhaps could be made to work
> with
> >>> >> changes to the proposal), EOS, large messages, compression
> >>> >> - Not sure: Control messages
> >>> >>
> >>> >> Does this match where other people are at? From my point of view for
> >>> >> something like this where we are changing the core data model we
> should
> >>> >> ideally get a handful of use cases to the point where they are
> really
> >>> well
> >>> >> thought out and we know they can use the feature. I think we have
> one
> >>> solid
> >>> >> one with tracing IDs, and I think maybe mirroring could be another
> with
> >>> >> some work. I'm less sure about the others.
> >>> >>
> >>> >> Bi-directional Mirroring
> >>> >> I think bidirectional mirroring could potentially become a
> compelling
> >>> use
> >>> >> case if we an find a way to make it work. If I understand it, the
> >>> proposal
> >>> >> is to add a header like "origin-cluster" with the cluster id of the
> >>> Kafka
> >>> >> cluster. Then modify the mirroring tool to disallow mirroring a
> message
> >>> >> back to it's own origin cluster. This basic idea is definitely
> workable
> >>> and
> >>> >> useful, but can we really get to a good implementation using
> headers?
> >>> >> I imagine the approach would be to add this header in the producer
> if
> >>> some
> >>> >> config is set. There are two problems:
> >>> >>
> >>> >> - You add a pretty big honking cluster ID to each message. For
> small
> >>> >> messages this maybe larger than the contents itself.
> >>> >> - What happens if someone somewhere forgets to set that config
> on
> >>> some
> >>> >> client somewhere?
> >>> >>
> >>> >> If someone forgets to set the config in some client somewhere I
> guess
> >>> you
> >>> >> have various options in mirror maker:
> >>> >>
> >>> >> 1. drop the data (pretty bad, now your data is out of sync)
> >>> >> 2. crash (pretty bad--how do you recover)
> >>> >> 3. insert some default (pretty bad since would also lead to data
> >>> loss if
> >>> >> that's not right)
> >>> >>
> >>> >> Basically although I think a cluster tag could be a good way to do
> >>> >> bi-directional mirroring, I'm not sure this naive approach I
> described
> >>> >> would get us to a usable implementation. This could be fixable. I
> think
> >>> if
> >>> >> there was some scheme to check required headers on the server that
> could
> >>> >> alleviate the missing header problem. Not really sure what to do
> about
> >>> the
> >>> >> message bloat problem, but maybe there could be some way to shrink
> the
> >>> >> cluster id?
> >>> >>
> >>> >> Cheers,
> >>> >>
> >>> >> -Jay
> >>> >>
> >>> >>
> >>> >> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> Michael.Pearce@ig.com
> >>> >
> >>> >> wrote:
> >>> >>
> >>> >>> @Jay, Ismael and a few others.
> >>> >>>
> >>> >>> Based on the discussion on kip-87, and the additional use case by
> >>> Matthias.
> >>> >>>
> >>> >>> Just for clarity do we now have consensus on the earlier points (1)
> >>> >>> headers are useful (2) headers should be native in Kafka?
> >>> >>>
> >>> >>> Cheers
> >>> >>> Mike
> >>> >>> ________________________________________
> >>> >>> From: radai <ra...@gmail.com>
> >>> >>> Sent: Monday, December 19, 2016 5:01 PM
> >>> >>> To: dev@kafka.apache.org
> >>> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>> >>>
> >>> >>> having said that im perfectly fine with something like a "no
> payload"
> >>> flag
> >>> >>> (as opposed to null payload), which would mean such a msg is not
> >>> returned
> >>> >>> from poll() unless the user code specifically "opts in" and would
> serve
> >>> >>> only to carry headers (making it a control message)
> >>> >>>
> >>> >>> On Mon, Dec 19, 2016 at 8:58 AM, radai <radai.rosenblatt@gmail.com
> >
> >>> wrote:
> >>> >>>
> >>> >>>> if all you want is to tag "end of stream" wouldnt a header on the
> last
> >>> >>> msg
> >>> >>>> in the stream be enough?
> >>> >>>>
> >>> >>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> >>> matthias@confluent.io>
> >>> >>>> wrote:
> >>> >>>>
> >>> >>>>> The details about headers for control messages are still to
> define.
> >>> But
> >>> >>>>> yes, the idea is to have some common default behavior that
> clients
> >>> would
> >>> >>>>> need to implement.
> >>> >>>>>
> >>> >>>>> The point is, that "regular headers" add meta data to regular
> >>> messages.
> >>> >>>>> Thus, those messages will be returned to the user via .poll().
> And
> >>> after
> >>> >>>>> the message is received the user can check if meta data is
> present
> >>> and
> >>> >>>>> read it.
> >>> >>>>>
> >>> >>>>> For control messages, we do not want those to pop up via .poll()
> as
> >>> >>>>> those are no regular messages. A client would need to opt-in to
> see
> >>> >>>>> those messages (either via poll() or maybe a callback). Thus, we
> need
> >>> >>>>> some special (standardized) header IDs that indicate control
> messages
> >>> >>>>> that should not be returned to the user via poll() by default.
> >>> >>>>>
> >>> >>>>>
> >>> >>>>> -Matthias
> >>> >>>>>
> >>> >>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
> >>> >>>>>> Matthias,
> >>> >>>>>>
> >>> >>>>>> Thanks for your input. I'm +1 on control messages as they seem
> to
> >>> be
> >>> >>>>> the
> >>> >>>>>> simplest way to implement watermarks (
> >>> >>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-
> streaming-102
> >>> ),
> >>> >>> a
> >>> >>>>>> feature that would add a lot of value to Kafka Streams IMHO.
> >>> >>>>>>
> >>> >>>>>> Your argument that the control-message indicator belongs in the
> >>> >>>>> client-only
> >>> >>>>>> section of the record format make sense. Just to make sure I
> >>> >>>>> understand,
> >>> >>>>>> are you suggesting that control messages would be indicated by a
> >>> >>>>> standard
> >>> >>>>>> reserved header? By standard, I mean that ALL Kafka consumers
> would
> >>> >>>>> know
> >>> >>>>>> to handle these messages differently (possibly just ignoring
> them).
> >>> >>>>> This
> >>> >>>>>> would need to be added to the specification of the consumer
> protocol
> >>> >>> so
> >>> >>>>>> that all Kafka clients implement it, right? I think it's a good
> >>> idea
> >>> >>>>> but
> >>> >>>>>> just checking.
> >>> >>>>>>
> >>> >>>>>> Cheers,
> >>> >>>>>>
> >>> >>>>>> Roger
> >>> >>>>>>
> >>> >>>>>>
> >>> >>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> >>> >>> matthias@confluent.io
> >>> >>>>>> wrote:
> >>> >>>>>>
> >>> >>>>>>> Yes and no. I did overload the term "control message".
> >>> >>>>>>>
> >>> >>>>>>> EOS control messages are for client-broker communication and
> thus
> >>> >>> never
> >>> >>>>>>> exposed to any application. And I think this is a good design
> >>> because
> >>> >>>>>>> broker needs to understand those control messages. Thus, this
> >>> should
> >>> >>> be
> >>> >>>>>>> a protocol change.
> >>> >>>>>>>
> >>> >>>>>>> The type of control messages I have in mind are for
> client-client
> >>> >>>>>>> (application-application) communication and the broker is
> agnostic
> >>> to
> >>> >>>>>>> them. Thus, it should not be a protocol change.
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> -Matthias
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>>
> >>> >>>>>>> On 12/14/16 9:42 AM, radai wrote:
> >>> >>>>>>>> arent control messages getting pushed as their own top level
> >>> >>> protocol
> >>> >>>>>>>> change (and a fairly massive one) for the transactions KIP ?
> >>> >>>>>>>>
> >>> >>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> >>> >>>>> matthias@confluent.io>
> >>> >>>>>>>> wrote:
> >>> >>>>>>>>
> >>> >>>>>>>>> Hi,
> >>> >>>>>>>>>
> >>> >>>>>>>>> I want to add a completely new angle to this discussion. For
> >>> this,
> >>> >>> I
> >>> >>>>>>>>> want to propose an extension for the headers feature that
> enables
> >>> >>> new
> >>> >>>>>>>>> uses cases -- and those new use cases might convince people
> to
> >>> >>>>> support
> >>> >>>>>>>>> headers (of course including the larger scoped proposal).
> >>> >>>>>>>>>
> >>> >>>>>>>>> Extended Proposal:
> >>> >>>>>>>>>
> >>> >>>>>>>>> Allow messages with a certain header key to be special
> "control
> >>> >>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an
> >>> application
> >>> >>>>> via
> >>> >>>>>>>>> .poll().
> >>> >>>>>>>>>
> >>> >>>>>>>>> Thus, a consumer client would automatically skip over those
> >>> >>>>> messages. If
> >>> >>>>>>>>> an application knows about embedded control messages, it can
> >>> "sing
> >>> >>>>> up"
> >>> >>>>>>>>> to those messages by the consumer client and either get a
> >>> callback
> >>> >>> or
> >>> >>>>>>>>> the consumer auto-drop for this messages gets disabled
> (allowing
> >>> to
> >>> >>>>>>>>> consumer those messages via poll()).
> >>> >>>>>>>>>
> >>> >>>>>>>>> (The details need further considerations/discussion. I just
> want
> >>> to
> >>> >>>>>>>>> sketch the main idea.)
> >>> >>>>>>>>>
> >>> >>>>>>>>> Usage:
> >>> >>>>>>>>>
> >>> >>>>>>>>> There is a shared topic (ie, used by multiple applications)
> and a
> >>> >>>>>>>>> producer application wants to embed a special message in the
> >>> topic
> >>> >>>>> for a
> >>> >>>>>>>>> dedicated consumer application. Because only one application
> will
> >>> >>>>>>>>> understand this message, it cannot be a regular message as
> this
> >>> >>> would
> >>> >>>>>>>>> break all applications that do not understand this message.
> The
> >>> >>>>> producer
> >>> >>>>>>>>> application would set a special metadata key and no consumer
> >>> >>>>> application
> >>> >>>>>>>>> would see this control message by default because they did
> not
> >>> >>> enable
> >>> >>>>>>>>> their consumer client to return this message in poll() (and
> the
> >>> >>>>> client
> >>> >>>>>>>>> would just drop this message with special metadata key).
> Only the
> >>> >>>>> single
> >>> >>>>>>>>> application that should receive this message, will subscribe
> to
> >>> >>> this
> >>> >>>>>>>>> message on its consumer client and process it.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Concrete Use Case: Kafka Streams
> >>> >>>>>>>>>
> >>> >>>>>>>>> In Kafka Streams, we would like to propagate "control
> messages"
> >>> >>> from
> >>> >>>>>>>>> subtopology to subtopology. There are multiple scenarios for
> >>> which
> >>> >>>>> this
> >>> >>>>>>>>> would be useful. For example, currently we do not guarantee a
> >>> >>>>>>>>> "consistent shutdown" of an application. By this, I mean that
> >>> input
> >>> >>>>>>>>> records might not be completely processed by the whole
> topology
> >>> >>>>> because
> >>> >>>>>>>>> the application shutdown happens "in between" and an
> intermediate
> >>> >>>>> result
> >>> >>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user
> would
> >>> see
> >>> >>>>> an
> >>> >>>>>>>>> committed offset of the source topic of the application, but
> no
> >>> >>>>>>>>> corresponding result record in the output topic.
> >>> >>>>>>>>>
> >>> >>>>>>>>> Having "shutdown markers" would allow us, to first stop the
> >>> >>> upstream
> >>> >>>>>>>>> subtopology and write this marker into the intermediate
> topic and
> >>> >>> the
> >>> >>>>>>>>> downstream subtopology would only shut down itself after is
> sees
> >>> >>> the
> >>> >>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that
> no
> >>> >>>>>>>>> "in-flight" messages got stuck in intermediate topics.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch
> >>> Processing).
> >>> >>>>>>>>> There was a discussion about the proposed metadata topic,
> and we
> >>> >>>>> could
> >>> >>>>>>>>> avoid this metadata topic if we would have "control
> messages".
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Right now, we cannot insert an "application control message"
> >>> >>> because
> >>> >>>>>>>>> Kafka Streams does not own all topics it read/writes and thus
> >>> might
> >>> >>>>>>>>> break other consumer application (as described above) if we
> >>> inject
> >>> >>>>>>>>> random messages that are not understood by other apps.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Of course, one can work around "embedded control messaged" by
> >>> using
> >>> >>>>> an
> >>> >>>>>>>>> additional topic to propagate control messaged between
> >>> application
> >>> >>>>> (as
> >>> >>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka
> Streams). But
> >>> >>>>> there
> >>> >>>>>>>>> are major concerns about adding this metadata topic in the
> KIP
> >>> and
> >>> >>>>> this
> >>> >>>>>>>>> shows that other application that need a similar pattern
> might
> >>> >>> profit
> >>> >>>>>>>>> from topic embedded "control messages", too.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> One last important consideration: those "control messages"
> are
> >>> used
> >>> >>>>> for
> >>> >>>>>>>>> client to client communication and are not understood by the
> >>> >>> broker.
> >>> >>>>>>>>> Thus, those messages should not be enabled within the message
> >>> >>> format
> >>> >>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> record
> >>> >>>>> headers
> >>> >>>>>>>>> would be a nice way to implement them. Because KIP-82 did
> >>> consider
> >>> >>>>> key
> >>> >>>>>>>>> namespaces for metatdata keys, this extension should not be
> an
> >>> own
> >>> >>>>> KIP
> >>> >>>>>>>>> but should be included in KIP-82 to reserve a namespace for
> >>> >>> "control
> >>> >>>>>>>>> message" in the first place.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> Sorry for the long email... Looking forward to your feedback.
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> -Matthias
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>>
> >>> >>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> >>> >>>>>>>>>> Hi Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
> >>> >>>>> (HTTP/JMS/
> >>> >>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
> >>> >>> (though I
> >>> >>>>>>>>> believe it contains the end2end uuid embedded in this id), on
> >>> >>>>> receiving
> >>> >>>>>>> the
> >>> >>>>>>>>> message at the receiving JVM the apm code takes this out, and
> >>> >>>>> continues
> >>> >>>>>>> its
> >>> >>>>>>>>> tracing on the that new thread. Both JVM's (and other
> languages
> >>> the
> >>> >>>>> APM
> >>> >>>>>>>>> tool supports) send this data async back to the central
> >>> controllers
> >>> >>>>>>> where
> >>> >>>>>>>>> the stiching togeather occurs. For this they need some header
> >>> space
> >>> >>>>> for
> >>> >>>>>>>>> them to put this id.
> >>> >>>>>>>>>> 101) Yes indeed we have a business transaction Id in the
> >>> payload.
> >>> >>>>>>> Though
> >>> >>>>>>>>> this is a system level tracing, that we need to have marry
> up.
> >>> Also
> >>> >>>>> as
> >>> >>>>>>> per
> >>> >>>>>>>>> note on end2end encryption we'd be unable to prove the flow
> if
> >>> the
> >>> >>>>>>> payload
> >>> >>>>>>>>> is encrypted as we'd not have access to this at certain
> points of
> >>> >>> the
> >>> >>>>>>> flow
> >>> >>>>>>>>> through the infrastructure/platform.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 103) As said we use this mechanism in IG very successfully,
> as
> >>> >>>>> stated
> >>> >>>>>>>>> per key we guarantee the transaction producing app to handle
> the
> >>> >>>>>>>>> transaction of a key at one DC unless at point of critical
> >>> failure
> >>> >>>>>>> where we
> >>> >>>>>>>>> have to flip processing to another. We care about key
> ordering.
> >>> >>>>>>>>>> I disagree on the offset comment for the partition solution
> >>> unless
> >>> >>>>> you
> >>> >>>>>>>>> do full ISR, or expensive full XA transactions even with
> >>> partitions
> >>> >>>>> you
> >>> >>>>>>>>> cannot fully guarantee offsets would match.
> >>> >>>>>>>>>> 105) Very much so, I need to have access at the platform
> level
> >>> to
> >>> >>>>> the
> >>> >>>>>>>>> other meta data all mentioned, without having to need to have
> >>> >>> access
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> encryption keys of the payload.
> >>> >>>>>>>>>> 106)
> >>> >>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to
> >>> have
> >>> >>> a
> >>> >>>>>>>>> global producerId register which would be very hard to
> >>> >>>>> enforce/ensure is
> >>> >>>>>>>>> current and correct, just to understand the message origins
> of
> >>> its
> >>> >>>>>>>>> region/az/cluster for routing.
> >>> >>>>>>>>>> The client wrapper version, producerId can be the same, as
> >>> >>> obviously
> >>> >>>>>>> the
> >>> >>>>>>>>> producer could upgrade its wrapper, as such we need to know
> what
> >>> >>>>> wrapper
> >>> >>>>>>>>> version the message is created with.
> >>> >>>>>>>>>> Likewise the IP address, as stated we can have our producer
> >>> move,
> >>> >>>>> where
> >>> >>>>>>>>> its IP would change.
> >>> >>>>>>>>>> 107)
> >>> >>>>>>>>>> UUID is set on the message by interceptors before actual
> >>> producer
> >>> >>>>>>>>> transport send. This is for platform level message dedupe
> >>> >>> guarantee,
> >>> >>>>> the
> >>> >>>>>>>>> business payload should be agnostic to this. Please see
> >>> >>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
> >>> >>>>> ection.html
> >>> >>>>>>>>> note this is not touching business payloads.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Hi, Michael,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Thanks for the reply. I find it very helpful.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Data lineage:
> >>> >>>>>>>>>> 100. I'd like to understand the APM use case a bit
> more. It
> >>> >>>>> sounds
> >>> >>>>>>>>> like
> >>> >>>>>>>>>> that those APM plugins can generate a transaction id
> that
> >>> we
> >>> >>>>> could
> >>> >>>>>>>>>> potentially put in the header of every message. How
> would
> >>> you
> >>> >>>>>>>>> typically
> >>> >>>>>>>>>> make use of such transaction ids? Are there other
> metadata
> >>> >>>>>>>>> associated with
> >>> >>>>>>>>>> the transaction id and if so, how are they propagated
> >>> >>>>> downstream?
> >>> >>>>>>>>>> 101. For the finance use case, if the concept of
> >>> transaction
> >>> >>> is
> >>> >>>>>>>>> important,
> >>> >>>>>>>>>> wouldn't it be typically included in the message
> payload
> >>> >>>>> instead of
> >>> >>>>>>>>> as an
> >>> >>>>>>>>>> optional header field?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 102. The data lineage that Altas and Navigator support
> >>> seems
> >>> >>> to
> >>> >>>>> be
> >>> >>>>>>>>> at the
> >>> >>>>>>>>>> dataset level, not per record level? So, not sure if
> per
> >>> >>> message
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> are relevant there.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Mirroring:
> >>> >>>>>>>>>> 103. The benefit of using separate partitions is that
> it
> >>> >>>>>>> potentially
> >>> >>>>>>>>> makes
> >>> >>>>>>>>>> it easy to preserve offsets during mirroring. This will
> >>> make
> >>> >>> it
> >>> >>>>>>>>> easier for
> >>> >>>>>>>>>> consumer to switch clusters. Currently, the consumers
> can
> >>> >>> switch
> >>> >>>>>>>>> clusters
> >>> >>>>>>>>>> by using the timestampToOffset() api, but it has to
> deal
> >>> with
> >>> >>>>>>>>> duplicates.
> >>> >>>>>>>>>> Good point on the issue with log compact and I am not
> sure
> >>> how
> >>> >>>>> to
> >>> >>>>>>>>> address
> >>> >>>>>>>>>> this. However, even if we mirror into the existing
> >>> partitions,
> >>> >>>>> the
> >>> >>>>>>>>> ordering
> >>> >>>>>>>>>> for messages generated from different clusters seems
> >>> >>>>>>>>> non-deterministic
> >>> >>>>>>>>>> anyway. So, it seems that the consumers already have to
> >>> deal
> >>> >>>>> with
> >>> >>>>>>>>> that? If
> >>> >>>>>>>>>> a topic is compacted, does that mean which messages are
> >>> >>>>> preserved
> >>> >>>>>>> is
> >>> >>>>>>>>> also
> >>> >>>>>>>>>> non-deterministic across clusters?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> 104. Good point on partition key.
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> End-to-end encryption:
> >>> >>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are
> >>> headers
> >>> >>>>>>>>> useful there?
> >>> >>>>>>>>>> Auditing:
> >>> >>>>>>>>>> 106. It seems other than the UUID, all other metadata
> are
> >>> per
> >>> >>>>>>>>> producer?
> >>> >>>>>>>>>> EOS:
> >>> >>>>>>>>>> 107. How are those UUIDs generated? I am not sure if
> they
> >>> can
> >>> >>> be
> >>> >>>>>>>>> generated
> >>> >>>>>>>>>> in the producer library. An application may send
> messages
> >>> >>>>> through a
> >>> >>>>>>>>> load
> >>> >>>>>>>>>> balancer and on retry, the same message could be
> routed to
> >>> a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> producer instance. So, it seems that the application
> has to
> >>> >>>>>>> generate
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> UUIDs. In that case, shouldn't the application just
> put the
> >>> >>>>> UUID in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> payload?
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Thanks,
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> Jun
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> >>> >>>>>>>>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>> wrote:
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> > Hi Jun.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Per Transaction Tracing / Data Lineage.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As Stated in the KIP this has the first use case of
> how
> >>> many
> >>> >>>>> APM
> >>> >>>>>>>>> tools now
> >>> >>>>>>>>>> > work.
> >>> >>>>>>>>>> > I would find it impossible for any one to argue this
> is
> >>> not
> >>> >>>>>>>>> important or a
> >>> >>>>>>>>>> > niche market as it has its own gartner report for
> this
> >>> >>> space.
> >>> >>>>>>> Such
> >>> >>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace,
> Hawqular
> >>> are
> >>> >>>>> but a
> >>> >>>>>>>>> few.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise these APM tools can help very rapidly track
> down
> >>> >>>>> issues
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > automatically capture metrics, perform actions based
> on
> >>> >>>>>>> unexpected
> >>> >>>>>>>>> behavior
> >>> >>>>>>>>>> > to auto recover services.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Before mentioning looking at aggregated stats, in
> these
> >>> >>> cases
> >>> >>>>>>> where
> >>> >>>>>>>>>> > actually on critical flows we cannot afford to have
> >>> >>> aggregated
> >>> >>>>>>>>> rolled up
> >>> >>>>>>>>>> > stats only.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > With the APM tool we use its actually able to detect
> a
> >>> >>> single
> >>> >>>>>>>>> transaction
> >>> >>>>>>>>>> > failure and capture the thread traces in the JVM
> where it
> >>> >>>>> failed
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > everything for us, to the point it sends us alerts
> where
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > giving the line number of the code that caused it,
> the
> >>> >>>>>>> transaction
> >>> >>>>>>>>> trace
> >>> >>>>>>>>>> > through all the services and endpoints (supported)
> upto
> >>> the
> >>> >>>>> point
> >>> >>>>>>>>> of
> >>> >>>>>>>>>> > failure, it can also capture the data in and out (so
> we
> >>> can
> >>> >>>>>>>>> replay).
> >>> >>>>>>>>>> > Because atm Kafka doesn't support us being able to
> stich
> >>> in
> >>> >>>>> these
> >>> >>>>>>>>> tracing
> >>> >>>>>>>>>> > transaction ids natively, we cannot get these
> benefits as
> >>> >>>>> such is
> >>> >>>>>>>>> limiting
> >>> >>>>>>>>>> > our ability support apps and monitor them to the same
> >>> >>>>> standards
> >>> >>>>>>> we
> >>> >>>>>>>>> come to
> >>> >>>>>>>>>> > expect when on a kafka flow.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > This actually ties in with Data Lineage, as the same
> >>> tracing
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>> > to back stich this. Essentially many times due to the
> >>> sums
> >>> >>> of
> >>> >>>>>>> money
> >>> >>>>>>>>>> > involved there are disputes, and typically as a
> financial
> >>> >>>>>>>>> institute the
> >>> >>>>>>>>>> > easiest and cleanest way to prove when disputes
> arise is
> >>> to
> >>> >>>>>>>>> present the
> >>> >>>>>>>>>> > actual flow and processes involved in a transaction.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise as Hadoop matures its evident this case is
> >>> >>>>> important, as
> >>> >>>>>>>>> tools
> >>> >>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator
> (cloudera
> >>> led)
> >>> >>>>> are
> >>> >>>>>>>>> evident
> >>> >>>>>>>>>> > also I believe the importance here is very much NOT
> just
> >>> a
> >>> >>>>>>>>> financial issue.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > From a MDM point of view any company wanting to care
> >>> about
> >>> >>>>> Data
> >>> >>>>>>>>> Quality
> >>> >>>>>>>>>> > and Data Governance - Data Lineage is a key piece in
> this
> >>> >>>>> puzzle.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Mirroring,
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As per the KIP in-fact this is exactly what we do re
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> to mirror
> >>> >>>>>>>>>> > a network of clusters between AZ's / Regions. We
> know a
> >>> >>>>>>>>> transaction for a
> >>> >>>>>>>>>> > key will be done within a AZ/Region, as such we
> know the
> >>> >>>>> write
> >>> >>>>>>> to
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>> > would be ordered per key. But we need eventual view
> of
> >>> that
> >>> >>>>>>> across
> >>> >>>>>>>>> in our
> >>> >>>>>>>>>> > other regions/az's. When we have complete AZ or
> Region
> >>> >>>>> failure we
> >>> >>>>>>>>> know
> >>> >>>>>>>>>> > there will be a brief interruption whilst those
> >>> transactions
> >>> >>>>> are
> >>> >>>>>>>>> moved to
> >>> >>>>>>>>>> > another region but we expect after it to continue.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As mentioned having separate Partions to do this
> starts
> >>> to
> >>> >>> get
> >>> >>>>>>>>>> > ugly/complicated for us:
> >>> >>>>>>>>>> > how would I do compaction where a key is in two
> >>> partitions?
> >>> >>>>>>>>>> > How do we balance consumers so where multiple
> partitions
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>> same key
> >>> >>>>>>>>>> > goto the same consumer
> >>> >>>>>>>>>> > What do you do if cluster 1 has 5 partitions but
> cluster
> >>> 20
> >>> >>>>> has
> >>> >>>>>>> 10
> >>> >>>>>>>>> because
> >>> >>>>>>>>>> > its larger kit in our more core DC's, as such key to
> >>> >>> partition
> >>> >>>>>>>>> mappings for
> >>> >>>>>>>>>> > consumers get even more complicated.
> >>> >>>>>>>>>> > What do you do if we add or remove a complete region
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Where as simple mirror will work we just need to
> ensure
> >>> we
> >>> >>>>> don't
> >>> >>>>>>>>> have a
> >>> >>>>>>>>>> > cycle which we can do with clusterId.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > We even have started to look at shortest path mirror
> >>> routing
> >>> >>>>>>> based
> >>> >>>>>>>>> on
> >>> >>>>>>>>>> > clusterId, if we also had the region and az info on
> the
> >>> >>>>>>> originating
> >>> >>>>>>>>>> > message, this we have not implemented but some ideas
> come
> >>> >>> from
> >>> >>>>>>>>> network
> >>> >>>>>>>>>> > routing, and also the dispatcher router in apache
> qpid.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Also we need to have data perimeters e.g. certain
> data
> >>> >>> cannot
> >>> >>>>>>> leave
> >>> >>>>>>>>>> > certain countries borders. We want this all
> automated so
> >>> >>> that
> >>> >>>>> at
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > platform level without having to touch or look at the
> >>> >>> business
> >>> >>>>>>>>> data inside
> >>> >>>>>>>>>> > we can have headers we can put tags into so that we
> can
> >>> >>> ensure
> >>> >>>>>>>>> this doesn't
> >>> >>>>>>>>>> > occur when we mirror. (actually links in to data
> lineage
> >>> /
> >>> >>>>>>> tracing
> >>> >>>>>>>>> as again
> >>> >>>>>>>>>> > we need to tag messages at a platform level) Examples
> >>> are we
> >>> >>>>> are
> >>> >>>>>>>>> not
> >>> >>>>>>>>>> > allowed Private customer details to leave
> Switzerland,
> >>> yet
> >>> >>> we
> >>> >>>>>>> need
> >>> >>>>>>>>> those
> >>> >>>>>>>>>> > systems integrated.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Lastly around mirroring we have a partionKey field,
> as
> >>> the
> >>> >>> key
> >>> >>>>>>>>> used for
> >>> >>>>>>>>>> > portioning logic != compaction key all the time but
> we
> >>> want
> >>> >>> to
> >>> >>>>>>>>> preserve it
> >>> >>>>>>>>>> > for when we mirror so that if source cluster
> partition
> >>> count
> >>> >>>>> !=
> >>> >>>>>>>>> destination
> >>> >>>>>>>>>> > cluster partition count we can honour the same
> >>> partitioning
> >>> >>>>>>> logic.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE End 2 End encryption
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As I believe mentioned just before, the solution you
> >>> mention
> >>> >>>>> just
> >>> >>>>>>>>> doesn't
> >>> >>>>>>>>>> > cut the mustard these days with many regulators. An
> >>> >>> operations
> >>> >>>>>>>>> person with
> >>> >>>>>>>>>> > access to the box should not be able to have access
> to
> >>> the
> >>> >>>>> data.
> >>> >>>>>>>>> Many now
> >>> >>>>>>>>>> > actually impose quite literally the implementation
> >>> expected
> >>> >>>>> being
> >>> >>>>>>>>> end2end
> >>> >>>>>>>>>> > encryption for certain data (Singapore for us is one
> >>> that I
> >>> >>> am
> >>> >>>>>>>>> most aware
> >>> >>>>>>>>>> > of). In fact we're even now needing encrypt the data
> and
> >>> >>> store
> >>> >>>>>>> the
> >>> >>>>>>>>> keys in
> >>> >>>>>>>>>> > HSM modules.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Likewise the performance penalty on encrypting
> >>> decrypting as
> >>> >>>>> you
> >>> >>>>>>>>> produce
> >>> >>>>>>>>>> > over wire, then again encrypt decrypt as the data is
> >>> stored
> >>> >>> on
> >>> >>>>>>> the
> >>> >>>>>>>>> brokers
> >>> >>>>>>>>>> > disks and back again, then again encrypted and
> decrypted
> >>> >>> back
> >>> >>>>>>> over
> >>> >>>>>>>>> the wire
> >>> >>>>>>>>>> > each time for each consumer all adds up, ignoring
> this
> >>> >>>>> doubling
> >>> >>>>>>>>> with mirror
> >>> >>>>>>>>>> > makers etc. simply encrypting the value once on
> write by
> >>> the
> >>> >>>>>>>>> client and
> >>> >>>>>>>>>> > again decrypting on consume by the consumer is far
> more
> >>> >>>>>>>>> performant, but
> >>> >>>>>>>>>> > then the routing and platform meta data needs to be
> >>> separate
> >>> >>>>>>> (thus
> >>> >>>>>>>>> headers)
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Auditing:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Our Auditing needs are:
> >>> >>>>>>>>>> > Producer Id,
> >>> >>>>>>>>>> > Origin Cluster Id that message first produced into
> >>> >>>>>>>>>> > Origin AZ - agreed we can derive this if we have
> cluster
> >>> id,
> >>> >>>>> but
> >>> >>>>>>>>> it makes
> >>> >>>>>>>>>> > resolving this for audit reporting a lot easier.
> >>> >>>>>>>>>> > Origin Region - agreed we can derive this if we have
> >>> cluster
> >>> >>>>> id,
> >>> >>>>>>>>> but it
> >>> >>>>>>>>>> > makes resolving this for audit reporting a lot
> easier.
> >>> >>>>>>>>>> > Unique Message Identification (this is not the same
> as
> >>> >>>>>>> transaction
> >>> >>>>>>>>>> > tracing) - note offset and partition are not the
> same, as
> >>> >>>>> when we
> >>> >>>>>>>>> mirror or
> >>> >>>>>>>>>> > have for what ever system failure duplicate send,
> >>> >>>>>>>>>> > Custom Client wrapper version (where organizations
> have
> >>> to
> >>> >>>>> wrap
> >>> >>>>>>>>> the kafka
> >>> >>>>>>>>>> > client for added features) so we know what version
> of the
> >>> >>>>> wrapper
> >>> >>>>>>>>> is used
> >>> >>>>>>>>>> > Producer IP address (in case of clients being in our
> >>> vm/open
> >>> >>>>>>> stack
> >>> >>>>>>>>> infra
> >>> >>>>>>>>>> > where they can move around, producer id will stay the
> >>> same
> >>> >>> but
> >>> >>>>>>>>> this would
> >>> >>>>>>>>>> > change)
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > RE Once and only once delivery case
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Using the same Message UUID for auditing we can
> achieve
> >>> this
> >>> >>>>>>> quite
> >>> >>>>>>>>> simply.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > As per how some other brokers do this (cough qpid,
> >>> artemis)
> >>> >>>>>>>>> message uuid
> >>> >>>>>>>>>> > are used to dedupe where message is sent and
> produced but
> >>> >>> the
> >>> >>>>>>>>> client didn't
> >>> >>>>>>>>>> > receive the ack, and there for replays the send, by
> >>> having a
> >>> >>>>>>>>> unique message
> >>> >>>>>>>>>> > id per message, this can be filtered out, on
> consumers
> >>> where
> >>> >>>>>>>>> message
> >>> >>>>>>>>>> > delivery may occur twice for what ever reasons a
> message
> >>> >>> uuid
> >>> >>>>> can
> >>> >>>>>>>>> be used
> >>> >>>>>>>>>> > to remove duplicates being deliverd , like wise we
> can do
> >>> >>>>> this in
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > mirrormakers so if we detect a dupe message we can
> avoid
> >>> >>>>>>>>> replicating it.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Cheers
> >>> >>>>>>>>>> > Mike
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
> >>> wrote:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Since this KIP affects message format, wire
> protocol,
> >>> >>>>> apis, I
> >>> >>>>>>>>> think
> >>> >>>>>>>>>> > it's
> >>> >>>>>>>>>> > worth spending a bit more time to nail down the
> >>> concrete
> >>> >>>>> use
> >>> >>>>>>>>> cases. It
> >>> >>>>>>>>>> > would be bad if we add this feature, but when
> start
> >>> >>>>>>>>> implementing it
> >>> >>>>>>>>>> > for say
> >>> >>>>>>>>>> > mirroring, we then realize that header is not the
> >>> best
> >>> >>>>>>>>> approach.
> >>> >>>>>>>>>> > Initially,
> >>> >>>>>>>>>> > I thought I was convinced of the use cases of
> headers
> >>> >>> and
> >>> >>>>> was
> >>> >>>>>>>>> trying to
> >>> >>>>>>>>>> > write down a few use cases to convince others.
> That's
> >>> >>>>> when I
> >>> >>>>>>>>> became
> >>> >>>>>>>>>> > less
> >>> >>>>>>>>>> > certain. For me to be convinced, I just want to
> see
> >>> two
> >>> >>>>>>> strong
> >>> >>>>>>>>> use
> >>> >>>>>>>>>> > cases
> >>> >>>>>>>>>> > (instead of 10 maybe use cases) in the
> third-party
> >>> >>> space.
> >>> >>>>> The
> >>> >>>>>>>>> reason is
> >>> >>>>>>>>>> > that when we discussed the use cases within a
> >>> company,
> >>> >>>>> often
> >>> >>>>>>>>> it ends
> >>> >>>>>>>>>> > with
> >>> >>>>>>>>>> > "we can't force everyone to use this standard
> since
> >>> we
> >>> >>> may
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>> > integrate with third-party tools".
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > At present, I am not sure why headers are useful
> for
> >>> >>>>> things
> >>> >>>>>>>>> like
> >>> >>>>>>>>>> > schemaId
> >>> >>>>>>>>>> > or encryption. In order to do anything useful to
> the
> >>> >>>>> value,
> >>> >>>>>>>>> one needs
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > know the schemaId or how data is encrypted, but
> >>> header
> >>> >>> is
> >>> >>>>>>>>> optional.
> >>> >>>>>>>>>> > But, I
> >>> >>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
> >>> >>> provides
> >>> >>>>>>> more
> >>> >>>>>>>>> details
> >>> >>>>>>>>>> > on
> >>> >>>>>>>>>> > the argument.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I am not very sure header is the best approach
> for
> >>> >>>>> mirroring
> >>> >>>>>>>>> either. If
> >>> >>>>>>>>>> > someone has thought about this more, I'd be
> happy to
> >>> >>> hear.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I can see the data lineage use case. I am just
> not
> >>> sure
> >>> >>>>> how
> >>> >>>>>>>>> widely
> >>> >>>>>>>>>> > applicable this is. If someone familiar with this
> >>> space
> >>> >>>>> can
> >>> >>>>>>>>> justify
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > a significant use case, say in the finance
> industry,
> >>> >>> this
> >>> >>>>>>>>> would be a
> >>> >>>>>>>>>> > strong
> >>> >>>>>>>>>> > use case.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > I can see the auditing use case. I am just not
> sure
> >>> if a
> >>> >>>>>>> native
> >>> >>>>>>>>>> > producer id
> >>> >>>>>>>>>> > solves that problem. If there are additional
> metadata
> >>> >>>>> that's
> >>> >>>>>>>>> worth
> >>> >>>>>>>>>> > collecting but not covered by the producer id,
> that
> >>> >>> would
> >>> >>>>>>> make
> >>> >>>>>>>>> this a
> >>> >>>>>>>>>> > strong use case.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Thanks,
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > Jun
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > > this KIP is about enabling headers, nothing
> more
> >>> >>> nothing
> >>> >>>>>>>>> less - so
> >>> >>>>>>>>>> > no,
> >>> >>>>>>>>>> > > broker-side use of headers is not in the KIP
> scope.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > obviously though, once you have headers
> potential
> >>> use
> >>> >>>>> cases
> >>> >>>>>>>>> could
> >>> >>>>>>>>>> > include
> >>> >>>>>>>>>> > > broker-side header-aware interceptors (which
> would
> >>> be
> >>> >>>>> the
> >>> >>>>>>>>> topic of
> >>> >>>>>>>>>> > other
> >>> >>>>>>>>>> > > future KIPs).
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > a trivially clear use case (to me) would be
> using
> >>> such
> >>> >>>>>>>>> broker-side
> >>> >>>>>>>>>> > > interceptors to enforce compliance with
> >>> organizational
> >>> >>>>>>>>> policies - it
> >>> >>>>>>>>>> > would
> >>> >>>>>>>>>> > > make our SREs lives much easier if instead of
> >>> >>>>> retroactively
> >>> >>>>>>>>>> > discovering
> >>> >>>>>>>>>> > > "rogue" topics/users those messages would have
> been
> >>> >>>>>>> rejected
> >>> >>>>>>>>>> > up-front.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > the kafka broker code is lacking any such
> >>> >>> extensibility
> >>> >>>>>>>>> support
> >>> >>>>>>>>>> > (beyond
> >>> >>>>>>>>>> > > maybe authorizer) which is why these use cases
> were
> >>> >>> left
> >>> >>>>>>> out
> >>> >>>>>>>>> of the
> >>> >>>>>>>>>> > "case
> >>> >>>>>>>>>> > > for headers" doc - broker extensibility is a
> >>> separate
> >>> >>>>>>>>> discussion.
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
> >>> >>>>>>>>> gwen@confluent.io>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > > Woah, I wasn't aware this is something we'll
> do.
> >>> It
> >>> >>>>>>> wasn't
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > KIP,
> >>> >>>>>>>>>> > > > right?
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > I guess we could do it the same way ACLs
> >>> currently
> >>> >>>>> work.
> >>> >>>>>>>>>> > > > I had in mind something that will allow
> admins to
> >>> >>>>> apply
> >>> >>>>>>>>> rules to
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > new create/delete/config topic APIs. So Todd
> can
> >>> >>>>> decide
> >>> >>>>>>> to
> >>> >>>>>>>>> reject
> >>> >>>>>>>>>> > > > "create topic" requests that ask for more
> than 40
> >>> >>>>>>>>> partitions, or
> >>> >>>>>>>>>> > > > require exactly 3 replicas, or no more than
> 50GB
> >>> >>>>>>> partition
> >>> >>>>>>>>> size,
> >>> >>>>>>>>>> > etc.
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are
> planning
> >>> to
> >>> >>>>> apply
> >>> >>>>>>>>> more
> >>> >>>>>>>>>> > rules
> >>> >>>>>>>>>> > > > to requests (and I think we should), we may
> want
> >>> a
> >>> >>> bit
> >>> >>>>>>>>> more generic
> >>> >>>>>>>>>> > > > design around that.
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > wrote:
> >>> >>>>>>>>>> > > > > "wouldn't you be in the business of making
> sure
> >>> >>>>>>> everyone
> >>> >>>>>>>>> uses
> >>> >>>>>>>>>> > them
> >>> >>>>>>>>>> > > > > properly?"
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > > thats where a broker-side plugin would come
> >>> handy
> >>> >>> -
> >>> >>>>> any
> >>> >>>>>>>>> incoming
> >>> >>>>>>>>>> > > message
> >>> >>>>>>>>>> > > > > that does not conform to org policy (read -
> >>> does
> >>> >>> not
> >>> >>>>>>>>> have the
> >>> >>>>>>>>>> > proper
> >>> >>>>>>>>>> > > > > headers) gets thrown out (with an error
> >>> returned
> >>> >>> to
> >>> >>>>>>> user)
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> Palino <
> >>> >>>>>>>>> tpalino@gmail.com>
> >>> >>>>>>>>>> > wrote:
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > >> Come on, I've done at least 2 talks on
> this
> >>> one
> >>> >>> :)
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> Producing counts to a topic is part of
> it, but
> >>> >>>>> that's
> >>> >>>>>>>>> only
> >>> >>>>>>>>>> > part. So
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> count you have 100 messages in topic A.
> When
> >>> you
> >>> >>>>>>> mirror
> >>> >>>>>>>>> topic A
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > > another
> >>> >>>>>>>>>> > > > >> cluster, you have 99 messages. Where was
> your
> >>> >>>>> problem?
> >>> >>>>>>>>> Or
> >>> >>>>>>>>>> > worse, you
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> 100 messages, but one producer duplicated
> >>> >>> messages
> >>> >>>>> and
> >>> >>>>>>>>> another
> >>> >>>>>>>>>> > one
> >>> >>>>>>>>>> > > lost
> >>> >>>>>>>>>> > > > >> messages. You need details about where the
> >>> >>> message
> >>> >>>>>>> came
> >>> >>>>>>>>> from in
> >>> >>>>>>>>>> > order
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> pinpoint problems when they happen. Source
> >>> >>> producer
> >>> >>>>>>>>> info, where
> >>> >>>>>>>>>> > it was
> >>> >>>>>>>>>> > > > >> produced into your infrastructure, and
> when it
> >>> >>> was
> >>> >>>>>>>>> produced.
> >>> >>>>>>>>>> > This
> >>> >>>>>>>>>> > > > requires
> >>> >>>>>>>>>> > > > >> you to add the information to the message.
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> And yes, you still need to maintain your
> >>> clients.
> >>> >>>>> So
> >>> >>>>>>>>> maybe my
> >>> >>>>>>>>>> > original
> >>> >>>>>>>>>> > > > >> example was not the best. My thoughts on
> not
> >>> >>>>> wanting
> >>> >>>>>>> to
> >>> >>>>>>>>> be
> >>> >>>>>>>>>> > responsible
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> message formats stands, because that's
> very
> >>> much
> >>> >>>>>>>>> separate from
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > client.
> >>> >>>>>>>>>> > > > >> As you know, we have our own internal
> client
> >>> >>>>> library
> >>> >>>>>>>>> that can
> >>> >>>>>>>>>> > insert
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> right headers, and right now inserts the
> right
> >>> >>>>> audit
> >>> >>>>>>>>>> > information into
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> message fields. If they exist, and
> assuming
> >>> the
> >>> >>>>>>> message
> >>> >>>>>>>>> is Avro
> >>> >>>>>>>>>> > > encoded.
> >>> >>>>>>>>>> > > > >> What if someone wants to use JSON instead
> for
> >>> a
> >>> >>>>> good
> >>> >>>>>>>>> reason?
> >>> >>>>>>>>>> > What if
> >>> >>>>>>>>>> > > > user X
> >>> >>>>>>>>>> > > > >> wants to encrypt messages, but user Y does
> >>> not?
> >>> >>>>>>>>> Maintaining the
> >>> >>>>>>>>>> > client
> >>> >>>>>>>>>> > > > >> library is still much easier than
> maintaining
> >>> the
> >>> >>>>>>>>> message
> >>> >>>>>>>>>> > formats.
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> -Todd
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> Shapira <
> >>> >>>>>>>>> gwen@confluent.io
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > wrote:
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> > Based on your last sentence, consider me
> >>> >>>>> convinced
> >>> >>>>>>> :)
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > I get why headers are critical for
> Mirroring
> >>> >>> (you
> >>> >>>>>>>>> need tags to
> >>> >>>>>>>>>> > > prevent
> >>> >>>>>>>>>> > > > >> > loops and sometimes to route messages
> to the
> >>> >>>>> correct
> >>> >>>>>>>>>> > destination).
> >>> >>>>>>>>>> > > > >> > But why do you need headers to audit?
> We are
> >>> >>>>>>> auditing
> >>> >>>>>>>>> by
> >>> >>>>>>>>>> > producing
> >>> >>>>>>>>>> > > > >> > counts to a side topic (and I was under
> the
> >>> >>>>>>>>> impression you do
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > >> > same), so we never need to modify the
> >>> message.
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > Another thing - after we added headers,
> >>> >>> wouldn't
> >>> >>>>> you
> >>> >>>>>>>>> be in the
> >>> >>>>>>>>>> > > > >> > business of making sure everyone uses
> them
> >>> >>>>> properly?
> >>> >>>>>>>>> Making
> >>> >>>>>>>>>> > sure
> >>> >>>>>>>>>> > > > >> > everyone includes the right headers you
> >>> need,
> >>> >>> not
> >>> >>>>>>>>> using the
> >>> >>>>>>>>>> > header
> >>> >>>>>>>>>> > > > >> > names you intend to use, etc. I don't
> think
> >>> the
> >>> >>>>>>>>> "policing"
> >>> >>>>>>>>>> > business
> >>> >>>>>>>>>> > > > >> > will ever go away.
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> Palino
> >>> <
> >>> >>>>>>>>>> > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > wrote:
> >>> >>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very
> happy
> >>> >>> with
> >>> >>>>> the
> >>> >>>>>>>>>> > workaround.
> >>> >>>>>>>>>> > > Avro
> >>> >>>>>>>>>> > > > >> means
> >>> >>>>>>>>>> > > > >> > > that I have to be concerned with the
> >>> format
> >>> >>> of
> >>> >>>>> the
> >>> >>>>>>>>> messages
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > order to
> >>> >>>>>>>>>> > > > >> > run
> >>> >>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring,
> >>> etc.).
> >>> >>>>> That
> >>> >>>>>>>>> means
> >>> >>>>>>>>>> > that I
> >>> >>>>>>>>>> > > have
> >>> >>>>>>>>>> > > > to
> >>> >>>>>>>>>> > > > >> > > handle the schemas, and I have to
> enforce
> >>> >>> rules
> >>> >>>>>>>>> about good
> >>> >>>>>>>>>> > > formats.
> >>> >>>>>>>>>> > > > >> This
> >>> >>>>>>>>>> > > > >> > is
> >>> >>>>>>>>>> > > > >> > > not something I want to be in the
> business
> >>> >>> of,
> >>> >>>>>>>>> because I
> >>> >>>>>>>>>> > should be
> >>> >>>>>>>>>> > > > able
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > > run a service infrastructure without
> >>> needing
> >>> >>>>> to be
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > weeds of
> >>> >>>>>>>>>> > > > >> > dealing
> >>> >>>>>>>>>> > > > >> > > with customer data formats.
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > Trust me, a sizable portion of my
> support
> >>> >>> time
> >>> >>>>> is
> >>> >>>>>>>>> spent
> >>> >>>>>>>>>> > dealing
> >>> >>>>>>>>>> > > with
> >>> >>>>>>>>>> > > > >> > schema
> >>> >>>>>>>>>> > > > >> > > issues. I really would like to get
> away
> >>> from
> >>> >>>>> that.
> >>> >>>>>>>>> Maybe
> >>> >>>>>>>>>> > I'd have
> >>> >>>>>>>>>> > > > more
> >>> >>>>>>>>>> > > > >> > time
> >>> >>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > -Todd
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
> >>> Shapira <
> >>> >>>>>>>>>> > gwen@confluent.io>
> >>> >>>>>>>>>> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
> >>> >>>>> workarounds
> >>> >>>>>>>>> (Avro
> >>> >>>>>>>>>> > container
> >>> >>>>>>>>>> > > > >> > >> format), so I'm not too excited
> about the
> >>> >>>>> extra
> >>> >>>>>>>>> work
> >>> >>>>>>>>>> > required to
> >>> >>>>>>>>>> > > do
> >>> >>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't
> >>> mind it
> >>> >>>>> if
> >>> >>>>>>>>> you do
> >>> >>>>>>>>>> > it...
> >>> >>>>>>>>>> > > > >> > >> I think the Apache convention for
> "good
> >>> >>> idea,
> >>> >>>>> but
> >>> >>>>>>>>> not
> >>> >>>>>>>>>> > willing to
> >>> >>>>>>>>>> > > > put
> >>> >>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway,
> >>> that's
> >>> >>>>> what
> >>> >>>>>>> I
> >>> >>>>>>>>> was
> >>> >>>>>>>>>> > trying to
> >>> >>>>>>>>>> > > > >> > >> convey :)
> >>> >>>>>>>>>> > > > >> > >>
> >>> >>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
> >>> Palino
> >>> >>> <
> >>> >>>>>>>>>> > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> > Well I guess my question for you,
> >>> then, is
> >>> >>>>> what
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > holding you
> >>> >>>>>>>>>> > > > back
> >>> >>>>>>>>>> > > > >> > from
> >>> >>>>>>>>>> > > > >> > >> > full support for headers? What's
> the
> >>> bit
> >>> >>>>> that
> >>> >>>>>>>>> you're
> >>> >>>>>>>>>> > missing
> >>> >>>>>>>>>> > > that
> >>> >>>>>>>>>> > > > >> has
> >>> >>>>>>>>>> > > > >> > you
> >>> >>>>>>>>>> > > > >> > >> > under a full +1?
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> > -Todd
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> Gwen
> >>> >>>>> Shapira <
> >>> >>>>>>>>>> > > gwen@confluent.io>
> >>> >>>>>>>>>> > > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> I know why people who support
> headers
> >>> >>>>> support
> >>> >>>>>>>>> them, and
> >>> >>>>>>>>>> > I've
> >>> >>>>>>>>>> > > > seen
> >>> >>>>>>>>>> > > > >> > what
> >>> >>>>>>>>>> > > > >> > >> >> the discussion is like.
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> This is why I'm asking people who
> are
> >>> >>>>> against
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > > > (especially
> >>> >>>>>>>>>> > > > >> > >> >> committers) what will make them
> change
> >>> >>>>> their
> >>> >>>>>>>>> mind - so
> >>> >>>>>>>>>> > we can
> >>> >>>>>>>>>> > > > get
> >>> >>>>>>>>>> > > > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> part over one way or another.
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at
> >>> Radai,
> >>> >>>>> Jun
> >>> >>>>>>>>> or you
> >>> >>>>>>>>>> > (Todd)...
> >>> >>>>>>>>>> > > > I am
> >>> >>>>>>>>>> > > > >> > >> >> just looking for something
> concrete we
> >>> >>> can
> >>> >>>>> do
> >>> >>>>>>>>> to move
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > >> discussion
> >>> >>>>>>>>>> > > > >> > >> >> along to the yummy design details
> >>> (which
> >>> >>> is
> >>> >>>>>>> the
> >>> >>>>>>>>>> > argument I
> >>> >>>>>>>>>> > > > really
> >>> >>>>>>>>>> > > > >> am
> >>> >>>>>>>>>> > > > >> > >> >> looking forward to).
> >>> >>>>>>>>>> > > > >> > >> >>
> >>> >>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM,
> Todd
> >>> >>>>> Palino <
> >>> >>>>>>>>>> > > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even
> >>> though
> >>> >>>>> I'm
> >>> >>>>>>>>> not a
> >>> >>>>>>>>>> > > > committer)...
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > I have always been a strong
> >>> supporter
> >>> >>> of
> >>> >>>>>>>>> introducing
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > concept
> >>> >>>>>>>>>> > > > >> > of an
> >>> >>>>>>>>>> > > > >> > >> >> > envelope to messages, which
> headers
> >>> >>>>>>>>> accomplishes. The
> >>> >>>>>>>>>> > > message
> >>> >>>>>>>>>> > > > key
> >>> >>>>>>>>>> > > > >> > is
> >>> >>>>>>>>>> > > > >> > >> >> > already an example of a piece of
> >>> >>> envelope
> >>> >>>>>>>>>> > information. By
> >>> >>>>>>>>>> > > > >> > providing a
> >>> >>>>>>>>>> > > > >> > >> >> means
> >>> >>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself,
> and
> >>> not
> >>> >>>>>>>>> relying on
> >>> >>>>>>>>>> > use-case
> >>> >>>>>>>>>> > > > >> > specific
> >>> >>>>>>>>>> > > > >> > >> >> > implementations, you make it
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>>> > components to
> >>> >>>>>>>>>> > > > >> > >> interoperate.
> >>> >>>>>>>>>> > > > >> > >> >> It
> >>> >>>>>>>>>> > > > >> > >> >> > simplifies development of all
> these
> >>> >>>>> things
> >>> >>>>>>>>> (message
> >>> >>>>>>>>>> > routing,
> >>> >>>>>>>>>> > > > >> > auditing,
> >>> >>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each
> one
> >>> does
> >>> >>>>> not
> >>> >>>>>>>>> have to
> >>> >>>>>>>>>> > reinvent
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> wheel.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > It also makes it much easier
> from a
> >>> >>>>> client
> >>> >>>>>>>>> point of
> >>> >>>>>>>>>> > view if
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> headers
> >>> >>>>>>>>>> > > > >> > >> >> are
> >>> >>>>>>>>>> > > > >> > >> >> > defined as part of the protocol
> >>> and/or
> >>> >>>>>>>>> message format
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > general
> >>> >>>>>>>>>> > > > >> > >> because
> >>> >>>>>>>>>> > > > >> > >> >> > you can easily produce and
> consume
> >>> >>>>> messages
> >>> >>>>>>>>> without
> >>> >>>>>>>>>> > having
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> take
> >>> >>>>>>>>>> > > > >> > >> into
> >>> >>>>>>>>>> > > > >> > >> >> > account specific cases. For
> >>> example, I
> >>> >>>>> want
> >>> >>>>>>>>> to route
> >>> >>>>>>>>>> > > messages,
> >>> >>>>>>>>>> > > > >> but
> >>> >>>>>>>>>> > > > >> > >> >> client A
> >>> >>>>>>>>>> > > > >> > >> >> > doesn't support the way audit
> >>> >>> implemented
> >>> >>>>>>>>> headers, and
> >>> >>>>>>>>>> > > client
> >>> >>>>>>>>>> > > > B
> >>> >>>>>>>>>> > > > >> > >> doesn't
> >>> >>>>>>>>>> > > > >> > >> >> > support the way encryption or
> >>> routing
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>> > headers,
> >>> >>>>>>>>>> > > so
> >>> >>>>>>>>>> > > > now
> >>> >>>>>>>>>> > > > >> > my
> >>> >>>>>>>>>> > > > >> > >> >> > application has to create some
> >>> really
> >>> >>>>>>> fragile
> >>> >>>>>>>>> (my
> >>> >>>>>>>>>> > > autocorrect
> >>> >>>>>>>>>> > > > >> just
> >>> >>>>>>>>>> > > > >> > >> tried
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> > make that "tragic", which is
> >>> probably
> >>> >>>>>>>>> appropriate
> >>> >>>>>>>>>> > too) code
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> > strip
> >>> >>>>>>>>>> > > > >> > >> >> > everything off, rather than just
> >>> >>>>> consuming
> >>> >>>>>>> the
> >>> >>>>>>>>>> > messages,
> >>> >>>>>>>>>> > > > picking
> >>> >>>>>>>>>> > > > >> > out
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> 1
> >>> >>>>>>>>>> > > > >> > >> >> > or 2 headers it's interested
> in, and
> >>> >>>>>>>>> performing its
> >>> >>>>>>>>>> > > function.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has
> been
> >>> >>> going
> >>> >>>>> on
> >>> >>>>>>>>> for a
> >>> >>>>>>>>>> > long time,
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > it's
> >>> >>>>>>>>>> > > > >> > >> >> > always "Oh, you came up with 2
> use
> >>> >>> cases,
> >>> >>>>>>> and
> >>> >>>>>>>>> yeah,
> >>> >>>>>>>>>> > those
> >>> >>>>>>>>>> > > use
> >>> >>>>>>>>>> > > > >> cases
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> > real things that someone would
> want
> >>> to
> >>> >>>>> do.
> >>> >>>>>>>>> Here's an
> >>> >>>>>>>>>> > > alternate
> >>> >>>>>>>>>> > > > >> way
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> >> > implement them so let's not do
> >>> >>> headers."
> >>> >>>>> If
> >>> >>>>>>>>> we have a
> >>> >>>>>>>>>> > few
> >>> >>>>>>>>>> > > use
> >>> >>>>>>>>>> > > > >> cases
> >>> >>>>>>>>>> > > > >> > >> that
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> > actually came up with, you can
> be
> >>> sure
> >>> >>>>> that
> >>> >>>>>>>>> over the
> >>> >>>>>>>>>> > next
> >>> >>>>>>>>>> > > year
> >>> >>>>>>>>>> > > > >> > >> there's a
> >>> >>>>>>>>>> > > > >> > >> >> > dozen others that we didn't
> think of
> >>> >>> that
> >>> >>>>>>>>> someone
> >>> >>>>>>>>>> > would like
> >>> >>>>>>>>>> > > > to
> >>> >>>>>>>>>> > > > >> > do. I
> >>> >>>>>>>>>> > > > >> > >> >> > really think it's time to stop
> >>> >>> rehashing
> >>> >>>>>>> this
> >>> >>>>>>>>>> > discussion and
> >>> >>>>>>>>>> > > > >> > instead
> >>> >>>>>>>>>> > > > >> > >> >> focus
> >>> >>>>>>>>>> > > > >> > >> >> > on a workable standard that we
> can
> >>> >>> adopt.
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > -Todd
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM,
> Todd
> >>> >>>>> Palino
> >>> >>>>>>> <
> >>> >>>>>>>>>> > > > tpalino@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >> C. per message encryption
> >>> >>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach
> is
> >>> that
> >>> >>>>> this
> >>> >>>>>>>>>> > significantly
> >>> >>>>>>>>>> > > > reduce
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression,
> >>> which
> >>> >>>>>>> happens
> >>> >>>>>>>>> on a
> >>> >>>>>>>>>> > set of
> >>> >>>>>>>>>> > > > >> > serialized
> >>> >>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to
> >>> enable
> >>> >>>>> SSL
> >>> >>>>>>>>> for wire
> >>> >>>>>>>>>> > > > encryption
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> rely
> >>> >>>>>>>>>> > > > >> > >> >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS)
> >>> for at
> >>> >>>>> rest
> >>> >>>>>>>>>> > encryption.
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient.
> While
> >>> >>> this
> >>> >>>>>>> does
> >>> >>>>>>>>> cover
> >>> >>>>>>>>>> > the case
> >>> >>>>>>>>>> > > > of
> >>> >>>>>>>>>> > > > >> > >> removing
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >> drive from the system, it will
> not
> >>> >>>>> satisfy
> >>> >>>>>>>>> most
> >>> >>>>>>>>>> > compliance
> >>> >>>>>>>>>> > > > >> > >> requirements
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever
> has
> >>> >>>>> access to
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > broker
> >>> >>>>>>>>>> > > itself
> >>> >>>>>>>>>> > > > >> > still
> >>> >>>>>>>>>> > > > >> > >> has
> >>> >>>>>>>>>> > > > >> > >> >> >> access to the unencrypted
> data. For
> >>> >>>>>>>>> end-to-end
> >>> >>>>>>>>>> > encryption
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> > need to
> >>> >>>>>>>>>> > > > >> > >> >> >> encrypt at the producer,
> before it
> >>> >>>>> enters
> >>> >>>>>>> the
> >>> >>>>>>>>>> > system, and
> >>> >>>>>>>>>> > > > >> decrypt
> >>> >>>>>>>>>> > > > >> > at
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the
> >>> system.
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> -Todd
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM,
> >>> radai
> >>> >>> <
> >>> >>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> another big plus of headers
> in the
> >>> >>>>>>> protocol
> >>> >>>>>>>>> is that
> >>> >>>>>>>>>> > it
> >>> >>>>>>>>>> > > would
> >>> >>>>>>>>>> > > > >> > enable
> >>> >>>>>>>>>> > > > >> > >> >> rapid
> >>> >>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of
> core
> >>> >>>>> kafka
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>> > reduce
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> number of
> >>> >>>>>>>>>> > > > >> > >> >> >>> future wire format changes
> >>> required.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a
> KIP
> >>> >>>>>>> represents
> >>> >>>>>>>>> use
> >>> >>>>>>>>>> > cases that
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> > not
> >>> >>>>>>>>>> > > > >> > >> 100%
> >>> >>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and
> some of
> >>> >>> them
> >>> >>>>>>>>> require
> >>> >>>>>>>>>> > rather
> >>> >>>>>>>>>> > > > invasive
> >>> >>>>>>>>>> > > > >> > wire
> >>> >>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a
> good
> >>> >>> recent
> >>> >>>>>>>>> example of
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > > > >> > kip-98.
> >>> >>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is
> expected
> >>> to
> >>> >>> be
> >>> >>>>> a
> >>> >>>>>>>>> very small
> >>> >>>>>>>>>> > > > fraction of
> >>> >>>>>>>>>> > > > >> > >> total
> >>> >>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes
> are
> >>> >>>>> invasive.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> every such wire format change
> >>> >>>>> translates
> >>> >>>>>>>>> into
> >>> >>>>>>>>>> > painful and
> >>> >>>>>>>>>> > > > slow
> >>> >>>>>>>>>> > > > >> > >> >> adoption of
> >>> >>>>>>>>>> > > > >> > >> >> >>> new versions.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
> >>> >>>>> currently
> >>> >>>>>>> in
> >>> >>>>>>>>> KIPs
> >>> >>>>>>>>>> > could be
> >>> >>>>>>>>>> > > > "spun
> >>> >>>>>>>>>> > > > >> > out"
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
> >>> >>>>> transmitting
> >>> >>>>>>>>> data over
> >>> >>>>>>>>>> > > > headers.
> >>> >>>>>>>>>> > > > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> keep the core wire format
> >>> stable(r),
> >>> >>>>> core
> >>> >>>>>>>>> codebase
> >>> >>>>>>>>>> > > smaller,
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > >> avoid
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats
> sometimes
> >>> >>>>> required
> >>> >>>>>>>>> to prove
> >>> >>>>>>>>>> > a
> >>> >>>>>>>>>> > > > certain
> >>> >>>>>>>>>> > > > >> > >> feature
> >>> >>>>>>>>>> > > > >> > >> >> is
> >>> >>>>>>>>>> > > > >> > >> >> >>> useful enough for a
> wide-enough
> >>> >>>>> audience
> >>> >>>>>>> to
> >>> >>>>>>>>> warrant
> >>> >>>>>>>>>> > a wire
> >>> >>>>>>>>>> > > > >> format
> >>> >>>>>>>>>> > > > >> > >> >> change
> >>> >>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
> beyond
> >>> >>>>> "mere"
> >>> >>>>>>>>> wire format
> >>> >>>>>>>>>> > > changes
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > im
> >>> >>>>>>>>>> > > > >> > >> not
> >>> >>>>>>>>>> > > > >> > >> >> >>> saying it could have been
> >>> completely
> >>> >>>>> done
> >>> >>>>>>>>> with
> >>> >>>>>>>>>> > headers,
> >>> >>>>>>>>>> > > but
> >>> >>>>>>>>>> > > > >> > >> >> exactly-once
> >>> >>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20
> AM,
> >>> Gwen
> >>> >>>>>>>>> Shapira <
> >>> >>>>>>>>>> > > > >> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> 10:24 AM,
> >>> >>>>> radai <
> >>> >>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
> >>> >>>>> organization,
> >>> >>>>>>>>> one could
> >>> >>>>>>>>>> > > always
> >>> >>>>>>>>>> > > > use
> >>> >>>>>>>>>> > > > >> > >> other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers"
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
> >>> >>>>>>> traditionally
> >>> >>>>>>>>> done
> >>> >>>>>>>>>> > but there
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> > now
> >>> >>>>>>>>>> > > > >> > >> >> cases
> >>> >>>>>>>>>> > > > >> > >> >> >>> > (read
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is
> not
> >>> >>>>>>> acceptable.
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > makes
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > >> useful
> >>> >>>>>>>>>> > > > >> > >> >> even
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for
> cases
> >>> >>> where
> >>> >>>>>>>>>> > > > one-container-fits-all
> >>> >>>>>>>>>> > > > >> > cannot
> >>> >>>>>>>>>> > > > >> > >> >> >>> apply.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use
> >>> cases
> >>> >>>>>>> listed,
> >>> >>>>>>>>> i dont
> >>> >>>>>>>>>> > want
> >>> >>>>>>>>>> > > > this to
> >>> >>>>>>>>>> > > > >> > >> devolve
> >>> >>>>>>>>>> > > > >> > >> >> >>> to a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular
> use
> >>> >>> cases
> >>> >>>>> - i
> >>> >>>>>>>>> think its
> >>> >>>>>>>>>> > > enough
> >>> >>>>>>>>>> > > > >> that
> >>> >>>>>>>>>> > > > >> > >> some
> >>> >>>>>>>>>> > > > >> > >> >> of
> >>> >>>>>>>>>> > > > >> > >> >> >>> them
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I think a main point of
> >>> contention
> >>> >>> is
> >>> >>>>>>>>> that: We
> >>> >>>>>>>>>> > > identified
> >>> >>>>>>>>>> > > > few
> >>> >>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are
> >>> useful,
> >>> >>>>> do
> >>> >>>>>>> we
> >>> >>>>>>>>> want
> >>> >>>>>>>>>> > Kafka to
> >>> >>>>>>>>>> > > > be a
> >>> >>>>>>>>>> > > > >> > >> system
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that supports those
> use-cases?
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
> >>> >>>>> record-level
> >>> >>>>>>>>> lineage
> >>> >>>>>>>>>> > is
> >>> >>>>>>>>>> > > though
> >>> >>>>>>>>>> > > > >> > since
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > overhead could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > be significant."
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record
> >>> level
> >>> >>>>>>>>> lineage. I
> >>> >>>>>>>>>> > don't
> >>> >>>>>>>>>> > > think
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > was
> >>> >>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think
> it
> >>> is
> >>> >>>>> safe
> >>> >>>>>>> to
> >>> >>>>>>>>> assume
> >>> >>>>>>>>>> > that
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > NSA
> >>> >>>>>>>>>> > > > >> > >> needed
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also
> know
> >>> >>> that
> >>> >>>>>>>>> certain
> >>> >>>>>>>>>> > financial
> >>> >>>>>>>>>> > > > >> > institutes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
> >>> >>> records
> >>> >>>>> at
> >>> >>>>>>> a
> >>> >>>>>>>>> record
> >>> >>>>>>>>>> > level
> >>> >>>>>>>>>> > > and
> >>> >>>>>>>>>> > > > >> > there
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > federal regulations that
> >>> absolutely
> >>> >>>>>>>>> require
> >>> >>>>>>>>>> > this. They
> >>> >>>>>>>>>> > > > also
> >>> >>>>>>>>>> > > > >> > need
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
> >>> >>>>> "touches"
> >>> >>>>>>> the
> >>> >>>>>>>>>> > messages and
> >>> >>>>>>>>>> > > > >> either
> >>> >>>>>>>>>> > > > >> > >> reads
> >>> >>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't
> have
> >>> >>>>>>> possibly
> >>> >>>>>>>>>> > modified the
> >>> >>>>>>>>>> > > > >> payload
> >>> >>>>>>>>>> > > > >> > >> >> itself.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > They use record level
> >>> encryption to
> >>> >>>>> do
> >>> >>>>>>>>> that -
> >>> >>>>>>>>>> > apps can
> >>> >>>>>>>>>> > > > read
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers
> but
> >>> >>> can't
> >>> >>>>>>>>> touch the
> >>> >>>>>>>>>> > payload.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those
> are
> >>> >>> corner
> >>> >>>>>>>>> cases and
> >>> >>>>>>>>>> > not worth
> >>> >>>>>>>>>> > > > >> adding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they
> >>> should
> >>> >>>>> use a
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> > > pubsub
> >>> >>>>>>>>>> > > > >> > message
> >>> >>>>>>>>>> > > > >> > >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the
> other
> >>> 1000
> >>> >>>>> that
> >>> >>>>>>>>> cater
> >>> >>>>>>>>>> > > > specifically
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > financial industry).
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a
> catch
> >>> 22:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific
> >>> use-case,
> >>> >>>>>>>>> someone can
> >>> >>>>>>>>>> > always
> >>> >>>>>>>>>> > > say
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > isn't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > interesting enough for
> Kafka.
> >>> If we
> >>> >>>>>>>>> discuss more
> >>> >>>>>>>>>> > general
> >>> >>>>>>>>>> > > > >> > trends,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we
> are not
> >>> >>> sure
> >>> >>>>>>> any
> >>> >>>>>>>>> of them
> >>> >>>>>>>>>> > really
> >>> >>>>>>>>>> > > > >> needs
> >>> >>>>>>>>>> > > > >> > >> >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just
> hand
> >>> >>>>> waving
> >>> >>>>>>>>> and not
> >>> >>>>>>>>>> > > > interesting.".
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I think discussing
> use-cases in
> >>> >>>>>>> specifics
> >>> >>>>>>>>> is super
> >>> >>>>>>>>>> > > > important
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> decide
> >>> >>>>>>>>>> > > > >> > >> >> >>> > implementation details for
> >>> headers
> >>> >>>>> (my
> >>> >>>>>>>>> use-cases
> >>> >>>>>>>>>> > lean
> >>> >>>>>>>>>> > > > toward
> >>> >>>>>>>>>> > > > >> > >> >> numerical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and
> object
> >>> >>>>> values,
> >>> >>>>>>>>> others
> >>> >>>>>>>>>> > differ),
> >>> >>>>>>>>>> > > > but I
> >>> >>>>>>>>>> > > > >> > >> think
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > need to answer the general
> "Are
> >>> we
> >>> >>>>> going
> >>> >>>>>>>>> to have
> >>> >>>>>>>>>> > > headers"
> >>> >>>>>>>>>> > > > >> > question
> >>> >>>>>>>>>> > > > >> > >> >> >>> > first.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the
> other
> >>> >>>>>>>>> committers in the
> >>> >>>>>>>>>> > > > discussion:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > What would it take to
> convince
> >>> you
> >>> >>>>> that
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > in Kafka
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> a
> >>> >>>>>>>>>> > > > >> > >> good
> >>> >>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can
> move
> >>> >>> ahead
> >>> >>>>>>> and
> >>> >>>>>>>>> try to
> >>> >>>>>>>>>> > agree
> >>> >>>>>>>>>> > > on
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> details?
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving
> the
> >>> goal
> >>> >>>>>>> posts
> >>> >>>>>>>>> and
> >>> >>>>>>>>>> > this is
> >>> >>>>>>>>>> > > > truly
> >>> >>>>>>>>>> > > > >> > >> >> exhausting.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly
> support
> >>> >>>>> adding
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > to Kafka
> >>> >>>>>>>>>> > > > >> > (+0.5?).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > The community can continue
> to
> >>> find
> >>> >>>>>>>>> workarounds to
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > issue
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > are some benefits to
> keeping the
> >>> >>>>> message
> >>> >>>>>>>>> format
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > > clients
> >>> >>>>>>>>>> > > > >> > >> simpler.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of
> >>> headers
> >>> >>>>> to
> >>> >>>>>>>>> many
> >>> >>>>>>>>>> > use-cases
> >>> >>>>>>>>>> > > and
> >>> >>>>>>>>>> > > > if
> >>> >>>>>>>>>> > > > >> we
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> > find a good and generally
> useful
> >>> >>> way
> >>> >>>>> to
> >>> >>>>>>>>> add it to
> >>> >>>>>>>>>> > Kafka,
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > will
> >>> >>>>>>>>>> > > > >> > >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for
> many -
> >>> >>> worthy
> >>> >>>>>>>>> goal in my
> >>> >>>>>>>>>> > eyes.
> >>> >>>>>>>>>> > > > >> > >> >> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible,
> but:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> polyglot
> >>> >>>>>>>>>> > topics.
> >>> >>>>>>>>>> > > > >> > >> especially if
> >>> >>>>>>>>>> > > > >> > >> >> >>> kafka
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
> >>> >>> something
> >>> >>>>>>> else.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would
> make
> >>> it
> >>> >>>>>>> harder
> >>> >>>>>>>>> to write
> >>> >>>>>>>>>> > > > portable
> >>> >>>>>>>>>> > > > >> > >> consumer
> >>> >>>>>>>>>> > > > >> > >> >> >>> code.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would
> mess
> >>> >>> with
> >>> >>>>>>>>> locality of
> >>> >>>>>>>>>> > > > consumption
> >>> >>>>>>>>>> > > > >> > >> >> >>> guarantees.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
> >>> >>>>>>>>> lineage/metadata is
> >>> >>>>>>>>>> > > > >> > >> billing/chargeback.
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough
> to
> >>> >>> simply
> >>> >>>>>>>>> record the
> >>> >>>>>>>>>> > point
> >>> >>>>>>>>>> > > of
> >>> >>>>>>>>>> > > > >> > origin,
> >>> >>>>>>>>>> > > > >> > >> but
> >>> >>>>>>>>>> > > > >> > >> >> >>> every
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think
> mirror
> >>> >>>>> maker)
> >>> >>>>>>>>> must also
> >>> >>>>>>>>>> > add a
> >>> >>>>>>>>>> > > > >> record
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> >> form a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing
> on
> >>> top
> >>> >>> of
> >>> >>>>>>>>> kafka - i
> >>> >>>>>>>>>> > know
> >>> >>>>>>>>>> > > samza
> >>> >>>>>>>>>> > > > >> has a
> >>> >>>>>>>>>> > > > >> > >> >> metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > map
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
> >>> >>>>> addition to
> >>> >>>>>>>>> user
> >>> >>>>>>>>>> > values.
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > perfect
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at
> 6:50
> >>> PM,
> >>> >>>>> Jun
> >>> >>>>>>>>> Rao <
> >>> >>>>>>>>>> > > > jun@confluent.io
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the
> first
> >>> two
> >>> >>>>>>>>> questions, it
> >>> >>>>>>>>>> > would
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > >> > helpful
> >>> >>>>>>>>>> > > > >> > >> >> if we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong
> use
> >>> cases
> >>> >>>>> for
> >>> >>>>>>>>> headers
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > space
> >>> >>>>>>>>>> > > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases
> >>> within an
> >>> >>>>>>>>> organization,
> >>> >>>>>>>>>> > one
> >>> >>>>>>>>>> > > > could
> >>> >>>>>>>>>> > > > >> > always
> >>> >>>>>>>>>> > > > >> > >> >> use
> >>> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as
> >>> company-wise
> >>> >>>>>>>>> containers to
> >>> >>>>>>>>>> > get
> >>> >>>>>>>>>> > > > around
> >>> >>>>>>>>>> > > > >> w/o
> >>> >>>>>>>>>> > > > >> > >> >> >>> headers. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> went through the use
> cases in
> >>> >>> the
> >>> >>>>> KIP
> >>> >>>>>>>>> and in
> >>> >>>>>>>>>> > Radai's
> >>> >>>>>>>>>> > > > wiki
> >>> >>>>>>>>>> > > > >> (
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> https://cwiki.apache.org/confl
> >>> >>>>>>>>>> > uence/display/KAFKA/A+
> >>> >>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> ).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> The following are the
> ones
> >>> that
> >>> >>>>> that
> >>> >>>>>>> I
> >>> >>>>>>>>>> > understand and
> >>> >>>>>>>>>> > > > >> could
> >>> >>>>>>>>>> > > > >> > be
> >>> >>>>>>>>>> > > > >> > >> in
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case
> >>> category.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
> >>> >>>>>>> content-type
> >>> >>>>>>>>> should
> >>> >>>>>>>>>> > be set
> >>> >>>>>>>>>> > > at
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> topic
> >>> >>>>>>>>>> > > > >> > >> >> >>> level.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing
> messages
> >>> with
> >>> >>>>>>>>> different
> >>> >>>>>>>>>> > content
> >>> >>>>>>>>>> > > > types
> >>> >>>>>>>>>> > > > >> > >> should be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
> >>> >>> useless
> >>> >>>>>>>>> without
> >>> >>>>>>>>>> > schema id,
> >>> >>>>>>>>>> > > it
> >>> >>>>>>>>>> > > > >> > seems
> >>> >>>>>>>>>> > > > >> > >> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > storing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together
> with
> >>> >>>>>>> serialized
> >>> >>>>>>>>> bytes
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > value
> >>> >>>>>>>>>> > > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> better?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this
> >>> approach is
> >>> >>>>> that
> >>> >>>>>>>>> this
> >>> >>>>>>>>>> > > > significantly
> >>> >>>>>>>>>> > > > >> > reduce
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of
> compression,
> >>> >>>>> which
> >>> >>>>>>>>> happens on
> >>> >>>>>>>>>> > a set
> >>> >>>>>>>>>> > > of
> >>> >>>>>>>>>> > > > >> > >> serialized
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative
> is
> >>> to
> >>> >>>>> enable
> >>> >>>>>>>>> SSL for
> >>> >>>>>>>>>> > wire
> >>> >>>>>>>>>> > > > >> > encryption
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> rely
> >>> >>>>>>>>>> > > > >> > >> >> >>> > on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g.
> >>> LUKS)
> >>> >>>>> for at
> >>> >>>>>>>>> rest
> >>> >>>>>>>>>> > > encryption.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for
> mirroring
> >>> >>> across
> >>> >>>>>>>>> Kafka
> >>> >>>>>>>>>> > clusters
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> This is actually
> interesting.
> >>> >>>>> Today,
> >>> >>>>>>>>> to avoid
> >>> >>>>>>>>>> > > > introducing
> >>> >>>>>>>>>> > > > >> > >> cycles
> >>> >>>>>>>>>> > > > >> > >> >> when
> >>> >>>>>>>>>> > > > >> > >> >> >>> > doing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data
> >>> centers,
> >>> >>> one
> >>> >>>>>>>>> would
> >>> >>>>>>>>>> > either have
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> set
> >>> >>>>>>>>>> > > > >> > up
> >>> >>>>>>>>>> > > > >> > >> two
> >>> >>>>>>>>>> > > > >> > >> >> >>> Kafka
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
> >>> >>>>> aggregate)
> >>> >>>>>>>>> per data
> >>> >>>>>>>>>> > center
> >>> >>>>>>>>>> > > or
> >>> >>>>>>>>>> > > > >> > rename
> >>> >>>>>>>>>> > > > >> > >> >> topics.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
> >>> headers,
> >>> >>>>> the
> >>> >>>>>>>>> producer
> >>> >>>>>>>>>> > could
> >>> >>>>>>>>>> > > tag
> >>> >>>>>>>>>> > > > >> each
> >>> >>>>>>>>>> > > > >> > >> >> message
> >>> >>>>>>>>>> > > > >> > >> >> >>> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID
> in
> >>> the
> >>> >>>>>>> header.
> >>> >>>>>>>>>> > MirrorMaker
> >>> >>>>>>>>>> > > > could
> >>> >>>>>>>>>> > > > >> > then
> >>> >>>>>>>>>> > > > >> > >> >> avoid
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a
> >>> cluster
> >>> >>> if
> >>> >>>>>>>>> they are
> >>> >>>>>>>>>> > tagged
> >>> >>>>>>>>>> > > with
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> same
> >>> >>>>>>>>>> > > > >> > >> >> >>> cluster
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> id.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative
> >>> approach
> >>> >>>>> is
> >>> >>>>>>> to
> >>> >>>>>>>>>> > introduce sth
> >>> >>>>>>>>>> > > > like
> >>> >>>>>>>>>> > > > >> > >> >> >>> hierarchical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages
> from
> >>> >>>>>>> different
> >>> >>>>>>>>>> > clusters in
> >>> >>>>>>>>>> > > > >> > different
> >>> >>>>>>>>>> > > > >> > >> >> >>> partitions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic.
> This
> >>> >>>>> approach
> >>> >>>>>>>>> avoids
> >>> >>>>>>>>>> > filtering
> >>> >>>>>>>>>> > > > out
> >>> >>>>>>>>>> > > > >> > >> unneeded
> >>> >>>>>>>>>> > > > >> > >> >> >>> data
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving
> >>> easier
> >>> >>> to
> >>> >>>>>>>>> support. It
> >>> >>>>>>>>>> > may
> >>> >>>>>>>>>> > > make
> >>> >>>>>>>>>> > > > >> > >> compaction
> >>> >>>>>>>>>> > > > >> > >> >> >>> > trickier
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> though since the same
> key may
> >>> >>>>> show up
> >>> >>>>>>>>> in
> >>> >>>>>>>>>> > different
> >>> >>>>>>>>>> > > > >> > partitions.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source
> >>> connector
> >>> >>>>> could
> >>> >>>>>>>>> store in
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > message
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
> >>> >>> record.
> >>> >>>>>>>>> Similarly,
> >>> >>>>>>>>>> > if a
> >>> >>>>>>>>>> > > > stream
> >>> >>>>>>>>>> > > > >> job
> >>> >>>>>>>>>> > > > >> > >> >> >>> transforms
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to
> >>> topic
> >>> >>> B,
> >>> >>>>> the
> >>> >>>>>>>>> library
> >>> >>>>>>>>>> > could
> >>> >>>>>>>>>> > > > >> include
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> source
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each
> of the
> >>> >>>>>>>>> transformed
> >>> >>>>>>>>>> > message in
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> header.
> >>> >>>>>>>>>> > > > >> > >> >> Not
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sure
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful
> >>> record-level
> >>> >>>>>>> lineage
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > though
> >>> >>>>>>>>>> > > since
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> overhead
> >>> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
> >>> >>>>>>>>> clientId/host/user in
> >>> >>>>>>>>>> > the
> >>> >>>>>>>>>> > > > header
> >>> >>>>>>>>>> > > > >> in
> >>> >>>>>>>>>> > > > >> > >> each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These
> metadata
> >>> are
> >>> >>>>>>>>> really at the
> >>> >>>>>>>>>> > > producer
> >>> >>>>>>>>>> > > > >> > level
> >>> >>>>>>>>>> > > > >> > >> >> though.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > So, a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to
> only
> >>> >>>>>>> include a
> >>> >>>>>>>>>> > "producerId"
> >>> >>>>>>>>>> > > > per
> >>> >>>>>>>>>> > > > >> > >> message
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > send
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producerId ->
> metadata
> >>> >>> mapping
> >>> >>>>>>>>>> > independently.
> >>> >>>>>>>>>> > > > KIP-98
> >>> >>>>>>>>>> > > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> actually
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such
> a
> >>> >>>>> producerId
> >>> >>>>>>>>> natively
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > >> > message.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure
> that
> >>> I
> >>> >>> am
> >>> >>>>>>> fully
> >>> >>>>>>>>>> > convinced of
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > strong
> >>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
> >>> >>> Perhaps
> >>> >>>>> we
> >>> >>>>>>>>> could
> >>> >>>>>>>>>> > discuss a
> >>> >>>>>>>>>> > > > bit
> >>> >>>>>>>>>> > > > >> > more
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > one
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing
> use
> >>> >>>>> cases.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal
> question
> >>> is
> >>> >>>>>>>>> whether header
> >>> >>>>>>>>>> > should
> >>> >>>>>>>>>> > > > be
> >>> >>>>>>>>>> > > > >> > >> exposed
> >>> >>>>>>>>>> > > > >> > >> >> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > stream
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such
> Kafka
> >>> >>>>> stream,
> >>> >>>>>>>>> Samza,
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > Spark
> >>> >>>>>>>>>> > > > >> > >> streaming.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems
> just
> >>> >>> deal
> >>> >>>>>>> with
> >>> >>>>>>>>>> > key/value
> >>> >>>>>>>>>> > > > pairs.
> >>> >>>>>>>>>> > > > >> > >> Should we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > expose a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there
> too
> >>> or
> >>> >>>>>>>>> somehow map
> >>> >>>>>>>>>> > header to
> >>> >>>>>>>>>> > > > key
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> >> value?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Jun
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at
> 3:35
> >>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>> > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a
> >>> period
> >>> >>>>> of a
> >>> >>>>>>>>> week,
> >>> >>>>>>>>>> > that there
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> no
> >>> >>>>>>>>>> > > > >> > >> >> concerns
> >>> >>>>>>>>>> > > > >> > >> >> >>> now
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2
> and
> >>> now
> >>> >>> we
> >>> >>>>>>> have
> >>> >>>>>>>>>> > agreement that
> >>> >>>>>>>>>> > > > >> > headers
> >>> >>>>>>>>>> > > > >> > >> are
> >>> >>>>>>>>>> > > > >> > >> >> >>> useful
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
> such if
> >>> >>> put
> >>> >>>>> to
> >>> >>>>>>> a
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>> > vote, this
> >>> >>>>>>>>>> > > > >> > wouldn't
> >>> >>>>>>>>>> > > > >> > >> be
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > reason
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > @
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of
> >>> getting
> >>> >>>>> this
> >>> >>>>>>>>> KIP
> >>> >>>>>>>>>> > moving past
> >>> >>>>>>>>>> > > > >> this,
> >>> >>>>>>>>>> > > > >> > we
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> state
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes
> space
> >>> >>> that
> >>> >>>>>>> can
> >>> >>>>>>>>> will be
> >>> >>>>>>>>>> > > > naturally
> >>> >>>>>>>>>> > > > >> > >> >> interpreted
> >>> >>>>>>>>>> > > > >> > >> >> >>> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing
> is
> >>> later
> >>> >>>>>>>>> wanted you can
> >>> >>>>>>>>>> > > easily
> >>> >>>>>>>>>> > > > >> split
> >>> >>>>>>>>>> > > > >> > >> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> into
> >>> >>>>>>>>>> > > > >> > >> >> >>> > two
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the
> >>> wire
> >>> >>>>>>> protocol
> >>> >>>>>>>>>> > > implementation
> >>> >>>>>>>>>> > > > >> this
> >>> >>>>>>>>>> > > > >> > >> makes
> >>> >>>>>>>>>> > > > >> > >> >> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don't
> >>> believe. Is
> >>> >>>>> this
> >>> >>>>>>>>>> > reasonable to
> >>> >>>>>>>>>> > > > all?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
> >>> therefor
> >>> >>>>> happy
> >>> >>>>>>>>> we keep
> >>> >>>>>>>>>> > with 32
> >>> >>>>>>>>>> > > > >> bits.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
> >>> >>>>>>>>>> > ignacio.solis@gmail.com on
> >>> >>>>>>>>>> > > > >> behalf
> >>> >>>>>>>>>> > > > >> > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> Ignacio
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
> >>> >>> ignacio.solis@gmail.com
> >>> >>>>> on
> >>> >>>>>>>>> behalf of
> >>> >>>>>>>>>> > > > >> > isolis@igso.net
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
> value
> >>> as
> >>> >>>>> byte[]
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String -
> Reluctant
> >>> >>> maybe
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the
> header
> >>> >>>>> system
> >>> >>>>>>>>> should
> >>> >>>>>>>>>> > take a
> >>> >>>>>>>>>> > > > single
> >>> >>>>>>>>>> > > > >> > >> int. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you
> >>> want
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>> > this as
> >>> >>>>>>>>>> > > to
> >>> >>>>>>>>>> > > > >> 16bit
> >>> >>>>>>>>>> > > > >> > >> >> numbers
> >>> >>>>>>>>>> > > > >> > >> >> >>> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > above go right
> ahead.
> >>> If
> >>> >>>>>>>>> somebody wants
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > argue
> >>> >>>>>>>>>> > > > >> for
> >>> >>>>>>>>>> > > > >> > 16
> >>> >>>>>>>>>> > > > >> > >> >> bits
> >>> >>>>>>>>>> > > > >> > >> >> >>> or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 64
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I
> >>> would
> >>> >>>>>>> listen.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key
> space
> >>> >>> into
> >>> >>>>>>>>> sub_key_1 and
> >>> >>>>>>>>>> > > > sub_key_2
> >>> >>>>>>>>>> > > > >> > >> makes no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sense to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > me at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we
> >>> going
> >>> >>> to
> >>> >>>>>>>>> start
> >>> >>>>>>>>>> > providing
> >>> >>>>>>>>>> > > > APIs to
> >>> >>>>>>>>>> > > > >> > get
> >>> >>>>>>>>>> > > > >> > >> all
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all
> the
> >>> >>>>>>>>> sub_key_2s? If
> >>> >>>>>>>>>> > there is
> >>> >>>>>>>>>> > > > no
> >>> >>>>>>>>>> > > > >> > >> >> >>> distinguishing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > functions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to
> >>> each
> >>> >>> one
> >>> >>>>>>>>> then they
> >>> >>>>>>>>>> > should
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > a
> >>> >>>>>>>>>> > > > >> > single
> >>> >>>>>>>>>> > > > >> > >> >> >>> value.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > At
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're
> doing
> >>> is
> >>> >>>>>>>>> equality.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer
> >>> wants
> >>> >>> to
> >>> >>>>>>>>> interpret
> >>> >>>>>>>>>> > this as
> >>> >>>>>>>>>> > > 2,
> >>> >>>>>>>>>> > > > 3
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> more
> >>> >>>>>>>>>> > > > >> > >> >> >>> values
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > different
> question. I
> >>> >>>>>>>>> personally think
> >>> >>>>>>>>>> > it's
> >>> >>>>>>>>>> > > all
> >>> >>>>>>>>>> > > > one
> >>> >>>>>>>>>> > > > >> > >> >> keyspace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned
> using
> >>> >>> some
> >>> >>>>>>>>> structure,
> >>> >>>>>>>>>> > but if
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> > want to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > sub-assign
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > parts
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's
> fine.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
> >>> >>> applies
> >>> >>>>> to
> >>> >>>>>>>>> strings.
> >>> >>>>>>>>>> > If
> >>> >>>>>>>>>> > > > somebody
> >>> >>>>>>>>>> > > > >> > >> argued
> >>> >>>>>>>>>> > > > >> > >> >> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would we be
> arguing to
> >>> >>>>> divide
> >>> >>>>>>> the
> >>> >>>>>>>>>> > strings with
> >>> >>>>>>>>>> > > > dots
> >>> >>>>>>>>>> > > > >> > ('.')
> >>> >>>>>>>>>> > > > >> > >> >> as a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them
> to
> >>> give
> >>> >>>>> us
> >>> >>>>>>> the
> >>> >>>>>>>>>> > different
> >>> >>>>>>>>>> > > name
> >>> >>>>>>>>>> > > > >> > segments
> >>> >>>>>>>>>> > > > >> > >> >> >>> > separately?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be
> performing
> >>> any
> >>> >>>>>>>>> actions on
> >>> >>>>>>>>>> > this key
> >>> >>>>>>>>>> > > > other
> >>> >>>>>>>>>> > > > >> > than
> >>> >>>>>>>>>> > > > >> > >> >> >>> > matching?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
> 2016 at
> >>> >>> 9:30
> >>> >>>>>>> AM,
> >>> >>>>>>>>> Michael
> >>> >>>>>>>>>> > > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> >>> concerns
> >>> >>> on
> >>> >>>>> 1
> >>> >>>>>>>>> and 2
> >>> >>>>>>>>>> > still?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this
> moving
> >>> >>> along a
> >>> >>>>>>> bit
> >>> >>>>>>>>> more
> >>> >>>>>>>>>> > I'd also
> >>> >>>>>>>>>> > > > like
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> ask
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> get
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last
> >>> points:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe
> we're
> >>> all
> >>> >>>>>>> roughly
> >>> >>>>>>>>> happy
> >>> >>>>>>>>>> > with the
> >>> >>>>>>>>>> > > > >> header
> >>> >>>>>>>>>> > > > >> > >> value
> >>> >>>>>>>>>> > > > >> > >> >> >>> > being a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> >>> consensus
> >>> >>> has
> >>> >>>>>>>>> been for an
> >>> >>>>>>>>>> > > > namespace
> >>> >>>>>>>>>> > > > >> > based
> >>> >>>>>>>>>> > > > >> > >> int
> >>> >>>>>>>>>> > > > >> > >> >> >>> > approach
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the
> >>> key.
> >>> >>> Any
> >>> >>>>>>>>> objections
> >>> >>>>>>>>>> > if this
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> > what
> >>> >>>>>>>>>> > > > >> > >> we
> >>> >>>>>>>>>> > > > >> > >> >> go
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
> >>> >>>>> assumption
> >>> >>>>>>> in
> >>> >>>>>>>>> (4) is
> >>> >>>>>>>>>> > > correct,
> >>> >>>>>>>>>> > > > >> > >> {int,int}
> >>> >>>>>>>>>> > > > >> > >> >> >>> keys.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both
> int's be
> >>> >>>>> int16 or
> >>> >>>>>>>>> int32?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them
> being
> >>> >>> int16(2
> >>> >>>>>>>>> bytes) as
> >>> >>>>>>>>>> > combined
> >>> >>>>>>>>>> > > > is
> >>> >>>>>>>>>> > > > >> > space
> >>> >>>>>>>>>> > > > >> > >> of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > per
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > original and
> gives
> >>> >>> plenty
> >>> >>>>> of
> >>> >>>>>>>>>> > combinations for
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> foreseeable,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead
> small.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any
> >>> benefit in
> >>> >>>>>>>>> another kip
> >>> >>>>>>>>>> > call to
> >>> >>>>>>>>>> > > > >> discuss
> >>> >>>>>>>>>> > > > >> > >> >> these at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > all?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>> ______________________________
> >>> >>>>>>>>>> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
> >>> >>>>>>>>> k.burstev@yandex.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> >>> November
> >>> >>> 18,
> >>> >>>>>>> 2016
> >>> >>>>>>>>>> > 7:07:07 AM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
> >>> >>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> >>> [DISCUSS]
> >>> >>>>> KIP-82
> >>> >>>>>>>>> - Add
> >>> >>>>>>>>>> > Record
> >>> >>>>>>>>>> > > > Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is
> worth
> >>> >>> also
> >>> >>>>> i
> >>> >>>>>>>>> agree. As
> >>> >>>>>>>>>> > a user:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
> Headers are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers
> >>> should
> >>> >>>>> be a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>> > > option
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
> 21:15,
> >>> >>>>> "Ignacio
> >>> >>>>>>>>> Solis" <
> >>> >>>>>>>>>> > > > >> isolis@igso.net
> >>> >>>>>>>>>> > > > >> > >:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
> Headers
> >>> are
> >>> >>>>>>>>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
> Headers
> >>> >>> should
> >>> >>>>> be
> >>> >>>>>>> a
> >>> >>>>>>>>> top
> >>> >>>>>>>>>> > level
> >>> >>>>>>>>>> > > option
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14,
> >>> 2016
> >>> >>> at
> >>> >>>>>>> 9:16
> >>> >>>>>>>>> AM,
> >>> >>>>>>>>>> > Michael
> >>> >>>>>>>>>> > > > Pearce
> >>> >>>>>>>>>> > > > >> <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
> >>> >>>>> details/examples
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > original
> >>> >>>>>>>>>> > > > proposal
> >>> >>>>>>>>>> > > > >> > for
> >>> >>>>>>>>>> > > > >> > >> key
> >>> >>>>>>>>>> > > > >> > >> >> >>> > spacing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> ,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > not
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new
> mentioned as
> >>> per
> >>> >>>>>>>>> discussion
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > >> idea.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need
> to
> >>> >>> update
> >>> >>>>>>> the
> >>> >>>>>>>>> kip,
> >>> >>>>>>>>>> > when we
> >>> >>>>>>>>>> > > get
> >>> >>>>>>>>>> > > > >> > >> agreement
> >>> >>>>>>>>>> > > > >> > >> >> >>> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > is a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > better
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach
> (which
> >>> >>> seems
> >>> >>>>> to
> >>> >>>>>>>>> be the
> >>> >>>>>>>>>> > case if I
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> > >> >> understood
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > general
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in
> the
> >>> >>>>>>>>> conversation)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the
> variable
> >>> >>> ints,
> >>> >>>>> at
> >>> >>>>>>>>> very
> >>> >>>>>>>>>> > early stage
> >>> >>>>>>>>>> > > > we
> >>> >>>>>>>>>> > > > >> did
> >>> >>>>>>>>>> > > > >> > >> think
> >>> >>>>>>>>>> > > > >> > >> >> >>> about
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added
> >>> complexity
> >>> >>>>> for
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > saving isn't
> >>> >>>>>>>>>> > > > >> worth
> >>> >>>>>>>>>> > > > >> > it.
> >>> >>>>>>>>>> > > > >> > >> >> I'd
> >>> >>>>>>>>>> > > > >> > >> >> >>> > rather
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> go
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to
> reduce
> >>> >>>>>>>>> overheads and
> >>> >>>>>>>>>> > size
> >>> >>>>>>>>>> > > int16
> >>> >>>>>>>>>> > > > >> > (2bytes)
> >>> >>>>>>>>>> > > > >> > >> >> keys
> >>> >>>>>>>>>> > > > >> > >> >> >>> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note
> of no
> >>> >>>>>>> headers,
> >>> >>>>>>>>> there
> >>> >>>>>>>>>> > is as
> >>> >>>>>>>>>> > > per
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > kip
> >>> >>>>>>>>>> > > > >> > >> as
> >>> >>>>>>>>>> > > > >> > >> >> we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > use an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to
> denote if
> >>> >>>>> headers
> >>> >>>>>>>>> are
> >>> >>>>>>>>>> > present or
> >>> >>>>>>>>>> > > > not as
> >>> >>>>>>>>>> > > > >> > such
> >>> >>>>>>>>>> > > > >> > >> >> >>> > provides a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > zero
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead
> >>> currently
> >>> >>> if
> >>> >>>>>>>>> headers are
> >>> >>>>>>>>>> > not
> >>> >>>>>>>>>> > > used.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as
> radai
> >>> >>>>> mentions
> >>> >>>>>>>>> would be
> >>> >>>>>>>>>> > good
> >>> >>>>>>>>>> > > > first
> >>> >>>>>>>>>> > > > >> > if we
> >>> >>>>>>>>>> > > > >> > >> >> can
> >>> >>>>>>>>>> > > > >> > >> >> >>> get
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have
> >>> general
> >>> >>>>>>>>> consensus that
> >>> >>>>>>>>>> > (1)
> >>> >>>>>>>>>> > > > headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we
> want
> >>> it
> >>> >>> as
> >>> >>>>> a
> >>> >>>>>>>>> top level
> >>> >>>>>>>>>> > entity.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to
> state the
> >>> >>>>>>> obvious i
> >>> >>>>>>>>>> > believe (1)
> >>> >>>>>>>>>> > > > >> headers
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a
> top
> >>> level
> >>> >>>>>>>>> entity.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>> ______________________________
> >>> >>>>>>>>>> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
> >>> Hoover <
> >>> >>>>>>>>>> > > roger.hoover@gmail.com
> >>> >>>>>>>>>> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent:
> Wednesday,
> >>> >>>>> November
> >>> >>>>>>>>> 9, 2016
> >>> >>>>>>>>>> > 9:10:47
> >>> >>>>>>>>>> > > > PM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
> >>> >>>>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
> >>> >>> [DISCUSS]
> >>> >>>>>>>>> KIP-82 - Add
> >>> >>>>>>>>>> > > Record
> >>> >>>>>>>>>> > > > >> > Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
> going a
> >>> >>>>> little
> >>> >>>>>>>>> in the
> >>> >>>>>>>>>> > weeds but
> >>> >>>>>>>>>> > > > >> thanks
> >>> >>>>>>>>>> > > > >> > >> for
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> replies
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a
> >>> prefix
> >>> >>>>> and
> >>> >>>>>>>>> {int,
> >>> >>>>>>>>>> > int} can
> >>> >>>>>>>>>> > > be
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> same.
> >>> >>>>>>>>>> > > > >> > >> >> It
> >>> >>>>>>>>>> > > > >> > >> >> >>> > doesn't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > look
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > like
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what
> the
> >>> KIP
> >>> >>> is
> >>> >>>>>>>>> saying the
> >>> >>>>>>>>>> > "Open"
> >>> >>>>>>>>>> > > > >> > section.
> >>> >>>>>>>>>> > > > >> > >> The
> >>> >>>>>>>>>> > > > >> > >> >> >>> > example
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shows
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New
> Relic and
> >>> >>>>> 210002
> >>> >>>>>>>>> for App
> >>> >>>>>>>>>> > Dynamics
> >>> >>>>>>>>>> > > > >> > implying
> >>> >>>>>>>>>> > > > >> > >> >> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > New
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization
> will
> >>> >>> have
> >>> >>>>>>>>> only a
> >>> >>>>>>>>>> > single
> >>> >>>>>>>>>> > > > header id
> >>> >>>>>>>>>> > > > >> > to
> >>> >>>>>>>>>> > > > >> > >> work
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with. Or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The
> >>> main
> >>> >>>>> point
> >>> >>>>>>>>> of a
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > or
> >>> >>>>>>>>>> > > > >> > prefix
> >>> >>>>>>>>>> > > > >> > >> is
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > reduce
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of
> >>> config
> >>> >>>>>>> mapping
> >>> >>>>>>>>> or
> >>> >>>>>>>>>> > > registration
> >>> >>>>>>>>>> > > > >> > >> depending
> >>> >>>>>>>>>> > > > >> > >> >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> how
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> namespaces/prefixes
> >>> >>>>> are
> >>> >>>>>>>>> managed.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to
> >>> hear
> >>> >>>>> more
> >>> >>>>>>>>> feedback
> >>> >>>>>>>>>> > on the
> >>> >>>>>>>>>> > > > >> > >> higher-level
> >>> >>>>>>>>>> > > > >> > >> >> >>> > questions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov
> 9,
> >>> 2016
> >>> >>> at
> >>> >>>>>>>>> 11:38 AM,
> >>> >>>>>>>>>> > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think
> this
> >>> >>>>> discussion
> >>> >>>>>>>>> is
> >>> >>>>>>>>>> > getting a
> >>> >>>>>>>>>> > > bit
> >>> >>>>>>>>>> > > > >> into
> >>> >>>>>>>>>> > > > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> weeds on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > technical
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> implementation
> >>> >>>>> details.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to
> >>> step
> >>> >>>>> back a
> >>> >>>>>>>>> minute
> >>> >>>>>>>>>> > and try
> >>> >>>>>>>>>> > > > and
> >>> >>>>>>>>>> > > > >> > >> establish
> >>> >>>>>>>>>> > > > >> > >> >> >>> > where we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > are in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger
> picture:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
> >>> >>> nacho's
> >>> >>>>>>> last
> >>> >>>>>>>>>> > paragraph)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we
> all
> >>> in
> >>> >>>>>>>>> agreement that
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > > are
> >>> >>>>>>>>>> > > > >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> worthwhile
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > useful
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to
> >>> have?
> >>> >>>>> this
> >>> >>>>>>>>> was
> >>> >>>>>>>>>> > contested
> >>> >>>>>>>>>> > > > early
> >>> >>>>>>>>>> > > > >> on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we
> all
> >>> in
> >>> >>>>>>>>> agreement on
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > as
> >>> >>>>>>>>>> > > > top
> >>> >>>>>>>>>> > > > >> > >> level
> >>> >>>>>>>>>> > > > >> > >> >> >>> entity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > vs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> squirreled-away in
> >>> >>>>> V?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there
> are
> >>> still
> >>> >>>>>>>>> concerns
> >>> >>>>>>>>>> > around
> >>> >>>>>>>>>> > > these
> >>> >>>>>>>>>> > > > #2
> >>> >>>>>>>>>> > > > >> > >> points
> >>> >>>>>>>>>> > > > >> > >> >> >>> (#jay?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now
> back
> >>> to
> >>> >>> our
> >>> >>>>>>>>> normal
> >>> >>>>>>>>>> > programming
> >>> >>>>>>>>>> > > > ...)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are
> >>> nice.
> >>> >>>>>>> having
> >>> >>>>>>>>> said
> >>> >>>>>>>>>> > that, its
> >>> >>>>>>>>>> > > > >> adding
> >>> >>>>>>>>>> > > > >> > >> >> >>> complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (see
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>> https://github.com/addthis/
> >>> >>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>> main/java/com/clearspring/
> >>> >>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
> google
> >>> >>>>> result)
> >>> >>>>>>>>> and would
> >>> >>>>>>>>>> > require
> >>> >>>>>>>>>> > > > >> anyone
> >>> >>>>>>>>>> > > > >> > >> >> writing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clients
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go?
> >>> Bash?
> >>> >>>>> ;-) )
> >>> >>>>>>>>> to
> >>> >>>>>>>>>> > > get/implement
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > >> same,
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain
> >>> (int
> >>> >>> vs
> >>> >>>>>>>>> string is
> >>> >>>>>>>>>> > order of
> >>> >>>>>>>>>> > > > >> > magnitude,
> >>> >>>>>>>>>> > > > >> > >> >> this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > isnt).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int
> >>> namespacing vs
> >>> >>>>>>> {int,
> >>> >>>>>>>>> int}
> >>> >>>>>>>>>> > > namespacing
> >>> >>>>>>>>>> > > > >> are
> >>> >>>>>>>>>> > > > >> > >> >> basically
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > same
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
> >>> >>>>> namespacing
> >>> >>>>>>>>> an int64
> >>> >>>>>>>>>> > and
> >>> >>>>>>>>>> > > > giving
> >>> >>>>>>>>>> > > > >> > people
> >>> >>>>>>>>>> > > > >> > >> >> while
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 2^32
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the
> part
> >>> i
> >>> >>>>> like
> >>> >>>>>>>>> about this
> >>> >>>>>>>>>> > is
> >>> >>>>>>>>>> > > > letting
> >>> >>>>>>>>>> > > > >> > >> people
> >>> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> large
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers
> with
> >>> one
> >>> >>>>>>>>> registration so
> >>> >>>>>>>>>> > they
> >>> >>>>>>>>>> > > > dont
> >>> >>>>>>>>>> > > > >> > have
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> come
> >>> >>>>>>>>>> > > > >> > >> >> >>> > back
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > every
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
> >>> >>> plugin/header
> >>> >>>>>>>>> they want to
> >>> >>>>>>>>>> > > > "reserve".
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
> Nov 9,
> >>> >>> 2016
> >>> >>>>> at
> >>> >>>>>>>>> 11:01 AM,
> >>> >>>>>>>>>> > Roger
> >>> >>>>>>>>>> > > > >> Hoover
> >>> >>>>>>>>>> > > > >> > <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> roger.hoover@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since
> some of
> >>> >>> the
> >>> >>>>>>>>> debate has
> >>> >>>>>>>>>> > been
> >>> >>>>>>>>>> > > about
> >>> >>>>>>>>>> > > > >> > >> overhead +
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> wondering if
> >>> we
> >>> >>>>> have
> >>> >>>>>>>>>> > considered a
> >>> >>>>>>>>>> > > > varint
> >>> >>>>>>>>>> > > > >> > >> encoding
> >>> >>>>>>>>>> > > > >> > >> >> (
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>> https://developers.google.com/
> >>> >>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the
> header
> >>> >>> length
> >>> >>>>>>>>> field (int32
> >>> >>>>>>>>>> > in the
> >>> >>>>>>>>>> > > > >> > proposal)
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
> >>> >>> headers,
> >>> >>>>>>> the
> >>> >>>>>>>>>> > overhead would
> >>> >>>>>>>>>> > > > be a
> >>> >>>>>>>>>> > > > >> > >> single
> >>> >>>>>>>>>> > > > >> > >> >> >>> byte
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128
> >>> would
> >>> >>>>> also
> >>> >>>>>>>>> need only a
> >>> >>>>>>>>>> > > single
> >>> >>>>>>>>>> > > > >> byte?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed,
> Nov
> >>> 9,
> >>> >>>>> 2016
> >>> >>>>>>> at
> >>> >>>>>>>>> 6:43 AM,
> >>> >>>>>>>>>> > > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com
> >>> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> @magnus -
> >>> and
> >>> >>>>> very
> >>> >>>>>>>>> dangerous
> >>> >>>>>>>>>> > (youre
> >>> >>>>>>>>>> > > > >> > >> essentially
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> arbitrary
> >>> code
> >>> >>>>> off
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > internet on
> >>> >>>>>>>>>> > > > your
> >>> >>>>>>>>>> > > > >> > >> servers
> >>> >>>>>>>>>> > > > >> > >> >> ...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > bad
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > idea
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > without
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> sandbox,
> >>> even
> >>> >>>>> with)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for
> it
> >>> >>> being
> >>> >>>>> a
> >>> >>>>>>>>> purely
> >>> >>>>>>>>>> > > > administrative
> >>> >>>>>>>>>> > > > >> > task
> >>> >>>>>>>>>> > > > >> > >> - i
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish
> it
> >>> >>> would,
> >>> >>>>>>>>> really,
> >>> >>>>>>>>>> > because
> >>> >>>>>>>>>> > > > then my
> >>> >>>>>>>>>> > > > >> > >> earlier
> >>> >>>>>>>>>> > > > >> > >> >> >>> > point on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >>> remapping
> >>> >>>>>>>>> process would
> >>> >>>>>>>>>> > be
> >>> >>>>>>>>>> > > > invalid,
> >>> >>>>>>>>>> > > > >> > but
> >>> >>>>>>>>>> > > > >> > >> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > linkedin,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the
> team
> >>> im
> >>> >>> in)
> >>> >>>>>>> run
> >>> >>>>>>>>> kafka
> >>> >>>>>>>>>> > as a
> >>> >>>>>>>>>> > > > service.
> >>> >>>>>>>>>> > > > >> > we
> >>> >>>>>>>>>> > > > >> > >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > really
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > know
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> (developing
> >>> >>>>>>>>> applications
> >>> >>>>>>>>>> > that use
> >>> >>>>>>>>>> > > > kafka)
> >>> >>>>>>>>>> > > > >> > are
> >>> >>>>>>>>>> > > > >> > >> up
> >>> >>>>>>>>>> > > > >> > >> >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > any
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > given
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
> >>> >>> possible
> >>> >>>>>>>>> (given the
> >>> >>>>>>>>>> > > > existance of
> >>> >>>>>>>>>> > > > >> > >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> and a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> ecosystem)
> >>> for
> >>> >>>>> some
> >>> >>>>>>>>>> > application to
> >>> >>>>>>>>>> > > > >> "equip"
> >>> >>>>>>>>>> > > > >> > >> their
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> producers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with
> the
> >>> >>>>> required
> >>> >>>>>>>>> plugin
> >>> >>>>>>>>>> > without us
> >>> >>>>>>>>>> > > > >> > knowing.
> >>> >>>>>>>>>> > > > >> > >> i
> >>> >>>>>>>>>> > > > >> > >> >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > mean
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i
> just
> >>> >>>>> want to
> >>> >>>>>>>>> make the
> >>> >>>>>>>>>> > point
> >>> >>>>>>>>>> > > > that
> >>> >>>>>>>>>> > > > >> > its
> >>> >>>>>>>>>> > > > >> > >> not
> >>> >>>>>>>>>> > > > >> > >> >> as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > simple
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
> >>> across a
> >>> >>>>>>>>> large-enough
> >>> >>>>>>>>>> > > > organization.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed,
> >>> Nov 9,
> >>> >>>>> 2016
> >>> >>>>>>>>> at 6:17
> >>> >>>>>>>>>> > AM,
> >>> >>>>>>>>>> > > > Magnus
> >>> >>>>>>>>>> > > > >> > >> Edenhill
> >>> >>>>>>>>>> > > > >> > >> >> <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> magnus@edenhill.se>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
> think
> >>> >>> there
> >>> >>>>> is
> >>> >>>>>>> a
> >>> >>>>>>>>> piece
> >>> >>>>>>>>>> > missing
> >>> >>>>>>>>>> > > in
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> >> Strings
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > where
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>> pro-Stringers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> reason
> >>> that
> >>> >>> by
> >>> >>>>>>>>> providing
> >>> >>>>>>>>>> > unique
> >>> >>>>>>>>>> > > > string
> >>> >>>>>>>>>> > > > >> > >> >> >>> identifiers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> everything
> >>> >>>>> will
> >>> >>>>>>>>> just
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> magically
> >>> >>> work
> >>> >>>>>>> for
> >>> >>>>>>>>> all
> >>> >>>>>>>>>> > parts of
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> > stream
> >>> >>>>>>>>>> > > > >> > >> >> >>> pipeline.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But
> the
> >>> >>>>> strings
> >>> >>>>>>>>> dont mean
> >>> >>>>>>>>>> > > anything
> >>> >>>>>>>>>> > > > by
> >>> >>>>>>>>>> > > > >> > >> >> themselves,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > while we
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> probably
> >>> >>>>> envision
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> auto
> >>> >>>>> plugin
> >>> >>>>>>>>> loader
> >>> >>>>>>>>>> > that
> >>> >>>>>>>>>> > > > >> downloads,
> >>> >>>>>>>>>> > > > >> > >> >> compiles,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > links
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> on-demand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as
> soon
> >>> as
> >>> >>>>>>> they're
> >>> >>>>>>>>> seen by
> >>> >>>>>>>>>> > a
> >>> >>>>>>>>>> > > > >> consumer, I
> >>> >>>>>>>>>> > > > >> > >> dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> really
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> see
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> something
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
> >>> dynamic
> >>> >>>>> (and
> >>> >>>>>>>>> fragile) in
> >>> >>>>>>>>>> > > > practice.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In
> the
> >>> real
> >>> >>>>> world
> >>> >>>>>>>>> an
> >>> >>>>>>>>>> > application
> >>> >>>>>>>>>> > > > will
> >>> >>>>>>>>>> > > > >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> configured
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a set
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to
> either
> >>> >>> add
> >>> >>>>>>>>> (producer)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or
> read
> >>> >>>>>>> (consumer)
> >>> >>>>>>>>> headers.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This
> is
> >>> an
> >>> >>>>>>>>> administrative
> >>> >>>>>>>>>> > task
> >>> >>>>>>>>>> > > > based
> >>> >>>>>>>>>> > > > >> on
> >>> >>>>>>>>>> > > > >> > >> what
> >>> >>>>>>>>>> > > > >> > >> >> >>> > features a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > client
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>> needs/provides
> >>> >>>>>>> and
> >>> >>>>>>>>> results
> >>> >>>>>>>>>> > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> >>> sort of
> >>> >>>>>>>>> configuration
> >>> >>>>>>>>>> > to
> >>> >>>>>>>>>> > > > enable
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> configure
> >>> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > desired
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since
> >>> this
> >>> >>>>> needs
> >>> >>>>>>>>> to be kept
> >>> >>>>>>>>>> > > > somewhat
> >>> >>>>>>>>>> > > > >> in
> >>> >>>>>>>>>> > > > >> > >> sync
> >>> >>>>>>>>>> > > > >> > >> >> >>> across
> >>> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no
> >>> point
> >>> >>> in
> >>> >>>>>>>>> having
> >>> >>>>>>>>>> > producers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
> >>> headers
> >>> >>> no
> >>> >>>>>>>>> consumers
> >>> >>>>>>>>>> > will
> >>> >>>>>>>>>> > > read,
> >>> >>>>>>>>>> > > > >> and
> >>> >>>>>>>>>> > > > >> > >> vice
> >>> >>>>>>>>>> > > > >> > >> >> >>> versa),
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > added
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> complexity
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
> >>> assigning
> >>> >>>>> an
> >>> >>>>>>> id
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for
> each
> >>> >>>>> plugin
> >>> >>>>>>> as
> >>> >>>>>>>>> it is
> >>> >>>>>>>>>> > being
> >>> >>>>>>>>>> > > > >> > configured
> >>> >>>>>>>>>> > > > >> > >> >> should
> >>> >>>>>>>>>> > > > >> > >> >> >>> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> /Magnus
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> 2016-11-09
> >>> >>>>> 13:06
> >>> >>>>>>>>> GMT+01:00
> >>> >>>>>>>>>> > > Michael
> >>> >>>>>>>>>> > > > >> > Pearce <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>> Michael.Pearce@ig.com>:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Just
> >>> >>>>>>>>> following/catching
> >>> >>>>>>>>>> > up on
> >>> >>>>>>>>>> > > > what
> >>> >>>>>>>>>> > > > >> > seems
> >>> >>>>>>>>>> > > > >> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > active
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> @Radai
> >>> >>>>> sorry if
> >>> >>>>>>>>> it may
> >>> >>>>>>>>>> > seem
> >>> >>>>>>>>>> > > > obvious
> >>> >>>>>>>>>> > > > >> > but
> >>> >>>>>>>>>> > > > >> > >> what
> >>> >>>>>>>>>> > > > >> > >> >> >>> does
> >>> >>>>>>>>>> > > > >> > >> >> >>> > MD
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > stand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My
> >>> take on
> >>> >>>>>>>>> String vs Int:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> will
> >>> >>> state
> >>> >>>>>>>>> first I am
> >>> >>>>>>>>>> > pro Int
> >>> >>>>>>>>>> > > > (16
> >>> >>>>>>>>>> > > > >> or
> >>> >>>>>>>>>> > > > >> > >> 32).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> do
> >>> >>> though
> >>> >>>>>>>>> playing
> >>> >>>>>>>>>> > devils
> >>> >>>>>>>>>> > > > advocate
> >>> >>>>>>>>>> > > > >> > see a
> >>> >>>>>>>>>> > > > >> > >> >> big
> >>> >>>>>>>>>> > > > >> > >> >> >>> plus
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> String
> >>> >>> keys,
> >>> >>>>>>>>> this is
> >>> >>>>>>>>>> > around
> >>> >>>>>>>>>> > > > >> > integrating
> >>> >>>>>>>>>> > > > >> > >> >> into an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > existing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As
> many
> >>> >>>>> other
> >>> >>>>>>>>> systems use
> >>> >>>>>>>>>> > > String
> >>> >>>>>>>>>> > > > >> based
> >>> >>>>>>>>>> > > > >> > >> >> headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> much
> >>> >>> easier
> >>> >>>>> for
> >>> >>>>>>>>> these to
> >>> >>>>>>>>>> > be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > into.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How
> >>> with
> >>> >>> Int
> >>> >>>>>>>>> based
> >>> >>>>>>>>>> > headers
> >>> >>>>>>>>>> > > could
> >>> >>>>>>>>>> > > > we
> >>> >>>>>>>>>> > > > >> > >> provide
> >>> >>>>>>>>>> > > > >> > >> >> a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>> integration
> >>> >>>>>>>>> simple /
> >>> >>>>>>>>>> > easy with
> >>> >>>>>>>>>> > > > >> > transition
> >>> >>>>>>>>>> > > > >> > >> >> flows
> >>> >>>>>>>>>> > > > >> > >> >> >>> > over
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> tough
> >>> >>> luck
> >>> >>>>>>>>> buddy
> >>> >>>>>>>>>> > you're on
> >>> >>>>>>>>>> > > your
> >>> >>>>>>>>>> > > > >> own
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> >>> simply
> >>> >>>>> hash
> >>> >>>>>>>>> the string
> >>> >>>>>>>>>> > into
> >>> >>>>>>>>>> > > int
> >>> >>>>>>>>>> > > > >> code
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> hope
> >>> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> convert
> >>> >>> back
> >>> >>>>>>>>> though?)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> http2
> >>> >>>>> style
> >>> >>>>>>> as
> >>> >>>>>>>>>> > mentioned by
> >>> >>>>>>>>>> > > > nacho.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> cheers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Mike
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > ______________________________
> >>> >>>>>>>>>> > > > >> > __________
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> From:
> >>> >>> radai
> >>> >>>>> <
> >>> >>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Sent:
> >>> >>>>>>> Wednesday,
> >>> >>>>>>>>>> > November 9,
> >>> >>>>>>>>>> > > 2016
> >>> >>>>>>>>>> > > > >> > 8:12 AM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
> >>> >>>>>>>>> dev@kafka.apache.org
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> Subject:
> >>> >>> Re:
> >>> >>>>>>>>> [DISCUSS]
> >>> >>>>>>>>>> > KIP-82 -
> >>> >>>>>>>>>> > > > Add
> >>> >>>>>>>>>> > > > >> > >> Record
> >>> >>>>>>>>>> > > > >> > >> >> >>> Headers
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> thinking
> >>> >>>>> about
> >>> >>>>>>>>> it some
> >>> >>>>>>>>>> > more,
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> best
> >>> >>>>>>>>>> > > > >> > >> way to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > transmit
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> remapping
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> data to
> >>> >>>>>>>>> consumers would
> >>> >>>>>>>>>> > be to
> >>> >>>>>>>>>> > > > put it
> >>> >>>>>>>>>> > > > >> > in
> >>> >>>>>>>>>> > > > >> > >> the
> >>> >>>>>>>>>> > > > >> > >> >> MD
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> response
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it
> >>> should
> >>> >>> be
> >>> >>>>>>>>> discussed
> >>> >>>>>>>>>> > now.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On
> Wed,
> >>> >>> Nov
> >>> >>>>> 9,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>> > 12:09
> >>> >>>>>>>>>> > > AM,
> >>> >>>>>>>>>> > > > >> > radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> >>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> im
> >>> not
> >>> >>>>>>> opposed
> >>> >>>>>>>>> to the
> >>> >>>>>>>>>> > idea of
> >>> >>>>>>>>>> > > > >> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> mapping.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > im
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> not
> >>> part
> >>> >>>>> of
> >>> >>>>>>>>> the "mvp"
> >>> >>>>>>>>>> > and,
> >>> >>>>>>>>>> > > > since
> >>> >>>>>>>>>> > > > >> it
> >>> >>>>>>>>>> > > > >> > >> >> requires
> >>> >>>>>>>>>> > > > >> > >> >> >>> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> wire
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > format
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> always
> >>> >>> be
> >>> >>>>>>>>> added later.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also,
> >>> >>> its
> >>> >>>>> not
> >>> >>>>>>>>> as
> >>> >>>>>>>>>> > simple as
> >>> >>>>>>>>>> > > just
> >>> >>>>>>>>>> > > > >> > >> >> configuring
> >>> >>>>>>>>>> > > > >> > >> >> >>> MM
> >>> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> say
> >>> i've
> >>> >>>>>>>>> implemented
> >>> >>>>>>>>>> > large
> >>> >>>>>>>>>> > > > message
> >>> >>>>>>>>>> > > > >> > >> >> support as
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> mirror
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> target
> >>> >>>>>>> cluster
> >>> >>>>>>>>> its been
> >>> >>>>>>>>>> > > > remapped
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> {999,1}.
> >>> >>>>>>>>>> > > > >> > >> >> >>> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also
> >>> >>> need
> >>> >>>>> to
> >>> >>>>>>>>> be told
> >>> >>>>>>>>>> > to look
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> the
> >>> >>>>>>>>>> > > > >> > >> large
> >>> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> under
> >>> >>>>>>> {999,1}.
> >>> >>>>>>>>> doable,
> >>> >>>>>>>>>> > but
> >>> >>>>>>>>>> > > > tricky.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>> 8,
> >>> >>>>>>>>> 2016 at
> >>> >>>>>>>>>> > 10:29
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> While
> >>> >>> you
> >>> >>>>>>> can
> >>> >>>>>>>>> do
> >>> >>>>>>>>>> > whatever
> >>> >>>>>>>>>> > > you
> >>> >>>>>>>>>> > > > >> want
> >>> >>>>>>>>>> > > > >> > >> with a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > your
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> what
> >>> >>> I'd
> >>> >>>>>>>>> expect is
> >>> >>>>>>>>>> > for each
> >>> >>>>>>>>>> > > > app
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> namespaces
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> So
> >>> if I
> >>> >>>>>>>>> accidentally
> >>> >>>>>>>>>> > used
> >>> >>>>>>>>>> > > 666
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> > my
> >>> >>>>>>>>>> > > > >> > >> HR
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> department,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > still
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> run
> >>> >>>>>>> RadaiApp,
> >>> >>>>>>>>> I can
> >>> >>>>>>>>>> > config
> >>> >>>>>>>>>> > > > >> > >> "namespace=42"
> >>> >>>>>>>>>> > > > >> > >> >> >>> for
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> everything
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> will
> >>> >>> look
> >>> >>>>>>>>> normal.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> This
> >>> >>>>> means
> >>> >>>>>>>>> you only
> >>> >>>>>>>>>> > need to
> >>> >>>>>>>>>> > > > sync
> >>> >>>>>>>>>> > > > >> > usage
> >>> >>>>>>>>>> > > > >> > >> >> >>> inside
> >>> >>>>>>>>>> > > > >> > >> >> >>> > your
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > own
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> organization.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> Still
> >>> >>>>> hard,
> >>> >>>>>>>>> but
> >>> >>>>>>>>>> > somewhat
> >>> >>>>>>>>>> > > > easier
> >>> >>>>>>>>>> > > > >> > than
> >>> >>>>>>>>>> > > > >> > >> >> syncing
> >>> >>>>>>>>>> > > > >> > >> >> >>> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> On
> >>> Tue,
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>> > at 10:07
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > >> radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> and
> >>> >>> we
> >>> >>>>> can
> >>> >>>>>>>>> start
> >>> >>>>>>>>>> > with
> >>> >>>>>>>>>> > > > >> {namespace,
> >>> >>>>>>>>>> > > > >> > >> id}
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> no
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> always
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> add
> >>> >>> it
> >>> >>>>>>>>> later on
> >>> >>>>>>>>>> > if/when
> >>> >>>>>>>>>> > > > >> > collisions
> >>> >>>>>>>>>> > > > >> > >> >> >>> actually
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> they'd
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > a
> >>> >>>>>>> problem).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> every
> >>> >>>>>>>>> interested
> >>> >>>>>>>>>> > party (so
> >>> >>>>>>>>>> > > > orgs
> >>> >>>>>>>>>> > > > >> > or
> >>> >>>>>>>>>> > > > >> > >> >> >>> > individuals)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > could
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> register
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>> prefix
> >>> >>>>> (0
> >>> >>>>>>> =
> >>> >>>>>>>>>> > reserved, 1 =
> >>> >>>>>>>>>> > > > >> > confluent
> >>> >>>>>>>>>> > > > >> > >> ...
> >>> >>>>>>>>>> > > > >> > >> >> >>> 666
> >>> >>>>>>>>>> > > > >> > >> >> >>> > = me
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> whatever
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> the
> >>> >>>>> 2nd ID
> >>> >>>>>>>>> - so once
> >>> >>>>>>>>>> > > > linkedin
> >>> >>>>>>>>>> > > > >> > >> >> registers,
> >>> >>>>>>>>>> > > > >> > >> >> >>> say
> >>> >>>>>>>>>> > > > >> > >> >> >>> > 3,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> free
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > to
> >>> >>> use
> >>> >>>>> {3,
> >>> >>>>>>>>> *} with a
> >>> >>>>>>>>>> > > > reasonable
> >>> >>>>>>>>>> > > > >> > >> >> >>> expectation
> >>> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> anything
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> else.
> >>> >>>>>>>>> further
> >>> >>>>>>>>>> > partitioning
> >>> >>>>>>>>>> > > > of
> >>> >>>>>>>>>> > > > >> > that *
> >>> >>>>>>>>>> > > > >> > >> >> >>> becomes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>> "upstream
> >>> >>>>>>>>>> > registration"
> >>> >>>>>>>>>> > > of a
> >>> >>>>>>>>>> > > > >> > >> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> only
> >>> >>>>>>>>>> > > > >> > >> >> >>> > has
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > On
> >>> >>> Tue,
> >>> >>>>>>> Nov
> >>> >>>>>>>>> 8, 2016
> >>> >>>>>>>>>> > at
> >>> >>>>>>>>>> > > 9:03
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > >> James
> >>> >>>>>>>>>> > > > >> > >> >> >>> Cheng <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>> >>>>> wushujames@gmail.com
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > On
> >>> >>>>> Nov
> >>> >>>>>>>>> 8, 2016,
> >>> >>>>>>>>>> > at 5:54
> >>> >>>>>>>>>> > > > PM,
> >>> >>>>>>>>>> > > > >> > Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>> gwen@confluent.io
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> Thank
> >>> >>>>>>>>> you so
> >>> >>>>>>>>>> > much for
> >>> >>>>>>>>>> > > > this
> >>> >>>>>>>>>> > > > >> > clear
> >>> >>>>>>>>>> > > > >> > >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> fair
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> arguments.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> I'm
> >>> >>>>> in
> >>> >>>>>>>>> favor of
> >>> >>>>>>>>>> > ints.
> >>> >>>>>>>>>> > > > Not a
> >>> >>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > but
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> Even
> >>> >>>>>>>>> more in
> >>> >>>>>>>>>> > favor of
> >>> >>>>>>>>>> > > > >> Magnus's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > decentralized
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Roger's
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> tweak:
> >>> >>>>>>>>> add a
> >>> >>>>>>>>>> > namespace
> >>> >>>>>>>>>> > > > for
> >>> >>>>>>>>>> > > > >> > >> headers.
> >>> >>>>>>>>>> > > > >> > >> >> >>> This
> >>> >>>>>>>>>> > > > >> > >> >> >>> > will
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > allow
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> just
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> use
> >>> >>>>>>>>> whatever IDs
> >>> >>>>>>>>>> > it
> >>> >>>>>>>>>> > > wants
> >>> >>>>>>>>>> > > > >> > >> >> internally,
> >>> >>>>>>>>>> > > > >> > >> >> >>> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> then
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > let
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>> deploying
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> the
> >>> >>>>> app
> >>> >>>>>>>>> figure
> >>> >>>>>>>>>> > out an
> >>> >>>>>>>>>> > > > >> > available
> >>> >>>>>>>>>> > > > >> > >> >> >>> namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > ID
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > So
> >>> >>>>>>>>>> > > > >> > io.confluent.schema-registry
> >>> >>>>>>>>>> > > > >> > >> can
> >>> >>>>>>>>>> > > > >> > >> >> be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>> deployment
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> and
> >>> >>>>>>> 0x57
> >>> >>>>>>>>> on
> >>> >>>>>>>>>> > yours, and
> >>> >>>>>>>>>> > > > the
> >>> >>>>>>>>>> > > > >> > poor
> >>> >>>>>>>>>> > > > >> > >> guys
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>> worry
> >>> >>>>>>>>> about that.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> Gwen,
> >>> >>>>> if
> >>> >>>>>>> I
> >>> >>>>>>>>>> > understand
> >>> >>>>>>>>>> > > your
> >>> >>>>>>>>>> > > > >> > example
> >>> >>>>>>>>>> > > > >> > >> >> >>> right, an
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > application
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> might
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> decide to
> >>> >>>>>>>>> use 0x01
> >>> >>>>>>>>>> > in one
> >>> >>>>>>>>>> > > > >> > >> deployment,
> >>> >>>>>>>>>> > > > >> > >> >> and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > means
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> message
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> is
> >>> >>>>>>> written
> >>> >>>>>>>>> into the
> >>> >>>>>>>>>> > > > broker, it
> >>> >>>>>>>>>> > > > >> > >> will be
> >>> >>>>>>>>>> > > > >> > >> >> >>> > saved on
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> specific
> >>> >>>>>>>>> namespace
> >>> >>>>>>>>>> > > (0x01).
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> If
> >>> >>> you
> >>> >>>>>>>>> were to
> >>> >>>>>>>>>> > mirror
> >>> >>>>>>>>>> > > that
> >>> >>>>>>>>>> > > > >> > message
> >>> >>>>>>>>>> > > > >> > >> >> into
> >>> >>>>>>>>>> > > > >> > >> >> >>> > another
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> accompany
> >>> >>>>>>>>> the
> >>> >>>>>>>>>> > message,
> >>> >>>>>>>>>> > > > right?
> >>> >>>>>>>>>> > > > >> > What
> >>> >>>>>>>>>> > > > >> > >> if
> >>> >>>>>>>>>> > > > >> > >> >> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> other
> >>> >>>>>>>>> cluster uses
> >>> >>>>>>>>>> > 0x57?
> >>> >>>>>>>>>> > > > They
> >>> >>>>>>>>>> > > > >> > won't
> >>> >>>>>>>>>> > > > >> > >> >> >>> > understand
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> I'm
> >>> >>>>> not
> >>> >>>>>>>>> sure
> >>> >>>>>>>>>> > that's an
> >>> >>>>>>>>>> > > > >> avoidable
> >>> >>>>>>>>>> > > > >> > >> >> >>> problem. I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > think it
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> in
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> order
> >>> >>>>> to
> >>> >>>>>>>>> share
> >>> >>>>>>>>>> > data, you
> >>> >>>>>>>>>> > > > have
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> also
> >>> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shared
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>> understanding of
> >>> >>>>>>>>>> > what the
> >>> >>>>>>>>>> > > > >> > >> namespaces
> >>> >>>>>>>>>> > > > >> > >> >> >>> mean.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> Which
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> sense,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>> because
> >>> >>>>>>> the
> >>> >>>>>>>>>> > alternate
> >>> >>>>>>>>>> > > > (sharing
> >>> >>>>>>>>>> > > > >> > >> >> *nothing*
> >>> >>>>>>>>>> > > > >> > >> >> >>> at
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> all)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> there
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> would
> >>> >>>>> be
> >>> >>>>>>>>> no way to
> >>> >>>>>>>>>> > > > understand
> >>> >>>>>>>>>> > > > >> > each
> >>> >>>>>>>>>> > > > >> > >> >> other.
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> -James
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>> Gwen
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> > On
> >>> >>>>> Tue,
> >>> >>>>>>>>> Nov 8,
> >>> >>>>>>>>>> > 2016 at
> >>> >>>>>>>>>> > > > 4:23
> >>> >>>>>>>>>> > > > >> > PM,
> >>> >>>>>>>>>> > > > >> > >> >> radai <
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> >>>>>>>>> radai.rosenblatt@gmail.com
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>> wrote:
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>> +1
> >>> >>>>> for
> >>> >>>>>>>>> sean's
> >>> >>>>>>>>>> > > document.
> >>> >>>>>>>>>> > > > it
> >>> >>>>>>>>>> > > > >> > >> covers
> >>> >>>>>>>>>> > > > >> > >> >> >>> pretty
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> much
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > all
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> trade-offs
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>>>>>> provides
> >>> >>>>>>>>>> > concrete
> >>> >>>>>>>>>> > > > figures
> >>> >>>>>>>>>> > > > >> to
> >>> >>>>>>>>>> > > > >> > >> argue
> >>> >>>>>>>>>> > > > >> > >> >> >>> about
> >>> >>>>>>>>>> > > > >> > >> >> >>> > :-)
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >>> >>
> >>> >>>>>>>>> (nit-picking -
> >>> >>>>>>>>>> > used
> >>> >>>>>>>>>> > > the
> >>> >>>>>>>>>> > > > >> same
> >>> >>>>>>>>>> > > > >> > >> xkcd
> >>> >>>>>>>>>> > > > >> > >> >> >>> twice,
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> also
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > trove
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > has
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
> >>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> >>>>> superceded
> >>> >>>>>>>>>> > > > >> > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >> > --
> >>> >>>>>>>>>> > > > >> > Gwen Shapira
> >>> >>>>>>>>>> > > > >> > Product Manager | Confluent
> >>> >>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
> @gwenshap
> >>> >>>>>>>>>> > > > >> > Follow us: Twitter | blog
> >>> >>>>>>>>>> > > > >> >
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> --
> >>> >>>>>>>>>> > > > >> *Todd Palino*
> >>> >>>>>>>>>> > > > >> Staff Site Reliability Engineer
> >>> >>>>>>>>>> > > > >> Data Infrastructure Streaming
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > > >> linkedin.com/in/toddpalino
> >>> >>>>>>>>>> > > > >>
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > > > --
> >>> >>>>>>>>>> > > > Gwen Shapira
> >>> >>>>>>>>>> > > > Product Manager | Confluent
> >>> >>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >>> >>>>>>>>>> > > > Follow us: Twitter | blog
> >>> >>>>>>>>>> > > >
> >>> >>>>>>>>>> > >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>> > The information contained in this email is strictly
> >>> >>>>> confidential
> >>> >>>>>>>>> and for
> >>> >>>>>>>>>> > the use of the addressee only, unless otherwise
> >>> indicated.
> >>> >>> If
> >>> >>>>> you
> >>> >>>>>>>>> are not
> >>> >>>>>>>>>> > the intended recipient, please do not read, copy,
> use or
> >>> >>>>> disclose
> >>> >>>>>>>>> to others
> >>> >>>>>>>>>> > this message or any attachment. Please also notify
> the
> >>> >>> sender
> >>> >>>>> by
> >>> >>>>>>>>> replying
> >>> >>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and
> >>> then
> >>> >>>>> delete
> >>> >>>>>>>>> the email
> >>> >>>>>>>>>> > and any copies of it. Opinions, conclusion (etc)
> that do
> >>> not
> >>> >>>>>>>>> relate to the
> >>> >>>>>>>>>> > official business of this company shall be
> understood as
> >>> >>>>> neither
> >>> >>>>>>>>> given nor
> >>> >>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets
> >>> Limited
> >>> >>> (a
> >>> >>>>>>>>> company
> >>> >>>>>>>>>> > registered in England and Wales, company number
> 04008957)
> >>> >>> and
> >>> >>>>> IG
> >>> >>>>>>>>> Index
> >>> >>>>>>>>>> > Limited (a company registered in England and Wales,
> >>> company
> >>> >>>>>>> number
> >>> >>>>>>>>>> > 01190902). Registered address at Cannon Bridge
> House, 25
> >>> >>>>> Dowgate
> >>> >>>>>>>>> Hill,
> >>> >>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register
> number
> >>> >>>>> 195355)
> >>> >>>>>>>>> and IG
> >>> >>>>>>>>>> > Index Limited (register number 114059) are
> authorised and
> >>> >>>>>>>>> regulated by the
> >>> >>>>>>>>>> > Financial Conduct Authority.
> >>> >>>>>>>>>> >
> >>> >>>>>>>>>>
> >>> >>>>>>>>>>
> >>> >>>>>>>>>> The information contained in this email is strictly
> confidential
> >>> >>> and
> >>> >>>>>>> for
> >>> >>>>>>>>> the use of the addressee only, unless otherwise indicated.
> If you
> >>> >>> are
> >>> >>>>>>> not
> >>> >>>>>>>>> the intended recipient, please do not read, copy, use or
> disclose
> >>> >>> to
> >>> >>>>>>> others
> >>> >>>>>>>>> this message or any attachment. Please also notify the
> sender by
> >>> >>>>>>> replying
> >>> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then
> delete
> >>> >>> the
> >>> >>>>>>> email
> >>> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
> >>> relate
> >>> >>>>> to
> >>> >>>>>>> the
> >>> >>>>>>>>> official business of this company shall be understood as
> neither
> >>> >>>>> given
> >>> >>>>>>> nor
> >>> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> >>> >>> company
> >>> >>>>>>>>> registered in England and Wales, company number 04008957)
> and IG
> >>> >>>>> Index
> >>> >>>>>>>>> Limited (a company registered in England and Wales, company
> >>> number
> >>> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >>> >>>>> Hill,
> >>> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> >>> >>>>> and IG
> >>> >>>>>>>>> Index Limited (register number 114059) are authorised and
> >>> regulated
> >>> >>>>> by
> >>> >>>>>>> the
> >>> >>>>>>>>> Financial Conduct Authority.
> >>> >>>>>>>>>
> >>> >>>>>>>
> >>> >>>>>
> >>> >>> The information contained in this email is strictly confidential
> and
> >>> for
> >>> >>> the use of the addressee only, unless otherwise indicated. If you
> are
> >>> not
> >>> >>> the intended recipient, please do not read, copy, use or disclose
> to
> >>> others
> >>> >>> this message or any attachment. Please also notify the sender by
> >>> replying
> >>> >>> to this email or by telephone (+44(020 7896 0011) and then delete
> the
> >>> email
> >>> >>> and any copies of it. Opinions, conclusion (etc) that do not
> relate to
> >>> the
> >>> >>> official business of this company shall be understood as neither
> given
> >>> nor
> >>> >>> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >>> >>> registered in England and Wales, company number 04008957) and IG
> Index
> >>> >>> Limited (a company registered in England and Wales, company number
> >>> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> >>> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and
> >>> IG
> >>> >>> Index Limited (register number 114059) are authorised and
> regulated by
> >>> the
> >>> >>> Financial Conduct Authority.
> >>> >>>
> >>> >
> >>> >
> >>>
> >>> The information contained in this email is strictly confidential and
> for
> >>> the use of the addressee only, unless otherwise indicated. If you are
> not
> >>> the intended recipient, please do not read, copy, use or disclose to
> others
> >>> this message or any attachment. Please also notify the sender by
> replying
> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >>> official business of this company shall be understood as neither given
> nor
> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >>> registered in England and Wales, company number 04008957) and IG Index
> >>> Limited (a company registered in England and Wales, company number
> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >>> Index Limited (register number 114059) are authorised and regulated by
> the
> >>> Financial Conduct Authority.
> >>>
> >
> >
> >
> > --
> > Nacho - Ignacio Solis - isolis@igso.net
> >
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Gwen Shapira <gw...@confluent.io>.
Can we also compromise on a multi-map?
One of my use-cases is tagging on multi-DC setup. It will be nice if
each "hop" can add "Tag" header with its own DC-name without having to
edit a single string that someone else wrote. For some financial
use-cases it will be a requirement.
Gwen
On Fri, Feb 10, 2017 at 12:19 PM, Ignacio Solis <is...@igso.net> wrote:
> I would be willing to compromise on Strings as keys if that moves all
> of us forward.
>
> Right now headers are blocking a few of the things we want to do.
>
> Nacho
>
> On Thu, Feb 9, 2017 at 2:14 PM, Jay Kreps <ja...@confluent.io> wrote:
>> I'm neutral positive on adding headers conceptually. I'm -1 on a proposal
>> using ints for header names. I just don't think a global namespace with
>> integer names is an acceptable end-user interface given that our goal is to
>> encourage a federated ecosystem of header users. Not trying to be a pain in
>> the ass, I just think it would be a bit of a wart.
>>
>> -Jay
>>
>> On Thu, Feb 9, 2017 at 12:12 AM, Michael Pearce <Mi...@ig.com>
>> wrote:
>>
>>> As we'd like to start work on this to make it into the next release in a
>>> few months.
>>>
>>> If we were to put this to the vote today who would actually give this a -1
>>> vote?
>>>
>>> Based on the number of identified usecases (some stronger than others)
>>>
>>> Also if we were to keep with int keys based on all the discussion thread
>>> for the pros vs cons who would give a -1 as it stands? Eg anyone believe it
>>> simply wouldn't work.
>>>
>>> If not so we make progress would it be sensible to put this to vote? So we
>>> can start work?
>>>
>>> Cheers
>>> Mike
>>>
>>>
>>> Sent using OWA for iPhone
>>> ________________________________________
>>> From: Jeroen van Disseldorp <je...@axual.io>
>>> Sent: Wednesday, February 1, 2017 2:43:45 PM
>>> To: dev@kafka.apache.org
>>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>>
>>> I'm lagging a bit here too, but want to respond to the usefulness of
>>> this KIP for two use cases: bidirectional mirroring and smart cluster
>>> switches.
>>>
>>> Bidirectional mirroring
>>>
>>> For a Dutch bank we have currently implemented bidirectional mirroring
>>> using our own headers, which we encode in the Value using a custom
>>> serializer. It encodes just one byte extra, containing a "copy/original"
>>> bit. When applications serialize an avro object, the bit is unset by
>>> default. Our mirroring mechanism subscribes to all topics, checks the
>>> bit, and when unset the bit is set and the message is forwarded to the
>>> other DC.
>>>
>>> The overhead of this is pretty low (1 byte) and it's cheap, since the
>>> mirroring mechanism does not need to fully deserialize the value, but
>>> just check+flip one bit in the byte[] value. On the receiving end, the
>>> mirroring mechanism sees the message for the second time, but notices
>>> the message is a copy already and ignores it.
>>>
>>> The mechanism has been in production since last year and it works really
>>> well. Its downside however is the incompatibility with avro serialized
>>> values, as the whole organization is mandated to use our own custom
>>> serde and we were forced to create a wrapper client library around the
>>> default kafka client. The callback we offer to applications is very
>>> similar to that proposed in the KIP, as every message our users get
>>> consists of <Metadata, K, V>.
>>>
>>> Additionally our header contains a message id and serialization
>>> timestamp (invented before 0.10.0.0). These fields could also be done
>>> away with when this KIP is implemented.
>>>
>>> Smart cluster switches
>>>
>>> Another case we are considering to use headers for is enabling "smart
>>> cluster switches" for consumers and producers in a way that would avoid
>>> duplicates and/or missed messages in most cases. This involves encoding
>>> a producer id (app name or id), producer instance id (uuid generated
>>> upon producer's start) and sequence number in every message. We expect
>>> to use one sequence per partition.
>>>
>>> These fields enable our consumers to keep track of which messages were
>>> seen without relying on offsets, which is a cluster-bound value.
>>> Consumers keep track of seen messages by keeping a small administration
>>> of ranges of sequence numbers per producer instance and partition. Since
>>> sequence numbers are always increasing per producer and partition, the
>>> memory consumption of such administration is pretty low, while being
>>> meaningful across clusters. After every poll() we can commit the
>>> administration to a topic in much the same way as we do with offsets.
>>> The difference is that this topic can be mirrored to the other cluster
>>> and upon a switch of consumer, all instances can use the topic to
>>> initialize themselves and skip all messages already processed.
>>>
>>> Hope these views help in the upcoming decision on this KIP.
>>>
>>> Regards
>>> Jeroen
>>>
>>>
>>>
>>> On 09/01/2017 20:40, Gwen Shapira wrote:
>>> > Mirroring + ClusterIDs:
>>> >
>>> > Kafka clusters already have an automatically generated ID that is
>>> > exposed to clients:
>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
>>> >
>>> > It sounds trivial to get MirrorMaker's consumer to grab this ID (which
>>> > always exists) from the source cluster and add it to the message
>>> > headers.
>>> >
>>> > Obviously adding stuff to the message makes it larger so this should
>>> > be a feature you should be able to turn on/off depending on the
>>> > replication requirements. If you need the feature - you pay the price.
>>> > I don't think there is an immediate need to put an effort into
>>> > shrinking the Cluster ID, but others can comment on that.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
>>> >> Hey Michael,
>>> >>
>>> >> Sorry for lagging, I've been out for the holiday break. What would help
>>> me
>>> >> would be to get to a consolidated list of use cases we've worked out and
>>> >> think add real value. I think getting down to the details really matters
>>> >> for these.
>>> >>
>>> >> The two lists I'm aware of are the one in the KIP and the one you gave
>>> in
>>> >> this email thread. Both seem to be more at the sketch level. Let me
>>> review
>>> >> the use cases and see if I have it right. If so let's get it in the
>>> design
>>> >> proposal.
>>> >>
>>> >> Your use cases:
>>> >>
>>> >> 1. Tracing IDs -- I get this use case and think I see how it would
>>> work.
>>> >> I think because the tracing is from a third party system there is
>>> some
>>> >> value to having a header versus a custom format to make this work
>>> fully
>>> >> transparently.
>>> >> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
>>> >> thinking below
>>> >> 3. End-to-end encryption -- I think what you are saying is that if
>>> you
>>> >> want "routing' fields that are accessible in transit without
>>> decrypting you
>>> >> need a scheme to leave this decrypted.I think the idea is to avoid
>>> >> decrypting to get a header, but I'm not sure if I get how this is
>>> better
>>> >> than a custom message format that doesn't encrypt the things you
>>> don't want
>>> >> encrypted since you have to be aware of the scheme anyway.
>>> >> 4. EOS -- I'm not convinced this is a good approach for EOS.
>>> >>
>>> >> Linked from KIP:
>>> >>
>>> >> 1. Schema-Registry-aware avro serialization -- not clear how this is
>>> >> better than keeping the schema id in the value
>>> >> 2. Polyglot kafka topics -- marginally better than a container
>>> format,
>>> >> but of course you still have to all agree on the the header and
>>> >> header=>serialization mapping so not sure if it's really better
>>> than a
>>> >> container format. Also not wild about polyglot topics :-)
>>> >> 3. End-to-End routing trail including mirroring --- I'd put this
>>> under
>>> >> tracing ids
>>> >> 4. Cross-service process tracing -- ditto
>>> >> 5. Signing -- I like this use case but it seems to depend on a
>>> feature
>>> >> we don't have and aren't proposing adding, right?
>>> >> 6. Per-record custom compression -- why do you want this?
>>> compressing
>>> >> per message won't work well compared to batch compression, right?
>>> >> 7. Large Message Support -- I think we can agree that although this
>>> >> might work it's clearly not the right way to support large messages
>>> in Kafka
>>> >>
>>> >> Matthias
>>> >>
>>> >> 1. Control messages --- Matthias, not sure if I get the motivation
>>> in
>>> >> streams for this. If we did want to do it, wouldn't it be easier to
>>> >> implement and by allowing users to send control messages (in the
>>> EOS sense)
>>> >> and adding a consumer config to allow either passing these on or
>>> >> suppressing them?
>>> >>
>>> >> So I if I were to summarize I'd list the use cases as follows:
>>> >>
>>> >> - Much better with headers: tracing IDs
>>> >> - Somewhat better: end-to-end encryption, polyglot topics
>>> >> - Don't buy: Mirroring (though perhaps could be made to work with
>>> >> changes to the proposal), EOS, large messages, compression
>>> >> - Not sure: Control messages
>>> >>
>>> >> Does this match where other people are at? From my point of view for
>>> >> something like this where we are changing the core data model we should
>>> >> ideally get a handful of use cases to the point where they are really
>>> well
>>> >> thought out and we know they can use the feature. I think we have one
>>> solid
>>> >> one with tracing IDs, and I think maybe mirroring could be another with
>>> >> some work. I'm less sure about the others.
>>> >>
>>> >> Bi-directional Mirroring
>>> >> I think bidirectional mirroring could potentially become a compelling
>>> use
>>> >> case if we an find a way to make it work. If I understand it, the
>>> proposal
>>> >> is to add a header like "origin-cluster" with the cluster id of the
>>> Kafka
>>> >> cluster. Then modify the mirroring tool to disallow mirroring a message
>>> >> back to it's own origin cluster. This basic idea is definitely workable
>>> and
>>> >> useful, but can we really get to a good implementation using headers?
>>> >> I imagine the approach would be to add this header in the producer if
>>> some
>>> >> config is set. There are two problems:
>>> >>
>>> >> - You add a pretty big honking cluster ID to each message. For small
>>> >> messages this maybe larger than the contents itself.
>>> >> - What happens if someone somewhere forgets to set that config on
>>> some
>>> >> client somewhere?
>>> >>
>>> >> If someone forgets to set the config in some client somewhere I guess
>>> you
>>> >> have various options in mirror maker:
>>> >>
>>> >> 1. drop the data (pretty bad, now your data is out of sync)
>>> >> 2. crash (pretty bad--how do you recover)
>>> >> 3. insert some default (pretty bad since would also lead to data
>>> loss if
>>> >> that's not right)
>>> >>
>>> >> Basically although I think a cluster tag could be a good way to do
>>> >> bi-directional mirroring, I'm not sure this naive approach I described
>>> >> would get us to a usable implementation. This could be fixable. I think
>>> if
>>> >> there was some scheme to check required headers on the server that could
>>> >> alleviate the missing header problem. Not really sure what to do about
>>> the
>>> >> message bloat problem, but maybe there could be some way to shrink the
>>> >> cluster id?
>>> >>
>>> >> Cheers,
>>> >>
>>> >> -Jay
>>> >>
>>> >>
>>> >> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Michael.Pearce@ig.com
>>> >
>>> >> wrote:
>>> >>
>>> >>> @Jay, Ismael and a few others.
>>> >>>
>>> >>> Based on the discussion on kip-87, and the additional use case by
>>> Matthias.
>>> >>>
>>> >>> Just for clarity do we now have consensus on the earlier points (1)
>>> >>> headers are useful (2) headers should be native in Kafka?
>>> >>>
>>> >>> Cheers
>>> >>> Mike
>>> >>> ________________________________________
>>> >>> From: radai <ra...@gmail.com>
>>> >>> Sent: Monday, December 19, 2016 5:01 PM
>>> >>> To: dev@kafka.apache.org
>>> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>> >>>
>>> >>> having said that im perfectly fine with something like a "no payload"
>>> flag
>>> >>> (as opposed to null payload), which would mean such a msg is not
>>> returned
>>> >>> from poll() unless the user code specifically "opts in" and would serve
>>> >>> only to carry headers (making it a control message)
>>> >>>
>>> >>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com>
>>> wrote:
>>> >>>
>>> >>>> if all you want is to tag "end of stream" wouldnt a header on the last
>>> >>> msg
>>> >>>> in the stream be enough?
>>> >>>>
>>> >>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
>>> matthias@confluent.io>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> The details about headers for control messages are still to define.
>>> But
>>> >>>>> yes, the idea is to have some common default behavior that clients
>>> would
>>> >>>>> need to implement.
>>> >>>>>
>>> >>>>> The point is, that "regular headers" add meta data to regular
>>> messages.
>>> >>>>> Thus, those messages will be returned to the user via .poll(). And
>>> after
>>> >>>>> the message is received the user can check if meta data is present
>>> and
>>> >>>>> read it.
>>> >>>>>
>>> >>>>> For control messages, we do not want those to pop up via .poll() as
>>> >>>>> those are no regular messages. A client would need to opt-in to see
>>> >>>>> those messages (either via poll() or maybe a callback). Thus, we need
>>> >>>>> some special (standardized) header IDs that indicate control messages
>>> >>>>> that should not be returned to the user via poll() by default.
>>> >>>>>
>>> >>>>>
>>> >>>>> -Matthias
>>> >>>>>
>>> >>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
>>> >>>>>> Matthias,
>>> >>>>>>
>>> >>>>>> Thanks for your input. I'm +1 on control messages as they seem to
>>> be
>>> >>>>> the
>>> >>>>>> simplest way to implement watermarks (
>>> >>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
>>> ),
>>> >>> a
>>> >>>>>> feature that would add a lot of value to Kafka Streams IMHO.
>>> >>>>>>
>>> >>>>>> Your argument that the control-message indicator belongs in the
>>> >>>>> client-only
>>> >>>>>> section of the record format make sense. Just to make sure I
>>> >>>>> understand,
>>> >>>>>> are you suggesting that control messages would be indicated by a
>>> >>>>> standard
>>> >>>>>> reserved header? By standard, I mean that ALL Kafka consumers would
>>> >>>>> know
>>> >>>>>> to handle these messages differently (possibly just ignoring them).
>>> >>>>> This
>>> >>>>>> would need to be added to the specification of the consumer protocol
>>> >>> so
>>> >>>>>> that all Kafka clients implement it, right? I think it's a good
>>> idea
>>> >>>>> but
>>> >>>>>> just checking.
>>> >>>>>>
>>> >>>>>> Cheers,
>>> >>>>>>
>>> >>>>>> Roger
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>>> >>> matthias@confluent.io
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> Yes and no. I did overload the term "control message".
>>> >>>>>>>
>>> >>>>>>> EOS control messages are for client-broker communication and thus
>>> >>> never
>>> >>>>>>> exposed to any application. And I think this is a good design
>>> because
>>> >>>>>>> broker needs to understand those control messages. Thus, this
>>> should
>>> >>> be
>>> >>>>>>> a protocol change.
>>> >>>>>>>
>>> >>>>>>> The type of control messages I have in mind are for client-client
>>> >>>>>>> (application-application) communication and the broker is agnostic
>>> to
>>> >>>>>>> them. Thus, it should not be a protocol change.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> -Matthias
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On 12/14/16 9:42 AM, radai wrote:
>>> >>>>>>>> arent control messages getting pushed as their own top level
>>> >>> protocol
>>> >>>>>>>> change (and a fairly massive one) for the transactions KIP ?
>>> >>>>>>>>
>>> >>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>>> >>>>> matthias@confluent.io>
>>> >>>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> Hi,
>>> >>>>>>>>>
>>> >>>>>>>>> I want to add a completely new angle to this discussion. For
>>> this,
>>> >>> I
>>> >>>>>>>>> want to propose an extension for the headers feature that enables
>>> >>> new
>>> >>>>>>>>> uses cases -- and those new use cases might convince people to
>>> >>>>> support
>>> >>>>>>>>> headers (of course including the larger scoped proposal).
>>> >>>>>>>>>
>>> >>>>>>>>> Extended Proposal:
>>> >>>>>>>>>
>>> >>>>>>>>> Allow messages with a certain header key to be special "control
>>> >>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an
>>> application
>>> >>>>> via
>>> >>>>>>>>> .poll().
>>> >>>>>>>>>
>>> >>>>>>>>> Thus, a consumer client would automatically skip over those
>>> >>>>> messages. If
>>> >>>>>>>>> an application knows about embedded control messages, it can
>>> "sing
>>> >>>>> up"
>>> >>>>>>>>> to those messages by the consumer client and either get a
>>> callback
>>> >>> or
>>> >>>>>>>>> the consumer auto-drop for this messages gets disabled (allowing
>>> to
>>> >>>>>>>>> consumer those messages via poll()).
>>> >>>>>>>>>
>>> >>>>>>>>> (The details need further considerations/discussion. I just want
>>> to
>>> >>>>>>>>> sketch the main idea.)
>>> >>>>>>>>>
>>> >>>>>>>>> Usage:
>>> >>>>>>>>>
>>> >>>>>>>>> There is a shared topic (ie, used by multiple applications) and a
>>> >>>>>>>>> producer application wants to embed a special message in the
>>> topic
>>> >>>>> for a
>>> >>>>>>>>> dedicated consumer application. Because only one application will
>>> >>>>>>>>> understand this message, it cannot be a regular message as this
>>> >>> would
>>> >>>>>>>>> break all applications that do not understand this message. The
>>> >>>>> producer
>>> >>>>>>>>> application would set a special metadata key and no consumer
>>> >>>>> application
>>> >>>>>>>>> would see this control message by default because they did not
>>> >>> enable
>>> >>>>>>>>> their consumer client to return this message in poll() (and the
>>> >>>>> client
>>> >>>>>>>>> would just drop this message with special metadata key). Only the
>>> >>>>> single
>>> >>>>>>>>> application that should receive this message, will subscribe to
>>> >>> this
>>> >>>>>>>>> message on its consumer client and process it.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> Concrete Use Case: Kafka Streams
>>> >>>>>>>>>
>>> >>>>>>>>> In Kafka Streams, we would like to propagate "control messages"
>>> >>> from
>>> >>>>>>>>> subtopology to subtopology. There are multiple scenarios for
>>> which
>>> >>>>> this
>>> >>>>>>>>> would be useful. For example, currently we do not guarantee a
>>> >>>>>>>>> "consistent shutdown" of an application. By this, I mean that
>>> input
>>> >>>>>>>>> records might not be completely processed by the whole topology
>>> >>>>> because
>>> >>>>>>>>> the application shutdown happens "in between" and an intermediate
>>> >>>>> result
>>> >>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user would
>>> see
>>> >>>>> an
>>> >>>>>>>>> committed offset of the source topic of the application, but no
>>> >>>>>>>>> corresponding result record in the output topic.
>>> >>>>>>>>>
>>> >>>>>>>>> Having "shutdown markers" would allow us, to first stop the
>>> >>> upstream
>>> >>>>>>>>> subtopology and write this marker into the intermediate topic and
>>> >>> the
>>> >>>>>>>>> downstream subtopology would only shut down itself after is sees
>>> >>> the
>>> >>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
>>> >>>>>>>>> "in-flight" messages got stuck in intermediate topics.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch
>>> Processing).
>>> >>>>>>>>> There was a discussion about the proposed metadata topic, and we
>>> >>>>> could
>>> >>>>>>>>> avoid this metadata topic if we would have "control messages".
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> Right now, we cannot insert an "application control message"
>>> >>> because
>>> >>>>>>>>> Kafka Streams does not own all topics it read/writes and thus
>>> might
>>> >>>>>>>>> break other consumer application (as described above) if we
>>> inject
>>> >>>>>>>>> random messages that are not understood by other apps.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> Of course, one can work around "embedded control messaged" by
>>> using
>>> >>>>> an
>>> >>>>>>>>> additional topic to propagate control messaged between
>>> application
>>> >>>>> (as
>>> >>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
>>> >>>>> there
>>> >>>>>>>>> are major concerns about adding this metadata topic in the KIP
>>> and
>>> >>>>> this
>>> >>>>>>>>> shows that other application that need a similar pattern might
>>> >>> profit
>>> >>>>>>>>> from topic embedded "control messages", too.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> One last important consideration: those "control messages" are
>>> used
>>> >>>>> for
>>> >>>>>>>>> client to client communication and are not understood by the
>>> >>> broker.
>>> >>>>>>>>> Thus, those messages should not be enabled within the message
>>> >>> format
>>> >>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
>>> >>>>> headers
>>> >>>>>>>>> would be a nice way to implement them. Because KIP-82 did
>>> consider
>>> >>>>> key
>>> >>>>>>>>> namespaces for metatdata keys, this extension should not be an
>>> own
>>> >>>>> KIP
>>> >>>>>>>>> but should be included in KIP-82 to reserve a namespace for
>>> >>> "control
>>> >>>>>>>>> message" in the first place.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> Sorry for the long email... Looking forward to your feedback.
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> -Matthias
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>>> >>>>>>>>>> Hi Jun
>>> >>>>>>>>>>
>>> >>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
>>> >>>>> (HTTP/JMS/
>>> >>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
>>> >>> (though I
>>> >>>>>>>>> believe it contains the end2end uuid embedded in this id), on
>>> >>>>> receiving
>>> >>>>>>> the
>>> >>>>>>>>> message at the receiving JVM the apm code takes this out, and
>>> >>>>> continues
>>> >>>>>>> its
>>> >>>>>>>>> tracing on the that new thread. Both JVM's (and other languages
>>> the
>>> >>>>> APM
>>> >>>>>>>>> tool supports) send this data async back to the central
>>> controllers
>>> >>>>>>> where
>>> >>>>>>>>> the stiching togeather occurs. For this they need some header
>>> space
>>> >>>>> for
>>> >>>>>>>>> them to put this id.
>>> >>>>>>>>>> 101) Yes indeed we have a business transaction Id in the
>>> payload.
>>> >>>>>>> Though
>>> >>>>>>>>> this is a system level tracing, that we need to have marry up.
>>> Also
>>> >>>>> as
>>> >>>>>>> per
>>> >>>>>>>>> note on end2end encryption we'd be unable to prove the flow if
>>> the
>>> >>>>>>> payload
>>> >>>>>>>>> is encrypted as we'd not have access to this at certain points of
>>> >>> the
>>> >>>>>>> flow
>>> >>>>>>>>> through the infrastructure/platform.
>>> >>>>>>>>>>
>>> >>>>>>>>>> 103) As said we use this mechanism in IG very successfully, as
>>> >>>>> stated
>>> >>>>>>>>> per key we guarantee the transaction producing app to handle the
>>> >>>>>>>>> transaction of a key at one DC unless at point of critical
>>> failure
>>> >>>>>>> where we
>>> >>>>>>>>> have to flip processing to another. We care about key ordering.
>>> >>>>>>>>>> I disagree on the offset comment for the partition solution
>>> unless
>>> >>>>> you
>>> >>>>>>>>> do full ISR, or expensive full XA transactions even with
>>> partitions
>>> >>>>> you
>>> >>>>>>>>> cannot fully guarantee offsets would match.
>>> >>>>>>>>>> 105) Very much so, I need to have access at the platform level
>>> to
>>> >>>>> the
>>> >>>>>>>>> other meta data all mentioned, without having to need to have
>>> >>> access
>>> >>>>> to
>>> >>>>>>> the
>>> >>>>>>>>> encryption keys of the payload.
>>> >>>>>>>>>> 106)
>>> >>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to
>>> have
>>> >>> a
>>> >>>>>>>>> global producerId register which would be very hard to
>>> >>>>> enforce/ensure is
>>> >>>>>>>>> current and correct, just to understand the message origins of
>>> its
>>> >>>>>>>>> region/az/cluster for routing.
>>> >>>>>>>>>> The client wrapper version, producerId can be the same, as
>>> >>> obviously
>>> >>>>>>> the
>>> >>>>>>>>> producer could upgrade its wrapper, as such we need to know what
>>> >>>>> wrapper
>>> >>>>>>>>> version the message is created with.
>>> >>>>>>>>>> Likewise the IP address, as stated we can have our producer
>>> move,
>>> >>>>> where
>>> >>>>>>>>> its IP would change.
>>> >>>>>>>>>> 107)
>>> >>>>>>>>>> UUID is set on the message by interceptors before actual
>>> producer
>>> >>>>>>>>> transport send. This is for platform level message dedupe
>>> >>> guarantee,
>>> >>>>> the
>>> >>>>>>>>> business payload should be agnostic to this. Please see
>>> >>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
>>> >>>>> ection.html
>>> >>>>>>>>> note this is not touching business payloads.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> Hi, Michael,
>>> >>>>>>>>>>
>>> >>>>>>>>>> Thanks for the reply. I find it very helpful.
>>> >>>>>>>>>>
>>> >>>>>>>>>> Data lineage:
>>> >>>>>>>>>> 100. I'd like to understand the APM use case a bit more. It
>>> >>>>> sounds
>>> >>>>>>>>> like
>>> >>>>>>>>>> that those APM plugins can generate a transaction id that
>>> we
>>> >>>>> could
>>> >>>>>>>>>> potentially put in the header of every message. How would
>>> you
>>> >>>>>>>>> typically
>>> >>>>>>>>>> make use of such transaction ids? Are there other metadata
>>> >>>>>>>>> associated with
>>> >>>>>>>>>> the transaction id and if so, how are they propagated
>>> >>>>> downstream?
>>> >>>>>>>>>> 101. For the finance use case, if the concept of
>>> transaction
>>> >>> is
>>> >>>>>>>>> important,
>>> >>>>>>>>>> wouldn't it be typically included in the message payload
>>> >>>>> instead of
>>> >>>>>>>>> as an
>>> >>>>>>>>>> optional header field?
>>> >>>>>>>>>>
>>> >>>>>>>>>> 102. The data lineage that Altas and Navigator support
>>> seems
>>> >>> to
>>> >>>>> be
>>> >>>>>>>>> at the
>>> >>>>>>>>>> dataset level, not per record level? So, not sure if per
>>> >>> message
>>> >>>>>>>>> headers
>>> >>>>>>>>>> are relevant there.
>>> >>>>>>>>>>
>>> >>>>>>>>>> Mirroring:
>>> >>>>>>>>>> 103. The benefit of using separate partitions is that it
>>> >>>>>>> potentially
>>> >>>>>>>>> makes
>>> >>>>>>>>>> it easy to preserve offsets during mirroring. This will
>>> make
>>> >>> it
>>> >>>>>>>>> easier for
>>> >>>>>>>>>> consumer to switch clusters. Currently, the consumers can
>>> >>> switch
>>> >>>>>>>>> clusters
>>> >>>>>>>>>> by using the timestampToOffset() api, but it has to deal
>>> with
>>> >>>>>>>>> duplicates.
>>> >>>>>>>>>> Good point on the issue with log compact and I am not sure
>>> how
>>> >>>>> to
>>> >>>>>>>>> address
>>> >>>>>>>>>> this. However, even if we mirror into the existing
>>> partitions,
>>> >>>>> the
>>> >>>>>>>>> ordering
>>> >>>>>>>>>> for messages generated from different clusters seems
>>> >>>>>>>>> non-deterministic
>>> >>>>>>>>>> anyway. So, it seems that the consumers already have to
>>> deal
>>> >>>>> with
>>> >>>>>>>>> that? If
>>> >>>>>>>>>> a topic is compacted, does that mean which messages are
>>> >>>>> preserved
>>> >>>>>>> is
>>> >>>>>>>>> also
>>> >>>>>>>>>> non-deterministic across clusters?
>>> >>>>>>>>>>
>>> >>>>>>>>>> 104. Good point on partition key.
>>> >>>>>>>>>>
>>> >>>>>>>>>> End-to-end encryption:
>>> >>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are
>>> headers
>>> >>>>>>>>> useful there?
>>> >>>>>>>>>> Auditing:
>>> >>>>>>>>>> 106. It seems other than the UUID, all other metadata are
>>> per
>>> >>>>>>>>> producer?
>>> >>>>>>>>>> EOS:
>>> >>>>>>>>>> 107. How are those UUIDs generated? I am not sure if they
>>> can
>>> >>> be
>>> >>>>>>>>> generated
>>> >>>>>>>>>> in the producer library. An application may send messages
>>> >>>>> through a
>>> >>>>>>>>> load
>>> >>>>>>>>>> balancer and on retry, the same message could be routed to
>>> a
>>> >>>>>>>>> different
>>> >>>>>>>>>> producer instance. So, it seems that the application has to
>>> >>>>>>> generate
>>> >>>>>>>>> the
>>> >>>>>>>>>> UUIDs. In that case, shouldn't the application just put the
>>> >>>>> UUID in
>>> >>>>>>>>> the
>>> >>>>>>>>>> payload?
>>> >>>>>>>>>>
>>> >>>>>>>>>> Thanks,
>>> >>>>>>>>>>
>>> >>>>>>>>>> Jun
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>>> >>>>>>>>> Michael.Pearce@ig.com>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> > Hi Jun.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Per Transaction Tracing / Data Lineage.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > As Stated in the KIP this has the first use case of how
>>> many
>>> >>>>> APM
>>> >>>>>>>>> tools now
>>> >>>>>>>>>> > work.
>>> >>>>>>>>>> > I would find it impossible for any one to argue this is
>>> not
>>> >>>>>>>>> important or a
>>> >>>>>>>>>> > niche market as it has its own gartner report for this
>>> >>> space.
>>> >>>>>>> Such
>>> >>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular
>>> are
>>> >>>>> but a
>>> >>>>>>>>> few.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Likewise these APM tools can help very rapidly track down
>>> >>>>> issues
>>> >>>>>>>>> and
>>> >>>>>>>>>> > automatically capture metrics, perform actions based on
>>> >>>>>>> unexpected
>>> >>>>>>>>> behavior
>>> >>>>>>>>>> > to auto recover services.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Before mentioning looking at aggregated stats, in these
>>> >>> cases
>>> >>>>>>> where
>>> >>>>>>>>>> > actually on critical flows we cannot afford to have
>>> >>> aggregated
>>> >>>>>>>>> rolled up
>>> >>>>>>>>>> > stats only.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > With the APM tool we use its actually able to detect a
>>> >>> single
>>> >>>>>>>>> transaction
>>> >>>>>>>>>> > failure and capture the thread traces in the JVM where it
>>> >>>>> failed
>>> >>>>>>>>> and
>>> >>>>>>>>>> > everything for us, to the point it sends us alerts where
>>> we
>>> >>>>> have
>>> >>>>>>>>> this
>>> >>>>>>>>>> > giving the line number of the code that caused it, the
>>> >>>>>>> transaction
>>> >>>>>>>>> trace
>>> >>>>>>>>>> > through all the services and endpoints (supported) upto
>>> the
>>> >>>>> point
>>> >>>>>>>>> of
>>> >>>>>>>>>> > failure, it can also capture the data in and out (so we
>>> can
>>> >>>>>>>>> replay).
>>> >>>>>>>>>> > Because atm Kafka doesn't support us being able to stich
>>> in
>>> >>>>> these
>>> >>>>>>>>> tracing
>>> >>>>>>>>>> > transaction ids natively, we cannot get these benefits as
>>> >>>>> such is
>>> >>>>>>>>> limiting
>>> >>>>>>>>>> > our ability support apps and monitor them to the same
>>> >>>>> standards
>>> >>>>>>> we
>>> >>>>>>>>> come to
>>> >>>>>>>>>> > expect when on a kafka flow.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > This actually ties in with Data Lineage, as the same
>>> tracing
>>> >>>>> can
>>> >>>>>>>>> be used
>>> >>>>>>>>>> > to back stich this. Essentially many times due to the
>>> sums
>>> >>> of
>>> >>>>>>> money
>>> >>>>>>>>>> > involved there are disputes, and typically as a financial
>>> >>>>>>>>> institute the
>>> >>>>>>>>>> > easiest and cleanest way to prove when disputes arise is
>>> to
>>> >>>>>>>>> present the
>>> >>>>>>>>>> > actual flow and processes involved in a transaction.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Likewise as Hadoop matures its evident this case is
>>> >>>>> important, as
>>> >>>>>>>>> tools
>>> >>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera
>>> led)
>>> >>>>> are
>>> >>>>>>>>> evident
>>> >>>>>>>>>> > also I believe the importance here is very much NOT just
>>> a
>>> >>>>>>>>> financial issue.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > From a MDM point of view any company wanting to care
>>> about
>>> >>>>> Data
>>> >>>>>>>>> Quality
>>> >>>>>>>>>> > and Data Governance - Data Lineage is a key piece in this
>>> >>>>> puzzle.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > RE Mirroring,
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > As per the KIP in-fact this is exactly what we do re
>>> cluster
>>> >>>>> id,
>>> >>>>>>>>> to mirror
>>> >>>>>>>>>> > a network of clusters between AZ's / Regions. We know a
>>> >>>>>>>>> transaction for a
>>> >>>>>>>>>> > key will be done within a AZ/Region, as such we know the
>>> >>>>> write
>>> >>>>>>> to
>>> >>>>>>>>> kafka
>>> >>>>>>>>>> > would be ordered per key. But we need eventual view of
>>> that
>>> >>>>>>> across
>>> >>>>>>>>> in our
>>> >>>>>>>>>> > other regions/az's. When we have complete AZ or Region
>>> >>>>> failure we
>>> >>>>>>>>> know
>>> >>>>>>>>>> > there will be a brief interruption whilst those
>>> transactions
>>> >>>>> are
>>> >>>>>>>>> moved to
>>> >>>>>>>>>> > another region but we expect after it to continue.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > As mentioned having separate Partions to do this starts
>>> to
>>> >>> get
>>> >>>>>>>>>> > ugly/complicated for us:
>>> >>>>>>>>>> > how would I do compaction where a key is in two
>>> partitions?
>>> >>>>>>>>>> > How do we balance consumers so where multiple partitions
>>> >>> with
>>> >>>>> the
>>> >>>>>>>>> same key
>>> >>>>>>>>>> > goto the same consumer
>>> >>>>>>>>>> > What do you do if cluster 1 has 5 partitions but cluster
>>> 20
>>> >>>>> has
>>> >>>>>>> 10
>>> >>>>>>>>> because
>>> >>>>>>>>>> > its larger kit in our more core DC's, as such key to
>>> >>> partition
>>> >>>>>>>>> mappings for
>>> >>>>>>>>>> > consumers get even more complicated.
>>> >>>>>>>>>> > What do you do if we add or remove a complete region
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Where as simple mirror will work we just need to ensure
>>> we
>>> >>>>> don't
>>> >>>>>>>>> have a
>>> >>>>>>>>>> > cycle which we can do with clusterId.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > We even have started to look at shortest path mirror
>>> routing
>>> >>>>>>> based
>>> >>>>>>>>> on
>>> >>>>>>>>>> > clusterId, if we also had the region and az info on the
>>> >>>>>>> originating
>>> >>>>>>>>>> > message, this we have not implemented but some ideas come
>>> >>> from
>>> >>>>>>>>> network
>>> >>>>>>>>>> > routing, and also the dispatcher router in apache qpid.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Also we need to have data perimeters e.g. certain data
>>> >>> cannot
>>> >>>>>>> leave
>>> >>>>>>>>>> > certain countries borders. We want this all automated so
>>> >>> that
>>> >>>>> at
>>> >>>>>>>>> the
>>> >>>>>>>>>> > platform level without having to touch or look at the
>>> >>> business
>>> >>>>>>>>> data inside
>>> >>>>>>>>>> > we can have headers we can put tags into so that we can
>>> >>> ensure
>>> >>>>>>>>> this doesn't
>>> >>>>>>>>>> > occur when we mirror. (actually links in to data lineage
>>> /
>>> >>>>>>> tracing
>>> >>>>>>>>> as again
>>> >>>>>>>>>> > we need to tag messages at a platform level) Examples
>>> are we
>>> >>>>> are
>>> >>>>>>>>> not
>>> >>>>>>>>>> > allowed Private customer details to leave Switzerland,
>>> yet
>>> >>> we
>>> >>>>>>> need
>>> >>>>>>>>> those
>>> >>>>>>>>>> > systems integrated.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Lastly around mirroring we have a partionKey field, as
>>> the
>>> >>> key
>>> >>>>>>>>> used for
>>> >>>>>>>>>> > portioning logic != compaction key all the time but we
>>> want
>>> >>> to
>>> >>>>>>>>> preserve it
>>> >>>>>>>>>> > for when we mirror so that if source cluster partition
>>> count
>>> >>>>> !=
>>> >>>>>>>>> destination
>>> >>>>>>>>>> > cluster partition count we can honour the same
>>> partitioning
>>> >>>>>>> logic.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > RE End 2 End encryption
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > As I believe mentioned just before, the solution you
>>> mention
>>> >>>>> just
>>> >>>>>>>>> doesn't
>>> >>>>>>>>>> > cut the mustard these days with many regulators. An
>>> >>> operations
>>> >>>>>>>>> person with
>>> >>>>>>>>>> > access to the box should not be able to have access to
>>> the
>>> >>>>> data.
>>> >>>>>>>>> Many now
>>> >>>>>>>>>> > actually impose quite literally the implementation
>>> expected
>>> >>>>> being
>>> >>>>>>>>> end2end
>>> >>>>>>>>>> > encryption for certain data (Singapore for us is one
>>> that I
>>> >>> am
>>> >>>>>>>>> most aware
>>> >>>>>>>>>> > of). In fact we're even now needing encrypt the data and
>>> >>> store
>>> >>>>>>> the
>>> >>>>>>>>> keys in
>>> >>>>>>>>>> > HSM modules.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Likewise the performance penalty on encrypting
>>> decrypting as
>>> >>>>> you
>>> >>>>>>>>> produce
>>> >>>>>>>>>> > over wire, then again encrypt decrypt as the data is
>>> stored
>>> >>> on
>>> >>>>>>> the
>>> >>>>>>>>> brokers
>>> >>>>>>>>>> > disks and back again, then again encrypted and decrypted
>>> >>> back
>>> >>>>>>> over
>>> >>>>>>>>> the wire
>>> >>>>>>>>>> > each time for each consumer all adds up, ignoring this
>>> >>>>> doubling
>>> >>>>>>>>> with mirror
>>> >>>>>>>>>> > makers etc. simply encrypting the value once on write by
>>> the
>>> >>>>>>>>> client and
>>> >>>>>>>>>> > again decrypting on consume by the consumer is far more
>>> >>>>>>>>> performant, but
>>> >>>>>>>>>> > then the routing and platform meta data needs to be
>>> separate
>>> >>>>>>> (thus
>>> >>>>>>>>> headers)
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > RE Auditing:
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Our Auditing needs are:
>>> >>>>>>>>>> > Producer Id,
>>> >>>>>>>>>> > Origin Cluster Id that message first produced into
>>> >>>>>>>>>> > Origin AZ - agreed we can derive this if we have cluster
>>> id,
>>> >>>>> but
>>> >>>>>>>>> it makes
>>> >>>>>>>>>> > resolving this for audit reporting a lot easier.
>>> >>>>>>>>>> > Origin Region - agreed we can derive this if we have
>>> cluster
>>> >>>>> id,
>>> >>>>>>>>> but it
>>> >>>>>>>>>> > makes resolving this for audit reporting a lot easier.
>>> >>>>>>>>>> > Unique Message Identification (this is not the same as
>>> >>>>>>> transaction
>>> >>>>>>>>>> > tracing) - note offset and partition are not the same, as
>>> >>>>> when we
>>> >>>>>>>>> mirror or
>>> >>>>>>>>>> > have for what ever system failure duplicate send,
>>> >>>>>>>>>> > Custom Client wrapper version (where organizations have
>>> to
>>> >>>>> wrap
>>> >>>>>>>>> the kafka
>>> >>>>>>>>>> > client for added features) so we know what version of the
>>> >>>>> wrapper
>>> >>>>>>>>> is used
>>> >>>>>>>>>> > Producer IP address (in case of clients being in our
>>> vm/open
>>> >>>>>>> stack
>>> >>>>>>>>> infra
>>> >>>>>>>>>> > where they can move around, producer id will stay the
>>> same
>>> >>> but
>>> >>>>>>>>> this would
>>> >>>>>>>>>> > change)
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > RE Once and only once delivery case
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Using the same Message UUID for auditing we can achieve
>>> this
>>> >>>>>>> quite
>>> >>>>>>>>> simply.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > As per how some other brokers do this (cough qpid,
>>> artemis)
>>> >>>>>>>>> message uuid
>>> >>>>>>>>>> > are used to dedupe where message is sent and produced but
>>> >>> the
>>> >>>>>>>>> client didn't
>>> >>>>>>>>>> > receive the ack, and there for replays the send, by
>>> having a
>>> >>>>>>>>> unique message
>>> >>>>>>>>>> > id per message, this can be filtered out, on consumers
>>> where
>>> >>>>>>>>> message
>>> >>>>>>>>>> > delivery may occur twice for what ever reasons a message
>>> >>> uuid
>>> >>>>> can
>>> >>>>>>>>> be used
>>> >>>>>>>>>> > to remove duplicates being deliverd , like wise we can do
>>> >>>>> this in
>>> >>>>>>>>> the
>>> >>>>>>>>>> > mirrormakers so if we detect a dupe message we can avoid
>>> >>>>>>>>> replicating it.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Cheers
>>> >>>>>>>>>> > Mike
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
>>> wrote:
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Since this KIP affects message format, wire protocol,
>>> >>>>> apis, I
>>> >>>>>>>>> think
>>> >>>>>>>>>> > it's
>>> >>>>>>>>>> > worth spending a bit more time to nail down the
>>> concrete
>>> >>>>> use
>>> >>>>>>>>> cases. It
>>> >>>>>>>>>> > would be bad if we add this feature, but when start
>>> >>>>>>>>> implementing it
>>> >>>>>>>>>> > for say
>>> >>>>>>>>>> > mirroring, we then realize that header is not the
>>> best
>>> >>>>>>>>> approach.
>>> >>>>>>>>>> > Initially,
>>> >>>>>>>>>> > I thought I was convinced of the use cases of headers
>>> >>> and
>>> >>>>> was
>>> >>>>>>>>> trying to
>>> >>>>>>>>>> > write down a few use cases to convince others. That's
>>> >>>>> when I
>>> >>>>>>>>> became
>>> >>>>>>>>>> > less
>>> >>>>>>>>>> > certain. For me to be convinced, I just want to see
>>> two
>>> >>>>>>> strong
>>> >>>>>>>>> use
>>> >>>>>>>>>> > cases
>>> >>>>>>>>>> > (instead of 10 maybe use cases) in the third-party
>>> >>> space.
>>> >>>>> The
>>> >>>>>>>>> reason is
>>> >>>>>>>>>> > that when we discussed the use cases within a
>>> company,
>>> >>>>> often
>>> >>>>>>>>> it ends
>>> >>>>>>>>>> > with
>>> >>>>>>>>>> > "we can't force everyone to use this standard since
>>> we
>>> >>> may
>>> >>>>>>>>> have to
>>> >>>>>>>>>> > integrate with third-party tools".
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > At present, I am not sure why headers are useful for
>>> >>>>> things
>>> >>>>>>>>> like
>>> >>>>>>>>>> > schemaId
>>> >>>>>>>>>> > or encryption. In order to do anything useful to the
>>> >>>>> value,
>>> >>>>>>>>> one needs
>>> >>>>>>>>>> > to
>>> >>>>>>>>>> > know the schemaId or how data is encrypted, but
>>> header
>>> >>> is
>>> >>>>>>>>> optional.
>>> >>>>>>>>>> > But, I
>>> >>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
>>> >>> provides
>>> >>>>>>> more
>>> >>>>>>>>> details
>>> >>>>>>>>>> > on
>>> >>>>>>>>>> > the argument.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > I am not very sure header is the best approach for
>>> >>>>> mirroring
>>> >>>>>>>>> either. If
>>> >>>>>>>>>> > someone has thought about this more, I'd be happy to
>>> >>> hear.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > I can see the data lineage use case. I am just not
>>> sure
>>> >>>>> how
>>> >>>>>>>>> widely
>>> >>>>>>>>>> > applicable this is. If someone familiar with this
>>> space
>>> >>>>> can
>>> >>>>>>>>> justify
>>> >>>>>>>>>> > this is
>>> >>>>>>>>>> > a significant use case, say in the finance industry,
>>> >>> this
>>> >>>>>>>>> would be a
>>> >>>>>>>>>> > strong
>>> >>>>>>>>>> > use case.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > I can see the auditing use case. I am just not sure
>>> if a
>>> >>>>>>> native
>>> >>>>>>>>>> > producer id
>>> >>>>>>>>>> > solves that problem. If there are additional metadata
>>> >>>>> that's
>>> >>>>>>>>> worth
>>> >>>>>>>>>> > collecting but not covered by the producer id, that
>>> >>> would
>>> >>>>>>> make
>>> >>>>>>>>> this a
>>> >>>>>>>>>> > strong use case.
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Thanks,
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > Jun
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>>> >>>>>>>>> radai.rosenblatt@gmail.com>
>>> >>>>>>>>>> > wrote:
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > > this KIP is about enabling headers, nothing more
>>> >>> nothing
>>> >>>>>>>>> less - so
>>> >>>>>>>>>> > no,
>>> >>>>>>>>>> > > broker-side use of headers is not in the KIP scope.
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> > > obviously though, once you have headers potential
>>> use
>>> >>>>> cases
>>> >>>>>>>>> could
>>> >>>>>>>>>> > include
>>> >>>>>>>>>> > > broker-side header-aware interceptors (which would
>>> be
>>> >>>>> the
>>> >>>>>>>>> topic of
>>> >>>>>>>>>> > other
>>> >>>>>>>>>> > > future KIPs).
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> > > a trivially clear use case (to me) would be using
>>> such
>>> >>>>>>>>> broker-side
>>> >>>>>>>>>> > > interceptors to enforce compliance with
>>> organizational
>>> >>>>>>>>> policies - it
>>> >>>>>>>>>> > would
>>> >>>>>>>>>> > > make our SREs lives much easier if instead of
>>> >>>>> retroactively
>>> >>>>>>>>>> > discovering
>>> >>>>>>>>>> > > "rogue" topics/users those messages would have been
>>> >>>>>>> rejected
>>> >>>>>>>>>> > up-front.
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> > > the kafka broker code is lacking any such
>>> >>> extensibility
>>> >>>>>>>>> support
>>> >>>>>>>>>> > (beyond
>>> >>>>>>>>>> > > maybe authorizer) which is why these use cases were
>>> >>> left
>>> >>>>>>> out
>>> >>>>>>>>> of the
>>> >>>>>>>>>> > "case
>>> >>>>>>>>>> > > for headers" doc - broker extensibility is a
>>> separate
>>> >>>>>>>>> discussion.
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
>>> >>>>>>>>> gwen@confluent.io>
>>> >>>>>>>>>> > wrote:
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> > > > Woah, I wasn't aware this is something we'll do.
>>> It
>>> >>>>>>> wasn't
>>> >>>>>>>>> in the
>>> >>>>>>>>>> > KIP,
>>> >>>>>>>>>> > > > right?
>>> >>>>>>>>>> > > >
>>> >>>>>>>>>> > > > I guess we could do it the same way ACLs
>>> currently
>>> >>>>> work.
>>> >>>>>>>>>> > > > I had in mind something that will allow admins to
>>> >>>>> apply
>>> >>>>>>>>> rules to
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > new create/delete/config topic APIs. So Todd can
>>> >>>>> decide
>>> >>>>>>> to
>>> >>>>>>>>> reject
>>> >>>>>>>>>> > > > "create topic" requests that ask for more than 40
>>> >>>>>>>>> partitions, or
>>> >>>>>>>>>> > > > require exactly 3 replicas, or no more than 50GB
>>> >>>>>>> partition
>>> >>>>>>>>> size,
>>> >>>>>>>>>> > etc.
>>> >>>>>>>>>> > > >
>>> >>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are planning
>>> to
>>> >>>>> apply
>>> >>>>>>>>> more
>>> >>>>>>>>>> > rules
>>> >>>>>>>>>> > > > to requests (and I think we should), we may want
>>> a
>>> >>> bit
>>> >>>>>>>>> more generic
>>> >>>>>>>>>> > > > design around that.
>>> >>>>>>>>>> > > >
>>> >>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>>> >>>>>>>>> radai.rosenblatt@gmail.com>
>>> >>>>>>>>>> > > wrote:
>>> >>>>>>>>>> > > > > "wouldn't you be in the business of making sure
>>> >>>>>>> everyone
>>> >>>>>>>>> uses
>>> >>>>>>>>>> > them
>>> >>>>>>>>>> > > > > properly?"
>>> >>>>>>>>>> > > > >
>>> >>>>>>>>>> > > > > thats where a broker-side plugin would come
>>> handy
>>> >>> -
>>> >>>>> any
>>> >>>>>>>>> incoming
>>> >>>>>>>>>> > > message
>>> >>>>>>>>>> > > > > that does not conform to org policy (read -
>>> does
>>> >>> not
>>> >>>>>>>>> have the
>>> >>>>>>>>>> > proper
>>> >>>>>>>>>> > > > > headers) gets thrown out (with an error
>>> returned
>>> >>> to
>>> >>>>>>> user)
>>> >>>>>>>>>> > > > >
>>> >>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
>>> >>>>>>>>> tpalino@gmail.com>
>>> >>>>>>>>>> > wrote:
>>> >>>>>>>>>> > > > >
>>> >>>>>>>>>> > > > >> Come on, I've done at least 2 talks on this
>>> one
>>> >>> :)
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >> Producing counts to a topic is part of it, but
>>> >>>>> that's
>>> >>>>>>>>> only
>>> >>>>>>>>>> > part. So
>>> >>>>>>>>>> > > you
>>> >>>>>>>>>> > > > >> count you have 100 messages in topic A. When
>>> you
>>> >>>>>>> mirror
>>> >>>>>>>>> topic A
>>> >>>>>>>>>> > to
>>> >>>>>>>>>> > > > another
>>> >>>>>>>>>> > > > >> cluster, you have 99 messages. Where was your
>>> >>>>> problem?
>>> >>>>>>>>> Or
>>> >>>>>>>>>> > worse, you
>>> >>>>>>>>>> > > > have
>>> >>>>>>>>>> > > > >> 100 messages, but one producer duplicated
>>> >>> messages
>>> >>>>> and
>>> >>>>>>>>> another
>>> >>>>>>>>>> > one
>>> >>>>>>>>>> > > lost
>>> >>>>>>>>>> > > > >> messages. You need details about where the
>>> >>> message
>>> >>>>>>> came
>>> >>>>>>>>> from in
>>> >>>>>>>>>> > order
>>> >>>>>>>>>> > > to
>>> >>>>>>>>>> > > > >> pinpoint problems when they happen. Source
>>> >>> producer
>>> >>>>>>>>> info, where
>>> >>>>>>>>>> > it was
>>> >>>>>>>>>> > > > >> produced into your infrastructure, and when it
>>> >>> was
>>> >>>>>>>>> produced.
>>> >>>>>>>>>> > This
>>> >>>>>>>>>> > > > requires
>>> >>>>>>>>>> > > > >> you to add the information to the message.
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >> And yes, you still need to maintain your
>>> clients.
>>> >>>>> So
>>> >>>>>>>>> maybe my
>>> >>>>>>>>>> > original
>>> >>>>>>>>>> > > > >> example was not the best. My thoughts on not
>>> >>>>> wanting
>>> >>>>>>> to
>>> >>>>>>>>> be
>>> >>>>>>>>>> > responsible
>>> >>>>>>>>>> > > > for
>>> >>>>>>>>>> > > > >> message formats stands, because that's very
>>> much
>>> >>>>>>>>> separate from
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > client.
>>> >>>>>>>>>> > > > >> As you know, we have our own internal client
>>> >>>>> library
>>> >>>>>>>>> that can
>>> >>>>>>>>>> > insert
>>> >>>>>>>>>> > > the
>>> >>>>>>>>>> > > > >> right headers, and right now inserts the right
>>> >>>>> audit
>>> >>>>>>>>>> > information into
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> message fields. If they exist, and assuming
>>> the
>>> >>>>>>> message
>>> >>>>>>>>> is Avro
>>> >>>>>>>>>> > > encoded.
>>> >>>>>>>>>> > > > >> What if someone wants to use JSON instead for
>>> a
>>> >>>>> good
>>> >>>>>>>>> reason?
>>> >>>>>>>>>> > What if
>>> >>>>>>>>>> > > > user X
>>> >>>>>>>>>> > > > >> wants to encrypt messages, but user Y does
>>> not?
>>> >>>>>>>>> Maintaining the
>>> >>>>>>>>>> > client
>>> >>>>>>>>>> > > > >> library is still much easier than maintaining
>>> the
>>> >>>>>>>>> message
>>> >>>>>>>>>> > formats.
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >> -Todd
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
>>> >>>>>>>>> gwen@confluent.io
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> > > wrote:
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >> > Based on your last sentence, consider me
>>> >>>>> convinced
>>> >>>>>>> :)
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >> > I get why headers are critical for Mirroring
>>> >>> (you
>>> >>>>>>>>> need tags to
>>> >>>>>>>>>> > > prevent
>>> >>>>>>>>>> > > > >> > loops and sometimes to route messages to the
>>> >>>>> correct
>>> >>>>>>>>>> > destination).
>>> >>>>>>>>>> > > > >> > But why do you need headers to audit? We are
>>> >>>>>>> auditing
>>> >>>>>>>>> by
>>> >>>>>>>>>> > producing
>>> >>>>>>>>>> > > > >> > counts to a side topic (and I was under the
>>> >>>>>>>>> impression you do
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > >> > same), so we never need to modify the
>>> message.
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >> > Another thing - after we added headers,
>>> >>> wouldn't
>>> >>>>> you
>>> >>>>>>>>> be in the
>>> >>>>>>>>>> > > > >> > business of making sure everyone uses them
>>> >>>>> properly?
>>> >>>>>>>>> Making
>>> >>>>>>>>>> > sure
>>> >>>>>>>>>> > > > >> > everyone includes the right headers you
>>> need,
>>> >>> not
>>> >>>>>>>>> using the
>>> >>>>>>>>>> > header
>>> >>>>>>>>>> > > > >> > names you intend to use, etc. I don't think
>>> the
>>> >>>>>>>>> "policing"
>>> >>>>>>>>>> > business
>>> >>>>>>>>>> > > > >> > will ever go away.
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino
>>> <
>>> >>>>>>>>>> > tpalino@gmail.com>
>>> >>>>>>>>>> > > > wrote:
>>> >>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
>>> >>> with
>>> >>>>> the
>>> >>>>>>>>>> > workaround.
>>> >>>>>>>>>> > > Avro
>>> >>>>>>>>>> > > > >> means
>>> >>>>>>>>>> > > > >> > > that I have to be concerned with the
>>> format
>>> >>> of
>>> >>>>> the
>>> >>>>>>>>> messages
>>> >>>>>>>>>> > in
>>> >>>>>>>>>> > > > order to
>>> >>>>>>>>>> > > > >> > run
>>> >>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring,
>>> etc.).
>>> >>>>> That
>>> >>>>>>>>> means
>>> >>>>>>>>>> > that I
>>> >>>>>>>>>> > > have
>>> >>>>>>>>>> > > > to
>>> >>>>>>>>>> > > > >> > > handle the schemas, and I have to enforce
>>> >>> rules
>>> >>>>>>>>> about good
>>> >>>>>>>>>> > > formats.
>>> >>>>>>>>>> > > > >> This
>>> >>>>>>>>>> > > > >> > is
>>> >>>>>>>>>> > > > >> > > not something I want to be in the business
>>> >>> of,
>>> >>>>>>>>> because I
>>> >>>>>>>>>> > should be
>>> >>>>>>>>>> > > > able
>>> >>>>>>>>>> > > > >> > to
>>> >>>>>>>>>> > > > >> > > run a service infrastructure without
>>> needing
>>> >>>>> to be
>>> >>>>>>>>> in the
>>> >>>>>>>>>> > weeds of
>>> >>>>>>>>>> > > > >> > dealing
>>> >>>>>>>>>> > > > >> > > with customer data formats.
>>> >>>>>>>>>> > > > >> > >
>>> >>>>>>>>>> > > > >> > > Trust me, a sizable portion of my support
>>> >>> time
>>> >>>>> is
>>> >>>>>>>>> spent
>>> >>>>>>>>>> > dealing
>>> >>>>>>>>>> > > with
>>> >>>>>>>>>> > > > >> > schema
>>> >>>>>>>>>> > > > >> > > issues. I really would like to get away
>>> from
>>> >>>>> that.
>>> >>>>>>>>> Maybe
>>> >>>>>>>>>> > I'd have
>>> >>>>>>>>>> > > > more
>>> >>>>>>>>>> > > > >> > time
>>> >>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
>>> >>>>>>>>>> > > > >> > >
>>> >>>>>>>>>> > > > >> > > -Todd
>>> >>>>>>>>>> > > > >> > >
>>> >>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
>>> Shapira <
>>> >>>>>>>>>> > gwen@confluent.io>
>>> >>>>>>>>>> > > > wrote:
>>> >>>>>>>>>> > > > >> > >
>>> >>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
>>> >>>>> workarounds
>>> >>>>>>>>> (Avro
>>> >>>>>>>>>> > container
>>> >>>>>>>>>> > > > >> > >> format), so I'm not too excited about the
>>> >>>>> extra
>>> >>>>>>>>> work
>>> >>>>>>>>>> > required to
>>> >>>>>>>>>> > > do
>>> >>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't
>>> mind it
>>> >>>>> if
>>> >>>>>>>>> you do
>>> >>>>>>>>>> > it...
>>> >>>>>>>>>> > > > >> > >> I think the Apache convention for "good
>>> >>> idea,
>>> >>>>> but
>>> >>>>>>>>> not
>>> >>>>>>>>>> > willing to
>>> >>>>>>>>>> > > > put
>>> >>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway,
>>> that's
>>> >>>>> what
>>> >>>>>>> I
>>> >>>>>>>>> was
>>> >>>>>>>>>> > trying to
>>> >>>>>>>>>> > > > >> > >> convey :)
>>> >>>>>>>>>> > > > >> > >>
>>> >>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
>>> Palino
>>> >>> <
>>> >>>>>>>>>> > tpalino@gmail.com>
>>> >>>>>>>>>> > > > >> wrote:
>>> >>>>>>>>>> > > > >> > >> > Well I guess my question for you,
>>> then, is
>>> >>>>> what
>>> >>>>>>>>> is
>>> >>>>>>>>>> > holding you
>>> >>>>>>>>>> > > > back
>>> >>>>>>>>>> > > > >> > from
>>> >>>>>>>>>> > > > >> > >> > full support for headers? What's the
>>> bit
>>> >>>>> that
>>> >>>>>>>>> you're
>>> >>>>>>>>>> > missing
>>> >>>>>>>>>> > > that
>>> >>>>>>>>>> > > > >> has
>>> >>>>>>>>>> > > > >> > you
>>> >>>>>>>>>> > > > >> > >> > under a full +1?
>>> >>>>>>>>>> > > > >> > >> >
>>> >>>>>>>>>> > > > >> > >> > -Todd
>>> >>>>>>>>>> > > > >> > >> >
>>> >>>>>>>>>> > > > >> > >> >
>>> >>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
>>> >>>>> Shapira <
>>> >>>>>>>>>> > > gwen@confluent.io>
>>> >>>>>>>>>> > > > >> > wrote:
>>> >>>>>>>>>> > > > >> > >> >
>>> >>>>>>>>>> > > > >> > >> >> I know why people who support headers
>>> >>>>> support
>>> >>>>>>>>> them, and
>>> >>>>>>>>>> > I've
>>> >>>>>>>>>> > > > seen
>>> >>>>>>>>>> > > > >> > what
>>> >>>>>>>>>> > > > >> > >> >> the discussion is like.
>>> >>>>>>>>>> > > > >> > >> >>
>>> >>>>>>>>>> > > > >> > >> >> This is why I'm asking people who are
>>> >>>>> against
>>> >>>>>>>>> headers
>>> >>>>>>>>>> > > > (especially
>>> >>>>>>>>>> > > > >> > >> >> committers) what will make them change
>>> >>>>> their
>>> >>>>>>>>> mind - so
>>> >>>>>>>>>> > we can
>>> >>>>>>>>>> > > > get
>>> >>>>>>>>>> > > > >> > this
>>> >>>>>>>>>> > > > >> > >> >> part over one way or another.
>>> >>>>>>>>>> > > > >> > >> >>
>>> >>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at
>>> Radai,
>>> >>>>> Jun
>>> >>>>>>>>> or you
>>> >>>>>>>>>> > (Todd)...
>>> >>>>>>>>>> > > > I am
>>> >>>>>>>>>> > > > >> > >> >> just looking for something concrete we
>>> >>> can
>>> >>>>> do
>>> >>>>>>>>> to move
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > >> discussion
>>> >>>>>>>>>> > > > >> > >> >> along to the yummy design details
>>> (which
>>> >>> is
>>> >>>>>>> the
>>> >>>>>>>>>> > argument I
>>> >>>>>>>>>> > > > really
>>> >>>>>>>>>> > > > >> am
>>> >>>>>>>>>> > > > >> > >> >> looking forward to).
>>> >>>>>>>>>> > > > >> > >> >>
>>> >>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
>>> >>>>> Palino <
>>> >>>>>>>>>> > > tpalino@gmail.com>
>>> >>>>>>>>>> > > > >> > wrote:
>>> >>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even
>>> though
>>> >>>>> I'm
>>> >>>>>>>>> not a
>>> >>>>>>>>>> > > > committer)...
>>> >>>>>>>>>> > > > >> > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> > I have always been a strong
>>> supporter
>>> >>> of
>>> >>>>>>>>> introducing
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > concept
>>> >>>>>>>>>> > > > >> > of an
>>> >>>>>>>>>> > > > >> > >> >> > envelope to messages, which headers
>>> >>>>>>>>> accomplishes. The
>>> >>>>>>>>>> > > message
>>> >>>>>>>>>> > > > key
>>> >>>>>>>>>> > > > >> > is
>>> >>>>>>>>>> > > > >> > >> >> > already an example of a piece of
>>> >>> envelope
>>> >>>>>>>>>> > information. By
>>> >>>>>>>>>> > > > >> > providing a
>>> >>>>>>>>>> > > > >> > >> >> means
>>> >>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself, and
>>> not
>>> >>>>>>>>> relying on
>>> >>>>>>>>>> > use-case
>>> >>>>>>>>>> > > > >> > specific
>>> >>>>>>>>>> > > > >> > >> >> > implementations, you make it much
>>> >>> easier
>>> >>>>> for
>>> >>>>>>>>>> > components to
>>> >>>>>>>>>> > > > >> > >> interoperate.
>>> >>>>>>>>>> > > > >> > >> >> It
>>> >>>>>>>>>> > > > >> > >> >> > simplifies development of all these
>>> >>>>> things
>>> >>>>>>>>> (message
>>> >>>>>>>>>> > routing,
>>> >>>>>>>>>> > > > >> > auditing,
>>> >>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each one
>>> does
>>> >>>>> not
>>> >>>>>>>>> have to
>>> >>>>>>>>>> > reinvent
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> wheel.
>>> >>>>>>>>>> > > > >> > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> > It also makes it much easier from a
>>> >>>>> client
>>> >>>>>>>>> point of
>>> >>>>>>>>>> > view if
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> headers
>>> >>>>>>>>>> > > > >> > >> >> are
>>> >>>>>>>>>> > > > >> > >> >> > defined as part of the protocol
>>> and/or
>>> >>>>>>>>> message format
>>> >>>>>>>>>> > in
>>> >>>>>>>>>> > > > general
>>> >>>>>>>>>> > > > >> > >> because
>>> >>>>>>>>>> > > > >> > >> >> > you can easily produce and consume
>>> >>>>> messages
>>> >>>>>>>>> without
>>> >>>>>>>>>> > having
>>> >>>>>>>>>> > > to
>>> >>>>>>>>>> > > > >> take
>>> >>>>>>>>>> > > > >> > >> into
>>> >>>>>>>>>> > > > >> > >> >> > account specific cases. For
>>> example, I
>>> >>>>> want
>>> >>>>>>>>> to route
>>> >>>>>>>>>> > > messages,
>>> >>>>>>>>>> > > > >> but
>>> >>>>>>>>>> > > > >> > >> >> client A
>>> >>>>>>>>>> > > > >> > >> >> > doesn't support the way audit
>>> >>> implemented
>>> >>>>>>>>> headers, and
>>> >>>>>>>>>> > > client
>>> >>>>>>>>>> > > > B
>>> >>>>>>>>>> > > > >> > >> doesn't
>>> >>>>>>>>>> > > > >> > >> >> > support the way encryption or
>>> routing
>>> >>>>>>>>> implemented
>>> >>>>>>>>>> > headers,
>>> >>>>>>>>>> > > so
>>> >>>>>>>>>> > > > now
>>> >>>>>>>>>> > > > >> > my
>>> >>>>>>>>>> > > > >> > >> >> > application has to create some
>>> really
>>> >>>>>>> fragile
>>> >>>>>>>>> (my
>>> >>>>>>>>>> > > autocorrect
>>> >>>>>>>>>> > > > >> just
>>> >>>>>>>>>> > > > >> > >> tried
>>> >>>>>>>>>> > > > >> > >> >> to
>>> >>>>>>>>>> > > > >> > >> >> > make that "tragic", which is
>>> probably
>>> >>>>>>>>> appropriate
>>> >>>>>>>>>> > too) code
>>> >>>>>>>>>> > > to
>>> >>>>>>>>>> > > > >> > strip
>>> >>>>>>>>>> > > > >> > >> >> > everything off, rather than just
>>> >>>>> consuming
>>> >>>>>>> the
>>> >>>>>>>>>> > messages,
>>> >>>>>>>>>> > > > picking
>>> >>>>>>>>>> > > > >> > out
>>> >>>>>>>>>> > > > >> > >> the
>>> >>>>>>>>>> > > > >> > >> >> 1
>>> >>>>>>>>>> > > > >> > >> >> > or 2 headers it's interested in, and
>>> >>>>>>>>> performing its
>>> >>>>>>>>>> > > function.
>>> >>>>>>>>>> > > > >> > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has been
>>> >>> going
>>> >>>>> on
>>> >>>>>>>>> for a
>>> >>>>>>>>>> > long time,
>>> >>>>>>>>>> > > > and
>>> >>>>>>>>>> > > > >> > it's
>>> >>>>>>>>>> > > > >> > >> >> > always "Oh, you came up with 2 use
>>> >>> cases,
>>> >>>>>>> and
>>> >>>>>>>>> yeah,
>>> >>>>>>>>>> > those
>>> >>>>>>>>>> > > use
>>> >>>>>>>>>> > > > >> cases
>>> >>>>>>>>>> > > > >> > >> are
>>> >>>>>>>>>> > > > >> > >> >> > real things that someone would want
>>> to
>>> >>>>> do.
>>> >>>>>>>>> Here's an
>>> >>>>>>>>>> > > alternate
>>> >>>>>>>>>> > > > >> way
>>> >>>>>>>>>> > > > >> > to
>>> >>>>>>>>>> > > > >> > >> >> > implement them so let's not do
>>> >>> headers."
>>> >>>>> If
>>> >>>>>>>>> we have a
>>> >>>>>>>>>> > few
>>> >>>>>>>>>> > > use
>>> >>>>>>>>>> > > > >> cases
>>> >>>>>>>>>> > > > >> > >> that
>>> >>>>>>>>>> > > > >> > >> >> we
>>> >>>>>>>>>> > > > >> > >> >> > actually came up with, you can be
>>> sure
>>> >>>>> that
>>> >>>>>>>>> over the
>>> >>>>>>>>>> > next
>>> >>>>>>>>>> > > year
>>> >>>>>>>>>> > > > >> > >> there's a
>>> >>>>>>>>>> > > > >> > >> >> > dozen others that we didn't think of
>>> >>> that
>>> >>>>>>>>> someone
>>> >>>>>>>>>> > would like
>>> >>>>>>>>>> > > > to
>>> >>>>>>>>>> > > > >> > do. I
>>> >>>>>>>>>> > > > >> > >> >> > really think it's time to stop
>>> >>> rehashing
>>> >>>>>>> this
>>> >>>>>>>>>> > discussion and
>>> >>>>>>>>>> > > > >> > instead
>>> >>>>>>>>>> > > > >> > >> >> focus
>>> >>>>>>>>>> > > > >> > >> >> > on a workable standard that we can
>>> >>> adopt.
>>> >>>>>>>>>> > > > >> > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> > -Todd
>>> >>>>>>>>>> > > > >> > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
>>> >>>>> Palino
>>> >>>>>>> <
>>> >>>>>>>>>> > > > tpalino@gmail.com>
>>> >>>>>>>>>> > > > >> > >> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> >> C. per message encryption
>>> >>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach is
>>> that
>>> >>>>> this
>>> >>>>>>>>>> > significantly
>>> >>>>>>>>>> > > > reduce
>>> >>>>>>>>>> > > > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression,
>>> which
>>> >>>>>>> happens
>>> >>>>>>>>> on a
>>> >>>>>>>>>> > set of
>>> >>>>>>>>>> > > > >> > serialized
>>> >>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to
>>> enable
>>> >>>>> SSL
>>> >>>>>>>>> for wire
>>> >>>>>>>>>> > > > encryption
>>> >>>>>>>>>> > > > >> and
>>> >>>>>>>>>> > > > >> > >> rely
>>> >>>>>>>>>> > > > >> > >> >> on
>>> >>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS)
>>> for at
>>> >>>>> rest
>>> >>>>>>>>>> > encryption.
>>> >>>>>>>>>> > > > >> > >> >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>
>>> >>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
>>> >>> this
>>> >>>>>>> does
>>> >>>>>>>>> cover
>>> >>>>>>>>>> > the case
>>> >>>>>>>>>> > > > of
>>> >>>>>>>>>> > > > >> > >> removing
>>> >>>>>>>>>> > > > >> > >> >> a
>>> >>>>>>>>>> > > > >> > >> >> >> drive from the system, it will not
>>> >>>>> satisfy
>>> >>>>>>>>> most
>>> >>>>>>>>>> > compliance
>>> >>>>>>>>>> > > > >> > >> requirements
>>> >>>>>>>>>> > > > >> > >> >> for
>>> >>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever has
>>> >>>>> access to
>>> >>>>>>>>> the
>>> >>>>>>>>>> > broker
>>> >>>>>>>>>> > > itself
>>> >>>>>>>>>> > > > >> > still
>>> >>>>>>>>>> > > > >> > >> has
>>> >>>>>>>>>> > > > >> > >> >> >> access to the unencrypted data. For
>>> >>>>>>>>> end-to-end
>>> >>>>>>>>>> > encryption
>>> >>>>>>>>>> > > you
>>> >>>>>>>>>> > > > >> > need to
>>> >>>>>>>>>> > > > >> > >> >> >> encrypt at the producer, before it
>>> >>>>> enters
>>> >>>>>>> the
>>> >>>>>>>>>> > system, and
>>> >>>>>>>>>> > > > >> decrypt
>>> >>>>>>>>>> > > > >> > at
>>> >>>>>>>>>> > > > >> > >> the
>>> >>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the
>>> system.
>>> >>>>>>>>>> > > > >> > >> >> >>
>>> >>>>>>>>>> > > > >> > >> >> >> -Todd
>>> >>>>>>>>>> > > > >> > >> >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>
>>> >>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM,
>>> radai
>>> >>> <
>>> >>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
>>> >>>>>>>>>> > > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> another big plus of headers in the
>>> >>>>>>> protocol
>>> >>>>>>>>> is that
>>> >>>>>>>>>> > it
>>> >>>>>>>>>> > > would
>>> >>>>>>>>>> > > > >> > enable
>>> >>>>>>>>>> > > > >> > >> >> rapid
>>> >>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of core
>>> >>>>> kafka
>>> >>>>>>>>> and would
>>> >>>>>>>>>> > reduce
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> number of
>>> >>>>>>>>>> > > > >> > >> >> >>> future wire format changes
>>> required.
>>> >>>>>>>>>> > > > >> > >> >> >>>
>>> >>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
>>> >>>>>>> represents
>>> >>>>>>>>> use
>>> >>>>>>>>>> > cases that
>>> >>>>>>>>>> > > > are
>>> >>>>>>>>>> > > > >> > not
>>> >>>>>>>>>> > > > >> > >> 100%
>>> >>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and some of
>>> >>> them
>>> >>>>>>>>> require
>>> >>>>>>>>>> > rather
>>> >>>>>>>>>> > > > invasive
>>> >>>>>>>>>> > > > >> > wire
>>> >>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a good
>>> >>> recent
>>> >>>>>>>>> example of
>>> >>>>>>>>>> > this is
>>> >>>>>>>>>> > > > >> > kip-98.
>>> >>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected
>>> to
>>> >>> be
>>> >>>>> a
>>> >>>>>>>>> very small
>>> >>>>>>>>>> > > > fraction of
>>> >>>>>>>>>> > > > >> > >> total
>>> >>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes are
>>> >>>>> invasive.
>>> >>>>>>>>>> > > > >> > >> >> >>>
>>> >>>>>>>>>> > > > >> > >> >> >>> every such wire format change
>>> >>>>> translates
>>> >>>>>>>>> into
>>> >>>>>>>>>> > painful and
>>> >>>>>>>>>> > > > slow
>>> >>>>>>>>>> > > > >> > >> >> adoption of
>>> >>>>>>>>>> > > > >> > >> >> >>> new versions.
>>> >>>>>>>>>> > > > >> > >> >> >>>
>>> >>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
>>> >>>>> currently
>>> >>>>>>> in
>>> >>>>>>>>> KIPs
>>> >>>>>>>>>> > could be
>>> >>>>>>>>>> > > > "spun
>>> >>>>>>>>>> > > > >> > out"
>>> >>>>>>>>>> > > > >> > >> >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
>>> >>>>> transmitting
>>> >>>>>>>>> data over
>>> >>>>>>>>>> > > > headers.
>>> >>>>>>>>>> > > > >> > this
>>> >>>>>>>>>> > > > >> > >> >> would
>>> >>>>>>>>>> > > > >> > >> >> >>> keep the core wire format
>>> stable(r),
>>> >>>>> core
>>> >>>>>>>>> codebase
>>> >>>>>>>>>> > > smaller,
>>> >>>>>>>>>> > > > and
>>> >>>>>>>>>> > > > >> > >> avoid
>>> >>>>>>>>>> > > > >> > >> >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
>>> >>>>> required
>>> >>>>>>>>> to prove
>>> >>>>>>>>>> > a
>>> >>>>>>>>>> > > > certain
>>> >>>>>>>>>> > > > >> > >> feature
>>> >>>>>>>>>> > > > >> > >> >> is
>>> >>>>>>>>>> > > > >> > >> >> >>> useful enough for a wide-enough
>>> >>>>> audience
>>> >>>>>>> to
>>> >>>>>>>>> warrant
>>> >>>>>>>>>> > a wire
>>> >>>>>>>>>> > > > >> format
>>> >>>>>>>>>> > > > >> > >> >> change
>>> >>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
>>> >>>>>>>>>> > > > >> > >> >> >>>
>>> >>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
>>> >>>>> "mere"
>>> >>>>>>>>> wire format
>>> >>>>>>>>>> > > changes
>>> >>>>>>>>>> > > > >> and
>>> >>>>>>>>>> > > > >> > im
>>> >>>>>>>>>> > > > >> > >> not
>>> >>>>>>>>>> > > > >> > >> >> >>> saying it could have been
>>> completely
>>> >>>>> done
>>> >>>>>>>>> with
>>> >>>>>>>>>> > headers,
>>> >>>>>>>>>> > > but
>>> >>>>>>>>>> > > > >> > >> >> exactly-once
>>> >>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
>>> >>>>>>>>>> > > > >> > >> >> >>>
>>> >>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM,
>>> Gwen
>>> >>>>>>>>> Shapira <
>>> >>>>>>>>>> > > > >> gwen@confluent.io
>>> >>>>>>>>>> > > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>>
>>> >>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
>>> >>>>> radai <
>>> >>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
>>> >>>>> organization,
>>> >>>>>>>>> one could
>>> >>>>>>>>>> > > always
>>> >>>>>>>>>> > > > use
>>> >>>>>>>>>> > > > >> > >> other
>>> >>>>>>>>>> > > > >> > >> >> >>> > > approaches such as
>>> company-wise
>>> >>>>>>>>> containers"
>>> >>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
>>> >>>>>>> traditionally
>>> >>>>>>>>> done
>>> >>>>>>>>>> > but there
>>> >>>>>>>>>> > > > are
>>> >>>>>>>>>> > > > >> > now
>>> >>>>>>>>>> > > > >> > >> >> cases
>>> >>>>>>>>>> > > > >> > >> >> >>> > (read
>>> >>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is not
>>> >>>>>>> acceptable.
>>> >>>>>>>>> this
>>> >>>>>>>>>> > makes
>>> >>>>>>>>>> > > > headers
>>> >>>>>>>>>> > > > >> > >> useful
>>> >>>>>>>>>> > > > >> > >> >> even
>>> >>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for cases
>>> >>> where
>>> >>>>>>>>>> > > > one-container-fits-all
>>> >>>>>>>>>> > > > >> > cannot
>>> >>>>>>>>>> > > > >> > >> >> >>> apply.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use
>>> cases
>>> >>>>>>> listed,
>>> >>>>>>>>> i dont
>>> >>>>>>>>>> > want
>>> >>>>>>>>>> > > > this to
>>> >>>>>>>>>> > > > >> > >> devolve
>>> >>>>>>>>>> > > > >> > >> >> >>> to a
>>> >>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular use
>>> >>> cases
>>> >>>>> - i
>>> >>>>>>>>> think its
>>> >>>>>>>>>> > > enough
>>> >>>>>>>>>> > > > >> that
>>> >>>>>>>>>> > > > >> > >> some
>>> >>>>>>>>>> > > > >> > >> >> of
>>> >>>>>>>>>> > > > >> > >> >> >>> them
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > I think a main point of
>>> contention
>>> >>> is
>>> >>>>>>>>> that: We
>>> >>>>>>>>>> > > identified
>>> >>>>>>>>>> > > > few
>>> >>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are
>>> useful,
>>> >>>>> do
>>> >>>>>>> we
>>> >>>>>>>>> want
>>> >>>>>>>>>> > Kafka to
>>> >>>>>>>>>> > > > be a
>>> >>>>>>>>>> > > > >> > >> system
>>> >>>>>>>>>> > > > >> > >> >> >>> > that supports those use-cases?
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
>>> >>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
>>> >>>>> record-level
>>> >>>>>>>>> lineage
>>> >>>>>>>>>> > is
>>> >>>>>>>>>> > > though
>>> >>>>>>>>>> > > > >> > since
>>> >>>>>>>>>> > > > >> > >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > overhead could
>>> >>>>>>>>>> > > > >> > >> >> >>> > be significant."
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record
>>> level
>>> >>>>>>>>> lineage. I
>>> >>>>>>>>>> > don't
>>> >>>>>>>>>> > > think
>>> >>>>>>>>>> > > > it
>>> >>>>>>>>>> > > > >> > was
>>> >>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think it
>>> is
>>> >>>>> safe
>>> >>>>>>> to
>>> >>>>>>>>> assume
>>> >>>>>>>>>> > that
>>> >>>>>>>>>> > > the
>>> >>>>>>>>>> > > > NSA
>>> >>>>>>>>>> > > > >> > >> needed
>>> >>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also know
>>> >>> that
>>> >>>>>>>>> certain
>>> >>>>>>>>>> > financial
>>> >>>>>>>>>> > > > >> > institutes
>>> >>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
>>> >>> records
>>> >>>>> at
>>> >>>>>>> a
>>> >>>>>>>>> record
>>> >>>>>>>>>> > level
>>> >>>>>>>>>> > > and
>>> >>>>>>>>>> > > > >> > there
>>> >>>>>>>>>> > > > >> > >> are
>>> >>>>>>>>>> > > > >> > >> >> >>> > federal regulations that
>>> absolutely
>>> >>>>>>>>> require
>>> >>>>>>>>>> > this. They
>>> >>>>>>>>>> > > > also
>>> >>>>>>>>>> > > > >> > need
>>> >>>>>>>>>> > > > >> > >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
>>> >>>>> "touches"
>>> >>>>>>> the
>>> >>>>>>>>>> > messages and
>>> >>>>>>>>>> > > > >> either
>>> >>>>>>>>>> > > > >> > >> reads
>>> >>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't have
>>> >>>>>>> possibly
>>> >>>>>>>>>> > modified the
>>> >>>>>>>>>> > > > >> payload
>>> >>>>>>>>>> > > > >> > >> >> itself.
>>> >>>>>>>>>> > > > >> > >> >> >>> > They use record level
>>> encryption to
>>> >>>>> do
>>> >>>>>>>>> that -
>>> >>>>>>>>>> > apps can
>>> >>>>>>>>>> > > > read
>>> >>>>>>>>>> > > > >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
>>> >>> can't
>>> >>>>>>>>> touch the
>>> >>>>>>>>>> > payload.
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those are
>>> >>> corner
>>> >>>>>>>>> cases and
>>> >>>>>>>>>> > not worth
>>> >>>>>>>>>> > > > >> adding
>>> >>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they
>>> should
>>> >>>>> use a
>>> >>>>>>>>> different
>>> >>>>>>>>>> > > pubsub
>>> >>>>>>>>>> > > > >> > message
>>> >>>>>>>>>> > > > >> > >> for
>>> >>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the other
>>> 1000
>>> >>>>> that
>>> >>>>>>>>> cater
>>> >>>>>>>>>> > > > specifically
>>> >>>>>>>>>> > > > >> to
>>> >>>>>>>>>> > > > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > financial industry).
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a catch
>>> 22:
>>> >>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific
>>> use-case,
>>> >>>>>>>>> someone can
>>> >>>>>>>>>> > always
>>> >>>>>>>>>> > > say
>>> >>>>>>>>>> > > > it
>>> >>>>>>>>>> > > > >> > isn't
>>> >>>>>>>>>> > > > >> > >> >> >>> > interesting enough for Kafka.
>>> If we
>>> >>>>>>>>> discuss more
>>> >>>>>>>>>> > general
>>> >>>>>>>>>> > > > >> > trends,
>>> >>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we are not
>>> >>> sure
>>> >>>>>>> any
>>> >>>>>>>>> of them
>>> >>>>>>>>>> > really
>>> >>>>>>>>>> > > > >> needs
>>> >>>>>>>>>> > > > >> > >> >> headers
>>> >>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just hand
>>> >>>>> waving
>>> >>>>>>>>> and not
>>> >>>>>>>>>> > > > interesting.".
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > I think discussing use-cases in
>>> >>>>>>> specifics
>>> >>>>>>>>> is super
>>> >>>>>>>>>> > > > important
>>> >>>>>>>>>> > > > >> to
>>> >>>>>>>>>> > > > >> > >> >> decide
>>> >>>>>>>>>> > > > >> > >> >> >>> > implementation details for
>>> headers
>>> >>>>> (my
>>> >>>>>>>>> use-cases
>>> >>>>>>>>>> > lean
>>> >>>>>>>>>> > > > toward
>>> >>>>>>>>>> > > > >> > >> >> numerical
>>> >>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and object
>>> >>>>> values,
>>> >>>>>>>>> others
>>> >>>>>>>>>> > differ),
>>> >>>>>>>>>> > > > but I
>>> >>>>>>>>>> > > > >> > >> think
>>> >>>>>>>>>> > > > >> > >> >> we
>>> >>>>>>>>>> > > > >> > >> >> >>> > need to answer the general "Are
>>> we
>>> >>>>> going
>>> >>>>>>>>> to have
>>> >>>>>>>>>> > > headers"
>>> >>>>>>>>>> > > > >> > question
>>> >>>>>>>>>> > > > >> > >> >> >>> > first.
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the other
>>> >>>>>>>>> committers in the
>>> >>>>>>>>>> > > > discussion:
>>> >>>>>>>>>> > > > >> > >> >> >>> > What would it take to convince
>>> you
>>> >>>>> that
>>> >>>>>>>>> headers
>>> >>>>>>>>>> > in Kafka
>>> >>>>>>>>>> > > > are
>>> >>>>>>>>>> > > > >> a
>>> >>>>>>>>>> > > > >> > >> good
>>> >>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can move
>>> >>> ahead
>>> >>>>>>> and
>>> >>>>>>>>> try to
>>> >>>>>>>>>> > agree
>>> >>>>>>>>>> > > on
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> >> details?
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving the
>>> goal
>>> >>>>>>> posts
>>> >>>>>>>>> and
>>> >>>>>>>>>> > this is
>>> >>>>>>>>>> > > > truly
>>> >>>>>>>>>> > > > >> > >> >> exhausting.
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly support
>>> >>>>> adding
>>> >>>>>>>>> headers
>>> >>>>>>>>>> > to Kafka
>>> >>>>>>>>>> > > > >> > (+0.5?).
>>> >>>>>>>>>> > > > >> > >> >> >>> > The community can continue to
>>> find
>>> >>>>>>>>> workarounds to
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > issue
>>> >>>>>>>>>> > > > >> and
>>> >>>>>>>>>> > > > >> > >> there
>>> >>>>>>>>>> > > > >> > >> >> >>> > are some benefits to keeping the
>>> >>>>> message
>>> >>>>>>>>> format
>>> >>>>>>>>>> > and
>>> >>>>>>>>>> > > > clients
>>> >>>>>>>>>> > > > >> > >> simpler.
>>> >>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of
>>> headers
>>> >>>>> to
>>> >>>>>>>>> many
>>> >>>>>>>>>> > use-cases
>>> >>>>>>>>>> > > and
>>> >>>>>>>>>> > > > if
>>> >>>>>>>>>> > > > >> we
>>> >>>>>>>>>> > > > >> > >> can
>>> >>>>>>>>>> > > > >> > >> >> >>> > find a good and generally useful
>>> >>> way
>>> >>>>> to
>>> >>>>>>>>> add it to
>>> >>>>>>>>>> > Kafka,
>>> >>>>>>>>>> > > > it
>>> >>>>>>>>>> > > > >> > will
>>> >>>>>>>>>> > > > >> > >> make
>>> >>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
>>> >>> worthy
>>> >>>>>>>>> goal in my
>>> >>>>>>>>>> > eyes.
>>> >>>>>>>>>> > > > >> > >> >> >>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
>>> >>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use
>>> cases
>>> >>>>> for
>>> >>>>>>>>> polyglot
>>> >>>>>>>>>> > topics.
>>> >>>>>>>>>> > > > >> > >> especially if
>>> >>>>>>>>>> > > > >> > >> >> >>> kafka
>>> >>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
>>> >>> something
>>> >>>>>>> else.
>>> >>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would make
>>> it
>>> >>>>>>> harder
>>> >>>>>>>>> to write
>>> >>>>>>>>>> > > > portable
>>> >>>>>>>>>> > > > >> > >> consumer
>>> >>>>>>>>>> > > > >> > >> >> >>> code.
>>> >>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would mess
>>> >>> with
>>> >>>>>>>>> locality of
>>> >>>>>>>>>> > > > consumption
>>> >>>>>>>>>> > > > >> > >> >> >>> guarantees.
>>> >>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
>>> >>>>>>>>> lineage/metadata is
>>> >>>>>>>>>> > > > >> > >> billing/chargeback.
>>> >>>>>>>>>> > > > >> > >> >> for
>>> >>>>>>>>>> > > > >> > >> >> >>> > that
>>> >>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough to
>>> >>> simply
>>> >>>>>>>>> record the
>>> >>>>>>>>>> > point
>>> >>>>>>>>>> > > of
>>> >>>>>>>>>> > > > >> > origin,
>>> >>>>>>>>>> > > > >> > >> but
>>> >>>>>>>>>> > > > >> > >> >> >>> every
>>> >>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think mirror
>>> >>>>> maker)
>>> >>>>>>>>> must also
>>> >>>>>>>>>> > add a
>>> >>>>>>>>>> > > > >> record
>>> >>>>>>>>>> > > > >> > to
>>> >>>>>>>>>> > > > >> > >> >> form a
>>> >>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
>>> >>>>>>>>>> > > > >> > >> >> >>> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing on
>>> top
>>> >>> of
>>> >>>>>>>>> kafka - i
>>> >>>>>>>>>> > know
>>> >>>>>>>>>> > > samza
>>> >>>>>>>>>> > > > >> has a
>>> >>>>>>>>>> > > > >> > >> >> metadata
>>> >>>>>>>>>> > > > >> > >> >> >>> > map
>>> >>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
>>> >>>>> addition to
>>> >>>>>>>>> user
>>> >>>>>>>>>> > values.
>>> >>>>>>>>>> > > > headers
>>> >>>>>>>>>> > > > >> > are
>>> >>>>>>>>>> > > > >> > >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > perfect
>>> >>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50
>>> PM,
>>> >>>>> Jun
>>> >>>>>>>>> Rao <
>>> >>>>>>>>>> > > > jun@confluent.io
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >> > >> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the first
>>> two
>>> >>>>>>>>> questions, it
>>> >>>>>>>>>> > would
>>> >>>>>>>>>> > > be
>>> >>>>>>>>>> > > > >> > helpful
>>> >>>>>>>>>> > > > >> > >> >> if we
>>> >>>>>>>>>> > > > >> > >> >> >>> > could
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use
>>> cases
>>> >>>>> for
>>> >>>>>>>>> headers
>>> >>>>>>>>>> > in the
>>> >>>>>>>>>> > > > space
>>> >>>>>>>>>> > > > >> > for
>>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases
>>> within an
>>> >>>>>>>>> organization,
>>> >>>>>>>>>> > one
>>> >>>>>>>>>> > > > could
>>> >>>>>>>>>> > > > >> > always
>>> >>>>>>>>>> > > > >> > >> >> use
>>> >>>>>>>>>> > > > >> > >> >> >>> > other
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as
>>> company-wise
>>> >>>>>>>>> containers to
>>> >>>>>>>>>> > get
>>> >>>>>>>>>> > > > around
>>> >>>>>>>>>> > > > >> w/o
>>> >>>>>>>>>> > > > >> > >> >> >>> headers. I
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> went through the use cases in
>>> >>> the
>>> >>>>> KIP
>>> >>>>>>>>> and in
>>> >>>>>>>>>> > Radai's
>>> >>>>>>>>>> > > > wiki
>>> >>>>>>>>>> > > > >> (
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> https://cwiki.apache.org/confl
>>> >>>>>>>>>> > uence/display/KAFKA/A+
>>> >>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> ).
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> The following are the ones
>>> that
>>> >>>>> that
>>> >>>>>>> I
>>> >>>>>>>>>> > understand and
>>> >>>>>>>>>> > > > >> could
>>> >>>>>>>>>> > > > >> > be
>>> >>>>>>>>>> > > > >> > >> in
>>> >>>>>>>>>> > > > >> > >> >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case
>>> category.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
>>> >>>>>>> content-type
>>> >>>>>>>>> should
>>> >>>>>>>>>> > be set
>>> >>>>>>>>>> > > at
>>> >>>>>>>>>> > > > >> the
>>> >>>>>>>>>> > > > >> > >> topic
>>> >>>>>>>>>> > > > >> > >> >> >>> level.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages
>>> with
>>> >>>>>>>>> different
>>> >>>>>>>>>> > content
>>> >>>>>>>>>> > > > types
>>> >>>>>>>>>> > > > >> > >> should be
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
>>> >>> useless
>>> >>>>>>>>> without
>>> >>>>>>>>>> > schema id,
>>> >>>>>>>>>> > > it
>>> >>>>>>>>>> > > > >> > seems
>>> >>>>>>>>>> > > > >> > >> that
>>> >>>>>>>>>> > > > >> > >> >> >>> > storing
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together with
>>> >>>>>>> serialized
>>> >>>>>>>>> bytes
>>> >>>>>>>>>> > in the
>>> >>>>>>>>>> > > > value
>>> >>>>>>>>>> > > > >> is
>>> >>>>>>>>>> > > > >> > >> >> better?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this
>>> approach is
>>> >>>>> that
>>> >>>>>>>>> this
>>> >>>>>>>>>> > > > significantly
>>> >>>>>>>>>> > > > >> > reduce
>>> >>>>>>>>>> > > > >> > >> >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
>>> >>>>> which
>>> >>>>>>>>> happens on
>>> >>>>>>>>>> > a set
>>> >>>>>>>>>> > > of
>>> >>>>>>>>>> > > > >> > >> serialized
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative is
>>> to
>>> >>>>> enable
>>> >>>>>>>>> SSL for
>>> >>>>>>>>>> > wire
>>> >>>>>>>>>> > > > >> > encryption
>>> >>>>>>>>>> > > > >> > >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> rely
>>> >>>>>>>>>> > > > >> > >> >> >>> > on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g.
>>> LUKS)
>>> >>>>> for at
>>> >>>>>>>>> rest
>>> >>>>>>>>>> > > encryption.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
>>> >>> across
>>> >>>>>>>>> Kafka
>>> >>>>>>>>>> > clusters
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> This is actually interesting.
>>> >>>>> Today,
>>> >>>>>>>>> to avoid
>>> >>>>>>>>>> > > > introducing
>>> >>>>>>>>>> > > > >> > >> cycles
>>> >>>>>>>>>> > > > >> > >> >> when
>>> >>>>>>>>>> > > > >> > >> >> >>> > doing
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data
>>> centers,
>>> >>> one
>>> >>>>>>>>> would
>>> >>>>>>>>>> > either have
>>> >>>>>>>>>> > > to
>>> >>>>>>>>>> > > > >> set
>>> >>>>>>>>>> > > > >> > up
>>> >>>>>>>>>> > > > >> > >> two
>>> >>>>>>>>>> > > > >> > >> >> >>> Kafka
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
>>> >>>>> aggregate)
>>> >>>>>>>>> per data
>>> >>>>>>>>>> > center
>>> >>>>>>>>>> > > or
>>> >>>>>>>>>> > > > >> > rename
>>> >>>>>>>>>> > > > >> > >> >> topics.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
>>> headers,
>>> >>>>> the
>>> >>>>>>>>> producer
>>> >>>>>>>>>> > could
>>> >>>>>>>>>> > > tag
>>> >>>>>>>>>> > > > >> each
>>> >>>>>>>>>> > > > >> > >> >> message
>>> >>>>>>>>>> > > > >> > >> >> >>> with
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID in
>>> the
>>> >>>>>>> header.
>>> >>>>>>>>>> > MirrorMaker
>>> >>>>>>>>>> > > > could
>>> >>>>>>>>>> > > > >> > then
>>> >>>>>>>>>> > > > >> > >> >> avoid
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a
>>> cluster
>>> >>> if
>>> >>>>>>>>> they are
>>> >>>>>>>>>> > tagged
>>> >>>>>>>>>> > > with
>>> >>>>>>>>>> > > > >> the
>>> >>>>>>>>>> > > > >> > >> same
>>> >>>>>>>>>> > > > >> > >> >> >>> cluster
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> id.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative
>>> approach
>>> >>>>> is
>>> >>>>>>> to
>>> >>>>>>>>>> > introduce sth
>>> >>>>>>>>>> > > > like
>>> >>>>>>>>>> > > > >> > >> >> >>> hierarchical
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages from
>>> >>>>>>> different
>>> >>>>>>>>>> > clusters in
>>> >>>>>>>>>> > > > >> > different
>>> >>>>>>>>>> > > > >> > >> >> >>> partitions
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic. This
>>> >>>>> approach
>>> >>>>>>>>> avoids
>>> >>>>>>>>>> > filtering
>>> >>>>>>>>>> > > > out
>>> >>>>>>>>>> > > > >> > >> unneeded
>>> >>>>>>>>>> > > > >> > >> >> >>> data
>>> >>>>>>>>>> > > > >> > >> >> >>> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving
>>> easier
>>> >>> to
>>> >>>>>>>>> support. It
>>> >>>>>>>>>> > may
>>> >>>>>>>>>> > > make
>>> >>>>>>>>>> > > > >> > >> compaction
>>> >>>>>>>>>> > > > >> > >> >> >>> > trickier
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> though since the same key may
>>> >>>>> show up
>>> >>>>>>>>> in
>>> >>>>>>>>>> > different
>>> >>>>>>>>>> > > > >> > partitions.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source
>>> connector
>>> >>>>> could
>>> >>>>>>>>> store in
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > message
>>> >>>>>>>>>> > > > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> metadata
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
>>> >>> record.
>>> >>>>>>>>> Similarly,
>>> >>>>>>>>>> > if a
>>> >>>>>>>>>> > > > stream
>>> >>>>>>>>>> > > > >> job
>>> >>>>>>>>>> > > > >> > >> >> >>> transforms
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to
>>> topic
>>> >>> B,
>>> >>>>> the
>>> >>>>>>>>> library
>>> >>>>>>>>>> > could
>>> >>>>>>>>>> > > > >> include
>>> >>>>>>>>>> > > > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> source
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each of the
>>> >>>>>>>>> transformed
>>> >>>>>>>>>> > message in
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> header.
>>> >>>>>>>>>> > > > >> > >> >> Not
>>> >>>>>>>>>> > > > >> > >> >> >>> > sure
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful
>>> record-level
>>> >>>>>>> lineage
>>> >>>>>>>>> is
>>> >>>>>>>>>> > though
>>> >>>>>>>>>> > > since
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> >> overhead
>>> >>>>>>>>>> > > > >> > >> >> >>> > could
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
>>> >>>>>>>>> clientId/host/user in
>>> >>>>>>>>>> > the
>>> >>>>>>>>>> > > > header
>>> >>>>>>>>>> > > > >> in
>>> >>>>>>>>>> > > > >> > >> each
>>> >>>>>>>>>> > > > >> > >> >> >>> > message
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These metadata
>>> are
>>> >>>>>>>>> really at the
>>> >>>>>>>>>> > > producer
>>> >>>>>>>>>> > > > >> > level
>>> >>>>>>>>>> > > > >> > >> >> though.
>>> >>>>>>>>>> > > > >> > >> >> >>> > So, a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to only
>>> >>>>>>> include a
>>> >>>>>>>>>> > "producerId"
>>> >>>>>>>>>> > > > per
>>> >>>>>>>>>> > > > >> > >> message
>>> >>>>>>>>>> > > > >> > >> >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> > send
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
>>> >>> mapping
>>> >>>>>>>>>> > independently.
>>> >>>>>>>>>> > > > KIP-98
>>> >>>>>>>>>> > > > >> is
>>> >>>>>>>>>> > > > >> > >> >> actually
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such a
>>> >>>>> producerId
>>> >>>>>>>>> natively
>>> >>>>>>>>>> > in the
>>> >>>>>>>>>> > > > >> > message.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that
>>> I
>>> >>> am
>>> >>>>>>> fully
>>> >>>>>>>>>> > convinced of
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > strong
>>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
>>> >>> Perhaps
>>> >>>>> we
>>> >>>>>>>>> could
>>> >>>>>>>>>> > discuss a
>>> >>>>>>>>>> > > > bit
>>> >>>>>>>>>> > > > >> > more
>>> >>>>>>>>>> > > > >> > >> to
>>> >>>>>>>>>> > > > >> > >> >> make
>>> >>>>>>>>>> > > > >> > >> >> >>> > one
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing use
>>> >>>>> cases.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal question
>>> is
>>> >>>>>>>>> whether header
>>> >>>>>>>>>> > should
>>> >>>>>>>>>> > > > be
>>> >>>>>>>>>> > > > >> > >> exposed
>>> >>>>>>>>>> > > > >> > >> >> in
>>> >>>>>>>>>> > > > >> > >> >> >>> > stream
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
>>> >>>>> stream,
>>> >>>>>>>>> Samza,
>>> >>>>>>>>>> > and
>>> >>>>>>>>>> > > Spark
>>> >>>>>>>>>> > > > >> > >> streaming.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems just
>>> >>> deal
>>> >>>>>>> with
>>> >>>>>>>>>> > key/value
>>> >>>>>>>>>> > > > pairs.
>>> >>>>>>>>>> > > > >> > >> Should we
>>> >>>>>>>>>> > > > >> > >> >> >>> > expose a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there too
>>> or
>>> >>>>>>>>> somehow map
>>> >>>>>>>>>> > header to
>>> >>>>>>>>>> > > > key
>>> >>>>>>>>>> > > > >> or
>>> >>>>>>>>>> > > > >> > >> >> value?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Jun
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35
>>> AM,
>>> >>>>>>>>> Michael
>>> >>>>>>>>>> > Pearce <
>>> >>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a
>>> period
>>> >>>>> of a
>>> >>>>>>>>> week,
>>> >>>>>>>>>> > that there
>>> >>>>>>>>>> > > > is
>>> >>>>>>>>>> > > > >> no
>>> >>>>>>>>>> > > > >> > >> >> concerns
>>> >>>>>>>>>> > > > >> > >> >> >>> now
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and
>>> now
>>> >>> we
>>> >>>>>>> have
>>> >>>>>>>>>> > agreement that
>>> >>>>>>>>>> > > > >> > headers
>>> >>>>>>>>>> > > > >> > >> are
>>> >>>>>>>>>> > > > >> > >> >> >>> useful
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
>>> >>> put
>>> >>>>> to
>>> >>>>>>> a
>>> >>>>>>>>> KIP
>>> >>>>>>>>>> > vote, this
>>> >>>>>>>>>> > > > >> > wouldn't
>>> >>>>>>>>>> > > > >> > >> be
>>> >>>>>>>>>> > > > >> > >> >> a
>>> >>>>>>>>>> > > > >> > >> >> >>> > reason
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > @
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of
>>> getting
>>> >>>>> this
>>> >>>>>>>>> KIP
>>> >>>>>>>>>> > moving past
>>> >>>>>>>>>> > > > >> this,
>>> >>>>>>>>>> > > > >> > we
>>> >>>>>>>>>> > > > >> > >> can
>>> >>>>>>>>>> > > > >> > >> >> >>> state
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
>>> >>> that
>>> >>>>>>> can
>>> >>>>>>>>> will be
>>> >>>>>>>>>> > > > naturally
>>> >>>>>>>>>> > > > >> > >> >> interpreted
>>> >>>>>>>>>> > > > >> > >> >> >>> as
>>> >>>>>>>>>> > > > >> > >> >> >>> > an
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is
>>> later
>>> >>>>>>>>> wanted you can
>>> >>>>>>>>>> > > easily
>>> >>>>>>>>>> > > > >> split
>>> >>>>>>>>>> > > > >> > >> this
>>> >>>>>>>>>> > > > >> > >> >> >>> into
>>> >>>>>>>>>> > > > >> > >> >> >>> > two
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the
>>> wire
>>> >>>>>>> protocol
>>> >>>>>>>>>> > > implementation
>>> >>>>>>>>>> > > > >> this
>>> >>>>>>>>>> > > > >> > >> makes
>>> >>>>>>>>>> > > > >> > >> >> no
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don't
>>> believe. Is
>>> >>>>> this
>>> >>>>>>>>>> > reasonable to
>>> >>>>>>>>>> > > > all?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
>>> therefor
>>> >>>>> happy
>>> >>>>>>>>> we keep
>>> >>>>>>>>>> > with 32
>>> >>>>>>>>>> > > > >> bits.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
>>> >>>>>>>>>> > ignacio.solis@gmail.com on
>>> >>>>>>>>>> > > > >> behalf
>>> >>>>>>>>>> > > > >> > of
>>> >>>>>>>>>> > > > >> > >> >> >>> Ignacio
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
>>> >>> ignacio.solis@gmail.com
>>> >>>>> on
>>> >>>>>>>>> behalf of
>>> >>>>>>>>>> > > > >> > isolis@igso.net
>>> >>>>>>>>>> > > > >> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value
>>> as
>>> >>>>> byte[]
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
>>> >>> maybe
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
>>> >>>>> system
>>> >>>>>>>>> should
>>> >>>>>>>>>> > take a
>>> >>>>>>>>>> > > > single
>>> >>>>>>>>>> > > > >> > >> int. I
>>> >>>>>>>>>> > > > >> > >> >> >>> think
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you
>>> want
>>> >>> to
>>> >>>>>>>>> interpret
>>> >>>>>>>>>> > this as
>>> >>>>>>>>>> > > to
>>> >>>>>>>>>> > > > >> 16bit
>>> >>>>>>>>>> > > > >> > >> >> numbers
>>> >>>>>>>>>> > > > >> > >> >> >>> in
>>> >>>>>>>>>> > > > >> > >> >> >>> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > above go right ahead.
>>> If
>>> >>>>>>>>> somebody wants
>>> >>>>>>>>>> > to
>>> >>>>>>>>>> > > argue
>>> >>>>>>>>>> > > > >> for
>>> >>>>>>>>>> > > > >> > 16
>>> >>>>>>>>>> > > > >> > >> >> bits
>>> >>>>>>>>>> > > > >> > >> >> >>> or
>>> >>>>>>>>>> > > > >> > >> >> >>> > 64
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I
>>> would
>>> >>>>>>> listen.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key space
>>> >>> into
>>> >>>>>>>>> sub_key_1 and
>>> >>>>>>>>>> > > > sub_key_2
>>> >>>>>>>>>> > > > >> > >> makes no
>>> >>>>>>>>>> > > > >> > >> >> >>> > sense to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > me at
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we
>>> going
>>> >>> to
>>> >>>>>>>>> start
>>> >>>>>>>>>> > providing
>>> >>>>>>>>>> > > > APIs to
>>> >>>>>>>>>> > > > >> > get
>>> >>>>>>>>>> > > > >> > >> all
>>> >>>>>>>>>> > > > >> > >> >> >>> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
>>> >>>>>>>>> sub_key_2s? If
>>> >>>>>>>>>> > there is
>>> >>>>>>>>>> > > > no
>>> >>>>>>>>>> > > > >> > >> >> >>> distinguishing
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > functions
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to
>>> each
>>> >>> one
>>> >>>>>>>>> then they
>>> >>>>>>>>>> > should
>>> >>>>>>>>>> > > be
>>> >>>>>>>>>> > > > a
>>> >>>>>>>>>> > > > >> > single
>>> >>>>>>>>>> > > > >> > >> >> >>> value.
>>> >>>>>>>>>> > > > >> > >> >> >>> > At
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're doing
>>> is
>>> >>>>>>>>> equality.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer
>>> wants
>>> >>> to
>>> >>>>>>>>> interpret
>>> >>>>>>>>>> > this as
>>> >>>>>>>>>> > > 2,
>>> >>>>>>>>>> > > > 3
>>> >>>>>>>>>> > > > >> or
>>> >>>>>>>>>> > > > >> > >> more
>>> >>>>>>>>>> > > > >> > >> >> >>> values
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > different question. I
>>> >>>>>>>>> personally think
>>> >>>>>>>>>> > it's
>>> >>>>>>>>>> > > all
>>> >>>>>>>>>> > > > one
>>> >>>>>>>>>> > > > >> > >> >> keyspace
>>> >>>>>>>>>> > > > >> > >> >> >>> > that is
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned using
>>> >>> some
>>> >>>>>>>>> structure,
>>> >>>>>>>>>> > but if
>>> >>>>>>>>>> > > you
>>> >>>>>>>>>> > > > >> > want to
>>> >>>>>>>>>> > > > >> > >> >> >>> > sub-assign
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > parts
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
>>> >>> applies
>>> >>>>> to
>>> >>>>>>>>> strings.
>>> >>>>>>>>>> > If
>>> >>>>>>>>>> > > > somebody
>>> >>>>>>>>>> > > > >> > >> argued
>>> >>>>>>>>>> > > > >> > >> >> for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would we be arguing to
>>> >>>>> divide
>>> >>>>>>> the
>>> >>>>>>>>>> > strings with
>>> >>>>>>>>>> > > > dots
>>> >>>>>>>>>> > > > >> > ('.')
>>> >>>>>>>>>> > > > >> > >> >> as a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them to
>>> give
>>> >>>>> us
>>> >>>>>>> the
>>> >>>>>>>>>> > different
>>> >>>>>>>>>> > > name
>>> >>>>>>>>>> > > > >> > segments
>>> >>>>>>>>>> > > > >> > >> >> >>> > separately?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be performing
>>> any
>>> >>>>>>>>> actions on
>>> >>>>>>>>>> > this key
>>> >>>>>>>>>> > > > other
>>> >>>>>>>>>> > > > >> > than
>>> >>>>>>>>>> > > > >> > >> >> >>> > matching?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
>>> >>> 9:30
>>> >>>>>>> AM,
>>> >>>>>>>>> Michael
>>> >>>>>>>>>> > > Pearce <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
>>> concerns
>>> >>> on
>>> >>>>> 1
>>> >>>>>>>>> and 2
>>> >>>>>>>>>> > still?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this moving
>>> >>> along a
>>> >>>>>>> bit
>>> >>>>>>>>> more
>>> >>>>>>>>>> > I'd also
>>> >>>>>>>>>> > > > like
>>> >>>>>>>>>> > > > >> to
>>> >>>>>>>>>> > > > >> > >> ask
>>> >>>>>>>>>> > > > >> > >> >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> get
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last
>>> points:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're
>>> all
>>> >>>>>>> roughly
>>> >>>>>>>>> happy
>>> >>>>>>>>>> > with the
>>> >>>>>>>>>> > > > >> header
>>> >>>>>>>>>> > > > >> > >> value
>>> >>>>>>>>>> > > > >> > >> >> >>> > being a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe
>>> consensus
>>> >>> has
>>> >>>>>>>>> been for an
>>> >>>>>>>>>> > > > namespace
>>> >>>>>>>>>> > > > >> > based
>>> >>>>>>>>>> > > > >> > >> int
>>> >>>>>>>>>> > > > >> > >> >> >>> > approach
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the
>>> key.
>>> >>> Any
>>> >>>>>>>>> objections
>>> >>>>>>>>>> > if this
>>> >>>>>>>>>> > > > is
>>> >>>>>>>>>> > > > >> > what
>>> >>>>>>>>>> > > > >> > >> we
>>> >>>>>>>>>> > > > >> > >> >> go
>>> >>>>>>>>>> > > > >> > >> >> >>> > with?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
>>> >>>>> assumption
>>> >>>>>>> in
>>> >>>>>>>>> (4) is
>>> >>>>>>>>>> > > correct,
>>> >>>>>>>>>> > > > >> > >> {int,int}
>>> >>>>>>>>>> > > > >> > >> >> >>> keys.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both int's be
>>> >>>>> int16 or
>>> >>>>>>>>> int32?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them being
>>> >>> int16(2
>>> >>>>>>>>> bytes) as
>>> >>>>>>>>>> > combined
>>> >>>>>>>>>> > > > is
>>> >>>>>>>>>> > > > >> > space
>>> >>>>>>>>>> > > > >> > >> of
>>> >>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > per
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > original and gives
>>> >>> plenty
>>> >>>>> of
>>> >>>>>>>>>> > combinations for
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> foreseeable,
>>> >>>>>>>>>> > > > >> > >> >> >>> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead small.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any
>>> benefit in
>>> >>>>>>>>> another kip
>>> >>>>>>>>>> > call to
>>> >>>>>>>>>> > > > >> discuss
>>> >>>>>>>>>> > > > >> > >> >> these at
>>> >>>>>>>>>> > > > >> > >> >> >>> > all?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>> ______________________________
>>> >>>>>>>>>> > __________
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
>>> >>>>>>>>> k.burstev@yandex.com>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
>>> November
>>> >>> 18,
>>> >>>>>>> 2016
>>> >>>>>>>>>> > 7:07:07 AM
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
>>> >>> dev@kafka.apache.org
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
>>> [DISCUSS]
>>> >>>>> KIP-82
>>> >>>>>>>>> - Add
>>> >>>>>>>>>> > Record
>>> >>>>>>>>>> > > > Headers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is worth
>>> >>> also
>>> >>>>> i
>>> >>>>>>>>> agree. As
>>> >>>>>>>>>> > a user:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
>>> >>>>>>>>> worthwhile
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers
>>> should
>>> >>>>> be a
>>> >>>>>>>>> top level
>>> >>>>>>>>>> > > option
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
>>> >>>>> "Ignacio
>>> >>>>>>>>> Solis" <
>>> >>>>>>>>>> > > > >> isolis@igso.net
>>> >>>>>>>>>> > > > >> > >:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers
>>> are
>>> >>>>>>>>> worthwhile
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
>>> >>> should
>>> >>>>> be
>>> >>>>>>> a
>>> >>>>>>>>> top
>>> >>>>>>>>>> > level
>>> >>>>>>>>>> > > option
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14,
>>> 2016
>>> >>> at
>>> >>>>>>> 9:16
>>> >>>>>>>>> AM,
>>> >>>>>>>>>> > Michael
>>> >>>>>>>>>> > > > Pearce
>>> >>>>>>>>>> > > > >> <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
>>> >>>>> details/examples
>>> >>>>>>>>> the
>>> >>>>>>>>>> > original
>>> >>>>>>>>>> > > > proposal
>>> >>>>>>>>>> > > > >> > for
>>> >>>>>>>>>> > > > >> > >> key
>>> >>>>>>>>>> > > > >> > >> >> >>> > spacing
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> ,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > not
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as
>>> per
>>> >>>>>>>>> discussion
>>> >>>>>>>>>> > namespace
>>> >>>>>>>>>> > > > >> idea.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need to
>>> >>> update
>>> >>>>>>> the
>>> >>>>>>>>> kip,
>>> >>>>>>>>>> > when we
>>> >>>>>>>>>> > > get
>>> >>>>>>>>>> > > > >> > >> agreement
>>> >>>>>>>>>> > > > >> > >> >> >>> this
>>> >>>>>>>>>> > > > >> > >> >> >>> > is a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > better
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach (which
>>> >>> seems
>>> >>>>> to
>>> >>>>>>>>> be the
>>> >>>>>>>>>> > case if I
>>> >>>>>>>>>> > > > have
>>> >>>>>>>>>> > > > >> > >> >> understood
>>> >>>>>>>>>> > > > >> > >> >> >>> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > general
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
>>> >>>>>>>>> conversation)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
>>> >>> ints,
>>> >>>>> at
>>> >>>>>>>>> very
>>> >>>>>>>>>> > early stage
>>> >>>>>>>>>> > > > we
>>> >>>>>>>>>> > > > >> did
>>> >>>>>>>>>> > > > >> > >> think
>>> >>>>>>>>>> > > > >> > >> >> >>> about
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added
>>> complexity
>>> >>>>> for
>>> >>>>>>>>> the
>>> >>>>>>>>>> > saving isn't
>>> >>>>>>>>>> > > > >> worth
>>> >>>>>>>>>> > > > >> > it.
>>> >>>>>>>>>> > > > >> > >> >> I'd
>>> >>>>>>>>>> > > > >> > >> >> >>> > rather
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> go
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
>>> >>>>>>>>> overheads and
>>> >>>>>>>>>> > size
>>> >>>>>>>>>> > > int16
>>> >>>>>>>>>> > > > >> > (2bytes)
>>> >>>>>>>>>> > > > >> > >> >> keys
>>> >>>>>>>>>> > > > >> > >> >> >>> as
>>> >>>>>>>>>> > > > >> > >> >> >>> > it
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
>>> >>>>>>> headers,
>>> >>>>>>>>> there
>>> >>>>>>>>>> > is as
>>> >>>>>>>>>> > > per
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > kip
>>> >>>>>>>>>> > > > >> > >> as
>>> >>>>>>>>>> > > > >> > >> >> we
>>> >>>>>>>>>> > > > >> > >> >> >>> > use an
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
>>> >>>>> headers
>>> >>>>>>>>> are
>>> >>>>>>>>>> > present or
>>> >>>>>>>>>> > > > not as
>>> >>>>>>>>>> > > > >> > such
>>> >>>>>>>>>> > > > >> > >> >> >>> > provides a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > zero
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead
>>> currently
>>> >>> if
>>> >>>>>>>>> headers are
>>> >>>>>>>>>> > not
>>> >>>>>>>>>> > > used.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
>>> >>>>> mentions
>>> >>>>>>>>> would be
>>> >>>>>>>>>> > good
>>> >>>>>>>>>> > > > first
>>> >>>>>>>>>> > > > >> > if we
>>> >>>>>>>>>> > > > >> > >> >> can
>>> >>>>>>>>>> > > > >> > >> >> >>> get
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > do
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have
>>> general
>>> >>>>>>>>> consensus that
>>> >>>>>>>>>> > (1)
>>> >>>>>>>>>> > > > headers
>>> >>>>>>>>>> > > > >> > are
>>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want
>>> it
>>> >>> as
>>> >>>>> a
>>> >>>>>>>>> top level
>>> >>>>>>>>>> > entity.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
>>> >>>>>>> obvious i
>>> >>>>>>>>>> > believe (1)
>>> >>>>>>>>>> > > > >> headers
>>> >>>>>>>>>> > > > >> > are
>>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a top
>>> level
>>> >>>>>>>>> entity.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>> ______________________________
>>> >>>>>>>>>> > __________
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
>>> Hoover <
>>> >>>>>>>>>> > > roger.hoover@gmail.com
>>> >>>>>>>>>> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
>>> >>>>> November
>>> >>>>>>>>> 9, 2016
>>> >>>>>>>>>> > 9:10:47
>>> >>>>>>>>>> > > > PM
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
>>> >>>>> dev@kafka.apache.org
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
>>> >>> [DISCUSS]
>>> >>>>>>>>> KIP-82 - Add
>>> >>>>>>>>>> > > Record
>>> >>>>>>>>>> > > > >> > Headers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
>>> >>>>> little
>>> >>>>>>>>> in the
>>> >>>>>>>>>> > weeds but
>>> >>>>>>>>>> > > > >> thanks
>>> >>>>>>>>>> > > > >> > >> for
>>> >>>>>>>>>> > > > >> > >> >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> replies
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a
>>> prefix
>>> >>>>> and
>>> >>>>>>>>> {int,
>>> >>>>>>>>>> > int} can
>>> >>>>>>>>>> > > be
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> same.
>>> >>>>>>>>>> > > > >> > >> >> It
>>> >>>>>>>>>> > > > >> > >> >> >>> > doesn't
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > look
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > like
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what the
>>> KIP
>>> >>> is
>>> >>>>>>>>> saying the
>>> >>>>>>>>>> > "Open"
>>> >>>>>>>>>> > > > >> > section.
>>> >>>>>>>>>> > > > >> > >> The
>>> >>>>>>>>>> > > > >> > >> >> >>> > example
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shows
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
>>> >>>>> 210002
>>> >>>>>>>>> for App
>>> >>>>>>>>>> > Dynamics
>>> >>>>>>>>>> > > > >> > implying
>>> >>>>>>>>>> > > > >> > >> >> that
>>> >>>>>>>>>> > > > >> > >> >> >>> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > New
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization will
>>> >>> have
>>> >>>>>>>>> only a
>>> >>>>>>>>>> > single
>>> >>>>>>>>>> > > > header id
>>> >>>>>>>>>> > > > >> > to
>>> >>>>>>>>>> > > > >> > >> work
>>> >>>>>>>>>> > > > >> > >> >> >>> > with. Or
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > is
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The
>>> main
>>> >>>>> point
>>> >>>>>>>>> of a
>>> >>>>>>>>>> > namespace
>>> >>>>>>>>>> > > or
>>> >>>>>>>>>> > > > >> > prefix
>>> >>>>>>>>>> > > > >> > >> is
>>> >>>>>>>>>> > > > >> > >> >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> > reduce
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of
>>> config
>>> >>>>>>> mapping
>>> >>>>>>>>> or
>>> >>>>>>>>>> > > registration
>>> >>>>>>>>>> > > > >> > >> depending
>>> >>>>>>>>>> > > > >> > >> >> on
>>> >>>>>>>>>> > > > >> > >> >> >>> how
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> namespaces/prefixes
>>> >>>>> are
>>> >>>>>>>>> managed.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to
>>> hear
>>> >>>>> more
>>> >>>>>>>>> feedback
>>> >>>>>>>>>> > on the
>>> >>>>>>>>>> > > > >> > >> higher-level
>>> >>>>>>>>>> > > > >> > >> >> >>> > questions
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9,
>>> 2016
>>> >>> at
>>> >>>>>>>>> 11:38 AM,
>>> >>>>>>>>>> > radai <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com
>>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think this
>>> >>>>> discussion
>>> >>>>>>>>> is
>>> >>>>>>>>>> > getting a
>>> >>>>>>>>>> > > bit
>>> >>>>>>>>>> > > > >> into
>>> >>>>>>>>>> > > > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> weeds on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > technical
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > implementation
>>> >>>>> details.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to
>>> step
>>> >>>>> back a
>>> >>>>>>>>> minute
>>> >>>>>>>>>> > and try
>>> >>>>>>>>>> > > > and
>>> >>>>>>>>>> > > > >> > >> establish
>>> >>>>>>>>>> > > > >> > >> >> >>> > where we
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > are in
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
>>> >>> nacho's
>>> >>>>>>> last
>>> >>>>>>>>>> > paragraph)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all
>>> in
>>> >>>>>>>>> agreement that
>>> >>>>>>>>>> > headers
>>> >>>>>>>>>> > > > are
>>> >>>>>>>>>> > > > >> a
>>> >>>>>>>>>> > > > >> > >> >> >>> worthwhile
>>> >>>>>>>>>> > > > >> > >> >> >>> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > useful
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to
>>> have?
>>> >>>>> this
>>> >>>>>>>>> was
>>> >>>>>>>>>> > contested
>>> >>>>>>>>>> > > > early
>>> >>>>>>>>>> > > > >> on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all
>>> in
>>> >>>>>>>>> agreement on
>>> >>>>>>>>>> > headers
>>> >>>>>>>>>> > > as
>>> >>>>>>>>>> > > > top
>>> >>>>>>>>>> > > > >> > >> level
>>> >>>>>>>>>> > > > >> > >> >> >>> entity
>>> >>>>>>>>>> > > > >> > >> >> >>> > vs
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > headers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> squirreled-away in
>>> >>>>> V?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there are
>>> still
>>> >>>>>>>>> concerns
>>> >>>>>>>>>> > around
>>> >>>>>>>>>> > > these
>>> >>>>>>>>>> > > > #2
>>> >>>>>>>>>> > > > >> > >> points
>>> >>>>>>>>>> > > > >> > >> >> >>> (#jay?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now back
>>> to
>>> >>> our
>>> >>>>>>>>> normal
>>> >>>>>>>>>> > programming
>>> >>>>>>>>>> > > > ...)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are
>>> nice.
>>> >>>>>>> having
>>> >>>>>>>>> said
>>> >>>>>>>>>> > that, its
>>> >>>>>>>>>> > > > >> adding
>>> >>>>>>>>>> > > > >> > >> >> >>> complexity
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> (see
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>> https://github.com/addthis/
>>> >>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>> main/java/com/clearspring/
>>> >>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
>>> >>>>> result)
>>> >>>>>>>>> and would
>>> >>>>>>>>>> > require
>>> >>>>>>>>>> > > > >> anyone
>>> >>>>>>>>>> > > > >> > >> >> writing
>>> >>>>>>>>>> > > > >> > >> >> >>> > other
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clients
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go?
>>> Bash?
>>> >>>>> ;-) )
>>> >>>>>>>>> to
>>> >>>>>>>>>> > > get/implement
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > >> same,
>>> >>>>>>>>>> > > > >> > >> >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain
>>> (int
>>> >>> vs
>>> >>>>>>>>> string is
>>> >>>>>>>>>> > order of
>>> >>>>>>>>>> > > > >> > magnitude,
>>> >>>>>>>>>> > > > >> > >> >> this
>>> >>>>>>>>>> > > > >> > >> >> >>> > isnt).
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int
>>> namespacing vs
>>> >>>>>>> {int,
>>> >>>>>>>>> int}
>>> >>>>>>>>>> > > namespacing
>>> >>>>>>>>>> > > > >> are
>>> >>>>>>>>>> > > > >> > >> >> basically
>>> >>>>>>>>>> > > > >> > >> >> >>> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > same
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
>>> >>>>> namespacing
>>> >>>>>>>>> an int64
>>> >>>>>>>>>> > and
>>> >>>>>>>>>> > > > giving
>>> >>>>>>>>>> > > > >> > people
>>> >>>>>>>>>> > > > >> > >> >> while
>>> >>>>>>>>>> > > > >> > >> >> >>> > 2^32
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the part
>>> i
>>> >>>>> like
>>> >>>>>>>>> about this
>>> >>>>>>>>>> > is
>>> >>>>>>>>>> > > > letting
>>> >>>>>>>>>> > > > >> > >> people
>>> >>>>>>>>>> > > > >> > >> >> >>> have a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> large
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers with
>>> one
>>> >>>>>>>>> registration so
>>> >>>>>>>>>> > they
>>> >>>>>>>>>> > > > dont
>>> >>>>>>>>>> > > > >> > have
>>> >>>>>>>>>> > > > >> > >> to
>>> >>>>>>>>>> > > > >> > >> >> come
>>> >>>>>>>>>> > > > >> > >> >> >>> > back
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > every
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
>>> >>> plugin/header
>>> >>>>>>>>> they want to
>>> >>>>>>>>>> > > > "reserve".
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
>>> >>> 2016
>>> >>>>> at
>>> >>>>>>>>> 11:01 AM,
>>> >>>>>>>>>> > Roger
>>> >>>>>>>>>> > > > >> Hoover
>>> >>>>>>>>>> > > > >> > <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> roger.hoover@gmail.com>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
>>> >>> the
>>> >>>>>>>>> debate has
>>> >>>>>>>>>> > been
>>> >>>>>>>>>> > > about
>>> >>>>>>>>>> > > > >> > >> overhead +
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if
>>> we
>>> >>>>> have
>>> >>>>>>>>>> > considered a
>>> >>>>>>>>>> > > > varint
>>> >>>>>>>>>> > > > >> > >> encoding
>>> >>>>>>>>>> > > > >> > >> >> (
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>> https://developers.google.com/
>>> >>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the header
>>> >>> length
>>> >>>>>>>>> field (int32
>>> >>>>>>>>>> > in the
>>> >>>>>>>>>> > > > >> > proposal)
>>> >>>>>>>>>> > > > >> > >> >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
>>> >>> headers,
>>> >>>>>>> the
>>> >>>>>>>>>> > overhead would
>>> >>>>>>>>>> > > > be a
>>> >>>>>>>>>> > > > >> > >> single
>>> >>>>>>>>>> > > > >> > >> >> >>> byte
>>> >>>>>>>>>> > > > >> > >> >> >>> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for each
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128
>>> would
>>> >>>>> also
>>> >>>>>>>>> need only a
>>> >>>>>>>>>> > > single
>>> >>>>>>>>>> > > > >> byte?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov
>>> 9,
>>> >>>>> 2016
>>> >>>>>>> at
>>> >>>>>>>>> 6:43 AM,
>>> >>>>>>>>>> > > radai <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com
>>> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus -
>>> and
>>> >>>>> very
>>> >>>>>>>>> dangerous
>>> >>>>>>>>>> > (youre
>>> >>>>>>>>>> > > > >> > >> essentially
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary
>>> code
>>> >>>>> off
>>> >>>>>>>>> the
>>> >>>>>>>>>> > internet on
>>> >>>>>>>>>> > > > your
>>> >>>>>>>>>> > > > >> > >> servers
>>> >>>>>>>>>> > > > >> > >> >> ...
>>> >>>>>>>>>> > > > >> > >> >> >>> > bad
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > idea
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > without
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox,
>>> even
>>> >>>>> with)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
>>> >>> being
>>> >>>>> a
>>> >>>>>>>>> purely
>>> >>>>>>>>>> > > > administrative
>>> >>>>>>>>>> > > > >> > task
>>> >>>>>>>>>> > > > >> > >> - i
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
>>> >>> would,
>>> >>>>>>>>> really,
>>> >>>>>>>>>> > because
>>> >>>>>>>>>> > > > then my
>>> >>>>>>>>>> > > > >> > >> earlier
>>> >>>>>>>>>> > > > >> > >> >> >>> > point on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>>> remapping
>>> >>>>>>>>> process would
>>> >>>>>>>>>> > be
>>> >>>>>>>>>> > > > invalid,
>>> >>>>>>>>>> > > > >> > but
>>> >>>>>>>>>> > > > >> > >> at
>>> >>>>>>>>>> > > > >> > >> >> >>> > linkedin,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team
>>> im
>>> >>> in)
>>> >>>>>>> run
>>> >>>>>>>>> kafka
>>> >>>>>>>>>> > as a
>>> >>>>>>>>>> > > > service.
>>> >>>>>>>>>> > > > >> > we
>>> >>>>>>>>>> > > > >> > >> dont
>>> >>>>>>>>>> > > > >> > >> >> >>> > really
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > know
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
>>> >>>>>>>>> applications
>>> >>>>>>>>>> > that use
>>> >>>>>>>>>> > > > kafka)
>>> >>>>>>>>>> > > > >> > are
>>> >>>>>>>>>> > > > >> > >> up
>>> >>>>>>>>>> > > > >> > >> >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> at
>>> >>>>>>>>>> > > > >> > >> >> >>> > any
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > given
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
>>> >>> possible
>>> >>>>>>>>> (given the
>>> >>>>>>>>>> > > > existance of
>>> >>>>>>>>>> > > > >> > >> headers
>>> >>>>>>>>>> > > > >> > >> >> >>> and a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem)
>>> for
>>> >>>>> some
>>> >>>>>>>>>> > application to
>>> >>>>>>>>>> > > > >> "equip"
>>> >>>>>>>>>> > > > >> > >> their
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> producers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
>>> >>>>> required
>>> >>>>>>>>> plugin
>>> >>>>>>>>>> > without us
>>> >>>>>>>>>> > > > >> > knowing.
>>> >>>>>>>>>> > > > >> > >> i
>>> >>>>>>>>>> > > > >> > >> >> dont
>>> >>>>>>>>>> > > > >> > >> >> >>> > mean
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
>>> >>>>> want to
>>> >>>>>>>>> make the
>>> >>>>>>>>>> > point
>>> >>>>>>>>>> > > > that
>>> >>>>>>>>>> > > > >> > its
>>> >>>>>>>>>> > > > >> > >> not
>>> >>>>>>>>>> > > > >> > >> >> as
>>> >>>>>>>>>> > > > >> > >> >> >>> > simple
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
>>> across a
>>> >>>>>>>>> large-enough
>>> >>>>>>>>>> > > > organization.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed,
>>> Nov 9,
>>> >>>>> 2016
>>> >>>>>>>>> at 6:17
>>> >>>>>>>>>> > AM,
>>> >>>>>>>>>> > > > Magnus
>>> >>>>>>>>>> > > > >> > >> Edenhill
>>> >>>>>>>>>> > > > >> > >> >> <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
>>> >>> there
>>> >>>>> is
>>> >>>>>>> a
>>> >>>>>>>>> piece
>>> >>>>>>>>>> > missing
>>> >>>>>>>>>> > > in
>>> >>>>>>>>>> > > > >> the
>>> >>>>>>>>>> > > > >> > >> >> Strings
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > where
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>> pro-Stringers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason
>>> that
>>> >>> by
>>> >>>>>>>>> providing
>>> >>>>>>>>>> > unique
>>> >>>>>>>>>> > > > string
>>> >>>>>>>>>> > > > >> > >> >> >>> identifiers
>>> >>>>>>>>>> > > > >> > >> >> >>> > for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> everything
>>> >>>>> will
>>> >>>>>>>>> just
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
>>> >>> work
>>> >>>>>>> for
>>> >>>>>>>>> all
>>> >>>>>>>>>> > parts of
>>> >>>>>>>>>> > > the
>>> >>>>>>>>>> > > > >> > stream
>>> >>>>>>>>>> > > > >> > >> >> >>> pipeline.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
>>> >>>>> strings
>>> >>>>>>>>> dont mean
>>> >>>>>>>>>> > > anything
>>> >>>>>>>>>> > > > by
>>> >>>>>>>>>> > > > >> > >> >> themselves,
>>> >>>>>>>>>> > > > >> > >> >> >>> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > while we
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
>>> >>>>> envision
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
>>> >>>>> plugin
>>> >>>>>>>>> loader
>>> >>>>>>>>>> > that
>>> >>>>>>>>>> > > > >> downloads,
>>> >>>>>>>>>> > > > >> > >> >> compiles,
>>> >>>>>>>>>> > > > >> > >> >> >>> > links
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon
>>> as
>>> >>>>>>> they're
>>> >>>>>>>>> seen by
>>> >>>>>>>>>> > a
>>> >>>>>>>>>> > > > >> consumer, I
>>> >>>>>>>>>> > > > >> > >> dont
>>> >>>>>>>>>> > > > >> > >> >> >>> really
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> see
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
>>> dynamic
>>> >>>>> (and
>>> >>>>>>>>> fragile) in
>>> >>>>>>>>>> > > > practice.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the
>>> real
>>> >>>>> world
>>> >>>>>>>>> an
>>> >>>>>>>>>> > application
>>> >>>>>>>>>> > > > will
>>> >>>>>>>>>> > > > >> be
>>> >>>>>>>>>> > > > >> > >> >> >>> configured
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> with
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a set
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
>>> >>> add
>>> >>>>>>>>> (producer)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
>>> >>>>>>> (consumer)
>>> >>>>>>>>> headers.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is
>>> an
>>> >>>>>>>>> administrative
>>> >>>>>>>>>> > task
>>> >>>>>>>>>> > > > based
>>> >>>>>>>>>> > > > >> on
>>> >>>>>>>>>> > > > >> > >> what
>>> >>>>>>>>>> > > > >> > >> >> >>> > features a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > client
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>> needs/provides
>>> >>>>>>> and
>>> >>>>>>>>> results
>>> >>>>>>>>>> > in
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
>>> sort of
>>> >>>>>>>>> configuration
>>> >>>>>>>>>> > to
>>> >>>>>>>>>> > > > enable
>>> >>>>>>>>>> > > > >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> configure
>>> >>>>>>>>>> > > > >> > >> >> >>> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > desired
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since
>>> this
>>> >>>>> needs
>>> >>>>>>>>> to be kept
>>> >>>>>>>>>> > > > somewhat
>>> >>>>>>>>>> > > > >> in
>>> >>>>>>>>>> > > > >> > >> sync
>>> >>>>>>>>>> > > > >> > >> >> >>> across
>>> >>>>>>>>>> > > > >> > >> >> >>> > an
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no
>>> point
>>> >>> in
>>> >>>>>>>>> having
>>> >>>>>>>>>> > producers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
>>> headers
>>> >>> no
>>> >>>>>>>>> consumers
>>> >>>>>>>>>> > will
>>> >>>>>>>>>> > > read,
>>> >>>>>>>>>> > > > >> and
>>> >>>>>>>>>> > > > >> > >> vice
>>> >>>>>>>>>> > > > >> > >> >> >>> versa),
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > added
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
>>> assigning
>>> >>>>> an
>>> >>>>>>> id
>>> >>>>>>>>>> > namespace
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
>>> >>>>> plugin
>>> >>>>>>> as
>>> >>>>>>>>> it is
>>> >>>>>>>>>> > being
>>> >>>>>>>>>> > > > >> > configured
>>> >>>>>>>>>> > > > >> > >> >> should
>>> >>>>>>>>>> > > > >> > >> >> >>> be
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> 2016-11-09
>>> >>>>> 13:06
>>> >>>>>>>>> GMT+01:00
>>> >>>>>>>>>> > > Michael
>>> >>>>>>>>>> > > > >> > Pearce <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>> Michael.Pearce@ig.com>:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
>>> >>>>>>>>> following/catching
>>> >>>>>>>>>> > up on
>>> >>>>>>>>>> > > > what
>>> >>>>>>>>>> > > > >> > seems
>>> >>>>>>>>>> > > > >> > >> to
>>> >>>>>>>>>> > > > >> > >> >> be
>>> >>>>>>>>>> > > > >> > >> >> >>> an
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > active
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
>>> >>>>> sorry if
>>> >>>>>>>>> it may
>>> >>>>>>>>>> > seem
>>> >>>>>>>>>> > > > obvious
>>> >>>>>>>>>> > > > >> > but
>>> >>>>>>>>>> > > > >> > >> what
>>> >>>>>>>>>> > > > >> > >> >> >>> does
>>> >>>>>>>>>> > > > >> > >> >> >>> > MD
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > stand
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My
>>> take on
>>> >>>>>>>>> String vs Int:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
>>> >>> state
>>> >>>>>>>>> first I am
>>> >>>>>>>>>> > pro Int
>>> >>>>>>>>>> > > > (16
>>> >>>>>>>>>> > > > >> or
>>> >>>>>>>>>> > > > >> > >> 32).
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
>>> >>> though
>>> >>>>>>>>> playing
>>> >>>>>>>>>> > devils
>>> >>>>>>>>>> > > > advocate
>>> >>>>>>>>>> > > > >> > see a
>>> >>>>>>>>>> > > > >> > >> >> big
>>> >>>>>>>>>> > > > >> > >> >> >>> plus
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
>>> >>> keys,
>>> >>>>>>>>> this is
>>> >>>>>>>>>> > around
>>> >>>>>>>>>> > > > >> > integrating
>>> >>>>>>>>>> > > > >> > >> >> into an
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > existing
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
>>> >>>>> other
>>> >>>>>>>>> systems use
>>> >>>>>>>>>> > > String
>>> >>>>>>>>>> > > > >> based
>>> >>>>>>>>>> > > > >> > >> >> headers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > it
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
>>> >>> easier
>>> >>>>> for
>>> >>>>>>>>> these to
>>> >>>>>>>>>> > be
>>> >>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > into.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How
>>> with
>>> >>> Int
>>> >>>>>>>>> based
>>> >>>>>>>>>> > headers
>>> >>>>>>>>>> > > could
>>> >>>>>>>>>> > > > we
>>> >>>>>>>>>> > > > >> > >> provide
>>> >>>>>>>>>> > > > >> > >> >> a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>> integration
>>> >>>>>>>>> simple /
>>> >>>>>>>>>> > easy with
>>> >>>>>>>>>> > > > >> > transition
>>> >>>>>>>>>> > > > >> > >> >> flows
>>> >>>>>>>>>> > > > >> > >> >> >>> > over
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
>>> >>> luck
>>> >>>>>>>>> buddy
>>> >>>>>>>>>> > you're on
>>> >>>>>>>>>> > > your
>>> >>>>>>>>>> > > > >> own
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
>>> simply
>>> >>>>> hash
>>> >>>>>>>>> the string
>>> >>>>>>>>>> > into
>>> >>>>>>>>>> > > int
>>> >>>>>>>>>> > > > >> code
>>> >>>>>>>>>> > > > >> > >> and
>>> >>>>>>>>>> > > > >> > >> >> hope
>>> >>>>>>>>>> > > > >> > >> >> >>> > for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> no
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
>>> >>> back
>>> >>>>>>>>> though?)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
>>> >>>>> style
>>> >>>>>>> as
>>> >>>>>>>>>> > mentioned by
>>> >>>>>>>>>> > > > nacho.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > ______________________________
>>> >>>>>>>>>> > > > >> > __________
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
>>> >>> radai
>>> >>>>> <
>>> >>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
>>> >>>>>>> Wednesday,
>>> >>>>>>>>>> > November 9,
>>> >>>>>>>>>> > > 2016
>>> >>>>>>>>>> > > > >> > 8:12 AM
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
>>> >>>>>>>>> dev@kafka.apache.org
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> Subject:
>>> >>> Re:
>>> >>>>>>>>> [DISCUSS]
>>> >>>>>>>>>> > KIP-82 -
>>> >>>>>>>>>> > > > Add
>>> >>>>>>>>>> > > > >> > >> Record
>>> >>>>>>>>>> > > > >> > >> >> >>> Headers
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> thinking
>>> >>>>> about
>>> >>>>>>>>> it some
>>> >>>>>>>>>> > more,
>>> >>>>>>>>>> > > the
>>> >>>>>>>>>> > > > >> best
>>> >>>>>>>>>> > > > >> > >> way to
>>> >>>>>>>>>> > > > >> > >> >> >>> > transmit
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
>>> >>>>>>>>> consumers would
>>> >>>>>>>>>> > be to
>>> >>>>>>>>>> > > > put it
>>> >>>>>>>>>> > > > >> > in
>>> >>>>>>>>>> > > > >> > >> the
>>> >>>>>>>>>> > > > >> > >> >> MD
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> response
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it
>>> should
>>> >>> be
>>> >>>>>>>>> discussed
>>> >>>>>>>>>> > now.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
>>> >>> Nov
>>> >>>>> 9,
>>> >>>>>>>>> 2016 at
>>> >>>>>>>>>> > 12:09
>>> >>>>>>>>>> > > AM,
>>> >>>>>>>>>> > > > >> > radai <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> >>>>>>> radai.rosenblatt@gmail.com
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im
>>> not
>>> >>>>>>> opposed
>>> >>>>>>>>> to the
>>> >>>>>>>>>> > idea of
>>> >>>>>>>>>> > > > >> > namespace
>>> >>>>>>>>>> > > > >> > >> >> >>> mapping.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> all
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > im
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not
>>> part
>>> >>>>> of
>>> >>>>>>>>> the "mvp"
>>> >>>>>>>>>> > and,
>>> >>>>>>>>>> > > > since
>>> >>>>>>>>>> > > > >> it
>>> >>>>>>>>>> > > > >> > >> >> requires
>>> >>>>>>>>>> > > > >> > >> >> >>> no
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> wire
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > format
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>> always
>>> >>> be
>>> >>>>>>>>> added later.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
>>> >>> its
>>> >>>>> not
>>> >>>>>>>>> as
>>> >>>>>>>>>> > simple as
>>> >>>>>>>>>> > > just
>>> >>>>>>>>>> > > > >> > >> >> configuring
>>> >>>>>>>>>> > > > >> > >> >> >>> MM
>>> >>>>>>>>>> > > > >> > >> >> >>> > to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> do
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say
>>> i've
>>> >>>>>>>>> implemented
>>> >>>>>>>>>> > large
>>> >>>>>>>>>> > > > message
>>> >>>>>>>>>> > > > >> > >> >> support as
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>> target
>>> >>>>>>> cluster
>>> >>>>>>>>> its been
>>> >>>>>>>>>> > > > remapped
>>> >>>>>>>>>> > > > >> to
>>> >>>>>>>>>> > > > >> > >> >> {999,1}.
>>> >>>>>>>>>> > > > >> > >> >> >>> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
>>> >>> need
>>> >>>>> to
>>> >>>>>>>>> be told
>>> >>>>>>>>>> > to look
>>> >>>>>>>>>> > > > for
>>> >>>>>>>>>> > > > >> the
>>> >>>>>>>>>> > > > >> > >> large
>>> >>>>>>>>>> > > > >> > >> >> >>> > message
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
>>> >>>>>>> {999,1}.
>>> >>>>>>>>> doable,
>>> >>>>>>>>>> > but
>>> >>>>>>>>>> > > > tricky.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On
>>> Tue,
>>> >>>>> Nov
>>> >>>>>>> 8,
>>> >>>>>>>>> 2016 at
>>> >>>>>>>>>> > 10:29
>>> >>>>>>>>>> > > > PM,
>>> >>>>>>>>>> > > > >> > Gwen
>>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>> gwen@confluent.io
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> While
>>> >>> you
>>> >>>>>>> can
>>> >>>>>>>>> do
>>> >>>>>>>>>> > whatever
>>> >>>>>>>>>> > > you
>>> >>>>>>>>>> > > > >> want
>>> >>>>>>>>>> > > > >> > >> with a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> namespace
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > your
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
>>> >>> I'd
>>> >>>>>>>>> expect is
>>> >>>>>>>>>> > for each
>>> >>>>>>>>>> > > > app
>>> >>>>>>>>>> > > > >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> namespaces
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So
>>> if I
>>> >>>>>>>>> accidentally
>>> >>>>>>>>>> > used
>>> >>>>>>>>>> > > 666
>>> >>>>>>>>>> > > > for
>>> >>>>>>>>>> > > > >> > my
>>> >>>>>>>>>> > > > >> > >> HR
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> department,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > still
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
>>> >>>>>>> RadaiApp,
>>> >>>>>>>>> I can
>>> >>>>>>>>>> > config
>>> >>>>>>>>>> > > > >> > >> "namespace=42"
>>> >>>>>>>>>> > > > >> > >> >> >>> for
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
>>> >>> look
>>> >>>>>>>>> normal.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
>>> >>>>> means
>>> >>>>>>>>> you only
>>> >>>>>>>>>> > need to
>>> >>>>>>>>>> > > > sync
>>> >>>>>>>>>> > > > >> > usage
>>> >>>>>>>>>> > > > >> > >> >> >>> inside
>>> >>>>>>>>>> > > > >> > >> >> >>> > your
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > own
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> Still
>>> >>>>> hard,
>>> >>>>>>>>> but
>>> >>>>>>>>>> > somewhat
>>> >>>>>>>>>> > > > easier
>>> >>>>>>>>>> > > > >> > than
>>> >>>>>>>>>> > > > >> > >> >> syncing
>>> >>>>>>>>>> > > > >> > >> >> >>> > with
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On
>>> Tue,
>>> >>>>> Nov
>>> >>>>>>>>> 8, 2016
>>> >>>>>>>>>> > at 10:07
>>> >>>>>>>>>> > > > PM,
>>> >>>>>>>>>> > > > >> > >> radai <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>>>>>> radai.rosenblatt@gmail.com>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> and
>>> >>> we
>>> >>>>> can
>>> >>>>>>>>> start
>>> >>>>>>>>>> > with
>>> >>>>>>>>>> > > > >> {namespace,
>>> >>>>>>>>>> > > > >> > >> id}
>>> >>>>>>>>>> > > > >> > >> >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> no
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> always
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> add
>>> >>> it
>>> >>>>>>>>> later on
>>> >>>>>>>>>> > if/when
>>> >>>>>>>>>> > > > >> > collisions
>>> >>>>>>>>>> > > > >> > >> >> >>> actually
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
>>> >>>>>>> problem).
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> every
>>> >>>>>>>>> interested
>>> >>>>>>>>>> > party (so
>>> >>>>>>>>>> > > > orgs
>>> >>>>>>>>>> > > > >> > or
>>> >>>>>>>>>> > > > >> > >> >> >>> > individuals)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > could
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > then
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > register
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> >>> prefix
>>> >>>>> (0
>>> >>>>>>> =
>>> >>>>>>>>>> > reserved, 1 =
>>> >>>>>>>>>> > > > >> > confluent
>>> >>>>>>>>>> > > > >> > >> ...
>>> >>>>>>>>>> > > > >> > >> >> >>> 666
>>> >>>>>>>>>> > > > >> > >> >> >>> > = me
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> the
>>> >>>>> 2nd ID
>>> >>>>>>>>> - so once
>>> >>>>>>>>>> > > > linkedin
>>> >>>>>>>>>> > > > >> > >> >> registers,
>>> >>>>>>>>>> > > > >> > >> >> >>> say
>>> >>>>>>>>>> > > > >> > >> >> >>> > 3,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > then
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
>>> >>> use
>>> >>>>> {3,
>>> >>>>>>>>> *} with a
>>> >>>>>>>>>> > > > reasonable
>>> >>>>>>>>>> > > > >> > >> >> >>> expectation
>>> >>>>>>>>>> > > > >> > >> >> >>> > to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> anything
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> else.
>>> >>>>>>>>> further
>>> >>>>>>>>>> > partitioning
>>> >>>>>>>>>> > > > of
>>> >>>>>>>>>> > > > >> > that *
>>> >>>>>>>>>> > > > >> > >> >> >>> becomes
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> >>>>> "upstream
>>> >>>>>>>>>> > registration"
>>> >>>>>>>>>> > > of a
>>> >>>>>>>>>> > > > >> > >> namespace
>>> >>>>>>>>>> > > > >> > >> >> >>> only
>>> >>>>>>>>>> > > > >> > >> >> >>> > has
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
>>> >>> Tue,
>>> >>>>>>> Nov
>>> >>>>>>>>> 8, 2016
>>> >>>>>>>>>> > at
>>> >>>>>>>>>> > > 9:03
>>> >>>>>>>>>> > > > PM,
>>> >>>>>>>>>> > > > >> > >> James
>>> >>>>>>>>>> > > > >> > >> >> >>> Cheng <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>> >>>>> wushujames@gmail.com
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> > On
>>> >>>>> Nov
>>> >>>>>>>>> 8, 2016,
>>> >>>>>>>>>> > at 5:54
>>> >>>>>>>>>> > > > PM,
>>> >>>>>>>>>> > > > >> > Gwen
>>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>> gwen@confluent.io
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>> Thank
>>> >>>>>>>>> you so
>>> >>>>>>>>>> > much for
>>> >>>>>>>>>> > > > this
>>> >>>>>>>>>> > > > >> > clear
>>> >>>>>>>>>> > > > >> > >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> fair
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> arguments.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>> I'm
>>> >>>>> in
>>> >>>>>>>>> favor of
>>> >>>>>>>>>> > ints.
>>> >>>>>>>>>> > > > Not a
>>> >>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
>>> >>>>>>>>>> > > > >> > >> >> >>> > but
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > in
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>> Even
>>> >>>>>>>>> more in
>>> >>>>>>>>>> > favor of
>>> >>>>>>>>>> > > > >> Magnus's
>>> >>>>>>>>>> > > > >> > >> >> >>> > decentralized
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>> tweak:
>>> >>>>>>>>> add a
>>> >>>>>>>>>> > namespace
>>> >>>>>>>>>> > > > for
>>> >>>>>>>>>> > > > >> > >> headers.
>>> >>>>>>>>>> > > > >> > >> >> >>> This
>>> >>>>>>>>>> > > > >> > >> >> >>> > will
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > allow
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > each
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>> use
>>> >>>>>>>>> whatever IDs
>>> >>>>>>>>>> > it
>>> >>>>>>>>>> > > wants
>>> >>>>>>>>>> > > > >> > >> >> internally,
>>> >>>>>>>>>> > > > >> > >> >> >>> and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> then
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > let
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> >>> deploying
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>> the
>>> >>>>> app
>>> >>>>>>>>> figure
>>> >>>>>>>>>> > out an
>>> >>>>>>>>>> > > > >> > available
>>> >>>>>>>>>> > > > >> > >> >> >>> namespace
>>> >>>>>>>>>> > > > >> > >> >> >>> > ID
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> > So
>>> >>>>>>>>>> > > > >> > io.confluent.schema-registry
>>> >>>>>>>>>> > > > >> > >> can
>>> >>>>>>>>>> > > > >> > >> >> be
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> >>>>> deployment
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>> and
>>> >>>>>>> 0x57
>>> >>>>>>>>> on
>>> >>>>>>>>>> > yours, and
>>> >>>>>>>>>> > > > the
>>> >>>>>>>>>> > > > >> > poor
>>> >>>>>>>>>> > > > >> > >> guys
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>> worry
>>> >>>>>>>>> about that.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>> Gwen,
>>> >>>>> if
>>> >>>>>>> I
>>> >>>>>>>>>> > understand
>>> >>>>>>>>>> > > your
>>> >>>>>>>>>> > > > >> > example
>>> >>>>>>>>>> > > > >> > >> >> >>> right, an
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > application
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>> decide to
>>> >>>>>>>>> use 0x01
>>> >>>>>>>>>> > in one
>>> >>>>>>>>>> > > > >> > >> deployment,
>>> >>>>>>>>>> > > > >> > >> >> and
>>> >>>>>>>>>> > > > >> > >> >> >>> > that
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > means
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > that
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> message
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> is
>>> >>>>>>> written
>>> >>>>>>>>> into the
>>> >>>>>>>>>> > > > broker, it
>>> >>>>>>>>>> > > > >> > >> will be
>>> >>>>>>>>>> > > > >> > >> >> >>> > saved on
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>> specific
>>> >>>>>>>>> namespace
>>> >>>>>>>>>> > > (0x01).
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> If
>>> >>> you
>>> >>>>>>>>> were to
>>> >>>>>>>>>> > mirror
>>> >>>>>>>>>> > > that
>>> >>>>>>>>>> > > > >> > message
>>> >>>>>>>>>> > > > >> > >> >> into
>>> >>>>>>>>>> > > > >> > >> >> >>> > another
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>> accompany
>>> >>>>>>>>> the
>>> >>>>>>>>>> > message,
>>> >>>>>>>>>> > > > right?
>>> >>>>>>>>>> > > > >> > What
>>> >>>>>>>>>> > > > >> > >> if
>>> >>>>>>>>>> > > > >> > >> >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>> other
>>> >>>>>>>>> cluster uses
>>> >>>>>>>>>> > 0x57?
>>> >>>>>>>>>> > > > They
>>> >>>>>>>>>> > > > >> > won't
>>> >>>>>>>>>> > > > >> > >> >> >>> > understand
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> I'm
>>> >>>>> not
>>> >>>>>>>>> sure
>>> >>>>>>>>>> > that's an
>>> >>>>>>>>>> > > > >> avoidable
>>> >>>>>>>>>> > > > >> > >> >> >>> problem. I
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > think it
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>> order
>>> >>>>> to
>>> >>>>>>>>> share
>>> >>>>>>>>>> > data, you
>>> >>>>>>>>>> > > > have
>>> >>>>>>>>>> > > > >> to
>>> >>>>>>>>>> > > > >> > >> also
>>> >>>>>>>>>> > > > >> > >> >> >>> have a
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shared
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>> understanding of
>>> >>>>>>>>>> > what the
>>> >>>>>>>>>> > > > >> > >> namespaces
>>> >>>>>>>>>> > > > >> > >> >> >>> mean.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> Which
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>> because
>>> >>>>>>> the
>>> >>>>>>>>>> > alternate
>>> >>>>>>>>>> > > > (sharing
>>> >>>>>>>>>> > > > >> > >> >> *nothing*
>>> >>>>>>>>>> > > > >> > >> >> >>> at
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> all)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>> would
>>> >>>>> be
>>> >>>>>>>>> no way to
>>> >>>>>>>>>> > > > understand
>>> >>>>>>>>>> > > > >> > each
>>> >>>>>>>>>> > > > >> > >> >> other.
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>> -James
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>> Gwen
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> > On
>>> >>>>> Tue,
>>> >>>>>>>>> Nov 8,
>>> >>>>>>>>>> > 2016 at
>>> >>>>>>>>>> > > > 4:23
>>> >>>>>>>>>> > > > >> > PM,
>>> >>>>>>>>>> > > > >> > >> >> radai <
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> >>>>>>>>> radai.rosenblatt@gmail.com
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>> wrote:
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>
>>> >>> +1
>>> >>>>> for
>>> >>>>>>>>> sean's
>>> >>>>>>>>>> > > document.
>>> >>>>>>>>>> > > > it
>>> >>>>>>>>>> > > > >> > >> covers
>>> >>>>>>>>>> > > > >> > >> >> >>> pretty
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> much
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > all
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>
>>> >>>>>>> provides
>>> >>>>>>>>>> > concrete
>>> >>>>>>>>>> > > > figures
>>> >>>>>>>>>> > > > >> to
>>> >>>>>>>>>> > > > >> > >> argue
>>> >>>>>>>>>> > > > >> > >> >> >>> about
>>> >>>>>>>>>> > > > >> > >> >> >>> > :-)
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> >>
>>> >>>>>>>>> (nit-picking -
>>> >>>>>>>>>> > used
>>> >>>>>>>>>> > > the
>>> >>>>>>>>>> > > > >> same
>>> >>>>>>>>>> > > > >> > >> xkcd
>>> >>>>>>>>>> > > > >> > >> >> >>> twice,
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> also
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > trove
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > has
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
>>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> >>>>> superceded
>>> >>>>>>>>>> > > > >> > >> >
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >> > --
>>> >>>>>>>>>> > > > >> > Gwen Shapira
>>> >>>>>>>>>> > > > >> > Product Manager | Confluent
>>> >>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>> >>>>>>>>>> > > > >> > Follow us: Twitter | blog
>>> >>>>>>>>>> > > > >> >
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >> --
>>> >>>>>>>>>> > > > >> *Todd Palino*
>>> >>>>>>>>>> > > > >> Staff Site Reliability Engineer
>>> >>>>>>>>>> > > > >> Data Infrastructure Streaming
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > > >> linkedin.com/in/toddpalino
>>> >>>>>>>>>> > > > >>
>>> >>>>>>>>>> > > >
>>> >>>>>>>>>> > > >
>>> >>>>>>>>>> > > >
>>> >>>>>>>>>> > > > --
>>> >>>>>>>>>> > > > Gwen Shapira
>>> >>>>>>>>>> > > > Product Manager | Confluent
>>> >>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>> >>>>>>>>>> > > > Follow us: Twitter | blog
>>> >>>>>>>>>> > > >
>>> >>>>>>>>>> > >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> >
>>> >>>>>>>>>> > The information contained in this email is strictly
>>> >>>>> confidential
>>> >>>>>>>>> and for
>>> >>>>>>>>>> > the use of the addressee only, unless otherwise
>>> indicated.
>>> >>> If
>>> >>>>> you
>>> >>>>>>>>> are not
>>> >>>>>>>>>> > the intended recipient, please do not read, copy, use or
>>> >>>>> disclose
>>> >>>>>>>>> to others
>>> >>>>>>>>>> > this message or any attachment. Please also notify the
>>> >>> sender
>>> >>>>> by
>>> >>>>>>>>> replying
>>> >>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and
>>> then
>>> >>>>> delete
>>> >>>>>>>>> the email
>>> >>>>>>>>>> > and any copies of it. Opinions, conclusion (etc) that do
>>> not
>>> >>>>>>>>> relate to the
>>> >>>>>>>>>> > official business of this company shall be understood as
>>> >>>>> neither
>>> >>>>>>>>> given nor
>>> >>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets
>>> Limited
>>> >>> (a
>>> >>>>>>>>> company
>>> >>>>>>>>>> > registered in England and Wales, company number 04008957)
>>> >>> and
>>> >>>>> IG
>>> >>>>>>>>> Index
>>> >>>>>>>>>> > Limited (a company registered in England and Wales,
>>> company
>>> >>>>>>> number
>>> >>>>>>>>>> > 01190902). Registered address at Cannon Bridge House, 25
>>> >>>>> Dowgate
>>> >>>>>>>>> Hill,
>>> >>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register number
>>> >>>>> 195355)
>>> >>>>>>>>> and IG
>>> >>>>>>>>>> > Index Limited (register number 114059) are authorised and
>>> >>>>>>>>> regulated by the
>>> >>>>>>>>>> > Financial Conduct Authority.
>>> >>>>>>>>>> >
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> The information contained in this email is strictly confidential
>>> >>> and
>>> >>>>>>> for
>>> >>>>>>>>> the use of the addressee only, unless otherwise indicated. If you
>>> >>> are
>>> >>>>>>> not
>>> >>>>>>>>> the intended recipient, please do not read, copy, use or disclose
>>> >>> to
>>> >>>>>>> others
>>> >>>>>>>>> this message or any attachment. Please also notify the sender by
>>> >>>>>>> replying
>>> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
>>> >>> the
>>> >>>>>>> email
>>> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
>>> relate
>>> >>>>> to
>>> >>>>>>> the
>>> >>>>>>>>> official business of this company shall be understood as neither
>>> >>>>> given
>>> >>>>>>> nor
>>> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
>>> >>> company
>>> >>>>>>>>> registered in England and Wales, company number 04008957) and IG
>>> >>>>> Index
>>> >>>>>>>>> Limited (a company registered in England and Wales, company
>>> number
>>> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>>> >>>>> Hill,
>>> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
>>> >>>>> and IG
>>> >>>>>>>>> Index Limited (register number 114059) are authorised and
>>> regulated
>>> >>>>> by
>>> >>>>>>> the
>>> >>>>>>>>> Financial Conduct Authority.
>>> >>>>>>>>>
>>> >>>>>>>
>>> >>>>>
>>> >>> The information contained in this email is strictly confidential and
>>> for
>>> >>> the use of the addressee only, unless otherwise indicated. If you are
>>> not
>>> >>> the intended recipient, please do not read, copy, use or disclose to
>>> others
>>> >>> this message or any attachment. Please also notify the sender by
>>> replying
>>> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
>>> email
>>> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
>>> the
>>> >>> official business of this company shall be understood as neither given
>>> nor
>>> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>>> >>> registered in England and Wales, company number 04008957) and IG Index
>>> >>> Limited (a company registered in England and Wales, company number
>>> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>>> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
>>> IG
>>> >>> Index Limited (register number 114059) are authorised and regulated by
>>> the
>>> >>> Financial Conduct Authority.
>>> >>>
>>> >
>>> >
>>>
>>> The information contained in this email is strictly confidential and for
>>> the use of the addressee only, unless otherwise indicated. If you are not
>>> the intended recipient, please do not read, copy, use or disclose to others
>>> this message or any attachment. Please also notify the sender by replying
>>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>>> official business of this company shall be understood as neither given nor
>>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>>> registered in England and Wales, company number 04008957) and IG Index
>>> Limited (a company registered in England and Wales, company number
>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>>> Index Limited (register number 114059) are authorised and regulated by the
>>> Financial Conduct Authority.
>>>
>
>
>
> --
> Nacho - Ignacio Solis - isolis@igso.net
>
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ignacio Solis <is...@igso.net>.
I would be willing to compromise on Strings as keys if that moves all
of us forward.
Right now headers are blocking a few of the things we want to do.
Nacho
On Thu, Feb 9, 2017 at 2:14 PM, Jay Kreps <ja...@confluent.io> wrote:
> I'm neutral positive on adding headers conceptually. I'm -1 on a proposal
> using ints for header names. I just don't think a global namespace with
> integer names is an acceptable end-user interface given that our goal is to
> encourage a federated ecosystem of header users. Not trying to be a pain in
> the ass, I just think it would be a bit of a wart.
>
> -Jay
>
> On Thu, Feb 9, 2017 at 12:12 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> As we'd like to start work on this to make it into the next release in a
>> few months.
>>
>> If we were to put this to the vote today who would actually give this a -1
>> vote?
>>
>> Based on the number of identified usecases (some stronger than others)
>>
>> Also if we were to keep with int keys based on all the discussion thread
>> for the pros vs cons who would give a -1 as it stands? Eg anyone believe it
>> simply wouldn't work.
>>
>> If not so we make progress would it be sensible to put this to vote? So we
>> can start work?
>>
>> Cheers
>> Mike
>>
>>
>> Sent using OWA for iPhone
>> ________________________________________
>> From: Jeroen van Disseldorp <je...@axual.io>
>> Sent: Wednesday, February 1, 2017 2:43:45 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>> I'm lagging a bit here too, but want to respond to the usefulness of
>> this KIP for two use cases: bidirectional mirroring and smart cluster
>> switches.
>>
>> Bidirectional mirroring
>>
>> For a Dutch bank we have currently implemented bidirectional mirroring
>> using our own headers, which we encode in the Value using a custom
>> serializer. It encodes just one byte extra, containing a "copy/original"
>> bit. When applications serialize an avro object, the bit is unset by
>> default. Our mirroring mechanism subscribes to all topics, checks the
>> bit, and when unset the bit is set and the message is forwarded to the
>> other DC.
>>
>> The overhead of this is pretty low (1 byte) and it's cheap, since the
>> mirroring mechanism does not need to fully deserialize the value, but
>> just check+flip one bit in the byte[] value. On the receiving end, the
>> mirroring mechanism sees the message for the second time, but notices
>> the message is a copy already and ignores it.
>>
>> The mechanism has been in production since last year and it works really
>> well. Its downside however is the incompatibility with avro serialized
>> values, as the whole organization is mandated to use our own custom
>> serde and we were forced to create a wrapper client library around the
>> default kafka client. The callback we offer to applications is very
>> similar to that proposed in the KIP, as every message our users get
>> consists of <Metadata, K, V>.
>>
>> Additionally our header contains a message id and serialization
>> timestamp (invented before 0.10.0.0). These fields could also be done
>> away with when this KIP is implemented.
>>
>> Smart cluster switches
>>
>> Another case we are considering to use headers for is enabling "smart
>> cluster switches" for consumers and producers in a way that would avoid
>> duplicates and/or missed messages in most cases. This involves encoding
>> a producer id (app name or id), producer instance id (uuid generated
>> upon producer's start) and sequence number in every message. We expect
>> to use one sequence per partition.
>>
>> These fields enable our consumers to keep track of which messages were
>> seen without relying on offsets, which is a cluster-bound value.
>> Consumers keep track of seen messages by keeping a small administration
>> of ranges of sequence numbers per producer instance and partition. Since
>> sequence numbers are always increasing per producer and partition, the
>> memory consumption of such administration is pretty low, while being
>> meaningful across clusters. After every poll() we can commit the
>> administration to a topic in much the same way as we do with offsets.
>> The difference is that this topic can be mirrored to the other cluster
>> and upon a switch of consumer, all instances can use the topic to
>> initialize themselves and skip all messages already processed.
>>
>> Hope these views help in the upcoming decision on this KIP.
>>
>> Regards
>> Jeroen
>>
>>
>>
>> On 09/01/2017 20:40, Gwen Shapira wrote:
>> > Mirroring + ClusterIDs:
>> >
>> > Kafka clusters already have an automatically generated ID that is
>> > exposed to clients:
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
>> >
>> > It sounds trivial to get MirrorMaker's consumer to grab this ID (which
>> > always exists) from the source cluster and add it to the message
>> > headers.
>> >
>> > Obviously adding stuff to the message makes it larger so this should
>> > be a feature you should be able to turn on/off depending on the
>> > replication requirements. If you need the feature - you pay the price.
>> > I don't think there is an immediate need to put an effort into
>> > shrinking the Cluster ID, but others can comment on that.
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
>> >> Hey Michael,
>> >>
>> >> Sorry for lagging, I've been out for the holiday break. What would help
>> me
>> >> would be to get to a consolidated list of use cases we've worked out and
>> >> think add real value. I think getting down to the details really matters
>> >> for these.
>> >>
>> >> The two lists I'm aware of are the one in the KIP and the one you gave
>> in
>> >> this email thread. Both seem to be more at the sketch level. Let me
>> review
>> >> the use cases and see if I have it right. If so let's get it in the
>> design
>> >> proposal.
>> >>
>> >> Your use cases:
>> >>
>> >> 1. Tracing IDs -- I get this use case and think I see how it would
>> work.
>> >> I think because the tracing is from a third party system there is
>> some
>> >> value to having a header versus a custom format to make this work
>> fully
>> >> transparently.
>> >> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
>> >> thinking below
>> >> 3. End-to-end encryption -- I think what you are saying is that if
>> you
>> >> want "routing' fields that are accessible in transit without
>> decrypting you
>> >> need a scheme to leave this decrypted.I think the idea is to avoid
>> >> decrypting to get a header, but I'm not sure if I get how this is
>> better
>> >> than a custom message format that doesn't encrypt the things you
>> don't want
>> >> encrypted since you have to be aware of the scheme anyway.
>> >> 4. EOS -- I'm not convinced this is a good approach for EOS.
>> >>
>> >> Linked from KIP:
>> >>
>> >> 1. Schema-Registry-aware avro serialization -- not clear how this is
>> >> better than keeping the schema id in the value
>> >> 2. Polyglot kafka topics -- marginally better than a container
>> format,
>> >> but of course you still have to all agree on the the header and
>> >> header=>serialization mapping so not sure if it's really better
>> than a
>> >> container format. Also not wild about polyglot topics :-)
>> >> 3. End-to-End routing trail including mirroring --- I'd put this
>> under
>> >> tracing ids
>> >> 4. Cross-service process tracing -- ditto
>> >> 5. Signing -- I like this use case but it seems to depend on a
>> feature
>> >> we don't have and aren't proposing adding, right?
>> >> 6. Per-record custom compression -- why do you want this?
>> compressing
>> >> per message won't work well compared to batch compression, right?
>> >> 7. Large Message Support -- I think we can agree that although this
>> >> might work it's clearly not the right way to support large messages
>> in Kafka
>> >>
>> >> Matthias
>> >>
>> >> 1. Control messages --- Matthias, not sure if I get the motivation
>> in
>> >> streams for this. If we did want to do it, wouldn't it be easier to
>> >> implement and by allowing users to send control messages (in the
>> EOS sense)
>> >> and adding a consumer config to allow either passing these on or
>> >> suppressing them?
>> >>
>> >> So I if I were to summarize I'd list the use cases as follows:
>> >>
>> >> - Much better with headers: tracing IDs
>> >> - Somewhat better: end-to-end encryption, polyglot topics
>> >> - Don't buy: Mirroring (though perhaps could be made to work with
>> >> changes to the proposal), EOS, large messages, compression
>> >> - Not sure: Control messages
>> >>
>> >> Does this match where other people are at? From my point of view for
>> >> something like this where we are changing the core data model we should
>> >> ideally get a handful of use cases to the point where they are really
>> well
>> >> thought out and we know they can use the feature. I think we have one
>> solid
>> >> one with tracing IDs, and I think maybe mirroring could be another with
>> >> some work. I'm less sure about the others.
>> >>
>> >> Bi-directional Mirroring
>> >> I think bidirectional mirroring could potentially become a compelling
>> use
>> >> case if we an find a way to make it work. If I understand it, the
>> proposal
>> >> is to add a header like "origin-cluster" with the cluster id of the
>> Kafka
>> >> cluster. Then modify the mirroring tool to disallow mirroring a message
>> >> back to it's own origin cluster. This basic idea is definitely workable
>> and
>> >> useful, but can we really get to a good implementation using headers?
>> >> I imagine the approach would be to add this header in the producer if
>> some
>> >> config is set. There are two problems:
>> >>
>> >> - You add a pretty big honking cluster ID to each message. For small
>> >> messages this maybe larger than the contents itself.
>> >> - What happens if someone somewhere forgets to set that config on
>> some
>> >> client somewhere?
>> >>
>> >> If someone forgets to set the config in some client somewhere I guess
>> you
>> >> have various options in mirror maker:
>> >>
>> >> 1. drop the data (pretty bad, now your data is out of sync)
>> >> 2. crash (pretty bad--how do you recover)
>> >> 3. insert some default (pretty bad since would also lead to data
>> loss if
>> >> that's not right)
>> >>
>> >> Basically although I think a cluster tag could be a good way to do
>> >> bi-directional mirroring, I'm not sure this naive approach I described
>> >> would get us to a usable implementation. This could be fixable. I think
>> if
>> >> there was some scheme to check required headers on the server that could
>> >> alleviate the missing header problem. Not really sure what to do about
>> the
>> >> message bloat problem, but maybe there could be some way to shrink the
>> >> cluster id?
>> >>
>> >> Cheers,
>> >>
>> >> -Jay
>> >>
>> >>
>> >> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Michael.Pearce@ig.com
>> >
>> >> wrote:
>> >>
>> >>> @Jay, Ismael and a few others.
>> >>>
>> >>> Based on the discussion on kip-87, and the additional use case by
>> Matthias.
>> >>>
>> >>> Just for clarity do we now have consensus on the earlier points (1)
>> >>> headers are useful (2) headers should be native in Kafka?
>> >>>
>> >>> Cheers
>> >>> Mike
>> >>> ________________________________________
>> >>> From: radai <ra...@gmail.com>
>> >>> Sent: Monday, December 19, 2016 5:01 PM
>> >>> To: dev@kafka.apache.org
>> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>> >>>
>> >>> having said that im perfectly fine with something like a "no payload"
>> flag
>> >>> (as opposed to null payload), which would mean such a msg is not
>> returned
>> >>> from poll() unless the user code specifically "opts in" and would serve
>> >>> only to carry headers (making it a control message)
>> >>>
>> >>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com>
>> wrote:
>> >>>
>> >>>> if all you want is to tag "end of stream" wouldnt a header on the last
>> >>> msg
>> >>>> in the stream be enough?
>> >>>>
>> >>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
>> matthias@confluent.io>
>> >>>> wrote:
>> >>>>
>> >>>>> The details about headers for control messages are still to define.
>> But
>> >>>>> yes, the idea is to have some common default behavior that clients
>> would
>> >>>>> need to implement.
>> >>>>>
>> >>>>> The point is, that "regular headers" add meta data to regular
>> messages.
>> >>>>> Thus, those messages will be returned to the user via .poll(). And
>> after
>> >>>>> the message is received the user can check if meta data is present
>> and
>> >>>>> read it.
>> >>>>>
>> >>>>> For control messages, we do not want those to pop up via .poll() as
>> >>>>> those are no regular messages. A client would need to opt-in to see
>> >>>>> those messages (either via poll() or maybe a callback). Thus, we need
>> >>>>> some special (standardized) header IDs that indicate control messages
>> >>>>> that should not be returned to the user via poll() by default.
>> >>>>>
>> >>>>>
>> >>>>> -Matthias
>> >>>>>
>> >>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
>> >>>>>> Matthias,
>> >>>>>>
>> >>>>>> Thanks for your input. I'm +1 on control messages as they seem to
>> be
>> >>>>> the
>> >>>>>> simplest way to implement watermarks (
>> >>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
>> ),
>> >>> a
>> >>>>>> feature that would add a lot of value to Kafka Streams IMHO.
>> >>>>>>
>> >>>>>> Your argument that the control-message indicator belongs in the
>> >>>>> client-only
>> >>>>>> section of the record format make sense. Just to make sure I
>> >>>>> understand,
>> >>>>>> are you suggesting that control messages would be indicated by a
>> >>>>> standard
>> >>>>>> reserved header? By standard, I mean that ALL Kafka consumers would
>> >>>>> know
>> >>>>>> to handle these messages differently (possibly just ignoring them).
>> >>>>> This
>> >>>>>> would need to be added to the specification of the consumer protocol
>> >>> so
>> >>>>>> that all Kafka clients implement it, right? I think it's a good
>> idea
>> >>>>> but
>> >>>>>> just checking.
>> >>>>>>
>> >>>>>> Cheers,
>> >>>>>>
>> >>>>>> Roger
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>> >>> matthias@confluent.io
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> Yes and no. I did overload the term "control message".
>> >>>>>>>
>> >>>>>>> EOS control messages are for client-broker communication and thus
>> >>> never
>> >>>>>>> exposed to any application. And I think this is a good design
>> because
>> >>>>>>> broker needs to understand those control messages. Thus, this
>> should
>> >>> be
>> >>>>>>> a protocol change.
>> >>>>>>>
>> >>>>>>> The type of control messages I have in mind are for client-client
>> >>>>>>> (application-application) communication and the broker is agnostic
>> to
>> >>>>>>> them. Thus, it should not be a protocol change.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> -Matthias
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 12/14/16 9:42 AM, radai wrote:
>> >>>>>>>> arent control messages getting pushed as their own top level
>> >>> protocol
>> >>>>>>>> change (and a fairly massive one) for the transactions KIP ?
>> >>>>>>>>
>> >>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>> >>>>> matthias@confluent.io>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>> Hi,
>> >>>>>>>>>
>> >>>>>>>>> I want to add a completely new angle to this discussion. For
>> this,
>> >>> I
>> >>>>>>>>> want to propose an extension for the headers feature that enables
>> >>> new
>> >>>>>>>>> uses cases -- and those new use cases might convince people to
>> >>>>> support
>> >>>>>>>>> headers (of course including the larger scoped proposal).
>> >>>>>>>>>
>> >>>>>>>>> Extended Proposal:
>> >>>>>>>>>
>> >>>>>>>>> Allow messages with a certain header key to be special "control
>> >>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an
>> application
>> >>>>> via
>> >>>>>>>>> .poll().
>> >>>>>>>>>
>> >>>>>>>>> Thus, a consumer client would automatically skip over those
>> >>>>> messages. If
>> >>>>>>>>> an application knows about embedded control messages, it can
>> "sing
>> >>>>> up"
>> >>>>>>>>> to those messages by the consumer client and either get a
>> callback
>> >>> or
>> >>>>>>>>> the consumer auto-drop for this messages gets disabled (allowing
>> to
>> >>>>>>>>> consumer those messages via poll()).
>> >>>>>>>>>
>> >>>>>>>>> (The details need further considerations/discussion. I just want
>> to
>> >>>>>>>>> sketch the main idea.)
>> >>>>>>>>>
>> >>>>>>>>> Usage:
>> >>>>>>>>>
>> >>>>>>>>> There is a shared topic (ie, used by multiple applications) and a
>> >>>>>>>>> producer application wants to embed a special message in the
>> topic
>> >>>>> for a
>> >>>>>>>>> dedicated consumer application. Because only one application will
>> >>>>>>>>> understand this message, it cannot be a regular message as this
>> >>> would
>> >>>>>>>>> break all applications that do not understand this message. The
>> >>>>> producer
>> >>>>>>>>> application would set a special metadata key and no consumer
>> >>>>> application
>> >>>>>>>>> would see this control message by default because they did not
>> >>> enable
>> >>>>>>>>> their consumer client to return this message in poll() (and the
>> >>>>> client
>> >>>>>>>>> would just drop this message with special metadata key). Only the
>> >>>>> single
>> >>>>>>>>> application that should receive this message, will subscribe to
>> >>> this
>> >>>>>>>>> message on its consumer client and process it.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Concrete Use Case: Kafka Streams
>> >>>>>>>>>
>> >>>>>>>>> In Kafka Streams, we would like to propagate "control messages"
>> >>> from
>> >>>>>>>>> subtopology to subtopology. There are multiple scenarios for
>> which
>> >>>>> this
>> >>>>>>>>> would be useful. For example, currently we do not guarantee a
>> >>>>>>>>> "consistent shutdown" of an application. By this, I mean that
>> input
>> >>>>>>>>> records might not be completely processed by the whole topology
>> >>>>> because
>> >>>>>>>>> the application shutdown happens "in between" and an intermediate
>> >>>>> result
>> >>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user would
>> see
>> >>>>> an
>> >>>>>>>>> committed offset of the source topic of the application, but no
>> >>>>>>>>> corresponding result record in the output topic.
>> >>>>>>>>>
>> >>>>>>>>> Having "shutdown markers" would allow us, to first stop the
>> >>> upstream
>> >>>>>>>>> subtopology and write this marker into the intermediate topic and
>> >>> the
>> >>>>>>>>> downstream subtopology would only shut down itself after is sees
>> >>> the
>> >>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
>> >>>>>>>>> "in-flight" messages got stuck in intermediate topics.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch
>> Processing).
>> >>>>>>>>> There was a discussion about the proposed metadata topic, and we
>> >>>>> could
>> >>>>>>>>> avoid this metadata topic if we would have "control messages".
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Right now, we cannot insert an "application control message"
>> >>> because
>> >>>>>>>>> Kafka Streams does not own all topics it read/writes and thus
>> might
>> >>>>>>>>> break other consumer application (as described above) if we
>> inject
>> >>>>>>>>> random messages that are not understood by other apps.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Of course, one can work around "embedded control messaged" by
>> using
>> >>>>> an
>> >>>>>>>>> additional topic to propagate control messaged between
>> application
>> >>>>> (as
>> >>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
>> >>>>> there
>> >>>>>>>>> are major concerns about adding this metadata topic in the KIP
>> and
>> >>>>> this
>> >>>>>>>>> shows that other application that need a similar pattern might
>> >>> profit
>> >>>>>>>>> from topic embedded "control messages", too.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> One last important consideration: those "control messages" are
>> used
>> >>>>> for
>> >>>>>>>>> client to client communication and are not understood by the
>> >>> broker.
>> >>>>>>>>> Thus, those messages should not be enabled within the message
>> >>> format
>> >>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
>> >>>>> headers
>> >>>>>>>>> would be a nice way to implement them. Because KIP-82 did
>> consider
>> >>>>> key
>> >>>>>>>>> namespaces for metatdata keys, this extension should not be an
>> own
>> >>>>> KIP
>> >>>>>>>>> but should be included in KIP-82 to reserve a namespace for
>> >>> "control
>> >>>>>>>>> message" in the first place.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Sorry for the long email... Looking forward to your feedback.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> -Matthias
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>> >>>>>>>>>> Hi Jun
>> >>>>>>>>>>
>> >>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
>> >>>>> (HTTP/JMS/
>> >>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
>> >>> (though I
>> >>>>>>>>> believe it contains the end2end uuid embedded in this id), on
>> >>>>> receiving
>> >>>>>>> the
>> >>>>>>>>> message at the receiving JVM the apm code takes this out, and
>> >>>>> continues
>> >>>>>>> its
>> >>>>>>>>> tracing on the that new thread. Both JVM's (and other languages
>> the
>> >>>>> APM
>> >>>>>>>>> tool supports) send this data async back to the central
>> controllers
>> >>>>>>> where
>> >>>>>>>>> the stiching togeather occurs. For this they need some header
>> space
>> >>>>> for
>> >>>>>>>>> them to put this id.
>> >>>>>>>>>> 101) Yes indeed we have a business transaction Id in the
>> payload.
>> >>>>>>> Though
>> >>>>>>>>> this is a system level tracing, that we need to have marry up.
>> Also
>> >>>>> as
>> >>>>>>> per
>> >>>>>>>>> note on end2end encryption we'd be unable to prove the flow if
>> the
>> >>>>>>> payload
>> >>>>>>>>> is encrypted as we'd not have access to this at certain points of
>> >>> the
>> >>>>>>> flow
>> >>>>>>>>> through the infrastructure/platform.
>> >>>>>>>>>>
>> >>>>>>>>>> 103) As said we use this mechanism in IG very successfully, as
>> >>>>> stated
>> >>>>>>>>> per key we guarantee the transaction producing app to handle the
>> >>>>>>>>> transaction of a key at one DC unless at point of critical
>> failure
>> >>>>>>> where we
>> >>>>>>>>> have to flip processing to another. We care about key ordering.
>> >>>>>>>>>> I disagree on the offset comment for the partition solution
>> unless
>> >>>>> you
>> >>>>>>>>> do full ISR, or expensive full XA transactions even with
>> partitions
>> >>>>> you
>> >>>>>>>>> cannot fully guarantee offsets would match.
>> >>>>>>>>>> 105) Very much so, I need to have access at the platform level
>> to
>> >>>>> the
>> >>>>>>>>> other meta data all mentioned, without having to need to have
>> >>> access
>> >>>>> to
>> >>>>>>> the
>> >>>>>>>>> encryption keys of the payload.
>> >>>>>>>>>> 106)
>> >>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to
>> have
>> >>> a
>> >>>>>>>>> global producerId register which would be very hard to
>> >>>>> enforce/ensure is
>> >>>>>>>>> current and correct, just to understand the message origins of
>> its
>> >>>>>>>>> region/az/cluster for routing.
>> >>>>>>>>>> The client wrapper version, producerId can be the same, as
>> >>> obviously
>> >>>>>>> the
>> >>>>>>>>> producer could upgrade its wrapper, as such we need to know what
>> >>>>> wrapper
>> >>>>>>>>> version the message is created with.
>> >>>>>>>>>> Likewise the IP address, as stated we can have our producer
>> move,
>> >>>>> where
>> >>>>>>>>> its IP would change.
>> >>>>>>>>>> 107)
>> >>>>>>>>>> UUID is set on the message by interceptors before actual
>> producer
>> >>>>>>>>> transport send. This is for platform level message dedupe
>> >>> guarantee,
>> >>>>> the
>> >>>>>>>>> business payload should be agnostic to this. Please see
>> >>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
>> >>>>> ection.html
>> >>>>>>>>> note this is not touching business payloads.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> Hi, Michael,
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks for the reply. I find it very helpful.
>> >>>>>>>>>>
>> >>>>>>>>>> Data lineage:
>> >>>>>>>>>> 100. I'd like to understand the APM use case a bit more. It
>> >>>>> sounds
>> >>>>>>>>> like
>> >>>>>>>>>> that those APM plugins can generate a transaction id that
>> we
>> >>>>> could
>> >>>>>>>>>> potentially put in the header of every message. How would
>> you
>> >>>>>>>>> typically
>> >>>>>>>>>> make use of such transaction ids? Are there other metadata
>> >>>>>>>>> associated with
>> >>>>>>>>>> the transaction id and if so, how are they propagated
>> >>>>> downstream?
>> >>>>>>>>>> 101. For the finance use case, if the concept of
>> transaction
>> >>> is
>> >>>>>>>>> important,
>> >>>>>>>>>> wouldn't it be typically included in the message payload
>> >>>>> instead of
>> >>>>>>>>> as an
>> >>>>>>>>>> optional header field?
>> >>>>>>>>>>
>> >>>>>>>>>> 102. The data lineage that Altas and Navigator support
>> seems
>> >>> to
>> >>>>> be
>> >>>>>>>>> at the
>> >>>>>>>>>> dataset level, not per record level? So, not sure if per
>> >>> message
>> >>>>>>>>> headers
>> >>>>>>>>>> are relevant there.
>> >>>>>>>>>>
>> >>>>>>>>>> Mirroring:
>> >>>>>>>>>> 103. The benefit of using separate partitions is that it
>> >>>>>>> potentially
>> >>>>>>>>> makes
>> >>>>>>>>>> it easy to preserve offsets during mirroring. This will
>> make
>> >>> it
>> >>>>>>>>> easier for
>> >>>>>>>>>> consumer to switch clusters. Currently, the consumers can
>> >>> switch
>> >>>>>>>>> clusters
>> >>>>>>>>>> by using the timestampToOffset() api, but it has to deal
>> with
>> >>>>>>>>> duplicates.
>> >>>>>>>>>> Good point on the issue with log compact and I am not sure
>> how
>> >>>>> to
>> >>>>>>>>> address
>> >>>>>>>>>> this. However, even if we mirror into the existing
>> partitions,
>> >>>>> the
>> >>>>>>>>> ordering
>> >>>>>>>>>> for messages generated from different clusters seems
>> >>>>>>>>> non-deterministic
>> >>>>>>>>>> anyway. So, it seems that the consumers already have to
>> deal
>> >>>>> with
>> >>>>>>>>> that? If
>> >>>>>>>>>> a topic is compacted, does that mean which messages are
>> >>>>> preserved
>> >>>>>>> is
>> >>>>>>>>> also
>> >>>>>>>>>> non-deterministic across clusters?
>> >>>>>>>>>>
>> >>>>>>>>>> 104. Good point on partition key.
>> >>>>>>>>>>
>> >>>>>>>>>> End-to-end encryption:
>> >>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are
>> headers
>> >>>>>>>>> useful there?
>> >>>>>>>>>> Auditing:
>> >>>>>>>>>> 106. It seems other than the UUID, all other metadata are
>> per
>> >>>>>>>>> producer?
>> >>>>>>>>>> EOS:
>> >>>>>>>>>> 107. How are those UUIDs generated? I am not sure if they
>> can
>> >>> be
>> >>>>>>>>> generated
>> >>>>>>>>>> in the producer library. An application may send messages
>> >>>>> through a
>> >>>>>>>>> load
>> >>>>>>>>>> balancer and on retry, the same message could be routed to
>> a
>> >>>>>>>>> different
>> >>>>>>>>>> producer instance. So, it seems that the application has to
>> >>>>>>> generate
>> >>>>>>>>> the
>> >>>>>>>>>> UUIDs. In that case, shouldn't the application just put the
>> >>>>> UUID in
>> >>>>>>>>> the
>> >>>>>>>>>> payload?
>> >>>>>>>>>>
>> >>>>>>>>>> Thanks,
>> >>>>>>>>>>
>> >>>>>>>>>> Jun
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>> >>>>>>>>> Michael.Pearce@ig.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> > Hi Jun.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Per Transaction Tracing / Data Lineage.
>> >>>>>>>>>> >
>> >>>>>>>>>> > As Stated in the KIP this has the first use case of how
>> many
>> >>>>> APM
>> >>>>>>>>> tools now
>> >>>>>>>>>> > work.
>> >>>>>>>>>> > I would find it impossible for any one to argue this is
>> not
>> >>>>>>>>> important or a
>> >>>>>>>>>> > niche market as it has its own gartner report for this
>> >>> space.
>> >>>>>>> Such
>> >>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular
>> are
>> >>>>> but a
>> >>>>>>>>> few.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Likewise these APM tools can help very rapidly track down
>> >>>>> issues
>> >>>>>>>>> and
>> >>>>>>>>>> > automatically capture metrics, perform actions based on
>> >>>>>>> unexpected
>> >>>>>>>>> behavior
>> >>>>>>>>>> > to auto recover services.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Before mentioning looking at aggregated stats, in these
>> >>> cases
>> >>>>>>> where
>> >>>>>>>>>> > actually on critical flows we cannot afford to have
>> >>> aggregated
>> >>>>>>>>> rolled up
>> >>>>>>>>>> > stats only.
>> >>>>>>>>>> >
>> >>>>>>>>>> > With the APM tool we use its actually able to detect a
>> >>> single
>> >>>>>>>>> transaction
>> >>>>>>>>>> > failure and capture the thread traces in the JVM where it
>> >>>>> failed
>> >>>>>>>>> and
>> >>>>>>>>>> > everything for us, to the point it sends us alerts where
>> we
>> >>>>> have
>> >>>>>>>>> this
>> >>>>>>>>>> > giving the line number of the code that caused it, the
>> >>>>>>> transaction
>> >>>>>>>>> trace
>> >>>>>>>>>> > through all the services and endpoints (supported) upto
>> the
>> >>>>> point
>> >>>>>>>>> of
>> >>>>>>>>>> > failure, it can also capture the data in and out (so we
>> can
>> >>>>>>>>> replay).
>> >>>>>>>>>> > Because atm Kafka doesn't support us being able to stich
>> in
>> >>>>> these
>> >>>>>>>>> tracing
>> >>>>>>>>>> > transaction ids natively, we cannot get these benefits as
>> >>>>> such is
>> >>>>>>>>> limiting
>> >>>>>>>>>> > our ability support apps and monitor them to the same
>> >>>>> standards
>> >>>>>>> we
>> >>>>>>>>> come to
>> >>>>>>>>>> > expect when on a kafka flow.
>> >>>>>>>>>> >
>> >>>>>>>>>> > This actually ties in with Data Lineage, as the same
>> tracing
>> >>>>> can
>> >>>>>>>>> be used
>> >>>>>>>>>> > to back stich this. Essentially many times due to the
>> sums
>> >>> of
>> >>>>>>> money
>> >>>>>>>>>> > involved there are disputes, and typically as a financial
>> >>>>>>>>> institute the
>> >>>>>>>>>> > easiest and cleanest way to prove when disputes arise is
>> to
>> >>>>>>>>> present the
>> >>>>>>>>>> > actual flow and processes involved in a transaction.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Likewise as Hadoop matures its evident this case is
>> >>>>> important, as
>> >>>>>>>>> tools
>> >>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera
>> led)
>> >>>>> are
>> >>>>>>>>> evident
>> >>>>>>>>>> > also I believe the importance here is very much NOT just
>> a
>> >>>>>>>>> financial issue.
>> >>>>>>>>>> >
>> >>>>>>>>>> > From a MDM point of view any company wanting to care
>> about
>> >>>>> Data
>> >>>>>>>>> Quality
>> >>>>>>>>>> > and Data Governance - Data Lineage is a key piece in this
>> >>>>> puzzle.
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > RE Mirroring,
>> >>>>>>>>>> >
>> >>>>>>>>>> > As per the KIP in-fact this is exactly what we do re
>> cluster
>> >>>>> id,
>> >>>>>>>>> to mirror
>> >>>>>>>>>> > a network of clusters between AZ's / Regions. We know a
>> >>>>>>>>> transaction for a
>> >>>>>>>>>> > key will be done within a AZ/Region, as such we know the
>> >>>>> write
>> >>>>>>> to
>> >>>>>>>>> kafka
>> >>>>>>>>>> > would be ordered per key. But we need eventual view of
>> that
>> >>>>>>> across
>> >>>>>>>>> in our
>> >>>>>>>>>> > other regions/az's. When we have complete AZ or Region
>> >>>>> failure we
>> >>>>>>>>> know
>> >>>>>>>>>> > there will be a brief interruption whilst those
>> transactions
>> >>>>> are
>> >>>>>>>>> moved to
>> >>>>>>>>>> > another region but we expect after it to continue.
>> >>>>>>>>>> >
>> >>>>>>>>>> > As mentioned having separate Partions to do this starts
>> to
>> >>> get
>> >>>>>>>>>> > ugly/complicated for us:
>> >>>>>>>>>> > how would I do compaction where a key is in two
>> partitions?
>> >>>>>>>>>> > How do we balance consumers so where multiple partitions
>> >>> with
>> >>>>> the
>> >>>>>>>>> same key
>> >>>>>>>>>> > goto the same consumer
>> >>>>>>>>>> > What do you do if cluster 1 has 5 partitions but cluster
>> 20
>> >>>>> has
>> >>>>>>> 10
>> >>>>>>>>> because
>> >>>>>>>>>> > its larger kit in our more core DC's, as such key to
>> >>> partition
>> >>>>>>>>> mappings for
>> >>>>>>>>>> > consumers get even more complicated.
>> >>>>>>>>>> > What do you do if we add or remove a complete region
>> >>>>>>>>>> >
>> >>>>>>>>>> > Where as simple mirror will work we just need to ensure
>> we
>> >>>>> don't
>> >>>>>>>>> have a
>> >>>>>>>>>> > cycle which we can do with clusterId.
>> >>>>>>>>>> >
>> >>>>>>>>>> > We even have started to look at shortest path mirror
>> routing
>> >>>>>>> based
>> >>>>>>>>> on
>> >>>>>>>>>> > clusterId, if we also had the region and az info on the
>> >>>>>>> originating
>> >>>>>>>>>> > message, this we have not implemented but some ideas come
>> >>> from
>> >>>>>>>>> network
>> >>>>>>>>>> > routing, and also the dispatcher router in apache qpid.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Also we need to have data perimeters e.g. certain data
>> >>> cannot
>> >>>>>>> leave
>> >>>>>>>>>> > certain countries borders. We want this all automated so
>> >>> that
>> >>>>> at
>> >>>>>>>>> the
>> >>>>>>>>>> > platform level without having to touch or look at the
>> >>> business
>> >>>>>>>>> data inside
>> >>>>>>>>>> > we can have headers we can put tags into so that we can
>> >>> ensure
>> >>>>>>>>> this doesn't
>> >>>>>>>>>> > occur when we mirror. (actually links in to data lineage
>> /
>> >>>>>>> tracing
>> >>>>>>>>> as again
>> >>>>>>>>>> > we need to tag messages at a platform level) Examples
>> are we
>> >>>>> are
>> >>>>>>>>> not
>> >>>>>>>>>> > allowed Private customer details to leave Switzerland,
>> yet
>> >>> we
>> >>>>>>> need
>> >>>>>>>>> those
>> >>>>>>>>>> > systems integrated.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Lastly around mirroring we have a partionKey field, as
>> the
>> >>> key
>> >>>>>>>>> used for
>> >>>>>>>>>> > portioning logic != compaction key all the time but we
>> want
>> >>> to
>> >>>>>>>>> preserve it
>> >>>>>>>>>> > for when we mirror so that if source cluster partition
>> count
>> >>>>> !=
>> >>>>>>>>> destination
>> >>>>>>>>>> > cluster partition count we can honour the same
>> partitioning
>> >>>>>>> logic.
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > RE End 2 End encryption
>> >>>>>>>>>> >
>> >>>>>>>>>> > As I believe mentioned just before, the solution you
>> mention
>> >>>>> just
>> >>>>>>>>> doesn't
>> >>>>>>>>>> > cut the mustard these days with many regulators. An
>> >>> operations
>> >>>>>>>>> person with
>> >>>>>>>>>> > access to the box should not be able to have access to
>> the
>> >>>>> data.
>> >>>>>>>>> Many now
>> >>>>>>>>>> > actually impose quite literally the implementation
>> expected
>> >>>>> being
>> >>>>>>>>> end2end
>> >>>>>>>>>> > encryption for certain data (Singapore for us is one
>> that I
>> >>> am
>> >>>>>>>>> most aware
>> >>>>>>>>>> > of). In fact we're even now needing encrypt the data and
>> >>> store
>> >>>>>>> the
>> >>>>>>>>> keys in
>> >>>>>>>>>> > HSM modules.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Likewise the performance penalty on encrypting
>> decrypting as
>> >>>>> you
>> >>>>>>>>> produce
>> >>>>>>>>>> > over wire, then again encrypt decrypt as the data is
>> stored
>> >>> on
>> >>>>>>> the
>> >>>>>>>>> brokers
>> >>>>>>>>>> > disks and back again, then again encrypted and decrypted
>> >>> back
>> >>>>>>> over
>> >>>>>>>>> the wire
>> >>>>>>>>>> > each time for each consumer all adds up, ignoring this
>> >>>>> doubling
>> >>>>>>>>> with mirror
>> >>>>>>>>>> > makers etc. simply encrypting the value once on write by
>> the
>> >>>>>>>>> client and
>> >>>>>>>>>> > again decrypting on consume by the consumer is far more
>> >>>>>>>>> performant, but
>> >>>>>>>>>> > then the routing and platform meta data needs to be
>> separate
>> >>>>>>> (thus
>> >>>>>>>>> headers)
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > RE Auditing:
>> >>>>>>>>>> >
>> >>>>>>>>>> > Our Auditing needs are:
>> >>>>>>>>>> > Producer Id,
>> >>>>>>>>>> > Origin Cluster Id that message first produced into
>> >>>>>>>>>> > Origin AZ - agreed we can derive this if we have cluster
>> id,
>> >>>>> but
>> >>>>>>>>> it makes
>> >>>>>>>>>> > resolving this for audit reporting a lot easier.
>> >>>>>>>>>> > Origin Region - agreed we can derive this if we have
>> cluster
>> >>>>> id,
>> >>>>>>>>> but it
>> >>>>>>>>>> > makes resolving this for audit reporting a lot easier.
>> >>>>>>>>>> > Unique Message Identification (this is not the same as
>> >>>>>>> transaction
>> >>>>>>>>>> > tracing) - note offset and partition are not the same, as
>> >>>>> when we
>> >>>>>>>>> mirror or
>> >>>>>>>>>> > have for what ever system failure duplicate send,
>> >>>>>>>>>> > Custom Client wrapper version (where organizations have
>> to
>> >>>>> wrap
>> >>>>>>>>> the kafka
>> >>>>>>>>>> > client for added features) so we know what version of the
>> >>>>> wrapper
>> >>>>>>>>> is used
>> >>>>>>>>>> > Producer IP address (in case of clients being in our
>> vm/open
>> >>>>>>> stack
>> >>>>>>>>> infra
>> >>>>>>>>>> > where they can move around, producer id will stay the
>> same
>> >>> but
>> >>>>>>>>> this would
>> >>>>>>>>>> > change)
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > RE Once and only once delivery case
>> >>>>>>>>>> >
>> >>>>>>>>>> > Using the same Message UUID for auditing we can achieve
>> this
>> >>>>>>> quite
>> >>>>>>>>> simply.
>> >>>>>>>>>> >
>> >>>>>>>>>> > As per how some other brokers do this (cough qpid,
>> artemis)
>> >>>>>>>>> message uuid
>> >>>>>>>>>> > are used to dedupe where message is sent and produced but
>> >>> the
>> >>>>>>>>> client didn't
>> >>>>>>>>>> > receive the ack, and there for replays the send, by
>> having a
>> >>>>>>>>> unique message
>> >>>>>>>>>> > id per message, this can be filtered out, on consumers
>> where
>> >>>>>>>>> message
>> >>>>>>>>>> > delivery may occur twice for what ever reasons a message
>> >>> uuid
>> >>>>> can
>> >>>>>>>>> be used
>> >>>>>>>>>> > to remove duplicates being deliverd , like wise we can do
>> >>>>> this in
>> >>>>>>>>> the
>> >>>>>>>>>> > mirrormakers so if we detect a dupe message we can avoid
>> >>>>>>>>> replicating it.
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > Cheers
>> >>>>>>>>>> > Mike
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
>> wrote:
>> >>>>>>>>>> >
>> >>>>>>>>>> > Since this KIP affects message format, wire protocol,
>> >>>>> apis, I
>> >>>>>>>>> think
>> >>>>>>>>>> > it's
>> >>>>>>>>>> > worth spending a bit more time to nail down the
>> concrete
>> >>>>> use
>> >>>>>>>>> cases. It
>> >>>>>>>>>> > would be bad if we add this feature, but when start
>> >>>>>>>>> implementing it
>> >>>>>>>>>> > for say
>> >>>>>>>>>> > mirroring, we then realize that header is not the
>> best
>> >>>>>>>>> approach.
>> >>>>>>>>>> > Initially,
>> >>>>>>>>>> > I thought I was convinced of the use cases of headers
>> >>> and
>> >>>>> was
>> >>>>>>>>> trying to
>> >>>>>>>>>> > write down a few use cases to convince others. That's
>> >>>>> when I
>> >>>>>>>>> became
>> >>>>>>>>>> > less
>> >>>>>>>>>> > certain. For me to be convinced, I just want to see
>> two
>> >>>>>>> strong
>> >>>>>>>>> use
>> >>>>>>>>>> > cases
>> >>>>>>>>>> > (instead of 10 maybe use cases) in the third-party
>> >>> space.
>> >>>>> The
>> >>>>>>>>> reason is
>> >>>>>>>>>> > that when we discussed the use cases within a
>> company,
>> >>>>> often
>> >>>>>>>>> it ends
>> >>>>>>>>>> > with
>> >>>>>>>>>> > "we can't force everyone to use this standard since
>> we
>> >>> may
>> >>>>>>>>> have to
>> >>>>>>>>>> > integrate with third-party tools".
>> >>>>>>>>>> >
>> >>>>>>>>>> > At present, I am not sure why headers are useful for
>> >>>>> things
>> >>>>>>>>> like
>> >>>>>>>>>> > schemaId
>> >>>>>>>>>> > or encryption. In order to do anything useful to the
>> >>>>> value,
>> >>>>>>>>> one needs
>> >>>>>>>>>> > to
>> >>>>>>>>>> > know the schemaId or how data is encrypted, but
>> header
>> >>> is
>> >>>>>>>>> optional.
>> >>>>>>>>>> > But, I
>> >>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
>> >>> provides
>> >>>>>>> more
>> >>>>>>>>> details
>> >>>>>>>>>> > on
>> >>>>>>>>>> > the argument.
>> >>>>>>>>>> >
>> >>>>>>>>>> > I am not very sure header is the best approach for
>> >>>>> mirroring
>> >>>>>>>>> either. If
>> >>>>>>>>>> > someone has thought about this more, I'd be happy to
>> >>> hear.
>> >>>>>>>>>> >
>> >>>>>>>>>> > I can see the data lineage use case. I am just not
>> sure
>> >>>>> how
>> >>>>>>>>> widely
>> >>>>>>>>>> > applicable this is. If someone familiar with this
>> space
>> >>>>> can
>> >>>>>>>>> justify
>> >>>>>>>>>> > this is
>> >>>>>>>>>> > a significant use case, say in the finance industry,
>> >>> this
>> >>>>>>>>> would be a
>> >>>>>>>>>> > strong
>> >>>>>>>>>> > use case.
>> >>>>>>>>>> >
>> >>>>>>>>>> > I can see the auditing use case. I am just not sure
>> if a
>> >>>>>>> native
>> >>>>>>>>>> > producer id
>> >>>>>>>>>> > solves that problem. If there are additional metadata
>> >>>>> that's
>> >>>>>>>>> worth
>> >>>>>>>>>> > collecting but not covered by the producer id, that
>> >>> would
>> >>>>>>> make
>> >>>>>>>>> this a
>> >>>>>>>>>> > strong use case.
>> >>>>>>>>>> >
>> >>>>>>>>>> > Thanks,
>> >>>>>>>>>> >
>> >>>>>>>>>> > Jun
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>> >>>>>>>>> radai.rosenblatt@gmail.com>
>> >>>>>>>>>> > wrote:
>> >>>>>>>>>> >
>> >>>>>>>>>> > > this KIP is about enabling headers, nothing more
>> >>> nothing
>> >>>>>>>>> less - so
>> >>>>>>>>>> > no,
>> >>>>>>>>>> > > broker-side use of headers is not in the KIP scope.
>> >>>>>>>>>> > >
>> >>>>>>>>>> > > obviously though, once you have headers potential
>> use
>> >>>>> cases
>> >>>>>>>>> could
>> >>>>>>>>>> > include
>> >>>>>>>>>> > > broker-side header-aware interceptors (which would
>> be
>> >>>>> the
>> >>>>>>>>> topic of
>> >>>>>>>>>> > other
>> >>>>>>>>>> > > future KIPs).
>> >>>>>>>>>> > >
>> >>>>>>>>>> > > a trivially clear use case (to me) would be using
>> such
>> >>>>>>>>> broker-side
>> >>>>>>>>>> > > interceptors to enforce compliance with
>> organizational
>> >>>>>>>>> policies - it
>> >>>>>>>>>> > would
>> >>>>>>>>>> > > make our SREs lives much easier if instead of
>> >>>>> retroactively
>> >>>>>>>>>> > discovering
>> >>>>>>>>>> > > "rogue" topics/users those messages would have been
>> >>>>>>> rejected
>> >>>>>>>>>> > up-front.
>> >>>>>>>>>> > >
>> >>>>>>>>>> > > the kafka broker code is lacking any such
>> >>> extensibility
>> >>>>>>>>> support
>> >>>>>>>>>> > (beyond
>> >>>>>>>>>> > > maybe authorizer) which is why these use cases were
>> >>> left
>> >>>>>>> out
>> >>>>>>>>> of the
>> >>>>>>>>>> > "case
>> >>>>>>>>>> > > for headers" doc - broker extensibility is a
>> separate
>> >>>>>>>>> discussion.
>> >>>>>>>>>> > >
>> >>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
>> >>>>>>>>> gwen@confluent.io>
>> >>>>>>>>>> > wrote:
>> >>>>>>>>>> > >
>> >>>>>>>>>> > > > Woah, I wasn't aware this is something we'll do.
>> It
>> >>>>>>> wasn't
>> >>>>>>>>> in the
>> >>>>>>>>>> > KIP,
>> >>>>>>>>>> > > > right?
>> >>>>>>>>>> > > >
>> >>>>>>>>>> > > > I guess we could do it the same way ACLs
>> currently
>> >>>>> work.
>> >>>>>>>>>> > > > I had in mind something that will allow admins to
>> >>>>> apply
>> >>>>>>>>> rules to
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > new create/delete/config topic APIs. So Todd can
>> >>>>> decide
>> >>>>>>> to
>> >>>>>>>>> reject
>> >>>>>>>>>> > > > "create topic" requests that ask for more than 40
>> >>>>>>>>> partitions, or
>> >>>>>>>>>> > > > require exactly 3 replicas, or no more than 50GB
>> >>>>>>> partition
>> >>>>>>>>> size,
>> >>>>>>>>>> > etc.
>> >>>>>>>>>> > > >
>> >>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are planning
>> to
>> >>>>> apply
>> >>>>>>>>> more
>> >>>>>>>>>> > rules
>> >>>>>>>>>> > > > to requests (and I think we should), we may want
>> a
>> >>> bit
>> >>>>>>>>> more generic
>> >>>>>>>>>> > > > design around that.
>> >>>>>>>>>> > > >
>> >>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>> >>>>>>>>> radai.rosenblatt@gmail.com>
>> >>>>>>>>>> > > wrote:
>> >>>>>>>>>> > > > > "wouldn't you be in the business of making sure
>> >>>>>>> everyone
>> >>>>>>>>> uses
>> >>>>>>>>>> > them
>> >>>>>>>>>> > > > > properly?"
>> >>>>>>>>>> > > > >
>> >>>>>>>>>> > > > > thats where a broker-side plugin would come
>> handy
>> >>> -
>> >>>>> any
>> >>>>>>>>> incoming
>> >>>>>>>>>> > > message
>> >>>>>>>>>> > > > > that does not conform to org policy (read -
>> does
>> >>> not
>> >>>>>>>>> have the
>> >>>>>>>>>> > proper
>> >>>>>>>>>> > > > > headers) gets thrown out (with an error
>> returned
>> >>> to
>> >>>>>>> user)
>> >>>>>>>>>> > > > >
>> >>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
>> >>>>>>>>> tpalino@gmail.com>
>> >>>>>>>>>> > wrote:
>> >>>>>>>>>> > > > >
>> >>>>>>>>>> > > > >> Come on, I've done at least 2 talks on this
>> one
>> >>> :)
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >> Producing counts to a topic is part of it, but
>> >>>>> that's
>> >>>>>>>>> only
>> >>>>>>>>>> > part. So
>> >>>>>>>>>> > > you
>> >>>>>>>>>> > > > >> count you have 100 messages in topic A. When
>> you
>> >>>>>>> mirror
>> >>>>>>>>> topic A
>> >>>>>>>>>> > to
>> >>>>>>>>>> > > > another
>> >>>>>>>>>> > > > >> cluster, you have 99 messages. Where was your
>> >>>>> problem?
>> >>>>>>>>> Or
>> >>>>>>>>>> > worse, you
>> >>>>>>>>>> > > > have
>> >>>>>>>>>> > > > >> 100 messages, but one producer duplicated
>> >>> messages
>> >>>>> and
>> >>>>>>>>> another
>> >>>>>>>>>> > one
>> >>>>>>>>>> > > lost
>> >>>>>>>>>> > > > >> messages. You need details about where the
>> >>> message
>> >>>>>>> came
>> >>>>>>>>> from in
>> >>>>>>>>>> > order
>> >>>>>>>>>> > > to
>> >>>>>>>>>> > > > >> pinpoint problems when they happen. Source
>> >>> producer
>> >>>>>>>>> info, where
>> >>>>>>>>>> > it was
>> >>>>>>>>>> > > > >> produced into your infrastructure, and when it
>> >>> was
>> >>>>>>>>> produced.
>> >>>>>>>>>> > This
>> >>>>>>>>>> > > > requires
>> >>>>>>>>>> > > > >> you to add the information to the message.
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >> And yes, you still need to maintain your
>> clients.
>> >>>>> So
>> >>>>>>>>> maybe my
>> >>>>>>>>>> > original
>> >>>>>>>>>> > > > >> example was not the best. My thoughts on not
>> >>>>> wanting
>> >>>>>>> to
>> >>>>>>>>> be
>> >>>>>>>>>> > responsible
>> >>>>>>>>>> > > > for
>> >>>>>>>>>> > > > >> message formats stands, because that's very
>> much
>> >>>>>>>>> separate from
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > client.
>> >>>>>>>>>> > > > >> As you know, we have our own internal client
>> >>>>> library
>> >>>>>>>>> that can
>> >>>>>>>>>> > insert
>> >>>>>>>>>> > > the
>> >>>>>>>>>> > > > >> right headers, and right now inserts the right
>> >>>>> audit
>> >>>>>>>>>> > information into
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> message fields. If they exist, and assuming
>> the
>> >>>>>>> message
>> >>>>>>>>> is Avro
>> >>>>>>>>>> > > encoded.
>> >>>>>>>>>> > > > >> What if someone wants to use JSON instead for
>> a
>> >>>>> good
>> >>>>>>>>> reason?
>> >>>>>>>>>> > What if
>> >>>>>>>>>> > > > user X
>> >>>>>>>>>> > > > >> wants to encrypt messages, but user Y does
>> not?
>> >>>>>>>>> Maintaining the
>> >>>>>>>>>> > client
>> >>>>>>>>>> > > > >> library is still much easier than maintaining
>> the
>> >>>>>>>>> message
>> >>>>>>>>>> > formats.
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >> -Todd
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
>> >>>>>>>>> gwen@confluent.io
>> >>>>>>>>>> > >
>> >>>>>>>>>> > > wrote:
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >> > Based on your last sentence, consider me
>> >>>>> convinced
>> >>>>>>> :)
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >> > I get why headers are critical for Mirroring
>> >>> (you
>> >>>>>>>>> need tags to
>> >>>>>>>>>> > > prevent
>> >>>>>>>>>> > > > >> > loops and sometimes to route messages to the
>> >>>>> correct
>> >>>>>>>>>> > destination).
>> >>>>>>>>>> > > > >> > But why do you need headers to audit? We are
>> >>>>>>> auditing
>> >>>>>>>>> by
>> >>>>>>>>>> > producing
>> >>>>>>>>>> > > > >> > counts to a side topic (and I was under the
>> >>>>>>>>> impression you do
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > >> > same), so we never need to modify the
>> message.
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >> > Another thing - after we added headers,
>> >>> wouldn't
>> >>>>> you
>> >>>>>>>>> be in the
>> >>>>>>>>>> > > > >> > business of making sure everyone uses them
>> >>>>> properly?
>> >>>>>>>>> Making
>> >>>>>>>>>> > sure
>> >>>>>>>>>> > > > >> > everyone includes the right headers you
>> need,
>> >>> not
>> >>>>>>>>> using the
>> >>>>>>>>>> > header
>> >>>>>>>>>> > > > >> > names you intend to use, etc. I don't think
>> the
>> >>>>>>>>> "policing"
>> >>>>>>>>>> > business
>> >>>>>>>>>> > > > >> > will ever go away.
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino
>> <
>> >>>>>>>>>> > tpalino@gmail.com>
>> >>>>>>>>>> > > > wrote:
>> >>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
>> >>> with
>> >>>>> the
>> >>>>>>>>>> > workaround.
>> >>>>>>>>>> > > Avro
>> >>>>>>>>>> > > > >> means
>> >>>>>>>>>> > > > >> > > that I have to be concerned with the
>> format
>> >>> of
>> >>>>> the
>> >>>>>>>>> messages
>> >>>>>>>>>> > in
>> >>>>>>>>>> > > > order to
>> >>>>>>>>>> > > > >> > run
>> >>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring,
>> etc.).
>> >>>>> That
>> >>>>>>>>> means
>> >>>>>>>>>> > that I
>> >>>>>>>>>> > > have
>> >>>>>>>>>> > > > to
>> >>>>>>>>>> > > > >> > > handle the schemas, and I have to enforce
>> >>> rules
>> >>>>>>>>> about good
>> >>>>>>>>>> > > formats.
>> >>>>>>>>>> > > > >> This
>> >>>>>>>>>> > > > >> > is
>> >>>>>>>>>> > > > >> > > not something I want to be in the business
>> >>> of,
>> >>>>>>>>> because I
>> >>>>>>>>>> > should be
>> >>>>>>>>>> > > > able
>> >>>>>>>>>> > > > >> > to
>> >>>>>>>>>> > > > >> > > run a service infrastructure without
>> needing
>> >>>>> to be
>> >>>>>>>>> in the
>> >>>>>>>>>> > weeds of
>> >>>>>>>>>> > > > >> > dealing
>> >>>>>>>>>> > > > >> > > with customer data formats.
>> >>>>>>>>>> > > > >> > >
>> >>>>>>>>>> > > > >> > > Trust me, a sizable portion of my support
>> >>> time
>> >>>>> is
>> >>>>>>>>> spent
>> >>>>>>>>>> > dealing
>> >>>>>>>>>> > > with
>> >>>>>>>>>> > > > >> > schema
>> >>>>>>>>>> > > > >> > > issues. I really would like to get away
>> from
>> >>>>> that.
>> >>>>>>>>> Maybe
>> >>>>>>>>>> > I'd have
>> >>>>>>>>>> > > > more
>> >>>>>>>>>> > > > >> > time
>> >>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
>> >>>>>>>>>> > > > >> > >
>> >>>>>>>>>> > > > >> > > -Todd
>> >>>>>>>>>> > > > >> > >
>> >>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
>> Shapira <
>> >>>>>>>>>> > gwen@confluent.io>
>> >>>>>>>>>> > > > wrote:
>> >>>>>>>>>> > > > >> > >
>> >>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
>> >>>>> workarounds
>> >>>>>>>>> (Avro
>> >>>>>>>>>> > container
>> >>>>>>>>>> > > > >> > >> format), so I'm not too excited about the
>> >>>>> extra
>> >>>>>>>>> work
>> >>>>>>>>>> > required to
>> >>>>>>>>>> > > do
>> >>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't
>> mind it
>> >>>>> if
>> >>>>>>>>> you do
>> >>>>>>>>>> > it...
>> >>>>>>>>>> > > > >> > >> I think the Apache convention for "good
>> >>> idea,
>> >>>>> but
>> >>>>>>>>> not
>> >>>>>>>>>> > willing to
>> >>>>>>>>>> > > > put
>> >>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway,
>> that's
>> >>>>> what
>> >>>>>>> I
>> >>>>>>>>> was
>> >>>>>>>>>> > trying to
>> >>>>>>>>>> > > > >> > >> convey :)
>> >>>>>>>>>> > > > >> > >>
>> >>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
>> Palino
>> >>> <
>> >>>>>>>>>> > tpalino@gmail.com>
>> >>>>>>>>>> > > > >> wrote:
>> >>>>>>>>>> > > > >> > >> > Well I guess my question for you,
>> then, is
>> >>>>> what
>> >>>>>>>>> is
>> >>>>>>>>>> > holding you
>> >>>>>>>>>> > > > back
>> >>>>>>>>>> > > > >> > from
>> >>>>>>>>>> > > > >> > >> > full support for headers? What's the
>> bit
>> >>>>> that
>> >>>>>>>>> you're
>> >>>>>>>>>> > missing
>> >>>>>>>>>> > > that
>> >>>>>>>>>> > > > >> has
>> >>>>>>>>>> > > > >> > you
>> >>>>>>>>>> > > > >> > >> > under a full +1?
>> >>>>>>>>>> > > > >> > >> >
>> >>>>>>>>>> > > > >> > >> > -Todd
>> >>>>>>>>>> > > > >> > >> >
>> >>>>>>>>>> > > > >> > >> >
>> >>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
>> >>>>> Shapira <
>> >>>>>>>>>> > > gwen@confluent.io>
>> >>>>>>>>>> > > > >> > wrote:
>> >>>>>>>>>> > > > >> > >> >
>> >>>>>>>>>> > > > >> > >> >> I know why people who support headers
>> >>>>> support
>> >>>>>>>>> them, and
>> >>>>>>>>>> > I've
>> >>>>>>>>>> > > > seen
>> >>>>>>>>>> > > > >> > what
>> >>>>>>>>>> > > > >> > >> >> the discussion is like.
>> >>>>>>>>>> > > > >> > >> >>
>> >>>>>>>>>> > > > >> > >> >> This is why I'm asking people who are
>> >>>>> against
>> >>>>>>>>> headers
>> >>>>>>>>>> > > > (especially
>> >>>>>>>>>> > > > >> > >> >> committers) what will make them change
>> >>>>> their
>> >>>>>>>>> mind - so
>> >>>>>>>>>> > we can
>> >>>>>>>>>> > > > get
>> >>>>>>>>>> > > > >> > this
>> >>>>>>>>>> > > > >> > >> >> part over one way or another.
>> >>>>>>>>>> > > > >> > >> >>
>> >>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at
>> Radai,
>> >>>>> Jun
>> >>>>>>>>> or you
>> >>>>>>>>>> > (Todd)...
>> >>>>>>>>>> > > > I am
>> >>>>>>>>>> > > > >> > >> >> just looking for something concrete we
>> >>> can
>> >>>>> do
>> >>>>>>>>> to move
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > >> discussion
>> >>>>>>>>>> > > > >> > >> >> along to the yummy design details
>> (which
>> >>> is
>> >>>>>>> the
>> >>>>>>>>>> > argument I
>> >>>>>>>>>> > > > really
>> >>>>>>>>>> > > > >> am
>> >>>>>>>>>> > > > >> > >> >> looking forward to).
>> >>>>>>>>>> > > > >> > >> >>
>> >>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
>> >>>>> Palino <
>> >>>>>>>>>> > > tpalino@gmail.com>
>> >>>>>>>>>> > > > >> > wrote:
>> >>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even
>> though
>> >>>>> I'm
>> >>>>>>>>> not a
>> >>>>>>>>>> > > > committer)...
>> >>>>>>>>>> > > > >> > >> >> >
>> >>>>>>>>>> > > > >> > >> >> > I have always been a strong
>> supporter
>> >>> of
>> >>>>>>>>> introducing
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > concept
>> >>>>>>>>>> > > > >> > of an
>> >>>>>>>>>> > > > >> > >> >> > envelope to messages, which headers
>> >>>>>>>>> accomplishes. The
>> >>>>>>>>>> > > message
>> >>>>>>>>>> > > > key
>> >>>>>>>>>> > > > >> > is
>> >>>>>>>>>> > > > >> > >> >> > already an example of a piece of
>> >>> envelope
>> >>>>>>>>>> > information. By
>> >>>>>>>>>> > > > >> > providing a
>> >>>>>>>>>> > > > >> > >> >> means
>> >>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself, and
>> not
>> >>>>>>>>> relying on
>> >>>>>>>>>> > use-case
>> >>>>>>>>>> > > > >> > specific
>> >>>>>>>>>> > > > >> > >> >> > implementations, you make it much
>> >>> easier
>> >>>>> for
>> >>>>>>>>>> > components to
>> >>>>>>>>>> > > > >> > >> interoperate.
>> >>>>>>>>>> > > > >> > >> >> It
>> >>>>>>>>>> > > > >> > >> >> > simplifies development of all these
>> >>>>> things
>> >>>>>>>>> (message
>> >>>>>>>>>> > routing,
>> >>>>>>>>>> > > > >> > auditing,
>> >>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each one
>> does
>> >>>>> not
>> >>>>>>>>> have to
>> >>>>>>>>>> > reinvent
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> wheel.
>> >>>>>>>>>> > > > >> > >> >> >
>> >>>>>>>>>> > > > >> > >> >> > It also makes it much easier from a
>> >>>>> client
>> >>>>>>>>> point of
>> >>>>>>>>>> > view if
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> headers
>> >>>>>>>>>> > > > >> > >> >> are
>> >>>>>>>>>> > > > >> > >> >> > defined as part of the protocol
>> and/or
>> >>>>>>>>> message format
>> >>>>>>>>>> > in
>> >>>>>>>>>> > > > general
>> >>>>>>>>>> > > > >> > >> because
>> >>>>>>>>>> > > > >> > >> >> > you can easily produce and consume
>> >>>>> messages
>> >>>>>>>>> without
>> >>>>>>>>>> > having
>> >>>>>>>>>> > > to
>> >>>>>>>>>> > > > >> take
>> >>>>>>>>>> > > > >> > >> into
>> >>>>>>>>>> > > > >> > >> >> > account specific cases. For
>> example, I
>> >>>>> want
>> >>>>>>>>> to route
>> >>>>>>>>>> > > messages,
>> >>>>>>>>>> > > > >> but
>> >>>>>>>>>> > > > >> > >> >> client A
>> >>>>>>>>>> > > > >> > >> >> > doesn't support the way audit
>> >>> implemented
>> >>>>>>>>> headers, and
>> >>>>>>>>>> > > client
>> >>>>>>>>>> > > > B
>> >>>>>>>>>> > > > >> > >> doesn't
>> >>>>>>>>>> > > > >> > >> >> > support the way encryption or
>> routing
>> >>>>>>>>> implemented
>> >>>>>>>>>> > headers,
>> >>>>>>>>>> > > so
>> >>>>>>>>>> > > > now
>> >>>>>>>>>> > > > >> > my
>> >>>>>>>>>> > > > >> > >> >> > application has to create some
>> really
>> >>>>>>> fragile
>> >>>>>>>>> (my
>> >>>>>>>>>> > > autocorrect
>> >>>>>>>>>> > > > >> just
>> >>>>>>>>>> > > > >> > >> tried
>> >>>>>>>>>> > > > >> > >> >> to
>> >>>>>>>>>> > > > >> > >> >> > make that "tragic", which is
>> probably
>> >>>>>>>>> appropriate
>> >>>>>>>>>> > too) code
>> >>>>>>>>>> > > to
>> >>>>>>>>>> > > > >> > strip
>> >>>>>>>>>> > > > >> > >> >> > everything off, rather than just
>> >>>>> consuming
>> >>>>>>> the
>> >>>>>>>>>> > messages,
>> >>>>>>>>>> > > > picking
>> >>>>>>>>>> > > > >> > out
>> >>>>>>>>>> > > > >> > >> the
>> >>>>>>>>>> > > > >> > >> >> 1
>> >>>>>>>>>> > > > >> > >> >> > or 2 headers it's interested in, and
>> >>>>>>>>> performing its
>> >>>>>>>>>> > > function.
>> >>>>>>>>>> > > > >> > >> >> >
>> >>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has been
>> >>> going
>> >>>>> on
>> >>>>>>>>> for a
>> >>>>>>>>>> > long time,
>> >>>>>>>>>> > > > and
>> >>>>>>>>>> > > > >> > it's
>> >>>>>>>>>> > > > >> > >> >> > always "Oh, you came up with 2 use
>> >>> cases,
>> >>>>>>> and
>> >>>>>>>>> yeah,
>> >>>>>>>>>> > those
>> >>>>>>>>>> > > use
>> >>>>>>>>>> > > > >> cases
>> >>>>>>>>>> > > > >> > >> are
>> >>>>>>>>>> > > > >> > >> >> > real things that someone would want
>> to
>> >>>>> do.
>> >>>>>>>>> Here's an
>> >>>>>>>>>> > > alternate
>> >>>>>>>>>> > > > >> way
>> >>>>>>>>>> > > > >> > to
>> >>>>>>>>>> > > > >> > >> >> > implement them so let's not do
>> >>> headers."
>> >>>>> If
>> >>>>>>>>> we have a
>> >>>>>>>>>> > few
>> >>>>>>>>>> > > use
>> >>>>>>>>>> > > > >> cases
>> >>>>>>>>>> > > > >> > >> that
>> >>>>>>>>>> > > > >> > >> >> we
>> >>>>>>>>>> > > > >> > >> >> > actually came up with, you can be
>> sure
>> >>>>> that
>> >>>>>>>>> over the
>> >>>>>>>>>> > next
>> >>>>>>>>>> > > year
>> >>>>>>>>>> > > > >> > >> there's a
>> >>>>>>>>>> > > > >> > >> >> > dozen others that we didn't think of
>> >>> that
>> >>>>>>>>> someone
>> >>>>>>>>>> > would like
>> >>>>>>>>>> > > > to
>> >>>>>>>>>> > > > >> > do. I
>> >>>>>>>>>> > > > >> > >> >> > really think it's time to stop
>> >>> rehashing
>> >>>>>>> this
>> >>>>>>>>>> > discussion and
>> >>>>>>>>>> > > > >> > instead
>> >>>>>>>>>> > > > >> > >> >> focus
>> >>>>>>>>>> > > > >> > >> >> > on a workable standard that we can
>> >>> adopt.
>> >>>>>>>>>> > > > >> > >> >> >
>> >>>>>>>>>> > > > >> > >> >> > -Todd
>> >>>>>>>>>> > > > >> > >> >> >
>> >>>>>>>>>> > > > >> > >> >> >
>> >>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
>> >>>>> Palino
>> >>>>>>> <
>> >>>>>>>>>> > > > tpalino@gmail.com>
>> >>>>>>>>>> > > > >> > >> wrote:
>> >>>>>>>>>> > > > >> > >> >> >
>> >>>>>>>>>> > > > >> > >> >> >> C. per message encryption
>> >>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach is
>> that
>> >>>>> this
>> >>>>>>>>>> > significantly
>> >>>>>>>>>> > > > reduce
>> >>>>>>>>>> > > > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression,
>> which
>> >>>>>>> happens
>> >>>>>>>>> on a
>> >>>>>>>>>> > set of
>> >>>>>>>>>> > > > >> > serialized
>> >>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to
>> enable
>> >>>>> SSL
>> >>>>>>>>> for wire
>> >>>>>>>>>> > > > encryption
>> >>>>>>>>>> > > > >> and
>> >>>>>>>>>> > > > >> > >> rely
>> >>>>>>>>>> > > > >> > >> >> on
>> >>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS)
>> for at
>> >>>>> rest
>> >>>>>>>>>> > encryption.
>> >>>>>>>>>> > > > >> > >> >> >>
>> >>>>>>>>>> > > > >> > >> >> >>
>> >>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
>> >>> this
>> >>>>>>> does
>> >>>>>>>>> cover
>> >>>>>>>>>> > the case
>> >>>>>>>>>> > > > of
>> >>>>>>>>>> > > > >> > >> removing
>> >>>>>>>>>> > > > >> > >> >> a
>> >>>>>>>>>> > > > >> > >> >> >> drive from the system, it will not
>> >>>>> satisfy
>> >>>>>>>>> most
>> >>>>>>>>>> > compliance
>> >>>>>>>>>> > > > >> > >> requirements
>> >>>>>>>>>> > > > >> > >> >> for
>> >>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever has
>> >>>>> access to
>> >>>>>>>>> the
>> >>>>>>>>>> > broker
>> >>>>>>>>>> > > itself
>> >>>>>>>>>> > > > >> > still
>> >>>>>>>>>> > > > >> > >> has
>> >>>>>>>>>> > > > >> > >> >> >> access to the unencrypted data. For
>> >>>>>>>>> end-to-end
>> >>>>>>>>>> > encryption
>> >>>>>>>>>> > > you
>> >>>>>>>>>> > > > >> > need to
>> >>>>>>>>>> > > > >> > >> >> >> encrypt at the producer, before it
>> >>>>> enters
>> >>>>>>> the
>> >>>>>>>>>> > system, and
>> >>>>>>>>>> > > > >> decrypt
>> >>>>>>>>>> > > > >> > at
>> >>>>>>>>>> > > > >> > >> the
>> >>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the
>> system.
>> >>>>>>>>>> > > > >> > >> >> >>
>> >>>>>>>>>> > > > >> > >> >> >> -Todd
>> >>>>>>>>>> > > > >> > >> >> >>
>> >>>>>>>>>> > > > >> > >> >> >>
>> >>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM,
>> radai
>> >>> <
>> >>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
>> >>>>>>>>>> > > > >> > >
>> >>>>>>>>>> > > > >> > >> >> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> another big plus of headers in the
>> >>>>>>> protocol
>> >>>>>>>>> is that
>> >>>>>>>>>> > it
>> >>>>>>>>>> > > would
>> >>>>>>>>>> > > > >> > enable
>> >>>>>>>>>> > > > >> > >> >> rapid
>> >>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of core
>> >>>>> kafka
>> >>>>>>>>> and would
>> >>>>>>>>>> > reduce
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> number of
>> >>>>>>>>>> > > > >> > >> >> >>> future wire format changes
>> required.
>> >>>>>>>>>> > > > >> > >> >> >>>
>> >>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
>> >>>>>>> represents
>> >>>>>>>>> use
>> >>>>>>>>>> > cases that
>> >>>>>>>>>> > > > are
>> >>>>>>>>>> > > > >> > not
>> >>>>>>>>>> > > > >> > >> 100%
>> >>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and some of
>> >>> them
>> >>>>>>>>> require
>> >>>>>>>>>> > rather
>> >>>>>>>>>> > > > invasive
>> >>>>>>>>>> > > > >> > wire
>> >>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a good
>> >>> recent
>> >>>>>>>>> example of
>> >>>>>>>>>> > this is
>> >>>>>>>>>> > > > >> > kip-98.
>> >>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected
>> to
>> >>> be
>> >>>>> a
>> >>>>>>>>> very small
>> >>>>>>>>>> > > > fraction of
>> >>>>>>>>>> > > > >> > >> total
>> >>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes are
>> >>>>> invasive.
>> >>>>>>>>>> > > > >> > >> >> >>>
>> >>>>>>>>>> > > > >> > >> >> >>> every such wire format change
>> >>>>> translates
>> >>>>>>>>> into
>> >>>>>>>>>> > painful and
>> >>>>>>>>>> > > > slow
>> >>>>>>>>>> > > > >> > >> >> adoption of
>> >>>>>>>>>> > > > >> > >> >> >>> new versions.
>> >>>>>>>>>> > > > >> > >> >> >>>
>> >>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
>> >>>>> currently
>> >>>>>>> in
>> >>>>>>>>> KIPs
>> >>>>>>>>>> > could be
>> >>>>>>>>>> > > > "spun
>> >>>>>>>>>> > > > >> > out"
>> >>>>>>>>>> > > > >> > >> >> and
>> >>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
>> >>>>> transmitting
>> >>>>>>>>> data over
>> >>>>>>>>>> > > > headers.
>> >>>>>>>>>> > > > >> > this
>> >>>>>>>>>> > > > >> > >> >> would
>> >>>>>>>>>> > > > >> > >> >> >>> keep the core wire format
>> stable(r),
>> >>>>> core
>> >>>>>>>>> codebase
>> >>>>>>>>>> > > smaller,
>> >>>>>>>>>> > > > and
>> >>>>>>>>>> > > > >> > >> avoid
>> >>>>>>>>>> > > > >> > >> >> the
>> >>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
>> >>>>> required
>> >>>>>>>>> to prove
>> >>>>>>>>>> > a
>> >>>>>>>>>> > > > certain
>> >>>>>>>>>> > > > >> > >> feature
>> >>>>>>>>>> > > > >> > >> >> is
>> >>>>>>>>>> > > > >> > >> >> >>> useful enough for a wide-enough
>> >>>>> audience
>> >>>>>>> to
>> >>>>>>>>> warrant
>> >>>>>>>>>> > a wire
>> >>>>>>>>>> > > > >> format
>> >>>>>>>>>> > > > >> > >> >> change
>> >>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
>> >>>>>>>>>> > > > >> > >> >> >>>
>> >>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
>> >>>>> "mere"
>> >>>>>>>>> wire format
>> >>>>>>>>>> > > changes
>> >>>>>>>>>> > > > >> and
>> >>>>>>>>>> > > > >> > im
>> >>>>>>>>>> > > > >> > >> not
>> >>>>>>>>>> > > > >> > >> >> >>> saying it could have been
>> completely
>> >>>>> done
>> >>>>>>>>> with
>> >>>>>>>>>> > headers,
>> >>>>>>>>>> > > but
>> >>>>>>>>>> > > > >> > >> >> exactly-once
>> >>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
>> >>>>>>>>>> > > > >> > >> >> >>>
>> >>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM,
>> Gwen
>> >>>>>>>>> Shapira <
>> >>>>>>>>>> > > > >> gwen@confluent.io
>> >>>>>>>>>> > > > >> > >
>> >>>>>>>>>> > > > >> > >> >> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>>
>> >>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
>> >>>>> radai <
>> >>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
>> >>>>> organization,
>> >>>>>>>>> one could
>> >>>>>>>>>> > > always
>> >>>>>>>>>> > > > use
>> >>>>>>>>>> > > > >> > >> other
>> >>>>>>>>>> > > > >> > >> >> >>> > > approaches such as
>> company-wise
>> >>>>>>>>> containers"
>> >>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
>> >>>>>>> traditionally
>> >>>>>>>>> done
>> >>>>>>>>>> > but there
>> >>>>>>>>>> > > > are
>> >>>>>>>>>> > > > >> > now
>> >>>>>>>>>> > > > >> > >> >> cases
>> >>>>>>>>>> > > > >> > >> >> >>> > (read
>> >>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is not
>> >>>>>>> acceptable.
>> >>>>>>>>> this
>> >>>>>>>>>> > makes
>> >>>>>>>>>> > > > headers
>> >>>>>>>>>> > > > >> > >> useful
>> >>>>>>>>>> > > > >> > >> >> even
>> >>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for cases
>> >>> where
>> >>>>>>>>>> > > > one-container-fits-all
>> >>>>>>>>>> > > > >> > cannot
>> >>>>>>>>>> > > > >> > >> >> >>> apply.
>> >>>>>>>>>> > > > >> > >> >> >>> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use
>> cases
>> >>>>>>> listed,
>> >>>>>>>>> i dont
>> >>>>>>>>>> > want
>> >>>>>>>>>> > > > this to
>> >>>>>>>>>> > > > >> > >> devolve
>> >>>>>>>>>> > > > >> > >> >> >>> to a
>> >>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular use
>> >>> cases
>> >>>>> - i
>> >>>>>>>>> think its
>> >>>>>>>>>> > > enough
>> >>>>>>>>>> > > > >> that
>> >>>>>>>>>> > > > >> > >> some
>> >>>>>>>>>> > > > >> > >> >> of
>> >>>>>>>>>> > > > >> > >> >> >>> them
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > I think a main point of
>> contention
>> >>> is
>> >>>>>>>>> that: We
>> >>>>>>>>>> > > identified
>> >>>>>>>>>> > > > few
>> >>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are
>> useful,
>> >>>>> do
>> >>>>>>> we
>> >>>>>>>>> want
>> >>>>>>>>>> > Kafka to
>> >>>>>>>>>> > > > be a
>> >>>>>>>>>> > > > >> > >> system
>> >>>>>>>>>> > > > >> > >> >> >>> > that supports those use-cases?
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
>> >>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
>> >>>>> record-level
>> >>>>>>>>> lineage
>> >>>>>>>>>> > is
>> >>>>>>>>>> > > though
>> >>>>>>>>>> > > > >> > since
>> >>>>>>>>>> > > > >> > >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > overhead could
>> >>>>>>>>>> > > > >> > >> >> >>> > be significant."
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record
>> level
>> >>>>>>>>> lineage. I
>> >>>>>>>>>> > don't
>> >>>>>>>>>> > > think
>> >>>>>>>>>> > > > it
>> >>>>>>>>>> > > > >> > was
>> >>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think it
>> is
>> >>>>> safe
>> >>>>>>> to
>> >>>>>>>>> assume
>> >>>>>>>>>> > that
>> >>>>>>>>>> > > the
>> >>>>>>>>>> > > > NSA
>> >>>>>>>>>> > > > >> > >> needed
>> >>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also know
>> >>> that
>> >>>>>>>>> certain
>> >>>>>>>>>> > financial
>> >>>>>>>>>> > > > >> > institutes
>> >>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
>> >>> records
>> >>>>> at
>> >>>>>>> a
>> >>>>>>>>> record
>> >>>>>>>>>> > level
>> >>>>>>>>>> > > and
>> >>>>>>>>>> > > > >> > there
>> >>>>>>>>>> > > > >> > >> are
>> >>>>>>>>>> > > > >> > >> >> >>> > federal regulations that
>> absolutely
>> >>>>>>>>> require
>> >>>>>>>>>> > this. They
>> >>>>>>>>>> > > > also
>> >>>>>>>>>> > > > >> > need
>> >>>>>>>>>> > > > >> > >> to
>> >>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
>> >>>>> "touches"
>> >>>>>>> the
>> >>>>>>>>>> > messages and
>> >>>>>>>>>> > > > >> either
>> >>>>>>>>>> > > > >> > >> reads
>> >>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't have
>> >>>>>>> possibly
>> >>>>>>>>>> > modified the
>> >>>>>>>>>> > > > >> payload
>> >>>>>>>>>> > > > >> > >> >> itself.
>> >>>>>>>>>> > > > >> > >> >> >>> > They use record level
>> encryption to
>> >>>>> do
>> >>>>>>>>> that -
>> >>>>>>>>>> > apps can
>> >>>>>>>>>> > > > read
>> >>>>>>>>>> > > > >> and
>> >>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
>> >>> can't
>> >>>>>>>>> touch the
>> >>>>>>>>>> > payload.
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those are
>> >>> corner
>> >>>>>>>>> cases and
>> >>>>>>>>>> > not worth
>> >>>>>>>>>> > > > >> adding
>> >>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they
>> should
>> >>>>> use a
>> >>>>>>>>> different
>> >>>>>>>>>> > > pubsub
>> >>>>>>>>>> > > > >> > message
>> >>>>>>>>>> > > > >> > >> for
>> >>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the other
>> 1000
>> >>>>> that
>> >>>>>>>>> cater
>> >>>>>>>>>> > > > specifically
>> >>>>>>>>>> > > > >> to
>> >>>>>>>>>> > > > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > financial industry).
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a catch
>> 22:
>> >>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific
>> use-case,
>> >>>>>>>>> someone can
>> >>>>>>>>>> > always
>> >>>>>>>>>> > > say
>> >>>>>>>>>> > > > it
>> >>>>>>>>>> > > > >> > isn't
>> >>>>>>>>>> > > > >> > >> >> >>> > interesting enough for Kafka.
>> If we
>> >>>>>>>>> discuss more
>> >>>>>>>>>> > general
>> >>>>>>>>>> > > > >> > trends,
>> >>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we are not
>> >>> sure
>> >>>>>>> any
>> >>>>>>>>> of them
>> >>>>>>>>>> > really
>> >>>>>>>>>> > > > >> needs
>> >>>>>>>>>> > > > >> > >> >> headers
>> >>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just hand
>> >>>>> waving
>> >>>>>>>>> and not
>> >>>>>>>>>> > > > interesting.".
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > I think discussing use-cases in
>> >>>>>>> specifics
>> >>>>>>>>> is super
>> >>>>>>>>>> > > > important
>> >>>>>>>>>> > > > >> to
>> >>>>>>>>>> > > > >> > >> >> decide
>> >>>>>>>>>> > > > >> > >> >> >>> > implementation details for
>> headers
>> >>>>> (my
>> >>>>>>>>> use-cases
>> >>>>>>>>>> > lean
>> >>>>>>>>>> > > > toward
>> >>>>>>>>>> > > > >> > >> >> numerical
>> >>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and object
>> >>>>> values,
>> >>>>>>>>> others
>> >>>>>>>>>> > differ),
>> >>>>>>>>>> > > > but I
>> >>>>>>>>>> > > > >> > >> think
>> >>>>>>>>>> > > > >> > >> >> we
>> >>>>>>>>>> > > > >> > >> >> >>> > need to answer the general "Are
>> we
>> >>>>> going
>> >>>>>>>>> to have
>> >>>>>>>>>> > > headers"
>> >>>>>>>>>> > > > >> > question
>> >>>>>>>>>> > > > >> > >> >> >>> > first.
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the other
>> >>>>>>>>> committers in the
>> >>>>>>>>>> > > > discussion:
>> >>>>>>>>>> > > > >> > >> >> >>> > What would it take to convince
>> you
>> >>>>> that
>> >>>>>>>>> headers
>> >>>>>>>>>> > in Kafka
>> >>>>>>>>>> > > > are
>> >>>>>>>>>> > > > >> a
>> >>>>>>>>>> > > > >> > >> good
>> >>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can move
>> >>> ahead
>> >>>>>>> and
>> >>>>>>>>> try to
>> >>>>>>>>>> > agree
>> >>>>>>>>>> > > on
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> >> details?
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving the
>> goal
>> >>>>>>> posts
>> >>>>>>>>> and
>> >>>>>>>>>> > this is
>> >>>>>>>>>> > > > truly
>> >>>>>>>>>> > > > >> > >> >> exhausting.
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly support
>> >>>>> adding
>> >>>>>>>>> headers
>> >>>>>>>>>> > to Kafka
>> >>>>>>>>>> > > > >> > (+0.5?).
>> >>>>>>>>>> > > > >> > >> >> >>> > The community can continue to
>> find
>> >>>>>>>>> workarounds to
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > issue
>> >>>>>>>>>> > > > >> and
>> >>>>>>>>>> > > > >> > >> there
>> >>>>>>>>>> > > > >> > >> >> >>> > are some benefits to keeping the
>> >>>>> message
>> >>>>>>>>> format
>> >>>>>>>>>> > and
>> >>>>>>>>>> > > > clients
>> >>>>>>>>>> > > > >> > >> simpler.
>> >>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of
>> headers
>> >>>>> to
>> >>>>>>>>> many
>> >>>>>>>>>> > use-cases
>> >>>>>>>>>> > > and
>> >>>>>>>>>> > > > if
>> >>>>>>>>>> > > > >> we
>> >>>>>>>>>> > > > >> > >> can
>> >>>>>>>>>> > > > >> > >> >> >>> > find a good and generally useful
>> >>> way
>> >>>>> to
>> >>>>>>>>> add it to
>> >>>>>>>>>> > Kafka,
>> >>>>>>>>>> > > > it
>> >>>>>>>>>> > > > >> > will
>> >>>>>>>>>> > > > >> > >> make
>> >>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
>> >>> worthy
>> >>>>>>>>> goal in my
>> >>>>>>>>>> > eyes.
>> >>>>>>>>>> > > > >> > >> >> >>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
>> >>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use
>> cases
>> >>>>> for
>> >>>>>>>>> polyglot
>> >>>>>>>>>> > topics.
>> >>>>>>>>>> > > > >> > >> especially if
>> >>>>>>>>>> > > > >> > >> >> >>> kafka
>> >>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
>> >>> something
>> >>>>>>> else.
>> >>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would make
>> it
>> >>>>>>> harder
>> >>>>>>>>> to write
>> >>>>>>>>>> > > > portable
>> >>>>>>>>>> > > > >> > >> consumer
>> >>>>>>>>>> > > > >> > >> >> >>> code.
>> >>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would mess
>> >>> with
>> >>>>>>>>> locality of
>> >>>>>>>>>> > > > consumption
>> >>>>>>>>>> > > > >> > >> >> >>> guarantees.
>> >>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
>> >>>>>>>>> lineage/metadata is
>> >>>>>>>>>> > > > >> > >> billing/chargeback.
>> >>>>>>>>>> > > > >> > >> >> for
>> >>>>>>>>>> > > > >> > >> >> >>> > that
>> >>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough to
>> >>> simply
>> >>>>>>>>> record the
>> >>>>>>>>>> > point
>> >>>>>>>>>> > > of
>> >>>>>>>>>> > > > >> > origin,
>> >>>>>>>>>> > > > >> > >> but
>> >>>>>>>>>> > > > >> > >> >> >>> every
>> >>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think mirror
>> >>>>> maker)
>> >>>>>>>>> must also
>> >>>>>>>>>> > add a
>> >>>>>>>>>> > > > >> record
>> >>>>>>>>>> > > > >> > to
>> >>>>>>>>>> > > > >> > >> >> form a
>> >>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
>> >>>>>>>>>> > > > >> > >> >> >>> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing on
>> top
>> >>> of
>> >>>>>>>>> kafka - i
>> >>>>>>>>>> > know
>> >>>>>>>>>> > > samza
>> >>>>>>>>>> > > > >> has a
>> >>>>>>>>>> > > > >> > >> >> metadata
>> >>>>>>>>>> > > > >> > >> >> >>> > map
>> >>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
>> >>>>> addition to
>> >>>>>>>>> user
>> >>>>>>>>>> > values.
>> >>>>>>>>>> > > > headers
>> >>>>>>>>>> > > > >> > are
>> >>>>>>>>>> > > > >> > >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > perfect
>> >>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
>> >>>>>>>>>> > > > >> > >> >> >>> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50
>> PM,
>> >>>>> Jun
>> >>>>>>>>> Rao <
>> >>>>>>>>>> > > > jun@confluent.io
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >> > >> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the first
>> two
>> >>>>>>>>> questions, it
>> >>>>>>>>>> > would
>> >>>>>>>>>> > > be
>> >>>>>>>>>> > > > >> > helpful
>> >>>>>>>>>> > > > >> > >> >> if we
>> >>>>>>>>>> > > > >> > >> >> >>> > could
>> >>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use
>> cases
>> >>>>> for
>> >>>>>>>>> headers
>> >>>>>>>>>> > in the
>> >>>>>>>>>> > > > space
>> >>>>>>>>>> > > > >> > for
>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
>> >>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases
>> within an
>> >>>>>>>>> organization,
>> >>>>>>>>>> > one
>> >>>>>>>>>> > > > could
>> >>>>>>>>>> > > > >> > always
>> >>>>>>>>>> > > > >> > >> >> use
>> >>>>>>>>>> > > > >> > >> >> >>> > other
>> >>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as
>> company-wise
>> >>>>>>>>> containers to
>> >>>>>>>>>> > get
>> >>>>>>>>>> > > > around
>> >>>>>>>>>> > > > >> w/o
>> >>>>>>>>>> > > > >> > >> >> >>> headers. I
>> >>>>>>>>>> > > > >> > >> >> >>> > >> went through the use cases in
>> >>> the
>> >>>>> KIP
>> >>>>>>>>> and in
>> >>>>>>>>>> > Radai's
>> >>>>>>>>>> > > > wiki
>> >>>>>>>>>> > > > >> (
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> https://cwiki.apache.org/confl
>> >>>>>>>>>> > uence/display/KAFKA/A+
>> >>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> ).
>> >>>>>>>>>> > > > >> > >> >> >>> > >> The following are the ones
>> that
>> >>>>> that
>> >>>>>>> I
>> >>>>>>>>>> > understand and
>> >>>>>>>>>> > > > >> could
>> >>>>>>>>>> > > > >> > be
>> >>>>>>>>>> > > > >> > >> in
>> >>>>>>>>>> > > > >> > >> >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case
>> category.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
>> >>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
>> >>>>>>> content-type
>> >>>>>>>>> should
>> >>>>>>>>>> > be set
>> >>>>>>>>>> > > at
>> >>>>>>>>>> > > > >> the
>> >>>>>>>>>> > > > >> > >> topic
>> >>>>>>>>>> > > > >> > >> >> >>> level.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages
>> with
>> >>>>>>>>> different
>> >>>>>>>>>> > content
>> >>>>>>>>>> > > > types
>> >>>>>>>>>> > > > >> > >> should be
>> >>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
>> >>> useless
>> >>>>>>>>> without
>> >>>>>>>>>> > schema id,
>> >>>>>>>>>> > > it
>> >>>>>>>>>> > > > >> > seems
>> >>>>>>>>>> > > > >> > >> that
>> >>>>>>>>>> > > > >> > >> >> >>> > storing
>> >>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together with
>> >>>>>>> serialized
>> >>>>>>>>> bytes
>> >>>>>>>>>> > in the
>> >>>>>>>>>> > > > value
>> >>>>>>>>>> > > > >> is
>> >>>>>>>>>> > > > >> > >> >> better?
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
>> >>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this
>> approach is
>> >>>>> that
>> >>>>>>>>> this
>> >>>>>>>>>> > > > significantly
>> >>>>>>>>>> > > > >> > reduce
>> >>>>>>>>>> > > > >> > >> >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
>> >>>>> which
>> >>>>>>>>> happens on
>> >>>>>>>>>> > a set
>> >>>>>>>>>> > > of
>> >>>>>>>>>> > > > >> > >> serialized
>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative is
>> to
>> >>>>> enable
>> >>>>>>>>> SSL for
>> >>>>>>>>>> > wire
>> >>>>>>>>>> > > > >> > encryption
>> >>>>>>>>>> > > > >> > >> and
>> >>>>>>>>>> > > > >> > >> >> >>> rely
>> >>>>>>>>>> > > > >> > >> >> >>> > on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g.
>> LUKS)
>> >>>>> for at
>> >>>>>>>>> rest
>> >>>>>>>>>> > > encryption.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
>> >>> across
>> >>>>>>>>> Kafka
>> >>>>>>>>>> > clusters
>> >>>>>>>>>> > > > >> > >> >> >>> > >> This is actually interesting.
>> >>>>> Today,
>> >>>>>>>>> to avoid
>> >>>>>>>>>> > > > introducing
>> >>>>>>>>>> > > > >> > >> cycles
>> >>>>>>>>>> > > > >> > >> >> when
>> >>>>>>>>>> > > > >> > >> >> >>> > doing
>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data
>> centers,
>> >>> one
>> >>>>>>>>> would
>> >>>>>>>>>> > either have
>> >>>>>>>>>> > > to
>> >>>>>>>>>> > > > >> set
>> >>>>>>>>>> > > > >> > up
>> >>>>>>>>>> > > > >> > >> two
>> >>>>>>>>>> > > > >> > >> >> >>> Kafka
>> >>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
>> >>>>> aggregate)
>> >>>>>>>>> per data
>> >>>>>>>>>> > center
>> >>>>>>>>>> > > or
>> >>>>>>>>>> > > > >> > rename
>> >>>>>>>>>> > > > >> > >> >> topics.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
>> headers,
>> >>>>> the
>> >>>>>>>>> producer
>> >>>>>>>>>> > could
>> >>>>>>>>>> > > tag
>> >>>>>>>>>> > > > >> each
>> >>>>>>>>>> > > > >> > >> >> message
>> >>>>>>>>>> > > > >> > >> >> >>> with
>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID in
>> the
>> >>>>>>> header.
>> >>>>>>>>>> > MirrorMaker
>> >>>>>>>>>> > > > could
>> >>>>>>>>>> > > > >> > then
>> >>>>>>>>>> > > > >> > >> >> avoid
>> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a
>> cluster
>> >>> if
>> >>>>>>>>> they are
>> >>>>>>>>>> > tagged
>> >>>>>>>>>> > > with
>> >>>>>>>>>> > > > >> the
>> >>>>>>>>>> > > > >> > >> same
>> >>>>>>>>>> > > > >> > >> >> >>> cluster
>> >>>>>>>>>> > > > >> > >> >> >>> > >> id.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative
>> approach
>> >>>>> is
>> >>>>>>> to
>> >>>>>>>>>> > introduce sth
>> >>>>>>>>>> > > > like
>> >>>>>>>>>> > > > >> > >> >> >>> hierarchical
>> >>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages from
>> >>>>>>> different
>> >>>>>>>>>> > clusters in
>> >>>>>>>>>> > > > >> > different
>> >>>>>>>>>> > > > >> > >> >> >>> partitions
>> >>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic. This
>> >>>>> approach
>> >>>>>>>>> avoids
>> >>>>>>>>>> > filtering
>> >>>>>>>>>> > > > out
>> >>>>>>>>>> > > > >> > >> unneeded
>> >>>>>>>>>> > > > >> > >> >> >>> data
>> >>>>>>>>>> > > > >> > >> >> >>> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving
>> easier
>> >>> to
>> >>>>>>>>> support. It
>> >>>>>>>>>> > may
>> >>>>>>>>>> > > make
>> >>>>>>>>>> > > > >> > >> compaction
>> >>>>>>>>>> > > > >> > >> >> >>> > trickier
>> >>>>>>>>>> > > > >> > >> >> >>> > >> though since the same key may
>> >>>>> show up
>> >>>>>>>>> in
>> >>>>>>>>>> > different
>> >>>>>>>>>> > > > >> > partitions.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
>> >>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source
>> connector
>> >>>>> could
>> >>>>>>>>> store in
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > message
>> >>>>>>>>>> > > > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> metadata
>> >>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
>> >>> record.
>> >>>>>>>>> Similarly,
>> >>>>>>>>>> > if a
>> >>>>>>>>>> > > > stream
>> >>>>>>>>>> > > > >> job
>> >>>>>>>>>> > > > >> > >> >> >>> transforms
>> >>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to
>> topic
>> >>> B,
>> >>>>> the
>> >>>>>>>>> library
>> >>>>>>>>>> > could
>> >>>>>>>>>> > > > >> include
>> >>>>>>>>>> > > > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> source
>> >>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each of the
>> >>>>>>>>> transformed
>> >>>>>>>>>> > message in
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> header.
>> >>>>>>>>>> > > > >> > >> >> Not
>> >>>>>>>>>> > > > >> > >> >> >>> > sure
>> >>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful
>> record-level
>> >>>>>>> lineage
>> >>>>>>>>> is
>> >>>>>>>>>> > though
>> >>>>>>>>>> > > since
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> >> overhead
>> >>>>>>>>>> > > > >> > >> >> >>> > could
>> >>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>> >>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
>> >>>>>>>>> clientId/host/user in
>> >>>>>>>>>> > the
>> >>>>>>>>>> > > > header
>> >>>>>>>>>> > > > >> in
>> >>>>>>>>>> > > > >> > >> each
>> >>>>>>>>>> > > > >> > >> >> >>> > message
>> >>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These metadata
>> are
>> >>>>>>>>> really at the
>> >>>>>>>>>> > > producer
>> >>>>>>>>>> > > > >> > level
>> >>>>>>>>>> > > > >> > >> >> though.
>> >>>>>>>>>> > > > >> > >> >> >>> > So, a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to only
>> >>>>>>> include a
>> >>>>>>>>>> > "producerId"
>> >>>>>>>>>> > > > per
>> >>>>>>>>>> > > > >> > >> message
>> >>>>>>>>>> > > > >> > >> >> and
>> >>>>>>>>>> > > > >> > >> >> >>> > send
>> >>>>>>>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
>> >>> mapping
>> >>>>>>>>>> > independently.
>> >>>>>>>>>> > > > KIP-98
>> >>>>>>>>>> > > > >> is
>> >>>>>>>>>> > > > >> > >> >> actually
>> >>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such a
>> >>>>> producerId
>> >>>>>>>>> natively
>> >>>>>>>>>> > in the
>> >>>>>>>>>> > > > >> > message.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that
>> I
>> >>> am
>> >>>>>>> fully
>> >>>>>>>>>> > convinced of
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > strong
>> >>>>>>>>>> > > > >> > >> >> >>> > third-party
>> >>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
>> >>> Perhaps
>> >>>>> we
>> >>>>>>>>> could
>> >>>>>>>>>> > discuss a
>> >>>>>>>>>> > > > bit
>> >>>>>>>>>> > > > >> > more
>> >>>>>>>>>> > > > >> > >> to
>> >>>>>>>>>> > > > >> > >> >> make
>> >>>>>>>>>> > > > >> > >> >> >>> > one
>> >>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing use
>> >>>>> cases.
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal question
>> is
>> >>>>>>>>> whether header
>> >>>>>>>>>> > should
>> >>>>>>>>>> > > > be
>> >>>>>>>>>> > > > >> > >> exposed
>> >>>>>>>>>> > > > >> > >> >> in
>> >>>>>>>>>> > > > >> > >> >> >>> > stream
>> >>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
>> >>>>> stream,
>> >>>>>>>>> Samza,
>> >>>>>>>>>> > and
>> >>>>>>>>>> > > Spark
>> >>>>>>>>>> > > > >> > >> streaming.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems just
>> >>> deal
>> >>>>>>> with
>> >>>>>>>>>> > key/value
>> >>>>>>>>>> > > > pairs.
>> >>>>>>>>>> > > > >> > >> Should we
>> >>>>>>>>>> > > > >> > >> >> >>> > expose a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there too
>> or
>> >>>>>>>>> somehow map
>> >>>>>>>>>> > header to
>> >>>>>>>>>> > > > key
>> >>>>>>>>>> > > > >> or
>> >>>>>>>>>> > > > >> > >> >> value?
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Jun
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35
>> AM,
>> >>>>>>>>> Michael
>> >>>>>>>>>> > Pearce <
>> >>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a
>> period
>> >>>>> of a
>> >>>>>>>>> week,
>> >>>>>>>>>> > that there
>> >>>>>>>>>> > > > is
>> >>>>>>>>>> > > > >> no
>> >>>>>>>>>> > > > >> > >> >> concerns
>> >>>>>>>>>> > > > >> > >> >> >>> now
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and
>> now
>> >>> we
>> >>>>>>> have
>> >>>>>>>>>> > agreement that
>> >>>>>>>>>> > > > >> > headers
>> >>>>>>>>>> > > > >> > >> are
>> >>>>>>>>>> > > > >> > >> >> >>> useful
>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
>> >>> put
>> >>>>> to
>> >>>>>>> a
>> >>>>>>>>> KIP
>> >>>>>>>>>> > vote, this
>> >>>>>>>>>> > > > >> > wouldn't
>> >>>>>>>>>> > > > >> > >> be
>> >>>>>>>>>> > > > >> > >> >> a
>> >>>>>>>>>> > > > >> > >> >> >>> > reason
>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > @
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of
>> getting
>> >>>>> this
>> >>>>>>>>> KIP
>> >>>>>>>>>> > moving past
>> >>>>>>>>>> > > > >> this,
>> >>>>>>>>>> > > > >> > we
>> >>>>>>>>>> > > > >> > >> can
>> >>>>>>>>>> > > > >> > >> >> >>> state
>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
>> >>> that
>> >>>>>>> can
>> >>>>>>>>> will be
>> >>>>>>>>>> > > > naturally
>> >>>>>>>>>> > > > >> > >> >> interpreted
>> >>>>>>>>>> > > > >> > >> >> >>> as
>> >>>>>>>>>> > > > >> > >> >> >>> > an
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is
>> later
>> >>>>>>>>> wanted you can
>> >>>>>>>>>> > > easily
>> >>>>>>>>>> > > > >> split
>> >>>>>>>>>> > > > >> > >> this
>> >>>>>>>>>> > > > >> > >> >> >>> into
>> >>>>>>>>>> > > > >> > >> >> >>> > two
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the
>> wire
>> >>>>>>> protocol
>> >>>>>>>>>> > > implementation
>> >>>>>>>>>> > > > >> this
>> >>>>>>>>>> > > > >> > >> makes
>> >>>>>>>>>> > > > >> > >> >> no
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don't
>> believe. Is
>> >>>>> this
>> >>>>>>>>>> > reasonable to
>> >>>>>>>>>> > > > all?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
>> therefor
>> >>>>> happy
>> >>>>>>>>> we keep
>> >>>>>>>>>> > with 32
>> >>>>>>>>>> > > > >> bits.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
>> >>>>>>>>>> > ignacio.solis@gmail.com on
>> >>>>>>>>>> > > > >> behalf
>> >>>>>>>>>> > > > >> > of
>> >>>>>>>>>> > > > >> > >> >> >>> Ignacio
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
>> >>> ignacio.solis@gmail.com
>> >>>>> on
>> >>>>>>>>> behalf of
>> >>>>>>>>>> > > > >> > isolis@igso.net
>> >>>>>>>>>> > > > >> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value
>> as
>> >>>>> byte[]
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
>> >>> maybe
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
>> >>>>> system
>> >>>>>>>>> should
>> >>>>>>>>>> > take a
>> >>>>>>>>>> > > > single
>> >>>>>>>>>> > > > >> > >> int. I
>> >>>>>>>>>> > > > >> > >> >> >>> think
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you
>> want
>> >>> to
>> >>>>>>>>> interpret
>> >>>>>>>>>> > this as
>> >>>>>>>>>> > > to
>> >>>>>>>>>> > > > >> 16bit
>> >>>>>>>>>> > > > >> > >> >> numbers
>> >>>>>>>>>> > > > >> > >> >> >>> in
>> >>>>>>>>>> > > > >> > >> >> >>> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > above go right ahead.
>> If
>> >>>>>>>>> somebody wants
>> >>>>>>>>>> > to
>> >>>>>>>>>> > > argue
>> >>>>>>>>>> > > > >> for
>> >>>>>>>>>> > > > >> > 16
>> >>>>>>>>>> > > > >> > >> >> bits
>> >>>>>>>>>> > > > >> > >> >> >>> or
>> >>>>>>>>>> > > > >> > >> >> >>> > 64
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I
>> would
>> >>>>>>> listen.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key space
>> >>> into
>> >>>>>>>>> sub_key_1 and
>> >>>>>>>>>> > > > sub_key_2
>> >>>>>>>>>> > > > >> > >> makes no
>> >>>>>>>>>> > > > >> > >> >> >>> > sense to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > me at
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we
>> going
>> >>> to
>> >>>>>>>>> start
>> >>>>>>>>>> > providing
>> >>>>>>>>>> > > > APIs to
>> >>>>>>>>>> > > > >> > get
>> >>>>>>>>>> > > > >> > >> all
>> >>>>>>>>>> > > > >> > >> >> >>> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
>> >>>>>>>>> sub_key_2s? If
>> >>>>>>>>>> > there is
>> >>>>>>>>>> > > > no
>> >>>>>>>>>> > > > >> > >> >> >>> distinguishing
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > functions
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to
>> each
>> >>> one
>> >>>>>>>>> then they
>> >>>>>>>>>> > should
>> >>>>>>>>>> > > be
>> >>>>>>>>>> > > > a
>> >>>>>>>>>> > > > >> > single
>> >>>>>>>>>> > > > >> > >> >> >>> value.
>> >>>>>>>>>> > > > >> > >> >> >>> > At
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're doing
>> is
>> >>>>>>>>> equality.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer
>> wants
>> >>> to
>> >>>>>>>>> interpret
>> >>>>>>>>>> > this as
>> >>>>>>>>>> > > 2,
>> >>>>>>>>>> > > > 3
>> >>>>>>>>>> > > > >> or
>> >>>>>>>>>> > > > >> > >> more
>> >>>>>>>>>> > > > >> > >> >> >>> values
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > different question. I
>> >>>>>>>>> personally think
>> >>>>>>>>>> > it's
>> >>>>>>>>>> > > all
>> >>>>>>>>>> > > > one
>> >>>>>>>>>> > > > >> > >> >> keyspace
>> >>>>>>>>>> > > > >> > >> >> >>> > that is
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned using
>> >>> some
>> >>>>>>>>> structure,
>> >>>>>>>>>> > but if
>> >>>>>>>>>> > > you
>> >>>>>>>>>> > > > >> > want to
>> >>>>>>>>>> > > > >> > >> >> >>> > sub-assign
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > parts
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
>> >>> applies
>> >>>>> to
>> >>>>>>>>> strings.
>> >>>>>>>>>> > If
>> >>>>>>>>>> > > > somebody
>> >>>>>>>>>> > > > >> > >> argued
>> >>>>>>>>>> > > > >> > >> >> for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would we be arguing to
>> >>>>> divide
>> >>>>>>> the
>> >>>>>>>>>> > strings with
>> >>>>>>>>>> > > > dots
>> >>>>>>>>>> > > > >> > ('.')
>> >>>>>>>>>> > > > >> > >> >> as a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them to
>> give
>> >>>>> us
>> >>>>>>> the
>> >>>>>>>>>> > different
>> >>>>>>>>>> > > name
>> >>>>>>>>>> > > > >> > segments
>> >>>>>>>>>> > > > >> > >> >> >>> > separately?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be performing
>> any
>> >>>>>>>>> actions on
>> >>>>>>>>>> > this key
>> >>>>>>>>>> > > > other
>> >>>>>>>>>> > > > >> > than
>> >>>>>>>>>> > > > >> > >> >> >>> > matching?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
>> >>> 9:30
>> >>>>>>> AM,
>> >>>>>>>>> Michael
>> >>>>>>>>>> > > Pearce <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
>> concerns
>> >>> on
>> >>>>> 1
>> >>>>>>>>> and 2
>> >>>>>>>>>> > still?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this moving
>> >>> along a
>> >>>>>>> bit
>> >>>>>>>>> more
>> >>>>>>>>>> > I'd also
>> >>>>>>>>>> > > > like
>> >>>>>>>>>> > > > >> to
>> >>>>>>>>>> > > > >> > >> ask
>> >>>>>>>>>> > > > >> > >> >> to
>> >>>>>>>>>> > > > >> > >> >> >>> get
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last
>> points:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're
>> all
>> >>>>>>> roughly
>> >>>>>>>>> happy
>> >>>>>>>>>> > with the
>> >>>>>>>>>> > > > >> header
>> >>>>>>>>>> > > > >> > >> value
>> >>>>>>>>>> > > > >> > >> >> >>> > being a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe
>> consensus
>> >>> has
>> >>>>>>>>> been for an
>> >>>>>>>>>> > > > namespace
>> >>>>>>>>>> > > > >> > based
>> >>>>>>>>>> > > > >> > >> int
>> >>>>>>>>>> > > > >> > >> >> >>> > approach
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the
>> key.
>> >>> Any
>> >>>>>>>>> objections
>> >>>>>>>>>> > if this
>> >>>>>>>>>> > > > is
>> >>>>>>>>>> > > > >> > what
>> >>>>>>>>>> > > > >> > >> we
>> >>>>>>>>>> > > > >> > >> >> go
>> >>>>>>>>>> > > > >> > >> >> >>> > with?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
>> >>>>> assumption
>> >>>>>>> in
>> >>>>>>>>> (4) is
>> >>>>>>>>>> > > correct,
>> >>>>>>>>>> > > > >> > >> {int,int}
>> >>>>>>>>>> > > > >> > >> >> >>> keys.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both int's be
>> >>>>> int16 or
>> >>>>>>>>> int32?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them being
>> >>> int16(2
>> >>>>>>>>> bytes) as
>> >>>>>>>>>> > combined
>> >>>>>>>>>> > > > is
>> >>>>>>>>>> > > > >> > space
>> >>>>>>>>>> > > > >> > >> of
>> >>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > per
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > original and gives
>> >>> plenty
>> >>>>> of
>> >>>>>>>>>> > combinations for
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> >> >>> foreseeable,
>> >>>>>>>>>> > > > >> > >> >> >>> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead small.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any
>> benefit in
>> >>>>>>>>> another kip
>> >>>>>>>>>> > call to
>> >>>>>>>>>> > > > >> discuss
>> >>>>>>>>>> > > > >> > >> >> these at
>> >>>>>>>>>> > > > >> > >> >> >>> > all?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>> ______________________________
>> >>>>>>>>>> > __________
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
>> >>>>>>>>> k.burstev@yandex.com>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
>> November
>> >>> 18,
>> >>>>>>> 2016
>> >>>>>>>>>> > 7:07:07 AM
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
>> >>> dev@kafka.apache.org
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
>> [DISCUSS]
>> >>>>> KIP-82
>> >>>>>>>>> - Add
>> >>>>>>>>>> > Record
>> >>>>>>>>>> > > > Headers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is worth
>> >>> also
>> >>>>> i
>> >>>>>>>>> agree. As
>> >>>>>>>>>> > a user:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
>> >>>>>>>>> worthwhile
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers
>> should
>> >>>>> be a
>> >>>>>>>>> top level
>> >>>>>>>>>> > > option
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
>> >>>>> "Ignacio
>> >>>>>>>>> Solis" <
>> >>>>>>>>>> > > > >> isolis@igso.net
>> >>>>>>>>>> > > > >> > >:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers
>> are
>> >>>>>>>>> worthwhile
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
>> >>> should
>> >>>>> be
>> >>>>>>> a
>> >>>>>>>>> top
>> >>>>>>>>>> > level
>> >>>>>>>>>> > > option
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14,
>> 2016
>> >>> at
>> >>>>>>> 9:16
>> >>>>>>>>> AM,
>> >>>>>>>>>> > Michael
>> >>>>>>>>>> > > > Pearce
>> >>>>>>>>>> > > > >> <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
>> >>>>> details/examples
>> >>>>>>>>> the
>> >>>>>>>>>> > original
>> >>>>>>>>>> > > > proposal
>> >>>>>>>>>> > > > >> > for
>> >>>>>>>>>> > > > >> > >> key
>> >>>>>>>>>> > > > >> > >> >> >>> > spacing
>> >>>>>>>>>> > > > >> > >> >> >>> > >> ,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > not
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as
>> per
>> >>>>>>>>> discussion
>> >>>>>>>>>> > namespace
>> >>>>>>>>>> > > > >> idea.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need to
>> >>> update
>> >>>>>>> the
>> >>>>>>>>> kip,
>> >>>>>>>>>> > when we
>> >>>>>>>>>> > > get
>> >>>>>>>>>> > > > >> > >> agreement
>> >>>>>>>>>> > > > >> > >> >> >>> this
>> >>>>>>>>>> > > > >> > >> >> >>> > is a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > better
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach (which
>> >>> seems
>> >>>>> to
>> >>>>>>>>> be the
>> >>>>>>>>>> > case if I
>> >>>>>>>>>> > > > have
>> >>>>>>>>>> > > > >> > >> >> understood
>> >>>>>>>>>> > > > >> > >> >> >>> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > general
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
>> >>>>>>>>> conversation)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
>> >>> ints,
>> >>>>> at
>> >>>>>>>>> very
>> >>>>>>>>>> > early stage
>> >>>>>>>>>> > > > we
>> >>>>>>>>>> > > > >> did
>> >>>>>>>>>> > > > >> > >> think
>> >>>>>>>>>> > > > >> > >> >> >>> about
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added
>> complexity
>> >>>>> for
>> >>>>>>>>> the
>> >>>>>>>>>> > saving isn't
>> >>>>>>>>>> > > > >> worth
>> >>>>>>>>>> > > > >> > it.
>> >>>>>>>>>> > > > >> > >> >> I'd
>> >>>>>>>>>> > > > >> > >> >> >>> > rather
>> >>>>>>>>>> > > > >> > >> >> >>> > >> go
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
>> >>>>>>>>> overheads and
>> >>>>>>>>>> > size
>> >>>>>>>>>> > > int16
>> >>>>>>>>>> > > > >> > (2bytes)
>> >>>>>>>>>> > > > >> > >> >> keys
>> >>>>>>>>>> > > > >> > >> >> >>> as
>> >>>>>>>>>> > > > >> > >> >> >>> > it
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
>> >>>>>>> headers,
>> >>>>>>>>> there
>> >>>>>>>>>> > is as
>> >>>>>>>>>> > > per
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > kip
>> >>>>>>>>>> > > > >> > >> as
>> >>>>>>>>>> > > > >> > >> >> we
>> >>>>>>>>>> > > > >> > >> >> >>> > use an
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
>> >>>>> headers
>> >>>>>>>>> are
>> >>>>>>>>>> > present or
>> >>>>>>>>>> > > > not as
>> >>>>>>>>>> > > > >> > such
>> >>>>>>>>>> > > > >> > >> >> >>> > provides a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > zero
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead
>> currently
>> >>> if
>> >>>>>>>>> headers are
>> >>>>>>>>>> > not
>> >>>>>>>>>> > > used.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
>> >>>>> mentions
>> >>>>>>>>> would be
>> >>>>>>>>>> > good
>> >>>>>>>>>> > > > first
>> >>>>>>>>>> > > > >> > if we
>> >>>>>>>>>> > > > >> > >> >> can
>> >>>>>>>>>> > > > >> > >> >> >>> get
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > do
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have
>> general
>> >>>>>>>>> consensus that
>> >>>>>>>>>> > (1)
>> >>>>>>>>>> > > > headers
>> >>>>>>>>>> > > > >> > are
>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>> >>>>>>>>>> > > > >> > >> >> >>> > >> and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want
>> it
>> >>> as
>> >>>>> a
>> >>>>>>>>> top level
>> >>>>>>>>>> > entity.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
>> >>>>>>> obvious i
>> >>>>>>>>>> > believe (1)
>> >>>>>>>>>> > > > >> headers
>> >>>>>>>>>> > > > >> > are
>> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a top
>> level
>> >>>>>>>>> entity.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>> ______________________________
>> >>>>>>>>>> > __________
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
>> Hoover <
>> >>>>>>>>>> > > roger.hoover@gmail.com
>> >>>>>>>>>> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
>> >>>>> November
>> >>>>>>>>> 9, 2016
>> >>>>>>>>>> > 9:10:47
>> >>>>>>>>>> > > > PM
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
>> >>>>> dev@kafka.apache.org
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
>> >>> [DISCUSS]
>> >>>>>>>>> KIP-82 - Add
>> >>>>>>>>>> > > Record
>> >>>>>>>>>> > > > >> > Headers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
>> >>>>> little
>> >>>>>>>>> in the
>> >>>>>>>>>> > weeds but
>> >>>>>>>>>> > > > >> thanks
>> >>>>>>>>>> > > > >> > >> for
>> >>>>>>>>>> > > > >> > >> >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> replies
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a
>> prefix
>> >>>>> and
>> >>>>>>>>> {int,
>> >>>>>>>>>> > int} can
>> >>>>>>>>>> > > be
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> same.
>> >>>>>>>>>> > > > >> > >> >> It
>> >>>>>>>>>> > > > >> > >> >> >>> > doesn't
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > look
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > like
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what the
>> KIP
>> >>> is
>> >>>>>>>>> saying the
>> >>>>>>>>>> > "Open"
>> >>>>>>>>>> > > > >> > section.
>> >>>>>>>>>> > > > >> > >> The
>> >>>>>>>>>> > > > >> > >> >> >>> > example
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shows
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
>> >>>>> 210002
>> >>>>>>>>> for App
>> >>>>>>>>>> > Dynamics
>> >>>>>>>>>> > > > >> > implying
>> >>>>>>>>>> > > > >> > >> >> that
>> >>>>>>>>>> > > > >> > >> >> >>> the
>> >>>>>>>>>> > > > >> > >> >> >>> > New
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization will
>> >>> have
>> >>>>>>>>> only a
>> >>>>>>>>>> > single
>> >>>>>>>>>> > > > header id
>> >>>>>>>>>> > > > >> > to
>> >>>>>>>>>> > > > >> > >> work
>> >>>>>>>>>> > > > >> > >> >> >>> > with. Or
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > is
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The
>> main
>> >>>>> point
>> >>>>>>>>> of a
>> >>>>>>>>>> > namespace
>> >>>>>>>>>> > > or
>> >>>>>>>>>> > > > >> > prefix
>> >>>>>>>>>> > > > >> > >> is
>> >>>>>>>>>> > > > >> > >> >> to
>> >>>>>>>>>> > > > >> > >> >> >>> > reduce
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of
>> config
>> >>>>>>> mapping
>> >>>>>>>>> or
>> >>>>>>>>>> > > registration
>> >>>>>>>>>> > > > >> > >> depending
>> >>>>>>>>>> > > > >> > >> >> on
>> >>>>>>>>>> > > > >> > >> >> >>> how
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> namespaces/prefixes
>> >>>>> are
>> >>>>>>>>> managed.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to
>> hear
>> >>>>> more
>> >>>>>>>>> feedback
>> >>>>>>>>>> > on the
>> >>>>>>>>>> > > > >> > >> higher-level
>> >>>>>>>>>> > > > >> > >> >> >>> > questions
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9,
>> 2016
>> >>> at
>> >>>>>>>>> 11:38 AM,
>> >>>>>>>>>> > radai <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com
>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think this
>> >>>>> discussion
>> >>>>>>>>> is
>> >>>>>>>>>> > getting a
>> >>>>>>>>>> > > bit
>> >>>>>>>>>> > > > >> into
>> >>>>>>>>>> > > > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> weeds on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > technical
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > implementation
>> >>>>> details.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to
>> step
>> >>>>> back a
>> >>>>>>>>> minute
>> >>>>>>>>>> > and try
>> >>>>>>>>>> > > > and
>> >>>>>>>>>> > > > >> > >> establish
>> >>>>>>>>>> > > > >> > >> >> >>> > where we
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > are in
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
>> >>> nacho's
>> >>>>>>> last
>> >>>>>>>>>> > paragraph)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all
>> in
>> >>>>>>>>> agreement that
>> >>>>>>>>>> > headers
>> >>>>>>>>>> > > > are
>> >>>>>>>>>> > > > >> a
>> >>>>>>>>>> > > > >> > >> >> >>> worthwhile
>> >>>>>>>>>> > > > >> > >> >> >>> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > useful
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to
>> have?
>> >>>>> this
>> >>>>>>>>> was
>> >>>>>>>>>> > contested
>> >>>>>>>>>> > > > early
>> >>>>>>>>>> > > > >> on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all
>> in
>> >>>>>>>>> agreement on
>> >>>>>>>>>> > headers
>> >>>>>>>>>> > > as
>> >>>>>>>>>> > > > top
>> >>>>>>>>>> > > > >> > >> level
>> >>>>>>>>>> > > > >> > >> >> >>> entity
>> >>>>>>>>>> > > > >> > >> >> >>> > vs
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > headers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> squirreled-away in
>> >>>>> V?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there are
>> still
>> >>>>>>>>> concerns
>> >>>>>>>>>> > around
>> >>>>>>>>>> > > these
>> >>>>>>>>>> > > > #2
>> >>>>>>>>>> > > > >> > >> points
>> >>>>>>>>>> > > > >> > >> >> >>> (#jay?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now back
>> to
>> >>> our
>> >>>>>>>>> normal
>> >>>>>>>>>> > programming
>> >>>>>>>>>> > > > ...)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are
>> nice.
>> >>>>>>> having
>> >>>>>>>>> said
>> >>>>>>>>>> > that, its
>> >>>>>>>>>> > > > >> adding
>> >>>>>>>>>> > > > >> > >> >> >>> complexity
>> >>>>>>>>>> > > > >> > >> >> >>> > >> (see
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>> https://github.com/addthis/
>> >>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>> main/java/com/clearspring/
>> >>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
>> >>>>> result)
>> >>>>>>>>> and would
>> >>>>>>>>>> > require
>> >>>>>>>>>> > > > >> anyone
>> >>>>>>>>>> > > > >> > >> >> writing
>> >>>>>>>>>> > > > >> > >> >> >>> > other
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > clients
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go?
>> Bash?
>> >>>>> ;-) )
>> >>>>>>>>> to
>> >>>>>>>>>> > > get/implement
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > >> same,
>> >>>>>>>>>> > > > >> > >> >> and
>> >>>>>>>>>> > > > >> > >> >> >>> for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain
>> (int
>> >>> vs
>> >>>>>>>>> string is
>> >>>>>>>>>> > order of
>> >>>>>>>>>> > > > >> > magnitude,
>> >>>>>>>>>> > > > >> > >> >> this
>> >>>>>>>>>> > > > >> > >> >> >>> > isnt).
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int
>> namespacing vs
>> >>>>>>> {int,
>> >>>>>>>>> int}
>> >>>>>>>>>> > > namespacing
>> >>>>>>>>>> > > > >> are
>> >>>>>>>>>> > > > >> > >> >> basically
>> >>>>>>>>>> > > > >> > >> >> >>> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > same
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
>> >>>>> namespacing
>> >>>>>>>>> an int64
>> >>>>>>>>>> > and
>> >>>>>>>>>> > > > giving
>> >>>>>>>>>> > > > >> > people
>> >>>>>>>>>> > > > >> > >> >> while
>> >>>>>>>>>> > > > >> > >> >> >>> > 2^32
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the part
>> i
>> >>>>> like
>> >>>>>>>>> about this
>> >>>>>>>>>> > is
>> >>>>>>>>>> > > > letting
>> >>>>>>>>>> > > > >> > >> people
>> >>>>>>>>>> > > > >> > >> >> >>> have a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> large
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers with
>> one
>> >>>>>>>>> registration so
>> >>>>>>>>>> > they
>> >>>>>>>>>> > > > dont
>> >>>>>>>>>> > > > >> > have
>> >>>>>>>>>> > > > >> > >> to
>> >>>>>>>>>> > > > >> > >> >> come
>> >>>>>>>>>> > > > >> > >> >> >>> > back
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > every
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
>> >>> plugin/header
>> >>>>>>>>> they want to
>> >>>>>>>>>> > > > "reserve".
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
>> >>> 2016
>> >>>>> at
>> >>>>>>>>> 11:01 AM,
>> >>>>>>>>>> > Roger
>> >>>>>>>>>> > > > >> Hoover
>> >>>>>>>>>> > > > >> > <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> roger.hoover@gmail.com>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
>> >>> the
>> >>>>>>>>> debate has
>> >>>>>>>>>> > been
>> >>>>>>>>>> > > about
>> >>>>>>>>>> > > > >> > >> overhead +
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if
>> we
>> >>>>> have
>> >>>>>>>>>> > considered a
>> >>>>>>>>>> > > > varint
>> >>>>>>>>>> > > > >> > >> encoding
>> >>>>>>>>>> > > > >> > >> >> (
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>> https://developers.google.com/
>> >>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the header
>> >>> length
>> >>>>>>>>> field (int32
>> >>>>>>>>>> > in the
>> >>>>>>>>>> > > > >> > proposal)
>> >>>>>>>>>> > > > >> > >> >> and
>> >>>>>>>>>> > > > >> > >> >> >>> for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > header
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
>> >>> headers,
>> >>>>>>> the
>> >>>>>>>>>> > overhead would
>> >>>>>>>>>> > > > be a
>> >>>>>>>>>> > > > >> > >> single
>> >>>>>>>>>> > > > >> > >> >> >>> byte
>> >>>>>>>>>> > > > >> > >> >> >>> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for each
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128
>> would
>> >>>>> also
>> >>>>>>>>> need only a
>> >>>>>>>>>> > > single
>> >>>>>>>>>> > > > >> byte?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov
>> 9,
>> >>>>> 2016
>> >>>>>>> at
>> >>>>>>>>> 6:43 AM,
>> >>>>>>>>>> > > radai <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com
>> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus -
>> and
>> >>>>> very
>> >>>>>>>>> dangerous
>> >>>>>>>>>> > (youre
>> >>>>>>>>>> > > > >> > >> essentially
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary
>> code
>> >>>>> off
>> >>>>>>>>> the
>> >>>>>>>>>> > internet on
>> >>>>>>>>>> > > > your
>> >>>>>>>>>> > > > >> > >> servers
>> >>>>>>>>>> > > > >> > >> >> ...
>> >>>>>>>>>> > > > >> > >> >> >>> > bad
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > idea
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > without
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox,
>> even
>> >>>>> with)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
>> >>> being
>> >>>>> a
>> >>>>>>>>> purely
>> >>>>>>>>>> > > > administrative
>> >>>>>>>>>> > > > >> > task
>> >>>>>>>>>> > > > >> > >> - i
>> >>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
>> >>> would,
>> >>>>>>>>> really,
>> >>>>>>>>>> > because
>> >>>>>>>>>> > > > then my
>> >>>>>>>>>> > > > >> > >> earlier
>> >>>>>>>>>> > > > >> > >> >> >>> > point on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>> remapping
>> >>>>>>>>> process would
>> >>>>>>>>>> > be
>> >>>>>>>>>> > > > invalid,
>> >>>>>>>>>> > > > >> > but
>> >>>>>>>>>> > > > >> > >> at
>> >>>>>>>>>> > > > >> > >> >> >>> > linkedin,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team
>> im
>> >>> in)
>> >>>>>>> run
>> >>>>>>>>> kafka
>> >>>>>>>>>> > as a
>> >>>>>>>>>> > > > service.
>> >>>>>>>>>> > > > >> > we
>> >>>>>>>>>> > > > >> > >> dont
>> >>>>>>>>>> > > > >> > >> >> >>> > really
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > know
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
>> >>>>>>>>> applications
>> >>>>>>>>>> > that use
>> >>>>>>>>>> > > > kafka)
>> >>>>>>>>>> > > > >> > are
>> >>>>>>>>>> > > > >> > >> up
>> >>>>>>>>>> > > > >> > >> >> to
>> >>>>>>>>>> > > > >> > >> >> >>> at
>> >>>>>>>>>> > > > >> > >> >> >>> > any
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > given
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
>> >>> possible
>> >>>>>>>>> (given the
>> >>>>>>>>>> > > > existance of
>> >>>>>>>>>> > > > >> > >> headers
>> >>>>>>>>>> > > > >> > >> >> >>> and a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem)
>> for
>> >>>>> some
>> >>>>>>>>>> > application to
>> >>>>>>>>>> > > > >> "equip"
>> >>>>>>>>>> > > > >> > >> their
>> >>>>>>>>>> > > > >> > >> >> >>> > >> producers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
>> >>>>> required
>> >>>>>>>>> plugin
>> >>>>>>>>>> > without us
>> >>>>>>>>>> > > > >> > knowing.
>> >>>>>>>>>> > > > >> > >> i
>> >>>>>>>>>> > > > >> > >> >> dont
>> >>>>>>>>>> > > > >> > >> >> >>> > mean
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
>> >>>>> want to
>> >>>>>>>>> make the
>> >>>>>>>>>> > point
>> >>>>>>>>>> > > > that
>> >>>>>>>>>> > > > >> > its
>> >>>>>>>>>> > > > >> > >> not
>> >>>>>>>>>> > > > >> > >> >> as
>> >>>>>>>>>> > > > >> > >> >> >>> > simple
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
>> across a
>> >>>>>>>>> large-enough
>> >>>>>>>>>> > > > organization.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed,
>> Nov 9,
>> >>>>> 2016
>> >>>>>>>>> at 6:17
>> >>>>>>>>>> > AM,
>> >>>>>>>>>> > > > Magnus
>> >>>>>>>>>> > > > >> > >> Edenhill
>> >>>>>>>>>> > > > >> > >> >> <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
>> >>> there
>> >>>>> is
>> >>>>>>> a
>> >>>>>>>>> piece
>> >>>>>>>>>> > missing
>> >>>>>>>>>> > > in
>> >>>>>>>>>> > > > >> the
>> >>>>>>>>>> > > > >> > >> >> Strings
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > where
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>> pro-Stringers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason
>> that
>> >>> by
>> >>>>>>>>> providing
>> >>>>>>>>>> > unique
>> >>>>>>>>>> > > > string
>> >>>>>>>>>> > > > >> > >> >> >>> identifiers
>> >>>>>>>>>> > > > >> > >> >> >>> > for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> everything
>> >>>>> will
>> >>>>>>>>> just
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
>> >>> work
>> >>>>>>> for
>> >>>>>>>>> all
>> >>>>>>>>>> > parts of
>> >>>>>>>>>> > > the
>> >>>>>>>>>> > > > >> > stream
>> >>>>>>>>>> > > > >> > >> >> >>> pipeline.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
>> >>>>> strings
>> >>>>>>>>> dont mean
>> >>>>>>>>>> > > anything
>> >>>>>>>>>> > > > by
>> >>>>>>>>>> > > > >> > >> >> themselves,
>> >>>>>>>>>> > > > >> > >> >> >>> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > while we
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
>> >>>>> envision
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
>> >>>>> plugin
>> >>>>>>>>> loader
>> >>>>>>>>>> > that
>> >>>>>>>>>> > > > >> downloads,
>> >>>>>>>>>> > > > >> > >> >> compiles,
>> >>>>>>>>>> > > > >> > >> >> >>> > links
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon
>> as
>> >>>>>>> they're
>> >>>>>>>>> seen by
>> >>>>>>>>>> > a
>> >>>>>>>>>> > > > >> consumer, I
>> >>>>>>>>>> > > > >> > >> dont
>> >>>>>>>>>> > > > >> > >> >> >>> really
>> >>>>>>>>>> > > > >> > >> >> >>> > >> see
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
>> dynamic
>> >>>>> (and
>> >>>>>>>>> fragile) in
>> >>>>>>>>>> > > > practice.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the
>> real
>> >>>>> world
>> >>>>>>>>> an
>> >>>>>>>>>> > application
>> >>>>>>>>>> > > > will
>> >>>>>>>>>> > > > >> be
>> >>>>>>>>>> > > > >> > >> >> >>> configured
>> >>>>>>>>>> > > > >> > >> >> >>> > >> with
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > a set
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
>> >>> add
>> >>>>>>>>> (producer)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
>> >>>>>>> (consumer)
>> >>>>>>>>> headers.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is
>> an
>> >>>>>>>>> administrative
>> >>>>>>>>>> > task
>> >>>>>>>>>> > > > based
>> >>>>>>>>>> > > > >> on
>> >>>>>>>>>> > > > >> > >> what
>> >>>>>>>>>> > > > >> > >> >> >>> > features a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > client
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>> needs/provides
>> >>>>>>> and
>> >>>>>>>>> results
>> >>>>>>>>>> > in
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
>> sort of
>> >>>>>>>>> configuration
>> >>>>>>>>>> > to
>> >>>>>>>>>> > > > enable
>> >>>>>>>>>> > > > >> and
>> >>>>>>>>>> > > > >> > >> >> >>> configure
>> >>>>>>>>>> > > > >> > >> >> >>> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > desired
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since
>> this
>> >>>>> needs
>> >>>>>>>>> to be kept
>> >>>>>>>>>> > > > somewhat
>> >>>>>>>>>> > > > >> in
>> >>>>>>>>>> > > > >> > >> sync
>> >>>>>>>>>> > > > >> > >> >> >>> across
>> >>>>>>>>>> > > > >> > >> >> >>> > an
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no
>> point
>> >>> in
>> >>>>>>>>> having
>> >>>>>>>>>> > producers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
>> headers
>> >>> no
>> >>>>>>>>> consumers
>> >>>>>>>>>> > will
>> >>>>>>>>>> > > read,
>> >>>>>>>>>> > > > >> and
>> >>>>>>>>>> > > > >> > >> vice
>> >>>>>>>>>> > > > >> > >> >> >>> versa),
>> >>>>>>>>>> > > > >> > >> >> >>> > >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > added
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
>> assigning
>> >>>>> an
>> >>>>>>> id
>> >>>>>>>>>> > namespace
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
>> >>>>> plugin
>> >>>>>>> as
>> >>>>>>>>> it is
>> >>>>>>>>>> > being
>> >>>>>>>>>> > > > >> > configured
>> >>>>>>>>>> > > > >> > >> >> should
>> >>>>>>>>>> > > > >> > >> >> >>> be
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> 2016-11-09
>> >>>>> 13:06
>> >>>>>>>>> GMT+01:00
>> >>>>>>>>>> > > Michael
>> >>>>>>>>>> > > > >> > Pearce <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
>> Michael.Pearce@ig.com>:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
>> >>>>>>>>> following/catching
>> >>>>>>>>>> > up on
>> >>>>>>>>>> > > > what
>> >>>>>>>>>> > > > >> > seems
>> >>>>>>>>>> > > > >> > >> to
>> >>>>>>>>>> > > > >> > >> >> be
>> >>>>>>>>>> > > > >> > >> >> >>> an
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > active
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
>> >>>>> sorry if
>> >>>>>>>>> it may
>> >>>>>>>>>> > seem
>> >>>>>>>>>> > > > obvious
>> >>>>>>>>>> > > > >> > but
>> >>>>>>>>>> > > > >> > >> what
>> >>>>>>>>>> > > > >> > >> >> >>> does
>> >>>>>>>>>> > > > >> > >> >> >>> > MD
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > stand
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My
>> take on
>> >>>>>>>>> String vs Int:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
>> >>> state
>> >>>>>>>>> first I am
>> >>>>>>>>>> > pro Int
>> >>>>>>>>>> > > > (16
>> >>>>>>>>>> > > > >> or
>> >>>>>>>>>> > > > >> > >> 32).
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
>> >>> though
>> >>>>>>>>> playing
>> >>>>>>>>>> > devils
>> >>>>>>>>>> > > > advocate
>> >>>>>>>>>> > > > >> > see a
>> >>>>>>>>>> > > > >> > >> >> big
>> >>>>>>>>>> > > > >> > >> >> >>> plus
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > with the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
>> >>> keys,
>> >>>>>>>>> this is
>> >>>>>>>>>> > around
>> >>>>>>>>>> > > > >> > integrating
>> >>>>>>>>>> > > > >> > >> >> into an
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > existing
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
>> >>>>> other
>> >>>>>>>>> systems use
>> >>>>>>>>>> > > String
>> >>>>>>>>>> > > > >> based
>> >>>>>>>>>> > > > >> > >> >> headers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > it
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
>> >>> easier
>> >>>>> for
>> >>>>>>>>> these to
>> >>>>>>>>>> > be
>> >>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > into.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How
>> with
>> >>> Int
>> >>>>>>>>> based
>> >>>>>>>>>> > headers
>> >>>>>>>>>> > > could
>> >>>>>>>>>> > > > we
>> >>>>>>>>>> > > > >> > >> provide
>> >>>>>>>>>> > > > >> > >> >> a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>> integration
>> >>>>>>>>> simple /
>> >>>>>>>>>> > easy with
>> >>>>>>>>>> > > > >> > transition
>> >>>>>>>>>> > > > >> > >> >> flows
>> >>>>>>>>>> > > > >> > >> >> >>> > over
>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
>> >>> luck
>> >>>>>>>>> buddy
>> >>>>>>>>>> > you're on
>> >>>>>>>>>> > > your
>> >>>>>>>>>> > > > >> own
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
>> simply
>> >>>>> hash
>> >>>>>>>>> the string
>> >>>>>>>>>> > into
>> >>>>>>>>>> > > int
>> >>>>>>>>>> > > > >> code
>> >>>>>>>>>> > > > >> > >> and
>> >>>>>>>>>> > > > >> > >> >> hope
>> >>>>>>>>>> > > > >> > >> >> >>> > for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> no
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
>> >>> back
>> >>>>>>>>> though?)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
>> >>>>> style
>> >>>>>>> as
>> >>>>>>>>>> > mentioned by
>> >>>>>>>>>> > > > nacho.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > ______________________________
>> >>>>>>>>>> > > > >> > __________
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
>> >>> radai
>> >>>>> <
>> >>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
>> >>>>>>> Wednesday,
>> >>>>>>>>>> > November 9,
>> >>>>>>>>>> > > 2016
>> >>>>>>>>>> > > > >> > 8:12 AM
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
>> >>>>>>>>> dev@kafka.apache.org
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> Subject:
>> >>> Re:
>> >>>>>>>>> [DISCUSS]
>> >>>>>>>>>> > KIP-82 -
>> >>>>>>>>>> > > > Add
>> >>>>>>>>>> > > > >> > >> Record
>> >>>>>>>>>> > > > >> > >> >> >>> Headers
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> thinking
>> >>>>> about
>> >>>>>>>>> it some
>> >>>>>>>>>> > more,
>> >>>>>>>>>> > > the
>> >>>>>>>>>> > > > >> best
>> >>>>>>>>>> > > > >> > >> way to
>> >>>>>>>>>> > > > >> > >> >> >>> > transmit
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
>> >>>>>>>>> consumers would
>> >>>>>>>>>> > be to
>> >>>>>>>>>> > > > put it
>> >>>>>>>>>> > > > >> > in
>> >>>>>>>>>> > > > >> > >> the
>> >>>>>>>>>> > > > >> > >> >> MD
>> >>>>>>>>>> > > > >> > >> >> >>> > >> response
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it
>> should
>> >>> be
>> >>>>>>>>> discussed
>> >>>>>>>>>> > now.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
>> >>> Nov
>> >>>>> 9,
>> >>>>>>>>> 2016 at
>> >>>>>>>>>> > 12:09
>> >>>>>>>>>> > > AM,
>> >>>>>>>>>> > > > >> > radai <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> >>>>>>> radai.rosenblatt@gmail.com
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im
>> not
>> >>>>>>> opposed
>> >>>>>>>>> to the
>> >>>>>>>>>> > idea of
>> >>>>>>>>>> > > > >> > namespace
>> >>>>>>>>>> > > > >> > >> >> >>> mapping.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> all
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > im
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not
>> part
>> >>>>> of
>> >>>>>>>>> the "mvp"
>> >>>>>>>>>> > and,
>> >>>>>>>>>> > > > since
>> >>>>>>>>>> > > > >> it
>> >>>>>>>>>> > > > >> > >> >> requires
>> >>>>>>>>>> > > > >> > >> >> >>> no
>> >>>>>>>>>> > > > >> > >> >> >>> > >> wire
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > format
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> always
>> >>> be
>> >>>>>>>>> added later.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
>> >>> its
>> >>>>> not
>> >>>>>>>>> as
>> >>>>>>>>>> > simple as
>> >>>>>>>>>> > > just
>> >>>>>>>>>> > > > >> > >> >> configuring
>> >>>>>>>>>> > > > >> > >> >> >>> MM
>> >>>>>>>>>> > > > >> > >> >> >>> > to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> do
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say
>> i've
>> >>>>>>>>> implemented
>> >>>>>>>>>> > large
>> >>>>>>>>>> > > > message
>> >>>>>>>>>> > > > >> > >> >> support as
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> target
>> >>>>>>> cluster
>> >>>>>>>>> its been
>> >>>>>>>>>> > > > remapped
>> >>>>>>>>>> > > > >> to
>> >>>>>>>>>> > > > >> > >> >> {999,1}.
>> >>>>>>>>>> > > > >> > >> >> >>> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
>> >>> need
>> >>>>> to
>> >>>>>>>>> be told
>> >>>>>>>>>> > to look
>> >>>>>>>>>> > > > for
>> >>>>>>>>>> > > > >> the
>> >>>>>>>>>> > > > >> > >> large
>> >>>>>>>>>> > > > >> > >> >> >>> > message
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
>> >>>>>>> {999,1}.
>> >>>>>>>>> doable,
>> >>>>>>>>>> > but
>> >>>>>>>>>> > > > tricky.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On
>> Tue,
>> >>>>> Nov
>> >>>>>>> 8,
>> >>>>>>>>> 2016 at
>> >>>>>>>>>> > 10:29
>> >>>>>>>>>> > > > PM,
>> >>>>>>>>>> > > > >> > Gwen
>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>> gwen@confluent.io
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> While
>> >>> you
>> >>>>>>> can
>> >>>>>>>>> do
>> >>>>>>>>>> > whatever
>> >>>>>>>>>> > > you
>> >>>>>>>>>> > > > >> want
>> >>>>>>>>>> > > > >> > >> with a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> namespace
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > your
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
>> >>> I'd
>> >>>>>>>>> expect is
>> >>>>>>>>>> > for each
>> >>>>>>>>>> > > > app
>> >>>>>>>>>> > > > >> to
>> >>>>>>>>>> > > > >> > >> >> >>> namespaces
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So
>> if I
>> >>>>>>>>> accidentally
>> >>>>>>>>>> > used
>> >>>>>>>>>> > > 666
>> >>>>>>>>>> > > > for
>> >>>>>>>>>> > > > >> > my
>> >>>>>>>>>> > > > >> > >> HR
>> >>>>>>>>>> > > > >> > >> >> >>> > >> department,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > still
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
>> >>>>>>> RadaiApp,
>> >>>>>>>>> I can
>> >>>>>>>>>> > config
>> >>>>>>>>>> > > > >> > >> "namespace=42"
>> >>>>>>>>>> > > > >> > >> >> >>> for
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
>> >>> look
>> >>>>>>>>> normal.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
>> >>>>> means
>> >>>>>>>>> you only
>> >>>>>>>>>> > need to
>> >>>>>>>>>> > > > sync
>> >>>>>>>>>> > > > >> > usage
>> >>>>>>>>>> > > > >> > >> >> >>> inside
>> >>>>>>>>>> > > > >> > >> >> >>> > your
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > own
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> Still
>> >>>>> hard,
>> >>>>>>>>> but
>> >>>>>>>>>> > somewhat
>> >>>>>>>>>> > > > easier
>> >>>>>>>>>> > > > >> > than
>> >>>>>>>>>> > > > >> > >> >> syncing
>> >>>>>>>>>> > > > >> > >> >> >>> > with
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On
>> Tue,
>> >>>>> Nov
>> >>>>>>>>> 8, 2016
>> >>>>>>>>>> > at 10:07
>> >>>>>>>>>> > > > PM,
>> >>>>>>>>>> > > > >> > >> radai <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>>>>>> radai.rosenblatt@gmail.com>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> and
>> >>> we
>> >>>>> can
>> >>>>>>>>> start
>> >>>>>>>>>> > with
>> >>>>>>>>>> > > > >> {namespace,
>> >>>>>>>>>> > > > >> > >> id}
>> >>>>>>>>>> > > > >> > >> >> and
>> >>>>>>>>>> > > > >> > >> >> >>> no
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> always
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> add
>> >>> it
>> >>>>>>>>> later on
>> >>>>>>>>>> > if/when
>> >>>>>>>>>> > > > >> > collisions
>> >>>>>>>>>> > > > >> > >> >> >>> actually
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
>> >>>>>>> problem).
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> every
>> >>>>>>>>> interested
>> >>>>>>>>>> > party (so
>> >>>>>>>>>> > > > orgs
>> >>>>>>>>>> > > > >> > or
>> >>>>>>>>>> > > > >> > >> >> >>> > individuals)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > could
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > then
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > register
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >>> prefix
>> >>>>> (0
>> >>>>>>> =
>> >>>>>>>>>> > reserved, 1 =
>> >>>>>>>>>> > > > >> > confluent
>> >>>>>>>>>> > > > >> > >> ...
>> >>>>>>>>>> > > > >> > >> >> >>> 666
>> >>>>>>>>>> > > > >> > >> >> >>> > = me
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> the
>> >>>>> 2nd ID
>> >>>>>>>>> - so once
>> >>>>>>>>>> > > > linkedin
>> >>>>>>>>>> > > > >> > >> >> registers,
>> >>>>>>>>>> > > > >> > >> >> >>> say
>> >>>>>>>>>> > > > >> > >> >> >>> > 3,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > then
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
>> >>> use
>> >>>>> {3,
>> >>>>>>>>> *} with a
>> >>>>>>>>>> > > > reasonable
>> >>>>>>>>>> > > > >> > >> >> >>> expectation
>> >>>>>>>>>> > > > >> > >> >> >>> > to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> anything
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> else.
>> >>>>>>>>> further
>> >>>>>>>>>> > partitioning
>> >>>>>>>>>> > > > of
>> >>>>>>>>>> > > > >> > that *
>> >>>>>>>>>> > > > >> > >> >> >>> becomes
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >>>>> "upstream
>> >>>>>>>>>> > registration"
>> >>>>>>>>>> > > of a
>> >>>>>>>>>> > > > >> > >> namespace
>> >>>>>>>>>> > > > >> > >> >> >>> only
>> >>>>>>>>>> > > > >> > >> >> >>> > has
>> >>>>>>>>>> > > > >> > >> >> >>> > >> to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
>> >>> Tue,
>> >>>>>>> Nov
>> >>>>>>>>> 8, 2016
>> >>>>>>>>>> > at
>> >>>>>>>>>> > > 9:03
>> >>>>>>>>>> > > > PM,
>> >>>>>>>>>> > > > >> > >> James
>> >>>>>>>>>> > > > >> > >> >> >>> Cheng <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >>>>> wushujames@gmail.com
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> > On
>> >>>>> Nov
>> >>>>>>>>> 8, 2016,
>> >>>>>>>>>> > at 5:54
>> >>>>>>>>>> > > > PM,
>> >>>>>>>>>> > > > >> > Gwen
>> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>> gwen@confluent.io
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>> Thank
>> >>>>>>>>> you so
>> >>>>>>>>>> > much for
>> >>>>>>>>>> > > > this
>> >>>>>>>>>> > > > >> > clear
>> >>>>>>>>>> > > > >> > >> and
>> >>>>>>>>>> > > > >> > >> >> >>> fair
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> arguments.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>> I'm
>> >>>>> in
>> >>>>>>>>> favor of
>> >>>>>>>>>> > ints.
>> >>>>>>>>>> > > > Not a
>> >>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
>> >>>>>>>>>> > > > >> > >> >> >>> > but
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > in
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>> Even
>> >>>>>>>>> more in
>> >>>>>>>>>> > favor of
>> >>>>>>>>>> > > > >> Magnus's
>> >>>>>>>>>> > > > >> > >> >> >>> > decentralized
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>> tweak:
>> >>>>>>>>> add a
>> >>>>>>>>>> > namespace
>> >>>>>>>>>> > > > for
>> >>>>>>>>>> > > > >> > >> headers.
>> >>>>>>>>>> > > > >> > >> >> >>> This
>> >>>>>>>>>> > > > >> > >> >> >>> > will
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > allow
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > each
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>> use
>> >>>>>>>>> whatever IDs
>> >>>>>>>>>> > it
>> >>>>>>>>>> > > wants
>> >>>>>>>>>> > > > >> > >> >> internally,
>> >>>>>>>>>> > > > >> > >> >> >>> and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> then
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > let
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >>> deploying
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>> the
>> >>>>> app
>> >>>>>>>>> figure
>> >>>>>>>>>> > out an
>> >>>>>>>>>> > > > >> > available
>> >>>>>>>>>> > > > >> > >> >> >>> namespace
>> >>>>>>>>>> > > > >> > >> >> >>> > ID
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > for the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> > So
>> >>>>>>>>>> > > > >> > io.confluent.schema-registry
>> >>>>>>>>>> > > > >> > >> can
>> >>>>>>>>>> > > > >> > >> >> be
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >>>>> deployment
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>> and
>> >>>>>>> 0x57
>> >>>>>>>>> on
>> >>>>>>>>>> > yours, and
>> >>>>>>>>>> > > > the
>> >>>>>>>>>> > > > >> > poor
>> >>>>>>>>>> > > > >> > >> guys
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>> worry
>> >>>>>>>>> about that.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>> Gwen,
>> >>>>> if
>> >>>>>>> I
>> >>>>>>>>>> > understand
>> >>>>>>>>>> > > your
>> >>>>>>>>>> > > > >> > example
>> >>>>>>>>>> > > > >> > >> >> >>> right, an
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > application
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>> decide to
>> >>>>>>>>> use 0x01
>> >>>>>>>>>> > in one
>> >>>>>>>>>> > > > >> > >> deployment,
>> >>>>>>>>>> > > > >> > >> >> and
>> >>>>>>>>>> > > > >> > >> >> >>> > that
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > means
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > that
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> message
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> is
>> >>>>>>> written
>> >>>>>>>>> into the
>> >>>>>>>>>> > > > broker, it
>> >>>>>>>>>> > > > >> > >> will be
>> >>>>>>>>>> > > > >> > >> >> >>> > saved on
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>> specific
>> >>>>>>>>> namespace
>> >>>>>>>>>> > > (0x01).
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> If
>> >>> you
>> >>>>>>>>> were to
>> >>>>>>>>>> > mirror
>> >>>>>>>>>> > > that
>> >>>>>>>>>> > > > >> > message
>> >>>>>>>>>> > > > >> > >> >> into
>> >>>>>>>>>> > > > >> > >> >> >>> > another
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>> accompany
>> >>>>>>>>> the
>> >>>>>>>>>> > message,
>> >>>>>>>>>> > > > right?
>> >>>>>>>>>> > > > >> > What
>> >>>>>>>>>> > > > >> > >> if
>> >>>>>>>>>> > > > >> > >> >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>> other
>> >>>>>>>>> cluster uses
>> >>>>>>>>>> > 0x57?
>> >>>>>>>>>> > > > They
>> >>>>>>>>>> > > > >> > won't
>> >>>>>>>>>> > > > >> > >> >> >>> > understand
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> I'm
>> >>>>> not
>> >>>>>>>>> sure
>> >>>>>>>>>> > that's an
>> >>>>>>>>>> > > > >> avoidable
>> >>>>>>>>>> > > > >> > >> >> >>> problem. I
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > think it
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>> order
>> >>>>> to
>> >>>>>>>>> share
>> >>>>>>>>>> > data, you
>> >>>>>>>>>> > > > have
>> >>>>>>>>>> > > > >> to
>> >>>>>>>>>> > > > >> > >> also
>> >>>>>>>>>> > > > >> > >> >> >>> have a
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > shared
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>> understanding of
>> >>>>>>>>>> > what the
>> >>>>>>>>>> > > > >> > >> namespaces
>> >>>>>>>>>> > > > >> > >> >> >>> mean.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> Which
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > I
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>> because
>> >>>>>>> the
>> >>>>>>>>>> > alternate
>> >>>>>>>>>> > > > (sharing
>> >>>>>>>>>> > > > >> > >> >> *nothing*
>> >>>>>>>>>> > > > >> > >> >> >>> at
>> >>>>>>>>>> > > > >> > >> >> >>> > >> all)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > would
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>> would
>> >>>>> be
>> >>>>>>>>> no way to
>> >>>>>>>>>> > > > understand
>> >>>>>>>>>> > > > >> > each
>> >>>>>>>>>> > > > >> > >> >> other.
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>> -James
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>> Gwen
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> > On
>> >>>>> Tue,
>> >>>>>>>>> Nov 8,
>> >>>>>>>>>> > 2016 at
>> >>>>>>>>>> > > > 4:23
>> >>>>>>>>>> > > > >> > PM,
>> >>>>>>>>>> > > > >> > >> >> radai <
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >>>>>>>>> radai.rosenblatt@gmail.com
>> >>>>>>>>>> > >
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>> wrote:
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>
>> >>> +1
>> >>>>> for
>> >>>>>>>>> sean's
>> >>>>>>>>>> > > document.
>> >>>>>>>>>> > > > it
>> >>>>>>>>>> > > > >> > >> covers
>> >>>>>>>>>> > > > >> > >> >> >>> pretty
>> >>>>>>>>>> > > > >> > >> >> >>> > >> much
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > all
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>
>> >>>>>>> provides
>> >>>>>>>>>> > concrete
>> >>>>>>>>>> > > > figures
>> >>>>>>>>>> > > > >> to
>> >>>>>>>>>> > > > >> > >> argue
>> >>>>>>>>>> > > > >> > >> >> >>> about
>> >>>>>>>>>> > > > >> > >> >> >>> > :-)
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >>
>> >>>>>>>>> (nit-picking -
>> >>>>>>>>>> > used
>> >>>>>>>>>> > > the
>> >>>>>>>>>> > > > >> same
>> >>>>>>>>>> > > > >> > >> xkcd
>> >>>>>>>>>> > > > >> > >> >> >>> twice,
>> >>>>>>>>>> > > > >> > >> >> >>> > >> also
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > trove
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > has
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
>> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >>>>> superceded
>> >>>>>>>>>> > > > >> > >> >
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >> > --
>> >>>>>>>>>> > > > >> > Gwen Shapira
>> >>>>>>>>>> > > > >> > Product Manager | Confluent
>> >>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
>> >>>>>>>>>> > > > >> > Follow us: Twitter | blog
>> >>>>>>>>>> > > > >> >
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >> --
>> >>>>>>>>>> > > > >> *Todd Palino*
>> >>>>>>>>>> > > > >> Staff Site Reliability Engineer
>> >>>>>>>>>> > > > >> Data Infrastructure Streaming
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > > >> linkedin.com/in/toddpalino
>> >>>>>>>>>> > > > >>
>> >>>>>>>>>> > > >
>> >>>>>>>>>> > > >
>> >>>>>>>>>> > > >
>> >>>>>>>>>> > > > --
>> >>>>>>>>>> > > > Gwen Shapira
>> >>>>>>>>>> > > > Product Manager | Confluent
>> >>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
>> >>>>>>>>>> > > > Follow us: Twitter | blog
>> >>>>>>>>>> > > >
>> >>>>>>>>>> > >
>> >>>>>>>>>> >
>> >>>>>>>>>> >
>> >>>>>>>>>> > The information contained in this email is strictly
>> >>>>> confidential
>> >>>>>>>>> and for
>> >>>>>>>>>> > the use of the addressee only, unless otherwise
>> indicated.
>> >>> If
>> >>>>> you
>> >>>>>>>>> are not
>> >>>>>>>>>> > the intended recipient, please do not read, copy, use or
>> >>>>> disclose
>> >>>>>>>>> to others
>> >>>>>>>>>> > this message or any attachment. Please also notify the
>> >>> sender
>> >>>>> by
>> >>>>>>>>> replying
>> >>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and
>> then
>> >>>>> delete
>> >>>>>>>>> the email
>> >>>>>>>>>> > and any copies of it. Opinions, conclusion (etc) that do
>> not
>> >>>>>>>>> relate to the
>> >>>>>>>>>> > official business of this company shall be understood as
>> >>>>> neither
>> >>>>>>>>> given nor
>> >>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets
>> Limited
>> >>> (a
>> >>>>>>>>> company
>> >>>>>>>>>> > registered in England and Wales, company number 04008957)
>> >>> and
>> >>>>> IG
>> >>>>>>>>> Index
>> >>>>>>>>>> > Limited (a company registered in England and Wales,
>> company
>> >>>>>>> number
>> >>>>>>>>>> > 01190902). Registered address at Cannon Bridge House, 25
>> >>>>> Dowgate
>> >>>>>>>>> Hill,
>> >>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register number
>> >>>>> 195355)
>> >>>>>>>>> and IG
>> >>>>>>>>>> > Index Limited (register number 114059) are authorised and
>> >>>>>>>>> regulated by the
>> >>>>>>>>>> > Financial Conduct Authority.
>> >>>>>>>>>> >
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> The information contained in this email is strictly confidential
>> >>> and
>> >>>>>>> for
>> >>>>>>>>> the use of the addressee only, unless otherwise indicated. If you
>> >>> are
>> >>>>>>> not
>> >>>>>>>>> the intended recipient, please do not read, copy, use or disclose
>> >>> to
>> >>>>>>> others
>> >>>>>>>>> this message or any attachment. Please also notify the sender by
>> >>>>>>> replying
>> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
>> >>> the
>> >>>>>>> email
>> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
>> relate
>> >>>>> to
>> >>>>>>> the
>> >>>>>>>>> official business of this company shall be understood as neither
>> >>>>> given
>> >>>>>>> nor
>> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
>> >>> company
>> >>>>>>>>> registered in England and Wales, company number 04008957) and IG
>> >>>>> Index
>> >>>>>>>>> Limited (a company registered in England and Wales, company
>> number
>> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> >>>>> Hill,
>> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> >>>>> and IG
>> >>>>>>>>> Index Limited (register number 114059) are authorised and
>> regulated
>> >>>>> by
>> >>>>>>> the
>> >>>>>>>>> Financial Conduct Authority.
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>> The information contained in this email is strictly confidential and
>> for
>> >>> the use of the addressee only, unless otherwise indicated. If you are
>> not
>> >>> the intended recipient, please do not read, copy, use or disclose to
>> others
>> >>> this message or any attachment. Please also notify the sender by
>> replying
>> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
>> email
>> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
>> the
>> >>> official business of this company shall be understood as neither given
>> nor
>> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> >>> registered in England and Wales, company number 04008957) and IG Index
>> >>> Limited (a company registered in England and Wales, company number
>> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
>> IG
>> >>> Index Limited (register number 114059) are authorised and regulated by
>> the
>> >>> Financial Conduct Authority.
>> >>>
>> >
>> >
>>
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
--
Nacho - Ignacio Solis - isolis@igso.net
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jay Kreps <ja...@confluent.io>.
I'm neutral positive on adding headers conceptually. I'm -1 on a proposal
using ints for header names. I just don't think a global namespace with
integer names is an acceptable end-user interface given that our goal is to
encourage a federated ecosystem of header users. Not trying to be a pain in
the ass, I just think it would be a bit of a wart.
-Jay
On Thu, Feb 9, 2017 at 12:12 AM, Michael Pearce <Mi...@ig.com>
wrote:
> As we'd like to start work on this to make it into the next release in a
> few months.
>
> If we were to put this to the vote today who would actually give this a -1
> vote?
>
> Based on the number of identified usecases (some stronger than others)
>
> Also if we were to keep with int keys based on all the discussion thread
> for the pros vs cons who would give a -1 as it stands? Eg anyone believe it
> simply wouldn't work.
>
> If not so we make progress would it be sensible to put this to vote? So we
> can start work?
>
> Cheers
> Mike
>
>
> Sent using OWA for iPhone
> ________________________________________
> From: Jeroen van Disseldorp <je...@axual.io>
> Sent: Wednesday, February 1, 2017 2:43:45 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> I'm lagging a bit here too, but want to respond to the usefulness of
> this KIP for two use cases: bidirectional mirroring and smart cluster
> switches.
>
> Bidirectional mirroring
>
> For a Dutch bank we have currently implemented bidirectional mirroring
> using our own headers, which we encode in the Value using a custom
> serializer. It encodes just one byte extra, containing a "copy/original"
> bit. When applications serialize an avro object, the bit is unset by
> default. Our mirroring mechanism subscribes to all topics, checks the
> bit, and when unset the bit is set and the message is forwarded to the
> other DC.
>
> The overhead of this is pretty low (1 byte) and it's cheap, since the
> mirroring mechanism does not need to fully deserialize the value, but
> just check+flip one bit in the byte[] value. On the receiving end, the
> mirroring mechanism sees the message for the second time, but notices
> the message is a copy already and ignores it.
>
> The mechanism has been in production since last year and it works really
> well. Its downside however is the incompatibility with avro serialized
> values, as the whole organization is mandated to use our own custom
> serde and we were forced to create a wrapper client library around the
> default kafka client. The callback we offer to applications is very
> similar to that proposed in the KIP, as every message our users get
> consists of <Metadata, K, V>.
>
> Additionally our header contains a message id and serialization
> timestamp (invented before 0.10.0.0). These fields could also be done
> away with when this KIP is implemented.
>
> Smart cluster switches
>
> Another case we are considering to use headers for is enabling "smart
> cluster switches" for consumers and producers in a way that would avoid
> duplicates and/or missed messages in most cases. This involves encoding
> a producer id (app name or id), producer instance id (uuid generated
> upon producer's start) and sequence number in every message. We expect
> to use one sequence per partition.
>
> These fields enable our consumers to keep track of which messages were
> seen without relying on offsets, which is a cluster-bound value.
> Consumers keep track of seen messages by keeping a small administration
> of ranges of sequence numbers per producer instance and partition. Since
> sequence numbers are always increasing per producer and partition, the
> memory consumption of such administration is pretty low, while being
> meaningful across clusters. After every poll() we can commit the
> administration to a topic in much the same way as we do with offsets.
> The difference is that this topic can be mirrored to the other cluster
> and upon a switch of consumer, all instances can use the topic to
> initialize themselves and skip all messages already processed.
>
> Hope these views help in the upcoming decision on this KIP.
>
> Regards
> Jeroen
>
>
>
> On 09/01/2017 20:40, Gwen Shapira wrote:
> > Mirroring + ClusterIDs:
> >
> > Kafka clusters already have an automatically generated ID that is
> > exposed to clients:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
> >
> > It sounds trivial to get MirrorMaker's consumer to grab this ID (which
> > always exists) from the source cluster and add it to the message
> > headers.
> >
> > Obviously adding stuff to the message makes it larger so this should
> > be a feature you should be able to turn on/off depending on the
> > replication requirements. If you need the feature - you pay the price.
> > I don't think there is an immediate need to put an effort into
> > shrinking the Cluster ID, but others can comment on that.
> >
> >
> >
> >
> >
> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> >> Hey Michael,
> >>
> >> Sorry for lagging, I've been out for the holiday break. What would help
> me
> >> would be to get to a consolidated list of use cases we've worked out and
> >> think add real value. I think getting down to the details really matters
> >> for these.
> >>
> >> The two lists I'm aware of are the one in the KIP and the one you gave
> in
> >> this email thread. Both seem to be more at the sketch level. Let me
> review
> >> the use cases and see if I have it right. If so let's get it in the
> design
> >> proposal.
> >>
> >> Your use cases:
> >>
> >> 1. Tracing IDs -- I get this use case and think I see how it would
> work.
> >> I think because the tracing is from a third party system there is
> some
> >> value to having a header versus a custom format to make this work
> fully
> >> transparently.
> >> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
> >> thinking below
> >> 3. End-to-end encryption -- I think what you are saying is that if
> you
> >> want "routing' fields that are accessible in transit without
> decrypting you
> >> need a scheme to leave this decrypted.I think the idea is to avoid
> >> decrypting to get a header, but I'm not sure if I get how this is
> better
> >> than a custom message format that doesn't encrypt the things you
> don't want
> >> encrypted since you have to be aware of the scheme anyway.
> >> 4. EOS -- I'm not convinced this is a good approach for EOS.
> >>
> >> Linked from KIP:
> >>
> >> 1. Schema-Registry-aware avro serialization -- not clear how this is
> >> better than keeping the schema id in the value
> >> 2. Polyglot kafka topics -- marginally better than a container
> format,
> >> but of course you still have to all agree on the the header and
> >> header=>serialization mapping so not sure if it's really better
> than a
> >> container format. Also not wild about polyglot topics :-)
> >> 3. End-to-End routing trail including mirroring --- I'd put this
> under
> >> tracing ids
> >> 4. Cross-service process tracing -- ditto
> >> 5. Signing -- I like this use case but it seems to depend on a
> feature
> >> we don't have and aren't proposing adding, right?
> >> 6. Per-record custom compression -- why do you want this?
> compressing
> >> per message won't work well compared to batch compression, right?
> >> 7. Large Message Support -- I think we can agree that although this
> >> might work it's clearly not the right way to support large messages
> in Kafka
> >>
> >> Matthias
> >>
> >> 1. Control messages --- Matthias, not sure if I get the motivation
> in
> >> streams for this. If we did want to do it, wouldn't it be easier to
> >> implement and by allowing users to send control messages (in the
> EOS sense)
> >> and adding a consumer config to allow either passing these on or
> >> suppressing them?
> >>
> >> So I if I were to summarize I'd list the use cases as follows:
> >>
> >> - Much better with headers: tracing IDs
> >> - Somewhat better: end-to-end encryption, polyglot topics
> >> - Don't buy: Mirroring (though perhaps could be made to work with
> >> changes to the proposal), EOS, large messages, compression
> >> - Not sure: Control messages
> >>
> >> Does this match where other people are at? From my point of view for
> >> something like this where we are changing the core data model we should
> >> ideally get a handful of use cases to the point where they are really
> well
> >> thought out and we know they can use the feature. I think we have one
> solid
> >> one with tracing IDs, and I think maybe mirroring could be another with
> >> some work. I'm less sure about the others.
> >>
> >> Bi-directional Mirroring
> >> I think bidirectional mirroring could potentially become a compelling
> use
> >> case if we an find a way to make it work. If I understand it, the
> proposal
> >> is to add a header like "origin-cluster" with the cluster id of the
> Kafka
> >> cluster. Then modify the mirroring tool to disallow mirroring a message
> >> back to it's own origin cluster. This basic idea is definitely workable
> and
> >> useful, but can we really get to a good implementation using headers?
> >> I imagine the approach would be to add this header in the producer if
> some
> >> config is set. There are two problems:
> >>
> >> - You add a pretty big honking cluster ID to each message. For small
> >> messages this maybe larger than the contents itself.
> >> - What happens if someone somewhere forgets to set that config on
> some
> >> client somewhere?
> >>
> >> If someone forgets to set the config in some client somewhere I guess
> you
> >> have various options in mirror maker:
> >>
> >> 1. drop the data (pretty bad, now your data is out of sync)
> >> 2. crash (pretty bad--how do you recover)
> >> 3. insert some default (pretty bad since would also lead to data
> loss if
> >> that's not right)
> >>
> >> Basically although I think a cluster tag could be a good way to do
> >> bi-directional mirroring, I'm not sure this naive approach I described
> >> would get us to a usable implementation. This could be fixable. I think
> if
> >> there was some scheme to check required headers on the server that could
> >> alleviate the missing header problem. Not really sure what to do about
> the
> >> message bloat problem, but maybe there could be some way to shrink the
> >> cluster id?
> >>
> >> Cheers,
> >>
> >> -Jay
> >>
> >>
> >> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Michael.Pearce@ig.com
> >
> >> wrote:
> >>
> >>> @Jay, Ismael and a few others.
> >>>
> >>> Based on the discussion on kip-87, and the additional use case by
> Matthias.
> >>>
> >>> Just for clarity do we now have consensus on the earlier points (1)
> >>> headers are useful (2) headers should be native in Kafka?
> >>>
> >>> Cheers
> >>> Mike
> >>> ________________________________________
> >>> From: radai <ra...@gmail.com>
> >>> Sent: Monday, December 19, 2016 5:01 PM
> >>> To: dev@kafka.apache.org
> >>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>>
> >>> having said that im perfectly fine with something like a "no payload"
> flag
> >>> (as opposed to null payload), which would mean such a msg is not
> returned
> >>> from poll() unless the user code specifically "opts in" and would serve
> >>> only to carry headers (making it a control message)
> >>>
> >>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com>
> wrote:
> >>>
> >>>> if all you want is to tag "end of stream" wouldnt a header on the last
> >>> msg
> >>>> in the stream be enough?
> >>>>
> >>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> matthias@confluent.io>
> >>>> wrote:
> >>>>
> >>>>> The details about headers for control messages are still to define.
> But
> >>>>> yes, the idea is to have some common default behavior that clients
> would
> >>>>> need to implement.
> >>>>>
> >>>>> The point is, that "regular headers" add meta data to regular
> messages.
> >>>>> Thus, those messages will be returned to the user via .poll(). And
> after
> >>>>> the message is received the user can check if meta data is present
> and
> >>>>> read it.
> >>>>>
> >>>>> For control messages, we do not want those to pop up via .poll() as
> >>>>> those are no regular messages. A client would need to opt-in to see
> >>>>> those messages (either via poll() or maybe a callback). Thus, we need
> >>>>> some special (standardized) header IDs that indicate control messages
> >>>>> that should not be returned to the user via poll() by default.
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
> >>>>>> Matthias,
> >>>>>>
> >>>>>> Thanks for your input. I'm +1 on control messages as they seem to
> be
> >>>>> the
> >>>>>> simplest way to implement watermarks (
> >>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102
> ),
> >>> a
> >>>>>> feature that would add a lot of value to Kafka Streams IMHO.
> >>>>>>
> >>>>>> Your argument that the control-message indicator belongs in the
> >>>>> client-only
> >>>>>> section of the record format make sense. Just to make sure I
> >>>>> understand,
> >>>>>> are you suggesting that control messages would be indicated by a
> >>>>> standard
> >>>>>> reserved header? By standard, I mean that ALL Kafka consumers would
> >>>>> know
> >>>>>> to handle these messages differently (possibly just ignoring them).
> >>>>> This
> >>>>>> would need to be added to the specification of the consumer protocol
> >>> so
> >>>>>> that all Kafka clients implement it, right? I think it's a good
> idea
> >>>>> but
> >>>>>> just checking.
> >>>>>>
> >>>>>> Cheers,
> >>>>>>
> >>>>>> Roger
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> >>> matthias@confluent.io
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Yes and no. I did overload the term "control message".
> >>>>>>>
> >>>>>>> EOS control messages are for client-broker communication and thus
> >>> never
> >>>>>>> exposed to any application. And I think this is a good design
> because
> >>>>>>> broker needs to understand those control messages. Thus, this
> should
> >>> be
> >>>>>>> a protocol change.
> >>>>>>>
> >>>>>>> The type of control messages I have in mind are for client-client
> >>>>>>> (application-application) communication and the broker is agnostic
> to
> >>>>>>> them. Thus, it should not be a protocol change.
> >>>>>>>
> >>>>>>>
> >>>>>>> -Matthias
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 12/14/16 9:42 AM, radai wrote:
> >>>>>>>> arent control messages getting pushed as their own top level
> >>> protocol
> >>>>>>>> change (and a fairly massive one) for the transactions KIP ?
> >>>>>>>>
> >>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> >>>>> matthias@confluent.io>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> I want to add a completely new angle to this discussion. For
> this,
> >>> I
> >>>>>>>>> want to propose an extension for the headers feature that enables
> >>> new
> >>>>>>>>> uses cases -- and those new use cases might convince people to
> >>>>> support
> >>>>>>>>> headers (of course including the larger scoped proposal).
> >>>>>>>>>
> >>>>>>>>> Extended Proposal:
> >>>>>>>>>
> >>>>>>>>> Allow messages with a certain header key to be special "control
> >>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an
> application
> >>>>> via
> >>>>>>>>> .poll().
> >>>>>>>>>
> >>>>>>>>> Thus, a consumer client would automatically skip over those
> >>>>> messages. If
> >>>>>>>>> an application knows about embedded control messages, it can
> "sing
> >>>>> up"
> >>>>>>>>> to those messages by the consumer client and either get a
> callback
> >>> or
> >>>>>>>>> the consumer auto-drop for this messages gets disabled (allowing
> to
> >>>>>>>>> consumer those messages via poll()).
> >>>>>>>>>
> >>>>>>>>> (The details need further considerations/discussion. I just want
> to
> >>>>>>>>> sketch the main idea.)
> >>>>>>>>>
> >>>>>>>>> Usage:
> >>>>>>>>>
> >>>>>>>>> There is a shared topic (ie, used by multiple applications) and a
> >>>>>>>>> producer application wants to embed a special message in the
> topic
> >>>>> for a
> >>>>>>>>> dedicated consumer application. Because only one application will
> >>>>>>>>> understand this message, it cannot be a regular message as this
> >>> would
> >>>>>>>>> break all applications that do not understand this message. The
> >>>>> producer
> >>>>>>>>> application would set a special metadata key and no consumer
> >>>>> application
> >>>>>>>>> would see this control message by default because they did not
> >>> enable
> >>>>>>>>> their consumer client to return this message in poll() (and the
> >>>>> client
> >>>>>>>>> would just drop this message with special metadata key). Only the
> >>>>> single
> >>>>>>>>> application that should receive this message, will subscribe to
> >>> this
> >>>>>>>>> message on its consumer client and process it.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Concrete Use Case: Kafka Streams
> >>>>>>>>>
> >>>>>>>>> In Kafka Streams, we would like to propagate "control messages"
> >>> from
> >>>>>>>>> subtopology to subtopology. There are multiple scenarios for
> which
> >>>>> this
> >>>>>>>>> would be useful. For example, currently we do not guarantee a
> >>>>>>>>> "consistent shutdown" of an application. By this, I mean that
> input
> >>>>>>>>> records might not be completely processed by the whole topology
> >>>>> because
> >>>>>>>>> the application shutdown happens "in between" and an intermediate
> >>>>> result
> >>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user would
> see
> >>>>> an
> >>>>>>>>> committed offset of the source topic of the application, but no
> >>>>>>>>> corresponding result record in the output topic.
> >>>>>>>>>
> >>>>>>>>> Having "shutdown markers" would allow us, to first stop the
> >>> upstream
> >>>>>>>>> subtopology and write this marker into the intermediate topic and
> >>> the
> >>>>>>>>> downstream subtopology would only shut down itself after is sees
> >>> the
> >>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
> >>>>>>>>> "in-flight" messages got stuck in intermediate topics.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch
> Processing).
> >>>>>>>>> There was a discussion about the proposed metadata topic, and we
> >>>>> could
> >>>>>>>>> avoid this metadata topic if we would have "control messages".
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Right now, we cannot insert an "application control message"
> >>> because
> >>>>>>>>> Kafka Streams does not own all topics it read/writes and thus
> might
> >>>>>>>>> break other consumer application (as described above) if we
> inject
> >>>>>>>>> random messages that are not understood by other apps.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Of course, one can work around "embedded control messaged" by
> using
> >>>>> an
> >>>>>>>>> additional topic to propagate control messaged between
> application
> >>>>> (as
> >>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
> >>>>> there
> >>>>>>>>> are major concerns about adding this metadata topic in the KIP
> and
> >>>>> this
> >>>>>>>>> shows that other application that need a similar pattern might
> >>> profit
> >>>>>>>>> from topic embedded "control messages", too.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> One last important consideration: those "control messages" are
> used
> >>>>> for
> >>>>>>>>> client to client communication and are not understood by the
> >>> broker.
> >>>>>>>>> Thus, those messages should not be enabled within the message
> >>> format
> >>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
> >>>>> headers
> >>>>>>>>> would be a nice way to implement them. Because KIP-82 did
> consider
> >>>>> key
> >>>>>>>>> namespaces for metatdata keys, this extension should not be an
> own
> >>>>> KIP
> >>>>>>>>> but should be included in KIP-82 to reserve a namespace for
> >>> "control
> >>>>>>>>> message" in the first place.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Sorry for the long email... Looking forward to your feedback.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -Matthias
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> >>>>>>>>>> Hi Jun
> >>>>>>>>>>
> >>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
> >>>>> (HTTP/JMS/
> >>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
> >>> (though I
> >>>>>>>>> believe it contains the end2end uuid embedded in this id), on
> >>>>> receiving
> >>>>>>> the
> >>>>>>>>> message at the receiving JVM the apm code takes this out, and
> >>>>> continues
> >>>>>>> its
> >>>>>>>>> tracing on the that new thread. Both JVM's (and other languages
> the
> >>>>> APM
> >>>>>>>>> tool supports) send this data async back to the central
> controllers
> >>>>>>> where
> >>>>>>>>> the stiching togeather occurs. For this they need some header
> space
> >>>>> for
> >>>>>>>>> them to put this id.
> >>>>>>>>>> 101) Yes indeed we have a business transaction Id in the
> payload.
> >>>>>>> Though
> >>>>>>>>> this is a system level tracing, that we need to have marry up.
> Also
> >>>>> as
> >>>>>>> per
> >>>>>>>>> note on end2end encryption we'd be unable to prove the flow if
> the
> >>>>>>> payload
> >>>>>>>>> is encrypted as we'd not have access to this at certain points of
> >>> the
> >>>>>>> flow
> >>>>>>>>> through the infrastructure/platform.
> >>>>>>>>>>
> >>>>>>>>>> 103) As said we use this mechanism in IG very successfully, as
> >>>>> stated
> >>>>>>>>> per key we guarantee the transaction producing app to handle the
> >>>>>>>>> transaction of a key at one DC unless at point of critical
> failure
> >>>>>>> where we
> >>>>>>>>> have to flip processing to another. We care about key ordering.
> >>>>>>>>>> I disagree on the offset comment for the partition solution
> unless
> >>>>> you
> >>>>>>>>> do full ISR, or expensive full XA transactions even with
> partitions
> >>>>> you
> >>>>>>>>> cannot fully guarantee offsets would match.
> >>>>>>>>>> 105) Very much so, I need to have access at the platform level
> to
> >>>>> the
> >>>>>>>>> other meta data all mentioned, without having to need to have
> >>> access
> >>>>> to
> >>>>>>> the
> >>>>>>>>> encryption keys of the payload.
> >>>>>>>>>> 106)
> >>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to
> have
> >>> a
> >>>>>>>>> global producerId register which would be very hard to
> >>>>> enforce/ensure is
> >>>>>>>>> current and correct, just to understand the message origins of
> its
> >>>>>>>>> region/az/cluster for routing.
> >>>>>>>>>> The client wrapper version, producerId can be the same, as
> >>> obviously
> >>>>>>> the
> >>>>>>>>> producer could upgrade its wrapper, as such we need to know what
> >>>>> wrapper
> >>>>>>>>> version the message is created with.
> >>>>>>>>>> Likewise the IP address, as stated we can have our producer
> move,
> >>>>> where
> >>>>>>>>> its IP would change.
> >>>>>>>>>> 107)
> >>>>>>>>>> UUID is set on the message by interceptors before actual
> producer
> >>>>>>>>> transport send. This is for platform level message dedupe
> >>> guarantee,
> >>>>> the
> >>>>>>>>> business payload should be agnostic to this. Please see
> >>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
> >>>>> ection.html
> >>>>>>>>> note this is not touching business payloads.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi, Michael,
> >>>>>>>>>>
> >>>>>>>>>> Thanks for the reply. I find it very helpful.
> >>>>>>>>>>
> >>>>>>>>>> Data lineage:
> >>>>>>>>>> 100. I'd like to understand the APM use case a bit more. It
> >>>>> sounds
> >>>>>>>>> like
> >>>>>>>>>> that those APM plugins can generate a transaction id that
> we
> >>>>> could
> >>>>>>>>>> potentially put in the header of every message. How would
> you
> >>>>>>>>> typically
> >>>>>>>>>> make use of such transaction ids? Are there other metadata
> >>>>>>>>> associated with
> >>>>>>>>>> the transaction id and if so, how are they propagated
> >>>>> downstream?
> >>>>>>>>>> 101. For the finance use case, if the concept of
> transaction
> >>> is
> >>>>>>>>> important,
> >>>>>>>>>> wouldn't it be typically included in the message payload
> >>>>> instead of
> >>>>>>>>> as an
> >>>>>>>>>> optional header field?
> >>>>>>>>>>
> >>>>>>>>>> 102. The data lineage that Altas and Navigator support
> seems
> >>> to
> >>>>> be
> >>>>>>>>> at the
> >>>>>>>>>> dataset level, not per record level? So, not sure if per
> >>> message
> >>>>>>>>> headers
> >>>>>>>>>> are relevant there.
> >>>>>>>>>>
> >>>>>>>>>> Mirroring:
> >>>>>>>>>> 103. The benefit of using separate partitions is that it
> >>>>>>> potentially
> >>>>>>>>> makes
> >>>>>>>>>> it easy to preserve offsets during mirroring. This will
> make
> >>> it
> >>>>>>>>> easier for
> >>>>>>>>>> consumer to switch clusters. Currently, the consumers can
> >>> switch
> >>>>>>>>> clusters
> >>>>>>>>>> by using the timestampToOffset() api, but it has to deal
> with
> >>>>>>>>> duplicates.
> >>>>>>>>>> Good point on the issue with log compact and I am not sure
> how
> >>>>> to
> >>>>>>>>> address
> >>>>>>>>>> this. However, even if we mirror into the existing
> partitions,
> >>>>> the
> >>>>>>>>> ordering
> >>>>>>>>>> for messages generated from different clusters seems
> >>>>>>>>> non-deterministic
> >>>>>>>>>> anyway. So, it seems that the consumers already have to
> deal
> >>>>> with
> >>>>>>>>> that? If
> >>>>>>>>>> a topic is compacted, does that mean which messages are
> >>>>> preserved
> >>>>>>> is
> >>>>>>>>> also
> >>>>>>>>>> non-deterministic across clusters?
> >>>>>>>>>>
> >>>>>>>>>> 104. Good point on partition key.
> >>>>>>>>>>
> >>>>>>>>>> End-to-end encryption:
> >>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are
> headers
> >>>>>>>>> useful there?
> >>>>>>>>>> Auditing:
> >>>>>>>>>> 106. It seems other than the UUID, all other metadata are
> per
> >>>>>>>>> producer?
> >>>>>>>>>> EOS:
> >>>>>>>>>> 107. How are those UUIDs generated? I am not sure if they
> can
> >>> be
> >>>>>>>>> generated
> >>>>>>>>>> in the producer library. An application may send messages
> >>>>> through a
> >>>>>>>>> load
> >>>>>>>>>> balancer and on retry, the same message could be routed to
> a
> >>>>>>>>> different
> >>>>>>>>>> producer instance. So, it seems that the application has to
> >>>>>>> generate
> >>>>>>>>> the
> >>>>>>>>>> UUIDs. In that case, shouldn't the application just put the
> >>>>> UUID in
> >>>>>>>>> the
> >>>>>>>>>> payload?
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Jun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> >>>>>>>>> Michael.Pearce@ig.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> > Hi Jun.
> >>>>>>>>>> >
> >>>>>>>>>> > Per Transaction Tracing / Data Lineage.
> >>>>>>>>>> >
> >>>>>>>>>> > As Stated in the KIP this has the first use case of how
> many
> >>>>> APM
> >>>>>>>>> tools now
> >>>>>>>>>> > work.
> >>>>>>>>>> > I would find it impossible for any one to argue this is
> not
> >>>>>>>>> important or a
> >>>>>>>>>> > niche market as it has its own gartner report for this
> >>> space.
> >>>>>>> Such
> >>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular
> are
> >>>>> but a
> >>>>>>>>> few.
> >>>>>>>>>> >
> >>>>>>>>>> > Likewise these APM tools can help very rapidly track down
> >>>>> issues
> >>>>>>>>> and
> >>>>>>>>>> > automatically capture metrics, perform actions based on
> >>>>>>> unexpected
> >>>>>>>>> behavior
> >>>>>>>>>> > to auto recover services.
> >>>>>>>>>> >
> >>>>>>>>>> > Before mentioning looking at aggregated stats, in these
> >>> cases
> >>>>>>> where
> >>>>>>>>>> > actually on critical flows we cannot afford to have
> >>> aggregated
> >>>>>>>>> rolled up
> >>>>>>>>>> > stats only.
> >>>>>>>>>> >
> >>>>>>>>>> > With the APM tool we use its actually able to detect a
> >>> single
> >>>>>>>>> transaction
> >>>>>>>>>> > failure and capture the thread traces in the JVM where it
> >>>>> failed
> >>>>>>>>> and
> >>>>>>>>>> > everything for us, to the point it sends us alerts where
> we
> >>>>> have
> >>>>>>>>> this
> >>>>>>>>>> > giving the line number of the code that caused it, the
> >>>>>>> transaction
> >>>>>>>>> trace
> >>>>>>>>>> > through all the services and endpoints (supported) upto
> the
> >>>>> point
> >>>>>>>>> of
> >>>>>>>>>> > failure, it can also capture the data in and out (so we
> can
> >>>>>>>>> replay).
> >>>>>>>>>> > Because atm Kafka doesn't support us being able to stich
> in
> >>>>> these
> >>>>>>>>> tracing
> >>>>>>>>>> > transaction ids natively, we cannot get these benefits as
> >>>>> such is
> >>>>>>>>> limiting
> >>>>>>>>>> > our ability support apps and monitor them to the same
> >>>>> standards
> >>>>>>> we
> >>>>>>>>> come to
> >>>>>>>>>> > expect when on a kafka flow.
> >>>>>>>>>> >
> >>>>>>>>>> > This actually ties in with Data Lineage, as the same
> tracing
> >>>>> can
> >>>>>>>>> be used
> >>>>>>>>>> > to back stich this. Essentially many times due to the
> sums
> >>> of
> >>>>>>> money
> >>>>>>>>>> > involved there are disputes, and typically as a financial
> >>>>>>>>> institute the
> >>>>>>>>>> > easiest and cleanest way to prove when disputes arise is
> to
> >>>>>>>>> present the
> >>>>>>>>>> > actual flow and processes involved in a transaction.
> >>>>>>>>>> >
> >>>>>>>>>> > Likewise as Hadoop matures its evident this case is
> >>>>> important, as
> >>>>>>>>> tools
> >>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera
> led)
> >>>>> are
> >>>>>>>>> evident
> >>>>>>>>>> > also I believe the importance here is very much NOT just
> a
> >>>>>>>>> financial issue.
> >>>>>>>>>> >
> >>>>>>>>>> > From a MDM point of view any company wanting to care
> about
> >>>>> Data
> >>>>>>>>> Quality
> >>>>>>>>>> > and Data Governance - Data Lineage is a key piece in this
> >>>>> puzzle.
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > RE Mirroring,
> >>>>>>>>>> >
> >>>>>>>>>> > As per the KIP in-fact this is exactly what we do re
> cluster
> >>>>> id,
> >>>>>>>>> to mirror
> >>>>>>>>>> > a network of clusters between AZ's / Regions. We know a
> >>>>>>>>> transaction for a
> >>>>>>>>>> > key will be done within a AZ/Region, as such we know the
> >>>>> write
> >>>>>>> to
> >>>>>>>>> kafka
> >>>>>>>>>> > would be ordered per key. But we need eventual view of
> that
> >>>>>>> across
> >>>>>>>>> in our
> >>>>>>>>>> > other regions/az's. When we have complete AZ or Region
> >>>>> failure we
> >>>>>>>>> know
> >>>>>>>>>> > there will be a brief interruption whilst those
> transactions
> >>>>> are
> >>>>>>>>> moved to
> >>>>>>>>>> > another region but we expect after it to continue.
> >>>>>>>>>> >
> >>>>>>>>>> > As mentioned having separate Partions to do this starts
> to
> >>> get
> >>>>>>>>>> > ugly/complicated for us:
> >>>>>>>>>> > how would I do compaction where a key is in two
> partitions?
> >>>>>>>>>> > How do we balance consumers so where multiple partitions
> >>> with
> >>>>> the
> >>>>>>>>> same key
> >>>>>>>>>> > goto the same consumer
> >>>>>>>>>> > What do you do if cluster 1 has 5 partitions but cluster
> 20
> >>>>> has
> >>>>>>> 10
> >>>>>>>>> because
> >>>>>>>>>> > its larger kit in our more core DC's, as such key to
> >>> partition
> >>>>>>>>> mappings for
> >>>>>>>>>> > consumers get even more complicated.
> >>>>>>>>>> > What do you do if we add or remove a complete region
> >>>>>>>>>> >
> >>>>>>>>>> > Where as simple mirror will work we just need to ensure
> we
> >>>>> don't
> >>>>>>>>> have a
> >>>>>>>>>> > cycle which we can do with clusterId.
> >>>>>>>>>> >
> >>>>>>>>>> > We even have started to look at shortest path mirror
> routing
> >>>>>>> based
> >>>>>>>>> on
> >>>>>>>>>> > clusterId, if we also had the region and az info on the
> >>>>>>> originating
> >>>>>>>>>> > message, this we have not implemented but some ideas come
> >>> from
> >>>>>>>>> network
> >>>>>>>>>> > routing, and also the dispatcher router in apache qpid.
> >>>>>>>>>> >
> >>>>>>>>>> > Also we need to have data perimeters e.g. certain data
> >>> cannot
> >>>>>>> leave
> >>>>>>>>>> > certain countries borders. We want this all automated so
> >>> that
> >>>>> at
> >>>>>>>>> the
> >>>>>>>>>> > platform level without having to touch or look at the
> >>> business
> >>>>>>>>> data inside
> >>>>>>>>>> > we can have headers we can put tags into so that we can
> >>> ensure
> >>>>>>>>> this doesn't
> >>>>>>>>>> > occur when we mirror. (actually links in to data lineage
> /
> >>>>>>> tracing
> >>>>>>>>> as again
> >>>>>>>>>> > we need to tag messages at a platform level) Examples
> are we
> >>>>> are
> >>>>>>>>> not
> >>>>>>>>>> > allowed Private customer details to leave Switzerland,
> yet
> >>> we
> >>>>>>> need
> >>>>>>>>> those
> >>>>>>>>>> > systems integrated.
> >>>>>>>>>> >
> >>>>>>>>>> > Lastly around mirroring we have a partionKey field, as
> the
> >>> key
> >>>>>>>>> used for
> >>>>>>>>>> > portioning logic != compaction key all the time but we
> want
> >>> to
> >>>>>>>>> preserve it
> >>>>>>>>>> > for when we mirror so that if source cluster partition
> count
> >>>>> !=
> >>>>>>>>> destination
> >>>>>>>>>> > cluster partition count we can honour the same
> partitioning
> >>>>>>> logic.
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > RE End 2 End encryption
> >>>>>>>>>> >
> >>>>>>>>>> > As I believe mentioned just before, the solution you
> mention
> >>>>> just
> >>>>>>>>> doesn't
> >>>>>>>>>> > cut the mustard these days with many regulators. An
> >>> operations
> >>>>>>>>> person with
> >>>>>>>>>> > access to the box should not be able to have access to
> the
> >>>>> data.
> >>>>>>>>> Many now
> >>>>>>>>>> > actually impose quite literally the implementation
> expected
> >>>>> being
> >>>>>>>>> end2end
> >>>>>>>>>> > encryption for certain data (Singapore for us is one
> that I
> >>> am
> >>>>>>>>> most aware
> >>>>>>>>>> > of). In fact we're even now needing encrypt the data and
> >>> store
> >>>>>>> the
> >>>>>>>>> keys in
> >>>>>>>>>> > HSM modules.
> >>>>>>>>>> >
> >>>>>>>>>> > Likewise the performance penalty on encrypting
> decrypting as
> >>>>> you
> >>>>>>>>> produce
> >>>>>>>>>> > over wire, then again encrypt decrypt as the data is
> stored
> >>> on
> >>>>>>> the
> >>>>>>>>> brokers
> >>>>>>>>>> > disks and back again, then again encrypted and decrypted
> >>> back
> >>>>>>> over
> >>>>>>>>> the wire
> >>>>>>>>>> > each time for each consumer all adds up, ignoring this
> >>>>> doubling
> >>>>>>>>> with mirror
> >>>>>>>>>> > makers etc. simply encrypting the value once on write by
> the
> >>>>>>>>> client and
> >>>>>>>>>> > again decrypting on consume by the consumer is far more
> >>>>>>>>> performant, but
> >>>>>>>>>> > then the routing and platform meta data needs to be
> separate
> >>>>>>> (thus
> >>>>>>>>> headers)
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > RE Auditing:
> >>>>>>>>>> >
> >>>>>>>>>> > Our Auditing needs are:
> >>>>>>>>>> > Producer Id,
> >>>>>>>>>> > Origin Cluster Id that message first produced into
> >>>>>>>>>> > Origin AZ - agreed we can derive this if we have cluster
> id,
> >>>>> but
> >>>>>>>>> it makes
> >>>>>>>>>> > resolving this for audit reporting a lot easier.
> >>>>>>>>>> > Origin Region - agreed we can derive this if we have
> cluster
> >>>>> id,
> >>>>>>>>> but it
> >>>>>>>>>> > makes resolving this for audit reporting a lot easier.
> >>>>>>>>>> > Unique Message Identification (this is not the same as
> >>>>>>> transaction
> >>>>>>>>>> > tracing) - note offset and partition are not the same, as
> >>>>> when we
> >>>>>>>>> mirror or
> >>>>>>>>>> > have for what ever system failure duplicate send,
> >>>>>>>>>> > Custom Client wrapper version (where organizations have
> to
> >>>>> wrap
> >>>>>>>>> the kafka
> >>>>>>>>>> > client for added features) so we know what version of the
> >>>>> wrapper
> >>>>>>>>> is used
> >>>>>>>>>> > Producer IP address (in case of clients being in our
> vm/open
> >>>>>>> stack
> >>>>>>>>> infra
> >>>>>>>>>> > where they can move around, producer id will stay the
> same
> >>> but
> >>>>>>>>> this would
> >>>>>>>>>> > change)
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > RE Once and only once delivery case
> >>>>>>>>>> >
> >>>>>>>>>> > Using the same Message UUID for auditing we can achieve
> this
> >>>>>>> quite
> >>>>>>>>> simply.
> >>>>>>>>>> >
> >>>>>>>>>> > As per how some other brokers do this (cough qpid,
> artemis)
> >>>>>>>>> message uuid
> >>>>>>>>>> > are used to dedupe where message is sent and produced but
> >>> the
> >>>>>>>>> client didn't
> >>>>>>>>>> > receive the ack, and there for replays the send, by
> having a
> >>>>>>>>> unique message
> >>>>>>>>>> > id per message, this can be filtered out, on consumers
> where
> >>>>>>>>> message
> >>>>>>>>>> > delivery may occur twice for what ever reasons a message
> >>> uuid
> >>>>> can
> >>>>>>>>> be used
> >>>>>>>>>> > to remove duplicates being deliverd , like wise we can do
> >>>>> this in
> >>>>>>>>> the
> >>>>>>>>>> > mirrormakers so if we detect a dupe message we can avoid
> >>>>>>>>> replicating it.
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > Cheers
> >>>>>>>>>> > Mike
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
> wrote:
> >>>>>>>>>> >
> >>>>>>>>>> > Since this KIP affects message format, wire protocol,
> >>>>> apis, I
> >>>>>>>>> think
> >>>>>>>>>> > it's
> >>>>>>>>>> > worth spending a bit more time to nail down the
> concrete
> >>>>> use
> >>>>>>>>> cases. It
> >>>>>>>>>> > would be bad if we add this feature, but when start
> >>>>>>>>> implementing it
> >>>>>>>>>> > for say
> >>>>>>>>>> > mirroring, we then realize that header is not the
> best
> >>>>>>>>> approach.
> >>>>>>>>>> > Initially,
> >>>>>>>>>> > I thought I was convinced of the use cases of headers
> >>> and
> >>>>> was
> >>>>>>>>> trying to
> >>>>>>>>>> > write down a few use cases to convince others. That's
> >>>>> when I
> >>>>>>>>> became
> >>>>>>>>>> > less
> >>>>>>>>>> > certain. For me to be convinced, I just want to see
> two
> >>>>>>> strong
> >>>>>>>>> use
> >>>>>>>>>> > cases
> >>>>>>>>>> > (instead of 10 maybe use cases) in the third-party
> >>> space.
> >>>>> The
> >>>>>>>>> reason is
> >>>>>>>>>> > that when we discussed the use cases within a
> company,
> >>>>> often
> >>>>>>>>> it ends
> >>>>>>>>>> > with
> >>>>>>>>>> > "we can't force everyone to use this standard since
> we
> >>> may
> >>>>>>>>> have to
> >>>>>>>>>> > integrate with third-party tools".
> >>>>>>>>>> >
> >>>>>>>>>> > At present, I am not sure why headers are useful for
> >>>>> things
> >>>>>>>>> like
> >>>>>>>>>> > schemaId
> >>>>>>>>>> > or encryption. In order to do anything useful to the
> >>>>> value,
> >>>>>>>>> one needs
> >>>>>>>>>> > to
> >>>>>>>>>> > know the schemaId or how data is encrypted, but
> header
> >>> is
> >>>>>>>>> optional.
> >>>>>>>>>> > But, I
> >>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
> >>> provides
> >>>>>>> more
> >>>>>>>>> details
> >>>>>>>>>> > on
> >>>>>>>>>> > the argument.
> >>>>>>>>>> >
> >>>>>>>>>> > I am not very sure header is the best approach for
> >>>>> mirroring
> >>>>>>>>> either. If
> >>>>>>>>>> > someone has thought about this more, I'd be happy to
> >>> hear.
> >>>>>>>>>> >
> >>>>>>>>>> > I can see the data lineage use case. I am just not
> sure
> >>>>> how
> >>>>>>>>> widely
> >>>>>>>>>> > applicable this is. If someone familiar with this
> space
> >>>>> can
> >>>>>>>>> justify
> >>>>>>>>>> > this is
> >>>>>>>>>> > a significant use case, say in the finance industry,
> >>> this
> >>>>>>>>> would be a
> >>>>>>>>>> > strong
> >>>>>>>>>> > use case.
> >>>>>>>>>> >
> >>>>>>>>>> > I can see the auditing use case. I am just not sure
> if a
> >>>>>>> native
> >>>>>>>>>> > producer id
> >>>>>>>>>> > solves that problem. If there are additional metadata
> >>>>> that's
> >>>>>>>>> worth
> >>>>>>>>>> > collecting but not covered by the producer id, that
> >>> would
> >>>>>>> make
> >>>>>>>>> this a
> >>>>>>>>>> > strong use case.
> >>>>>>>>>> >
> >>>>>>>>>> > Thanks,
> >>>>>>>>>> >
> >>>>>>>>>> > Jun
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>>>>>>>>> > wrote:
> >>>>>>>>>> >
> >>>>>>>>>> > > this KIP is about enabling headers, nothing more
> >>> nothing
> >>>>>>>>> less - so
> >>>>>>>>>> > no,
> >>>>>>>>>> > > broker-side use of headers is not in the KIP scope.
> >>>>>>>>>> > >
> >>>>>>>>>> > > obviously though, once you have headers potential
> use
> >>>>> cases
> >>>>>>>>> could
> >>>>>>>>>> > include
> >>>>>>>>>> > > broker-side header-aware interceptors (which would
> be
> >>>>> the
> >>>>>>>>> topic of
> >>>>>>>>>> > other
> >>>>>>>>>> > > future KIPs).
> >>>>>>>>>> > >
> >>>>>>>>>> > > a trivially clear use case (to me) would be using
> such
> >>>>>>>>> broker-side
> >>>>>>>>>> > > interceptors to enforce compliance with
> organizational
> >>>>>>>>> policies - it
> >>>>>>>>>> > would
> >>>>>>>>>> > > make our SREs lives much easier if instead of
> >>>>> retroactively
> >>>>>>>>>> > discovering
> >>>>>>>>>> > > "rogue" topics/users those messages would have been
> >>>>>>> rejected
> >>>>>>>>>> > up-front.
> >>>>>>>>>> > >
> >>>>>>>>>> > > the kafka broker code is lacking any such
> >>> extensibility
> >>>>>>>>> support
> >>>>>>>>>> > (beyond
> >>>>>>>>>> > > maybe authorizer) which is why these use cases were
> >>> left
> >>>>>>> out
> >>>>>>>>> of the
> >>>>>>>>>> > "case
> >>>>>>>>>> > > for headers" doc - broker extensibility is a
> separate
> >>>>>>>>> discussion.
> >>>>>>>>>> > >
> >>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
> >>>>>>>>> gwen@confluent.io>
> >>>>>>>>>> > wrote:
> >>>>>>>>>> > >
> >>>>>>>>>> > > > Woah, I wasn't aware this is something we'll do.
> It
> >>>>>>> wasn't
> >>>>>>>>> in the
> >>>>>>>>>> > KIP,
> >>>>>>>>>> > > > right?
> >>>>>>>>>> > > >
> >>>>>>>>>> > > > I guess we could do it the same way ACLs
> currently
> >>>>> work.
> >>>>>>>>>> > > > I had in mind something that will allow admins to
> >>>>> apply
> >>>>>>>>> rules to
> >>>>>>>>>> > the
> >>>>>>>>>> > > > new create/delete/config topic APIs. So Todd can
> >>>>> decide
> >>>>>>> to
> >>>>>>>>> reject
> >>>>>>>>>> > > > "create topic" requests that ask for more than 40
> >>>>>>>>> partitions, or
> >>>>>>>>>> > > > require exactly 3 replicas, or no more than 50GB
> >>>>>>> partition
> >>>>>>>>> size,
> >>>>>>>>>> > etc.
> >>>>>>>>>> > > >
> >>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are planning
> to
> >>>>> apply
> >>>>>>>>> more
> >>>>>>>>>> > rules
> >>>>>>>>>> > > > to requests (and I think we should), we may want
> a
> >>> bit
> >>>>>>>>> more generic
> >>>>>>>>>> > > > design around that.
> >>>>>>>>>> > > >
> >>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>>>>>>>>> > > wrote:
> >>>>>>>>>> > > > > "wouldn't you be in the business of making sure
> >>>>>>> everyone
> >>>>>>>>> uses
> >>>>>>>>>> > them
> >>>>>>>>>> > > > > properly?"
> >>>>>>>>>> > > > >
> >>>>>>>>>> > > > > thats where a broker-side plugin would come
> handy
> >>> -
> >>>>> any
> >>>>>>>>> incoming
> >>>>>>>>>> > > message
> >>>>>>>>>> > > > > that does not conform to org policy (read -
> does
> >>> not
> >>>>>>>>> have the
> >>>>>>>>>> > proper
> >>>>>>>>>> > > > > headers) gets thrown out (with an error
> returned
> >>> to
> >>>>>>> user)
> >>>>>>>>>> > > > >
> >>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
> >>>>>>>>> tpalino@gmail.com>
> >>>>>>>>>> > wrote:
> >>>>>>>>>> > > > >
> >>>>>>>>>> > > > >> Come on, I've done at least 2 talks on this
> one
> >>> :)
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >> Producing counts to a topic is part of it, but
> >>>>> that's
> >>>>>>>>> only
> >>>>>>>>>> > part. So
> >>>>>>>>>> > > you
> >>>>>>>>>> > > > >> count you have 100 messages in topic A. When
> you
> >>>>>>> mirror
> >>>>>>>>> topic A
> >>>>>>>>>> > to
> >>>>>>>>>> > > > another
> >>>>>>>>>> > > > >> cluster, you have 99 messages. Where was your
> >>>>> problem?
> >>>>>>>>> Or
> >>>>>>>>>> > worse, you
> >>>>>>>>>> > > > have
> >>>>>>>>>> > > > >> 100 messages, but one producer duplicated
> >>> messages
> >>>>> and
> >>>>>>>>> another
> >>>>>>>>>> > one
> >>>>>>>>>> > > lost
> >>>>>>>>>> > > > >> messages. You need details about where the
> >>> message
> >>>>>>> came
> >>>>>>>>> from in
> >>>>>>>>>> > order
> >>>>>>>>>> > > to
> >>>>>>>>>> > > > >> pinpoint problems when they happen. Source
> >>> producer
> >>>>>>>>> info, where
> >>>>>>>>>> > it was
> >>>>>>>>>> > > > >> produced into your infrastructure, and when it
> >>> was
> >>>>>>>>> produced.
> >>>>>>>>>> > This
> >>>>>>>>>> > > > requires
> >>>>>>>>>> > > > >> you to add the information to the message.
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >> And yes, you still need to maintain your
> clients.
> >>>>> So
> >>>>>>>>> maybe my
> >>>>>>>>>> > original
> >>>>>>>>>> > > > >> example was not the best. My thoughts on not
> >>>>> wanting
> >>>>>>> to
> >>>>>>>>> be
> >>>>>>>>>> > responsible
> >>>>>>>>>> > > > for
> >>>>>>>>>> > > > >> message formats stands, because that's very
> much
> >>>>>>>>> separate from
> >>>>>>>>>> > the
> >>>>>>>>>> > > > client.
> >>>>>>>>>> > > > >> As you know, we have our own internal client
> >>>>> library
> >>>>>>>>> that can
> >>>>>>>>>> > insert
> >>>>>>>>>> > > the
> >>>>>>>>>> > > > >> right headers, and right now inserts the right
> >>>>> audit
> >>>>>>>>>> > information into
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> message fields. If they exist, and assuming
> the
> >>>>>>> message
> >>>>>>>>> is Avro
> >>>>>>>>>> > > encoded.
> >>>>>>>>>> > > > >> What if someone wants to use JSON instead for
> a
> >>>>> good
> >>>>>>>>> reason?
> >>>>>>>>>> > What if
> >>>>>>>>>> > > > user X
> >>>>>>>>>> > > > >> wants to encrypt messages, but user Y does
> not?
> >>>>>>>>> Maintaining the
> >>>>>>>>>> > client
> >>>>>>>>>> > > > >> library is still much easier than maintaining
> the
> >>>>>>>>> message
> >>>>>>>>>> > formats.
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >> -Todd
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
> >>>>>>>>> gwen@confluent.io
> >>>>>>>>>> > >
> >>>>>>>>>> > > wrote:
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >> > Based on your last sentence, consider me
> >>>>> convinced
> >>>>>>> :)
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >> > I get why headers are critical for Mirroring
> >>> (you
> >>>>>>>>> need tags to
> >>>>>>>>>> > > prevent
> >>>>>>>>>> > > > >> > loops and sometimes to route messages to the
> >>>>> correct
> >>>>>>>>>> > destination).
> >>>>>>>>>> > > > >> > But why do you need headers to audit? We are
> >>>>>>> auditing
> >>>>>>>>> by
> >>>>>>>>>> > producing
> >>>>>>>>>> > > > >> > counts to a side topic (and I was under the
> >>>>>>>>> impression you do
> >>>>>>>>>> > the
> >>>>>>>>>> > > > >> > same), so we never need to modify the
> message.
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >> > Another thing - after we added headers,
> >>> wouldn't
> >>>>> you
> >>>>>>>>> be in the
> >>>>>>>>>> > > > >> > business of making sure everyone uses them
> >>>>> properly?
> >>>>>>>>> Making
> >>>>>>>>>> > sure
> >>>>>>>>>> > > > >> > everyone includes the right headers you
> need,
> >>> not
> >>>>>>>>> using the
> >>>>>>>>>> > header
> >>>>>>>>>> > > > >> > names you intend to use, etc. I don't think
> the
> >>>>>>>>> "policing"
> >>>>>>>>>> > business
> >>>>>>>>>> > > > >> > will ever go away.
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino
> <
> >>>>>>>>>> > tpalino@gmail.com>
> >>>>>>>>>> > > > wrote:
> >>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
> >>> with
> >>>>> the
> >>>>>>>>>> > workaround.
> >>>>>>>>>> > > Avro
> >>>>>>>>>> > > > >> means
> >>>>>>>>>> > > > >> > > that I have to be concerned with the
> format
> >>> of
> >>>>> the
> >>>>>>>>> messages
> >>>>>>>>>> > in
> >>>>>>>>>> > > > order to
> >>>>>>>>>> > > > >> > run
> >>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring,
> etc.).
> >>>>> That
> >>>>>>>>> means
> >>>>>>>>>> > that I
> >>>>>>>>>> > > have
> >>>>>>>>>> > > > to
> >>>>>>>>>> > > > >> > > handle the schemas, and I have to enforce
> >>> rules
> >>>>>>>>> about good
> >>>>>>>>>> > > formats.
> >>>>>>>>>> > > > >> This
> >>>>>>>>>> > > > >> > is
> >>>>>>>>>> > > > >> > > not something I want to be in the business
> >>> of,
> >>>>>>>>> because I
> >>>>>>>>>> > should be
> >>>>>>>>>> > > > able
> >>>>>>>>>> > > > >> > to
> >>>>>>>>>> > > > >> > > run a service infrastructure without
> needing
> >>>>> to be
> >>>>>>>>> in the
> >>>>>>>>>> > weeds of
> >>>>>>>>>> > > > >> > dealing
> >>>>>>>>>> > > > >> > > with customer data formats.
> >>>>>>>>>> > > > >> > >
> >>>>>>>>>> > > > >> > > Trust me, a sizable portion of my support
> >>> time
> >>>>> is
> >>>>>>>>> spent
> >>>>>>>>>> > dealing
> >>>>>>>>>> > > with
> >>>>>>>>>> > > > >> > schema
> >>>>>>>>>> > > > >> > > issues. I really would like to get away
> from
> >>>>> that.
> >>>>>>>>> Maybe
> >>>>>>>>>> > I'd have
> >>>>>>>>>> > > > more
> >>>>>>>>>> > > > >> > time
> >>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
> >>>>>>>>>> > > > >> > >
> >>>>>>>>>> > > > >> > > -Todd
> >>>>>>>>>> > > > >> > >
> >>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
> Shapira <
> >>>>>>>>>> > gwen@confluent.io>
> >>>>>>>>>> > > > wrote:
> >>>>>>>>>> > > > >> > >
> >>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
> >>>>> workarounds
> >>>>>>>>> (Avro
> >>>>>>>>>> > container
> >>>>>>>>>> > > > >> > >> format), so I'm not too excited about the
> >>>>> extra
> >>>>>>>>> work
> >>>>>>>>>> > required to
> >>>>>>>>>> > > do
> >>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't
> mind it
> >>>>> if
> >>>>>>>>> you do
> >>>>>>>>>> > it...
> >>>>>>>>>> > > > >> > >> I think the Apache convention for "good
> >>> idea,
> >>>>> but
> >>>>>>>>> not
> >>>>>>>>>> > willing to
> >>>>>>>>>> > > > put
> >>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway,
> that's
> >>>>> what
> >>>>>>> I
> >>>>>>>>> was
> >>>>>>>>>> > trying to
> >>>>>>>>>> > > > >> > >> convey :)
> >>>>>>>>>> > > > >> > >>
> >>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
> Palino
> >>> <
> >>>>>>>>>> > tpalino@gmail.com>
> >>>>>>>>>> > > > >> wrote:
> >>>>>>>>>> > > > >> > >> > Well I guess my question for you,
> then, is
> >>>>> what
> >>>>>>>>> is
> >>>>>>>>>> > holding you
> >>>>>>>>>> > > > back
> >>>>>>>>>> > > > >> > from
> >>>>>>>>>> > > > >> > >> > full support for headers? What's the
> bit
> >>>>> that
> >>>>>>>>> you're
> >>>>>>>>>> > missing
> >>>>>>>>>> > > that
> >>>>>>>>>> > > > >> has
> >>>>>>>>>> > > > >> > you
> >>>>>>>>>> > > > >> > >> > under a full +1?
> >>>>>>>>>> > > > >> > >> >
> >>>>>>>>>> > > > >> > >> > -Todd
> >>>>>>>>>> > > > >> > >> >
> >>>>>>>>>> > > > >> > >> >
> >>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
> >>>>> Shapira <
> >>>>>>>>>> > > gwen@confluent.io>
> >>>>>>>>>> > > > >> > wrote:
> >>>>>>>>>> > > > >> > >> >
> >>>>>>>>>> > > > >> > >> >> I know why people who support headers
> >>>>> support
> >>>>>>>>> them, and
> >>>>>>>>>> > I've
> >>>>>>>>>> > > > seen
> >>>>>>>>>> > > > >> > what
> >>>>>>>>>> > > > >> > >> >> the discussion is like.
> >>>>>>>>>> > > > >> > >> >>
> >>>>>>>>>> > > > >> > >> >> This is why I'm asking people who are
> >>>>> against
> >>>>>>>>> headers
> >>>>>>>>>> > > > (especially
> >>>>>>>>>> > > > >> > >> >> committers) what will make them change
> >>>>> their
> >>>>>>>>> mind - so
> >>>>>>>>>> > we can
> >>>>>>>>>> > > > get
> >>>>>>>>>> > > > >> > this
> >>>>>>>>>> > > > >> > >> >> part over one way or another.
> >>>>>>>>>> > > > >> > >> >>
> >>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at
> Radai,
> >>>>> Jun
> >>>>>>>>> or you
> >>>>>>>>>> > (Todd)...
> >>>>>>>>>> > > > I am
> >>>>>>>>>> > > > >> > >> >> just looking for something concrete we
> >>> can
> >>>>> do
> >>>>>>>>> to move
> >>>>>>>>>> > the
> >>>>>>>>>> > > > >> discussion
> >>>>>>>>>> > > > >> > >> >> along to the yummy design details
> (which
> >>> is
> >>>>>>> the
> >>>>>>>>>> > argument I
> >>>>>>>>>> > > > really
> >>>>>>>>>> > > > >> am
> >>>>>>>>>> > > > >> > >> >> looking forward to).
> >>>>>>>>>> > > > >> > >> >>
> >>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
> >>>>> Palino <
> >>>>>>>>>> > > tpalino@gmail.com>
> >>>>>>>>>> > > > >> > wrote:
> >>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even
> though
> >>>>> I'm
> >>>>>>>>> not a
> >>>>>>>>>> > > > committer)...
> >>>>>>>>>> > > > >> > >> >> >
> >>>>>>>>>> > > > >> > >> >> > I have always been a strong
> supporter
> >>> of
> >>>>>>>>> introducing
> >>>>>>>>>> > the
> >>>>>>>>>> > > > concept
> >>>>>>>>>> > > > >> > of an
> >>>>>>>>>> > > > >> > >> >> > envelope to messages, which headers
> >>>>>>>>> accomplishes. The
> >>>>>>>>>> > > message
> >>>>>>>>>> > > > key
> >>>>>>>>>> > > > >> > is
> >>>>>>>>>> > > > >> > >> >> > already an example of a piece of
> >>> envelope
> >>>>>>>>>> > information. By
> >>>>>>>>>> > > > >> > providing a
> >>>>>>>>>> > > > >> > >> >> means
> >>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself, and
> not
> >>>>>>>>> relying on
> >>>>>>>>>> > use-case
> >>>>>>>>>> > > > >> > specific
> >>>>>>>>>> > > > >> > >> >> > implementations, you make it much
> >>> easier
> >>>>> for
> >>>>>>>>>> > components to
> >>>>>>>>>> > > > >> > >> interoperate.
> >>>>>>>>>> > > > >> > >> >> It
> >>>>>>>>>> > > > >> > >> >> > simplifies development of all these
> >>>>> things
> >>>>>>>>> (message
> >>>>>>>>>> > routing,
> >>>>>>>>>> > > > >> > auditing,
> >>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each one
> does
> >>>>> not
> >>>>>>>>> have to
> >>>>>>>>>> > reinvent
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> wheel.
> >>>>>>>>>> > > > >> > >> >> >
> >>>>>>>>>> > > > >> > >> >> > It also makes it much easier from a
> >>>>> client
> >>>>>>>>> point of
> >>>>>>>>>> > view if
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> headers
> >>>>>>>>>> > > > >> > >> >> are
> >>>>>>>>>> > > > >> > >> >> > defined as part of the protocol
> and/or
> >>>>>>>>> message format
> >>>>>>>>>> > in
> >>>>>>>>>> > > > general
> >>>>>>>>>> > > > >> > >> because
> >>>>>>>>>> > > > >> > >> >> > you can easily produce and consume
> >>>>> messages
> >>>>>>>>> without
> >>>>>>>>>> > having
> >>>>>>>>>> > > to
> >>>>>>>>>> > > > >> take
> >>>>>>>>>> > > > >> > >> into
> >>>>>>>>>> > > > >> > >> >> > account specific cases. For
> example, I
> >>>>> want
> >>>>>>>>> to route
> >>>>>>>>>> > > messages,
> >>>>>>>>>> > > > >> but
> >>>>>>>>>> > > > >> > >> >> client A
> >>>>>>>>>> > > > >> > >> >> > doesn't support the way audit
> >>> implemented
> >>>>>>>>> headers, and
> >>>>>>>>>> > > client
> >>>>>>>>>> > > > B
> >>>>>>>>>> > > > >> > >> doesn't
> >>>>>>>>>> > > > >> > >> >> > support the way encryption or
> routing
> >>>>>>>>> implemented
> >>>>>>>>>> > headers,
> >>>>>>>>>> > > so
> >>>>>>>>>> > > > now
> >>>>>>>>>> > > > >> > my
> >>>>>>>>>> > > > >> > >> >> > application has to create some
> really
> >>>>>>> fragile
> >>>>>>>>> (my
> >>>>>>>>>> > > autocorrect
> >>>>>>>>>> > > > >> just
> >>>>>>>>>> > > > >> > >> tried
> >>>>>>>>>> > > > >> > >> >> to
> >>>>>>>>>> > > > >> > >> >> > make that "tragic", which is
> probably
> >>>>>>>>> appropriate
> >>>>>>>>>> > too) code
> >>>>>>>>>> > > to
> >>>>>>>>>> > > > >> > strip
> >>>>>>>>>> > > > >> > >> >> > everything off, rather than just
> >>>>> consuming
> >>>>>>> the
> >>>>>>>>>> > messages,
> >>>>>>>>>> > > > picking
> >>>>>>>>>> > > > >> > out
> >>>>>>>>>> > > > >> > >> the
> >>>>>>>>>> > > > >> > >> >> 1
> >>>>>>>>>> > > > >> > >> >> > or 2 headers it's interested in, and
> >>>>>>>>> performing its
> >>>>>>>>>> > > function.
> >>>>>>>>>> > > > >> > >> >> >
> >>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has been
> >>> going
> >>>>> on
> >>>>>>>>> for a
> >>>>>>>>>> > long time,
> >>>>>>>>>> > > > and
> >>>>>>>>>> > > > >> > it's
> >>>>>>>>>> > > > >> > >> >> > always "Oh, you came up with 2 use
> >>> cases,
> >>>>>>> and
> >>>>>>>>> yeah,
> >>>>>>>>>> > those
> >>>>>>>>>> > > use
> >>>>>>>>>> > > > >> cases
> >>>>>>>>>> > > > >> > >> are
> >>>>>>>>>> > > > >> > >> >> > real things that someone would want
> to
> >>>>> do.
> >>>>>>>>> Here's an
> >>>>>>>>>> > > alternate
> >>>>>>>>>> > > > >> way
> >>>>>>>>>> > > > >> > to
> >>>>>>>>>> > > > >> > >> >> > implement them so let's not do
> >>> headers."
> >>>>> If
> >>>>>>>>> we have a
> >>>>>>>>>> > few
> >>>>>>>>>> > > use
> >>>>>>>>>> > > > >> cases
> >>>>>>>>>> > > > >> > >> that
> >>>>>>>>>> > > > >> > >> >> we
> >>>>>>>>>> > > > >> > >> >> > actually came up with, you can be
> sure
> >>>>> that
> >>>>>>>>> over the
> >>>>>>>>>> > next
> >>>>>>>>>> > > year
> >>>>>>>>>> > > > >> > >> there's a
> >>>>>>>>>> > > > >> > >> >> > dozen others that we didn't think of
> >>> that
> >>>>>>>>> someone
> >>>>>>>>>> > would like
> >>>>>>>>>> > > > to
> >>>>>>>>>> > > > >> > do. I
> >>>>>>>>>> > > > >> > >> >> > really think it's time to stop
> >>> rehashing
> >>>>>>> this
> >>>>>>>>>> > discussion and
> >>>>>>>>>> > > > >> > instead
> >>>>>>>>>> > > > >> > >> >> focus
> >>>>>>>>>> > > > >> > >> >> > on a workable standard that we can
> >>> adopt.
> >>>>>>>>>> > > > >> > >> >> >
> >>>>>>>>>> > > > >> > >> >> > -Todd
> >>>>>>>>>> > > > >> > >> >> >
> >>>>>>>>>> > > > >> > >> >> >
> >>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
> >>>>> Palino
> >>>>>>> <
> >>>>>>>>>> > > > tpalino@gmail.com>
> >>>>>>>>>> > > > >> > >> wrote:
> >>>>>>>>>> > > > >> > >> >> >
> >>>>>>>>>> > > > >> > >> >> >> C. per message encryption
> >>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach is
> that
> >>>>> this
> >>>>>>>>>> > significantly
> >>>>>>>>>> > > > reduce
> >>>>>>>>>> > > > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression,
> which
> >>>>>>> happens
> >>>>>>>>> on a
> >>>>>>>>>> > set of
> >>>>>>>>>> > > > >> > serialized
> >>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to
> enable
> >>>>> SSL
> >>>>>>>>> for wire
> >>>>>>>>>> > > > encryption
> >>>>>>>>>> > > > >> and
> >>>>>>>>>> > > > >> > >> rely
> >>>>>>>>>> > > > >> > >> >> on
> >>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS)
> for at
> >>>>> rest
> >>>>>>>>>> > encryption.
> >>>>>>>>>> > > > >> > >> >> >>
> >>>>>>>>>> > > > >> > >> >> >>
> >>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
> >>> this
> >>>>>>> does
> >>>>>>>>> cover
> >>>>>>>>>> > the case
> >>>>>>>>>> > > > of
> >>>>>>>>>> > > > >> > >> removing
> >>>>>>>>>> > > > >> > >> >> a
> >>>>>>>>>> > > > >> > >> >> >> drive from the system, it will not
> >>>>> satisfy
> >>>>>>>>> most
> >>>>>>>>>> > compliance
> >>>>>>>>>> > > > >> > >> requirements
> >>>>>>>>>> > > > >> > >> >> for
> >>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever has
> >>>>> access to
> >>>>>>>>> the
> >>>>>>>>>> > broker
> >>>>>>>>>> > > itself
> >>>>>>>>>> > > > >> > still
> >>>>>>>>>> > > > >> > >> has
> >>>>>>>>>> > > > >> > >> >> >> access to the unencrypted data. For
> >>>>>>>>> end-to-end
> >>>>>>>>>> > encryption
> >>>>>>>>>> > > you
> >>>>>>>>>> > > > >> > need to
> >>>>>>>>>> > > > >> > >> >> >> encrypt at the producer, before it
> >>>>> enters
> >>>>>>> the
> >>>>>>>>>> > system, and
> >>>>>>>>>> > > > >> decrypt
> >>>>>>>>>> > > > >> > at
> >>>>>>>>>> > > > >> > >> the
> >>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the
> system.
> >>>>>>>>>> > > > >> > >> >> >>
> >>>>>>>>>> > > > >> > >> >> >> -Todd
> >>>>>>>>>> > > > >> > >> >> >>
> >>>>>>>>>> > > > >> > >> >> >>
> >>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM,
> radai
> >>> <
> >>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
> >>>>>>>>>> > > > >> > >
> >>>>>>>>>> > > > >> > >> >> wrote:
> >>>>>>>>>> > > > >> > >> >> >>
> >>>>>>>>>> > > > >> > >> >> >>> another big plus of headers in the
> >>>>>>> protocol
> >>>>>>>>> is that
> >>>>>>>>>> > it
> >>>>>>>>>> > > would
> >>>>>>>>>> > > > >> > enable
> >>>>>>>>>> > > > >> > >> >> rapid
> >>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of core
> >>>>> kafka
> >>>>>>>>> and would
> >>>>>>>>>> > reduce
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> number of
> >>>>>>>>>> > > > >> > >> >> >>> future wire format changes
> required.
> >>>>>>>>>> > > > >> > >> >> >>>
> >>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
> >>>>>>> represents
> >>>>>>>>> use
> >>>>>>>>>> > cases that
> >>>>>>>>>> > > > are
> >>>>>>>>>> > > > >> > not
> >>>>>>>>>> > > > >> > >> 100%
> >>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and some of
> >>> them
> >>>>>>>>> require
> >>>>>>>>>> > rather
> >>>>>>>>>> > > > invasive
> >>>>>>>>>> > > > >> > wire
> >>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a good
> >>> recent
> >>>>>>>>> example of
> >>>>>>>>>> > this is
> >>>>>>>>>> > > > >> > kip-98.
> >>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected
> to
> >>> be
> >>>>> a
> >>>>>>>>> very small
> >>>>>>>>>> > > > fraction of
> >>>>>>>>>> > > > >> > >> total
> >>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes are
> >>>>> invasive.
> >>>>>>>>>> > > > >> > >> >> >>>
> >>>>>>>>>> > > > >> > >> >> >>> every such wire format change
> >>>>> translates
> >>>>>>>>> into
> >>>>>>>>>> > painful and
> >>>>>>>>>> > > > slow
> >>>>>>>>>> > > > >> > >> >> adoption of
> >>>>>>>>>> > > > >> > >> >> >>> new versions.
> >>>>>>>>>> > > > >> > >> >> >>>
> >>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
> >>>>> currently
> >>>>>>> in
> >>>>>>>>> KIPs
> >>>>>>>>>> > could be
> >>>>>>>>>> > > > "spun
> >>>>>>>>>> > > > >> > out"
> >>>>>>>>>> > > > >> > >> >> and
> >>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
> >>>>> transmitting
> >>>>>>>>> data over
> >>>>>>>>>> > > > headers.
> >>>>>>>>>> > > > >> > this
> >>>>>>>>>> > > > >> > >> >> would
> >>>>>>>>>> > > > >> > >> >> >>> keep the core wire format
> stable(r),
> >>>>> core
> >>>>>>>>> codebase
> >>>>>>>>>> > > smaller,
> >>>>>>>>>> > > > and
> >>>>>>>>>> > > > >> > >> avoid
> >>>>>>>>>> > > > >> > >> >> the
> >>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
> >>>>> required
> >>>>>>>>> to prove
> >>>>>>>>>> > a
> >>>>>>>>>> > > > certain
> >>>>>>>>>> > > > >> > >> feature
> >>>>>>>>>> > > > >> > >> >> is
> >>>>>>>>>> > > > >> > >> >> >>> useful enough for a wide-enough
> >>>>> audience
> >>>>>>> to
> >>>>>>>>> warrant
> >>>>>>>>>> > a wire
> >>>>>>>>>> > > > >> format
> >>>>>>>>>> > > > >> > >> >> change
> >>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
> >>>>>>>>>> > > > >> > >> >> >>>
> >>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
> >>>>> "mere"
> >>>>>>>>> wire format
> >>>>>>>>>> > > changes
> >>>>>>>>>> > > > >> and
> >>>>>>>>>> > > > >> > im
> >>>>>>>>>> > > > >> > >> not
> >>>>>>>>>> > > > >> > >> >> >>> saying it could have been
> completely
> >>>>> done
> >>>>>>>>> with
> >>>>>>>>>> > headers,
> >>>>>>>>>> > > but
> >>>>>>>>>> > > > >> > >> >> exactly-once
> >>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
> >>>>>>>>>> > > > >> > >> >> >>>
> >>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM,
> Gwen
> >>>>>>>>> Shapira <
> >>>>>>>>>> > > > >> gwen@confluent.io
> >>>>>>>>>> > > > >> > >
> >>>>>>>>>> > > > >> > >> >> wrote:
> >>>>>>>>>> > > > >> > >> >> >>>
> >>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
> >>>>> radai <
> >>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
> >>>>> organization,
> >>>>>>>>> one could
> >>>>>>>>>> > > always
> >>>>>>>>>> > > > use
> >>>>>>>>>> > > > >> > >> other
> >>>>>>>>>> > > > >> > >> >> >>> > > approaches such as
> company-wise
> >>>>>>>>> containers"
> >>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
> >>>>>>> traditionally
> >>>>>>>>> done
> >>>>>>>>>> > but there
> >>>>>>>>>> > > > are
> >>>>>>>>>> > > > >> > now
> >>>>>>>>>> > > > >> > >> >> cases
> >>>>>>>>>> > > > >> > >> >> >>> > (read
> >>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is not
> >>>>>>> acceptable.
> >>>>>>>>> this
> >>>>>>>>>> > makes
> >>>>>>>>>> > > > headers
> >>>>>>>>>> > > > >> > >> useful
> >>>>>>>>>> > > > >> > >> >> even
> >>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for cases
> >>> where
> >>>>>>>>>> > > > one-container-fits-all
> >>>>>>>>>> > > > >> > cannot
> >>>>>>>>>> > > > >> > >> >> >>> apply.
> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use
> cases
> >>>>>>> listed,
> >>>>>>>>> i dont
> >>>>>>>>>> > want
> >>>>>>>>>> > > > this to
> >>>>>>>>>> > > > >> > >> devolve
> >>>>>>>>>> > > > >> > >> >> >>> to a
> >>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular use
> >>> cases
> >>>>> - i
> >>>>>>>>> think its
> >>>>>>>>>> > > enough
> >>>>>>>>>> > > > >> that
> >>>>>>>>>> > > > >> > >> some
> >>>>>>>>>> > > > >> > >> >> of
> >>>>>>>>>> > > > >> > >> >> >>> them
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > I think a main point of
> contention
> >>> is
> >>>>>>>>> that: We
> >>>>>>>>>> > > identified
> >>>>>>>>>> > > > few
> >>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are
> useful,
> >>>>> do
> >>>>>>> we
> >>>>>>>>> want
> >>>>>>>>>> > Kafka to
> >>>>>>>>>> > > > be a
> >>>>>>>>>> > > > >> > >> system
> >>>>>>>>>> > > > >> > >> >> >>> > that supports those use-cases?
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
> >>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
> >>>>> record-level
> >>>>>>>>> lineage
> >>>>>>>>>> > is
> >>>>>>>>>> > > though
> >>>>>>>>>> > > > >> > since
> >>>>>>>>>> > > > >> > >> the
> >>>>>>>>>> > > > >> > >> >> >>> > overhead could
> >>>>>>>>>> > > > >> > >> >> >>> > be significant."
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record
> level
> >>>>>>>>> lineage. I
> >>>>>>>>>> > don't
> >>>>>>>>>> > > think
> >>>>>>>>>> > > > it
> >>>>>>>>>> > > > >> > was
> >>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think it
> is
> >>>>> safe
> >>>>>>> to
> >>>>>>>>> assume
> >>>>>>>>>> > that
> >>>>>>>>>> > > the
> >>>>>>>>>> > > > NSA
> >>>>>>>>>> > > > >> > >> needed
> >>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also know
> >>> that
> >>>>>>>>> certain
> >>>>>>>>>> > financial
> >>>>>>>>>> > > > >> > institutes
> >>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
> >>> records
> >>>>> at
> >>>>>>> a
> >>>>>>>>> record
> >>>>>>>>>> > level
> >>>>>>>>>> > > and
> >>>>>>>>>> > > > >> > there
> >>>>>>>>>> > > > >> > >> are
> >>>>>>>>>> > > > >> > >> >> >>> > federal regulations that
> absolutely
> >>>>>>>>> require
> >>>>>>>>>> > this. They
> >>>>>>>>>> > > > also
> >>>>>>>>>> > > > >> > need
> >>>>>>>>>> > > > >> > >> to
> >>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
> >>>>> "touches"
> >>>>>>> the
> >>>>>>>>>> > messages and
> >>>>>>>>>> > > > >> either
> >>>>>>>>>> > > > >> > >> reads
> >>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't have
> >>>>>>> possibly
> >>>>>>>>>> > modified the
> >>>>>>>>>> > > > >> payload
> >>>>>>>>>> > > > >> > >> >> itself.
> >>>>>>>>>> > > > >> > >> >> >>> > They use record level
> encryption to
> >>>>> do
> >>>>>>>>> that -
> >>>>>>>>>> > apps can
> >>>>>>>>>> > > > read
> >>>>>>>>>> > > > >> and
> >>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
> >>> can't
> >>>>>>>>> touch the
> >>>>>>>>>> > payload.
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those are
> >>> corner
> >>>>>>>>> cases and
> >>>>>>>>>> > not worth
> >>>>>>>>>> > > > >> adding
> >>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they
> should
> >>>>> use a
> >>>>>>>>> different
> >>>>>>>>>> > > pubsub
> >>>>>>>>>> > > > >> > message
> >>>>>>>>>> > > > >> > >> for
> >>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the other
> 1000
> >>>>> that
> >>>>>>>>> cater
> >>>>>>>>>> > > > specifically
> >>>>>>>>>> > > > >> to
> >>>>>>>>>> > > > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> > financial industry).
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a catch
> 22:
> >>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific
> use-case,
> >>>>>>>>> someone can
> >>>>>>>>>> > always
> >>>>>>>>>> > > say
> >>>>>>>>>> > > > it
> >>>>>>>>>> > > > >> > isn't
> >>>>>>>>>> > > > >> > >> >> >>> > interesting enough for Kafka.
> If we
> >>>>>>>>> discuss more
> >>>>>>>>>> > general
> >>>>>>>>>> > > > >> > trends,
> >>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we are not
> >>> sure
> >>>>>>> any
> >>>>>>>>> of them
> >>>>>>>>>> > really
> >>>>>>>>>> > > > >> needs
> >>>>>>>>>> > > > >> > >> >> headers
> >>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just hand
> >>>>> waving
> >>>>>>>>> and not
> >>>>>>>>>> > > > interesting.".
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > I think discussing use-cases in
> >>>>>>> specifics
> >>>>>>>>> is super
> >>>>>>>>>> > > > important
> >>>>>>>>>> > > > >> to
> >>>>>>>>>> > > > >> > >> >> decide
> >>>>>>>>>> > > > >> > >> >> >>> > implementation details for
> headers
> >>>>> (my
> >>>>>>>>> use-cases
> >>>>>>>>>> > lean
> >>>>>>>>>> > > > toward
> >>>>>>>>>> > > > >> > >> >> numerical
> >>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and object
> >>>>> values,
> >>>>>>>>> others
> >>>>>>>>>> > differ),
> >>>>>>>>>> > > > but I
> >>>>>>>>>> > > > >> > >> think
> >>>>>>>>>> > > > >> > >> >> we
> >>>>>>>>>> > > > >> > >> >> >>> > need to answer the general "Are
> we
> >>>>> going
> >>>>>>>>> to have
> >>>>>>>>>> > > headers"
> >>>>>>>>>> > > > >> > question
> >>>>>>>>>> > > > >> > >> >> >>> > first.
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the other
> >>>>>>>>> committers in the
> >>>>>>>>>> > > > discussion:
> >>>>>>>>>> > > > >> > >> >> >>> > What would it take to convince
> you
> >>>>> that
> >>>>>>>>> headers
> >>>>>>>>>> > in Kafka
> >>>>>>>>>> > > > are
> >>>>>>>>>> > > > >> a
> >>>>>>>>>> > > > >> > >> good
> >>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can move
> >>> ahead
> >>>>>>> and
> >>>>>>>>> try to
> >>>>>>>>>> > agree
> >>>>>>>>>> > > on
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> >> details?
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving the
> goal
> >>>>>>> posts
> >>>>>>>>> and
> >>>>>>>>>> > this is
> >>>>>>>>>> > > > truly
> >>>>>>>>>> > > > >> > >> >> exhausting.
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly support
> >>>>> adding
> >>>>>>>>> headers
> >>>>>>>>>> > to Kafka
> >>>>>>>>>> > > > >> > (+0.5?).
> >>>>>>>>>> > > > >> > >> >> >>> > The community can continue to
> find
> >>>>>>>>> workarounds to
> >>>>>>>>>> > the
> >>>>>>>>>> > > > issue
> >>>>>>>>>> > > > >> and
> >>>>>>>>>> > > > >> > >> there
> >>>>>>>>>> > > > >> > >> >> >>> > are some benefits to keeping the
> >>>>> message
> >>>>>>>>> format
> >>>>>>>>>> > and
> >>>>>>>>>> > > > clients
> >>>>>>>>>> > > > >> > >> simpler.
> >>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of
> headers
> >>>>> to
> >>>>>>>>> many
> >>>>>>>>>> > use-cases
> >>>>>>>>>> > > and
> >>>>>>>>>> > > > if
> >>>>>>>>>> > > > >> we
> >>>>>>>>>> > > > >> > >> can
> >>>>>>>>>> > > > >> > >> >> >>> > find a good and generally useful
> >>> way
> >>>>> to
> >>>>>>>>> add it to
> >>>>>>>>>> > Kafka,
> >>>>>>>>>> > > > it
> >>>>>>>>>> > > > >> > will
> >>>>>>>>>> > > > >> > >> make
> >>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
> >>> worthy
> >>>>>>>>> goal in my
> >>>>>>>>>> > eyes.
> >>>>>>>>>> > > > >> > >> >> >>> >
> >>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
> >>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use
> cases
> >>>>> for
> >>>>>>>>> polyglot
> >>>>>>>>>> > topics.
> >>>>>>>>>> > > > >> > >> especially if
> >>>>>>>>>> > > > >> > >> >> >>> kafka
> >>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
> >>> something
> >>>>>>> else.
> >>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would make
> it
> >>>>>>> harder
> >>>>>>>>> to write
> >>>>>>>>>> > > > portable
> >>>>>>>>>> > > > >> > >> consumer
> >>>>>>>>>> > > > >> > >> >> >>> code.
> >>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would mess
> >>> with
> >>>>>>>>> locality of
> >>>>>>>>>> > > > consumption
> >>>>>>>>>> > > > >> > >> >> >>> guarantees.
> >>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
> >>>>>>>>> lineage/metadata is
> >>>>>>>>>> > > > >> > >> billing/chargeback.
> >>>>>>>>>> > > > >> > >> >> for
> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough to
> >>> simply
> >>>>>>>>> record the
> >>>>>>>>>> > point
> >>>>>>>>>> > > of
> >>>>>>>>>> > > > >> > origin,
> >>>>>>>>>> > > > >> > >> but
> >>>>>>>>>> > > > >> > >> >> >>> every
> >>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think mirror
> >>>>> maker)
> >>>>>>>>> must also
> >>>>>>>>>> > add a
> >>>>>>>>>> > > > >> record
> >>>>>>>>>> > > > >> > to
> >>>>>>>>>> > > > >> > >> >> form a
> >>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing on
> top
> >>> of
> >>>>>>>>> kafka - i
> >>>>>>>>>> > know
> >>>>>>>>>> > > samza
> >>>>>>>>>> > > > >> has a
> >>>>>>>>>> > > > >> > >> >> metadata
> >>>>>>>>>> > > > >> > >> >> >>> > map
> >>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
> >>>>> addition to
> >>>>>>>>> user
> >>>>>>>>>> > values.
> >>>>>>>>>> > > > headers
> >>>>>>>>>> > > > >> > are
> >>>>>>>>>> > > > >> > >> the
> >>>>>>>>>> > > > >> > >> >> >>> > perfect
> >>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50
> PM,
> >>>>> Jun
> >>>>>>>>> Rao <
> >>>>>>>>>> > > > jun@confluent.io
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >> > >> wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the first
> two
> >>>>>>>>> questions, it
> >>>>>>>>>> > would
> >>>>>>>>>> > > be
> >>>>>>>>>> > > > >> > helpful
> >>>>>>>>>> > > > >> > >> >> if we
> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use
> cases
> >>>>> for
> >>>>>>>>> headers
> >>>>>>>>>> > in the
> >>>>>>>>>> > > > space
> >>>>>>>>>> > > > >> > for
> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases
> within an
> >>>>>>>>> organization,
> >>>>>>>>>> > one
> >>>>>>>>>> > > > could
> >>>>>>>>>> > > > >> > always
> >>>>>>>>>> > > > >> > >> >> use
> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as
> company-wise
> >>>>>>>>> containers to
> >>>>>>>>>> > get
> >>>>>>>>>> > > > around
> >>>>>>>>>> > > > >> w/o
> >>>>>>>>>> > > > >> > >> >> >>> headers. I
> >>>>>>>>>> > > > >> > >> >> >>> > >> went through the use cases in
> >>> the
> >>>>> KIP
> >>>>>>>>> and in
> >>>>>>>>>> > Radai's
> >>>>>>>>>> > > > wiki
> >>>>>>>>>> > > > >> (
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> https://cwiki.apache.org/confl
> >>>>>>>>>> > uence/display/KAFKA/A+
> >>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> >>>>>>>>>> > > > >> > >> >> >>> > >> ).
> >>>>>>>>>> > > > >> > >> >> >>> > >> The following are the ones
> that
> >>>>> that
> >>>>>>> I
> >>>>>>>>>> > understand and
> >>>>>>>>>> > > > >> could
> >>>>>>>>>> > > > >> > be
> >>>>>>>>>> > > > >> > >> in
> >>>>>>>>>> > > > >> > >> >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case
> category.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
> >>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
> >>>>>>> content-type
> >>>>>>>>> should
> >>>>>>>>>> > be set
> >>>>>>>>>> > > at
> >>>>>>>>>> > > > >> the
> >>>>>>>>>> > > > >> > >> topic
> >>>>>>>>>> > > > >> > >> >> >>> level.
> >>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages
> with
> >>>>>>>>> different
> >>>>>>>>>> > content
> >>>>>>>>>> > > > types
> >>>>>>>>>> > > > >> > >> should be
> >>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
> >>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
> >>> useless
> >>>>>>>>> without
> >>>>>>>>>> > schema id,
> >>>>>>>>>> > > it
> >>>>>>>>>> > > > >> > seems
> >>>>>>>>>> > > > >> > >> that
> >>>>>>>>>> > > > >> > >> >> >>> > storing
> >>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together with
> >>>>>>> serialized
> >>>>>>>>> bytes
> >>>>>>>>>> > in the
> >>>>>>>>>> > > > value
> >>>>>>>>>> > > > >> is
> >>>>>>>>>> > > > >> > >> >> better?
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
> >>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this
> approach is
> >>>>> that
> >>>>>>>>> this
> >>>>>>>>>> > > > significantly
> >>>>>>>>>> > > > >> > reduce
> >>>>>>>>>> > > > >> > >> >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
> >>>>> which
> >>>>>>>>> happens on
> >>>>>>>>>> > a set
> >>>>>>>>>> > > of
> >>>>>>>>>> > > > >> > >> serialized
> >>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative is
> to
> >>>>> enable
> >>>>>>>>> SSL for
> >>>>>>>>>> > wire
> >>>>>>>>>> > > > >> > encryption
> >>>>>>>>>> > > > >> > >> and
> >>>>>>>>>> > > > >> > >> >> >>> rely
> >>>>>>>>>> > > > >> > >> >> >>> > on
> >>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g.
> LUKS)
> >>>>> for at
> >>>>>>>>> rest
> >>>>>>>>>> > > encryption.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
> >>> across
> >>>>>>>>> Kafka
> >>>>>>>>>> > clusters
> >>>>>>>>>> > > > >> > >> >> >>> > >> This is actually interesting.
> >>>>> Today,
> >>>>>>>>> to avoid
> >>>>>>>>>> > > > introducing
> >>>>>>>>>> > > > >> > >> cycles
> >>>>>>>>>> > > > >> > >> >> when
> >>>>>>>>>> > > > >> > >> >> >>> > doing
> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data
> centers,
> >>> one
> >>>>>>>>> would
> >>>>>>>>>> > either have
> >>>>>>>>>> > > to
> >>>>>>>>>> > > > >> set
> >>>>>>>>>> > > > >> > up
> >>>>>>>>>> > > > >> > >> two
> >>>>>>>>>> > > > >> > >> >> >>> Kafka
> >>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
> >>>>> aggregate)
> >>>>>>>>> per data
> >>>>>>>>>> > center
> >>>>>>>>>> > > or
> >>>>>>>>>> > > > >> > rename
> >>>>>>>>>> > > > >> > >> >> topics.
> >>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
> headers,
> >>>>> the
> >>>>>>>>> producer
> >>>>>>>>>> > could
> >>>>>>>>>> > > tag
> >>>>>>>>>> > > > >> each
> >>>>>>>>>> > > > >> > >> >> message
> >>>>>>>>>> > > > >> > >> >> >>> with
> >>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID in
> the
> >>>>>>> header.
> >>>>>>>>>> > MirrorMaker
> >>>>>>>>>> > > > could
> >>>>>>>>>> > > > >> > then
> >>>>>>>>>> > > > >> > >> >> avoid
> >>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a
> cluster
> >>> if
> >>>>>>>>> they are
> >>>>>>>>>> > tagged
> >>>>>>>>>> > > with
> >>>>>>>>>> > > > >> the
> >>>>>>>>>> > > > >> > >> same
> >>>>>>>>>> > > > >> > >> >> >>> cluster
> >>>>>>>>>> > > > >> > >> >> >>> > >> id.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative
> approach
> >>>>> is
> >>>>>>> to
> >>>>>>>>>> > introduce sth
> >>>>>>>>>> > > > like
> >>>>>>>>>> > > > >> > >> >> >>> hierarchical
> >>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages from
> >>>>>>> different
> >>>>>>>>>> > clusters in
> >>>>>>>>>> > > > >> > different
> >>>>>>>>>> > > > >> > >> >> >>> partitions
> >>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic. This
> >>>>> approach
> >>>>>>>>> avoids
> >>>>>>>>>> > filtering
> >>>>>>>>>> > > > out
> >>>>>>>>>> > > > >> > >> unneeded
> >>>>>>>>>> > > > >> > >> >> >>> data
> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving
> easier
> >>> to
> >>>>>>>>> support. It
> >>>>>>>>>> > may
> >>>>>>>>>> > > make
> >>>>>>>>>> > > > >> > >> compaction
> >>>>>>>>>> > > > >> > >> >> >>> > trickier
> >>>>>>>>>> > > > >> > >> >> >>> > >> though since the same key may
> >>>>> show up
> >>>>>>>>> in
> >>>>>>>>>> > different
> >>>>>>>>>> > > > >> > partitions.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
> >>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source
> connector
> >>>>> could
> >>>>>>>>> store in
> >>>>>>>>>> > the
> >>>>>>>>>> > > > message
> >>>>>>>>>> > > > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> metadata
> >>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
> >>> record.
> >>>>>>>>> Similarly,
> >>>>>>>>>> > if a
> >>>>>>>>>> > > > stream
> >>>>>>>>>> > > > >> job
> >>>>>>>>>> > > > >> > >> >> >>> transforms
> >>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to
> topic
> >>> B,
> >>>>> the
> >>>>>>>>> library
> >>>>>>>>>> > could
> >>>>>>>>>> > > > >> include
> >>>>>>>>>> > > > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> source
> >>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each of the
> >>>>>>>>> transformed
> >>>>>>>>>> > message in
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> header.
> >>>>>>>>>> > > > >> > >> >> Not
> >>>>>>>>>> > > > >> > >> >> >>> > sure
> >>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful
> record-level
> >>>>>>> lineage
> >>>>>>>>> is
> >>>>>>>>>> > though
> >>>>>>>>>> > > since
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> >> overhead
> >>>>>>>>>> > > > >> > >> >> >>> > could
> >>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> >>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
> >>>>>>>>> clientId/host/user in
> >>>>>>>>>> > the
> >>>>>>>>>> > > > header
> >>>>>>>>>> > > > >> in
> >>>>>>>>>> > > > >> > >> each
> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These metadata
> are
> >>>>>>>>> really at the
> >>>>>>>>>> > > producer
> >>>>>>>>>> > > > >> > level
> >>>>>>>>>> > > > >> > >> >> though.
> >>>>>>>>>> > > > >> > >> >> >>> > So, a
> >>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to only
> >>>>>>> include a
> >>>>>>>>>> > "producerId"
> >>>>>>>>>> > > > per
> >>>>>>>>>> > > > >> > >> message
> >>>>>>>>>> > > > >> > >> >> and
> >>>>>>>>>> > > > >> > >> >> >>> > send
> >>>>>>>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
> >>> mapping
> >>>>>>>>>> > independently.
> >>>>>>>>>> > > > KIP-98
> >>>>>>>>>> > > > >> is
> >>>>>>>>>> > > > >> > >> >> actually
> >>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such a
> >>>>> producerId
> >>>>>>>>> natively
> >>>>>>>>>> > in the
> >>>>>>>>>> > > > >> > message.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that
> I
> >>> am
> >>>>>>> fully
> >>>>>>>>>> > convinced of
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > strong
> >>>>>>>>>> > > > >> > >> >> >>> > third-party
> >>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
> >>> Perhaps
> >>>>> we
> >>>>>>>>> could
> >>>>>>>>>> > discuss a
> >>>>>>>>>> > > > bit
> >>>>>>>>>> > > > >> > more
> >>>>>>>>>> > > > >> > >> to
> >>>>>>>>>> > > > >> > >> >> make
> >>>>>>>>>> > > > >> > >> >> >>> > one
> >>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing use
> >>>>> cases.
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal question
> is
> >>>>>>>>> whether header
> >>>>>>>>>> > should
> >>>>>>>>>> > > > be
> >>>>>>>>>> > > > >> > >> exposed
> >>>>>>>>>> > > > >> > >> >> in
> >>>>>>>>>> > > > >> > >> >> >>> > stream
> >>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
> >>>>> stream,
> >>>>>>>>> Samza,
> >>>>>>>>>> > and
> >>>>>>>>>> > > Spark
> >>>>>>>>>> > > > >> > >> streaming.
> >>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems just
> >>> deal
> >>>>>>> with
> >>>>>>>>>> > key/value
> >>>>>>>>>> > > > pairs.
> >>>>>>>>>> > > > >> > >> Should we
> >>>>>>>>>> > > > >> > >> >> >>> > expose a
> >>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there too
> or
> >>>>>>>>> somehow map
> >>>>>>>>>> > header to
> >>>>>>>>>> > > > key
> >>>>>>>>>> > > > >> or
> >>>>>>>>>> > > > >> > >> >> value?
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> Jun
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35
> AM,
> >>>>>>>>> Michael
> >>>>>>>>>> > Pearce <
> >>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> >>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a
> period
> >>>>> of a
> >>>>>>>>> week,
> >>>>>>>>>> > that there
> >>>>>>>>>> > > > is
> >>>>>>>>>> > > > >> no
> >>>>>>>>>> > > > >> > >> >> concerns
> >>>>>>>>>> > > > >> > >> >> >>> now
> >>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and
> now
> >>> we
> >>>>>>> have
> >>>>>>>>>> > agreement that
> >>>>>>>>>> > > > >> > headers
> >>>>>>>>>> > > > >> > >> are
> >>>>>>>>>> > > > >> > >> >> >>> useful
> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
> >>> put
> >>>>> to
> >>>>>>> a
> >>>>>>>>> KIP
> >>>>>>>>>> > vote, this
> >>>>>>>>>> > > > >> > wouldn't
> >>>>>>>>>> > > > >> > >> be
> >>>>>>>>>> > > > >> > >> >> a
> >>>>>>>>>> > > > >> > >> >> >>> > reason
> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > @
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
> >>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of
> getting
> >>>>> this
> >>>>>>>>> KIP
> >>>>>>>>>> > moving past
> >>>>>>>>>> > > > >> this,
> >>>>>>>>>> > > > >> > we
> >>>>>>>>>> > > > >> > >> can
> >>>>>>>>>> > > > >> > >> >> >>> state
> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
> >>> that
> >>>>>>> can
> >>>>>>>>> will be
> >>>>>>>>>> > > > naturally
> >>>>>>>>>> > > > >> > >> >> interpreted
> >>>>>>>>>> > > > >> > >> >> >>> as
> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is
> later
> >>>>>>>>> wanted you can
> >>>>>>>>>> > > easily
> >>>>>>>>>> > > > >> split
> >>>>>>>>>> > > > >> > >> this
> >>>>>>>>>> > > > >> > >> >> >>> into
> >>>>>>>>>> > > > >> > >> >> >>> > two
> >>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the
> wire
> >>>>>>> protocol
> >>>>>>>>>> > > implementation
> >>>>>>>>>> > > > >> this
> >>>>>>>>>> > > > >> > >> makes
> >>>>>>>>>> > > > >> > >> >> no
> >>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don't
> believe. Is
> >>>>> this
> >>>>>>>>>> > reasonable to
> >>>>>>>>>> > > > all?
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
> therefor
> >>>>> happy
> >>>>>>>>> we keep
> >>>>>>>>>> > with 32
> >>>>>>>>>> > > > >> bits.
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
> >>>>>>>>>> > ignacio.solis@gmail.com on
> >>>>>>>>>> > > > >> behalf
> >>>>>>>>>> > > > >> > of
> >>>>>>>>>> > > > >> > >> >> >>> Ignacio
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
> >>> ignacio.solis@gmail.com
> >>>>> on
> >>>>>>>>> behalf of
> >>>>>>>>>> > > > >> > isolis@igso.net
> >>>>>>>>>> > > > >> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value
> as
> >>>>> byte[]
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> >>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
> >>> maybe
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
> >>>>> system
> >>>>>>>>> should
> >>>>>>>>>> > take a
> >>>>>>>>>> > > > single
> >>>>>>>>>> > > > >> > >> int. I
> >>>>>>>>>> > > > >> > >> >> >>> think
> >>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
> >>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you
> want
> >>> to
> >>>>>>>>> interpret
> >>>>>>>>>> > this as
> >>>>>>>>>> > > to
> >>>>>>>>>> > > > >> 16bit
> >>>>>>>>>> > > > >> > >> >> numbers
> >>>>>>>>>> > > > >> > >> >> >>> in
> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer
> >>>>>>>>>> > > > >> > >> >> >>> > >> > above go right ahead.
> If
> >>>>>>>>> somebody wants
> >>>>>>>>>> > to
> >>>>>>>>>> > > argue
> >>>>>>>>>> > > > >> for
> >>>>>>>>>> > > > >> > 16
> >>>>>>>>>> > > > >> > >> >> bits
> >>>>>>>>>> > > > >> > >> >> >>> or
> >>>>>>>>>> > > > >> > >> >> >>> > 64
> >>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I
> would
> >>>>>>> listen.
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key space
> >>> into
> >>>>>>>>> sub_key_1 and
> >>>>>>>>>> > > > sub_key_2
> >>>>>>>>>> > > > >> > >> makes no
> >>>>>>>>>> > > > >> > >> >> >>> > sense to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > me at
> >>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we
> going
> >>> to
> >>>>>>>>> start
> >>>>>>>>>> > providing
> >>>>>>>>>> > > > APIs to
> >>>>>>>>>> > > > >> > get
> >>>>>>>>>> > > > >> > >> all
> >>>>>>>>>> > > > >> > >> >> >>> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
> >>>>>>>>> sub_key_2s? If
> >>>>>>>>>> > there is
> >>>>>>>>>> > > > no
> >>>>>>>>>> > > > >> > >> >> >>> distinguishing
> >>>>>>>>>> > > > >> > >> >> >>> > >> > functions
> >>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to
> each
> >>> one
> >>>>>>>>> then they
> >>>>>>>>>> > should
> >>>>>>>>>> > > be
> >>>>>>>>>> > > > a
> >>>>>>>>>> > > > >> > single
> >>>>>>>>>> > > > >> > >> >> >>> value.
> >>>>>>>>>> > > > >> > >> >> >>> > At
> >>>>>>>>>> > > > >> > >> >> >>> > >> > this
> >>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're doing
> is
> >>>>>>>>> equality.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer
> wants
> >>> to
> >>>>>>>>> interpret
> >>>>>>>>>> > this as
> >>>>>>>>>> > > 2,
> >>>>>>>>>> > > > 3
> >>>>>>>>>> > > > >> or
> >>>>>>>>>> > > > >> > >> more
> >>>>>>>>>> > > > >> > >> >> >>> values
> >>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > different question. I
> >>>>>>>>> personally think
> >>>>>>>>>> > it's
> >>>>>>>>>> > > all
> >>>>>>>>>> > > > one
> >>>>>>>>>> > > > >> > >> >> keyspace
> >>>>>>>>>> > > > >> > >> >> >>> > that is
> >>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned using
> >>> some
> >>>>>>>>> structure,
> >>>>>>>>>> > but if
> >>>>>>>>>> > > you
> >>>>>>>>>> > > > >> > want to
> >>>>>>>>>> > > > >> > >> >> >>> > sub-assign
> >>>>>>>>>> > > > >> > >> >> >>> > >> > parts
> >>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
> >>> applies
> >>>>> to
> >>>>>>>>> strings.
> >>>>>>>>>> > If
> >>>>>>>>>> > > > somebody
> >>>>>>>>>> > > > >> > >> argued
> >>>>>>>>>> > > > >> > >> >> for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > would we be arguing to
> >>>>> divide
> >>>>>>> the
> >>>>>>>>>> > strings with
> >>>>>>>>>> > > > dots
> >>>>>>>>>> > > > >> > ('.')
> >>>>>>>>>> > > > >> > >> >> as a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them to
> give
> >>>>> us
> >>>>>>> the
> >>>>>>>>>> > different
> >>>>>>>>>> > > name
> >>>>>>>>>> > > > >> > segments
> >>>>>>>>>> > > > >> > >> >> >>> > separately?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be performing
> any
> >>>>>>>>> actions on
> >>>>>>>>>> > this key
> >>>>>>>>>> > > > other
> >>>>>>>>>> > > > >> > than
> >>>>>>>>>> > > > >> > >> >> >>> > matching?
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
> >>> 9:30
> >>>>>>> AM,
> >>>>>>>>> Michael
> >>>>>>>>>> > > Pearce <
> >>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> concerns
> >>> on
> >>>>> 1
> >>>>>>>>> and 2
> >>>>>>>>>> > still?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this moving
> >>> along a
> >>>>>>> bit
> >>>>>>>>> more
> >>>>>>>>>> > I'd also
> >>>>>>>>>> > > > like
> >>>>>>>>>> > > > >> to
> >>>>>>>>>> > > > >> > >> ask
> >>>>>>>>>> > > > >> > >> >> to
> >>>>>>>>>> > > > >> > >> >> >>> get
> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last
> points:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're
> all
> >>>>>>> roughly
> >>>>>>>>> happy
> >>>>>>>>>> > with the
> >>>>>>>>>> > > > >> header
> >>>>>>>>>> > > > >> > >> value
> >>>>>>>>>> > > > >> > >> >> >>> > being a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> consensus
> >>> has
> >>>>>>>>> been for an
> >>>>>>>>>> > > > namespace
> >>>>>>>>>> > > > >> > based
> >>>>>>>>>> > > > >> > >> int
> >>>>>>>>>> > > > >> > >> >> >>> > approach
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the
> key.
> >>> Any
> >>>>>>>>> objections
> >>>>>>>>>> > if this
> >>>>>>>>>> > > > is
> >>>>>>>>>> > > > >> > what
> >>>>>>>>>> > > > >> > >> we
> >>>>>>>>>> > > > >> > >> >> go
> >>>>>>>>>> > > > >> > >> >> >>> > with?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
> >>>>> assumption
> >>>>>>> in
> >>>>>>>>> (4) is
> >>>>>>>>>> > > correct,
> >>>>>>>>>> > > > >> > >> {int,int}
> >>>>>>>>>> > > > >> > >> >> >>> keys.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both int's be
> >>>>> int16 or
> >>>>>>>>> int32?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them being
> >>> int16(2
> >>>>>>>>> bytes) as
> >>>>>>>>>> > combined
> >>>>>>>>>> > > > is
> >>>>>>>>>> > > > >> > space
> >>>>>>>>>> > > > >> > >> of
> >>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
> >>>>>>>>>> > > > >> > >> >> >>> > >> > per
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > original and gives
> >>> plenty
> >>>>> of
> >>>>>>>>>> > combinations for
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> >> >>> foreseeable,
> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead small.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any
> benefit in
> >>>>>>>>> another kip
> >>>>>>>>>> > call to
> >>>>>>>>>> > > > >> discuss
> >>>>>>>>>> > > > >> > >> >> these at
> >>>>>>>>>> > > > >> > >> >> >>> > all?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>> ______________________________
> >>>>>>>>>> > __________
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
> >>>>>>>>> k.burstev@yandex.com>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> November
> >>> 18,
> >>>>>>> 2016
> >>>>>>>>>> > 7:07:07 AM
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
> >>> dev@kafka.apache.org
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> [DISCUSS]
> >>>>> KIP-82
> >>>>>>>>> - Add
> >>>>>>>>>> > Record
> >>>>>>>>>> > > > Headers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is worth
> >>> also
> >>>>> i
> >>>>>>>>> agree. As
> >>>>>>>>>> > a user:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
> >>>>>>>>> worthwhile
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers
> should
> >>>>> be a
> >>>>>>>>> top level
> >>>>>>>>>> > > option
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
> >>>>> "Ignacio
> >>>>>>>>> Solis" <
> >>>>>>>>>> > > > >> isolis@igso.net
> >>>>>>>>>> > > > >> > >:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers
> are
> >>>>>>>>> worthwhile
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
> >>> should
> >>>>> be
> >>>>>>> a
> >>>>>>>>> top
> >>>>>>>>>> > level
> >>>>>>>>>> > > option
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14,
> 2016
> >>> at
> >>>>>>> 9:16
> >>>>>>>>> AM,
> >>>>>>>>>> > Michael
> >>>>>>>>>> > > > Pearce
> >>>>>>>>>> > > > >> <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
> >>>>> details/examples
> >>>>>>>>> the
> >>>>>>>>>> > original
> >>>>>>>>>> > > > proposal
> >>>>>>>>>> > > > >> > for
> >>>>>>>>>> > > > >> > >> key
> >>>>>>>>>> > > > >> > >> >> >>> > spacing
> >>>>>>>>>> > > > >> > >> >> >>> > >> ,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > not
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as
> per
> >>>>>>>>> discussion
> >>>>>>>>>> > namespace
> >>>>>>>>>> > > > >> idea.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need to
> >>> update
> >>>>>>> the
> >>>>>>>>> kip,
> >>>>>>>>>> > when we
> >>>>>>>>>> > > get
> >>>>>>>>>> > > > >> > >> agreement
> >>>>>>>>>> > > > >> > >> >> >>> this
> >>>>>>>>>> > > > >> > >> >> >>> > is a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > better
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach (which
> >>> seems
> >>>>> to
> >>>>>>>>> be the
> >>>>>>>>>> > case if I
> >>>>>>>>>> > > > have
> >>>>>>>>>> > > > >> > >> >> understood
> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > general
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
> >>>>>>>>> conversation)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
> >>> ints,
> >>>>> at
> >>>>>>>>> very
> >>>>>>>>>> > early stage
> >>>>>>>>>> > > > we
> >>>>>>>>>> > > > >> did
> >>>>>>>>>> > > > >> > >> think
> >>>>>>>>>> > > > >> > >> >> >>> about
> >>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added
> complexity
> >>>>> for
> >>>>>>>>> the
> >>>>>>>>>> > saving isn't
> >>>>>>>>>> > > > >> worth
> >>>>>>>>>> > > > >> > it.
> >>>>>>>>>> > > > >> > >> >> I'd
> >>>>>>>>>> > > > >> > >> >> >>> > rather
> >>>>>>>>>> > > > >> > >> >> >>> > >> go
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
> >>>>>>>>> overheads and
> >>>>>>>>>> > size
> >>>>>>>>>> > > int16
> >>>>>>>>>> > > > >> > (2bytes)
> >>>>>>>>>> > > > >> > >> >> keys
> >>>>>>>>>> > > > >> > >> >> >>> as
> >>>>>>>>>> > > > >> > >> >> >>> > it
> >>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
> >>>>>>> headers,
> >>>>>>>>> there
> >>>>>>>>>> > is as
> >>>>>>>>>> > > per
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > kip
> >>>>>>>>>> > > > >> > >> as
> >>>>>>>>>> > > > >> > >> >> we
> >>>>>>>>>> > > > >> > >> >> >>> > use an
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
> >>>>> headers
> >>>>>>>>> are
> >>>>>>>>>> > present or
> >>>>>>>>>> > > > not as
> >>>>>>>>>> > > > >> > such
> >>>>>>>>>> > > > >> > >> >> >>> > provides a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > zero
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead
> currently
> >>> if
> >>>>>>>>> headers are
> >>>>>>>>>> > not
> >>>>>>>>>> > > used.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
> >>>>> mentions
> >>>>>>>>> would be
> >>>>>>>>>> > good
> >>>>>>>>>> > > > first
> >>>>>>>>>> > > > >> > if we
> >>>>>>>>>> > > > >> > >> >> can
> >>>>>>>>>> > > > >> > >> >> >>> get
> >>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > do
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have
> general
> >>>>>>>>> consensus that
> >>>>>>>>>> > (1)
> >>>>>>>>>> > > > headers
> >>>>>>>>>> > > > >> > are
> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>>>>>>>>> > > > >> > >> >> >>> > >> and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want
> it
> >>> as
> >>>>> a
> >>>>>>>>> top level
> >>>>>>>>>> > entity.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
> >>>>>>> obvious i
> >>>>>>>>>> > believe (1)
> >>>>>>>>>> > > > >> headers
> >>>>>>>>>> > > > >> > are
> >>>>>>>>>> > > > >> > >> >> >>> > worthwhile
> >>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a top
> level
> >>>>>>>>> entity.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>> ______________________________
> >>>>>>>>>> > __________
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
> Hoover <
> >>>>>>>>>> > > roger.hoover@gmail.com
> >>>>>>>>>> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
> >>>>> November
> >>>>>>>>> 9, 2016
> >>>>>>>>>> > 9:10:47
> >>>>>>>>>> > > > PM
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
> >>>>> dev@kafka.apache.org
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
> >>> [DISCUSS]
> >>>>>>>>> KIP-82 - Add
> >>>>>>>>>> > > Record
> >>>>>>>>>> > > > >> > Headers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
> >>>>> little
> >>>>>>>>> in the
> >>>>>>>>>> > weeds but
> >>>>>>>>>> > > > >> thanks
> >>>>>>>>>> > > > >> > >> for
> >>>>>>>>>> > > > >> > >> >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> replies
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a
> prefix
> >>>>> and
> >>>>>>>>> {int,
> >>>>>>>>>> > int} can
> >>>>>>>>>> > > be
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> same.
> >>>>>>>>>> > > > >> > >> >> It
> >>>>>>>>>> > > > >> > >> >> >>> > doesn't
> >>>>>>>>>> > > > >> > >> >> >>> > >> > look
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > like
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what the
> KIP
> >>> is
> >>>>>>>>> saying the
> >>>>>>>>>> > "Open"
> >>>>>>>>>> > > > >> > section.
> >>>>>>>>>> > > > >> > >> The
> >>>>>>>>>> > > > >> > >> >> >>> > example
> >>>>>>>>>> > > > >> > >> >> >>> > >> > shows
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
> >>>>> 210002
> >>>>>>>>> for App
> >>>>>>>>>> > Dynamics
> >>>>>>>>>> > > > >> > implying
> >>>>>>>>>> > > > >> > >> >> that
> >>>>>>>>>> > > > >> > >> >> >>> the
> >>>>>>>>>> > > > >> > >> >> >>> > New
> >>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization will
> >>> have
> >>>>>>>>> only a
> >>>>>>>>>> > single
> >>>>>>>>>> > > > header id
> >>>>>>>>>> > > > >> > to
> >>>>>>>>>> > > > >> > >> work
> >>>>>>>>>> > > > >> > >> >> >>> > with. Or
> >>>>>>>>>> > > > >> > >> >> >>> > >> > is
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The
> main
> >>>>> point
> >>>>>>>>> of a
> >>>>>>>>>> > namespace
> >>>>>>>>>> > > or
> >>>>>>>>>> > > > >> > prefix
> >>>>>>>>>> > > > >> > >> is
> >>>>>>>>>> > > > >> > >> >> to
> >>>>>>>>>> > > > >> > >> >> >>> > reduce
> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of
> config
> >>>>>>> mapping
> >>>>>>>>> or
> >>>>>>>>>> > > registration
> >>>>>>>>>> > > > >> > >> depending
> >>>>>>>>>> > > > >> > >> >> on
> >>>>>>>>>> > > > >> > >> >> >>> how
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> namespaces/prefixes
> >>>>> are
> >>>>>>>>> managed.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to
> hear
> >>>>> more
> >>>>>>>>> feedback
> >>>>>>>>>> > on the
> >>>>>>>>>> > > > >> > >> higher-level
> >>>>>>>>>> > > > >> > >> >> >>> > questions
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9,
> 2016
> >>> at
> >>>>>>>>> 11:38 AM,
> >>>>>>>>>> > radai <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com
> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think this
> >>>>> discussion
> >>>>>>>>> is
> >>>>>>>>>> > getting a
> >>>>>>>>>> > > bit
> >>>>>>>>>> > > > >> into
> >>>>>>>>>> > > > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> weeds on
> >>>>>>>>>> > > > >> > >> >> >>> > >> > technical
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > implementation
> >>>>> details.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to
> step
> >>>>> back a
> >>>>>>>>> minute
> >>>>>>>>>> > and try
> >>>>>>>>>> > > > and
> >>>>>>>>>> > > > >> > >> establish
> >>>>>>>>>> > > > >> > >> >> >>> > where we
> >>>>>>>>>> > > > >> > >> >> >>> > >> > are in
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
> >>> nacho's
> >>>>>>> last
> >>>>>>>>>> > paragraph)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all
> in
> >>>>>>>>> agreement that
> >>>>>>>>>> > headers
> >>>>>>>>>> > > > are
> >>>>>>>>>> > > > >> a
> >>>>>>>>>> > > > >> > >> >> >>> worthwhile
> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > useful
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to
> have?
> >>>>> this
> >>>>>>>>> was
> >>>>>>>>>> > contested
> >>>>>>>>>> > > > early
> >>>>>>>>>> > > > >> on
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all
> in
> >>>>>>>>> agreement on
> >>>>>>>>>> > headers
> >>>>>>>>>> > > as
> >>>>>>>>>> > > > top
> >>>>>>>>>> > > > >> > >> level
> >>>>>>>>>> > > > >> > >> >> >>> entity
> >>>>>>>>>> > > > >> > >> >> >>> > vs
> >>>>>>>>>> > > > >> > >> >> >>> > >> > headers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> squirreled-away in
> >>>>> V?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there are
> still
> >>>>>>>>> concerns
> >>>>>>>>>> > around
> >>>>>>>>>> > > these
> >>>>>>>>>> > > > #2
> >>>>>>>>>> > > > >> > >> points
> >>>>>>>>>> > > > >> > >> >> >>> (#jay?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now back
> to
> >>> our
> >>>>>>>>> normal
> >>>>>>>>>> > programming
> >>>>>>>>>> > > > ...)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are
> nice.
> >>>>>>> having
> >>>>>>>>> said
> >>>>>>>>>> > that, its
> >>>>>>>>>> > > > >> adding
> >>>>>>>>>> > > > >> > >> >> >>> complexity
> >>>>>>>>>> > > > >> > >> >> >>> > >> (see
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>> https://github.com/addthis/
> >>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>> main/java/com/clearspring/
> >>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
> >>>>> result)
> >>>>>>>>> and would
> >>>>>>>>>> > require
> >>>>>>>>>> > > > >> anyone
> >>>>>>>>>> > > > >> > >> >> writing
> >>>>>>>>>> > > > >> > >> >> >>> > other
> >>>>>>>>>> > > > >> > >> >> >>> > >> > clients
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go?
> Bash?
> >>>>> ;-) )
> >>>>>>>>> to
> >>>>>>>>>> > > get/implement
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > >> same,
> >>>>>>>>>> > > > >> > >> >> and
> >>>>>>>>>> > > > >> > >> >> >>> for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain
> (int
> >>> vs
> >>>>>>>>> string is
> >>>>>>>>>> > order of
> >>>>>>>>>> > > > >> > magnitude,
> >>>>>>>>>> > > > >> > >> >> this
> >>>>>>>>>> > > > >> > >> >> >>> > isnt).
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int
> namespacing vs
> >>>>>>> {int,
> >>>>>>>>> int}
> >>>>>>>>>> > > namespacing
> >>>>>>>>>> > > > >> are
> >>>>>>>>>> > > > >> > >> >> basically
> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > same
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
> >>>>> namespacing
> >>>>>>>>> an int64
> >>>>>>>>>> > and
> >>>>>>>>>> > > > giving
> >>>>>>>>>> > > > >> > people
> >>>>>>>>>> > > > >> > >> >> while
> >>>>>>>>>> > > > >> > >> >> >>> > 2^32
> >>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the part
> i
> >>>>> like
> >>>>>>>>> about this
> >>>>>>>>>> > is
> >>>>>>>>>> > > > letting
> >>>>>>>>>> > > > >> > >> people
> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>>>>>>>>> > > > >> > >> >> >>> > >> large
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers with
> one
> >>>>>>>>> registration so
> >>>>>>>>>> > they
> >>>>>>>>>> > > > dont
> >>>>>>>>>> > > > >> > have
> >>>>>>>>>> > > > >> > >> to
> >>>>>>>>>> > > > >> > >> >> come
> >>>>>>>>>> > > > >> > >> >> >>> > back
> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > every
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
> >>> plugin/header
> >>>>>>>>> they want to
> >>>>>>>>>> > > > "reserve".
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
> >>> 2016
> >>>>> at
> >>>>>>>>> 11:01 AM,
> >>>>>>>>>> > Roger
> >>>>>>>>>> > > > >> Hoover
> >>>>>>>>>> > > > >> > <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> roger.hoover@gmail.com>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
> >>> the
> >>>>>>>>> debate has
> >>>>>>>>>> > been
> >>>>>>>>>> > > about
> >>>>>>>>>> > > > >> > >> overhead +
> >>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if
> we
> >>>>> have
> >>>>>>>>>> > considered a
> >>>>>>>>>> > > > varint
> >>>>>>>>>> > > > >> > >> encoding
> >>>>>>>>>> > > > >> > >> >> (
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>> https://developers.google.com/
> >>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the header
> >>> length
> >>>>>>>>> field (int32
> >>>>>>>>>> > in the
> >>>>>>>>>> > > > >> > proposal)
> >>>>>>>>>> > > > >> > >> >> and
> >>>>>>>>>> > > > >> > >> >> >>> for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > header
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
> >>> headers,
> >>>>>>> the
> >>>>>>>>>> > overhead would
> >>>>>>>>>> > > > be a
> >>>>>>>>>> > > > >> > >> single
> >>>>>>>>>> > > > >> > >> >> >>> byte
> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > for each
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128
> would
> >>>>> also
> >>>>>>>>> need only a
> >>>>>>>>>> > > single
> >>>>>>>>>> > > > >> byte?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov
> 9,
> >>>>> 2016
> >>>>>>> at
> >>>>>>>>> 6:43 AM,
> >>>>>>>>>> > > radai <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com
> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus -
> and
> >>>>> very
> >>>>>>>>> dangerous
> >>>>>>>>>> > (youre
> >>>>>>>>>> > > > >> > >> essentially
> >>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary
> code
> >>>>> off
> >>>>>>>>> the
> >>>>>>>>>> > internet on
> >>>>>>>>>> > > > your
> >>>>>>>>>> > > > >> > >> servers
> >>>>>>>>>> > > > >> > >> >> ...
> >>>>>>>>>> > > > >> > >> >> >>> > bad
> >>>>>>>>>> > > > >> > >> >> >>> > >> > idea
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > without
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox,
> even
> >>>>> with)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
> >>> being
> >>>>> a
> >>>>>>>>> purely
> >>>>>>>>>> > > > administrative
> >>>>>>>>>> > > > >> > task
> >>>>>>>>>> > > > >> > >> - i
> >>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
> >>> would,
> >>>>>>>>> really,
> >>>>>>>>>> > because
> >>>>>>>>>> > > > then my
> >>>>>>>>>> > > > >> > >> earlier
> >>>>>>>>>> > > > >> > >> >> >>> > point on
> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> remapping
> >>>>>>>>> process would
> >>>>>>>>>> > be
> >>>>>>>>>> > > > invalid,
> >>>>>>>>>> > > > >> > but
> >>>>>>>>>> > > > >> > >> at
> >>>>>>>>>> > > > >> > >> >> >>> > linkedin,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team
> im
> >>> in)
> >>>>>>> run
> >>>>>>>>> kafka
> >>>>>>>>>> > as a
> >>>>>>>>>> > > > service.
> >>>>>>>>>> > > > >> > we
> >>>>>>>>>> > > > >> > >> dont
> >>>>>>>>>> > > > >> > >> >> >>> > really
> >>>>>>>>>> > > > >> > >> >> >>> > >> > know
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
> >>>>>>>>> applications
> >>>>>>>>>> > that use
> >>>>>>>>>> > > > kafka)
> >>>>>>>>>> > > > >> > are
> >>>>>>>>>> > > > >> > >> up
> >>>>>>>>>> > > > >> > >> >> to
> >>>>>>>>>> > > > >> > >> >> >>> at
> >>>>>>>>>> > > > >> > >> >> >>> > any
> >>>>>>>>>> > > > >> > >> >> >>> > >> > given
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
> >>> possible
> >>>>>>>>> (given the
> >>>>>>>>>> > > > existance of
> >>>>>>>>>> > > > >> > >> headers
> >>>>>>>>>> > > > >> > >> >> >>> and a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem)
> for
> >>>>> some
> >>>>>>>>>> > application to
> >>>>>>>>>> > > > >> "equip"
> >>>>>>>>>> > > > >> > >> their
> >>>>>>>>>> > > > >> > >> >> >>> > >> producers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
> >>>>> required
> >>>>>>>>> plugin
> >>>>>>>>>> > without us
> >>>>>>>>>> > > > >> > knowing.
> >>>>>>>>>> > > > >> > >> i
> >>>>>>>>>> > > > >> > >> >> dont
> >>>>>>>>>> > > > >> > >> >> >>> > mean
> >>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
> >>>>> want to
> >>>>>>>>> make the
> >>>>>>>>>> > point
> >>>>>>>>>> > > > that
> >>>>>>>>>> > > > >> > its
> >>>>>>>>>> > > > >> > >> not
> >>>>>>>>>> > > > >> > >> >> as
> >>>>>>>>>> > > > >> > >> >> >>> > simple
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
> across a
> >>>>>>>>> large-enough
> >>>>>>>>>> > > > organization.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed,
> Nov 9,
> >>>>> 2016
> >>>>>>>>> at 6:17
> >>>>>>>>>> > AM,
> >>>>>>>>>> > > > Magnus
> >>>>>>>>>> > > > >> > >> Edenhill
> >>>>>>>>>> > > > >> > >> >> <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
> >>> there
> >>>>> is
> >>>>>>> a
> >>>>>>>>> piece
> >>>>>>>>>> > missing
> >>>>>>>>>> > > in
> >>>>>>>>>> > > > >> the
> >>>>>>>>>> > > > >> > >> >> Strings
> >>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > where
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> pro-Stringers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason
> that
> >>> by
> >>>>>>>>> providing
> >>>>>>>>>> > unique
> >>>>>>>>>> > > > string
> >>>>>>>>>> > > > >> > >> >> >>> identifiers
> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> everything
> >>>>> will
> >>>>>>>>> just
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
> >>> work
> >>>>>>> for
> >>>>>>>>> all
> >>>>>>>>>> > parts of
> >>>>>>>>>> > > the
> >>>>>>>>>> > > > >> > stream
> >>>>>>>>>> > > > >> > >> >> >>> pipeline.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
> >>>>> strings
> >>>>>>>>> dont mean
> >>>>>>>>>> > > anything
> >>>>>>>>>> > > > by
> >>>>>>>>>> > > > >> > >> >> themselves,
> >>>>>>>>>> > > > >> > >> >> >>> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > while we
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
> >>>>> envision
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
> >>>>> plugin
> >>>>>>>>> loader
> >>>>>>>>>> > that
> >>>>>>>>>> > > > >> downloads,
> >>>>>>>>>> > > > >> > >> >> compiles,
> >>>>>>>>>> > > > >> > >> >> >>> > links
> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon
> as
> >>>>>>> they're
> >>>>>>>>> seen by
> >>>>>>>>>> > a
> >>>>>>>>>> > > > >> consumer, I
> >>>>>>>>>> > > > >> > >> dont
> >>>>>>>>>> > > > >> > >> >> >>> really
> >>>>>>>>>> > > > >> > >> >> >>> > >> see
> >>>>>>>>>> > > > >> > >> >> >>> > >> > a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
> dynamic
> >>>>> (and
> >>>>>>>>> fragile) in
> >>>>>>>>>> > > > practice.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the
> real
> >>>>> world
> >>>>>>>>> an
> >>>>>>>>>> > application
> >>>>>>>>>> > > > will
> >>>>>>>>>> > > > >> be
> >>>>>>>>>> > > > >> > >> >> >>> configured
> >>>>>>>>>> > > > >> > >> >> >>> > >> with
> >>>>>>>>>> > > > >> > >> >> >>> > >> > a set
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
> >>> add
> >>>>>>>>> (producer)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
> >>>>>>> (consumer)
> >>>>>>>>> headers.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is
> an
> >>>>>>>>> administrative
> >>>>>>>>>> > task
> >>>>>>>>>> > > > based
> >>>>>>>>>> > > > >> on
> >>>>>>>>>> > > > >> > >> what
> >>>>>>>>>> > > > >> > >> >> >>> > features a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > client
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>> needs/provides
> >>>>>>> and
> >>>>>>>>> results
> >>>>>>>>>> > in
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> sort of
> >>>>>>>>> configuration
> >>>>>>>>>> > to
> >>>>>>>>>> > > > enable
> >>>>>>>>>> > > > >> and
> >>>>>>>>>> > > > >> > >> >> >>> configure
> >>>>>>>>>> > > > >> > >> >> >>> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > desired
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since
> this
> >>>>> needs
> >>>>>>>>> to be kept
> >>>>>>>>>> > > > somewhat
> >>>>>>>>>> > > > >> in
> >>>>>>>>>> > > > >> > >> sync
> >>>>>>>>>> > > > >> > >> >> >>> across
> >>>>>>>>>> > > > >> > >> >> >>> > an
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no
> point
> >>> in
> >>>>>>>>> having
> >>>>>>>>>> > producers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
> headers
> >>> no
> >>>>>>>>> consumers
> >>>>>>>>>> > will
> >>>>>>>>>> > > read,
> >>>>>>>>>> > > > >> and
> >>>>>>>>>> > > > >> > >> vice
> >>>>>>>>>> > > > >> > >> >> >>> versa),
> >>>>>>>>>> > > > >> > >> >> >>> > >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > added
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
> assigning
> >>>>> an
> >>>>>>> id
> >>>>>>>>>> > namespace
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
> >>>>> plugin
> >>>>>>> as
> >>>>>>>>> it is
> >>>>>>>>>> > being
> >>>>>>>>>> > > > >> > configured
> >>>>>>>>>> > > > >> > >> >> should
> >>>>>>>>>> > > > >> > >> >> >>> be
> >>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> 2016-11-09
> >>>>> 13:06
> >>>>>>>>> GMT+01:00
> >>>>>>>>>> > > Michael
> >>>>>>>>>> > > > >> > Pearce <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > >
> Michael.Pearce@ig.com>:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
> >>>>>>>>> following/catching
> >>>>>>>>>> > up on
> >>>>>>>>>> > > > what
> >>>>>>>>>> > > > >> > seems
> >>>>>>>>>> > > > >> > >> to
> >>>>>>>>>> > > > >> > >> >> be
> >>>>>>>>>> > > > >> > >> >> >>> an
> >>>>>>>>>> > > > >> > >> >> >>> > >> > active
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
> >>>>> sorry if
> >>>>>>>>> it may
> >>>>>>>>>> > seem
> >>>>>>>>>> > > > obvious
> >>>>>>>>>> > > > >> > but
> >>>>>>>>>> > > > >> > >> what
> >>>>>>>>>> > > > >> > >> >> >>> does
> >>>>>>>>>> > > > >> > >> >> >>> > MD
> >>>>>>>>>> > > > >> > >> >> >>> > >> > stand
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My
> take on
> >>>>>>>>> String vs Int:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
> >>> state
> >>>>>>>>> first I am
> >>>>>>>>>> > pro Int
> >>>>>>>>>> > > > (16
> >>>>>>>>>> > > > >> or
> >>>>>>>>>> > > > >> > >> 32).
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
> >>> though
> >>>>>>>>> playing
> >>>>>>>>>> > devils
> >>>>>>>>>> > > > advocate
> >>>>>>>>>> > > > >> > see a
> >>>>>>>>>> > > > >> > >> >> big
> >>>>>>>>>> > > > >> > >> >> >>> plus
> >>>>>>>>>> > > > >> > >> >> >>> > >> > with the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
> >>> keys,
> >>>>>>>>> this is
> >>>>>>>>>> > around
> >>>>>>>>>> > > > >> > integrating
> >>>>>>>>>> > > > >> > >> >> into an
> >>>>>>>>>> > > > >> > >> >> >>> > >> > existing
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
> >>>>> other
> >>>>>>>>> systems use
> >>>>>>>>>> > > String
> >>>>>>>>>> > > > >> based
> >>>>>>>>>> > > > >> > >> >> headers
> >>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > it
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
> >>> easier
> >>>>> for
> >>>>>>>>> these to
> >>>>>>>>>> > be
> >>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
> >>>>>>>>>> > > > >> > >> >> >>> > >> > into.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How
> with
> >>> Int
> >>>>>>>>> based
> >>>>>>>>>> > headers
> >>>>>>>>>> > > could
> >>>>>>>>>> > > > we
> >>>>>>>>>> > > > >> > >> provide
> >>>>>>>>>> > > > >> > >> >> a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>> integration
> >>>>>>>>> simple /
> >>>>>>>>>> > easy with
> >>>>>>>>>> > > > >> > transition
> >>>>>>>>>> > > > >> > >> >> flows
> >>>>>>>>>> > > > >> > >> >> >>> > over
> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
> >>> luck
> >>>>>>>>> buddy
> >>>>>>>>>> > you're on
> >>>>>>>>>> > > your
> >>>>>>>>>> > > > >> own
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> simply
> >>>>> hash
> >>>>>>>>> the string
> >>>>>>>>>> > into
> >>>>>>>>>> > > int
> >>>>>>>>>> > > > >> code
> >>>>>>>>>> > > > >> > >> and
> >>>>>>>>>> > > > >> > >> >> hope
> >>>>>>>>>> > > > >> > >> >> >>> > for
> >>>>>>>>>> > > > >> > >> >> >>> > >> no
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
> >>> back
> >>>>>>>>> though?)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
> >>>>> style
> >>>>>>> as
> >>>>>>>>>> > mentioned by
> >>>>>>>>>> > > > nacho.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > ______________________________
> >>>>>>>>>> > > > >> > __________
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
> >>> radai
> >>>>> <
> >>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
> >>>>>>> Wednesday,
> >>>>>>>>>> > November 9,
> >>>>>>>>>> > > 2016
> >>>>>>>>>> > > > >> > 8:12 AM
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
> >>>>>>>>> dev@kafka.apache.org
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Subject:
> >>> Re:
> >>>>>>>>> [DISCUSS]
> >>>>>>>>>> > KIP-82 -
> >>>>>>>>>> > > > Add
> >>>>>>>>>> > > > >> > >> Record
> >>>>>>>>>> > > > >> > >> >> >>> Headers
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> thinking
> >>>>> about
> >>>>>>>>> it some
> >>>>>>>>>> > more,
> >>>>>>>>>> > > the
> >>>>>>>>>> > > > >> best
> >>>>>>>>>> > > > >> > >> way to
> >>>>>>>>>> > > > >> > >> >> >>> > transmit
> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > header
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
> >>>>>>>>> consumers would
> >>>>>>>>>> > be to
> >>>>>>>>>> > > > put it
> >>>>>>>>>> > > > >> > in
> >>>>>>>>>> > > > >> > >> the
> >>>>>>>>>> > > > >> > >> >> MD
> >>>>>>>>>> > > > >> > >> >> >>> > >> response
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it
> should
> >>> be
> >>>>>>>>> discussed
> >>>>>>>>>> > now.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
> >>> Nov
> >>>>> 9,
> >>>>>>>>> 2016 at
> >>>>>>>>>> > 12:09
> >>>>>>>>>> > > AM,
> >>>>>>>>>> > > > >> > radai <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> >>>>>>> radai.rosenblatt@gmail.com
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im
> not
> >>>>>>> opposed
> >>>>>>>>> to the
> >>>>>>>>>> > idea of
> >>>>>>>>>> > > > >> > namespace
> >>>>>>>>>> > > > >> > >> >> >>> mapping.
> >>>>>>>>>> > > > >> > >> >> >>> > >> all
> >>>>>>>>>> > > > >> > >> >> >>> > >> > im
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not
> part
> >>>>> of
> >>>>>>>>> the "mvp"
> >>>>>>>>>> > and,
> >>>>>>>>>> > > > since
> >>>>>>>>>> > > > >> it
> >>>>>>>>>> > > > >> > >> >> requires
> >>>>>>>>>> > > > >> > >> >> >>> no
> >>>>>>>>>> > > > >> > >> >> >>> > >> wire
> >>>>>>>>>> > > > >> > >> >> >>> > >> > format
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> always
> >>> be
> >>>>>>>>> added later.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
> >>> its
> >>>>> not
> >>>>>>>>> as
> >>>>>>>>>> > simple as
> >>>>>>>>>> > > just
> >>>>>>>>>> > > > >> > >> >> configuring
> >>>>>>>>>> > > > >> > >> >> >>> MM
> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>>>>>>>>> > > > >> > >> >> >>> > >> do
> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say
> i've
> >>>>>>>>> implemented
> >>>>>>>>>> > large
> >>>>>>>>>> > > > message
> >>>>>>>>>> > > > >> > >> >> support as
> >>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > on
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> target
> >>>>>>> cluster
> >>>>>>>>> its been
> >>>>>>>>>> > > > remapped
> >>>>>>>>>> > > > >> to
> >>>>>>>>>> > > > >> > >> >> {999,1}.
> >>>>>>>>>> > > > >> > >> >> >>> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
> >>> need
> >>>>> to
> >>>>>>>>> be told
> >>>>>>>>>> > to look
> >>>>>>>>>> > > > for
> >>>>>>>>>> > > > >> the
> >>>>>>>>>> > > > >> > >> large
> >>>>>>>>>> > > > >> > >> >> >>> > message
> >>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
> >>>>>>> {999,1}.
> >>>>>>>>> doable,
> >>>>>>>>>> > but
> >>>>>>>>>> > > > tricky.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On
> Tue,
> >>>>> Nov
> >>>>>>> 8,
> >>>>>>>>> 2016 at
> >>>>>>>>>> > 10:29
> >>>>>>>>>> > > > PM,
> >>>>>>>>>> > > > >> > Gwen
> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
> gwen@confluent.io
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> While
> >>> you
> >>>>>>> can
> >>>>>>>>> do
> >>>>>>>>>> > whatever
> >>>>>>>>>> > > you
> >>>>>>>>>> > > > >> want
> >>>>>>>>>> > > > >> > >> with a
> >>>>>>>>>> > > > >> > >> >> >>> > >> namespace
> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > your
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
> >>> I'd
> >>>>>>>>> expect is
> >>>>>>>>>> > for each
> >>>>>>>>>> > > > app
> >>>>>>>>>> > > > >> to
> >>>>>>>>>> > > > >> > >> >> >>> namespaces
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So
> if I
> >>>>>>>>> accidentally
> >>>>>>>>>> > used
> >>>>>>>>>> > > 666
> >>>>>>>>>> > > > for
> >>>>>>>>>> > > > >> > my
> >>>>>>>>>> > > > >> > >> HR
> >>>>>>>>>> > > > >> > >> >> >>> > >> department,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > still
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
> >>>>>>> RadaiApp,
> >>>>>>>>> I can
> >>>>>>>>>> > config
> >>>>>>>>>> > > > >> > >> "namespace=42"
> >>>>>>>>>> > > > >> > >> >> >>> for
> >>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
> >>> look
> >>>>>>>>> normal.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
> >>>>> means
> >>>>>>>>> you only
> >>>>>>>>>> > need to
> >>>>>>>>>> > > > sync
> >>>>>>>>>> > > > >> > usage
> >>>>>>>>>> > > > >> > >> >> >>> inside
> >>>>>>>>>> > > > >> > >> >> >>> > your
> >>>>>>>>>> > > > >> > >> >> >>> > >> > own
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> Still
> >>>>> hard,
> >>>>>>>>> but
> >>>>>>>>>> > somewhat
> >>>>>>>>>> > > > easier
> >>>>>>>>>> > > > >> > than
> >>>>>>>>>> > > > >> > >> >> syncing
> >>>>>>>>>> > > > >> > >> >> >>> > with
> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On
> Tue,
> >>>>> Nov
> >>>>>>>>> 8, 2016
> >>>>>>>>>> > at 10:07
> >>>>>>>>>> > > > PM,
> >>>>>>>>>> > > > >> > >> radai <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>>>>>> radai.rosenblatt@gmail.com>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> and
> >>> we
> >>>>> can
> >>>>>>>>> start
> >>>>>>>>>> > with
> >>>>>>>>>> > > > >> {namespace,
> >>>>>>>>>> > > > >> > >> id}
> >>>>>>>>>> > > > >> > >> >> and
> >>>>>>>>>> > > > >> > >> >> >>> no
> >>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> always
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> add
> >>> it
> >>>>>>>>> later on
> >>>>>>>>>> > if/when
> >>>>>>>>>> > > > >> > collisions
> >>>>>>>>>> > > > >> > >> >> >>> actually
> >>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
> >>>>>>> problem).
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> every
> >>>>>>>>> interested
> >>>>>>>>>> > party (so
> >>>>>>>>>> > > > orgs
> >>>>>>>>>> > > > >> > or
> >>>>>>>>>> > > > >> > >> >> >>> > individuals)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > could
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > then
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > register
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >>> prefix
> >>>>> (0
> >>>>>>> =
> >>>>>>>>>> > reserved, 1 =
> >>>>>>>>>> > > > >> > confluent
> >>>>>>>>>> > > > >> > >> ...
> >>>>>>>>>> > > > >> > >> >> >>> 666
> >>>>>>>>>> > > > >> > >> >> >>> > = me
> >>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> the
> >>>>> 2nd ID
> >>>>>>>>> - so once
> >>>>>>>>>> > > > linkedin
> >>>>>>>>>> > > > >> > >> >> registers,
> >>>>>>>>>> > > > >> > >> >> >>> say
> >>>>>>>>>> > > > >> > >> >> >>> > 3,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > then
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
> >>> use
> >>>>> {3,
> >>>>>>>>> *} with a
> >>>>>>>>>> > > > reasonable
> >>>>>>>>>> > > > >> > >> >> >>> expectation
> >>>>>>>>>> > > > >> > >> >> >>> > to
> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> anything
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> else.
> >>>>>>>>> further
> >>>>>>>>>> > partitioning
> >>>>>>>>>> > > > of
> >>>>>>>>>> > > > >> > that *
> >>>>>>>>>> > > > >> > >> >> >>> becomes
> >>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >>>>> "upstream
> >>>>>>>>>> > registration"
> >>>>>>>>>> > > of a
> >>>>>>>>>> > > > >> > >> namespace
> >>>>>>>>>> > > > >> > >> >> >>> only
> >>>>>>>>>> > > > >> > >> >> >>> > has
> >>>>>>>>>> > > > >> > >> >> >>> > >> to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
> >>> Tue,
> >>>>>>> Nov
> >>>>>>>>> 8, 2016
> >>>>>>>>>> > at
> >>>>>>>>>> > > 9:03
> >>>>>>>>>> > > > PM,
> >>>>>>>>>> > > > >> > >> James
> >>>>>>>>>> > > > >> > >> >> >>> Cheng <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
> >>>>> wushujames@gmail.com
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> > On
> >>>>> Nov
> >>>>>>>>> 8, 2016,
> >>>>>>>>>> > at 5:54
> >>>>>>>>>> > > > PM,
> >>>>>>>>>> > > > >> > Gwen
> >>>>>>>>>> > > > >> > >> >> >>> Shapira <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
> gwen@confluent.io
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>> Thank
> >>>>>>>>> you so
> >>>>>>>>>> > much for
> >>>>>>>>>> > > > this
> >>>>>>>>>> > > > >> > clear
> >>>>>>>>>> > > > >> > >> and
> >>>>>>>>>> > > > >> > >> >> >>> fair
> >>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> arguments.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>> I'm
> >>>>> in
> >>>>>>>>> favor of
> >>>>>>>>>> > ints.
> >>>>>>>>>> > > > Not a
> >>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
> >>>>>>>>>> > > > >> > >> >> >>> > but
> >>>>>>>>>> > > > >> > >> >> >>> > >> > in
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>> Even
> >>>>>>>>> more in
> >>>>>>>>>> > favor of
> >>>>>>>>>> > > > >> Magnus's
> >>>>>>>>>> > > > >> > >> >> >>> > decentralized
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>> tweak:
> >>>>>>>>> add a
> >>>>>>>>>> > namespace
> >>>>>>>>>> > > > for
> >>>>>>>>>> > > > >> > >> headers.
> >>>>>>>>>> > > > >> > >> >> >>> This
> >>>>>>>>>> > > > >> > >> >> >>> > will
> >>>>>>>>>> > > > >> > >> >> >>> > >> > allow
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > each
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>> use
> >>>>>>>>> whatever IDs
> >>>>>>>>>> > it
> >>>>>>>>>> > > wants
> >>>>>>>>>> > > > >> > >> >> internally,
> >>>>>>>>>> > > > >> > >> >> >>> and
> >>>>>>>>>> > > > >> > >> >> >>> > >> then
> >>>>>>>>>> > > > >> > >> >> >>> > >> > let
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>> deploying
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>> the
> >>>>> app
> >>>>>>>>> figure
> >>>>>>>>>> > out an
> >>>>>>>>>> > > > >> > available
> >>>>>>>>>> > > > >> > >> >> >>> namespace
> >>>>>>>>>> > > > >> > >> >> >>> > ID
> >>>>>>>>>> > > > >> > >> >> >>> > >> > for the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> > So
> >>>>>>>>>> > > > >> > io.confluent.schema-registry
> >>>>>>>>>> > > > >> > >> can
> >>>>>>>>>> > > > >> > >> >> be
> >>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>>>> deployment
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>> and
> >>>>>>> 0x57
> >>>>>>>>> on
> >>>>>>>>>> > yours, and
> >>>>>>>>>> > > > the
> >>>>>>>>>> > > > >> > poor
> >>>>>>>>>> > > > >> > >> guys
> >>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > app
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>> worry
> >>>>>>>>> about that.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>> Gwen,
> >>>>> if
> >>>>>>> I
> >>>>>>>>>> > understand
> >>>>>>>>>> > > your
> >>>>>>>>>> > > > >> > example
> >>>>>>>>>> > > > >> > >> >> >>> right, an
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > application
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>> decide to
> >>>>>>>>> use 0x01
> >>>>>>>>>> > in one
> >>>>>>>>>> > > > >> > >> deployment,
> >>>>>>>>>> > > > >> > >> >> and
> >>>>>>>>>> > > > >> > >> >> >>> > that
> >>>>>>>>>> > > > >> > >> >> >>> > >> > means
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > that
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> message
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> is
> >>>>>>> written
> >>>>>>>>> into the
> >>>>>>>>>> > > > broker, it
> >>>>>>>>>> > > > >> > >> will be
> >>>>>>>>>> > > > >> > >> >> >>> > saved on
> >>>>>>>>>> > > > >> > >> >> >>> > >> > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>> specific
> >>>>>>>>> namespace
> >>>>>>>>>> > > (0x01).
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> If
> >>> you
> >>>>>>>>> were to
> >>>>>>>>>> > mirror
> >>>>>>>>>> > > that
> >>>>>>>>>> > > > >> > message
> >>>>>>>>>> > > > >> > >> >> into
> >>>>>>>>>> > > > >> > >> >> >>> > another
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>> accompany
> >>>>>>>>> the
> >>>>>>>>>> > message,
> >>>>>>>>>> > > > right?
> >>>>>>>>>> > > > >> > What
> >>>>>>>>>> > > > >> > >> if
> >>>>>>>>>> > > > >> > >> >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>> other
> >>>>>>>>> cluster uses
> >>>>>>>>>> > 0x57?
> >>>>>>>>>> > > > They
> >>>>>>>>>> > > > >> > won't
> >>>>>>>>>> > > > >> > >> >> >>> > understand
> >>>>>>>>>> > > > >> > >> >> >>> > >> > each
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> I'm
> >>>>> not
> >>>>>>>>> sure
> >>>>>>>>>> > that's an
> >>>>>>>>>> > > > >> avoidable
> >>>>>>>>>> > > > >> > >> >> >>> problem. I
> >>>>>>>>>> > > > >> > >> >> >>> > >> > think it
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>> order
> >>>>> to
> >>>>>>>>> share
> >>>>>>>>>> > data, you
> >>>>>>>>>> > > > have
> >>>>>>>>>> > > > >> to
> >>>>>>>>>> > > > >> > >> also
> >>>>>>>>>> > > > >> > >> >> >>> have a
> >>>>>>>>>> > > > >> > >> >> >>> > >> > shared
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>> understanding of
> >>>>>>>>>> > what the
> >>>>>>>>>> > > > >> > >> namespaces
> >>>>>>>>>> > > > >> > >> >> >>> mean.
> >>>>>>>>>> > > > >> > >> >> >>> > >> Which
> >>>>>>>>>> > > > >> > >> >> >>> > >> > I
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > think
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>> because
> >>>>>>> the
> >>>>>>>>>> > alternate
> >>>>>>>>>> > > > (sharing
> >>>>>>>>>> > > > >> > >> >> *nothing*
> >>>>>>>>>> > > > >> > >> >> >>> at
> >>>>>>>>>> > > > >> > >> >> >>> > >> all)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > would
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>> would
> >>>>> be
> >>>>>>>>> no way to
> >>>>>>>>>> > > > understand
> >>>>>>>>>> > > > >> > each
> >>>>>>>>>> > > > >> > >> >> other.
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>> -James
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>> Gwen
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> > On
> >>>>> Tue,
> >>>>>>>>> Nov 8,
> >>>>>>>>>> > 2016 at
> >>>>>>>>>> > > > 4:23
> >>>>>>>>>> > > > >> > PM,
> >>>>>>>>>> > > > >> > >> >> radai <
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >>>>>>>>> radai.rosenblatt@gmail.com
> >>>>>>>>>> > >
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>> wrote:
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>
> >>> +1
> >>>>> for
> >>>>>>>>> sean's
> >>>>>>>>>> > > document.
> >>>>>>>>>> > > > it
> >>>>>>>>>> > > > >> > >> covers
> >>>>>>>>>> > > > >> > >> >> >>> pretty
> >>>>>>>>>> > > > >> > >> >> >>> > >> much
> >>>>>>>>>> > > > >> > >> >> >>> > >> > all
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > the
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>
> >>>>>>> provides
> >>>>>>>>>> > concrete
> >>>>>>>>>> > > > figures
> >>>>>>>>>> > > > >> to
> >>>>>>>>>> > > > >> > >> argue
> >>>>>>>>>> > > > >> > >> >> >>> about
> >>>>>>>>>> > > > >> > >> >> >>> > :-)
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
> >>
> >>>>>>>>> (nit-picking -
> >>>>>>>>>> > used
> >>>>>>>>>> > > the
> >>>>>>>>>> > > > >> same
> >>>>>>>>>> > > > >> > >> xkcd
> >>>>>>>>>> > > > >> > >> >> >>> twice,
> >>>>>>>>>> > > > >> > >> >> >>> > >> also
> >>>>>>>>>> > > > >> > >> >> >>> > >> > trove
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > has
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
> >>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>>>> superceded
> >>>>>>>>>> > > > >> > >> >
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >> > --
> >>>>>>>>>> > > > >> > Gwen Shapira
> >>>>>>>>>> > > > >> > Product Manager | Confluent
> >>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >>>>>>>>>> > > > >> > Follow us: Twitter | blog
> >>>>>>>>>> > > > >> >
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >> --
> >>>>>>>>>> > > > >> *Todd Palino*
> >>>>>>>>>> > > > >> Staff Site Reliability Engineer
> >>>>>>>>>> > > > >> Data Infrastructure Streaming
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > > >> linkedin.com/in/toddpalino
> >>>>>>>>>> > > > >>
> >>>>>>>>>> > > >
> >>>>>>>>>> > > >
> >>>>>>>>>> > > >
> >>>>>>>>>> > > > --
> >>>>>>>>>> > > > Gwen Shapira
> >>>>>>>>>> > > > Product Manager | Confluent
> >>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >>>>>>>>>> > > > Follow us: Twitter | blog
> >>>>>>>>>> > > >
> >>>>>>>>>> > >
> >>>>>>>>>> >
> >>>>>>>>>> >
> >>>>>>>>>> > The information contained in this email is strictly
> >>>>> confidential
> >>>>>>>>> and for
> >>>>>>>>>> > the use of the addressee only, unless otherwise
> indicated.
> >>> If
> >>>>> you
> >>>>>>>>> are not
> >>>>>>>>>> > the intended recipient, please do not read, copy, use or
> >>>>> disclose
> >>>>>>>>> to others
> >>>>>>>>>> > this message or any attachment. Please also notify the
> >>> sender
> >>>>> by
> >>>>>>>>> replying
> >>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and
> then
> >>>>> delete
> >>>>>>>>> the email
> >>>>>>>>>> > and any copies of it. Opinions, conclusion (etc) that do
> not
> >>>>>>>>> relate to the
> >>>>>>>>>> > official business of this company shall be understood as
> >>>>> neither
> >>>>>>>>> given nor
> >>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets
> Limited
> >>> (a
> >>>>>>>>> company
> >>>>>>>>>> > registered in England and Wales, company number 04008957)
> >>> and
> >>>>> IG
> >>>>>>>>> Index
> >>>>>>>>>> > Limited (a company registered in England and Wales,
> company
> >>>>>>> number
> >>>>>>>>>> > 01190902). Registered address at Cannon Bridge House, 25
> >>>>> Dowgate
> >>>>>>>>> Hill,
> >>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register number
> >>>>> 195355)
> >>>>>>>>> and IG
> >>>>>>>>>> > Index Limited (register number 114059) are authorised and
> >>>>>>>>> regulated by the
> >>>>>>>>>> > Financial Conduct Authority.
> >>>>>>>>>> >
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> The information contained in this email is strictly confidential
> >>> and
> >>>>>>> for
> >>>>>>>>> the use of the addressee only, unless otherwise indicated. If you
> >>> are
> >>>>>>> not
> >>>>>>>>> the intended recipient, please do not read, copy, use or disclose
> >>> to
> >>>>>>> others
> >>>>>>>>> this message or any attachment. Please also notify the sender by
> >>>>>>> replying
> >>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
> >>> the
> >>>>>>> email
> >>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not
> relate
> >>>>> to
> >>>>>>> the
> >>>>>>>>> official business of this company shall be understood as neither
> >>>>> given
> >>>>>>> nor
> >>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
> >>> company
> >>>>>>>>> registered in England and Wales, company number 04008957) and IG
> >>>>> Index
> >>>>>>>>> Limited (a company registered in England and Wales, company
> number
> >>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> >>>>> Hill,
> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> >>>>> and IG
> >>>>>>>>> Index Limited (register number 114059) are authorised and
> regulated
> >>>>> by
> >>>>>>> the
> >>>>>>>>> Financial Conduct Authority.
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>> The information contained in this email is strictly confidential and
> for
> >>> the use of the addressee only, unless otherwise indicated. If you are
> not
> >>> the intended recipient, please do not read, copy, use or disclose to
> others
> >>> this message or any attachment. Please also notify the sender by
> replying
> >>> to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> >>> and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> >>> official business of this company shall be understood as neither given
> nor
> >>> endorsed by it. IG is a trading name of IG Markets Limited (a company
> >>> registered in England and Wales, company number 04008957) and IG Index
> >>> Limited (a company registered in England and Wales, company number
> >>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> >>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> >>> Index Limited (register number 114059) are authorised and regulated by
> the
> >>> Financial Conduct Authority.
> >>>
> >
> >
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
As we'd like to start work on this to make it into the next release in a few months.
If we were to put this to the vote today who would actually give this a -1 vote?
Based on the number of identified usecases (some stronger than others)
Also if we were to keep with int keys based on all the discussion thread for the pros vs cons who would give a -1 as it stands? Eg anyone believe it simply wouldn't work.
If not so we make progress would it be sensible to put this to vote? So we can start work?
Cheers
Mike
Sent using OWA for iPhone
________________________________________
From: Jeroen van Disseldorp <je...@axual.io>
Sent: Wednesday, February 1, 2017 2:43:45 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
I'm lagging a bit here too, but want to respond to the usefulness of
this KIP for two use cases: bidirectional mirroring and smart cluster
switches.
Bidirectional mirroring
For a Dutch bank we have currently implemented bidirectional mirroring
using our own headers, which we encode in the Value using a custom
serializer. It encodes just one byte extra, containing a "copy/original"
bit. When applications serialize an avro object, the bit is unset by
default. Our mirroring mechanism subscribes to all topics, checks the
bit, and when unset the bit is set and the message is forwarded to the
other DC.
The overhead of this is pretty low (1 byte) and it's cheap, since the
mirroring mechanism does not need to fully deserialize the value, but
just check+flip one bit in the byte[] value. On the receiving end, the
mirroring mechanism sees the message for the second time, but notices
the message is a copy already and ignores it.
The mechanism has been in production since last year and it works really
well. Its downside however is the incompatibility with avro serialized
values, as the whole organization is mandated to use our own custom
serde and we were forced to create a wrapper client library around the
default kafka client. The callback we offer to applications is very
similar to that proposed in the KIP, as every message our users get
consists of <Metadata, K, V>.
Additionally our header contains a message id and serialization
timestamp (invented before 0.10.0.0). These fields could also be done
away with when this KIP is implemented.
Smart cluster switches
Another case we are considering to use headers for is enabling "smart
cluster switches" for consumers and producers in a way that would avoid
duplicates and/or missed messages in most cases. This involves encoding
a producer id (app name or id), producer instance id (uuid generated
upon producer's start) and sequence number in every message. We expect
to use one sequence per partition.
These fields enable our consumers to keep track of which messages were
seen without relying on offsets, which is a cluster-bound value.
Consumers keep track of seen messages by keeping a small administration
of ranges of sequence numbers per producer instance and partition. Since
sequence numbers are always increasing per producer and partition, the
memory consumption of such administration is pretty low, while being
meaningful across clusters. After every poll() we can commit the
administration to a topic in much the same way as we do with offsets.
The difference is that this topic can be mirrored to the other cluster
and upon a switch of consumer, all instances can use the topic to
initialize themselves and skip all messages already processed.
Hope these views help in the upcoming decision on this KIP.
Regards
Jeroen
On 09/01/2017 20:40, Gwen Shapira wrote:
> Mirroring + ClusterIDs:
>
> Kafka clusters already have an automatically generated ID that is
> exposed to clients:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
>
> It sounds trivial to get MirrorMaker's consumer to grab this ID (which
> always exists) from the source cluster and add it to the message
> headers.
>
> Obviously adding stuff to the message makes it larger so this should
> be a feature you should be able to turn on/off depending on the
> replication requirements. If you need the feature - you pay the price.
> I don't think there is an immediate need to put an effort into
> shrinking the Cluster ID, but others can comment on that.
>
>
>
>
>
> On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
>> Hey Michael,
>>
>> Sorry for lagging, I've been out for the holiday break. What would help me
>> would be to get to a consolidated list of use cases we've worked out and
>> think add real value. I think getting down to the details really matters
>> for these.
>>
>> The two lists I'm aware of are the one in the KIP and the one you gave in
>> this email thread. Both seem to be more at the sketch level. Let me review
>> the use cases and see if I have it right. If so let's get it in the design
>> proposal.
>>
>> Your use cases:
>>
>> 1. Tracing IDs -- I get this use case and think I see how it would work.
>> I think because the tracing is from a third party system there is some
>> value to having a header versus a custom format to make this work fully
>> transparently.
>> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
>> thinking below
>> 3. End-to-end encryption -- I think what you are saying is that if you
>> want "routing' fields that are accessible in transit without decrypting you
>> need a scheme to leave this decrypted.I think the idea is to avoid
>> decrypting to get a header, but I'm not sure if I get how this is better
>> than a custom message format that doesn't encrypt the things you don't want
>> encrypted since you have to be aware of the scheme anyway.
>> 4. EOS -- I'm not convinced this is a good approach for EOS.
>>
>> Linked from KIP:
>>
>> 1. Schema-Registry-aware avro serialization -- not clear how this is
>> better than keeping the schema id in the value
>> 2. Polyglot kafka topics -- marginally better than a container format,
>> but of course you still have to all agree on the the header and
>> header=>serialization mapping so not sure if it's really better than a
>> container format. Also not wild about polyglot topics :-)
>> 3. End-to-End routing trail including mirroring --- I'd put this under
>> tracing ids
>> 4. Cross-service process tracing -- ditto
>> 5. Signing -- I like this use case but it seems to depend on a feature
>> we don't have and aren't proposing adding, right?
>> 6. Per-record custom compression -- why do you want this? compressing
>> per message won't work well compared to batch compression, right?
>> 7. Large Message Support -- I think we can agree that although this
>> might work it's clearly not the right way to support large messages in Kafka
>>
>> Matthias
>>
>> 1. Control messages --- Matthias, not sure if I get the motivation in
>> streams for this. If we did want to do it, wouldn't it be easier to
>> implement and by allowing users to send control messages (in the EOS sense)
>> and adding a consumer config to allow either passing these on or
>> suppressing them?
>>
>> So I if I were to summarize I'd list the use cases as follows:
>>
>> - Much better with headers: tracing IDs
>> - Somewhat better: end-to-end encryption, polyglot topics
>> - Don't buy: Mirroring (though perhaps could be made to work with
>> changes to the proposal), EOS, large messages, compression
>> - Not sure: Control messages
>>
>> Does this match where other people are at? From my point of view for
>> something like this where we are changing the core data model we should
>> ideally get a handful of use cases to the point where they are really well
>> thought out and we know they can use the feature. I think we have one solid
>> one with tracing IDs, and I think maybe mirroring could be another with
>> some work. I'm less sure about the others.
>>
>> Bi-directional Mirroring
>> I think bidirectional mirroring could potentially become a compelling use
>> case if we an find a way to make it work. If I understand it, the proposal
>> is to add a header like "origin-cluster" with the cluster id of the Kafka
>> cluster. Then modify the mirroring tool to disallow mirroring a message
>> back to it's own origin cluster. This basic idea is definitely workable and
>> useful, but can we really get to a good implementation using headers?
>> I imagine the approach would be to add this header in the producer if some
>> config is set. There are two problems:
>>
>> - You add a pretty big honking cluster ID to each message. For small
>> messages this maybe larger than the contents itself.
>> - What happens if someone somewhere forgets to set that config on some
>> client somewhere?
>>
>> If someone forgets to set the config in some client somewhere I guess you
>> have various options in mirror maker:
>>
>> 1. drop the data (pretty bad, now your data is out of sync)
>> 2. crash (pretty bad--how do you recover)
>> 3. insert some default (pretty bad since would also lead to data loss if
>> that's not right)
>>
>> Basically although I think a cluster tag could be a good way to do
>> bi-directional mirroring, I'm not sure this naive approach I described
>> would get us to a usable implementation. This could be fixable. I think if
>> there was some scheme to check required headers on the server that could
>> alleviate the missing header problem. Not really sure what to do about the
>> message bloat problem, but maybe there could be some way to shrink the
>> cluster id?
>>
>> Cheers,
>>
>> -Jay
>>
>>
>> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Mi...@ig.com>
>> wrote:
>>
>>> @Jay, Ismael and a few others.
>>>
>>> Based on the discussion on kip-87, and the additional use case by Matthias.
>>>
>>> Just for clarity do we now have consensus on the earlier points (1)
>>> headers are useful (2) headers should be native in Kafka?
>>>
>>> Cheers
>>> Mike
>>> ________________________________________
>>> From: radai <ra...@gmail.com>
>>> Sent: Monday, December 19, 2016 5:01 PM
>>> To: dev@kafka.apache.org
>>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>>
>>> having said that im perfectly fine with something like a "no payload" flag
>>> (as opposed to null payload), which would mean such a msg is not returned
>>> from poll() unless the user code specifically "opts in" and would serve
>>> only to carry headers (making it a control message)
>>>
>>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com> wrote:
>>>
>>>> if all you want is to tag "end of stream" wouldnt a header on the last
>>> msg
>>>> in the stream be enough?
>>>>
>>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <ma...@confluent.io>
>>>> wrote:
>>>>
>>>>> The details about headers for control messages are still to define. But
>>>>> yes, the idea is to have some common default behavior that clients would
>>>>> need to implement.
>>>>>
>>>>> The point is, that "regular headers" add meta data to regular messages.
>>>>> Thus, those messages will be returned to the user via .poll(). And after
>>>>> the message is received the user can check if meta data is present and
>>>>> read it.
>>>>>
>>>>> For control messages, we do not want those to pop up via .poll() as
>>>>> those are no regular messages. A client would need to opt-in to see
>>>>> those messages (either via poll() or maybe a callback). Thus, we need
>>>>> some special (standardized) header IDs that indicate control messages
>>>>> that should not be returned to the user via poll() by default.
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
>>>>>> Matthias,
>>>>>>
>>>>>> Thanks for your input. I'm +1 on control messages as they seem to be
>>>>> the
>>>>>> simplest way to implement watermarks (
>>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102),
>>> a
>>>>>> feature that would add a lot of value to Kafka Streams IMHO.
>>>>>>
>>>>>> Your argument that the control-message indicator belongs in the
>>>>> client-only
>>>>>> section of the record format make sense. Just to make sure I
>>>>> understand,
>>>>>> are you suggesting that control messages would be indicated by a
>>>>> standard
>>>>>> reserved header? By standard, I mean that ALL Kafka consumers would
>>>>> know
>>>>>> to handle these messages differently (possibly just ignoring them).
>>>>> This
>>>>>> would need to be added to the specification of the consumer protocol
>>> so
>>>>>> that all Kafka clients implement it, right? I think it's a good idea
>>>>> but
>>>>>> just checking.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>>> matthias@confluent.io
>>>>>> wrote:
>>>>>>
>>>>>>> Yes and no. I did overload the term "control message".
>>>>>>>
>>>>>>> EOS control messages are for client-broker communication and thus
>>> never
>>>>>>> exposed to any application. And I think this is a good design because
>>>>>>> broker needs to understand those control messages. Thus, this should
>>> be
>>>>>>> a protocol change.
>>>>>>>
>>>>>>> The type of control messages I have in mind are for client-client
>>>>>>> (application-application) communication and the broker is agnostic to
>>>>>>> them. Thus, it should not be a protocol change.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12/14/16 9:42 AM, radai wrote:
>>>>>>>> arent control messages getting pushed as their own top level
>>> protocol
>>>>>>>> change (and a fairly massive one) for the transactions KIP ?
>>>>>>>>
>>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>>>>> matthias@confluent.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I want to add a completely new angle to this discussion. For this,
>>> I
>>>>>>>>> want to propose an extension for the headers feature that enables
>>> new
>>>>>>>>> uses cases -- and those new use cases might convince people to
>>>>> support
>>>>>>>>> headers (of course including the larger scoped proposal).
>>>>>>>>>
>>>>>>>>> Extended Proposal:
>>>>>>>>>
>>>>>>>>> Allow messages with a certain header key to be special "control
>>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an application
>>>>> via
>>>>>>>>> .poll().
>>>>>>>>>
>>>>>>>>> Thus, a consumer client would automatically skip over those
>>>>> messages. If
>>>>>>>>> an application knows about embedded control messages, it can "sing
>>>>> up"
>>>>>>>>> to those messages by the consumer client and either get a callback
>>> or
>>>>>>>>> the consumer auto-drop for this messages gets disabled (allowing to
>>>>>>>>> consumer those messages via poll()).
>>>>>>>>>
>>>>>>>>> (The details need further considerations/discussion. I just want to
>>>>>>>>> sketch the main idea.)
>>>>>>>>>
>>>>>>>>> Usage:
>>>>>>>>>
>>>>>>>>> There is a shared topic (ie, used by multiple applications) and a
>>>>>>>>> producer application wants to embed a special message in the topic
>>>>> for a
>>>>>>>>> dedicated consumer application. Because only one application will
>>>>>>>>> understand this message, it cannot be a regular message as this
>>> would
>>>>>>>>> break all applications that do not understand this message. The
>>>>> producer
>>>>>>>>> application would set a special metadata key and no consumer
>>>>> application
>>>>>>>>> would see this control message by default because they did not
>>> enable
>>>>>>>>> their consumer client to return this message in poll() (and the
>>>>> client
>>>>>>>>> would just drop this message with special metadata key). Only the
>>>>> single
>>>>>>>>> application that should receive this message, will subscribe to
>>> this
>>>>>>>>> message on its consumer client and process it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Concrete Use Case: Kafka Streams
>>>>>>>>>
>>>>>>>>> In Kafka Streams, we would like to propagate "control messages"
>>> from
>>>>>>>>> subtopology to subtopology. There are multiple scenarios for which
>>>>> this
>>>>>>>>> would be useful. For example, currently we do not guarantee a
>>>>>>>>> "consistent shutdown" of an application. By this, I mean that input
>>>>>>>>> records might not be completely processed by the whole topology
>>>>> because
>>>>>>>>> the application shutdown happens "in between" and an intermediate
>>>>> result
>>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user would see
>>>>> an
>>>>>>>>> committed offset of the source topic of the application, but no
>>>>>>>>> corresponding result record in the output topic.
>>>>>>>>>
>>>>>>>>> Having "shutdown markers" would allow us, to first stop the
>>> upstream
>>>>>>>>> subtopology and write this marker into the intermediate topic and
>>> the
>>>>>>>>> downstream subtopology would only shut down itself after is sees
>>> the
>>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
>>>>>>>>> "in-flight" messages got stuck in intermediate topics.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch Processing).
>>>>>>>>> There was a discussion about the proposed metadata topic, and we
>>>>> could
>>>>>>>>> avoid this metadata topic if we would have "control messages".
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Right now, we cannot insert an "application control message"
>>> because
>>>>>>>>> Kafka Streams does not own all topics it read/writes and thus might
>>>>>>>>> break other consumer application (as described above) if we inject
>>>>>>>>> random messages that are not understood by other apps.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Of course, one can work around "embedded control messaged" by using
>>>>> an
>>>>>>>>> additional topic to propagate control messaged between application
>>>>> (as
>>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
>>>>> there
>>>>>>>>> are major concerns about adding this metadata topic in the KIP and
>>>>> this
>>>>>>>>> shows that other application that need a similar pattern might
>>> profit
>>>>>>>>> from topic embedded "control messages", too.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> One last important consideration: those "control messages" are used
>>>>> for
>>>>>>>>> client to client communication and are not understood by the
>>> broker.
>>>>>>>>> Thus, those messages should not be enabled within the message
>>> format
>>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
>>>>> headers
>>>>>>>>> would be a nice way to implement them. Because KIP-82 did consider
>>>>> key
>>>>>>>>> namespaces for metatdata keys, this extension should not be an own
>>>>> KIP
>>>>>>>>> but should be included in KIP-82 to reserve a namespace for
>>> "control
>>>>>>>>> message" in the first place.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry for the long email... Looking forward to your feedback.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>>>>>>>>>> Hi Jun
>>>>>>>>>>
>>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
>>>>> (HTTP/JMS/
>>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
>>> (though I
>>>>>>>>> believe it contains the end2end uuid embedded in this id), on
>>>>> receiving
>>>>>>> the
>>>>>>>>> message at the receiving JVM the apm code takes this out, and
>>>>> continues
>>>>>>> its
>>>>>>>>> tracing on the that new thread. Both JVM's (and other languages the
>>>>> APM
>>>>>>>>> tool supports) send this data async back to the central controllers
>>>>>>> where
>>>>>>>>> the stiching togeather occurs. For this they need some header space
>>>>> for
>>>>>>>>> them to put this id.
>>>>>>>>>> 101) Yes indeed we have a business transaction Id in the payload.
>>>>>>> Though
>>>>>>>>> this is a system level tracing, that we need to have marry up. Also
>>>>> as
>>>>>>> per
>>>>>>>>> note on end2end encryption we'd be unable to prove the flow if the
>>>>>>> payload
>>>>>>>>> is encrypted as we'd not have access to this at certain points of
>>> the
>>>>>>> flow
>>>>>>>>> through the infrastructure/platform.
>>>>>>>>>>
>>>>>>>>>> 103) As said we use this mechanism in IG very successfully, as
>>>>> stated
>>>>>>>>> per key we guarantee the transaction producing app to handle the
>>>>>>>>> transaction of a key at one DC unless at point of critical failure
>>>>>>> where we
>>>>>>>>> have to flip processing to another. We care about key ordering.
>>>>>>>>>> I disagree on the offset comment for the partition solution unless
>>>>> you
>>>>>>>>> do full ISR, or expensive full XA transactions even with partitions
>>>>> you
>>>>>>>>> cannot fully guarantee offsets would match.
>>>>>>>>>> 105) Very much so, I need to have access at the platform level to
>>>>> the
>>>>>>>>> other meta data all mentioned, without having to need to have
>>> access
>>>>> to
>>>>>>> the
>>>>>>>>> encryption keys of the payload.
>>>>>>>>>> 106)
>>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we'd need to have
>>> a
>>>>>>>>> global producerId register which would be very hard to
>>>>> enforce/ensure is
>>>>>>>>> current and correct, just to understand the message origins of its
>>>>>>>>> region/az/cluster for routing.
>>>>>>>>>> The client wrapper version, producerId can be the same, as
>>> obviously
>>>>>>> the
>>>>>>>>> producer could upgrade its wrapper, as such we need to know what
>>>>> wrapper
>>>>>>>>> version the message is created with.
>>>>>>>>>> Likewise the IP address, as stated we can have our producer move,
>>>>> where
>>>>>>>>> its IP would change.
>>>>>>>>>> 107)
>>>>>>>>>> UUID is set on the message by interceptors before actual producer
>>>>>>>>> transport send. This is for platform level message dedupe
>>> guarantee,
>>>>> the
>>>>>>>>> business payload should be agnostic to this. Please see
>>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
>>>>> ection.html
>>>>>>>>> note this is not touching business payloads.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi, Michael,
>>>>>>>>>>
>>>>>>>>>> Thanks for the reply. I find it very helpful.
>>>>>>>>>>
>>>>>>>>>> Data lineage:
>>>>>>>>>> 100. I'd like to understand the APM use case a bit more. It
>>>>> sounds
>>>>>>>>> like
>>>>>>>>>> that those APM plugins can generate a transaction id that we
>>>>> could
>>>>>>>>>> potentially put in the header of every message. How would you
>>>>>>>>> typically
>>>>>>>>>> make use of such transaction ids? Are there other metadata
>>>>>>>>> associated with
>>>>>>>>>> the transaction id and if so, how are they propagated
>>>>> downstream?
>>>>>>>>>> 101. For the finance use case, if the concept of transaction
>>> is
>>>>>>>>> important,
>>>>>>>>>> wouldn't it be typically included in the message payload
>>>>> instead of
>>>>>>>>> as an
>>>>>>>>>> optional header field?
>>>>>>>>>>
>>>>>>>>>> 102. The data lineage that Altas and Navigator support seems
>>> to
>>>>> be
>>>>>>>>> at the
>>>>>>>>>> dataset level, not per record level? So, not sure if per
>>> message
>>>>>>>>> headers
>>>>>>>>>> are relevant there.
>>>>>>>>>>
>>>>>>>>>> Mirroring:
>>>>>>>>>> 103. The benefit of using separate partitions is that it
>>>>>>> potentially
>>>>>>>>> makes
>>>>>>>>>> it easy to preserve offsets during mirroring. This will make
>>> it
>>>>>>>>> easier for
>>>>>>>>>> consumer to switch clusters. Currently, the consumers can
>>> switch
>>>>>>>>> clusters
>>>>>>>>>> by using the timestampToOffset() api, but it has to deal with
>>>>>>>>> duplicates.
>>>>>>>>>> Good point on the issue with log compact and I am not sure how
>>>>> to
>>>>>>>>> address
>>>>>>>>>> this. However, even if we mirror into the existing partitions,
>>>>> the
>>>>>>>>> ordering
>>>>>>>>>> for messages generated from different clusters seems
>>>>>>>>> non-deterministic
>>>>>>>>>> anyway. So, it seems that the consumers already have to deal
>>>>> with
>>>>>>>>> that? If
>>>>>>>>>> a topic is compacted, does that mean which messages are
>>>>> preserved
>>>>>>> is
>>>>>>>>> also
>>>>>>>>>> non-deterministic across clusters?
>>>>>>>>>>
>>>>>>>>>> 104. Good point on partition key.
>>>>>>>>>>
>>>>>>>>>> End-to-end encryption:
>>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are headers
>>>>>>>>> useful there?
>>>>>>>>>> Auditing:
>>>>>>>>>> 106. It seems other than the UUID, all other metadata are per
>>>>>>>>> producer?
>>>>>>>>>> EOS:
>>>>>>>>>> 107. How are those UUIDs generated? I am not sure if they can
>>> be
>>>>>>>>> generated
>>>>>>>>>> in the producer library. An application may send messages
>>>>> through a
>>>>>>>>> load
>>>>>>>>>> balancer and on retry, the same message could be routed to a
>>>>>>>>> different
>>>>>>>>>> producer instance. So, it seems that the application has to
>>>>>>> generate
>>>>>>>>> the
>>>>>>>>>> UUIDs. In that case, shouldn't the application just put the
>>>>> UUID in
>>>>>>>>> the
>>>>>>>>>> payload?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Jun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>>>>>>>>> Michael.Pearce@ig.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> > Hi Jun.
>>>>>>>>>> >
>>>>>>>>>> > Per Transaction Tracing / Data Lineage.
>>>>>>>>>> >
>>>>>>>>>> > As Stated in the KIP this has the first use case of how many
>>>>> APM
>>>>>>>>> tools now
>>>>>>>>>> > work.
>>>>>>>>>> > I would find it impossible for any one to argue this is not
>>>>>>>>> important or a
>>>>>>>>>> > niche market as it has its own gartner report for this
>>> space.
>>>>>>> Such
>>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular are
>>>>> but a
>>>>>>>>> few.
>>>>>>>>>> >
>>>>>>>>>> > Likewise these APM tools can help very rapidly track down
>>>>> issues
>>>>>>>>> and
>>>>>>>>>> > automatically capture metrics, perform actions based on
>>>>>>> unexpected
>>>>>>>>> behavior
>>>>>>>>>> > to auto recover services.
>>>>>>>>>> >
>>>>>>>>>> > Before mentioning looking at aggregated stats, in these
>>> cases
>>>>>>> where
>>>>>>>>>> > actually on critical flows we cannot afford to have
>>> aggregated
>>>>>>>>> rolled up
>>>>>>>>>> > stats only.
>>>>>>>>>> >
>>>>>>>>>> > With the APM tool we use its actually able to detect a
>>> single
>>>>>>>>> transaction
>>>>>>>>>> > failure and capture the thread traces in the JVM where it
>>>>> failed
>>>>>>>>> and
>>>>>>>>>> > everything for us, to the point it sends us alerts where we
>>>>> have
>>>>>>>>> this
>>>>>>>>>> > giving the line number of the code that caused it, the
>>>>>>> transaction
>>>>>>>>> trace
>>>>>>>>>> > through all the services and endpoints (supported) upto the
>>>>> point
>>>>>>>>> of
>>>>>>>>>> > failure, it can also capture the data in and out (so we can
>>>>>>>>> replay).
>>>>>>>>>> > Because atm Kafka doesn't support us being able to stich in
>>>>> these
>>>>>>>>> tracing
>>>>>>>>>> > transaction ids natively, we cannot get these benefits as
>>>>> such is
>>>>>>>>> limiting
>>>>>>>>>> > our ability support apps and monitor them to the same
>>>>> standards
>>>>>>> we
>>>>>>>>> come to
>>>>>>>>>> > expect when on a kafka flow.
>>>>>>>>>> >
>>>>>>>>>> > This actually ties in with Data Lineage, as the same tracing
>>>>> can
>>>>>>>>> be used
>>>>>>>>>> > to back stich this. Essentially many times due to the sums
>>> of
>>>>>>> money
>>>>>>>>>> > involved there are disputes, and typically as a financial
>>>>>>>>> institute the
>>>>>>>>>> > easiest and cleanest way to prove when disputes arise is to
>>>>>>>>> present the
>>>>>>>>>> > actual flow and processes involved in a transaction.
>>>>>>>>>> >
>>>>>>>>>> > Likewise as Hadoop matures its evident this case is
>>>>> important, as
>>>>>>>>> tools
>>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera led)
>>>>> are
>>>>>>>>> evident
>>>>>>>>>> > also I believe the importance here is very much NOT just a
>>>>>>>>> financial issue.
>>>>>>>>>> >
>>>>>>>>>> > From a MDM point of view any company wanting to care about
>>>>> Data
>>>>>>>>> Quality
>>>>>>>>>> > and Data Governance - Data Lineage is a key piece in this
>>>>> puzzle.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE Mirroring,
>>>>>>>>>> >
>>>>>>>>>> > As per the KIP in-fact this is exactly what we do re cluster
>>>>> id,
>>>>>>>>> to mirror
>>>>>>>>>> > a network of clusters between AZ's / Regions. We know a
>>>>>>>>> transaction for a
>>>>>>>>>> > key will be done within a AZ/Region, as such we know the
>>>>> write
>>>>>>> to
>>>>>>>>> kafka
>>>>>>>>>> > would be ordered per key. But we need eventual view of that
>>>>>>> across
>>>>>>>>> in our
>>>>>>>>>> > other regions/az's. When we have complete AZ or Region
>>>>> failure we
>>>>>>>>> know
>>>>>>>>>> > there will be a brief interruption whilst those transactions
>>>>> are
>>>>>>>>> moved to
>>>>>>>>>> > another region but we expect after it to continue.
>>>>>>>>>> >
>>>>>>>>>> > As mentioned having separate Partions to do this starts to
>>> get
>>>>>>>>>> > ugly/complicated for us:
>>>>>>>>>> > how would I do compaction where a key is in two partitions?
>>>>>>>>>> > How do we balance consumers so where multiple partitions
>>> with
>>>>> the
>>>>>>>>> same key
>>>>>>>>>> > goto the same consumer
>>>>>>>>>> > What do you do if cluster 1 has 5 partitions but cluster 20
>>>>> has
>>>>>>> 10
>>>>>>>>> because
>>>>>>>>>> > its larger kit in our more core DC's, as such key to
>>> partition
>>>>>>>>> mappings for
>>>>>>>>>> > consumers get even more complicated.
>>>>>>>>>> > What do you do if we add or remove a complete region
>>>>>>>>>> >
>>>>>>>>>> > Where as simple mirror will work we just need to ensure we
>>>>> don't
>>>>>>>>> have a
>>>>>>>>>> > cycle which we can do with clusterId.
>>>>>>>>>> >
>>>>>>>>>> > We even have started to look at shortest path mirror routing
>>>>>>> based
>>>>>>>>> on
>>>>>>>>>> > clusterId, if we also had the region and az info on the
>>>>>>> originating
>>>>>>>>>> > message, this we have not implemented but some ideas come
>>> from
>>>>>>>>> network
>>>>>>>>>> > routing, and also the dispatcher router in apache qpid.
>>>>>>>>>> >
>>>>>>>>>> > Also we need to have data perimeters e.g. certain data
>>> cannot
>>>>>>> leave
>>>>>>>>>> > certain countries borders. We want this all automated so
>>> that
>>>>> at
>>>>>>>>> the
>>>>>>>>>> > platform level without having to touch or look at the
>>> business
>>>>>>>>> data inside
>>>>>>>>>> > we can have headers we can put tags into so that we can
>>> ensure
>>>>>>>>> this doesn't
>>>>>>>>>> > occur when we mirror. (actually links in to data lineage /
>>>>>>> tracing
>>>>>>>>> as again
>>>>>>>>>> > we need to tag messages at a platform level) Examples are we
>>>>> are
>>>>>>>>> not
>>>>>>>>>> > allowed Private customer details to leave Switzerland, yet
>>> we
>>>>>>> need
>>>>>>>>> those
>>>>>>>>>> > systems integrated.
>>>>>>>>>> >
>>>>>>>>>> > Lastly around mirroring we have a partionKey field, as the
>>> key
>>>>>>>>> used for
>>>>>>>>>> > portioning logic != compaction key all the time but we want
>>> to
>>>>>>>>> preserve it
>>>>>>>>>> > for when we mirror so that if source cluster partition count
>>>>> !=
>>>>>>>>> destination
>>>>>>>>>> > cluster partition count we can honour the same partitioning
>>>>>>> logic.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE End 2 End encryption
>>>>>>>>>> >
>>>>>>>>>> > As I believe mentioned just before, the solution you mention
>>>>> just
>>>>>>>>> doesn't
>>>>>>>>>> > cut the mustard these days with many regulators. An
>>> operations
>>>>>>>>> person with
>>>>>>>>>> > access to the box should not be able to have access to the
>>>>> data.
>>>>>>>>> Many now
>>>>>>>>>> > actually impose quite literally the implementation expected
>>>>> being
>>>>>>>>> end2end
>>>>>>>>>> > encryption for certain data (Singapore for us is one that I
>>> am
>>>>>>>>> most aware
>>>>>>>>>> > of). In fact we're even now needing encrypt the data and
>>> store
>>>>>>> the
>>>>>>>>> keys in
>>>>>>>>>> > HSM modules.
>>>>>>>>>> >
>>>>>>>>>> > Likewise the performance penalty on encrypting decrypting as
>>>>> you
>>>>>>>>> produce
>>>>>>>>>> > over wire, then again encrypt decrypt as the data is stored
>>> on
>>>>>>> the
>>>>>>>>> brokers
>>>>>>>>>> > disks and back again, then again encrypted and decrypted
>>> back
>>>>>>> over
>>>>>>>>> the wire
>>>>>>>>>> > each time for each consumer all adds up, ignoring this
>>>>> doubling
>>>>>>>>> with mirror
>>>>>>>>>> > makers etc. simply encrypting the value once on write by the
>>>>>>>>> client and
>>>>>>>>>> > again decrypting on consume by the consumer is far more
>>>>>>>>> performant, but
>>>>>>>>>> > then the routing and platform meta data needs to be separate
>>>>>>> (thus
>>>>>>>>> headers)
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE Auditing:
>>>>>>>>>> >
>>>>>>>>>> > Our Auditing needs are:
>>>>>>>>>> > Producer Id,
>>>>>>>>>> > Origin Cluster Id that message first produced into
>>>>>>>>>> > Origin AZ - agreed we can derive this if we have cluster id,
>>>>> but
>>>>>>>>> it makes
>>>>>>>>>> > resolving this for audit reporting a lot easier.
>>>>>>>>>> > Origin Region - agreed we can derive this if we have cluster
>>>>> id,
>>>>>>>>> but it
>>>>>>>>>> > makes resolving this for audit reporting a lot easier.
>>>>>>>>>> > Unique Message Identification (this is not the same as
>>>>>>> transaction
>>>>>>>>>> > tracing) - note offset and partition are not the same, as
>>>>> when we
>>>>>>>>> mirror or
>>>>>>>>>> > have for what ever system failure duplicate send,
>>>>>>>>>> > Custom Client wrapper version (where organizations have to
>>>>> wrap
>>>>>>>>> the kafka
>>>>>>>>>> > client for added features) so we know what version of the
>>>>> wrapper
>>>>>>>>> is used
>>>>>>>>>> > Producer IP address (in case of clients being in our vm/open
>>>>>>> stack
>>>>>>>>> infra
>>>>>>>>>> > where they can move around, producer id will stay the same
>>> but
>>>>>>>>> this would
>>>>>>>>>> > change)
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE Once and only once delivery case
>>>>>>>>>> >
>>>>>>>>>> > Using the same Message UUID for auditing we can achieve this
>>>>>>> quite
>>>>>>>>> simply.
>>>>>>>>>> >
>>>>>>>>>> > As per how some other brokers do this (cough qpid, artemis)
>>>>>>>>> message uuid
>>>>>>>>>> > are used to dedupe where message is sent and produced but
>>> the
>>>>>>>>> client didn't
>>>>>>>>>> > receive the ack, and there for replays the send, by having a
>>>>>>>>> unique message
>>>>>>>>>> > id per message, this can be filtered out, on consumers where
>>>>>>>>> message
>>>>>>>>>> > delivery may occur twice for what ever reasons a message
>>> uuid
>>>>> can
>>>>>>>>> be used
>>>>>>>>>> > to remove duplicates being deliverd , like wise we can do
>>>>> this in
>>>>>>>>> the
>>>>>>>>>> > mirrormakers so if we detect a dupe message we can avoid
>>>>>>>>> replicating it.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Cheers
>>>>>>>>>> > Mike
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Since this KIP affects message format, wire protocol,
>>>>> apis, I
>>>>>>>>> think
>>>>>>>>>> > it's
>>>>>>>>>> > worth spending a bit more time to nail down the concrete
>>>>> use
>>>>>>>>> cases. It
>>>>>>>>>> > would be bad if we add this feature, but when start
>>>>>>>>> implementing it
>>>>>>>>>> > for say
>>>>>>>>>> > mirroring, we then realize that header is not the best
>>>>>>>>> approach.
>>>>>>>>>> > Initially,
>>>>>>>>>> > I thought I was convinced of the use cases of headers
>>> and
>>>>> was
>>>>>>>>> trying to
>>>>>>>>>> > write down a few use cases to convince others. That's
>>>>> when I
>>>>>>>>> became
>>>>>>>>>> > less
>>>>>>>>>> > certain. For me to be convinced, I just want to see two
>>>>>>> strong
>>>>>>>>> use
>>>>>>>>>> > cases
>>>>>>>>>> > (instead of 10 maybe use cases) in the third-party
>>> space.
>>>>> The
>>>>>>>>> reason is
>>>>>>>>>> > that when we discussed the use cases within a company,
>>>>> often
>>>>>>>>> it ends
>>>>>>>>>> > with
>>>>>>>>>> > "we can't force everyone to use this standard since we
>>> may
>>>>>>>>> have to
>>>>>>>>>> > integrate with third-party tools".
>>>>>>>>>> >
>>>>>>>>>> > At present, I am not sure why headers are useful for
>>>>> things
>>>>>>>>> like
>>>>>>>>>> > schemaId
>>>>>>>>>> > or encryption. In order to do anything useful to the
>>>>> value,
>>>>>>>>> one needs
>>>>>>>>>> > to
>>>>>>>>>> > know the schemaId or how data is encrypted, but header
>>> is
>>>>>>>>> optional.
>>>>>>>>>> > But, I
>>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
>>> provides
>>>>>>> more
>>>>>>>>> details
>>>>>>>>>> > on
>>>>>>>>>> > the argument.
>>>>>>>>>> >
>>>>>>>>>> > I am not very sure header is the best approach for
>>>>> mirroring
>>>>>>>>> either. If
>>>>>>>>>> > someone has thought about this more, I'd be happy to
>>> hear.
>>>>>>>>>> >
>>>>>>>>>> > I can see the data lineage use case. I am just not sure
>>>>> how
>>>>>>>>> widely
>>>>>>>>>> > applicable this is. If someone familiar with this space
>>>>> can
>>>>>>>>> justify
>>>>>>>>>> > this is
>>>>>>>>>> > a significant use case, say in the finance industry,
>>> this
>>>>>>>>> would be a
>>>>>>>>>> > strong
>>>>>>>>>> > use case.
>>>>>>>>>> >
>>>>>>>>>> > I can see the auditing use case. I am just not sure if a
>>>>>>> native
>>>>>>>>>> > producer id
>>>>>>>>>> > solves that problem. If there are additional metadata
>>>>> that's
>>>>>>>>> worth
>>>>>>>>>> > collecting but not covered by the producer id, that
>>> would
>>>>>>> make
>>>>>>>>> this a
>>>>>>>>>> > strong use case.
>>>>>>>>>> >
>>>>>>>>>> > Thanks,
>>>>>>>>>> >
>>>>>>>>>> > Jun
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>>> > wrote:
>>>>>>>>>> >
>>>>>>>>>> > > this KIP is about enabling headers, nothing more
>>> nothing
>>>>>>>>> less - so
>>>>>>>>>> > no,
>>>>>>>>>> > > broker-side use of headers is not in the KIP scope.
>>>>>>>>>> > >
>>>>>>>>>> > > obviously though, once you have headers potential use
>>>>> cases
>>>>>>>>> could
>>>>>>>>>> > include
>>>>>>>>>> > > broker-side header-aware interceptors (which would be
>>>>> the
>>>>>>>>> topic of
>>>>>>>>>> > other
>>>>>>>>>> > > future KIPs).
>>>>>>>>>> > >
>>>>>>>>>> > > a trivially clear use case (to me) would be using such
>>>>>>>>> broker-side
>>>>>>>>>> > > interceptors to enforce compliance with organizational
>>>>>>>>> policies - it
>>>>>>>>>> > would
>>>>>>>>>> > > make our SREs lives much easier if instead of
>>>>> retroactively
>>>>>>>>>> > discovering
>>>>>>>>>> > > "rogue" topics/users those messages would have been
>>>>>>> rejected
>>>>>>>>>> > up-front.
>>>>>>>>>> > >
>>>>>>>>>> > > the kafka broker code is lacking any such
>>> extensibility
>>>>>>>>> support
>>>>>>>>>> > (beyond
>>>>>>>>>> > > maybe authorizer) which is why these use cases were
>>> left
>>>>>>> out
>>>>>>>>> of the
>>>>>>>>>> > "case
>>>>>>>>>> > > for headers" doc - broker extensibility is a separate
>>>>>>>>> discussion.
>>>>>>>>>> > >
>>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
>>>>>>>>> gwen@confluent.io>
>>>>>>>>>> > wrote:
>>>>>>>>>> > >
>>>>>>>>>> > > > Woah, I wasn't aware this is something we'll do. It
>>>>>>> wasn't
>>>>>>>>> in the
>>>>>>>>>> > KIP,
>>>>>>>>>> > > > right?
>>>>>>>>>> > > >
>>>>>>>>>> > > > I guess we could do it the same way ACLs currently
>>>>> work.
>>>>>>>>>> > > > I had in mind something that will allow admins to
>>>>> apply
>>>>>>>>> rules to
>>>>>>>>>> > the
>>>>>>>>>> > > > new create/delete/config topic APIs. So Todd can
>>>>> decide
>>>>>>> to
>>>>>>>>> reject
>>>>>>>>>> > > > "create topic" requests that ask for more than 40
>>>>>>>>> partitions, or
>>>>>>>>>> > > > require exactly 3 replicas, or no more than 50GB
>>>>>>> partition
>>>>>>>>> size,
>>>>>>>>>> > etc.
>>>>>>>>>> > > >
>>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are planning to
>>>>> apply
>>>>>>>>> more
>>>>>>>>>> > rules
>>>>>>>>>> > > > to requests (and I think we should), we may want a
>>> bit
>>>>>>>>> more generic
>>>>>>>>>> > > > design around that.
>>>>>>>>>> > > >
>>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>>> > > wrote:
>>>>>>>>>> > > > > "wouldn't you be in the business of making sure
>>>>>>> everyone
>>>>>>>>> uses
>>>>>>>>>> > them
>>>>>>>>>> > > > > properly?"
>>>>>>>>>> > > > >
>>>>>>>>>> > > > > thats where a broker-side plugin would come handy
>>> -
>>>>> any
>>>>>>>>> incoming
>>>>>>>>>> > > message
>>>>>>>>>> > > > > that does not conform to org policy (read - does
>>> not
>>>>>>>>> have the
>>>>>>>>>> > proper
>>>>>>>>>> > > > > headers) gets thrown out (with an error returned
>>> to
>>>>>>> user)
>>>>>>>>>> > > > >
>>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
>>>>>>>>> tpalino@gmail.com>
>>>>>>>>>> > wrote:
>>>>>>>>>> > > > >
>>>>>>>>>> > > > >> Come on, I've done at least 2 talks on this one
>>> :)
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> Producing counts to a topic is part of it, but
>>>>> that's
>>>>>>>>> only
>>>>>>>>>> > part. So
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> count you have 100 messages in topic A. When you
>>>>>>> mirror
>>>>>>>>> topic A
>>>>>>>>>> > to
>>>>>>>>>> > > > another
>>>>>>>>>> > > > >> cluster, you have 99 messages. Where was your
>>>>> problem?
>>>>>>>>> Or
>>>>>>>>>> > worse, you
>>>>>>>>>> > > > have
>>>>>>>>>> > > > >> 100 messages, but one producer duplicated
>>> messages
>>>>> and
>>>>>>>>> another
>>>>>>>>>> > one
>>>>>>>>>> > > lost
>>>>>>>>>> > > > >> messages. You need details about where the
>>> message
>>>>>>> came
>>>>>>>>> from in
>>>>>>>>>> > order
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> pinpoint problems when they happen. Source
>>> producer
>>>>>>>>> info, where
>>>>>>>>>> > it was
>>>>>>>>>> > > > >> produced into your infrastructure, and when it
>>> was
>>>>>>>>> produced.
>>>>>>>>>> > This
>>>>>>>>>> > > > requires
>>>>>>>>>> > > > >> you to add the information to the message.
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> And yes, you still need to maintain your clients.
>>>>> So
>>>>>>>>> maybe my
>>>>>>>>>> > original
>>>>>>>>>> > > > >> example was not the best. My thoughts on not
>>>>> wanting
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>> > responsible
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> message formats stands, because that's very much
>>>>>>>>> separate from
>>>>>>>>>> > the
>>>>>>>>>> > > > client.
>>>>>>>>>> > > > >> As you know, we have our own internal client
>>>>> library
>>>>>>>>> that can
>>>>>>>>>> > insert
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> right headers, and right now inserts the right
>>>>> audit
>>>>>>>>>> > information into
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> message fields. If they exist, and assuming the
>>>>>>> message
>>>>>>>>> is Avro
>>>>>>>>>> > > encoded.
>>>>>>>>>> > > > >> What if someone wants to use JSON instead for a
>>>>> good
>>>>>>>>> reason?
>>>>>>>>>> > What if
>>>>>>>>>> > > > user X
>>>>>>>>>> > > > >> wants to encrypt messages, but user Y does not?
>>>>>>>>> Maintaining the
>>>>>>>>>> > client
>>>>>>>>>> > > > >> library is still much easier than maintaining the
>>>>>>>>> message
>>>>>>>>>> > formats.
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> -Todd
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
>>>>>>>>> gwen@confluent.io
>>>>>>>>>> > >
>>>>>>>>>> > > wrote:
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> > Based on your last sentence, consider me
>>>>> convinced
>>>>>>> :)
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > I get why headers are critical for Mirroring
>>> (you
>>>>>>>>> need tags to
>>>>>>>>>> > > prevent
>>>>>>>>>> > > > >> > loops and sometimes to route messages to the
>>>>> correct
>>>>>>>>>> > destination).
>>>>>>>>>> > > > >> > But why do you need headers to audit? We are
>>>>>>> auditing
>>>>>>>>> by
>>>>>>>>>> > producing
>>>>>>>>>> > > > >> > counts to a side topic (and I was under the
>>>>>>>>> impression you do
>>>>>>>>>> > the
>>>>>>>>>> > > > >> > same), so we never need to modify the message.
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > Another thing - after we added headers,
>>> wouldn't
>>>>> you
>>>>>>>>> be in the
>>>>>>>>>> > > > >> > business of making sure everyone uses them
>>>>> properly?
>>>>>>>>> Making
>>>>>>>>>> > sure
>>>>>>>>>> > > > >> > everyone includes the right headers you need,
>>> not
>>>>>>>>> using the
>>>>>>>>>> > header
>>>>>>>>>> > > > >> > names you intend to use, etc. I don't think the
>>>>>>>>> "policing"
>>>>>>>>>> > business
>>>>>>>>>> > > > >> > will ever go away.
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <
>>>>>>>>>> > tpalino@gmail.com>
>>>>>>>>>> > > > wrote:
>>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
>>> with
>>>>> the
>>>>>>>>>> > workaround.
>>>>>>>>>> > > Avro
>>>>>>>>>> > > > >> means
>>>>>>>>>> > > > >> > > that I have to be concerned with the format
>>> of
>>>>> the
>>>>>>>>> messages
>>>>>>>>>> > in
>>>>>>>>>> > > > order to
>>>>>>>>>> > > > >> > run
>>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring, etc.).
>>>>> That
>>>>>>>>> means
>>>>>>>>>> > that I
>>>>>>>>>> > > have
>>>>>>>>>> > > > to
>>>>>>>>>> > > > >> > > handle the schemas, and I have to enforce
>>> rules
>>>>>>>>> about good
>>>>>>>>>> > > formats.
>>>>>>>>>> > > > >> This
>>>>>>>>>> > > > >> > is
>>>>>>>>>> > > > >> > > not something I want to be in the business
>>> of,
>>>>>>>>> because I
>>>>>>>>>> > should be
>>>>>>>>>> > > > able
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > > run a service infrastructure without needing
>>>>> to be
>>>>>>>>> in the
>>>>>>>>>> > weeds of
>>>>>>>>>> > > > >> > dealing
>>>>>>>>>> > > > >> > > with customer data formats.
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > > Trust me, a sizable portion of my support
>>> time
>>>>> is
>>>>>>>>> spent
>>>>>>>>>> > dealing
>>>>>>>>>> > > with
>>>>>>>>>> > > > >> > schema
>>>>>>>>>> > > > >> > > issues. I really would like to get away from
>>>>> that.
>>>>>>>>> Maybe
>>>>>>>>>> > I'd have
>>>>>>>>>> > > > more
>>>>>>>>>> > > > >> > time
>>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > > -Todd
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <
>>>>>>>>>> > gwen@confluent.io>
>>>>>>>>>> > > > wrote:
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
>>>>> workarounds
>>>>>>>>> (Avro
>>>>>>>>>> > container
>>>>>>>>>> > > > >> > >> format), so I'm not too excited about the
>>>>> extra
>>>>>>>>> work
>>>>>>>>>> > required to
>>>>>>>>>> > > do
>>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't mind it
>>>>> if
>>>>>>>>> you do
>>>>>>>>>> > it...
>>>>>>>>>> > > > >> > >> I think the Apache convention for "good
>>> idea,
>>>>> but
>>>>>>>>> not
>>>>>>>>>> > willing to
>>>>>>>>>> > > > put
>>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway, that's
>>>>> what
>>>>>>> I
>>>>>>>>> was
>>>>>>>>>> > trying to
>>>>>>>>>> > > > >> > >> convey :)
>>>>>>>>>> > > > >> > >>
>>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino
>>> <
>>>>>>>>>> > tpalino@gmail.com>
>>>>>>>>>> > > > >> wrote:
>>>>>>>>>> > > > >> > >> > Well I guess my question for you, then, is
>>>>> what
>>>>>>>>> is
>>>>>>>>>> > holding you
>>>>>>>>>> > > > back
>>>>>>>>>> > > > >> > from
>>>>>>>>>> > > > >> > >> > full support for headers? What's the bit
>>>>> that
>>>>>>>>> you're
>>>>>>>>>> > missing
>>>>>>>>>> > > that
>>>>>>>>>> > > > >> has
>>>>>>>>>> > > > >> > you
>>>>>>>>>> > > > >> > >> > under a full +1?
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> > -Todd
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
>>>>> Shapira <
>>>>>>>>>> > > gwen@confluent.io>
>>>>>>>>>> > > > >> > wrote:
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> >> I know why people who support headers
>>>>> support
>>>>>>>>> them, and
>>>>>>>>>> > I've
>>>>>>>>>> > > > seen
>>>>>>>>>> > > > >> > what
>>>>>>>>>> > > > >> > >> >> the discussion is like.
>>>>>>>>>> > > > >> > >> >>
>>>>>>>>>> > > > >> > >> >> This is why I'm asking people who are
>>>>> against
>>>>>>>>> headers
>>>>>>>>>> > > > (especially
>>>>>>>>>> > > > >> > >> >> committers) what will make them change
>>>>> their
>>>>>>>>> mind - so
>>>>>>>>>> > we can
>>>>>>>>>> > > > get
>>>>>>>>>> > > > >> > this
>>>>>>>>>> > > > >> > >> >> part over one way or another.
>>>>>>>>>> > > > >> > >> >>
>>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at Radai,
>>>>> Jun
>>>>>>>>> or you
>>>>>>>>>> > (Todd)...
>>>>>>>>>> > > > I am
>>>>>>>>>> > > > >> > >> >> just looking for something concrete we
>>> can
>>>>> do
>>>>>>>>> to move
>>>>>>>>>> > the
>>>>>>>>>> > > > >> discussion
>>>>>>>>>> > > > >> > >> >> along to the yummy design details (which
>>> is
>>>>>>> the
>>>>>>>>>> > argument I
>>>>>>>>>> > > > really
>>>>>>>>>> > > > >> am
>>>>>>>>>> > > > >> > >> >> looking forward to).
>>>>>>>>>> > > > >> > >> >>
>>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
>>>>> Palino <
>>>>>>>>>> > > tpalino@gmail.com>
>>>>>>>>>> > > > >> > wrote:
>>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even though
>>>>> I'm
>>>>>>>>> not a
>>>>>>>>>> > > > committer)...
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > I have always been a strong supporter
>>> of
>>>>>>>>> introducing
>>>>>>>>>> > the
>>>>>>>>>> > > > concept
>>>>>>>>>> > > > >> > of an
>>>>>>>>>> > > > >> > >> >> > envelope to messages, which headers
>>>>>>>>> accomplishes. The
>>>>>>>>>> > > message
>>>>>>>>>> > > > key
>>>>>>>>>> > > > >> > is
>>>>>>>>>> > > > >> > >> >> > already an example of a piece of
>>> envelope
>>>>>>>>>> > information. By
>>>>>>>>>> > > > >> > providing a
>>>>>>>>>> > > > >> > >> >> means
>>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself, and not
>>>>>>>>> relying on
>>>>>>>>>> > use-case
>>>>>>>>>> > > > >> > specific
>>>>>>>>>> > > > >> > >> >> > implementations, you make it much
>>> easier
>>>>> for
>>>>>>>>>> > components to
>>>>>>>>>> > > > >> > >> interoperate.
>>>>>>>>>> > > > >> > >> >> It
>>>>>>>>>> > > > >> > >> >> > simplifies development of all these
>>>>> things
>>>>>>>>> (message
>>>>>>>>>> > routing,
>>>>>>>>>> > > > >> > auditing,
>>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each one does
>>>>> not
>>>>>>>>> have to
>>>>>>>>>> > reinvent
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> wheel.
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > It also makes it much easier from a
>>>>> client
>>>>>>>>> point of
>>>>>>>>>> > view if
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> headers
>>>>>>>>>> > > > >> > >> >> are
>>>>>>>>>> > > > >> > >> >> > defined as part of the protocol and/or
>>>>>>>>> message format
>>>>>>>>>> > in
>>>>>>>>>> > > > general
>>>>>>>>>> > > > >> > >> because
>>>>>>>>>> > > > >> > >> >> > you can easily produce and consume
>>>>> messages
>>>>>>>>> without
>>>>>>>>>> > having
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> take
>>>>>>>>>> > > > >> > >> into
>>>>>>>>>> > > > >> > >> >> > account specific cases. For example, I
>>>>> want
>>>>>>>>> to route
>>>>>>>>>> > > messages,
>>>>>>>>>> > > > >> but
>>>>>>>>>> > > > >> > >> >> client A
>>>>>>>>>> > > > >> > >> >> > doesn't support the way audit
>>> implemented
>>>>>>>>> headers, and
>>>>>>>>>> > > client
>>>>>>>>>> > > > B
>>>>>>>>>> > > > >> > >> doesn't
>>>>>>>>>> > > > >> > >> >> > support the way encryption or routing
>>>>>>>>> implemented
>>>>>>>>>> > headers,
>>>>>>>>>> > > so
>>>>>>>>>> > > > now
>>>>>>>>>> > > > >> > my
>>>>>>>>>> > > > >> > >> >> > application has to create some really
>>>>>>> fragile
>>>>>>>>> (my
>>>>>>>>>> > > autocorrect
>>>>>>>>>> > > > >> just
>>>>>>>>>> > > > >> > >> tried
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> > make that "tragic", which is probably
>>>>>>>>> appropriate
>>>>>>>>>> > too) code
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> > strip
>>>>>>>>>> > > > >> > >> >> > everything off, rather than just
>>>>> consuming
>>>>>>> the
>>>>>>>>>> > messages,
>>>>>>>>>> > > > picking
>>>>>>>>>> > > > >> > out
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> 1
>>>>>>>>>> > > > >> > >> >> > or 2 headers it's interested in, and
>>>>>>>>> performing its
>>>>>>>>>> > > function.
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has been
>>> going
>>>>> on
>>>>>>>>> for a
>>>>>>>>>> > long time,
>>>>>>>>>> > > > and
>>>>>>>>>> > > > >> > it's
>>>>>>>>>> > > > >> > >> >> > always "Oh, you came up with 2 use
>>> cases,
>>>>>>> and
>>>>>>>>> yeah,
>>>>>>>>>> > those
>>>>>>>>>> > > use
>>>>>>>>>> > > > >> cases
>>>>>>>>>> > > > >> > >> are
>>>>>>>>>> > > > >> > >> >> > real things that someone would want to
>>>>> do.
>>>>>>>>> Here's an
>>>>>>>>>> > > alternate
>>>>>>>>>> > > > >> way
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > >> >> > implement them so let's not do
>>> headers."
>>>>> If
>>>>>>>>> we have a
>>>>>>>>>> > few
>>>>>>>>>> > > use
>>>>>>>>>> > > > >> cases
>>>>>>>>>> > > > >> > >> that
>>>>>>>>>> > > > >> > >> >> we
>>>>>>>>>> > > > >> > >> >> > actually came up with, you can be sure
>>>>> that
>>>>>>>>> over the
>>>>>>>>>> > next
>>>>>>>>>> > > year
>>>>>>>>>> > > > >> > >> there's a
>>>>>>>>>> > > > >> > >> >> > dozen others that we didn't think of
>>> that
>>>>>>>>> someone
>>>>>>>>>> > would like
>>>>>>>>>> > > > to
>>>>>>>>>> > > > >> > do. I
>>>>>>>>>> > > > >> > >> >> > really think it's time to stop
>>> rehashing
>>>>>>> this
>>>>>>>>>> > discussion and
>>>>>>>>>> > > > >> > instead
>>>>>>>>>> > > > >> > >> >> focus
>>>>>>>>>> > > > >> > >> >> > on a workable standard that we can
>>> adopt.
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > -Todd
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
>>>>> Palino
>>>>>>> <
>>>>>>>>>> > > > tpalino@gmail.com>
>>>>>>>>>> > > > >> > >> wrote:
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> >> C. per message encryption
>>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach is that
>>>>> this
>>>>>>>>>> > significantly
>>>>>>>>>> > > > reduce
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression, which
>>>>>>> happens
>>>>>>>>> on a
>>>>>>>>>> > set of
>>>>>>>>>> > > > >> > serialized
>>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to enable
>>>>> SSL
>>>>>>>>> for wire
>>>>>>>>>> > > > encryption
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> rely
>>>>>>>>>> > > > >> > >> >> on
>>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS) for at
>>>>> rest
>>>>>>>>>> > encryption.
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
>>> this
>>>>>>> does
>>>>>>>>> cover
>>>>>>>>>> > the case
>>>>>>>>>> > > > of
>>>>>>>>>> > > > >> > >> removing
>>>>>>>>>> > > > >> > >> >> a
>>>>>>>>>> > > > >> > >> >> >> drive from the system, it will not
>>>>> satisfy
>>>>>>>>> most
>>>>>>>>>> > compliance
>>>>>>>>>> > > > >> > >> requirements
>>>>>>>>>> > > > >> > >> >> for
>>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever has
>>>>> access to
>>>>>>>>> the
>>>>>>>>>> > broker
>>>>>>>>>> > > itself
>>>>>>>>>> > > > >> > still
>>>>>>>>>> > > > >> > >> has
>>>>>>>>>> > > > >> > >> >> >> access to the unencrypted data. For
>>>>>>>>> end-to-end
>>>>>>>>>> > encryption
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> > need to
>>>>>>>>>> > > > >> > >> >> >> encrypt at the producer, before it
>>>>> enters
>>>>>>> the
>>>>>>>>>> > system, and
>>>>>>>>>> > > > >> decrypt
>>>>>>>>>> > > > >> > at
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the system.
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >> -Todd
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai
>>> <
>>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > >> >> wrote:
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >>> another big plus of headers in the
>>>>>>> protocol
>>>>>>>>> is that
>>>>>>>>>> > it
>>>>>>>>>> > > would
>>>>>>>>>> > > > >> > enable
>>>>>>>>>> > > > >> > >> >> rapid
>>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of core
>>>>> kafka
>>>>>>>>> and would
>>>>>>>>>> > reduce
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> number of
>>>>>>>>>> > > > >> > >> >> >>> future wire format changes required.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
>>>>>>> represents
>>>>>>>>> use
>>>>>>>>>> > cases that
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> > not
>>>>>>>>>> > > > >> > >> 100%
>>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and some of
>>> them
>>>>>>>>> require
>>>>>>>>>> > rather
>>>>>>>>>> > > > invasive
>>>>>>>>>> > > > >> > wire
>>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a good
>>> recent
>>>>>>>>> example of
>>>>>>>>>> > this is
>>>>>>>>>> > > > >> > kip-98.
>>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected to
>>> be
>>>>> a
>>>>>>>>> very small
>>>>>>>>>> > > > fraction of
>>>>>>>>>> > > > >> > >> total
>>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes are
>>>>> invasive.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> every such wire format change
>>>>> translates
>>>>>>>>> into
>>>>>>>>>> > painful and
>>>>>>>>>> > > > slow
>>>>>>>>>> > > > >> > >> >> adoption of
>>>>>>>>>> > > > >> > >> >> >>> new versions.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
>>>>> currently
>>>>>>> in
>>>>>>>>> KIPs
>>>>>>>>>> > could be
>>>>>>>>>> > > > "spun
>>>>>>>>>> > > > >> > out"
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
>>>>> transmitting
>>>>>>>>> data over
>>>>>>>>>> > > > headers.
>>>>>>>>>> > > > >> > this
>>>>>>>>>> > > > >> > >> >> would
>>>>>>>>>> > > > >> > >> >> >>> keep the core wire format stable(r),
>>>>> core
>>>>>>>>> codebase
>>>>>>>>>> > > smaller,
>>>>>>>>>> > > > and
>>>>>>>>>> > > > >> > >> avoid
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
>>>>> required
>>>>>>>>> to prove
>>>>>>>>>> > a
>>>>>>>>>> > > > certain
>>>>>>>>>> > > > >> > >> feature
>>>>>>>>>> > > > >> > >> >> is
>>>>>>>>>> > > > >> > >> >> >>> useful enough for a wide-enough
>>>>> audience
>>>>>>> to
>>>>>>>>> warrant
>>>>>>>>>> > a wire
>>>>>>>>>> > > > >> format
>>>>>>>>>> > > > >> > >> >> change
>>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
>>>>> "mere"
>>>>>>>>> wire format
>>>>>>>>>> > > changes
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > im
>>>>>>>>>> > > > >> > >> not
>>>>>>>>>> > > > >> > >> >> >>> saying it could have been completely
>>>>> done
>>>>>>>>> with
>>>>>>>>>> > headers,
>>>>>>>>>> > > but
>>>>>>>>>> > > > >> > >> >> exactly-once
>>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen
>>>>>>>>> Shapira <
>>>>>>>>>> > > > >> gwen@confluent.io
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > >> >> wrote:
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
>>>>> radai <
>>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
>>>>> organization,
>>>>>>>>> one could
>>>>>>>>>> > > always
>>>>>>>>>> > > > use
>>>>>>>>>> > > > >> > >> other
>>>>>>>>>> > > > >> > >> >> >>> > > approaches such as company-wise
>>>>>>>>> containers"
>>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
>>>>>>> traditionally
>>>>>>>>> done
>>>>>>>>>> > but there
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> > now
>>>>>>>>>> > > > >> > >> >> cases
>>>>>>>>>> > > > >> > >> >> >>> > (read
>>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is not
>>>>>>> acceptable.
>>>>>>>>> this
>>>>>>>>>> > makes
>>>>>>>>>> > > > headers
>>>>>>>>>> > > > >> > >> useful
>>>>>>>>>> > > > >> > >> >> even
>>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for cases
>>> where
>>>>>>>>>> > > > one-container-fits-all
>>>>>>>>>> > > > >> > cannot
>>>>>>>>>> > > > >> > >> >> >>> apply.
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use cases
>>>>>>> listed,
>>>>>>>>> i dont
>>>>>>>>>> > want
>>>>>>>>>> > > > this to
>>>>>>>>>> > > > >> > >> devolve
>>>>>>>>>> > > > >> > >> >> >>> to a
>>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular use
>>> cases
>>>>> - i
>>>>>>>>> think its
>>>>>>>>>> > > enough
>>>>>>>>>> > > > >> that
>>>>>>>>>> > > > >> > >> some
>>>>>>>>>> > > > >> > >> >> of
>>>>>>>>>> > > > >> > >> >> >>> them
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I think a main point of contention
>>> is
>>>>>>>>> that: We
>>>>>>>>>> > > identified
>>>>>>>>>> > > > few
>>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are useful,
>>>>> do
>>>>>>> we
>>>>>>>>> want
>>>>>>>>>> > Kafka to
>>>>>>>>>> > > > be a
>>>>>>>>>> > > > >> > >> system
>>>>>>>>>> > > > >> > >> >> >>> > that supports those use-cases?
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
>>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
>>>>> record-level
>>>>>>>>> lineage
>>>>>>>>>> > is
>>>>>>>>>> > > though
>>>>>>>>>> > > > >> > since
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > overhead could
>>>>>>>>>> > > > >> > >> >> >>> > be significant."
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record level
>>>>>>>>> lineage. I
>>>>>>>>>> > don't
>>>>>>>>>> > > think
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > was
>>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think it is
>>>>> safe
>>>>>>> to
>>>>>>>>> assume
>>>>>>>>>> > that
>>>>>>>>>> > > the
>>>>>>>>>> > > > NSA
>>>>>>>>>> > > > >> > >> needed
>>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also know
>>> that
>>>>>>>>> certain
>>>>>>>>>> > financial
>>>>>>>>>> > > > >> > institutes
>>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
>>> records
>>>>> at
>>>>>>> a
>>>>>>>>> record
>>>>>>>>>> > level
>>>>>>>>>> > > and
>>>>>>>>>> > > > >> > there
>>>>>>>>>> > > > >> > >> are
>>>>>>>>>> > > > >> > >> >> >>> > federal regulations that absolutely
>>>>>>>>> require
>>>>>>>>>> > this. They
>>>>>>>>>> > > > also
>>>>>>>>>> > > > >> > need
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
>>>>> "touches"
>>>>>>> the
>>>>>>>>>> > messages and
>>>>>>>>>> > > > >> either
>>>>>>>>>> > > > >> > >> reads
>>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't have
>>>>>>> possibly
>>>>>>>>>> > modified the
>>>>>>>>>> > > > >> payload
>>>>>>>>>> > > > >> > >> >> itself.
>>>>>>>>>> > > > >> > >> >> >>> > They use record level encryption to
>>>>> do
>>>>>>>>> that -
>>>>>>>>>> > apps can
>>>>>>>>>> > > > read
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
>>> can't
>>>>>>>>> touch the
>>>>>>>>>> > payload.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those are
>>> corner
>>>>>>>>> cases and
>>>>>>>>>> > not worth
>>>>>>>>>> > > > >> adding
>>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they should
>>>>> use a
>>>>>>>>> different
>>>>>>>>>> > > pubsub
>>>>>>>>>> > > > >> > message
>>>>>>>>>> > > > >> > >> for
>>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the other 1000
>>>>> that
>>>>>>>>> cater
>>>>>>>>>> > > > specifically
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > financial industry).
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a catch 22:
>>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific use-case,
>>>>>>>>> someone can
>>>>>>>>>> > always
>>>>>>>>>> > > say
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > isn't
>>>>>>>>>> > > > >> > >> >> >>> > interesting enough for Kafka. If we
>>>>>>>>> discuss more
>>>>>>>>>> > general
>>>>>>>>>> > > > >> > trends,
>>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we are not
>>> sure
>>>>>>> any
>>>>>>>>> of them
>>>>>>>>>> > really
>>>>>>>>>> > > > >> needs
>>>>>>>>>> > > > >> > >> >> headers
>>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just hand
>>>>> waving
>>>>>>>>> and not
>>>>>>>>>> > > > interesting.".
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I think discussing use-cases in
>>>>>>> specifics
>>>>>>>>> is super
>>>>>>>>>> > > > important
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> >> decide
>>>>>>>>>> > > > >> > >> >> >>> > implementation details for headers
>>>>> (my
>>>>>>>>> use-cases
>>>>>>>>>> > lean
>>>>>>>>>> > > > toward
>>>>>>>>>> > > > >> > >> >> numerical
>>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and object
>>>>> values,
>>>>>>>>> others
>>>>>>>>>> > differ),
>>>>>>>>>> > > > but I
>>>>>>>>>> > > > >> > >> think
>>>>>>>>>> > > > >> > >> >> we
>>>>>>>>>> > > > >> > >> >> >>> > need to answer the general "Are we
>>>>> going
>>>>>>>>> to have
>>>>>>>>>> > > headers"
>>>>>>>>>> > > > >> > question
>>>>>>>>>> > > > >> > >> >> >>> > first.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the other
>>>>>>>>> committers in the
>>>>>>>>>> > > > discussion:
>>>>>>>>>> > > > >> > >> >> >>> > What would it take to convince you
>>>>> that
>>>>>>>>> headers
>>>>>>>>>> > in Kafka
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> a
>>>>>>>>>> > > > >> > >> good
>>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can move
>>> ahead
>>>>>>> and
>>>>>>>>> try to
>>>>>>>>>> > agree
>>>>>>>>>> > > on
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> >> details?
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving the goal
>>>>>>> posts
>>>>>>>>> and
>>>>>>>>>> > this is
>>>>>>>>>> > > > truly
>>>>>>>>>> > > > >> > >> >> exhausting.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly support
>>>>> adding
>>>>>>>>> headers
>>>>>>>>>> > to Kafka
>>>>>>>>>> > > > >> > (+0.5?).
>>>>>>>>>> > > > >> > >> >> >>> > The community can continue to find
>>>>>>>>> workarounds to
>>>>>>>>>> > the
>>>>>>>>>> > > > issue
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> there
>>>>>>>>>> > > > >> > >> >> >>> > are some benefits to keeping the
>>>>> message
>>>>>>>>> format
>>>>>>>>>> > and
>>>>>>>>>> > > > clients
>>>>>>>>>> > > > >> > >> simpler.
>>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of headers
>>>>> to
>>>>>>>>> many
>>>>>>>>>> > use-cases
>>>>>>>>>> > > and
>>>>>>>>>> > > > if
>>>>>>>>>> > > > >> we
>>>>>>>>>> > > > >> > >> can
>>>>>>>>>> > > > >> > >> >> >>> > find a good and generally useful
>>> way
>>>>> to
>>>>>>>>> add it to
>>>>>>>>>> > Kafka,
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > will
>>>>>>>>>> > > > >> > >> make
>>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
>>> worthy
>>>>>>>>> goal in my
>>>>>>>>>> > eyes.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
>>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use cases
>>>>> for
>>>>>>>>> polyglot
>>>>>>>>>> > topics.
>>>>>>>>>> > > > >> > >> especially if
>>>>>>>>>> > > > >> > >> >> >>> kafka
>>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
>>> something
>>>>>>> else.
>>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would make it
>>>>>>> harder
>>>>>>>>> to write
>>>>>>>>>> > > > portable
>>>>>>>>>> > > > >> > >> consumer
>>>>>>>>>> > > > >> > >> >> >>> code.
>>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would mess
>>> with
>>>>>>>>> locality of
>>>>>>>>>> > > > consumption
>>>>>>>>>> > > > >> > >> >> >>> guarantees.
>>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
>>>>>>>>> lineage/metadata is
>>>>>>>>>> > > > >> > >> billing/chargeback.
>>>>>>>>>> > > > >> > >> >> for
>>>>>>>>>> > > > >> > >> >> >>> > that
>>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough to
>>> simply
>>>>>>>>> record the
>>>>>>>>>> > point
>>>>>>>>>> > > of
>>>>>>>>>> > > > >> > origin,
>>>>>>>>>> > > > >> > >> but
>>>>>>>>>> > > > >> > >> >> >>> every
>>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think mirror
>>>>> maker)
>>>>>>>>> must also
>>>>>>>>>> > add a
>>>>>>>>>> > > > >> record
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > >> >> form a
>>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing on top
>>> of
>>>>>>>>> kafka - i
>>>>>>>>>> > know
>>>>>>>>>> > > samza
>>>>>>>>>> > > > >> has a
>>>>>>>>>> > > > >> > >> >> metadata
>>>>>>>>>> > > > >> > >> >> >>> > map
>>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
>>>>> addition to
>>>>>>>>> user
>>>>>>>>>> > values.
>>>>>>>>>> > > > headers
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > perfect
>>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM,
>>>>> Jun
>>>>>>>>> Rao <
>>>>>>>>>> > > > jun@confluent.io
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the first two
>>>>>>>>> questions, it
>>>>>>>>>> > would
>>>>>>>>>> > > be
>>>>>>>>>> > > > >> > helpful
>>>>>>>>>> > > > >> > >> >> if we
>>>>>>>>>> > > > >> > >> >> >>> > could
>>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use cases
>>>>> for
>>>>>>>>> headers
>>>>>>>>>> > in the
>>>>>>>>>> > > > space
>>>>>>>>>> > > > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > third-party
>>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases within an
>>>>>>>>> organization,
>>>>>>>>>> > one
>>>>>>>>>> > > > could
>>>>>>>>>> > > > >> > always
>>>>>>>>>> > > > >> > >> >> use
>>>>>>>>>> > > > >> > >> >> >>> > other
>>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as company-wise
>>>>>>>>> containers to
>>>>>>>>>> > get
>>>>>>>>>> > > > around
>>>>>>>>>> > > > >> w/o
>>>>>>>>>> > > > >> > >> >> >>> headers. I
>>>>>>>>>> > > > >> > >> >> >>> > >> went through the use cases in
>>> the
>>>>> KIP
>>>>>>>>> and in
>>>>>>>>>> > Radai's
>>>>>>>>>> > > > wiki
>>>>>>>>>> > > > >> (
>>>>>>>>>> > > > >> > >> >> >>> > >> https://cwiki.apache.org/confl
>>>>>>>>>> > uence/display/KAFKA/A+
>>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> ).
>>>>>>>>>> > > > >> > >> >> >>> > >> The following are the ones that
>>>>> that
>>>>>>> I
>>>>>>>>>> > understand and
>>>>>>>>>> > > > >> could
>>>>>>>>>> > > > >> > be
>>>>>>>>>> > > > >> > >> in
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case category.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
>>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
>>>>>>> content-type
>>>>>>>>> should
>>>>>>>>>> > be set
>>>>>>>>>> > > at
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> topic
>>>>>>>>>> > > > >> > >> >> >>> level.
>>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages with
>>>>>>>>> different
>>>>>>>>>> > content
>>>>>>>>>> > > > types
>>>>>>>>>> > > > >> > >> should be
>>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
>>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
>>> useless
>>>>>>>>> without
>>>>>>>>>> > schema id,
>>>>>>>>>> > > it
>>>>>>>>>> > > > >> > seems
>>>>>>>>>> > > > >> > >> that
>>>>>>>>>> > > > >> > >> >> >>> > storing
>>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together with
>>>>>>> serialized
>>>>>>>>> bytes
>>>>>>>>>> > in the
>>>>>>>>>> > > > value
>>>>>>>>>> > > > >> is
>>>>>>>>>> > > > >> > >> >> better?
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
>>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this approach is
>>>>> that
>>>>>>>>> this
>>>>>>>>>> > > > significantly
>>>>>>>>>> > > > >> > reduce
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
>>>>> which
>>>>>>>>> happens on
>>>>>>>>>> > a set
>>>>>>>>>> > > of
>>>>>>>>>> > > > >> > >> serialized
>>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative is to
>>>>> enable
>>>>>>>>> SSL for
>>>>>>>>>> > wire
>>>>>>>>>> > > > >> > encryption
>>>>>>>>>> > > > >> > >> and
>>>>>>>>>> > > > >> > >> >> >>> rely
>>>>>>>>>> > > > >> > >> >> >>> > on
>>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g. LUKS)
>>>>> for at
>>>>>>>>> rest
>>>>>>>>>> > > encryption.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
>>> across
>>>>>>>>> Kafka
>>>>>>>>>> > clusters
>>>>>>>>>> > > > >> > >> >> >>> > >> This is actually interesting.
>>>>> Today,
>>>>>>>>> to avoid
>>>>>>>>>> > > > introducing
>>>>>>>>>> > > > >> > >> cycles
>>>>>>>>>> > > > >> > >> >> when
>>>>>>>>>> > > > >> > >> >> >>> > doing
>>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data centers,
>>> one
>>>>>>>>> would
>>>>>>>>>> > either have
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> set
>>>>>>>>>> > > > >> > up
>>>>>>>>>> > > > >> > >> two
>>>>>>>>>> > > > >> > >> >> >>> Kafka
>>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
>>>>> aggregate)
>>>>>>>>> per data
>>>>>>>>>> > center
>>>>>>>>>> > > or
>>>>>>>>>> > > > >> > rename
>>>>>>>>>> > > > >> > >> >> topics.
>>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With headers,
>>>>> the
>>>>>>>>> producer
>>>>>>>>>> > could
>>>>>>>>>> > > tag
>>>>>>>>>> > > > >> each
>>>>>>>>>> > > > >> > >> >> message
>>>>>>>>>> > > > >> > >> >> >>> with
>>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID in the
>>>>>>> header.
>>>>>>>>>> > MirrorMaker
>>>>>>>>>> > > > could
>>>>>>>>>> > > > >> > then
>>>>>>>>>> > > > >> > >> >> avoid
>>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a cluster
>>> if
>>>>>>>>> they are
>>>>>>>>>> > tagged
>>>>>>>>>> > > with
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> same
>>>>>>>>>> > > > >> > >> >> >>> cluster
>>>>>>>>>> > > > >> > >> >> >>> > >> id.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative approach
>>>>> is
>>>>>>> to
>>>>>>>>>> > introduce sth
>>>>>>>>>> > > > like
>>>>>>>>>> > > > >> > >> >> >>> hierarchical
>>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages from
>>>>>>> different
>>>>>>>>>> > clusters in
>>>>>>>>>> > > > >> > different
>>>>>>>>>> > > > >> > >> >> >>> partitions
>>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic. This
>>>>> approach
>>>>>>>>> avoids
>>>>>>>>>> > filtering
>>>>>>>>>> > > > out
>>>>>>>>>> > > > >> > >> unneeded
>>>>>>>>>> > > > >> > >> >> >>> data
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving easier
>>> to
>>>>>>>>> support. It
>>>>>>>>>> > may
>>>>>>>>>> > > make
>>>>>>>>>> > > > >> > >> compaction
>>>>>>>>>> > > > >> > >> >> >>> > trickier
>>>>>>>>>> > > > >> > >> >> >>> > >> though since the same key may
>>>>> show up
>>>>>>>>> in
>>>>>>>>>> > different
>>>>>>>>>> > > > >> > partitions.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
>>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source connector
>>>>> could
>>>>>>>>> store in
>>>>>>>>>> > the
>>>>>>>>>> > > > message
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> metadata
>>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
>>> record.
>>>>>>>>> Similarly,
>>>>>>>>>> > if a
>>>>>>>>>> > > > stream
>>>>>>>>>> > > > >> job
>>>>>>>>>> > > > >> > >> >> >>> transforms
>>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to topic
>>> B,
>>>>> the
>>>>>>>>> library
>>>>>>>>>> > could
>>>>>>>>>> > > > >> include
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> source
>>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each of the
>>>>>>>>> transformed
>>>>>>>>>> > message in
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> header.
>>>>>>>>>> > > > >> > >> >> Not
>>>>>>>>>> > > > >> > >> >> >>> > sure
>>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful record-level
>>>>>>> lineage
>>>>>>>>> is
>>>>>>>>>> > though
>>>>>>>>>> > > since
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> >> overhead
>>>>>>>>>> > > > >> > >> >> >>> > could
>>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
>>>>>>>>> clientId/host/user in
>>>>>>>>>> > the
>>>>>>>>>> > > > header
>>>>>>>>>> > > > >> in
>>>>>>>>>> > > > >> > >> each
>>>>>>>>>> > > > >> > >> >> >>> > message
>>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These metadata are
>>>>>>>>> really at the
>>>>>>>>>> > > producer
>>>>>>>>>> > > > >> > level
>>>>>>>>>> > > > >> > >> >> though.
>>>>>>>>>> > > > >> > >> >> >>> > So, a
>>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to only
>>>>>>> include a
>>>>>>>>>> > "producerId"
>>>>>>>>>> > > > per
>>>>>>>>>> > > > >> > >> message
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> > send
>>>>>>>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
>>> mapping
>>>>>>>>>> > independently.
>>>>>>>>>> > > > KIP-98
>>>>>>>>>> > > > >> is
>>>>>>>>>> > > > >> > >> >> actually
>>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such a
>>>>> producerId
>>>>>>>>> natively
>>>>>>>>>> > in the
>>>>>>>>>> > > > >> > message.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that I
>>> am
>>>>>>> fully
>>>>>>>>>> > convinced of
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > strong
>>>>>>>>>> > > > >> > >> >> >>> > third-party
>>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
>>> Perhaps
>>>>> we
>>>>>>>>> could
>>>>>>>>>> > discuss a
>>>>>>>>>> > > > bit
>>>>>>>>>> > > > >> > more
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> make
>>>>>>>>>> > > > >> > >> >> >>> > one
>>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing use
>>>>> cases.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal question is
>>>>>>>>> whether header
>>>>>>>>>> > should
>>>>>>>>>> > > > be
>>>>>>>>>> > > > >> > >> exposed
>>>>>>>>>> > > > >> > >> >> in
>>>>>>>>>> > > > >> > >> >> >>> > stream
>>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
>>>>> stream,
>>>>>>>>> Samza,
>>>>>>>>>> > and
>>>>>>>>>> > > Spark
>>>>>>>>>> > > > >> > >> streaming.
>>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems just
>>> deal
>>>>>>> with
>>>>>>>>>> > key/value
>>>>>>>>>> > > > pairs.
>>>>>>>>>> > > > >> > >> Should we
>>>>>>>>>> > > > >> > >> >> >>> > expose a
>>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there too or
>>>>>>>>> somehow map
>>>>>>>>>> > header to
>>>>>>>>>> > > > key
>>>>>>>>>> > > > >> or
>>>>>>>>>> > > > >> > >> >> value?
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> Jun
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM,
>>>>>>>>> Michael
>>>>>>>>>> > Pearce <
>>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a period
>>>>> of a
>>>>>>>>> week,
>>>>>>>>>> > that there
>>>>>>>>>> > > > is
>>>>>>>>>> > > > >> no
>>>>>>>>>> > > > >> > >> >> concerns
>>>>>>>>>> > > > >> > >> >> >>> now
>>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and now
>>> we
>>>>>>> have
>>>>>>>>>> > agreement that
>>>>>>>>>> > > > >> > headers
>>>>>>>>>> > > > >> > >> are
>>>>>>>>>> > > > >> > >> >> >>> useful
>>>>>>>>>> > > > >> > >> >> >>> > >> and
>>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
>>> put
>>>>> to
>>>>>>> a
>>>>>>>>> KIP
>>>>>>>>>> > vote, this
>>>>>>>>>> > > > >> > wouldn't
>>>>>>>>>> > > > >> > >> be
>>>>>>>>>> > > > >> > >> >> a
>>>>>>>>>> > > > >> > >> >> >>> > reason
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > @
>>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
>>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of getting
>>>>> this
>>>>>>>>> KIP
>>>>>>>>>> > moving past
>>>>>>>>>> > > > >> this,
>>>>>>>>>> > > > >> > we
>>>>>>>>>> > > > >> > >> can
>>>>>>>>>> > > > >> > >> >> >>> state
>>>>>>>>>> > > > >> > >> >> >>> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
>>> that
>>>>>>> can
>>>>>>>>> will be
>>>>>>>>>> > > > naturally
>>>>>>>>>> > > > >> > >> >> interpreted
>>>>>>>>>> > > > >> > >> >> >>> as
>>>>>>>>>> > > > >> > >> >> >>> > an
>>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is later
>>>>>>>>> wanted you can
>>>>>>>>>> > > easily
>>>>>>>>>> > > > >> split
>>>>>>>>>> > > > >> > >> this
>>>>>>>>>> > > > >> > >> >> >>> into
>>>>>>>>>> > > > >> > >> >> >>> > two
>>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the wire
>>>>>>> protocol
>>>>>>>>>> > > implementation
>>>>>>>>>> > > > >> this
>>>>>>>>>> > > > >> > >> makes
>>>>>>>>>> > > > >> > >> >> no
>>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don't believe. Is
>>>>> this
>>>>>>>>>> > reasonable to
>>>>>>>>>> > > > all?
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4 therefor
>>>>> happy
>>>>>>>>> we keep
>>>>>>>>>> > with 32
>>>>>>>>>> > > > >> bits.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
>>>>>>>>>> > ignacio.solis@gmail.com on
>>>>>>>>>> > > > >> behalf
>>>>>>>>>> > > > >> > of
>>>>>>>>>> > > > >> > >> >> >>> Ignacio
>>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
>>> ignacio.solis@gmail.com
>>>>> on
>>>>>>>>> behalf of
>>>>>>>>>> > > > >> > isolis@igso.net
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> >> >>> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value as
>>>>> byte[]
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
>>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
>>> maybe
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
>>>>> system
>>>>>>>>> should
>>>>>>>>>> > take a
>>>>>>>>>> > > > single
>>>>>>>>>> > > > >> > >> int. I
>>>>>>>>>> > > > >> > >> >> >>> think
>>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
>>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you want
>>> to
>>>>>>>>> interpret
>>>>>>>>>> > this as
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> 16bit
>>>>>>>>>> > > > >> > >> >> numbers
>>>>>>>>>> > > > >> > >> >> >>> in
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > layer
>>>>>>>>>> > > > >> > >> >> >>> > >> > above go right ahead. If
>>>>>>>>> somebody wants
>>>>>>>>>> > to
>>>>>>>>>> > > argue
>>>>>>>>>> > > > >> for
>>>>>>>>>> > > > >> > 16
>>>>>>>>>> > > > >> > >> >> bits
>>>>>>>>>> > > > >> > >> >> >>> or
>>>>>>>>>> > > > >> > >> >> >>> > 64
>>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
>>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I would
>>>>>>> listen.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
>>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key space
>>> into
>>>>>>>>> sub_key_1 and
>>>>>>>>>> > > > sub_key_2
>>>>>>>>>> > > > >> > >> makes no
>>>>>>>>>> > > > >> > >> >> >>> > sense to
>>>>>>>>>> > > > >> > >> >> >>> > >> > me at
>>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we going
>>> to
>>>>>>>>> start
>>>>>>>>>> > providing
>>>>>>>>>> > > > APIs to
>>>>>>>>>> > > > >> > get
>>>>>>>>>> > > > >> > >> all
>>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
>>>>>>>>> sub_key_2s? If
>>>>>>>>>> > there is
>>>>>>>>>> > > > no
>>>>>>>>>> > > > >> > >> >> >>> distinguishing
>>>>>>>>>> > > > >> > >> >> >>> > >> > functions
>>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to each
>>> one
>>>>>>>>> then they
>>>>>>>>>> > should
>>>>>>>>>> > > be
>>>>>>>>>> > > > a
>>>>>>>>>> > > > >> > single
>>>>>>>>>> > > > >> > >> >> >>> value.
>>>>>>>>>> > > > >> > >> >> >>> > At
>>>>>>>>>> > > > >> > >> >> >>> > >> > this
>>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're doing is
>>>>>>>>> equality.
>>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer wants
>>> to
>>>>>>>>> interpret
>>>>>>>>>> > this as
>>>>>>>>>> > > 2,
>>>>>>>>>> > > > 3
>>>>>>>>>> > > > >> or
>>>>>>>>>> > > > >> > >> more
>>>>>>>>>> > > > >> > >> >> >>> values
>>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
>>>>>>>>>> > > > >> > >> >> >>> > >> > different question. I
>>>>>>>>> personally think
>>>>>>>>>> > it's
>>>>>>>>>> > > all
>>>>>>>>>> > > > one
>>>>>>>>>> > > > >> > >> >> keyspace
>>>>>>>>>> > > > >> > >> >> >>> > that is
>>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned using
>>> some
>>>>>>>>> structure,
>>>>>>>>>> > but if
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> > want to
>>>>>>>>>> > > > >> > >> >> >>> > sub-assign
>>>>>>>>>> > > > >> > >> >> >>> > >> > parts
>>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
>>> applies
>>>>> to
>>>>>>>>> strings.
>>>>>>>>>> > If
>>>>>>>>>> > > > somebody
>>>>>>>>>> > > > >> > >> argued
>>>>>>>>>> > > > >> > >> >> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
>>>>>>>>>> > > > >> > >> >> >>> > >> > would we be arguing to
>>>>> divide
>>>>>>> the
>>>>>>>>>> > strings with
>>>>>>>>>> > > > dots
>>>>>>>>>> > > > >> > ('.')
>>>>>>>>>> > > > >> > >> >> as a
>>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
>>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them to give
>>>>> us
>>>>>>> the
>>>>>>>>>> > different
>>>>>>>>>> > > name
>>>>>>>>>> > > > >> > segments
>>>>>>>>>> > > > >> > >> >> >>> > separately?
>>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be performing any
>>>>>>>>> actions on
>>>>>>>>>> > this key
>>>>>>>>>> > > > other
>>>>>>>>>> > > > >> > than
>>>>>>>>>> > > > >> > >> >> >>> > matching?
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
>>> 9:30
>>>>>>> AM,
>>>>>>>>> Michael
>>>>>>>>>> > > Pearce <
>>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any concerns
>>> on
>>>>> 1
>>>>>>>>> and 2
>>>>>>>>>> > still?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
>>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this moving
>>> along a
>>>>>>> bit
>>>>>>>>> more
>>>>>>>>>> > I'd also
>>>>>>>>>> > > > like
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> ask
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> >>> get
>>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last points:
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're all
>>>>>>> roughly
>>>>>>>>> happy
>>>>>>>>>> > with the
>>>>>>>>>> > > > >> header
>>>>>>>>>> > > > >> > >> value
>>>>>>>>>> > > > >> > >> >> >>> > being a
>>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe consensus
>>> has
>>>>>>>>> been for an
>>>>>>>>>> > > > namespace
>>>>>>>>>> > > > >> > based
>>>>>>>>>> > > > >> > >> int
>>>>>>>>>> > > > >> > >> >> >>> > approach
>>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the key.
>>> Any
>>>>>>>>> objections
>>>>>>>>>> > if this
>>>>>>>>>> > > > is
>>>>>>>>>> > > > >> > what
>>>>>>>>>> > > > >> > >> we
>>>>>>>>>> > > > >> > >> >> go
>>>>>>>>>> > > > >> > >> >> >>> > with?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
>>>>> assumption
>>>>>>> in
>>>>>>>>> (4) is
>>>>>>>>>> > > correct,
>>>>>>>>>> > > > >> > >> {int,int}
>>>>>>>>>> > > > >> > >> >> >>> keys.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both int's be
>>>>> int16 or
>>>>>>>>> int32?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them being
>>> int16(2
>>>>>>>>> bytes) as
>>>>>>>>>> > combined
>>>>>>>>>> > > > is
>>>>>>>>>> > > > >> > space
>>>>>>>>>> > > > >> > >> of
>>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
>>>>>>>>>> > > > >> > >> >> >>> > >> > per
>>>>>>>>>> > > > >> > >> >> >>> > >> > > original and gives
>>> plenty
>>>>> of
>>>>>>>>>> > combinations for
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> >> >>> foreseeable,
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead small.
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any benefit in
>>>>>>>>> another kip
>>>>>>>>>> > call to
>>>>>>>>>> > > > >> discuss
>>>>>>>>>> > > > >> > >> >> these at
>>>>>>>>>> > > > >> > >> >> >>> > all?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>> ______________________________
>>>>>>>>>> > __________
>>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
>>>>>>>>> k.burstev@yandex.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday, November
>>> 18,
>>>>>>> 2016
>>>>>>>>>> > 7:07:07 AM
>>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
>>> dev@kafka.apache.org
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re: [DISCUSS]
>>>>> KIP-82
>>>>>>>>> - Add
>>>>>>>>>> > Record
>>>>>>>>>> > > > Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is worth
>>> also
>>>>> i
>>>>>>>>> agree. As
>>>>>>>>>> > a user:
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
>>>>>>>>> worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers should
>>>>> be a
>>>>>>>>> top level
>>>>>>>>>> > > option
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
>>>>> "Ignacio
>>>>>>>>> Solis" <
>>>>>>>>>> > > > >> isolis@igso.net
>>>>>>>>>> > > > >> > >:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers are
>>>>>>>>> worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
>>> should
>>>>> be
>>>>>>> a
>>>>>>>>> top
>>>>>>>>>> > level
>>>>>>>>>> > > option
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14, 2016
>>> at
>>>>>>> 9:16
>>>>>>>>> AM,
>>>>>>>>>> > Michael
>>>>>>>>>> > > > Pearce
>>>>>>>>>> > > > >> <
>>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
>>>>> details/examples
>>>>>>>>> the
>>>>>>>>>> > original
>>>>>>>>>> > > > proposal
>>>>>>>>>> > > > >> > for
>>>>>>>>>> > > > >> > >> key
>>>>>>>>>> > > > >> > >> >> >>> > spacing
>>>>>>>>>> > > > >> > >> >> >>> > >> ,
>>>>>>>>>> > > > >> > >> >> >>> > >> > not
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as per
>>>>>>>>> discussion
>>>>>>>>>> > namespace
>>>>>>>>>> > > > >> idea.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need to
>>> update
>>>>>>> the
>>>>>>>>> kip,
>>>>>>>>>> > when we
>>>>>>>>>> > > get
>>>>>>>>>> > > > >> > >> agreement
>>>>>>>>>> > > > >> > >> >> >>> this
>>>>>>>>>> > > > >> > >> >> >>> > is a
>>>>>>>>>> > > > >> > >> >> >>> > >> > better
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach (which
>>> seems
>>>>> to
>>>>>>>>> be the
>>>>>>>>>> > case if I
>>>>>>>>>> > > > have
>>>>>>>>>> > > > >> > >> >> understood
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > general
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
>>>>>>>>> conversation)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
>>> ints,
>>>>> at
>>>>>>>>> very
>>>>>>>>>> > early stage
>>>>>>>>>> > > > we
>>>>>>>>>> > > > >> did
>>>>>>>>>> > > > >> > >> think
>>>>>>>>>> > > > >> > >> >> >>> about
>>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
>>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added complexity
>>>>> for
>>>>>>>>> the
>>>>>>>>>> > saving isn't
>>>>>>>>>> > > > >> worth
>>>>>>>>>> > > > >> > it.
>>>>>>>>>> > > > >> > >> >> I'd
>>>>>>>>>> > > > >> > >> >> >>> > rather
>>>>>>>>>> > > > >> > >> >> >>> > >> go
>>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
>>>>>>>>> overheads and
>>>>>>>>>> > size
>>>>>>>>>> > > int16
>>>>>>>>>> > > > >> > (2bytes)
>>>>>>>>>> > > > >> > >> >> keys
>>>>>>>>>> > > > >> > >> >> >>> as
>>>>>>>>>> > > > >> > >> >> >>> > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
>>>>>>> headers,
>>>>>>>>> there
>>>>>>>>>> > is as
>>>>>>>>>> > > per
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > kip
>>>>>>>>>> > > > >> > >> as
>>>>>>>>>> > > > >> > >> >> we
>>>>>>>>>> > > > >> > >> >> >>> > use an
>>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
>>>>> headers
>>>>>>>>> are
>>>>>>>>>> > present or
>>>>>>>>>> > > > not as
>>>>>>>>>> > > > >> > such
>>>>>>>>>> > > > >> > >> >> >>> > provides a
>>>>>>>>>> > > > >> > >> >> >>> > >> > zero
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead currently
>>> if
>>>>>>>>> headers are
>>>>>>>>>> > not
>>>>>>>>>> > > used.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
>>>>> mentions
>>>>>>>>> would be
>>>>>>>>>> > good
>>>>>>>>>> > > > first
>>>>>>>>>> > > > >> > if we
>>>>>>>>>> > > > >> > >> >> can
>>>>>>>>>> > > > >> > >> >> >>> get
>>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
>>>>>>>>>> > > > >> > >> >> >>> > >> > > do
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have general
>>>>>>>>> consensus that
>>>>>>>>>> > (1)
>>>>>>>>>> > > > headers
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want it
>>> as
>>>>> a
>>>>>>>>> top level
>>>>>>>>>> > entity.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
>>>>>>> obvious i
>>>>>>>>>> > believe (1)
>>>>>>>>>> > > > >> headers
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a top level
>>>>>>>>> entity.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> ______________________________
>>>>>>>>>> > __________
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger Hoover <
>>>>>>>>>> > > roger.hoover@gmail.com
>>>>>>>>>> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
>>>>> November
>>>>>>>>> 9, 2016
>>>>>>>>>> > 9:10:47
>>>>>>>>>> > > > PM
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
>>>>> dev@kafka.apache.org
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
>>> [DISCUSS]
>>>>>>>>> KIP-82 - Add
>>>>>>>>>> > > Record
>>>>>>>>>> > > > >> > Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
>>>>> little
>>>>>>>>> in the
>>>>>>>>>> > weeds but
>>>>>>>>>> > > > >> thanks
>>>>>>>>>> > > > >> > >> for
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> replies
>>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a prefix
>>>>> and
>>>>>>>>> {int,
>>>>>>>>>> > int} can
>>>>>>>>>> > > be
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> same.
>>>>>>>>>> > > > >> > >> >> It
>>>>>>>>>> > > > >> > >> >> >>> > doesn't
>>>>>>>>>> > > > >> > >> >> >>> > >> > look
>>>>>>>>>> > > > >> > >> >> >>> > >> > > like
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what the KIP
>>> is
>>>>>>>>> saying the
>>>>>>>>>> > "Open"
>>>>>>>>>> > > > >> > section.
>>>>>>>>>> > > > >> > >> The
>>>>>>>>>> > > > >> > >> >> >>> > example
>>>>>>>>>> > > > >> > >> >> >>> > >> > shows
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
>>>>> 210002
>>>>>>>>> for App
>>>>>>>>>> > Dynamics
>>>>>>>>>> > > > >> > implying
>>>>>>>>>> > > > >> > >> >> that
>>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>>> > > > >> > >> >> >>> > New
>>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization will
>>> have
>>>>>>>>> only a
>>>>>>>>>> > single
>>>>>>>>>> > > > header id
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > >> work
>>>>>>>>>> > > > >> > >> >> >>> > with. Or
>>>>>>>>>> > > > >> > >> >> >>> > >> > is
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The main
>>>>> point
>>>>>>>>> of a
>>>>>>>>>> > namespace
>>>>>>>>>> > > or
>>>>>>>>>> > > > >> > prefix
>>>>>>>>>> > > > >> > >> is
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> >>> > reduce
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of config
>>>>>>> mapping
>>>>>>>>> or
>>>>>>>>>> > > registration
>>>>>>>>>> > > > >> > >> depending
>>>>>>>>>> > > > >> > >> >> on
>>>>>>>>>> > > > >> > >> >> >>> how
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> namespaces/prefixes
>>>>> are
>>>>>>>>> managed.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to hear
>>>>> more
>>>>>>>>> feedback
>>>>>>>>>> > on the
>>>>>>>>>> > > > >> > >> higher-level
>>>>>>>>>> > > > >> > >> >> >>> > questions
>>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9, 2016
>>> at
>>>>>>>>> 11:38 AM,
>>>>>>>>>> > radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think this
>>>>> discussion
>>>>>>>>> is
>>>>>>>>>> > getting a
>>>>>>>>>> > > bit
>>>>>>>>>> > > > >> into
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> weeds on
>>>>>>>>>> > > > >> > >> >> >>> > >> > technical
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > implementation
>>>>> details.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to step
>>>>> back a
>>>>>>>>> minute
>>>>>>>>>> > and try
>>>>>>>>>> > > > and
>>>>>>>>>> > > > >> > >> establish
>>>>>>>>>> > > > >> > >> >> >>> > where we
>>>>>>>>>> > > > >> > >> >> >>> > >> > are in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
>>> nacho's
>>>>>>> last
>>>>>>>>>> > paragraph)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all in
>>>>>>>>> agreement that
>>>>>>>>>> > headers
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> a
>>>>>>>>>> > > > >> > >> >> >>> worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > useful
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to have?
>>>>> this
>>>>>>>>> was
>>>>>>>>>> > contested
>>>>>>>>>> > > > early
>>>>>>>>>> > > > >> on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all in
>>>>>>>>> agreement on
>>>>>>>>>> > headers
>>>>>>>>>> > > as
>>>>>>>>>> > > > top
>>>>>>>>>> > > > >> > >> level
>>>>>>>>>> > > > >> > >> >> >>> entity
>>>>>>>>>> > > > >> > >> >> >>> > vs
>>>>>>>>>> > > > >> > >> >> >>> > >> > headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > squirreled-away in
>>>>> V?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there are still
>>>>>>>>> concerns
>>>>>>>>>> > around
>>>>>>>>>> > > these
>>>>>>>>>> > > > #2
>>>>>>>>>> > > > >> > >> points
>>>>>>>>>> > > > >> > >> >> >>> (#jay?
>>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now back to
>>> our
>>>>>>>>> normal
>>>>>>>>>> > programming
>>>>>>>>>> > > > ...)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are nice.
>>>>>>> having
>>>>>>>>> said
>>>>>>>>>> > that, its
>>>>>>>>>> > > > >> adding
>>>>>>>>>> > > > >> > >> >> >>> complexity
>>>>>>>>>> > > > >> > >> >> >>> > >> (see
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> https://github.com/addthis/
>>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> main/java/com/clearspring/
>>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
>>>>> result)
>>>>>>>>> and would
>>>>>>>>>> > require
>>>>>>>>>> > > > >> anyone
>>>>>>>>>> > > > >> > >> >> writing
>>>>>>>>>> > > > >> > >> >> >>> > other
>>>>>>>>>> > > > >> > >> >> >>> > >> > clients
>>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go? Bash?
>>>>> ;-) )
>>>>>>>>> to
>>>>>>>>>> > > get/implement
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> same,
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain (int
>>> vs
>>>>>>>>> string is
>>>>>>>>>> > order of
>>>>>>>>>> > > > >> > magnitude,
>>>>>>>>>> > > > >> > >> >> this
>>>>>>>>>> > > > >> > >> >> >>> > isnt).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int namespacing vs
>>>>>>> {int,
>>>>>>>>> int}
>>>>>>>>>> > > namespacing
>>>>>>>>>> > > > >> are
>>>>>>>>>> > > > >> > >> >> basically
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > same
>>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
>>>>> namespacing
>>>>>>>>> an int64
>>>>>>>>>> > and
>>>>>>>>>> > > > giving
>>>>>>>>>> > > > >> > people
>>>>>>>>>> > > > >> > >> >> while
>>>>>>>>>> > > > >> > >> >> >>> > 2^32
>>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
>>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the part i
>>>>> like
>>>>>>>>> about this
>>>>>>>>>> > is
>>>>>>>>>> > > > letting
>>>>>>>>>> > > > >> > >> people
>>>>>>>>>> > > > >> > >> >> >>> have a
>>>>>>>>>> > > > >> > >> >> >>> > >> large
>>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers with one
>>>>>>>>> registration so
>>>>>>>>>> > they
>>>>>>>>>> > > > dont
>>>>>>>>>> > > > >> > have
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> come
>>>>>>>>>> > > > >> > >> >> >>> > back
>>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > every
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
>>> plugin/header
>>>>>>>>> they want to
>>>>>>>>>> > > > "reserve".
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
>>> 2016
>>>>> at
>>>>>>>>> 11:01 AM,
>>>>>>>>>> > Roger
>>>>>>>>>> > > > >> Hoover
>>>>>>>>>> > > > >> > <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > roger.hoover@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
>>> the
>>>>>>>>> debate has
>>>>>>>>>> > been
>>>>>>>>>> > > about
>>>>>>>>>> > > > >> > >> overhead +
>>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if we
>>>>> have
>>>>>>>>>> > considered a
>>>>>>>>>> > > > varint
>>>>>>>>>> > > > >> > >> encoding
>>>>>>>>>> > > > >> > >> >> (
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> https://developers.google.com/
>>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
>>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the header
>>> length
>>>>>>>>> field (int32
>>>>>>>>>> > in the
>>>>>>>>>> > > > >> > proposal)
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
>>> headers,
>>>>>>> the
>>>>>>>>>> > overhead would
>>>>>>>>>> > > > be a
>>>>>>>>>> > > > >> > >> single
>>>>>>>>>> > > > >> > >> >> >>> byte
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > for each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128 would
>>>>> also
>>>>>>>>> need only a
>>>>>>>>>> > > single
>>>>>>>>>> > > > >> byte?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov 9,
>>>>> 2016
>>>>>>> at
>>>>>>>>> 6:43 AM,
>>>>>>>>>> > > radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus - and
>>>>> very
>>>>>>>>> dangerous
>>>>>>>>>> > (youre
>>>>>>>>>> > > > >> > >> essentially
>>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary code
>>>>> off
>>>>>>>>> the
>>>>>>>>>> > internet on
>>>>>>>>>> > > > your
>>>>>>>>>> > > > >> > >> servers
>>>>>>>>>> > > > >> > >> >> ...
>>>>>>>>>> > > > >> > >> >> >>> > bad
>>>>>>>>>> > > > >> > >> >> >>> > >> > idea
>>>>>>>>>> > > > >> > >> >> >>> > >> > > without
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox, even
>>>>> with)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
>>> being
>>>>> a
>>>>>>>>> purely
>>>>>>>>>> > > > administrative
>>>>>>>>>> > > > >> > task
>>>>>>>>>> > > > >> > >> - i
>>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
>>> would,
>>>>>>>>> really,
>>>>>>>>>> > because
>>>>>>>>>> > > > then my
>>>>>>>>>> > > > >> > >> earlier
>>>>>>>>>> > > > >> > >> >> >>> > point on
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the remapping
>>>>>>>>> process would
>>>>>>>>>> > be
>>>>>>>>>> > > > invalid,
>>>>>>>>>> > > > >> > but
>>>>>>>>>> > > > >> > >> at
>>>>>>>>>> > > > >> > >> >> >>> > linkedin,
>>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team im
>>> in)
>>>>>>> run
>>>>>>>>> kafka
>>>>>>>>>> > as a
>>>>>>>>>> > > > service.
>>>>>>>>>> > > > >> > we
>>>>>>>>>> > > > >> > >> dont
>>>>>>>>>> > > > >> > >> >> >>> > really
>>>>>>>>>> > > > >> > >> >> >>> > >> > know
>>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
>>>>>>>>> applications
>>>>>>>>>> > that use
>>>>>>>>>> > > > kafka)
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> up
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> >>> at
>>>>>>>>>> > > > >> > >> >> >>> > any
>>>>>>>>>> > > > >> > >> >> >>> > >> > given
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
>>> possible
>>>>>>>>> (given the
>>>>>>>>>> > > > existance of
>>>>>>>>>> > > > >> > >> headers
>>>>>>>>>> > > > >> > >> >> >>> and a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem) for
>>>>> some
>>>>>>>>>> > application to
>>>>>>>>>> > > > >> "equip"
>>>>>>>>>> > > > >> > >> their
>>>>>>>>>> > > > >> > >> >> >>> > >> producers
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
>>>>> required
>>>>>>>>> plugin
>>>>>>>>>> > without us
>>>>>>>>>> > > > >> > knowing.
>>>>>>>>>> > > > >> > >> i
>>>>>>>>>> > > > >> > >> >> dont
>>>>>>>>>> > > > >> > >> >> >>> > mean
>>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
>>>>> want to
>>>>>>>>> make the
>>>>>>>>>> > point
>>>>>>>>>> > > > that
>>>>>>>>>> > > > >> > its
>>>>>>>>>> > > > >> > >> not
>>>>>>>>>> > > > >> > >> >> as
>>>>>>>>>> > > > >> > >> >> >>> > simple
>>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync across a
>>>>>>>>> large-enough
>>>>>>>>>> > > > organization.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed, Nov 9,
>>>>> 2016
>>>>>>>>> at 6:17
>>>>>>>>>> > AM,
>>>>>>>>>> > > > Magnus
>>>>>>>>>> > > > >> > >> Edenhill
>>>>>>>>>> > > > >> > >> >> <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
>>> there
>>>>> is
>>>>>>> a
>>>>>>>>> piece
>>>>>>>>>> > missing
>>>>>>>>>> > > in
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> >> Strings
>>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > where
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> pro-Stringers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason that
>>> by
>>>>>>>>> providing
>>>>>>>>>> > unique
>>>>>>>>>> > > > string
>>>>>>>>>> > > > >> > >> >> >>> identifiers
>>>>>>>>>> > > > >> > >> >> >>> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > everything
>>>>> will
>>>>>>>>> just
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
>>> work
>>>>>>> for
>>>>>>>>> all
>>>>>>>>>> > parts of
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> > stream
>>>>>>>>>> > > > >> > >> >> >>> pipeline.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
>>>>> strings
>>>>>>>>> dont mean
>>>>>>>>>> > > anything
>>>>>>>>>> > > > by
>>>>>>>>>> > > > >> > >> >> themselves,
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > while we
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
>>>>> envision
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
>>>>> plugin
>>>>>>>>> loader
>>>>>>>>>> > that
>>>>>>>>>> > > > >> downloads,
>>>>>>>>>> > > > >> > >> >> compiles,
>>>>>>>>>> > > > >> > >> >> >>> > links
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon as
>>>>>>> they're
>>>>>>>>> seen by
>>>>>>>>>> > a
>>>>>>>>>> > > > >> consumer, I
>>>>>>>>>> > > > >> > >> dont
>>>>>>>>>> > > > >> > >> >> >>> really
>>>>>>>>>> > > > >> > >> >> >>> > >> see
>>>>>>>>>> > > > >> > >> >> >>> > >> > a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so dynamic
>>>>> (and
>>>>>>>>> fragile) in
>>>>>>>>>> > > > practice.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the real
>>>>> world
>>>>>>>>> an
>>>>>>>>>> > application
>>>>>>>>>> > > > will
>>>>>>>>>> > > > >> be
>>>>>>>>>> > > > >> > >> >> >>> configured
>>>>>>>>>> > > > >> > >> >> >>> > >> with
>>>>>>>>>> > > > >> > >> >> >>> > >> > a set
>>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
>>> add
>>>>>>>>> (producer)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
>>>>>>> (consumer)
>>>>>>>>> headers.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is an
>>>>>>>>> administrative
>>>>>>>>>> > task
>>>>>>>>>> > > > based
>>>>>>>>>> > > > >> on
>>>>>>>>>> > > > >> > >> what
>>>>>>>>>> > > > >> > >> >> >>> > features a
>>>>>>>>>> > > > >> > >> >> >>> > >> > client
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> needs/provides
>>>>>>> and
>>>>>>>>> results
>>>>>>>>>> > in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some sort of
>>>>>>>>> configuration
>>>>>>>>>> > to
>>>>>>>>>> > > > enable
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> >> >>> configure
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > desired
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since this
>>>>> needs
>>>>>>>>> to be kept
>>>>>>>>>> > > > somewhat
>>>>>>>>>> > > > >> in
>>>>>>>>>> > > > >> > >> sync
>>>>>>>>>> > > > >> > >> >> >>> across
>>>>>>>>>> > > > >> > >> >> >>> > an
>>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no point
>>> in
>>>>>>>>> having
>>>>>>>>>> > producers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add headers
>>> no
>>>>>>>>> consumers
>>>>>>>>>> > will
>>>>>>>>>> > > read,
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> vice
>>>>>>>>>> > > > >> > >> >> >>> versa),
>>>>>>>>>> > > > >> > >> >> >>> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > added
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of assigning
>>>>> an
>>>>>>> id
>>>>>>>>>> > namespace
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
>>>>> plugin
>>>>>>> as
>>>>>>>>> it is
>>>>>>>>>> > being
>>>>>>>>>> > > > >> > configured
>>>>>>>>>> > > > >> > >> >> should
>>>>>>>>>> > > > >> > >> >> >>> be
>>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > 2016-11-09
>>>>> 13:06
>>>>>>>>> GMT+01:00
>>>>>>>>>> > > Michael
>>>>>>>>>> > > > >> > Pearce <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Michael.Pearce@ig.com>:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
>>>>>>>>> following/catching
>>>>>>>>>> > up on
>>>>>>>>>> > > > what
>>>>>>>>>> > > > >> > seems
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> be
>>>>>>>>>> > > > >> > >> >> >>> an
>>>>>>>>>> > > > >> > >> >> >>> > >> > active
>>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
>>>>> sorry if
>>>>>>>>> it may
>>>>>>>>>> > seem
>>>>>>>>>> > > > obvious
>>>>>>>>>> > > > >> > but
>>>>>>>>>> > > > >> > >> what
>>>>>>>>>> > > > >> > >> >> >>> does
>>>>>>>>>> > > > >> > >> >> >>> > MD
>>>>>>>>>> > > > >> > >> >> >>> > >> > stand
>>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My take on
>>>>>>>>> String vs Int:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
>>> state
>>>>>>>>> first I am
>>>>>>>>>> > pro Int
>>>>>>>>>> > > > (16
>>>>>>>>>> > > > >> or
>>>>>>>>>> > > > >> > >> 32).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
>>> though
>>>>>>>>> playing
>>>>>>>>>> > devils
>>>>>>>>>> > > > advocate
>>>>>>>>>> > > > >> > see a
>>>>>>>>>> > > > >> > >> >> big
>>>>>>>>>> > > > >> > >> >> >>> plus
>>>>>>>>>> > > > >> > >> >> >>> > >> > with the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
>>> keys,
>>>>>>>>> this is
>>>>>>>>>> > around
>>>>>>>>>> > > > >> > integrating
>>>>>>>>>> > > > >> > >> >> into an
>>>>>>>>>> > > > >> > >> >> >>> > >> > existing
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
>>>>> other
>>>>>>>>> systems use
>>>>>>>>>> > > String
>>>>>>>>>> > > > >> based
>>>>>>>>>> > > > >> > >> >> headers
>>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
>>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
>>> easier
>>>>> for
>>>>>>>>> these to
>>>>>>>>>> > be
>>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
>>>>>>>>>> > > > >> > >> >> >>> > >> > into.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How with
>>> Int
>>>>>>>>> based
>>>>>>>>>> > headers
>>>>>>>>>> > > could
>>>>>>>>>> > > > we
>>>>>>>>>> > > > >> > >> provide
>>>>>>>>>> > > > >> > >> >> a
>>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> integration
>>>>>>>>> simple /
>>>>>>>>>> > easy with
>>>>>>>>>> > > > >> > transition
>>>>>>>>>> > > > >> > >> >> flows
>>>>>>>>>> > > > >> > >> >> >>> > over
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
>>> luck
>>>>>>>>> buddy
>>>>>>>>>> > you're on
>>>>>>>>>> > > your
>>>>>>>>>> > > > >> own
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * simply
>>>>> hash
>>>>>>>>> the string
>>>>>>>>>> > into
>>>>>>>>>> > > int
>>>>>>>>>> > > > >> code
>>>>>>>>>> > > > >> > >> and
>>>>>>>>>> > > > >> > >> >> hope
>>>>>>>>>> > > > >> > >> >> >>> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> no
>>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
>>> back
>>>>>>>>> though?)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
>>>>> style
>>>>>>> as
>>>>>>>>>> > mentioned by
>>>>>>>>>> > > > nacho.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > ______________________________
>>>>>>>>>> > > > >> > __________
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
>>> radai
>>>>> <
>>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
>>>>>>> Wednesday,
>>>>>>>>>> > November 9,
>>>>>>>>>> > > 2016
>>>>>>>>>> > > > >> > 8:12 AM
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
>>>>>>>>> dev@kafka.apache.org
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Subject:
>>> Re:
>>>>>>>>> [DISCUSS]
>>>>>>>>>> > KIP-82 -
>>>>>>>>>> > > > Add
>>>>>>>>>> > > > >> > >> Record
>>>>>>>>>> > > > >> > >> >> >>> Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > thinking
>>>>> about
>>>>>>>>> it some
>>>>>>>>>> > more,
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> best
>>>>>>>>>> > > > >> > >> way to
>>>>>>>>>> > > > >> > >> >> >>> > transmit
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
>>>>>>>>> consumers would
>>>>>>>>>> > be to
>>>>>>>>>> > > > put it
>>>>>>>>>> > > > >> > in
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> MD
>>>>>>>>>> > > > >> > >> >> >>> > >> response
>>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it should
>>> be
>>>>>>>>> discussed
>>>>>>>>>> > now.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
>>> Nov
>>>>> 9,
>>>>>>>>> 2016 at
>>>>>>>>>> > 12:09
>>>>>>>>>> > > AM,
>>>>>>>>>> > > > >> > radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>> radai.rosenblatt@gmail.com
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im not
>>>>>>> opposed
>>>>>>>>> to the
>>>>>>>>>> > idea of
>>>>>>>>>> > > > >> > namespace
>>>>>>>>>> > > > >> > >> >> >>> mapping.
>>>>>>>>>> > > > >> > >> >> >>> > >> all
>>>>>>>>>> > > > >> > >> >> >>> > >> > im
>>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not part
>>>>> of
>>>>>>>>> the "mvp"
>>>>>>>>>> > and,
>>>>>>>>>> > > > since
>>>>>>>>>> > > > >> it
>>>>>>>>>> > > > >> > >> >> requires
>>>>>>>>>> > > > >> > >> >> >>> no
>>>>>>>>>> > > > >> > >> >> >>> > >> wire
>>>>>>>>>> > > > >> > >> >> >>> > >> > format
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > always
>>> be
>>>>>>>>> added later.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
>>> its
>>>>> not
>>>>>>>>> as
>>>>>>>>>> > simple as
>>>>>>>>>> > > just
>>>>>>>>>> > > > >> > >> >> configuring
>>>>>>>>>> > > > >> > >> >> >>> MM
>>>>>>>>>> > > > >> > >> >> >>> > to
>>>>>>>>>> > > > >> > >> >> >>> > >> do
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say i've
>>>>>>>>> implemented
>>>>>>>>>> > large
>>>>>>>>>> > > > message
>>>>>>>>>> > > > >> > >> >> support as
>>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > target
>>>>>>> cluster
>>>>>>>>> its been
>>>>>>>>>> > > > remapped
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> >> {999,1}.
>>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
>>> need
>>>>> to
>>>>>>>>> be told
>>>>>>>>>> > to look
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> large
>>>>>>>>>> > > > >> > >> >> >>> > message
>>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
>>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
>>>>>>> {999,1}.
>>>>>>>>> doable,
>>>>>>>>>> > but
>>>>>>>>>> > > > tricky.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On Tue,
>>>>> Nov
>>>>>>> 8,
>>>>>>>>> 2016 at
>>>>>>>>>> > 10:29
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > Gwen
>>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> gwen@confluent.io
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> While
>>> you
>>>>>>> can
>>>>>>>>> do
>>>>>>>>>> > whatever
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> want
>>>>>>>>>> > > > >> > >> with a
>>>>>>>>>> > > > >> > >> >> >>> > >> namespace
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > your
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
>>> I'd
>>>>>>>>> expect is
>>>>>>>>>> > for each
>>>>>>>>>> > > > app
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> >> >>> namespaces
>>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So if I
>>>>>>>>> accidentally
>>>>>>>>>> > used
>>>>>>>>>> > > 666
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> > my
>>>>>>>>>> > > > >> > >> HR
>>>>>>>>>> > > > >> > >> >> >>> > >> department,
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > still
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
>>>>>>> RadaiApp,
>>>>>>>>> I can
>>>>>>>>>> > config
>>>>>>>>>> > > > >> > >> "namespace=42"
>>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
>>> look
>>>>>>>>> normal.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
>>>>> means
>>>>>>>>> you only
>>>>>>>>>> > need to
>>>>>>>>>> > > > sync
>>>>>>>>>> > > > >> > usage
>>>>>>>>>> > > > >> > >> >> >>> inside
>>>>>>>>>> > > > >> > >> >> >>> > your
>>>>>>>>>> > > > >> > >> >> >>> > >> > own
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> Still
>>>>> hard,
>>>>>>>>> but
>>>>>>>>>> > somewhat
>>>>>>>>>> > > > easier
>>>>>>>>>> > > > >> > than
>>>>>>>>>> > > > >> > >> >> syncing
>>>>>>>>>> > > > >> > >> >> >>> > with
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On Tue,
>>>>> Nov
>>>>>>>>> 8, 2016
>>>>>>>>>> > at 10:07
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > >> radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > and
>>> we
>>>>> can
>>>>>>>>> start
>>>>>>>>>> > with
>>>>>>>>>> > > > >> {namespace,
>>>>>>>>>> > > > >> > >> id}
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> no
>>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> always
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > add
>>> it
>>>>>>>>> later on
>>>>>>>>>> > if/when
>>>>>>>>>> > > > >> > collisions
>>>>>>>>>> > > > >> > >> >> >>> actually
>>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
>>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
>>>>>>> problem).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > every
>>>>>>>>> interested
>>>>>>>>>> > party (so
>>>>>>>>>> > > > orgs
>>>>>>>>>> > > > >> > or
>>>>>>>>>> > > > >> > >> >> >>> > individuals)
>>>>>>>>>> > > > >> > >> >> >>> > >> > could
>>>>>>>>>> > > > >> > >> >> >>> > >> > > then
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > register
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> prefix
>>>>> (0
>>>>>>> =
>>>>>>>>>> > reserved, 1 =
>>>>>>>>>> > > > >> > confluent
>>>>>>>>>> > > > >> > >> ...
>>>>>>>>>> > > > >> > >> >> >>> 666
>>>>>>>>>> > > > >> > >> >> >>> > = me
>>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
>>>>>>>>>> > > > >> > >> >> >>> > >> > > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > the
>>>>> 2nd ID
>>>>>>>>> - so once
>>>>>>>>>> > > > linkedin
>>>>>>>>>> > > > >> > >> >> registers,
>>>>>>>>>> > > > >> > >> >> >>> say
>>>>>>>>>> > > > >> > >> >> >>> > 3,
>>>>>>>>>> > > > >> > >> >> >>> > >> > then
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
>>> use
>>>>> {3,
>>>>>>>>> *} with a
>>>>>>>>>> > > > reasonable
>>>>>>>>>> > > > >> > >> >> >>> expectation
>>>>>>>>>> > > > >> > >> >> >>> > to
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > anything
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > else.
>>>>>>>>> further
>>>>>>>>>> > partitioning
>>>>>>>>>> > > > of
>>>>>>>>>> > > > >> > that *
>>>>>>>>>> > > > >> > >> >> >>> becomes
>>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>> "upstream
>>>>>>>>>> > registration"
>>>>>>>>>> > > of a
>>>>>>>>>> > > > >> > >> namespace
>>>>>>>>>> > > > >> > >> >> >>> only
>>>>>>>>>> > > > >> > >> >> >>> > has
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
>>> Tue,
>>>>>>> Nov
>>>>>>>>> 8, 2016
>>>>>>>>>> > at
>>>>>>>>>> > > 9:03
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > >> James
>>>>>>>>>> > > > >> > >> >> >>> Cheng <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>> wushujames@gmail.com
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>>>>> Nov
>>>>>>>>> 8, 2016,
>>>>>>>>>> > at 5:54
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > Gwen
>>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > gwen@confluent.io
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>> Thank
>>>>>>>>> you so
>>>>>>>>>> > much for
>>>>>>>>>> > > > this
>>>>>>>>>> > > > >> > clear
>>>>>>>>>> > > > >> > >> and
>>>>>>>>>> > > > >> > >> >> >>> fair
>>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > arguments.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> I'm
>>>>> in
>>>>>>>>> favor of
>>>>>>>>>> > ints.
>>>>>>>>>> > > > Not a
>>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
>>>>>>>>>> > > > >> > >> >> >>> > but
>>>>>>>>>> > > > >> > >> >> >>> > >> > in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> Even
>>>>>>>>> more in
>>>>>>>>>> > favor of
>>>>>>>>>> > > > >> Magnus's
>>>>>>>>>> > > > >> > >> >> >>> > decentralized
>>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>> tweak:
>>>>>>>>> add a
>>>>>>>>>> > namespace
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> > >> headers.
>>>>>>>>>> > > > >> > >> >> >>> This
>>>>>>>>>> > > > >> > >> >> >>> > will
>>>>>>>>>> > > > >> > >> >> >>> > >> > allow
>>>>>>>>>> > > > >> > >> >> >>> > >> > > each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> use
>>>>>>>>> whatever IDs
>>>>>>>>>> > it
>>>>>>>>>> > > wants
>>>>>>>>>> > > > >> > >> >> internally,
>>>>>>>>>> > > > >> > >> >> >>> and
>>>>>>>>>> > > > >> > >> >> >>> > >> then
>>>>>>>>>> > > > >> > >> >> >>> > >> > let
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> deploying
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> the
>>>>> app
>>>>>>>>> figure
>>>>>>>>>> > out an
>>>>>>>>>> > > > >> > available
>>>>>>>>>> > > > >> > >> >> >>> namespace
>>>>>>>>>> > > > >> > >> >> >>> > ID
>>>>>>>>>> > > > >> > >> >> >>> > >> > for the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > So
>>>>>>>>>> > > > >> > io.confluent.schema-registry
>>>>>>>>>> > > > >> > >> can
>>>>>>>>>> > > > >> > >> >> be
>>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>> deployment
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> and
>>>>>>> 0x57
>>>>>>>>> on
>>>>>>>>>> > yours, and
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > poor
>>>>>>>>>> > > > >> > >> guys
>>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>> worry
>>>>>>>>> about that.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> Gwen,
>>>>> if
>>>>>>> I
>>>>>>>>>> > understand
>>>>>>>>>> > > your
>>>>>>>>>> > > > >> > example
>>>>>>>>>> > > > >> > >> >> >>> right, an
>>>>>>>>>> > > > >> > >> >> >>> > >> > > application
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> decide to
>>>>>>>>> use 0x01
>>>>>>>>>> > in one
>>>>>>>>>> > > > >> > >> deployment,
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > means
>>>>>>>>>> > > > >> > >> >> >>> > >> > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> message
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> is
>>>>>>> written
>>>>>>>>> into the
>>>>>>>>>> > > > broker, it
>>>>>>>>>> > > > >> > >> will be
>>>>>>>>>> > > > >> > >> >> >>> > saved on
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> specific
>>>>>>>>> namespace
>>>>>>>>>> > > (0x01).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> If
>>> you
>>>>>>>>> were to
>>>>>>>>>> > mirror
>>>>>>>>>> > > that
>>>>>>>>>> > > > >> > message
>>>>>>>>>> > > > >> > >> >> into
>>>>>>>>>> > > > >> > >> >> >>> > another
>>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> accompany
>>>>>>>>> the
>>>>>>>>>> > message,
>>>>>>>>>> > > > right?
>>>>>>>>>> > > > >> > What
>>>>>>>>>> > > > >> > >> if
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> other
>>>>>>>>> cluster uses
>>>>>>>>>> > 0x57?
>>>>>>>>>> > > > They
>>>>>>>>>> > > > >> > won't
>>>>>>>>>> > > > >> > >> >> >>> > understand
>>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> I'm
>>>>> not
>>>>>>>>> sure
>>>>>>>>>> > that's an
>>>>>>>>>> > > > >> avoidable
>>>>>>>>>> > > > >> > >> >> >>> problem. I
>>>>>>>>>> > > > >> > >> >> >>> > >> > think it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> order
>>>>> to
>>>>>>>>> share
>>>>>>>>>> > data, you
>>>>>>>>>> > > > have
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> also
>>>>>>>>>> > > > >> > >> >> >>> have a
>>>>>>>>>> > > > >> > >> >> >>> > >> > shared
>>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> understanding of
>>>>>>>>>> > what the
>>>>>>>>>> > > > >> > >> namespaces
>>>>>>>>>> > > > >> > >> >> >>> mean.
>>>>>>>>>> > > > >> > >> >> >>> > >> Which
>>>>>>>>>> > > > >> > >> >> >>> > >> > I
>>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> because
>>>>>>> the
>>>>>>>>>> > alternate
>>>>>>>>>> > > > (sharing
>>>>>>>>>> > > > >> > >> >> *nothing*
>>>>>>>>>> > > > >> > >> >> >>> at
>>>>>>>>>> > > > >> > >> >> >>> > >> all)
>>>>>>>>>> > > > >> > >> >> >>> > >> > would
>>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> would
>>>>> be
>>>>>>>>> no way to
>>>>>>>>>> > > > understand
>>>>>>>>>> > > > >> > each
>>>>>>>>>> > > > >> > >> >> other.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> -James
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> Gwen
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>>>>> Tue,
>>>>>>>>> Nov 8,
>>>>>>>>>> > 2016 at
>>>>>>>>>> > > > 4:23
>>>>>>>>>> > > > >> > PM,
>>>>>>>>>> > > > >> > >> >> radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> radai.rosenblatt@gmail.com
>>>>>>>>>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>> +1
>>>>> for
>>>>>>>>> sean's
>>>>>>>>>> > > document.
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > >> covers
>>>>>>>>>> > > > >> > >> >> >>> pretty
>>>>>>>>>> > > > >> > >> >> >>> > >> much
>>>>>>>>>> > > > >> > >> >> >>> > >> > all
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>>>>>> provides
>>>>>>>>>> > concrete
>>>>>>>>>> > > > figures
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> argue
>>>>>>>>>> > > > >> > >> >> >>> about
>>>>>>>>>> > > > >> > >> >> >>> > :-)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>>>>>>>> (nit-picking -
>>>>>>>>>> > used
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> same
>>>>>>>>>> > > > >> > >> xkcd
>>>>>>>>>> > > > >> > >> >> >>> twice,
>>>>>>>>>> > > > >> > >> >> >>> > >> also
>>>>>>>>>> > > > >> > >> >> >>> > >> > trove
>>>>>>>>>> > > > >> > >> >> >>> > >> > > has
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>> superceded
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > --
>>>>>>>>>> > > > >> > Gwen Shapira
>>>>>>>>>> > > > >> > Product Manager | Confluent
>>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>>>>>>>>> > > > >> > Follow us: Twitter | blog
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> --
>>>>>>>>>> > > > >> *Todd Palino*
>>>>>>>>>> > > > >> Staff Site Reliability Engineer
>>>>>>>>>> > > > >> Data Infrastructure Streaming
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> linkedin.com/in/toddpalino
>>>>>>>>>> > > > >>
>>>>>>>>>> > > >
>>>>>>>>>> > > >
>>>>>>>>>> > > >
>>>>>>>>>> > > > --
>>>>>>>>>> > > > Gwen Shapira
>>>>>>>>>> > > > Product Manager | Confluent
>>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>>>>>>>>> > > > Follow us: Twitter | blog
>>>>>>>>>> > > >
>>>>>>>>>> > >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > The information contained in this email is strictly
>>>>> confidential
>>>>>>>>> and for
>>>>>>>>>> > the use of the addressee only, unless otherwise indicated.
>>> If
>>>>> you
>>>>>>>>> are not
>>>>>>>>>> > the intended recipient, please do not read, copy, use or
>>>>> disclose
>>>>>>>>> to others
>>>>>>>>>> > this message or any attachment. Please also notify the
>>> sender
>>>>> by
>>>>>>>>> replying
>>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and then
>>>>> delete
>>>>>>>>> the email
>>>>>>>>>> > and any copies of it. Opinions, conclusion (etc) that do not
>>>>>>>>> relate to the
>>>>>>>>>> > official business of this company shall be understood as
>>>>> neither
>>>>>>>>> given nor
>>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets Limited
>>> (a
>>>>>>>>> company
>>>>>>>>>> > registered in England and Wales, company number 04008957)
>>> and
>>>>> IG
>>>>>>>>> Index
>>>>>>>>>> > Limited (a company registered in England and Wales, company
>>>>>>> number
>>>>>>>>>> > 01190902). Registered address at Cannon Bridge House, 25
>>>>> Dowgate
>>>>>>>>> Hill,
>>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register number
>>>>> 195355)
>>>>>>>>> and IG
>>>>>>>>>> > Index Limited (register number 114059) are authorised and
>>>>>>>>> regulated by the
>>>>>>>>>> > Financial Conduct Authority.
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The information contained in this email is strictly confidential
>>> and
>>>>>>> for
>>>>>>>>> the use of the addressee only, unless otherwise indicated. If you
>>> are
>>>>>>> not
>>>>>>>>> the intended recipient, please do not read, copy, use or disclose
>>> to
>>>>>>> others
>>>>>>>>> this message or any attachment. Please also notify the sender by
>>>>>>> replying
>>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
>>> the
>>>>>>> email
>>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not relate
>>>>> to
>>>>>>> the
>>>>>>>>> official business of this company shall be understood as neither
>>>>> given
>>>>>>> nor
>>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
>>> company
>>>>>>>>> registered in England and Wales, company number 04008957) and IG
>>>>> Index
>>>>>>>>> Limited (a company registered in England and Wales, company number
>>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>>>>> Hill,
>>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
>>>>> and IG
>>>>>>>>> Index Limited (register number 114059) are authorised and regulated
>>>>> by
>>>>>>> the
>>>>>>>>> Financial Conduct Authority.
>>>>>>>>>
>>>>>>>
>>>>>
>>> The information contained in this email is strictly confidential and for
>>> the use of the addressee only, unless otherwise indicated. If you are not
>>> the intended recipient, please do not read, copy, use or disclose to others
>>> this message or any attachment. Please also notify the sender by replying
>>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>>> official business of this company shall be understood as neither given nor
>>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>>> registered in England and Wales, company number 04008957) and IG Index
>>> Limited (a company registered in England and Wales, company number
>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>>> Index Limited (register number 114059) are authorised and regulated by the
>>> Financial Conduct Authority.
>>>
>
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jeroen van Disseldorp <je...@axual.io>.
I'm lagging a bit here too, but want to respond to the usefulness of
this KIP for two use cases: bidirectional mirroring and smart cluster
switches.
Bidirectional mirroring
For a Dutch bank we have currently implemented bidirectional mirroring
using our own headers, which we encode in the Value using a custom
serializer. It encodes just one byte extra, containing a "copy/original"
bit. When applications serialize an avro object, the bit is unset by
default. Our mirroring mechanism subscribes to all topics, checks the
bit, and when unset the bit is set and the message is forwarded to the
other DC.
The overhead of this is pretty low (1 byte) and it's cheap, since the
mirroring mechanism does not need to fully deserialize the value, but
just check+flip one bit in the byte[] value. On the receiving end, the
mirroring mechanism sees the message for the second time, but notices
the message is a copy already and ignores it.
The mechanism has been in production since last year and it works really
well. Its downside however is the incompatibility with avro serialized
values, as the whole organization is mandated to use our own custom
serde and we were forced to create a wrapper client library around the
default kafka client. The callback we offer to applications is very
similar to that proposed in the KIP, as every message our users get
consists of <Metadata, K, V>.
Additionally our header contains a message id and serialization
timestamp (invented before 0.10.0.0). These fields could also be done
away with when this KIP is implemented.
Smart cluster switches
Another case we are considering to use headers for is enabling "smart
cluster switches" for consumers and producers in a way that would avoid
duplicates and/or missed messages in most cases. This involves encoding
a producer id (app name or id), producer instance id (uuid generated
upon producer's start) and sequence number in every message. We expect
to use one sequence per partition.
These fields enable our consumers to keep track of which messages were
seen without relying on offsets, which is a cluster-bound value.
Consumers keep track of seen messages by keeping a small administration
of ranges of sequence numbers per producer instance and partition. Since
sequence numbers are always increasing per producer and partition, the
memory consumption of such administration is pretty low, while being
meaningful across clusters. After every poll() we can commit the
administration to a topic in much the same way as we do with offsets.
The difference is that this topic can be mirrored to the other cluster
and upon a switch of consumer, all instances can use the topic to
initialize themselves and skip all messages already processed.
Hope these views help in the upcoming decision on this KIP.
Regards
Jeroen
On 09/01/2017 20:40, Gwen Shapira wrote:
> Mirroring + ClusterIDs:
>
> Kafka clusters already have an automatically generated ID that is
> exposed to clients:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
>
> It sounds trivial to get MirrorMaker's consumer to grab this ID (which
> always exists) from the source cluster and add it to the message
> headers.
>
> Obviously adding stuff to the message makes it larger so this should
> be a feature you should be able to turn on/off depending on the
> replication requirements. If you need the feature - you pay the price.
> I don't think there is an immediate need to put an effort into
> shrinking the Cluster ID, but others can comment on that.
>
>
>
>
>
> On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
>> Hey Michael,
>>
>> Sorry for lagging, I've been out for the holiday break. What would help me
>> would be to get to a consolidated list of use cases we've worked out and
>> think add real value. I think getting down to the details really matters
>> for these.
>>
>> The two lists I'm aware of are the one in the KIP and the one you gave in
>> this email thread. Both seem to be more at the sketch level. Let me review
>> the use cases and see if I have it right. If so let's get it in the design
>> proposal.
>>
>> Your use cases:
>>
>> 1. Tracing IDs -- I get this use case and think I see how it would work.
>> I think because the tracing is from a third party system there is some
>> value to having a header versus a custom format to make this work fully
>> transparently.
>> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
>> thinking below
>> 3. End-to-end encryption -- I think what you are saying is that if you
>> want "routing' fields that are accessible in transit without decrypting you
>> need a scheme to leave this decrypted.I think the idea is to avoid
>> decrypting to get a header, but I'm not sure if I get how this is better
>> than a custom message format that doesn't encrypt the things you don't want
>> encrypted since you have to be aware of the scheme anyway.
>> 4. EOS -- I'm not convinced this is a good approach for EOS.
>>
>> Linked from KIP:
>>
>> 1. Schema-Registry-aware avro serialization -- not clear how this is
>> better than keeping the schema id in the value
>> 2. Polyglot kafka topics -- marginally better than a container format,
>> but of course you still have to all agree on the the header and
>> header=>serialization mapping so not sure if it's really better than a
>> container format. Also not wild about polyglot topics :-)
>> 3. End-to-End routing trail including mirroring --- I'd put this under
>> tracing ids
>> 4. Cross-service process tracing -- ditto
>> 5. Signing -- I like this use case but it seems to depend on a feature
>> we don't have and aren't proposing adding, right?
>> 6. Per-record custom compression -- why do you want this? compressing
>> per message won't work well compared to batch compression, right?
>> 7. Large Message Support -- I think we can agree that although this
>> might work it's clearly not the right way to support large messages in Kafka
>>
>> Matthias
>>
>> 1. Control messages --- Matthias, not sure if I get the motivation in
>> streams for this. If we did want to do it, wouldn't it be easier to
>> implement and by allowing users to send control messages (in the EOS sense)
>> and adding a consumer config to allow either passing these on or
>> suppressing them?
>>
>> So I if I were to summarize I'd list the use cases as follows:
>>
>> - Much better with headers: tracing IDs
>> - Somewhat better: end-to-end encryption, polyglot topics
>> - Don't buy: Mirroring (though perhaps could be made to work with
>> changes to the proposal), EOS, large messages, compression
>> - Not sure: Control messages
>>
>> Does this match where other people are at? From my point of view for
>> something like this where we are changing the core data model we should
>> ideally get a handful of use cases to the point where they are really well
>> thought out and we know they can use the feature. I think we have one solid
>> one with tracing IDs, and I think maybe mirroring could be another with
>> some work. I'm less sure about the others.
>>
>> Bi-directional Mirroring
>> I think bidirectional mirroring could potentially become a compelling use
>> case if we an find a way to make it work. If I understand it, the proposal
>> is to add a header like "origin-cluster" with the cluster id of the Kafka
>> cluster. Then modify the mirroring tool to disallow mirroring a message
>> back to it's own origin cluster. This basic idea is definitely workable and
>> useful, but can we really get to a good implementation using headers?
>> I imagine the approach would be to add this header in the producer if some
>> config is set. There are two problems:
>>
>> - You add a pretty big honking cluster ID to each message. For small
>> messages this maybe larger than the contents itself.
>> - What happens if someone somewhere forgets to set that config on some
>> client somewhere?
>>
>> If someone forgets to set the config in some client somewhere I guess you
>> have various options in mirror maker:
>>
>> 1. drop the data (pretty bad, now your data is out of sync)
>> 2. crash (pretty bad--how do you recover)
>> 3. insert some default (pretty bad since would also lead to data loss if
>> that's not right)
>>
>> Basically although I think a cluster tag could be a good way to do
>> bi-directional mirroring, I'm not sure this naive approach I described
>> would get us to a usable implementation. This could be fixable. I think if
>> there was some scheme to check required headers on the server that could
>> alleviate the missing header problem. Not really sure what to do about the
>> message bloat problem, but maybe there could be some way to shrink the
>> cluster id?
>>
>> Cheers,
>>
>> -Jay
>>
>>
>> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Mi...@ig.com>
>> wrote:
>>
>>> @Jay, Ismael and a few others.
>>>
>>> Based on the discussion on kip-87, and the additional use case by Matthias.
>>>
>>> Just for clarity do we now have consensus on the earlier points (1)
>>> headers are useful (2) headers should be native in Kafka?
>>>
>>> Cheers
>>> Mike
>>> ________________________________________
>>> From: radai <ra...@gmail.com>
>>> Sent: Monday, December 19, 2016 5:01 PM
>>> To: dev@kafka.apache.org
>>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>>
>>> having said that im perfectly fine with something like a "no payload" flag
>>> (as opposed to null payload), which would mean such a msg is not returned
>>> from poll() unless the user code specifically "opts in" and would serve
>>> only to carry headers (making it a control message)
>>>
>>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com> wrote:
>>>
>>>> if all you want is to tag "end of stream" wouldnt a header on the last
>>> msg
>>>> in the stream be enough?
>>>>
>>>> On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <ma...@confluent.io>
>>>> wrote:
>>>>
>>>>> The details about headers for control messages are still to define. But
>>>>> yes, the idea is to have some common default behavior that clients would
>>>>> need to implement.
>>>>>
>>>>> The point is, that "regular headers" add meta data to regular messages.
>>>>> Thus, those messages will be returned to the user via .poll(). And after
>>>>> the message is received the user can check if meta data is present and
>>>>> read it.
>>>>>
>>>>> For control messages, we do not want those to pop up via .poll() as
>>>>> those are no regular messages. A client would need to opt-in to see
>>>>> those messages (either via poll() or maybe a callback). Thus, we need
>>>>> some special (standardized) header IDs that indicate control messages
>>>>> that should not be returned to the user via poll() by default.
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 12/17/16 9:37 PM, Roger Hoover wrote:
>>>>>> Matthias,
>>>>>>
>>>>>> Thanks for your input. I'm +1 on control messages as they seem to be
>>>>> the
>>>>>> simplest way to implement watermarks (
>>>>>> https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102),
>>> a
>>>>>> feature that would add a lot of value to Kafka Streams IMHO.
>>>>>>
>>>>>> Your argument that the control-message indicator belongs in the
>>>>> client-only
>>>>>> section of the record format make sense. Just to make sure I
>>>>> understand,
>>>>>> are you suggesting that control messages would be indicated by a
>>>>> standard
>>>>>> reserved header? By standard, I mean that ALL Kafka consumers would
>>>>> know
>>>>>> to handle these messages differently (possibly just ignoring them).
>>>>> This
>>>>>> would need to be added to the specification of the consumer protocol
>>> so
>>>>>> that all Kafka clients implement it, right? I think it's a good idea
>>>>> but
>>>>>> just checking.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Roger
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>>> matthias@confluent.io
>>>>>> wrote:
>>>>>>
>>>>>>> Yes and no. I did overload the term "control message".
>>>>>>>
>>>>>>> EOS control messages are for client-broker communication and thus
>>> never
>>>>>>> exposed to any application. And I think this is a good design because
>>>>>>> broker needs to understand those control messages. Thus, this should
>>> be
>>>>>>> a protocol change.
>>>>>>>
>>>>>>> The type of control messages I have in mind are for client-client
>>>>>>> (application-application) communication and the broker is agnostic to
>>>>>>> them. Thus, it should not be a protocol change.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 12/14/16 9:42 AM, radai wrote:
>>>>>>>> arent control messages getting pushed as their own top level
>>> protocol
>>>>>>>> change (and a fairly massive one) for the transactions KIP ?
>>>>>>>>
>>>>>>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>>>>> matthias@confluent.io>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I want to add a completely new angle to this discussion. For this,
>>> I
>>>>>>>>> want to propose an extension for the headers feature that enables
>>> new
>>>>>>>>> uses cases -- and those new use cases might convince people to
>>>>> support
>>>>>>>>> headers (of course including the larger scoped proposal).
>>>>>>>>>
>>>>>>>>> Extended Proposal:
>>>>>>>>>
>>>>>>>>> Allow messages with a certain header key to be special "control
>>>>>>>>> messages" (w/ o w/o payload) that are not exposed to an application
>>>>> via
>>>>>>>>> .poll().
>>>>>>>>>
>>>>>>>>> Thus, a consumer client would automatically skip over those
>>>>> messages. If
>>>>>>>>> an application knows about embedded control messages, it can "sing
>>>>> up"
>>>>>>>>> to those messages by the consumer client and either get a callback
>>> or
>>>>>>>>> the consumer auto-drop for this messages gets disabled (allowing to
>>>>>>>>> consumer those messages via poll()).
>>>>>>>>>
>>>>>>>>> (The details need further considerations/discussion. I just want to
>>>>>>>>> sketch the main idea.)
>>>>>>>>>
>>>>>>>>> Usage:
>>>>>>>>>
>>>>>>>>> There is a shared topic (ie, used by multiple applications) and a
>>>>>>>>> producer application wants to embed a special message in the topic
>>>>> for a
>>>>>>>>> dedicated consumer application. Because only one application will
>>>>>>>>> understand this message, it cannot be a regular message as this
>>> would
>>>>>>>>> break all applications that do not understand this message. The
>>>>> producer
>>>>>>>>> application would set a special metadata key and no consumer
>>>>> application
>>>>>>>>> would see this control message by default because they did not
>>> enable
>>>>>>>>> their consumer client to return this message in poll() (and the
>>>>> client
>>>>>>>>> would just drop this message with special metadata key). Only the
>>>>> single
>>>>>>>>> application that should receive this message, will subscribe to
>>> this
>>>>>>>>> message on its consumer client and process it.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Concrete Use Case: Kafka Streams
>>>>>>>>>
>>>>>>>>> In Kafka Streams, we would like to propagate "control messages"
>>> from
>>>>>>>>> subtopology to subtopology. There are multiple scenarios for which
>>>>> this
>>>>>>>>> would be useful. For example, currently we do not guarantee a
>>>>>>>>> "consistent shutdown" of an application. By this, I mean that input
>>>>>>>>> records might not be completely processed by the whole topology
>>>>> because
>>>>>>>>> the application shutdown happens "in between" and an intermediate
>>>>> result
>>>>>>>>> topic gets "stock" in an intermediate topic. Thus, a user would see
>>>>> an
>>>>>>>>> committed offset of the source topic of the application, but no
>>>>>>>>> corresponding result record in the output topic.
>>>>>>>>>
>>>>>>>>> Having "shutdown markers" would allow us, to first stop the
>>> upstream
>>>>>>>>> subtopology and write this marker into the intermediate topic and
>>> the
>>>>>>>>> downstream subtopology would only shut down itself after is sees
>>> the
>>>>>>>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
>>>>>>>>> "in-flight" messages got stuck in intermediate topics.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> A similar usage would be for KIP-95 (Incremental Batch Processing).
>>>>>>>>> There was a discussion about the proposed metadata topic, and we
>>>>> could
>>>>>>>>> avoid this metadata topic if we would have "control messages".
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Right now, we cannot insert an "application control message"
>>> because
>>>>>>>>> Kafka Streams does not own all topics it read/writes and thus might
>>>>>>>>> break other consumer application (as described above) if we inject
>>>>>>>>> random messages that are not understood by other apps.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Of course, one can work around "embedded control messaged" by using
>>>>> an
>>>>>>>>> additional topic to propagate control messaged between application
>>>>> (as
>>>>>>>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
>>>>> there
>>>>>>>>> are major concerns about adding this metadata topic in the KIP and
>>>>> this
>>>>>>>>> shows that other application that need a similar pattern might
>>> profit
>>>>>>>>> from topic embedded "control messages", too.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> One last important consideration: those "control messages" are used
>>>>> for
>>>>>>>>> client to client communication and are not understood by the
>>> broker.
>>>>>>>>> Thus, those messages should not be enabled within the message
>>> format
>>>>>>>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
>>>>> headers
>>>>>>>>> would be a nice way to implement them. Because KIP-82 did consider
>>>>> key
>>>>>>>>> namespaces for metatdata keys, this extension should not be an own
>>>>> KIP
>>>>>>>>> but should be included in KIP-82 to reserve a namespace for
>>> "control
>>>>>>>>> message" in the first place.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sorry for the long email... Looking forward to your feedback.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Matthias
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>>>>>>>>>> Hi Jun
>>>>>>>>>>
>>>>>>>>>> 100) each time a transaction exits a jvm for a remote system
>>>>> (HTTP/JMS/
>>>>>>>>> Hopefully one day kafka) the APM tools stich in a unique id
>>> (though I
>>>>>>>>> believe it contains the end2end uuid embedded in this id), on
>>>>> receiving
>>>>>>> the
>>>>>>>>> message at the receiving JVM the apm code takes this out, and
>>>>> continues
>>>>>>> its
>>>>>>>>> tracing on the that new thread. Both JVM\u2019s (and other languages the
>>>>> APM
>>>>>>>>> tool supports) send this data async back to the central controllers
>>>>>>> where
>>>>>>>>> the stiching togeather occurs. For this they need some header space
>>>>> for
>>>>>>>>> them to put this id.
>>>>>>>>>> 101) Yes indeed we have a business transaction Id in the payload.
>>>>>>> Though
>>>>>>>>> this is a system level tracing, that we need to have marry up. Also
>>>>> as
>>>>>>> per
>>>>>>>>> note on end2end encryption we\u2019d be unable to prove the flow if the
>>>>>>> payload
>>>>>>>>> is encrypted as we\u2019d not have access to this at certain points of
>>> the
>>>>>>> flow
>>>>>>>>> through the infrastructure/platform.
>>>>>>>>>>
>>>>>>>>>> 103) As said we use this mechanism in IG very successfully, as
>>>>> stated
>>>>>>>>> per key we guarantee the transaction producing app to handle the
>>>>>>>>> transaction of a key at one DC unless at point of critical failure
>>>>>>> where we
>>>>>>>>> have to flip processing to another. We care about key ordering.
>>>>>>>>>> I disagree on the offset comment for the partition solution unless
>>>>> you
>>>>>>>>> do full ISR, or expensive full XA transactions even with partitions
>>>>> you
>>>>>>>>> cannot fully guarantee offsets would match.
>>>>>>>>>> 105) Very much so, I need to have access at the platform level to
>>>>> the
>>>>>>>>> other meta data all mentioned, without having to need to have
>>> access
>>>>> to
>>>>>>> the
>>>>>>>>> encryption keys of the payload.
>>>>>>>>>> 106)
>>>>>>>>>> Techincally yes for AZ/Region/Cluster, but then we\u2019d need to have
>>> a
>>>>>>>>> global producerId register which would be very hard to
>>>>> enforce/ensure is
>>>>>>>>> current and correct, just to understand the message origins of its
>>>>>>>>> region/az/cluster for routing.
>>>>>>>>>> The client wrapper version, producerId can be the same, as
>>> obviously
>>>>>>> the
>>>>>>>>> producer could upgrade its wrapper, as such we need to know what
>>>>> wrapper
>>>>>>>>> version the message is created with.
>>>>>>>>>> Likewise the IP address, as stated we can have our producer move,
>>>>> where
>>>>>>>>> its IP would change.
>>>>>>>>>> 107)
>>>>>>>>>> UUID is set on the message by interceptors before actual producer
>>>>>>>>> transport send. This is for platform level message dedupe
>>> guarantee,
>>>>> the
>>>>>>>>> business payload should be agnostic to this. Please see
>>>>>>>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
>>>>> ection.html
>>>>>>>>> note this is not touching business payloads.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi, Michael,
>>>>>>>>>>
>>>>>>>>>> Thanks for the reply. I find it very helpful.
>>>>>>>>>>
>>>>>>>>>> Data lineage:
>>>>>>>>>> 100. I'd like to understand the APM use case a bit more. It
>>>>> sounds
>>>>>>>>> like
>>>>>>>>>> that those APM plugins can generate a transaction id that we
>>>>> could
>>>>>>>>>> potentially put in the header of every message. How would you
>>>>>>>>> typically
>>>>>>>>>> make use of such transaction ids? Are there other metadata
>>>>>>>>> associated with
>>>>>>>>>> the transaction id and if so, how are they propagated
>>>>> downstream?
>>>>>>>>>> 101. For the finance use case, if the concept of transaction
>>> is
>>>>>>>>> important,
>>>>>>>>>> wouldn't it be typically included in the message payload
>>>>> instead of
>>>>>>>>> as an
>>>>>>>>>> optional header field?
>>>>>>>>>>
>>>>>>>>>> 102. The data lineage that Altas and Navigator support seems
>>> to
>>>>> be
>>>>>>>>> at the
>>>>>>>>>> dataset level, not per record level? So, not sure if per
>>> message
>>>>>>>>> headers
>>>>>>>>>> are relevant there.
>>>>>>>>>>
>>>>>>>>>> Mirroring:
>>>>>>>>>> 103. The benefit of using separate partitions is that it
>>>>>>> potentially
>>>>>>>>> makes
>>>>>>>>>> it easy to preserve offsets during mirroring. This will make
>>> it
>>>>>>>>> easier for
>>>>>>>>>> consumer to switch clusters. Currently, the consumers can
>>> switch
>>>>>>>>> clusters
>>>>>>>>>> by using the timestampToOffset() api, but it has to deal with
>>>>>>>>> duplicates.
>>>>>>>>>> Good point on the issue with log compact and I am not sure how
>>>>> to
>>>>>>>>> address
>>>>>>>>>> this. However, even if we mirror into the existing partitions,
>>>>> the
>>>>>>>>> ordering
>>>>>>>>>> for messages generated from different clusters seems
>>>>>>>>> non-deterministic
>>>>>>>>>> anyway. So, it seems that the consumers already have to deal
>>>>> with
>>>>>>>>> that? If
>>>>>>>>>> a topic is compacted, does that mean which messages are
>>>>> preserved
>>>>>>> is
>>>>>>>>> also
>>>>>>>>>> non-deterministic across clusters?
>>>>>>>>>>
>>>>>>>>>> 104. Good point on partition key.
>>>>>>>>>>
>>>>>>>>>> End-to-end encryption:
>>>>>>>>>> 105. So, it seems end-to-end encryption is useful. Are headers
>>>>>>>>> useful there?
>>>>>>>>>> Auditing:
>>>>>>>>>> 106. It seems other than the UUID, all other metadata are per
>>>>>>>>> producer?
>>>>>>>>>> EOS:
>>>>>>>>>> 107. How are those UUIDs generated? I am not sure if they can
>>> be
>>>>>>>>> generated
>>>>>>>>>> in the producer library. An application may send messages
>>>>> through a
>>>>>>>>> load
>>>>>>>>>> balancer and on retry, the same message could be routed to a
>>>>>>>>> different
>>>>>>>>>> producer instance. So, it seems that the application has to
>>>>>>> generate
>>>>>>>>> the
>>>>>>>>>> UUIDs. In that case, shouldn't the application just put the
>>>>> UUID in
>>>>>>>>> the
>>>>>>>>>> payload?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Jun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>>>>>>>>> Michael.Pearce@ig.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> > Hi Jun.
>>>>>>>>>> >
>>>>>>>>>> > Per Transaction Tracing / Data Lineage.
>>>>>>>>>> >
>>>>>>>>>> > As Stated in the KIP this has the first use case of how many
>>>>> APM
>>>>>>>>> tools now
>>>>>>>>>> > work.
>>>>>>>>>> > I would find it impossible for any one to argue this is not
>>>>>>>>> important or a
>>>>>>>>>> > niche market as it has its own gartner report for this
>>> space.
>>>>>>> Such
>>>>>>>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular are
>>>>> but a
>>>>>>>>> few.
>>>>>>>>>> >
>>>>>>>>>> > Likewise these APM tools can help very rapidly track down
>>>>> issues
>>>>>>>>> and
>>>>>>>>>> > automatically capture metrics, perform actions based on
>>>>>>> unexpected
>>>>>>>>> behavior
>>>>>>>>>> > to auto recover services.
>>>>>>>>>> >
>>>>>>>>>> > Before mentioning looking at aggregated stats, in these
>>> cases
>>>>>>> where
>>>>>>>>>> > actually on critical flows we cannot afford to have
>>> aggregated
>>>>>>>>> rolled up
>>>>>>>>>> > stats only.
>>>>>>>>>> >
>>>>>>>>>> > With the APM tool we use its actually able to detect a
>>> single
>>>>>>>>> transaction
>>>>>>>>>> > failure and capture the thread traces in the JVM where it
>>>>> failed
>>>>>>>>> and
>>>>>>>>>> > everything for us, to the point it sends us alerts where we
>>>>> have
>>>>>>>>> this
>>>>>>>>>> > giving the line number of the code that caused it, the
>>>>>>> transaction
>>>>>>>>> trace
>>>>>>>>>> > through all the services and endpoints (supported) upto the
>>>>> point
>>>>>>>>> of
>>>>>>>>>> > failure, it can also capture the data in and out (so we can
>>>>>>>>> replay).
>>>>>>>>>> > Because atm Kafka doesn\u2019t support us being able to stich in
>>>>> these
>>>>>>>>> tracing
>>>>>>>>>> > transaction ids natively, we cannot get these benefits as
>>>>> such is
>>>>>>>>> limiting
>>>>>>>>>> > our ability support apps and monitor them to the same
>>>>> standards
>>>>>>> we
>>>>>>>>> come to
>>>>>>>>>> > expect when on a kafka flow.
>>>>>>>>>> >
>>>>>>>>>> > This actually ties in with Data Lineage, as the same tracing
>>>>> can
>>>>>>>>> be used
>>>>>>>>>> > to back stich this. Essentially many times due to the sums
>>> of
>>>>>>> money
>>>>>>>>>> > involved there are disputes, and typically as a financial
>>>>>>>>> institute the
>>>>>>>>>> > easiest and cleanest way to prove when disputes arise is to
>>>>>>>>> present the
>>>>>>>>>> > actual flow and processes involved in a transaction.
>>>>>>>>>> >
>>>>>>>>>> > Likewise as Hadoop matures its evident this case is
>>>>> important, as
>>>>>>>>> tools
>>>>>>>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera led)
>>>>> are
>>>>>>>>> evident
>>>>>>>>>> > also I believe the importance here is very much NOT just a
>>>>>>>>> financial issue.
>>>>>>>>>> >
>>>>>>>>>> > From a MDM point of view any company wanting to care about
>>>>> Data
>>>>>>>>> Quality
>>>>>>>>>> > and Data Governance - Data Lineage is a key piece in this
>>>>> puzzle.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE Mirroring,
>>>>>>>>>> >
>>>>>>>>>> > As per the KIP in-fact this is exactly what we do re cluster
>>>>> id,
>>>>>>>>> to mirror
>>>>>>>>>> > a network of clusters between AZ\u2019s / Regions. We know a
>>>>>>>>> transaction for a
>>>>>>>>>> > key will be done within a AZ/Region, as such we know the
>>>>> write
>>>>>>> to
>>>>>>>>> kafka
>>>>>>>>>> > would be ordered per key. But we need eventual view of that
>>>>>>> across
>>>>>>>>> in our
>>>>>>>>>> > other regions/az\u2019s. When we have complete AZ or Region
>>>>> failure we
>>>>>>>>> know
>>>>>>>>>> > there will be a brief interruption whilst those transactions
>>>>> are
>>>>>>>>> moved to
>>>>>>>>>> > another region but we expect after it to continue.
>>>>>>>>>> >
>>>>>>>>>> > As mentioned having separate Partions to do this starts to
>>> get
>>>>>>>>>> > ugly/complicated for us:
>>>>>>>>>> > how would I do compaction where a key is in two partitions?
>>>>>>>>>> > How do we balance consumers so where multiple partitions
>>> with
>>>>> the
>>>>>>>>> same key
>>>>>>>>>> > goto the same consumer
>>>>>>>>>> > What do you do if cluster 1 has 5 partitions but cluster 20
>>>>> has
>>>>>>> 10
>>>>>>>>> because
>>>>>>>>>> > its larger kit in our more core DC\u2019s, as such key to
>>> partition
>>>>>>>>> mappings for
>>>>>>>>>> > consumers get even more complicated.
>>>>>>>>>> > What do you do if we add or remove a complete region
>>>>>>>>>> >
>>>>>>>>>> > Where as simple mirror will work we just need to ensure we
>>>>> don\u2019t
>>>>>>>>> have a
>>>>>>>>>> > cycle which we can do with clusterId.
>>>>>>>>>> >
>>>>>>>>>> > We even have started to look at shortest path mirror routing
>>>>>>> based
>>>>>>>>> on
>>>>>>>>>> > clusterId, if we also had the region and az info on the
>>>>>>> originating
>>>>>>>>>> > message, this we have not implemented but some ideas come
>>> from
>>>>>>>>> network
>>>>>>>>>> > routing, and also the dispatcher router in apache qpid.
>>>>>>>>>> >
>>>>>>>>>> > Also we need to have data perimeters e.g. certain data
>>> cannot
>>>>>>> leave
>>>>>>>>>> > certain countries borders. We want this all automated so
>>> that
>>>>> at
>>>>>>>>> the
>>>>>>>>>> > platform level without having to touch or look at the
>>> business
>>>>>>>>> data inside
>>>>>>>>>> > we can have headers we can put tags into so that we can
>>> ensure
>>>>>>>>> this doesn\u2019t
>>>>>>>>>> > occur when we mirror. (actually links in to data lineage /
>>>>>>> tracing
>>>>>>>>> as again
>>>>>>>>>> > we need to tag messages at a platform level) Examples are we
>>>>> are
>>>>>>>>> not
>>>>>>>>>> > allowed Private customer details to leave Switzerland, yet
>>> we
>>>>>>> need
>>>>>>>>> those
>>>>>>>>>> > systems integrated.
>>>>>>>>>> >
>>>>>>>>>> > Lastly around mirroring we have a partionKey field, as the
>>> key
>>>>>>>>> used for
>>>>>>>>>> > portioning logic != compaction key all the time but we want
>>> to
>>>>>>>>> preserve it
>>>>>>>>>> > for when we mirror so that if source cluster partition count
>>>>> !=
>>>>>>>>> destination
>>>>>>>>>> > cluster partition count we can honour the same partitioning
>>>>>>> logic.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE End 2 End encryption
>>>>>>>>>> >
>>>>>>>>>> > As I believe mentioned just before, the solution you mention
>>>>> just
>>>>>>>>> doesn\u2019t
>>>>>>>>>> > cut the mustard these days with many regulators. An
>>> operations
>>>>>>>>> person with
>>>>>>>>>> > access to the box should not be able to have access to the
>>>>> data.
>>>>>>>>> Many now
>>>>>>>>>> > actually impose quite literally the implementation expected
>>>>> being
>>>>>>>>> end2end
>>>>>>>>>> > encryption for certain data (Singapore for us is one that I
>>> am
>>>>>>>>> most aware
>>>>>>>>>> > of). In fact we\u2019re even now needing encrypt the data and
>>> store
>>>>>>> the
>>>>>>>>> keys in
>>>>>>>>>> > HSM modules.
>>>>>>>>>> >
>>>>>>>>>> > Likewise the performance penalty on encrypting decrypting as
>>>>> you
>>>>>>>>> produce
>>>>>>>>>> > over wire, then again encrypt decrypt as the data is stored
>>> on
>>>>>>> the
>>>>>>>>> brokers
>>>>>>>>>> > disks and back again, then again encrypted and decrypted
>>> back
>>>>>>> over
>>>>>>>>> the wire
>>>>>>>>>> > each time for each consumer all adds up, ignoring this
>>>>> doubling
>>>>>>>>> with mirror
>>>>>>>>>> > makers etc. simply encrypting the value once on write by the
>>>>>>>>> client and
>>>>>>>>>> > again decrypting on consume by the consumer is far more
>>>>>>>>> performant, but
>>>>>>>>>> > then the routing and platform meta data needs to be separate
>>>>>>> (thus
>>>>>>>>> headers)
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE Auditing:
>>>>>>>>>> >
>>>>>>>>>> > Our Auditing needs are:
>>>>>>>>>> > Producer Id,
>>>>>>>>>> > Origin Cluster Id that message first produced into
>>>>>>>>>> > Origin AZ \u2013 agreed we can derive this if we have cluster id,
>>>>> but
>>>>>>>>> it makes
>>>>>>>>>> > resolving this for audit reporting a lot easier.
>>>>>>>>>> > Origin Region \u2013 agreed we can derive this if we have cluster
>>>>> id,
>>>>>>>>> but it
>>>>>>>>>> > makes resolving this for audit reporting a lot easier.
>>>>>>>>>> > Unique Message Identification (this is not the same as
>>>>>>> transaction
>>>>>>>>>> > tracing) \u2013 note offset and partition are not the same, as
>>>>> when we
>>>>>>>>> mirror or
>>>>>>>>>> > have for what ever system failure duplicate send,
>>>>>>>>>> > Custom Client wrapper version (where organizations have to
>>>>> wrap
>>>>>>>>> the kafka
>>>>>>>>>> > client for added features) so we know what version of the
>>>>> wrapper
>>>>>>>>> is used
>>>>>>>>>> > Producer IP address (in case of clients being in our vm/open
>>>>>>> stack
>>>>>>>>> infra
>>>>>>>>>> > where they can move around, producer id will stay the same
>>> but
>>>>>>>>> this would
>>>>>>>>>> > change)
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > RE Once and only once delivery case
>>>>>>>>>> >
>>>>>>>>>> > Using the same Message UUID for auditing we can achieve this
>>>>>>> quite
>>>>>>>>> simply.
>>>>>>>>>> >
>>>>>>>>>> > As per how some other brokers do this (cough qpid, artemis)
>>>>>>>>> message uuid
>>>>>>>>>> > are used to dedupe where message is sent and produced but
>>> the
>>>>>>>>> client didn\u2019t
>>>>>>>>>> > receive the ack, and there for replays the send, by having a
>>>>>>>>> unique message
>>>>>>>>>> > id per message, this can be filtered out, on consumers where
>>>>>>>>> message
>>>>>>>>>> > delivery may occur twice for what ever reasons a message
>>> uuid
>>>>> can
>>>>>>>>> be used
>>>>>>>>>> > to remove duplicates being deliverd , like wise we can do
>>>>> this in
>>>>>>>>> the
>>>>>>>>>> > mirrormakers so if we detect a dupe message we can avoid
>>>>>>>>> replicating it.
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > Cheers
>>>>>>>>>> > Mike
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Since this KIP affects message format, wire protocol,
>>>>> apis, I
>>>>>>>>> think
>>>>>>>>>> > it's
>>>>>>>>>> > worth spending a bit more time to nail down the concrete
>>>>> use
>>>>>>>>> cases. It
>>>>>>>>>> > would be bad if we add this feature, but when start
>>>>>>>>> implementing it
>>>>>>>>>> > for say
>>>>>>>>>> > mirroring, we then realize that header is not the best
>>>>>>>>> approach.
>>>>>>>>>> > Initially,
>>>>>>>>>> > I thought I was convinced of the use cases of headers
>>> and
>>>>> was
>>>>>>>>> trying to
>>>>>>>>>> > write down a few use cases to convince others. That's
>>>>> when I
>>>>>>>>> became
>>>>>>>>>> > less
>>>>>>>>>> > certain. For me to be convinced, I just want to see two
>>>>>>> strong
>>>>>>>>> use
>>>>>>>>>> > cases
>>>>>>>>>> > (instead of 10 maybe use cases) in the third-party
>>> space.
>>>>> The
>>>>>>>>> reason is
>>>>>>>>>> > that when we discussed the use cases within a company,
>>>>> often
>>>>>>>>> it ends
>>>>>>>>>> > with
>>>>>>>>>> > "we can't force everyone to use this standard since we
>>> may
>>>>>>>>> have to
>>>>>>>>>> > integrate with third-party tools".
>>>>>>>>>> >
>>>>>>>>>> > At present, I am not sure why headers are useful for
>>>>> things
>>>>>>>>> like
>>>>>>>>>> > schemaId
>>>>>>>>>> > or encryption. In order to do anything useful to the
>>>>> value,
>>>>>>>>> one needs
>>>>>>>>>> > to
>>>>>>>>>> > know the schemaId or how data is encrypted, but header
>>> is
>>>>>>>>> optional.
>>>>>>>>>> > But, I
>>>>>>>>>> > can be convinced if someone (Radai, Sean, Todd?)
>>> provides
>>>>>>> more
>>>>>>>>> details
>>>>>>>>>> > on
>>>>>>>>>> > the argument.
>>>>>>>>>> >
>>>>>>>>>> > I am not very sure header is the best approach for
>>>>> mirroring
>>>>>>>>> either. If
>>>>>>>>>> > someone has thought about this more, I'd be happy to
>>> hear.
>>>>>>>>>> >
>>>>>>>>>> > I can see the data lineage use case. I am just not sure
>>>>> how
>>>>>>>>> widely
>>>>>>>>>> > applicable this is. If someone familiar with this space
>>>>> can
>>>>>>>>> justify
>>>>>>>>>> > this is
>>>>>>>>>> > a significant use case, say in the finance industry,
>>> this
>>>>>>>>> would be a
>>>>>>>>>> > strong
>>>>>>>>>> > use case.
>>>>>>>>>> >
>>>>>>>>>> > I can see the auditing use case. I am just not sure if a
>>>>>>> native
>>>>>>>>>> > producer id
>>>>>>>>>> > solves that problem. If there are additional metadata
>>>>> that's
>>>>>>>>> worth
>>>>>>>>>> > collecting but not covered by the producer id, that
>>> would
>>>>>>> make
>>>>>>>>> this a
>>>>>>>>>> > strong use case.
>>>>>>>>>> >
>>>>>>>>>> > Thanks,
>>>>>>>>>> >
>>>>>>>>>> > Jun
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>>> > wrote:
>>>>>>>>>> >
>>>>>>>>>> > > this KIP is about enabling headers, nothing more
>>> nothing
>>>>>>>>> less - so
>>>>>>>>>> > no,
>>>>>>>>>> > > broker-side use of headers is not in the KIP scope.
>>>>>>>>>> > >
>>>>>>>>>> > > obviously though, once you have headers potential use
>>>>> cases
>>>>>>>>> could
>>>>>>>>>> > include
>>>>>>>>>> > > broker-side header-aware interceptors (which would be
>>>>> the
>>>>>>>>> topic of
>>>>>>>>>> > other
>>>>>>>>>> > > future KIPs).
>>>>>>>>>> > >
>>>>>>>>>> > > a trivially clear use case (to me) would be using such
>>>>>>>>> broker-side
>>>>>>>>>> > > interceptors to enforce compliance with organizational
>>>>>>>>> policies - it
>>>>>>>>>> > would
>>>>>>>>>> > > make our SREs lives much easier if instead of
>>>>> retroactively
>>>>>>>>>> > discovering
>>>>>>>>>> > > "rogue" topics/users those messages would have been
>>>>>>> rejected
>>>>>>>>>> > up-front.
>>>>>>>>>> > >
>>>>>>>>>> > > the kafka broker code is lacking any such
>>> extensibility
>>>>>>>>> support
>>>>>>>>>> > (beyond
>>>>>>>>>> > > maybe authorizer) which is why these use cases were
>>> left
>>>>>>> out
>>>>>>>>> of the
>>>>>>>>>> > "case
>>>>>>>>>> > > for headers" doc - broker extensibility is a separate
>>>>>>>>> discussion.
>>>>>>>>>> > >
>>>>>>>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
>>>>>>>>> gwen@confluent.io>
>>>>>>>>>> > wrote:
>>>>>>>>>> > >
>>>>>>>>>> > > > Woah, I wasn't aware this is something we'll do. It
>>>>>>> wasn't
>>>>>>>>> in the
>>>>>>>>>> > KIP,
>>>>>>>>>> > > > right?
>>>>>>>>>> > > >
>>>>>>>>>> > > > I guess we could do it the same way ACLs currently
>>>>> work.
>>>>>>>>>> > > > I had in mind something that will allow admins to
>>>>> apply
>>>>>>>>> rules to
>>>>>>>>>> > the
>>>>>>>>>> > > > new create/delete/config topic APIs. So Todd can
>>>>> decide
>>>>>>> to
>>>>>>>>> reject
>>>>>>>>>> > > > "create topic" requests that ask for more than 40
>>>>>>>>> partitions, or
>>>>>>>>>> > > > require exactly 3 replicas, or no more than 50GB
>>>>>>> partition
>>>>>>>>> size,
>>>>>>>>>> > etc.
>>>>>>>>>> > > >
>>>>>>>>>> > > > ACLs were added a bit ad-hoc, if we are planning to
>>>>> apply
>>>>>>>>> more
>>>>>>>>>> > rules
>>>>>>>>>> > > > to requests (and I think we should), we may want a
>>> bit
>>>>>>>>> more generic
>>>>>>>>>> > > > design around that.
>>>>>>>>>> > > >
>>>>>>>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>>> > > wrote:
>>>>>>>>>> > > > > "wouldn't you be in the business of making sure
>>>>>>> everyone
>>>>>>>>> uses
>>>>>>>>>> > them
>>>>>>>>>> > > > > properly?"
>>>>>>>>>> > > > >
>>>>>>>>>> > > > > thats where a broker-side plugin would come handy
>>> -
>>>>> any
>>>>>>>>> incoming
>>>>>>>>>> > > message
>>>>>>>>>> > > > > that does not conform to org policy (read - does
>>> not
>>>>>>>>> have the
>>>>>>>>>> > proper
>>>>>>>>>> > > > > headers) gets thrown out (with an error returned
>>> to
>>>>>>> user)
>>>>>>>>>> > > > >
>>>>>>>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
>>>>>>>>> tpalino@gmail.com>
>>>>>>>>>> > wrote:
>>>>>>>>>> > > > >
>>>>>>>>>> > > > >> Come on, I\u2019ve done at least 2 talks on this one
>>> :)
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> Producing counts to a topic is part of it, but
>>>>> that\u2019s
>>>>>>>>> only
>>>>>>>>>> > part. So
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> count you have 100 messages in topic A. When you
>>>>>>> mirror
>>>>>>>>> topic A
>>>>>>>>>> > to
>>>>>>>>>> > > > another
>>>>>>>>>> > > > >> cluster, you have 99 messages. Where was your
>>>>> problem?
>>>>>>>>> Or
>>>>>>>>>> > worse, you
>>>>>>>>>> > > > have
>>>>>>>>>> > > > >> 100 messages, but one producer duplicated
>>> messages
>>>>> and
>>>>>>>>> another
>>>>>>>>>> > one
>>>>>>>>>> > > lost
>>>>>>>>>> > > > >> messages. You need details about where the
>>> message
>>>>>>> came
>>>>>>>>> from in
>>>>>>>>>> > order
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> pinpoint problems when they happen. Source
>>> producer
>>>>>>>>> info, where
>>>>>>>>>> > it was
>>>>>>>>>> > > > >> produced into your infrastructure, and when it
>>> was
>>>>>>>>> produced.
>>>>>>>>>> > This
>>>>>>>>>> > > > requires
>>>>>>>>>> > > > >> you to add the information to the message.
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> And yes, you still need to maintain your clients.
>>>>> So
>>>>>>>>> maybe my
>>>>>>>>>> > original
>>>>>>>>>> > > > >> example was not the best. My thoughts on not
>>>>> wanting
>>>>>>> to
>>>>>>>>> be
>>>>>>>>>> > responsible
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> message formats stands, because that\u2019s very much
>>>>>>>>> separate from
>>>>>>>>>> > the
>>>>>>>>>> > > > client.
>>>>>>>>>> > > > >> As you know, we have our own internal client
>>>>> library
>>>>>>>>> that can
>>>>>>>>>> > insert
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> right headers, and right now inserts the right
>>>>> audit
>>>>>>>>>> > information into
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> message fields. If they exist, and assuming the
>>>>>>> message
>>>>>>>>> is Avro
>>>>>>>>>> > > encoded.
>>>>>>>>>> > > > >> What if someone wants to use JSON instead for a
>>>>> good
>>>>>>>>> reason?
>>>>>>>>>> > What if
>>>>>>>>>> > > > user X
>>>>>>>>>> > > > >> wants to encrypt messages, but user Y does not?
>>>>>>>>> Maintaining the
>>>>>>>>>> > client
>>>>>>>>>> > > > >> library is still much easier than maintaining the
>>>>>>>>> message
>>>>>>>>>> > formats.
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> -Todd
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
>>>>>>>>> gwen@confluent.io
>>>>>>>>>> > >
>>>>>>>>>> > > wrote:
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> > Based on your last sentence, consider me
>>>>> convinced
>>>>>>> :)
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > I get why headers are critical for Mirroring
>>> (you
>>>>>>>>> need tags to
>>>>>>>>>> > > prevent
>>>>>>>>>> > > > >> > loops and sometimes to route messages to the
>>>>> correct
>>>>>>>>>> > destination).
>>>>>>>>>> > > > >> > But why do you need headers to audit? We are
>>>>>>> auditing
>>>>>>>>> by
>>>>>>>>>> > producing
>>>>>>>>>> > > > >> > counts to a side topic (and I was under the
>>>>>>>>> impression you do
>>>>>>>>>> > the
>>>>>>>>>> > > > >> > same), so we never need to modify the message.
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > Another thing - after we added headers,
>>> wouldn't
>>>>> you
>>>>>>>>> be in the
>>>>>>>>>> > > > >> > business of making sure everyone uses them
>>>>> properly?
>>>>>>>>> Making
>>>>>>>>>> > sure
>>>>>>>>>> > > > >> > everyone includes the right headers you need,
>>> not
>>>>>>>>> using the
>>>>>>>>>> > header
>>>>>>>>>> > > > >> > names you intend to use, etc. I don't think the
>>>>>>>>> "policing"
>>>>>>>>>> > business
>>>>>>>>>> > > > >> > will ever go away.
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <
>>>>>>>>>> > tpalino@gmail.com>
>>>>>>>>>> > > > wrote:
>>>>>>>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
>>> with
>>>>> the
>>>>>>>>>> > workaround.
>>>>>>>>>> > > Avro
>>>>>>>>>> > > > >> means
>>>>>>>>>> > > > >> > > that I have to be concerned with the format
>>> of
>>>>> the
>>>>>>>>> messages
>>>>>>>>>> > in
>>>>>>>>>> > > > order to
>>>>>>>>>> > > > >> > run
>>>>>>>>>> > > > >> > > the infrastructure (audit, mirroring, etc.).
>>>>> That
>>>>>>>>> means
>>>>>>>>>> > that I
>>>>>>>>>> > > have
>>>>>>>>>> > > > to
>>>>>>>>>> > > > >> > > handle the schemas, and I have to enforce
>>> rules
>>>>>>>>> about good
>>>>>>>>>> > > formats.
>>>>>>>>>> > > > >> This
>>>>>>>>>> > > > >> > is
>>>>>>>>>> > > > >> > > not something I want to be in the business
>>> of,
>>>>>>>>> because I
>>>>>>>>>> > should be
>>>>>>>>>> > > > able
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > > run a service infrastructure without needing
>>>>> to be
>>>>>>>>> in the
>>>>>>>>>> > weeds of
>>>>>>>>>> > > > >> > dealing
>>>>>>>>>> > > > >> > > with customer data formats.
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > > Trust me, a sizable portion of my support
>>> time
>>>>> is
>>>>>>>>> spent
>>>>>>>>>> > dealing
>>>>>>>>>> > > with
>>>>>>>>>> > > > >> > schema
>>>>>>>>>> > > > >> > > issues. I really would like to get away from
>>>>> that.
>>>>>>>>> Maybe
>>>>>>>>>> > I'd have
>>>>>>>>>> > > > more
>>>>>>>>>> > > > >> > time
>>>>>>>>>> > > > >> > > for other hobbies. Like writing. ;)
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > > -Todd
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <
>>>>>>>>>> > gwen@confluent.io>
>>>>>>>>>> > > > wrote:
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > >> I'm pretty satisfied with the current
>>>>> workarounds
>>>>>>>>> (Avro
>>>>>>>>>> > container
>>>>>>>>>> > > > >> > >> format), so I'm not too excited about the
>>>>> extra
>>>>>>>>> work
>>>>>>>>>> > required to
>>>>>>>>>> > > do
>>>>>>>>>> > > > >> > >> headers in Kafka. I absolutely don't mind it
>>>>> if
>>>>>>>>> you do
>>>>>>>>>> > it...
>>>>>>>>>> > > > >> > >> I think the Apache convention for "good
>>> idea,
>>>>> but
>>>>>>>>> not
>>>>>>>>>> > willing to
>>>>>>>>>> > > > put
>>>>>>>>>> > > > >> > >> any work toward it" is +0.5? anyway, that's
>>>>> what
>>>>>>> I
>>>>>>>>> was
>>>>>>>>>> > trying to
>>>>>>>>>> > > > >> > >> convey :)
>>>>>>>>>> > > > >> > >>
>>>>>>>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino
>>> <
>>>>>>>>>> > tpalino@gmail.com>
>>>>>>>>>> > > > >> wrote:
>>>>>>>>>> > > > >> > >> > Well I guess my question for you, then, is
>>>>> what
>>>>>>>>> is
>>>>>>>>>> > holding you
>>>>>>>>>> > > > back
>>>>>>>>>> > > > >> > from
>>>>>>>>>> > > > >> > >> > full support for headers? What\u2019s the bit
>>>>> that
>>>>>>>>> you\u2019re
>>>>>>>>>> > missing
>>>>>>>>>> > > that
>>>>>>>>>> > > > >> has
>>>>>>>>>> > > > >> > you
>>>>>>>>>> > > > >> > >> > under a full +1?
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> > -Todd
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
>>>>> Shapira <
>>>>>>>>>> > > gwen@confluent.io>
>>>>>>>>>> > > > >> > wrote:
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> >> I know why people who support headers
>>>>> support
>>>>>>>>> them, and
>>>>>>>>>> > I've
>>>>>>>>>> > > > seen
>>>>>>>>>> > > > >> > what
>>>>>>>>>> > > > >> > >> >> the discussion is like.
>>>>>>>>>> > > > >> > >> >>
>>>>>>>>>> > > > >> > >> >> This is why I'm asking people who are
>>>>> against
>>>>>>>>> headers
>>>>>>>>>> > > > (especially
>>>>>>>>>> > > > >> > >> >> committers) what will make them change
>>>>> their
>>>>>>>>> mind - so
>>>>>>>>>> > we can
>>>>>>>>>> > > > get
>>>>>>>>>> > > > >> > this
>>>>>>>>>> > > > >> > >> >> part over one way or another.
>>>>>>>>>> > > > >> > >> >>
>>>>>>>>>> > > > >> > >> >> If I sound frustrated it is not at Radai,
>>>>> Jun
>>>>>>>>> or you
>>>>>>>>>> > (Todd)...
>>>>>>>>>> > > > I am
>>>>>>>>>> > > > >> > >> >> just looking for something concrete we
>>> can
>>>>> do
>>>>>>>>> to move
>>>>>>>>>> > the
>>>>>>>>>> > > > >> discussion
>>>>>>>>>> > > > >> > >> >> along to the yummy design details (which
>>> is
>>>>>>> the
>>>>>>>>>> > argument I
>>>>>>>>>> > > > really
>>>>>>>>>> > > > >> am
>>>>>>>>>> > > > >> > >> >> looking forward to).
>>>>>>>>>> > > > >> > >> >>
>>>>>>>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
>>>>> Palino <
>>>>>>>>>> > > tpalino@gmail.com>
>>>>>>>>>> > > > >> > wrote:
>>>>>>>>>> > > > >> > >> >> > So, Gwen, to your question (even though
>>>>> I\u2019m
>>>>>>>>> not a
>>>>>>>>>> > > > committer)...
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > I have always been a strong supporter
>>> of
>>>>>>>>> introducing
>>>>>>>>>> > the
>>>>>>>>>> > > > concept
>>>>>>>>>> > > > >> > of an
>>>>>>>>>> > > > >> > >> >> > envelope to messages, which headers
>>>>>>>>> accomplishes. The
>>>>>>>>>> > > message
>>>>>>>>>> > > > key
>>>>>>>>>> > > > >> > is
>>>>>>>>>> > > > >> > >> >> > already an example of a piece of
>>> envelope
>>>>>>>>>> > information. By
>>>>>>>>>> > > > >> > providing a
>>>>>>>>>> > > > >> > >> >> means
>>>>>>>>>> > > > >> > >> >> > to do this within Kafka itself, and not
>>>>>>>>> relying on
>>>>>>>>>> > use-case
>>>>>>>>>> > > > >> > specific
>>>>>>>>>> > > > >> > >> >> > implementations, you make it much
>>> easier
>>>>> for
>>>>>>>>>> > components to
>>>>>>>>>> > > > >> > >> interoperate.
>>>>>>>>>> > > > >> > >> >> It
>>>>>>>>>> > > > >> > >> >> > simplifies development of all these
>>>>> things
>>>>>>>>> (message
>>>>>>>>>> > routing,
>>>>>>>>>> > > > >> > auditing,
>>>>>>>>>> > > > >> > >> >> > encryption, etc.) because each one does
>>>>> not
>>>>>>>>> have to
>>>>>>>>>> > reinvent
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> wheel.
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > It also makes it much easier from a
>>>>> client
>>>>>>>>> point of
>>>>>>>>>> > view if
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> headers
>>>>>>>>>> > > > >> > >> >> are
>>>>>>>>>> > > > >> > >> >> > defined as part of the protocol and/or
>>>>>>>>> message format
>>>>>>>>>> > in
>>>>>>>>>> > > > general
>>>>>>>>>> > > > >> > >> because
>>>>>>>>>> > > > >> > >> >> > you can easily produce and consume
>>>>> messages
>>>>>>>>> without
>>>>>>>>>> > having
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> take
>>>>>>>>>> > > > >> > >> into
>>>>>>>>>> > > > >> > >> >> > account specific cases. For example, I
>>>>> want
>>>>>>>>> to route
>>>>>>>>>> > > messages,
>>>>>>>>>> > > > >> but
>>>>>>>>>> > > > >> > >> >> client A
>>>>>>>>>> > > > >> > >> >> > doesn\u2019t support the way audit
>>> implemented
>>>>>>>>> headers, and
>>>>>>>>>> > > client
>>>>>>>>>> > > > B
>>>>>>>>>> > > > >> > >> doesn\u2019t
>>>>>>>>>> > > > >> > >> >> > support the way encryption or routing
>>>>>>>>> implemented
>>>>>>>>>> > headers,
>>>>>>>>>> > > so
>>>>>>>>>> > > > now
>>>>>>>>>> > > > >> > my
>>>>>>>>>> > > > >> > >> >> > application has to create some really
>>>>>>> fragile
>>>>>>>>> (my
>>>>>>>>>> > > autocorrect
>>>>>>>>>> > > > >> just
>>>>>>>>>> > > > >> > >> tried
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> > make that \u201ctragic\u201d, which is probably
>>>>>>>>> appropriate
>>>>>>>>>> > too) code
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> > strip
>>>>>>>>>> > > > >> > >> >> > everything off, rather than just
>>>>> consuming
>>>>>>> the
>>>>>>>>>> > messages,
>>>>>>>>>> > > > picking
>>>>>>>>>> > > > >> > out
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> 1
>>>>>>>>>> > > > >> > >> >> > or 2 headers it\u2019s interested in, and
>>>>>>>>> performing its
>>>>>>>>>> > > function.
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > Honestly, this discussion has been
>>> going
>>>>> on
>>>>>>>>> for a
>>>>>>>>>> > long time,
>>>>>>>>>> > > > and
>>>>>>>>>> > > > >> > it\u2019s
>>>>>>>>>> > > > >> > >> >> > always \u201cOh, you came up with 2 use
>>> cases,
>>>>>>> and
>>>>>>>>> yeah,
>>>>>>>>>> > those
>>>>>>>>>> > > use
>>>>>>>>>> > > > >> cases
>>>>>>>>>> > > > >> > >> are
>>>>>>>>>> > > > >> > >> >> > real things that someone would want to
>>>>> do.
>>>>>>>>> Here\u2019s an
>>>>>>>>>> > > alternate
>>>>>>>>>> > > > >> way
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > >> >> > implement them so let\u2019s not do
>>> headers.\u201d
>>>>> If
>>>>>>>>> we have a
>>>>>>>>>> > few
>>>>>>>>>> > > use
>>>>>>>>>> > > > >> cases
>>>>>>>>>> > > > >> > >> that
>>>>>>>>>> > > > >> > >> >> we
>>>>>>>>>> > > > >> > >> >> > actually came up with, you can be sure
>>>>> that
>>>>>>>>> over the
>>>>>>>>>> > next
>>>>>>>>>> > > year
>>>>>>>>>> > > > >> > >> there\u2019s a
>>>>>>>>>> > > > >> > >> >> > dozen others that we didn\u2019t think of
>>> that
>>>>>>>>> someone
>>>>>>>>>> > would like
>>>>>>>>>> > > > to
>>>>>>>>>> > > > >> > do. I
>>>>>>>>>> > > > >> > >> >> > really think it\u2019s time to stop
>>> rehashing
>>>>>>> this
>>>>>>>>>> > discussion and
>>>>>>>>>> > > > >> > instead
>>>>>>>>>> > > > >> > >> >> focus
>>>>>>>>>> > > > >> > >> >> > on a workable standard that we can
>>> adopt.
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > -Todd
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
>>>>> Palino
>>>>>>> <
>>>>>>>>>> > > > tpalino@gmail.com>
>>>>>>>>>> > > > >> > >> wrote:
>>>>>>>>>> > > > >> > >> >> >
>>>>>>>>>> > > > >> > >> >> >> C. per message encryption
>>>>>>>>>> > > > >> > >> >> >>> One drawback of this approach is that
>>>>> this
>>>>>>>>>> > significantly
>>>>>>>>>> > > > reduce
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> effectiveness of compression, which
>>>>>>> happens
>>>>>>>>> on a
>>>>>>>>>> > set of
>>>>>>>>>> > > > >> > serialized
>>>>>>>>>> > > > >> > >> >> >>> messages. An alternative is to enable
>>>>> SSL
>>>>>>>>> for wire
>>>>>>>>>> > > > encryption
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> rely
>>>>>>>>>> > > > >> > >> >> on
>>>>>>>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS) for at
>>>>> rest
>>>>>>>>>> > encryption.
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
>>> this
>>>>>>> does
>>>>>>>>> cover
>>>>>>>>>> > the case
>>>>>>>>>> > > > of
>>>>>>>>>> > > > >> > >> removing
>>>>>>>>>> > > > >> > >> >> a
>>>>>>>>>> > > > >> > >> >> >> drive from the system, it will not
>>>>> satisfy
>>>>>>>>> most
>>>>>>>>>> > compliance
>>>>>>>>>> > > > >> > >> requirements
>>>>>>>>>> > > > >> > >> >> for
>>>>>>>>>> > > > >> > >> >> >> encryption of data as whoever has
>>>>> access to
>>>>>>>>> the
>>>>>>>>>> > broker
>>>>>>>>>> > > itself
>>>>>>>>>> > > > >> > still
>>>>>>>>>> > > > >> > >> has
>>>>>>>>>> > > > >> > >> >> >> access to the unencrypted data. For
>>>>>>>>> end-to-end
>>>>>>>>>> > encryption
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> > need to
>>>>>>>>>> > > > >> > >> >> >> encrypt at the producer, before it
>>>>> enters
>>>>>>> the
>>>>>>>>>> > system, and
>>>>>>>>>> > > > >> decrypt
>>>>>>>>>> > > > >> > at
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> >> consumer, after it exits the system.
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >> -Todd
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai
>>> <
>>>>>>>>>> > > > >> radai.rosenblatt@gmail.com
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > >> >> wrote:
>>>>>>>>>> > > > >> > >> >> >>
>>>>>>>>>> > > > >> > >> >> >>> another big plus of headers in the
>>>>>>> protocol
>>>>>>>>> is that
>>>>>>>>>> > it
>>>>>>>>>> > > would
>>>>>>>>>> > > > >> > enable
>>>>>>>>>> > > > >> > >> >> rapid
>>>>>>>>>> > > > >> > >> >> >>> iteration on ideas outside of core
>>>>> kafka
>>>>>>>>> and would
>>>>>>>>>> > reduce
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> number of
>>>>>>>>>> > > > >> > >> >> >>> future wire format changes required.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
>>>>>>> represents
>>>>>>>>> use
>>>>>>>>>> > cases that
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> > not
>>>>>>>>>> > > > >> > >> 100%
>>>>>>>>>> > > > >> > >> >> >>> relevant to all users, and some of
>>> them
>>>>>>>>> require
>>>>>>>>>> > rather
>>>>>>>>>> > > > invasive
>>>>>>>>>> > > > >> > wire
>>>>>>>>>> > > > >> > >> >> >>> protocol changes. a thing a good
>>> recent
>>>>>>>>> example of
>>>>>>>>>> > this is
>>>>>>>>>> > > > >> > kip-98.
>>>>>>>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected to
>>> be
>>>>> a
>>>>>>>>> very small
>>>>>>>>>> > > > fraction of
>>>>>>>>>> > > > >> > >> total
>>>>>>>>>> > > > >> > >> >> >>> traffic and yet the changes are
>>>>> invasive.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> every such wire format change
>>>>> translates
>>>>>>>>> into
>>>>>>>>>> > painful and
>>>>>>>>>> > > > slow
>>>>>>>>>> > > > >> > >> >> adoption of
>>>>>>>>>> > > > >> > >> >> >>> new versions.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> i think a lot of functionality
>>>>> currently
>>>>>>> in
>>>>>>>>> KIPs
>>>>>>>>>> > could be
>>>>>>>>>> > > > "spun
>>>>>>>>>> > > > >> > out"
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> implemented as opt-in plugins
>>>>> transmitting
>>>>>>>>> data over
>>>>>>>>>> > > > headers.
>>>>>>>>>> > > > >> > this
>>>>>>>>>> > > > >> > >> >> would
>>>>>>>>>> > > > >> > >> >> >>> keep the core wire format stable(r),
>>>>> core
>>>>>>>>> codebase
>>>>>>>>>> > > smaller,
>>>>>>>>>> > > > and
>>>>>>>>>> > > > >> > >> avoid
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
>>>>> required
>>>>>>>>> to prove
>>>>>>>>>> > a
>>>>>>>>>> > > > certain
>>>>>>>>>> > > > >> > >> feature
>>>>>>>>>> > > > >> > >> >> is
>>>>>>>>>> > > > >> > >> >> >>> useful enough for a wide-enough
>>>>> audience
>>>>>>> to
>>>>>>>>> warrant
>>>>>>>>>> > a wire
>>>>>>>>>> > > > >> format
>>>>>>>>>> > > > >> > >> >> change
>>>>>>>>>> > > > >> > >> >> >>> and code complexity additions.
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
>>>>> "mere"
>>>>>>>>> wire format
>>>>>>>>>> > > changes
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > im
>>>>>>>>>> > > > >> > >> not
>>>>>>>>>> > > > >> > >> >> >>> saying it could have been completely
>>>>> done
>>>>>>>>> with
>>>>>>>>>> > headers,
>>>>>>>>>> > > but
>>>>>>>>>> > > > >> > >> >> exactly-once
>>>>>>>>>> > > > >> > >> >> >>> delivery certainly could)
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen
>>>>>>>>> Shapira <
>>>>>>>>>> > > > >> gwen@confluent.io
>>>>>>>>>> > > > >> > >
>>>>>>>>>> > > > >> > >> >> wrote:
>>>>>>>>>> > > > >> > >> >> >>>
>>>>>>>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
>>>>> radai <
>>>>>>>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > > "For use cases within an
>>>>> organization,
>>>>>>>>> one could
>>>>>>>>>> > > always
>>>>>>>>>> > > > use
>>>>>>>>>> > > > >> > >> other
>>>>>>>>>> > > > >> > >> >> >>> > > approaches such as company-wise
>>>>>>>>> containers"
>>>>>>>>>> > > > >> > >> >> >>> > > this is what linkedin has
>>>>>>> traditionally
>>>>>>>>> done
>>>>>>>>>> > but there
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> > now
>>>>>>>>>> > > > >> > >> >> cases
>>>>>>>>>> > > > >> > >> >> >>> > (read
>>>>>>>>>> > > > >> > >> >> >>> > > - topics) where this is not
>>>>>>> acceptable.
>>>>>>>>> this
>>>>>>>>>> > makes
>>>>>>>>>> > > > headers
>>>>>>>>>> > > > >> > >> useful
>>>>>>>>>> > > > >> > >> >> even
>>>>>>>>>> > > > >> > >> >> >>> > > within single orgs for cases
>>> where
>>>>>>>>>> > > > one-container-fits-all
>>>>>>>>>> > > > >> > cannot
>>>>>>>>>> > > > >> > >> >> >>> apply.
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > > as for the particular use cases
>>>>>>> listed,
>>>>>>>>> i dont
>>>>>>>>>> > want
>>>>>>>>>> > > > this to
>>>>>>>>>> > > > >> > >> devolve
>>>>>>>>>> > > > >> > >> >> >>> to a
>>>>>>>>>> > > > >> > >> >> >>> > > discussion of particular use
>>> cases
>>>>> - i
>>>>>>>>> think its
>>>>>>>>>> > > enough
>>>>>>>>>> > > > >> that
>>>>>>>>>> > > > >> > >> some
>>>>>>>>>> > > > >> > >> >> of
>>>>>>>>>> > > > >> > >> >> >>> them
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I think a main point of contention
>>> is
>>>>>>>>> that: We
>>>>>>>>>> > > identified
>>>>>>>>>> > > > few
>>>>>>>>>> > > > >> > >> >> >>> > use-cases where headers are useful,
>>>>> do
>>>>>>> we
>>>>>>>>> want
>>>>>>>>>> > Kafka to
>>>>>>>>>> > > > be a
>>>>>>>>>> > > > >> > >> system
>>>>>>>>>> > > > >> > >> >> >>> > that supports those use-cases?
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > For example, Jun said:
>>>>>>>>>> > > > >> > >> >> >>> > "Not sure how widely useful
>>>>> record-level
>>>>>>>>> lineage
>>>>>>>>>> > is
>>>>>>>>>> > > though
>>>>>>>>>> > > > >> > since
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > overhead could
>>>>>>>>>> > > > >> > >> >> >>> > be significant."
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > We know NiFi supports record level
>>>>>>>>> lineage. I
>>>>>>>>>> > don't
>>>>>>>>>> > > think
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > was
>>>>>>>>>> > > > >> > >> >> >>> > developed for lols, I think it is
>>>>> safe
>>>>>>> to
>>>>>>>>> assume
>>>>>>>>>> > that
>>>>>>>>>> > > the
>>>>>>>>>> > > > NSA
>>>>>>>>>> > > > >> > >> needed
>>>>>>>>>> > > > >> > >> >> >>> > that functionality. We also know
>>> that
>>>>>>>>> certain
>>>>>>>>>> > financial
>>>>>>>>>> > > > >> > institutes
>>>>>>>>>> > > > >> > >> >> >>> > need to track tampering with
>>> records
>>>>> at
>>>>>>> a
>>>>>>>>> record
>>>>>>>>>> > level
>>>>>>>>>> > > and
>>>>>>>>>> > > > >> > there
>>>>>>>>>> > > > >> > >> are
>>>>>>>>>> > > > >> > >> >> >>> > federal regulations that absolutely
>>>>>>>>> require
>>>>>>>>>> > this. They
>>>>>>>>>> > > > also
>>>>>>>>>> > > > >> > need
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > prove that routing apps that
>>>>> "touches"
>>>>>>> the
>>>>>>>>>> > messages and
>>>>>>>>>> > > > >> either
>>>>>>>>>> > > > >> > >> reads
>>>>>>>>>> > > > >> > >> >> >>> > or updates headers couldn't have
>>>>>>> possibly
>>>>>>>>>> > modified the
>>>>>>>>>> > > > >> payload
>>>>>>>>>> > > > >> > >> >> itself.
>>>>>>>>>> > > > >> > >> >> >>> > They use record level encryption to
>>>>> do
>>>>>>>>> that -
>>>>>>>>>> > apps can
>>>>>>>>>> > > > read
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
>>> can't
>>>>>>>>> touch the
>>>>>>>>>> > payload.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > We can totally say "those are
>>> corner
>>>>>>>>> cases and
>>>>>>>>>> > not worth
>>>>>>>>>> > > > >> adding
>>>>>>>>>> > > > >> > >> >> >>> > headers to Kafka for", they should
>>>>> use a
>>>>>>>>> different
>>>>>>>>>> > > pubsub
>>>>>>>>>> > > > >> > message
>>>>>>>>>> > > > >> > >> for
>>>>>>>>>> > > > >> > >> >> >>> > that (Nifi or one of the other 1000
>>>>> that
>>>>>>>>> cater
>>>>>>>>>> > > > specifically
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > financial industry).
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > But this gets us into a catch 22:
>>>>>>>>>> > > > >> > >> >> >>> > If we discuss a specific use-case,
>>>>>>>>> someone can
>>>>>>>>>> > always
>>>>>>>>>> > > say
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > isn't
>>>>>>>>>> > > > >> > >> >> >>> > interesting enough for Kafka. If we
>>>>>>>>> discuss more
>>>>>>>>>> > general
>>>>>>>>>> > > > >> > trends,
>>>>>>>>>> > > > >> > >> >> >>> > others can say "well, we are not
>>> sure
>>>>>>> any
>>>>>>>>> of them
>>>>>>>>>> > really
>>>>>>>>>> > > > >> needs
>>>>>>>>>> > > > >> > >> >> headers
>>>>>>>>>> > > > >> > >> >> >>> > specifically. This is just hand
>>>>> waving
>>>>>>>>> and not
>>>>>>>>>> > > > interesting.".
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I think discussing use-cases in
>>>>>>> specifics
>>>>>>>>> is super
>>>>>>>>>> > > > important
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> >> decide
>>>>>>>>>> > > > >> > >> >> >>> > implementation details for headers
>>>>> (my
>>>>>>>>> use-cases
>>>>>>>>>> > lean
>>>>>>>>>> > > > toward
>>>>>>>>>> > > > >> > >> >> numerical
>>>>>>>>>> > > > >> > >> >> >>> > keys with namespaces and object
>>>>> values,
>>>>>>>>> others
>>>>>>>>>> > differ),
>>>>>>>>>> > > > but I
>>>>>>>>>> > > > >> > >> think
>>>>>>>>>> > > > >> > >> >> we
>>>>>>>>>> > > > >> > >> >> >>> > need to answer the general "Are we
>>>>> going
>>>>>>>>> to have
>>>>>>>>>> > > headers"
>>>>>>>>>> > > > >> > question
>>>>>>>>>> > > > >> > >> >> >>> > first.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I'd love to hear from the other
>>>>>>>>> committers in the
>>>>>>>>>> > > > discussion:
>>>>>>>>>> > > > >> > >> >> >>> > What would it take to convince you
>>>>> that
>>>>>>>>> headers
>>>>>>>>>> > in Kafka
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> a
>>>>>>>>>> > > > >> > >> good
>>>>>>>>>> > > > >> > >> >> >>> > idea in general, so we can move
>>> ahead
>>>>>>> and
>>>>>>>>> try to
>>>>>>>>>> > agree
>>>>>>>>>> > > on
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> >> details?
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > I feel like we keep moving the goal
>>>>>>> posts
>>>>>>>>> and
>>>>>>>>>> > this is
>>>>>>>>>> > > > truly
>>>>>>>>>> > > > >> > >> >> exhausting.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > For the record, I mildly support
>>>>> adding
>>>>>>>>> headers
>>>>>>>>>> > to Kafka
>>>>>>>>>> > > > >> > (+0.5?).
>>>>>>>>>> > > > >> > >> >> >>> > The community can continue to find
>>>>>>>>> workarounds to
>>>>>>>>>> > the
>>>>>>>>>> > > > issue
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> there
>>>>>>>>>> > > > >> > >> >> >>> > are some benefits to keeping the
>>>>> message
>>>>>>>>> format
>>>>>>>>>> > and
>>>>>>>>>> > > > clients
>>>>>>>>>> > > > >> > >> simpler.
>>>>>>>>>> > > > >> > >> >> >>> > But I see the usefulness of headers
>>>>> to
>>>>>>>>> many
>>>>>>>>>> > use-cases
>>>>>>>>>> > > and
>>>>>>>>>> > > > if
>>>>>>>>>> > > > >> we
>>>>>>>>>> > > > >> > >> can
>>>>>>>>>> > > > >> > >> >> >>> > find a good and generally useful
>>> way
>>>>> to
>>>>>>>>> add it to
>>>>>>>>>> > Kafka,
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > will
>>>>>>>>>> > > > >> > >> make
>>>>>>>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
>>> worthy
>>>>>>>>> goal in my
>>>>>>>>>> > eyes.
>>>>>>>>>> > > > >> > >> >> >>> >
>>>>>>>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
>>>>>>>>>> > > > >> > >> >> >>> > > A+B. i think there are use cases
>>>>> for
>>>>>>>>> polyglot
>>>>>>>>>> > topics.
>>>>>>>>>> > > > >> > >> especially if
>>>>>>>>>> > > > >> > >> >> >>> kafka
>>>>>>>>>> > > > >> > >> >> >>> > > is being used to "trunk"
>>> something
>>>>>>> else.
>>>>>>>>>> > > > >> > >> >> >>> > > D. multiple topics would make it
>>>>>>> harder
>>>>>>>>> to write
>>>>>>>>>> > > > portable
>>>>>>>>>> > > > >> > >> consumer
>>>>>>>>>> > > > >> > >> >> >>> code.
>>>>>>>>>> > > > >> > >> >> >>> > > partition remapping would mess
>>> with
>>>>>>>>> locality of
>>>>>>>>>> > > > consumption
>>>>>>>>>> > > > >> > >> >> >>> guarantees.
>>>>>>>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
>>>>>>>>> lineage/metadata is
>>>>>>>>>> > > > >> > >> billing/chargeback.
>>>>>>>>>> > > > >> > >> >> for
>>>>>>>>>> > > > >> > >> >> >>> > that
>>>>>>>>>> > > > >> > >> >> >>> > > use case it is not enough to
>>> simply
>>>>>>>>> record the
>>>>>>>>>> > point
>>>>>>>>>> > > of
>>>>>>>>>> > > > >> > origin,
>>>>>>>>>> > > > >> > >> but
>>>>>>>>>> > > > >> > >> >> >>> every
>>>>>>>>>> > > > >> > >> >> >>> > > replication stop (think mirror
>>>>> maker)
>>>>>>>>> must also
>>>>>>>>>> > add a
>>>>>>>>>> > > > >> record
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > >> >> form a
>>>>>>>>>> > > > >> > >> >> >>> > > "transit log".
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > > as for stream processing on top
>>> of
>>>>>>>>> kafka - i
>>>>>>>>>> > know
>>>>>>>>>> > > samza
>>>>>>>>>> > > > >> has a
>>>>>>>>>> > > > >> > >> >> metadata
>>>>>>>>>> > > > >> > >> >> >>> > map
>>>>>>>>>> > > > >> > >> >> >>> > > which they carry around in
>>>>> addition to
>>>>>>>>> user
>>>>>>>>>> > values.
>>>>>>>>>> > > > headers
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > perfect
>>>>>>>>>> > > > >> > >> >> >>> > > fit for these things.
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM,
>>>>> Jun
>>>>>>>>> Rao <
>>>>>>>>>> > > > jun@confluent.io
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> In order to answer the first two
>>>>>>>>> questions, it
>>>>>>>>>> > would
>>>>>>>>>> > > be
>>>>>>>>>> > > > >> > helpful
>>>>>>>>>> > > > >> > >> >> if we
>>>>>>>>>> > > > >> > >> >> >>> > could
>>>>>>>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use cases
>>>>> for
>>>>>>>>> headers
>>>>>>>>>> > in the
>>>>>>>>>> > > > space
>>>>>>>>>> > > > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > third-party
>>>>>>>>>> > > > >> > >> >> >>> > >> vendors. For use cases within an
>>>>>>>>> organization,
>>>>>>>>>> > one
>>>>>>>>>> > > > could
>>>>>>>>>> > > > >> > always
>>>>>>>>>> > > > >> > >> >> use
>>>>>>>>>> > > > >> > >> >> >>> > other
>>>>>>>>>> > > > >> > >> >> >>> > >> approaches such as company-wise
>>>>>>>>> containers to
>>>>>>>>>> > get
>>>>>>>>>> > > > around
>>>>>>>>>> > > > >> w/o
>>>>>>>>>> > > > >> > >> >> >>> headers. I
>>>>>>>>>> > > > >> > >> >> >>> > >> went through the use cases in
>>> the
>>>>> KIP
>>>>>>>>> and in
>>>>>>>>>> > Radai's
>>>>>>>>>> > > > wiki
>>>>>>>>>> > > > >> (
>>>>>>>>>> > > > >> > >> >> >>> > >> https://cwiki.apache.org/confl
>>>>>>>>>> > uence/display/KAFKA/A+
>>>>>>>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> ).
>>>>>>>>>> > > > >> > >> >> >>> > >> The following are the ones that
>>>>> that
>>>>>>> I
>>>>>>>>>> > understand and
>>>>>>>>>> > > > >> could
>>>>>>>>>> > > > >> > be
>>>>>>>>>> > > > >> > >> in
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> third-party use case category.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> A. content-type
>>>>>>>>>> > > > >> > >> >> >>> > >> It seems that in general,
>>>>>>> content-type
>>>>>>>>> should
>>>>>>>>>> > be set
>>>>>>>>>> > > at
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> topic
>>>>>>>>>> > > > >> > >> >> >>> level.
>>>>>>>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages with
>>>>>>>>> different
>>>>>>>>>> > content
>>>>>>>>>> > > > types
>>>>>>>>>> > > > >> > >> should be
>>>>>>>>>> > > > >> > >> >> >>> > >> encouraged.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> B. schema id
>>>>>>>>>> > > > >> > >> >> >>> > >> Since the value is mostly
>>> useless
>>>>>>>>> without
>>>>>>>>>> > schema id,
>>>>>>>>>> > > it
>>>>>>>>>> > > > >> > seems
>>>>>>>>>> > > > >> > >> that
>>>>>>>>>> > > > >> > >> >> >>> > storing
>>>>>>>>>> > > > >> > >> >> >>> > >> the schema id together with
>>>>>>> serialized
>>>>>>>>> bytes
>>>>>>>>>> > in the
>>>>>>>>>> > > > value
>>>>>>>>>> > > > >> is
>>>>>>>>>> > > > >> > >> >> better?
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> C. per message encryption
>>>>>>>>>> > > > >> > >> >> >>> > >> One drawback of this approach is
>>>>> that
>>>>>>>>> this
>>>>>>>>>> > > > significantly
>>>>>>>>>> > > > >> > reduce
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
>>>>> which
>>>>>>>>> happens on
>>>>>>>>>> > a set
>>>>>>>>>> > > of
>>>>>>>>>> > > > >> > >> serialized
>>>>>>>>>> > > > >> > >> >> >>> > >> messages. An alternative is to
>>>>> enable
>>>>>>>>> SSL for
>>>>>>>>>> > wire
>>>>>>>>>> > > > >> > encryption
>>>>>>>>>> > > > >> > >> and
>>>>>>>>>> > > > >> > >> >> >>> rely
>>>>>>>>>> > > > >> > >> >> >>> > on
>>>>>>>>>> > > > >> > >> >> >>> > >> the storage system (e.g. LUKS)
>>>>> for at
>>>>>>>>> rest
>>>>>>>>>> > > encryption.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
>>> across
>>>>>>>>> Kafka
>>>>>>>>>> > clusters
>>>>>>>>>> > > > >> > >> >> >>> > >> This is actually interesting.
>>>>> Today,
>>>>>>>>> to avoid
>>>>>>>>>> > > > introducing
>>>>>>>>>> > > > >> > >> cycles
>>>>>>>>>> > > > >> > >> >> when
>>>>>>>>>> > > > >> > >> >> >>> > doing
>>>>>>>>>> > > > >> > >> >> >>> > >> mirroring across data centers,
>>> one
>>>>>>>>> would
>>>>>>>>>> > either have
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> set
>>>>>>>>>> > > > >> > up
>>>>>>>>>> > > > >> > >> two
>>>>>>>>>> > > > >> > >> >> >>> Kafka
>>>>>>>>>> > > > >> > >> >> >>> > >> clusters (a local and an
>>>>> aggregate)
>>>>>>>>> per data
>>>>>>>>>> > center
>>>>>>>>>> > > or
>>>>>>>>>> > > > >> > rename
>>>>>>>>>> > > > >> > >> >> topics.
>>>>>>>>>> > > > >> > >> >> >>> > >> Neither is ideal. With headers,
>>>>> the
>>>>>>>>> producer
>>>>>>>>>> > could
>>>>>>>>>> > > tag
>>>>>>>>>> > > > >> each
>>>>>>>>>> > > > >> > >> >> message
>>>>>>>>>> > > > >> > >> >> >>> with
>>>>>>>>>> > > > >> > >> >> >>> > >> the producing cluster ID in the
>>>>>>> header.
>>>>>>>>>> > MirrorMaker
>>>>>>>>>> > > > could
>>>>>>>>>> > > > >> > then
>>>>>>>>>> > > > >> > >> >> avoid
>>>>>>>>>> > > > >> > >> >> >>> > >> mirroring messages to a cluster
>>> if
>>>>>>>>> they are
>>>>>>>>>> > tagged
>>>>>>>>>> > > with
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> same
>>>>>>>>>> > > > >> > >> >> >>> cluster
>>>>>>>>>> > > > >> > >> >> >>> > >> id.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> However, an alternative approach
>>>>> is
>>>>>>> to
>>>>>>>>>> > introduce sth
>>>>>>>>>> > > > like
>>>>>>>>>> > > > >> > >> >> >>> hierarchical
>>>>>>>>>> > > > >> > >> >> >>> > >> topic and store messages from
>>>>>>> different
>>>>>>>>>> > clusters in
>>>>>>>>>> > > > >> > different
>>>>>>>>>> > > > >> > >> >> >>> partitions
>>>>>>>>>> > > > >> > >> >> >>> > >> under the same topic. This
>>>>> approach
>>>>>>>>> avoids
>>>>>>>>>> > filtering
>>>>>>>>>> > > > out
>>>>>>>>>> > > > >> > >> unneeded
>>>>>>>>>> > > > >> > >> >> >>> data
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> makes offset preserving easier
>>> to
>>>>>>>>> support. It
>>>>>>>>>> > may
>>>>>>>>>> > > make
>>>>>>>>>> > > > >> > >> compaction
>>>>>>>>>> > > > >> > >> >> >>> > trickier
>>>>>>>>>> > > > >> > >> >> >>> > >> though since the same key may
>>>>> show up
>>>>>>>>> in
>>>>>>>>>> > different
>>>>>>>>>> > > > >> > partitions.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> E. record-level lineage
>>>>>>>>>> > > > >> > >> >> >>> > >> For example, a source connector
>>>>> could
>>>>>>>>> store in
>>>>>>>>>> > the
>>>>>>>>>> > > > message
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> metadata
>>>>>>>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
>>> record.
>>>>>>>>> Similarly,
>>>>>>>>>> > if a
>>>>>>>>>> > > > stream
>>>>>>>>>> > > > >> job
>>>>>>>>>> > > > >> > >> >> >>> transforms
>>>>>>>>>> > > > >> > >> >> >>> > >> messages from topic A to topic
>>> B,
>>>>> the
>>>>>>>>> library
>>>>>>>>>> > could
>>>>>>>>>> > > > >> include
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> source
>>>>>>>>>> > > > >> > >> >> >>> > >> message offset in each of the
>>>>>>>>> transformed
>>>>>>>>>> > message in
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> header.
>>>>>>>>>> > > > >> > >> >> Not
>>>>>>>>>> > > > >> > >> >> >>> > sure
>>>>>>>>>> > > > >> > >> >> >>> > >> how widely useful record-level
>>>>>>> lineage
>>>>>>>>> is
>>>>>>>>>> > though
>>>>>>>>>> > > since
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> >> overhead
>>>>>>>>>> > > > >> > >> >> >>> > could
>>>>>>>>>> > > > >> > >> >> >>> > >> be significant.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>>>>>>>>>> > > > >> > >> >> >>> > >> We could put things like
>>>>>>>>> clientId/host/user in
>>>>>>>>>> > the
>>>>>>>>>> > > > header
>>>>>>>>>> > > > >> in
>>>>>>>>>> > > > >> > >> each
>>>>>>>>>> > > > >> > >> >> >>> > message
>>>>>>>>>> > > > >> > >> >> >>> > >> for auditing. These metadata are
>>>>>>>>> really at the
>>>>>>>>>> > > producer
>>>>>>>>>> > > > >> > level
>>>>>>>>>> > > > >> > >> >> though.
>>>>>>>>>> > > > >> > >> >> >>> > So, a
>>>>>>>>>> > > > >> > >> >> >>> > >> more efficient way is to only
>>>>>>> include a
>>>>>>>>>> > "producerId"
>>>>>>>>>> > > > per
>>>>>>>>>> > > > >> > >> message
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> > send
>>>>>>>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
>>> mapping
>>>>>>>>>> > independently.
>>>>>>>>>> > > > KIP-98
>>>>>>>>>> > > > >> is
>>>>>>>>>> > > > >> > >> >> actually
>>>>>>>>>> > > > >> > >> >> >>> > >> proposing including such a
>>>>> producerId
>>>>>>>>> natively
>>>>>>>>>> > in the
>>>>>>>>>> > > > >> > message.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that I
>>> am
>>>>>>> fully
>>>>>>>>>> > convinced of
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > strong
>>>>>>>>>> > > > >> > >> >> >>> > third-party
>>>>>>>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
>>> Perhaps
>>>>> we
>>>>>>>>> could
>>>>>>>>>> > discuss a
>>>>>>>>>> > > > bit
>>>>>>>>>> > > > >> > more
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> make
>>>>>>>>>> > > > >> > >> >> >>> > one
>>>>>>>>>> > > > >> > >> >> >>> > >> or two really convincing use
>>>>> cases.
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> Another orthogonal question is
>>>>>>>>> whether header
>>>>>>>>>> > should
>>>>>>>>>> > > > be
>>>>>>>>>> > > > >> > >> exposed
>>>>>>>>>> > > > >> > >> >> in
>>>>>>>>>> > > > >> > >> >> >>> > stream
>>>>>>>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
>>>>> stream,
>>>>>>>>> Samza,
>>>>>>>>>> > and
>>>>>>>>>> > > Spark
>>>>>>>>>> > > > >> > >> streaming.
>>>>>>>>>> > > > >> > >> >> >>> > >> Currently, those systems just
>>> deal
>>>>>>> with
>>>>>>>>>> > key/value
>>>>>>>>>> > > > pairs.
>>>>>>>>>> > > > >> > >> Should we
>>>>>>>>>> > > > >> > >> >> >>> > expose a
>>>>>>>>>> > > > >> > >> >> >>> > >> third thing header there too or
>>>>>>>>> somehow map
>>>>>>>>>> > header to
>>>>>>>>>> > > > key
>>>>>>>>>> > > > >> or
>>>>>>>>>> > > > >> > >> >> value?
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> Thanks,
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> Jun
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM,
>>>>>>>>> Michael
>>>>>>>>>> > Pearce <
>>>>>>>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > I assume, that after a period
>>>>> of a
>>>>>>>>> week,
>>>>>>>>>> > that there
>>>>>>>>>> > > > is
>>>>>>>>>> > > > >> no
>>>>>>>>>> > > > >> > >> >> concerns
>>>>>>>>>> > > > >> > >> >> >>> now
>>>>>>>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and now
>>> we
>>>>>>> have
>>>>>>>>>> > agreement that
>>>>>>>>>> > > > >> > headers
>>>>>>>>>> > > > >> > >> are
>>>>>>>>>> > > > >> > >> >> >>> useful
>>>>>>>>>> > > > >> > >> >> >>> > >> and
>>>>>>>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
>>> put
>>>>> to
>>>>>>> a
>>>>>>>>> KIP
>>>>>>>>>> > vote, this
>>>>>>>>>> > > > >> > wouldn\u2019t
>>>>>>>>>> > > > >> > >> be
>>>>>>>>>> > > > >> > >> >> a
>>>>>>>>>> > > > >> > >> >> >>> > reason
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > reject.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > @
>>>>>>>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
>>>>>>>>>> > > > >> > >> >> >>> > >> > I think for purpose of getting
>>>>> this
>>>>>>>>> KIP
>>>>>>>>>> > moving past
>>>>>>>>>> > > > >> this,
>>>>>>>>>> > > > >> > we
>>>>>>>>>> > > > >> > >> can
>>>>>>>>>> > > > >> > >> >> >>> state
>>>>>>>>>> > > > >> > >> >> >>> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
>>> that
>>>>>>> can
>>>>>>>>> will be
>>>>>>>>>> > > > naturally
>>>>>>>>>> > > > >> > >> >> interpreted
>>>>>>>>>> > > > >> > >> >> >>> as
>>>>>>>>>> > > > >> > >> >> >>> > an
>>>>>>>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is later
>>>>>>>>> wanted you can
>>>>>>>>>> > > easily
>>>>>>>>>> > > > >> split
>>>>>>>>>> > > > >> > >> this
>>>>>>>>>> > > > >> > >> >> >>> into
>>>>>>>>>> > > > >> > >> >> >>> > two
>>>>>>>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the wire
>>>>>>> protocol
>>>>>>>>>> > > implementation
>>>>>>>>>> > > > >> this
>>>>>>>>>> > > > >> > >> makes
>>>>>>>>>> > > > >> > >> >> no
>>>>>>>>>> > > > >> > >> >> >>> > >> > difference I don\u2019t believe. Is
>>>>> this
>>>>>>>>>> > reasonable to
>>>>>>>>>> > > > all?
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4 therefor
>>>>> happy
>>>>>>>>> we keep
>>>>>>>>>> > with 32
>>>>>>>>>> > > > >> bits.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
>>>>>>>>>> > ignacio.solis@gmail.com on
>>>>>>>>>> > > > >> behalf
>>>>>>>>>> > > > >> > of
>>>>>>>>>> > > > >> > >> >> >>> Ignacio
>>>>>>>>>> > > > >> > >> >> >>> > >> > Solis" <
>>> ignacio.solis@gmail.com
>>>>> on
>>>>>>>>> behalf of
>>>>>>>>>> > > > >> > isolis@igso.net
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> > >> >> >>> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > Summary:
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value as
>>>>> byte[]
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
>>>>>>>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>>>>>>>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
>>> maybe
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
>>>>> system
>>>>>>>>> should
>>>>>>>>>> > take a
>>>>>>>>>> > > > single
>>>>>>>>>> > > > >> > >> int. I
>>>>>>>>>> > > > >> > >> >> >>> think
>>>>>>>>>> > > > >> > >> >> >>> > >> > 32bits is
>>>>>>>>>> > > > >> > >> >> >>> > >> > a good size, if you want
>>> to
>>>>>>>>> interpret
>>>>>>>>>> > this as
>>>>>>>>>> > > to
>>>>>>>>>> > > > >> 16bit
>>>>>>>>>> > > > >> > >> >> numbers
>>>>>>>>>> > > > >> > >> >> >>> in
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > layer
>>>>>>>>>> > > > >> > >> >> >>> > >> > above go right ahead. If
>>>>>>>>> somebody wants
>>>>>>>>>> > to
>>>>>>>>>> > > argue
>>>>>>>>>> > > > >> for
>>>>>>>>>> > > > >> > 16
>>>>>>>>>> > > > >> > >> >> bits
>>>>>>>>>> > > > >> > >> >> >>> or
>>>>>>>>>> > > > >> > >> >> >>> > 64
>>>>>>>>>> > > > >> > >> >> >>> > >> > bits of
>>>>>>>>>> > > > >> > >> >> >>> > >> > header key space I would
>>>>>>> listen.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > Discussion:
>>>>>>>>>> > > > >> > >> >> >>> > >> > Dividing the key space
>>> into
>>>>>>>>> sub_key_1 and
>>>>>>>>>> > > > sub_key_2
>>>>>>>>>> > > > >> > >> makes no
>>>>>>>>>> > > > >> > >> >> >>> > sense to
>>>>>>>>>> > > > >> > >> >> >>> > >> > me at
>>>>>>>>>> > > > >> > >> >> >>> > >> > this layer. Are we going
>>> to
>>>>>>>>> start
>>>>>>>>>> > providing
>>>>>>>>>> > > > APIs to
>>>>>>>>>> > > > >> > get
>>>>>>>>>> > > > >> > >> all
>>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
>>>>>>>>> sub_key_2s? If
>>>>>>>>>> > there is
>>>>>>>>>> > > > no
>>>>>>>>>> > > > >> > >> >> >>> distinguishing
>>>>>>>>>> > > > >> > >> >> >>> > >> > functions
>>>>>>>>>> > > > >> > >> >> >>> > >> > that are applied to each
>>> one
>>>>>>>>> then they
>>>>>>>>>> > should
>>>>>>>>>> > > be
>>>>>>>>>> > > > a
>>>>>>>>>> > > > >> > single
>>>>>>>>>> > > > >> > >> >> >>> value.
>>>>>>>>>> > > > >> > >> >> >>> > At
>>>>>>>>>> > > > >> > >> >> >>> > >> > this
>>>>>>>>>> > > > >> > >> >> >>> > >> > layer all we're doing is
>>>>>>>>> equality.
>>>>>>>>>> > > > >> > >> >> >>> > >> > If the above layer wants
>>> to
>>>>>>>>> interpret
>>>>>>>>>> > this as
>>>>>>>>>> > > 2,
>>>>>>>>>> > > > 3
>>>>>>>>>> > > > >> or
>>>>>>>>>> > > > >> > >> more
>>>>>>>>>> > > > >> > >> >> >>> values
>>>>>>>>>> > > > >> > >> >> >>> > >> > that's a
>>>>>>>>>> > > > >> > >> >> >>> > >> > different question. I
>>>>>>>>> personally think
>>>>>>>>>> > it's
>>>>>>>>>> > > all
>>>>>>>>>> > > > one
>>>>>>>>>> > > > >> > >> >> keyspace
>>>>>>>>>> > > > >> > >> >> >>> > that is
>>>>>>>>>> > > > >> > >> >> >>> > >> > getting assigned using
>>> some
>>>>>>>>> structure,
>>>>>>>>>> > but if
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> > want to
>>>>>>>>>> > > > >> > >> >> >>> > sub-assign
>>>>>>>>>> > > > >> > >> >> >>> > >> > parts
>>>>>>>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > The same discussion
>>> applies
>>>>> to
>>>>>>>>> strings.
>>>>>>>>>> > If
>>>>>>>>>> > > > somebody
>>>>>>>>>> > > > >> > >> argued
>>>>>>>>>> > > > >> > >> >> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > strings,
>>>>>>>>>> > > > >> > >> >> >>> > >> > would we be arguing to
>>>>> divide
>>>>>>> the
>>>>>>>>>> > strings with
>>>>>>>>>> > > > dots
>>>>>>>>>> > > > >> > ('.')
>>>>>>>>>> > > > >> > >> >> as a
>>>>>>>>>> > > > >> > >> >> >>> > >> > requirement?
>>>>>>>>>> > > > >> > >> >> >>> > >> > Would we want them to give
>>>>> us
>>>>>>> the
>>>>>>>>>> > different
>>>>>>>>>> > > name
>>>>>>>>>> > > > >> > segments
>>>>>>>>>> > > > >> > >> >> >>> > separately?
>>>>>>>>>> > > > >> > >> >> >>> > >> > Would we be performing any
>>>>>>>>> actions on
>>>>>>>>>> > this key
>>>>>>>>>> > > > other
>>>>>>>>>> > > > >> > than
>>>>>>>>>> > > > >> > >> >> >>> > matching?
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > Nacho
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
>>> 9:30
>>>>>>> AM,
>>>>>>>>> Michael
>>>>>>>>>> > > Pearce <
>>>>>>>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > #jay #jun any concerns
>>> on
>>>>> 1
>>>>>>>>> and 2
>>>>>>>>>> > still?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > @all
>>>>>>>>>> > > > >> > >> >> >>> > >> > > To get this moving
>>> along a
>>>>>>> bit
>>>>>>>>> more
>>>>>>>>>> > I'd also
>>>>>>>>>> > > > like
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> ask
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> >>> get
>>>>>>>>>> > > > >> > >> >> >>> > >> > clarity on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the below last points:
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're all
>>>>>>> roughly
>>>>>>>>> happy
>>>>>>>>>> > with the
>>>>>>>>>> > > > >> header
>>>>>>>>>> > > > >> > >> value
>>>>>>>>>> > > > >> > >> >> >>> > being a
>>>>>>>>>> > > > >> > >> >> >>> > >> > byte[]?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 4) I believe consensus
>>> has
>>>>>>>>> been for an
>>>>>>>>>> > > > namespace
>>>>>>>>>> > > > >> > based
>>>>>>>>>> > > > >> > >> int
>>>>>>>>>> > > > >> > >> >> >>> > approach
>>>>>>>>>> > > > >> > >> >> >>> > >> > > {int,int} for the key.
>>> Any
>>>>>>>>> objections
>>>>>>>>>> > if this
>>>>>>>>>> > > > is
>>>>>>>>>> > > > >> > what
>>>>>>>>>> > > > >> > >> we
>>>>>>>>>> > > > >> > >> >> go
>>>>>>>>>> > > > >> > >> >> >>> > with?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
>>>>> assumption
>>>>>>> in
>>>>>>>>> (4) is
>>>>>>>>>> > > correct,
>>>>>>>>>> > > > >> > >> {int,int}
>>>>>>>>>> > > > >> > >> >> >>> keys.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Should both int's be
>>>>> int16 or
>>>>>>>>> int32?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > I'm for them being
>>> int16(2
>>>>>>>>> bytes) as
>>>>>>>>>> > combined
>>>>>>>>>> > > > is
>>>>>>>>>> > > > >> > space
>>>>>>>>>> > > > >> > >> of
>>>>>>>>>> > > > >> > >> >> >>> > 4bytes as
>>>>>>>>>> > > > >> > >> >> >>> > >> > per
>>>>>>>>>> > > > >> > >> >> >>> > >> > > original and gives
>>> plenty
>>>>> of
>>>>>>>>>> > combinations for
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> >> >>> foreseeable,
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > keeps
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the overhead small.
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Do we see any benefit in
>>>>>>>>> another kip
>>>>>>>>>> > call to
>>>>>>>>>> > > > >> discuss
>>>>>>>>>> > > > >> > >> >> these at
>>>>>>>>>> > > > >> > >> >> >>> > all?
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Cheers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Mike
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>> ______________________________
>>>>>>>>>> > __________
>>>>>>>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
>>>>>>>>> k.burstev@yandex.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Sent: Friday, November
>>> 18,
>>>>>>> 2016
>>>>>>>>>> > 7:07:07 AM
>>>>>>>>>> > > > >> > >> >> >>> > >> > > To:
>>> dev@kafka.apache.org
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Subject: Re: [DISCUSS]
>>>>> KIP-82
>>>>>>>>> - Add
>>>>>>>>>> > Record
>>>>>>>>>> > > > Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > For what it is worth
>>> also
>>>>> i
>>>>>>>>> agree. As
>>>>>>>>>> > a user:
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
>>>>>>>>> worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers should
>>>>> be a
>>>>>>>>> top level
>>>>>>>>>> > > option
>>>>>>>>>> > > > >> > >> >> >>> > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
>>>>> "Ignacio
>>>>>>>>> Solis" <
>>>>>>>>>> > > > >> isolis@igso.net
>>>>>>>>>> > > > >> > >:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers are
>>>>>>>>> worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
>>> should
>>>>> be
>>>>>>> a
>>>>>>>>> top
>>>>>>>>>> > level
>>>>>>>>>> > > option
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14, 2016
>>> at
>>>>>>> 9:16
>>>>>>>>> AM,
>>>>>>>>>> > Michael
>>>>>>>>>> > > > Pearce
>>>>>>>>>> > > > >> <
>>>>>>>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> The kip
>>>>> details/examples
>>>>>>>>> the
>>>>>>>>>> > original
>>>>>>>>>> > > > proposal
>>>>>>>>>> > > > >> > for
>>>>>>>>>> > > > >> > >> key
>>>>>>>>>> > > > >> > >> >> >>> > spacing
>>>>>>>>>> > > > >> > >> >> >>> > >> ,
>>>>>>>>>> > > > >> > >> >> >>> > >> > not
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as per
>>>>>>>>> discussion
>>>>>>>>>> > namespace
>>>>>>>>>> > > > >> idea.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> We will need to
>>> update
>>>>>>> the
>>>>>>>>> kip,
>>>>>>>>>> > when we
>>>>>>>>>> > > get
>>>>>>>>>> > > > >> > >> agreement
>>>>>>>>>> > > > >> > >> >> >>> this
>>>>>>>>>> > > > >> > >> >> >>> > is a
>>>>>>>>>> > > > >> > >> >> >>> > >> > better
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> approach (which
>>> seems
>>>>> to
>>>>>>>>> be the
>>>>>>>>>> > case if I
>>>>>>>>>> > > > have
>>>>>>>>>> > > > >> > >> >> understood
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > general
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
>>>>>>>>> conversation)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
>>> ints,
>>>>> at
>>>>>>>>> very
>>>>>>>>>> > early stage
>>>>>>>>>> > > > we
>>>>>>>>>> > > > >> did
>>>>>>>>>> > > > >> > >> think
>>>>>>>>>> > > > >> > >> >> >>> about
>>>>>>>>>> > > > >> > >> >> >>> > >> > this. I
>>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the added complexity
>>>>> for
>>>>>>>>> the
>>>>>>>>>> > saving isn't
>>>>>>>>>> > > > >> worth
>>>>>>>>>> > > > >> > it.
>>>>>>>>>> > > > >> > >> >> I'd
>>>>>>>>>> > > > >> > >> >> >>> > rather
>>>>>>>>>> > > > >> > >> >> >>> > >> go
>>>>>>>>>> > > > >> > >> >> >>> > >> > > with, if
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
>>>>>>>>> overheads and
>>>>>>>>>> > size
>>>>>>>>>> > > int16
>>>>>>>>>> > > > >> > (2bytes)
>>>>>>>>>> > > > >> > >> >> keys
>>>>>>>>>> > > > >> > >> >> >>> as
>>>>>>>>>> > > > >> > >> >> >>> > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > keeps it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> simple.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
>>>>>>> headers,
>>>>>>>>> there
>>>>>>>>>> > is as
>>>>>>>>>> > > per
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > kip
>>>>>>>>>> > > > >> > >> as
>>>>>>>>>> > > > >> > >> >> we
>>>>>>>>>> > > > >> > >> >> >>> > use an
>>>>>>>>>> > > > >> > >> >> >>> > >> > > attribute
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
>>>>> headers
>>>>>>>>> are
>>>>>>>>>> > present or
>>>>>>>>>> > > > not as
>>>>>>>>>> > > > >> > such
>>>>>>>>>> > > > >> > >> >> >>> > provides a
>>>>>>>>>> > > > >> > >> >> >>> > >> > zero
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead currently
>>> if
>>>>>>>>> headers are
>>>>>>>>>> > not
>>>>>>>>>> > > used.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
>>>>> mentions
>>>>>>>>> would be
>>>>>>>>>> > good
>>>>>>>>>> > > > first
>>>>>>>>>> > > > >> > if we
>>>>>>>>>> > > > >> > >> >> can
>>>>>>>>>> > > > >> > >> >> >>> get
>>>>>>>>>> > > > >> > >> >> >>> > >> > clarity if
>>>>>>>>>> > > > >> > >> >> >>> > >> > > do
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> we now have general
>>>>>>>>> consensus that
>>>>>>>>>> > (1)
>>>>>>>>>> > > > headers
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > useful,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want it
>>> as
>>>>> a
>>>>>>>>> top level
>>>>>>>>>> > entity.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
>>>>>>> obvious i
>>>>>>>>>> > believe (1)
>>>>>>>>>> > > > >> headers
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> >> >>> > worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > >> > and (2)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> agree as a top level
>>>>>>>>> entity.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Mike
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>> ______________________________
>>>>>>>>>> > __________
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> From: Roger Hoover <
>>>>>>>>>> > > roger.hoover@gmail.com
>>>>>>>>>> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
>>>>> November
>>>>>>>>> 9, 2016
>>>>>>>>>> > 9:10:47
>>>>>>>>>> > > > PM
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> To:
>>>>> dev@kafka.apache.org
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
>>> [DISCUSS]
>>>>>>>>> KIP-82 - Add
>>>>>>>>>> > > Record
>>>>>>>>>> > > > >> > Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
>>>>> little
>>>>>>>>> in the
>>>>>>>>>> > weeds but
>>>>>>>>>> > > > >> thanks
>>>>>>>>>> > > > >> > >> for
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> replies
>>>>>>>>>> > > > >> > >> >> >>> > >> > > regarding
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> varint.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a prefix
>>>>> and
>>>>>>>>> {int,
>>>>>>>>>> > int} can
>>>>>>>>>> > > be
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> same.
>>>>>>>>>> > > > >> > >> >> It
>>>>>>>>>> > > > >> > >> >> >>> > doesn't
>>>>>>>>>> > > > >> > >> >> >>> > >> > look
>>>>>>>>>> > > > >> > >> >> >>> > >> > > like
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> that's what the KIP
>>> is
>>>>>>>>> saying the
>>>>>>>>>> > "Open"
>>>>>>>>>> > > > >> > section.
>>>>>>>>>> > > > >> > >> The
>>>>>>>>>> > > > >> > >> >> >>> > example
>>>>>>>>>> > > > >> > >> >> >>> > >> > shows
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
>>>>> 210002
>>>>>>>>> for App
>>>>>>>>>> > Dynamics
>>>>>>>>>> > > > >> > implying
>>>>>>>>>> > > > >> > >> >> that
>>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>>> > > > >> > >> >> >>> > New
>>>>>>>>>> > > > >> > >> >> >>> > >> > Relic
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> organization will
>>> have
>>>>>>>>> only a
>>>>>>>>>> > single
>>>>>>>>>> > > > header id
>>>>>>>>>> > > > >> > to
>>>>>>>>>> > > > >> > >> work
>>>>>>>>>> > > > >> > >> >> >>> > with. Or
>>>>>>>>>> > > > >> > >> >> >>> > >> > is
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 2100001
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The main
>>>>> point
>>>>>>>>> of a
>>>>>>>>>> > namespace
>>>>>>>>>> > > or
>>>>>>>>>> > > > >> > prefix
>>>>>>>>>> > > > >> > >> is
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> >>> > reduce
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> overhead of config
>>>>>>> mapping
>>>>>>>>> or
>>>>>>>>>> > > registration
>>>>>>>>>> > > > >> > >> depending
>>>>>>>>>> > > > >> > >> >> on
>>>>>>>>>> > > > >> > >> >> >>> how
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> namespaces/prefixes
>>>>> are
>>>>>>>>> managed.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Would love to hear
>>>>> more
>>>>>>>>> feedback
>>>>>>>>>> > on the
>>>>>>>>>> > > > >> > >> higher-level
>>>>>>>>>> > > > >> > >> >> >>> > questions
>>>>>>>>>> > > > >> > >> >> >>> > >> > > though...
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Roger
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9, 2016
>>> at
>>>>>>>>> 11:38 AM,
>>>>>>>>>> > radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I think this
>>>>> discussion
>>>>>>>>> is
>>>>>>>>>> > getting a
>>>>>>>>>> > > bit
>>>>>>>>>> > > > >> into
>>>>>>>>>> > > > >> > the
>>>>>>>>>> > > > >> > >> >> >>> weeds on
>>>>>>>>>> > > > >> > >> >> >>> > >> > technical
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > implementation
>>>>> details.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to step
>>>>> back a
>>>>>>>>> minute
>>>>>>>>>> > and try
>>>>>>>>>> > > > and
>>>>>>>>>> > > > >> > >> establish
>>>>>>>>>> > > > >> > >> >> >>> > where we
>>>>>>>>>> > > > >> > >> >> >>> > >> > are in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
>>> nacho's
>>>>>>> last
>>>>>>>>>> > paragraph)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all in
>>>>>>>>> agreement that
>>>>>>>>>> > headers
>>>>>>>>>> > > > are
>>>>>>>>>> > > > >> a
>>>>>>>>>> > > > >> > >> >> >>> worthwhile
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > useful
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > addition to have?
>>>>> this
>>>>>>>>> was
>>>>>>>>>> > contested
>>>>>>>>>> > > > early
>>>>>>>>>> > > > >> on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all in
>>>>>>>>> agreement on
>>>>>>>>>> > headers
>>>>>>>>>> > > as
>>>>>>>>>> > > > top
>>>>>>>>>> > > > >> > >> level
>>>>>>>>>> > > > >> > >> >> >>> entity
>>>>>>>>>> > > > >> > >> >> >>> > vs
>>>>>>>>>> > > > >> > >> >> >>> > >> > headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > squirreled-away in
>>>>> V?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > if there are still
>>>>>>>>> concerns
>>>>>>>>>> > around
>>>>>>>>>> > > these
>>>>>>>>>> > > > #2
>>>>>>>>>> > > > >> > >> points
>>>>>>>>>> > > > >> > >> >> >>> (#jay?
>>>>>>>>>> > > > >> > >> >> >>> > >> > #jun?)?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > (and now back to
>>> our
>>>>>>>>> normal
>>>>>>>>>> > programming
>>>>>>>>>> > > > ...)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > varints are nice.
>>>>>>> having
>>>>>>>>> said
>>>>>>>>>> > that, its
>>>>>>>>>> > > > >> adding
>>>>>>>>>> > > > >> > >> >> >>> complexity
>>>>>>>>>> > > > >> > >> >> >>> > >> (see
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> https://github.com/addthis/
>>>>>>>>>> > > > >> > >> >> stream-lib/blob/master/src/
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>> main/java/com/clearspring/
>>>>>>>>>> > > > >> > >> >> analytics/util/Varint.java
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
>>>>> result)
>>>>>>>>> and would
>>>>>>>>>> > require
>>>>>>>>>> > > > >> anyone
>>>>>>>>>> > > > >> > >> >> writing
>>>>>>>>>> > > > >> > >> >> >>> > other
>>>>>>>>>> > > > >> > >> >> >>> > >> > clients
>>>>>>>>>> > > > >> > >> >> >>> > >> > > (C?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go? Bash?
>>>>> ;-) )
>>>>>>>>> to
>>>>>>>>>> > > get/implement
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > >> same,
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > relatively
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > little gain (int
>>> vs
>>>>>>>>> string is
>>>>>>>>>> > order of
>>>>>>>>>> > > > >> > magnitude,
>>>>>>>>>> > > > >> > >> >> this
>>>>>>>>>> > > > >> > >> >> >>> > isnt).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > int namespacing vs
>>>>>>> {int,
>>>>>>>>> int}
>>>>>>>>>> > > namespacing
>>>>>>>>>> > > > >> are
>>>>>>>>>> > > > >> > >> >> basically
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > same
>>>>>>>>>> > > > >> > >> >> >>> > >> > > thing -
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > youre just
>>>>> namespacing
>>>>>>>>> an int64
>>>>>>>>>> > and
>>>>>>>>>> > > > giving
>>>>>>>>>> > > > >> > people
>>>>>>>>>> > > > >> > >> >> while
>>>>>>>>>> > > > >> > >> >> >>> > 2^32
>>>>>>>>>> > > > >> > >> >> >>> > >> > ranges
>>>>>>>>>> > > > >> > >> >> >>> > >> > > at a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > time. the part i
>>>>> like
>>>>>>>>> about this
>>>>>>>>>> > is
>>>>>>>>>> > > > letting
>>>>>>>>>> > > > >> > >> people
>>>>>>>>>> > > > >> > >> >> >>> have a
>>>>>>>>>> > > > >> > >> >> >>> > >> large
>>>>>>>>>> > > > >> > >> >> >>> > >> > > swath of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > numbers with one
>>>>>>>>> registration so
>>>>>>>>>> > they
>>>>>>>>>> > > > dont
>>>>>>>>>> > > > >> > have
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> come
>>>>>>>>>> > > > >> > >> >> >>> > back
>>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > every
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > single
>>> plugin/header
>>>>>>>>> they want to
>>>>>>>>>> > > > "reserve".
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
>>> 2016
>>>>> at
>>>>>>>>> 11:01 AM,
>>>>>>>>>> > Roger
>>>>>>>>>> > > > >> Hoover
>>>>>>>>>> > > > >> > <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > roger.hoover@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
>>> the
>>>>>>>>> debate has
>>>>>>>>>> > been
>>>>>>>>>> > > about
>>>>>>>>>> > > > >> > >> overhead +
>>>>>>>>>> > > > >> > >> >> >>> > >> > performance, I'm
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if we
>>>>> have
>>>>>>>>>> > considered a
>>>>>>>>>> > > > varint
>>>>>>>>>> > > > >> > >> encoding
>>>>>>>>>> > > > >> > >> >> (
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> https://developers.google.com/
>>>>>>>>>> > > > >> > >> >> protocol-buffers/docs/
>>>>>>>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > the header
>>> length
>>>>>>>>> field (int32
>>>>>>>>>> > in the
>>>>>>>>>> > > > >> > proposal)
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > ids? If
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > you
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
>>> headers,
>>>>>>> the
>>>>>>>>>> > overhead would
>>>>>>>>>> > > > be a
>>>>>>>>>> > > > >> > >> single
>>>>>>>>>> > > > >> > >> >> >>> byte
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > for each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128 would
>>>>> also
>>>>>>>>> need only a
>>>>>>>>>> > > single
>>>>>>>>>> > > > >> byte?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov 9,
>>>>> 2016
>>>>>>> at
>>>>>>>>> 6:43 AM,
>>>>>>>>>> > > radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus - and
>>>>> very
>>>>>>>>> dangerous
>>>>>>>>>> > (youre
>>>>>>>>>> > > > >> > >> essentially
>>>>>>>>>> > > > >> > >> >> >>> > >> > downloading and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > executing
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary code
>>>>> off
>>>>>>>>> the
>>>>>>>>>> > internet on
>>>>>>>>>> > > > your
>>>>>>>>>> > > > >> > >> servers
>>>>>>>>>> > > > >> > >> >> ...
>>>>>>>>>> > > > >> > >> >> >>> > bad
>>>>>>>>>> > > > >> > >> >> >>> > >> > idea
>>>>>>>>>> > > > >> > >> >> >>> > >> > > without
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox, even
>>>>> with)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
>>> being
>>>>> a
>>>>>>>>> purely
>>>>>>>>>> > > > administrative
>>>>>>>>>> > > > >> > task
>>>>>>>>>> > > > >> > >> - i
>>>>>>>>>> > > > >> > >> >> >>> > >> disagree.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
>>> would,
>>>>>>>>> really,
>>>>>>>>>> > because
>>>>>>>>>> > > > then my
>>>>>>>>>> > > > >> > >> earlier
>>>>>>>>>> > > > >> > >> >> >>> > point on
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > complexity
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the remapping
>>>>>>>>> process would
>>>>>>>>>> > be
>>>>>>>>>> > > > invalid,
>>>>>>>>>> > > > >> > but
>>>>>>>>>> > > > >> > >> at
>>>>>>>>>> > > > >> > >> >> >>> > linkedin,
>>>>>>>>>> > > > >> > >> >> >>> > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > example,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > we
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team im
>>> in)
>>>>>>> run
>>>>>>>>> kafka
>>>>>>>>>> > as a
>>>>>>>>>> > > > service.
>>>>>>>>>> > > > >> > we
>>>>>>>>>> > > > >> > >> dont
>>>>>>>>>> > > > >> > >> >> >>> > really
>>>>>>>>>> > > > >> > >> >> >>> > >> > know
>>>>>>>>>> > > > >> > >> >> >>> > >> > > what our
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > users
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
>>>>>>>>> applications
>>>>>>>>>> > that use
>>>>>>>>>> > > > kafka)
>>>>>>>>>> > > > >> > are
>>>>>>>>>> > > > >> > >> up
>>>>>>>>>> > > > >> > >> >> to
>>>>>>>>>> > > > >> > >> >> >>> at
>>>>>>>>>> > > > >> > >> >> >>> > any
>>>>>>>>>> > > > >> > >> >> >>> > >> > given
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> moment.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
>>> possible
>>>>>>>>> (given the
>>>>>>>>>> > > > existance of
>>>>>>>>>> > > > >> > >> headers
>>>>>>>>>> > > > >> > >> >> >>> and a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > corresponding
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem) for
>>>>> some
>>>>>>>>>> > application to
>>>>>>>>>> > > > >> "equip"
>>>>>>>>>> > > > >> > >> their
>>>>>>>>>> > > > >> > >> >> >>> > >> producers
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > consumers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
>>>>> required
>>>>>>>>> plugin
>>>>>>>>>> > without us
>>>>>>>>>> > > > >> > knowing.
>>>>>>>>>> > > > >> > >> i
>>>>>>>>>> > > > >> > >> >> dont
>>>>>>>>>> > > > >> > >> >> >>> > mean
>>>>>>>>>> > > > >> > >> >> >>> > >> > to imply
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> thats
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
>>>>> want to
>>>>>>>>> make the
>>>>>>>>>> > point
>>>>>>>>>> > > > that
>>>>>>>>>> > > > >> > its
>>>>>>>>>> > > > >> > >> not
>>>>>>>>>> > > > >> > >> >> as
>>>>>>>>>> > > > >> > >> >> >>> > simple
>>>>>>>>>> > > > >> > >> >> >>> > >> > > keeping it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > sync across a
>>>>>>>>> large-enough
>>>>>>>>>> > > > organization.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed, Nov 9,
>>>>> 2016
>>>>>>>>> at 6:17
>>>>>>>>>> > AM,
>>>>>>>>>> > > > Magnus
>>>>>>>>>> > > > >> > >> Edenhill
>>>>>>>>>> > > > >> > >> >> <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
>>> there
>>>>> is
>>>>>>> a
>>>>>>>>> piece
>>>>>>>>>> > missing
>>>>>>>>>> > > in
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> >> Strings
>>>>>>>>>> > > > >> > >> >> >>> > >> > discussion,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > where
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> pro-Stringers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason that
>>> by
>>>>>>>>> providing
>>>>>>>>>> > unique
>>>>>>>>>> > > > string
>>>>>>>>>> > > > >> > >> >> >>> identifiers
>>>>>>>>>> > > > >> > >> >> >>> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > everything
>>>>> will
>>>>>>>>> just
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
>>> work
>>>>>>> for
>>>>>>>>> all
>>>>>>>>>> > parts of
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> > stream
>>>>>>>>>> > > > >> > >> >> >>> pipeline.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
>>>>> strings
>>>>>>>>> dont mean
>>>>>>>>>> > > anything
>>>>>>>>>> > > > by
>>>>>>>>>> > > > >> > >> >> themselves,
>>>>>>>>>> > > > >> > >> >> >>> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > while we
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> could
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
>>>>> envision
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
>>>>> plugin
>>>>>>>>> loader
>>>>>>>>>> > that
>>>>>>>>>> > > > >> downloads,
>>>>>>>>>> > > > >> > >> >> compiles,
>>>>>>>>>> > > > >> > >> >> >>> > links
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > runs
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon as
>>>>>>> they're
>>>>>>>>> seen by
>>>>>>>>>> > a
>>>>>>>>>> > > > >> consumer, I
>>>>>>>>>> > > > >> > >> dont
>>>>>>>>>> > > > >> > >> >> >>> really
>>>>>>>>>> > > > >> > >> >> >>> > >> see
>>>>>>>>>> > > > >> > >> >> >>> > >> > a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > use-case
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > so dynamic
>>>>> (and
>>>>>>>>> fragile) in
>>>>>>>>>> > > > practice.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the real
>>>>> world
>>>>>>>>> an
>>>>>>>>>> > application
>>>>>>>>>> > > > will
>>>>>>>>>> > > > >> be
>>>>>>>>>> > > > >> > >> >> >>> configured
>>>>>>>>>> > > > >> > >> >> >>> > >> with
>>>>>>>>>> > > > >> > >> >> >>> > >> > a set
>>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
>>> add
>>>>>>>>> (producer)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
>>>>>>> (consumer)
>>>>>>>>> headers.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is an
>>>>>>>>> administrative
>>>>>>>>>> > task
>>>>>>>>>> > > > based
>>>>>>>>>> > > > >> on
>>>>>>>>>> > > > >> > >> what
>>>>>>>>>> > > > >> > >> >> >>> > features a
>>>>>>>>>> > > > >> > >> >> >>> > >> > client
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>> needs/provides
>>>>>>> and
>>>>>>>>> results
>>>>>>>>>> > in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > some sort of
>>>>>>>>> configuration
>>>>>>>>>> > to
>>>>>>>>>> > > > enable
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> >> >>> configure
>>>>>>>>>> > > > >> > >> >> >>> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > desired
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since this
>>>>> needs
>>>>>>>>> to be kept
>>>>>>>>>> > > > somewhat
>>>>>>>>>> > > > >> in
>>>>>>>>>> > > > >> > >> sync
>>>>>>>>>> > > > >> > >> >> >>> across
>>>>>>>>>> > > > >> > >> >> >>> > an
>>>>>>>>>> > > > >> > >> >> >>> > >> > > organisation
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no point
>>> in
>>>>>>>>> having
>>>>>>>>>> > producers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > add headers
>>> no
>>>>>>>>> consumers
>>>>>>>>>> > will
>>>>>>>>>> > > read,
>>>>>>>>>> > > > >> and
>>>>>>>>>> > > > >> > >> vice
>>>>>>>>>> > > > >> > >> >> >>> versa),
>>>>>>>>>> > > > >> > >> >> >>> > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > added
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > of assigning
>>>>> an
>>>>>>> id
>>>>>>>>>> > namespace
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
>>>>> plugin
>>>>>>> as
>>>>>>>>> it is
>>>>>>>>>> > being
>>>>>>>>>> > > > >> > configured
>>>>>>>>>> > > > >> > >> >> should
>>>>>>>>>> > > > >> > >> >> >>> be
>>>>>>>>>> > > > >> > >> >> >>> > >> > tolerable.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > 2016-11-09
>>>>> 13:06
>>>>>>>>> GMT+01:00
>>>>>>>>>> > > Michael
>>>>>>>>>> > > > >> > Pearce <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > Michael.Pearce@ig.com>:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
>>>>>>>>> following/catching
>>>>>>>>>> > up on
>>>>>>>>>> > > > what
>>>>>>>>>> > > > >> > seems
>>>>>>>>>> > > > >> > >> to
>>>>>>>>>> > > > >> > >> >> be
>>>>>>>>>> > > > >> > >> >> >>> an
>>>>>>>>>> > > > >> > >> >> >>> > >> > active
>>>>>>>>>> > > > >> > >> >> >>> > >> > > night :)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
>>>>> sorry if
>>>>>>>>> it may
>>>>>>>>>> > seem
>>>>>>>>>> > > > obvious
>>>>>>>>>> > > > >> > but
>>>>>>>>>> > > > >> > >> what
>>>>>>>>>> > > > >> > >> >> >>> does
>>>>>>>>>> > > > >> > >> >> >>> > MD
>>>>>>>>>> > > > >> > >> >> >>> > >> > stand
>>>>>>>>>> > > > >> > >> >> >>> > >> > > for?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My take on
>>>>>>>>> String vs Int:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
>>> state
>>>>>>>>> first I am
>>>>>>>>>> > pro Int
>>>>>>>>>> > > > (16
>>>>>>>>>> > > > >> or
>>>>>>>>>> > > > >> > >> 32).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
>>> though
>>>>>>>>> playing
>>>>>>>>>> > devils
>>>>>>>>>> > > > advocate
>>>>>>>>>> > > > >> > see a
>>>>>>>>>> > > > >> > >> >> big
>>>>>>>>>> > > > >> > >> >> >>> plus
>>>>>>>>>> > > > >> > >> >> >>> > >> > with the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > argument
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
>>> keys,
>>>>>>>>> this is
>>>>>>>>>> > around
>>>>>>>>>> > > > >> > integrating
>>>>>>>>>> > > > >> > >> >> into an
>>>>>>>>>> > > > >> > >> >> >>> > >> > existing
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
>>>>> other
>>>>>>>>> systems use
>>>>>>>>>> > > String
>>>>>>>>>> > > > >> based
>>>>>>>>>> > > > >> > >> >> headers
>>>>>>>>>> > > > >> > >> >> >>> > >> (Flume,
>>>>>>>>>> > > > >> > >> >> >>> > >> > JMS)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
>>> easier
>>>>> for
>>>>>>>>> these to
>>>>>>>>>> > be
>>>>>>>>>> > > > >> > >> >> >>> > incorporated/integrated
>>>>>>>>>> > > > >> > >> >> >>> > >> > into.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How with
>>> Int
>>>>>>>>> based
>>>>>>>>>> > headers
>>>>>>>>>> > > could
>>>>>>>>>> > > > we
>>>>>>>>>> > > > >> > >> provide
>>>>>>>>>> > > > >> > >> >> a
>>>>>>>>>> > > > >> > >> >> >>> > >> > way/guidence to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> make
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > this
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>> integration
>>>>>>>>> simple /
>>>>>>>>>> > easy with
>>>>>>>>>> > > > >> > transition
>>>>>>>>>> > > > >> > >> >> flows
>>>>>>>>>> > > > >> > >> >> >>> > over
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > kafka?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
>>> luck
>>>>>>>>> buddy
>>>>>>>>>> > you're on
>>>>>>>>>> > > your
>>>>>>>>>> > > > >> own
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * simply
>>>>> hash
>>>>>>>>> the string
>>>>>>>>>> > into
>>>>>>>>>> > > int
>>>>>>>>>> > > > >> code
>>>>>>>>>> > > > >> > >> and
>>>>>>>>>> > > > >> > >> >> hope
>>>>>>>>>> > > > >> > >> >> >>> > for
>>>>>>>>>> > > > >> > >> >> >>> > >> no
>>>>>>>>>> > > > >> > >> >> >>> > >> > > collisions
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
>>> back
>>>>>>>>> though?)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
>>>>> style
>>>>>>> as
>>>>>>>>>> > mentioned by
>>>>>>>>>> > > > nacho.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > ______________________________
>>>>>>>>>> > > > >> > __________
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
>>> radai
>>>>> <
>>>>>>>>>> > > > >> > radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
>>>>>>> Wednesday,
>>>>>>>>>> > November 9,
>>>>>>>>>> > > 2016
>>>>>>>>>> > > > >> > 8:12 AM
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
>>>>>>>>> dev@kafka.apache.org
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Subject:
>>> Re:
>>>>>>>>> [DISCUSS]
>>>>>>>>>> > KIP-82 -
>>>>>>>>>> > > > Add
>>>>>>>>>> > > > >> > >> Record
>>>>>>>>>> > > > >> > >> >> >>> Headers
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > thinking
>>>>> about
>>>>>>>>> it some
>>>>>>>>>> > more,
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> best
>>>>>>>>>> > > > >> > >> way to
>>>>>>>>>> > > > >> > >> >> >>> > transmit
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
>>>>>>>>> consumers would
>>>>>>>>>> > be to
>>>>>>>>>> > > > put it
>>>>>>>>>> > > > >> > in
>>>>>>>>>> > > > >> > >> the
>>>>>>>>>> > > > >> > >> >> MD
>>>>>>>>>> > > > >> > >> >> >>> > >> response
>>>>>>>>>> > > > >> > >> >> >>> > >> > > payload,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> so
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it should
>>> be
>>>>>>>>> discussed
>>>>>>>>>> > now.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
>>> Nov
>>>>> 9,
>>>>>>>>> 2016 at
>>>>>>>>>> > 12:09
>>>>>>>>>> > > AM,
>>>>>>>>>> > > > >> > radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >>
>>>>>>> radai.rosenblatt@gmail.com
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im not
>>>>>>> opposed
>>>>>>>>> to the
>>>>>>>>>> > idea of
>>>>>>>>>> > > > >> > namespace
>>>>>>>>>> > > > >> > >> >> >>> mapping.
>>>>>>>>>> > > > >> > >> >> >>> > >> all
>>>>>>>>>> > > > >> > >> >> >>> > >> > im
>>>>>>>>>> > > > >> > >> >> >>> > >> > > saying
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> is
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not part
>>>>> of
>>>>>>>>> the "mvp"
>>>>>>>>>> > and,
>>>>>>>>>> > > > since
>>>>>>>>>> > > > >> it
>>>>>>>>>> > > > >> > >> >> requires
>>>>>>>>>> > > > >> > >> >> >>> no
>>>>>>>>>> > > > >> > >> >> >>> > >> wire
>>>>>>>>>> > > > >> > >> >> >>> > >> > format
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > change,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > always
>>> be
>>>>>>>>> added later.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
>>> its
>>>>> not
>>>>>>>>> as
>>>>>>>>>> > simple as
>>>>>>>>>> > > just
>>>>>>>>>> > > > >> > >> >> configuring
>>>>>>>>>> > > > >> > >> >> >>> MM
>>>>>>>>>> > > > >> > >> >> >>> > to
>>>>>>>>>> > > > >> > >> >> >>> > >> do
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > transform:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say i've
>>>>>>>>> implemented
>>>>>>>>>> > large
>>>>>>>>>> > > > message
>>>>>>>>>> > > > >> > >> >> support as
>>>>>>>>>> > > > >> > >> >> >>> > >> > {666,1} and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> some
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > target
>>>>>>> cluster
>>>>>>>>> its been
>>>>>>>>>> > > > remapped
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> >> {999,1}.
>>>>>>>>>> > > > >> > >> >> >>> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > consumer
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> plugin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > code
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
>>> need
>>>>> to
>>>>>>>>> be told
>>>>>>>>>> > to look
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> the
>>>>>>>>>> > > > >> > >> large
>>>>>>>>>> > > > >> > >> >> >>> > message
>>>>>>>>>> > > > >> > >> >> >>> > >> > "part X
>>>>>>>>>> > > > >> > >> >> >>> > >> > > of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> Y"
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > header
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
>>>>>>> {999,1}.
>>>>>>>>> doable,
>>>>>>>>>> > but
>>>>>>>>>> > > > tricky.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On Tue,
>>>>> Nov
>>>>>>> 8,
>>>>>>>>> 2016 at
>>>>>>>>>> > 10:29
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > Gwen
>>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> gwen@confluent.io
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> While
>>> you
>>>>>>> can
>>>>>>>>> do
>>>>>>>>>> > whatever
>>>>>>>>>> > > you
>>>>>>>>>> > > > >> want
>>>>>>>>>> > > > >> > >> with a
>>>>>>>>>> > > > >> > >> >> >>> > >> namespace
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > your
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > code,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
>>> I'd
>>>>>>>>> expect is
>>>>>>>>>> > for each
>>>>>>>>>> > > > app
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> >> >>> namespaces
>>>>>>>>>> > > > >> > >> >> >>> > >> > > configurable...
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So if I
>>>>>>>>> accidentally
>>>>>>>>>> > used
>>>>>>>>>> > > 666
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> > my
>>>>>>>>>> > > > >> > >> HR
>>>>>>>>>> > > > >> > >> >> >>> > >> department,
>>>>>>>>>> > > > >> > >> >> >>> > >> > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > still
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > want
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
>>>>>>> RadaiApp,
>>>>>>>>> I can
>>>>>>>>>> > config
>>>>>>>>>> > > > >> > >> "namespace=42"
>>>>>>>>>> > > > >> > >> >> >>> for
>>>>>>>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
>>> look
>>>>>>>>> normal.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
>>>>> means
>>>>>>>>> you only
>>>>>>>>>> > need to
>>>>>>>>>> > > > sync
>>>>>>>>>> > > > >> > usage
>>>>>>>>>> > > > >> > >> >> >>> inside
>>>>>>>>>> > > > >> > >> >> >>> > your
>>>>>>>>>> > > > >> > >> >> >>> > >> > own
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> Still
>>>>> hard,
>>>>>>>>> but
>>>>>>>>>> > somewhat
>>>>>>>>>> > > > easier
>>>>>>>>>> > > > >> > than
>>>>>>>>>> > > > >> > >> >> syncing
>>>>>>>>>> > > > >> > >> >> >>> > with
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > entire
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > world.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On Tue,
>>>>> Nov
>>>>>>>>> 8, 2016
>>>>>>>>>> > at 10:07
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > >> radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>>>>>> radai.rosenblatt@gmail.com>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > and
>>> we
>>>>> can
>>>>>>>>> start
>>>>>>>>>> > with
>>>>>>>>>> > > > >> {namespace,
>>>>>>>>>> > > > >> > >> id}
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> no
>>>>>>>>>> > > > >> > >> >> >>> > >> > re-mapping
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > support
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> always
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > add
>>> it
>>>>>>>>> later on
>>>>>>>>>> > if/when
>>>>>>>>>> > > > >> > collisions
>>>>>>>>>> > > > >> > >> >> >>> actually
>>>>>>>>>> > > > >> > >> >> >>> > >> > happen (i
>>>>>>>>>> > > > >> > >> >> >>> > >> > > dont
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > think
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
>>>>>>> problem).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > every
>>>>>>>>> interested
>>>>>>>>>> > party (so
>>>>>>>>>> > > > orgs
>>>>>>>>>> > > > >> > or
>>>>>>>>>> > > > >> > >> >> >>> > individuals)
>>>>>>>>>> > > > >> > >> >> >>> > >> > could
>>>>>>>>>> > > > >> > >> >> >>> > >> > > then
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > register
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>> prefix
>>>>> (0
>>>>>>> =
>>>>>>>>>> > reserved, 1 =
>>>>>>>>>> > > > >> > confluent
>>>>>>>>>> > > > >> > >> ...
>>>>>>>>>> > > > >> > >> >> >>> 666
>>>>>>>>>> > > > >> > >> >> >>> > = me
>>>>>>>>>> > > > >> > >> >> >>> > >> > :-) )
>>>>>>>>>> > > > >> > >> >> >>> > >> > > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> do
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > the
>>>>> 2nd ID
>>>>>>>>> - so once
>>>>>>>>>> > > > linkedin
>>>>>>>>>> > > > >> > >> >> registers,
>>>>>>>>>> > > > >> > >> >> >>> say
>>>>>>>>>> > > > >> > >> >> >>> > 3,
>>>>>>>>>> > > > >> > >> >> >>> > >> > then
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
>>> use
>>>>> {3,
>>>>>>>>> *} with a
>>>>>>>>>> > > > reasonable
>>>>>>>>>> > > > >> > >> >> >>> expectation
>>>>>>>>>> > > > >> > >> >> >>> > to
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > collide
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > anything
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > else.
>>>>>>>>> further
>>>>>>>>>> > partitioning
>>>>>>>>>> > > > of
>>>>>>>>>> > > > >> > that *
>>>>>>>>>> > > > >> > >> >> >>> becomes
>>>>>>>>>> > > > >> > >> >> >>> > >> > linkedin's
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > problem,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>> "upstream
>>>>>>>>>> > registration"
>>>>>>>>>> > > of a
>>>>>>>>>> > > > >> > >> namespace
>>>>>>>>>> > > > >> > >> >> >>> only
>>>>>>>>>> > > > >> > >> >> >>> > has
>>>>>>>>>> > > > >> > >> >> >>> > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > happen
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
>>> Tue,
>>>>>>> Nov
>>>>>>>>> 8, 2016
>>>>>>>>>> > at
>>>>>>>>>> > > 9:03
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > >> James
>>>>>>>>>> > > > >> > >> >> >>> Cheng <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > >
>>>>> wushujames@gmail.com
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>>>>> Nov
>>>>>>>>> 8, 2016,
>>>>>>>>>> > at 5:54
>>>>>>>>>> > > > PM,
>>>>>>>>>> > > > >> > Gwen
>>>>>>>>>> > > > >> > >> >> >>> Shapira <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > gwen@confluent.io
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>> Thank
>>>>>>>>> you so
>>>>>>>>>> > much for
>>>>>>>>>> > > > this
>>>>>>>>>> > > > >> > clear
>>>>>>>>>> > > > >> > >> and
>>>>>>>>>> > > > >> > >> >> >>> fair
>>>>>>>>>> > > > >> > >> >> >>> > >> > summary of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > arguments.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> I'm
>>>>> in
>>>>>>>>> favor of
>>>>>>>>>> > ints.
>>>>>>>>>> > > > Not a
>>>>>>>>>> > > > >> > >> >> >>> deal-breaker,
>>>>>>>>>> > > > >> > >> >> >>> > but
>>>>>>>>>> > > > >> > >> >> >>> > >> > in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > favor.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> Even
>>>>>>>>> more in
>>>>>>>>>> > favor of
>>>>>>>>>> > > > >> Magnus's
>>>>>>>>>> > > > >> > >> >> >>> > decentralized
>>>>>>>>>> > > > >> > >> >> >>> > >> > > suggestion
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>> tweak:
>>>>>>>>> add a
>>>>>>>>>> > namespace
>>>>>>>>>> > > > for
>>>>>>>>>> > > > >> > >> headers.
>>>>>>>>>> > > > >> > >> >> >>> This
>>>>>>>>>> > > > >> > >> >> >>> > will
>>>>>>>>>> > > > >> > >> >> >>> > >> > allow
>>>>>>>>>> > > > >> > >> >> >>> > >> > > each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> use
>>>>>>>>> whatever IDs
>>>>>>>>>> > it
>>>>>>>>>> > > wants
>>>>>>>>>> > > > >> > >> >> internally,
>>>>>>>>>> > > > >> > >> >> >>> and
>>>>>>>>>> > > > >> > >> >> >>> > >> then
>>>>>>>>>> > > > >> > >> >> >>> > >> > let
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > admin
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>> deploying
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> the
>>>>> app
>>>>>>>>> figure
>>>>>>>>>> > out an
>>>>>>>>>> > > > >> > available
>>>>>>>>>> > > > >> > >> >> >>> namespace
>>>>>>>>>> > > > >> > >> >> >>> > ID
>>>>>>>>>> > > > >> > >> >> >>> > >> > for the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > live
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > So
>>>>>>>>>> > > > >> > io.confluent.schema-registry
>>>>>>>>>> > > > >> > >> can
>>>>>>>>>> > > > >> > >> >> be
>>>>>>>>>> > > > >> > >> >> >>> > >> > namespace
>>>>>>>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> my
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>> deployment
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> and
>>>>>>> 0x57
>>>>>>>>> on
>>>>>>>>>> > yours, and
>>>>>>>>>> > > > the
>>>>>>>>>> > > > >> > poor
>>>>>>>>>> > > > >> > >> guys
>>>>>>>>>> > > > >> > >> >> >>> > >> > developing the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > don't
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>> worry
>>>>>>>>> about that.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> Gwen,
>>>>> if
>>>>>>> I
>>>>>>>>>> > understand
>>>>>>>>>> > > your
>>>>>>>>>> > > > >> > example
>>>>>>>>>> > > > >> > >> >> >>> right, an
>>>>>>>>>> > > > >> > >> >> >>> > >> > > application
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> decide to
>>>>>>>>> use 0x01
>>>>>>>>>> > in one
>>>>>>>>>> > > > >> > >> deployment,
>>>>>>>>>> > > > >> > >> >> and
>>>>>>>>>> > > > >> > >> >> >>> > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > means
>>>>>>>>>> > > > >> > >> >> >>> > >> > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > once
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> message
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> is
>>>>>>> written
>>>>>>>>> into the
>>>>>>>>>> > > > broker, it
>>>>>>>>>> > > > >> > >> will be
>>>>>>>>>> > > > >> > >> >> >>> > saved on
>>>>>>>>>> > > > >> > >> >> >>> > >> > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > broker
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > with
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> specific
>>>>>>>>> namespace
>>>>>>>>>> > > (0x01).
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> If
>>> you
>>>>>>>>> were to
>>>>>>>>>> > mirror
>>>>>>>>>> > > that
>>>>>>>>>> > > > >> > message
>>>>>>>>>> > > > >> > >> >> into
>>>>>>>>>> > > > >> > >> >> >>> > another
>>>>>>>>>> > > > >> > >> >> >>> > >> > > cluster,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> accompany
>>>>>>>>> the
>>>>>>>>>> > message,
>>>>>>>>>> > > > right?
>>>>>>>>>> > > > >> > What
>>>>>>>>>> > > > >> > >> if
>>>>>>>>>> > > > >> > >> >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > deployers of
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > same
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> other
>>>>>>>>> cluster uses
>>>>>>>>>> > 0x57?
>>>>>>>>>> > > > They
>>>>>>>>>> > > > >> > won't
>>>>>>>>>> > > > >> > >> >> >>> > understand
>>>>>>>>>> > > > >> > >> >> >>> > >> > each
>>>>>>>>>> > > > >> > >> >> >>> > >> > > other?
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> I'm
>>>>> not
>>>>>>>>> sure
>>>>>>>>>> > that's an
>>>>>>>>>> > > > >> avoidable
>>>>>>>>>> > > > >> > >> >> >>> problem. I
>>>>>>>>>> > > > >> > >> >> >>> > >> > think it
>>>>>>>>>> > > > >> > >> >> >>> > >> > > simply
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > means
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> order
>>>>> to
>>>>>>>>> share
>>>>>>>>>> > data, you
>>>>>>>>>> > > > have
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> also
>>>>>>>>>> > > > >> > >> >> >>> have a
>>>>>>>>>> > > > >> > >> >> >>> > >> > shared
>>>>>>>>>> > > > >> > >> >> >>> > >> > > (agreed
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>> understanding of
>>>>>>>>>> > what the
>>>>>>>>>> > > > >> > >> namespaces
>>>>>>>>>> > > > >> > >> >> >>> mean.
>>>>>>>>>> > > > >> > >> >> >>> > >> Which
>>>>>>>>>> > > > >> > >> >> >>> > >> > I
>>>>>>>>>> > > > >> > >> >> >>> > >> > > think
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > makes
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>> because
>>>>>>> the
>>>>>>>>>> > alternate
>>>>>>>>>> > > > (sharing
>>>>>>>>>> > > > >> > >> >> *nothing*
>>>>>>>>>> > > > >> > >> >> >>> at
>>>>>>>>>> > > > >> > >> >> >>> > >> all)
>>>>>>>>>> > > > >> > >> >> >>> > >> > would
>>>>>>>>>> > > > >> > >> >> >>> > >> > > mean
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > that
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> would
>>>>> be
>>>>>>>>> no way to
>>>>>>>>>> > > > understand
>>>>>>>>>> > > > >> > each
>>>>>>>>>> > > > >> > >> >> other.
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> -James
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>> Gwen
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>>>>> Tue,
>>>>>>>>> Nov 8,
>>>>>>>>>> > 2016 at
>>>>>>>>>> > > > 4:23
>>>>>>>>>> > > > >> > PM,
>>>>>>>>>> > > > >> > >> >> radai <
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>>>>>>>>> radai.rosenblatt@gmail.com
>>>>>>>>>> > >
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>>> wrote:
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>> +1
>>>>> for
>>>>>>>>> sean's
>>>>>>>>>> > > document.
>>>>>>>>>> > > > it
>>>>>>>>>> > > > >> > >> covers
>>>>>>>>>> > > > >> > >> >> >>> pretty
>>>>>>>>>> > > > >> > >> >> >>> > >> much
>>>>>>>>>> > > > >> > >> >> >>> > >> > all
>>>>>>>>>> > > > >> > >> >> >>> > >> > > the
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>>>>>> provides
>>>>>>>>>> > concrete
>>>>>>>>>> > > > figures
>>>>>>>>>> > > > >> to
>>>>>>>>>> > > > >> > >> argue
>>>>>>>>>> > > > >> > >> >> >>> about
>>>>>>>>>> > > > >> > >> >> >>> > :-)
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>>>>>>>>> (nit-picking -
>>>>>>>>>> > used
>>>>>>>>>> > > the
>>>>>>>>>> > > > >> same
>>>>>>>>>> > > > >> > >> xkcd
>>>>>>>>>> > > > >> > >> >> >>> twice,
>>>>>>>>>> > > > >> > >> >> >>> > >> also
>>>>>>>>>> > > > >> > >> >> >>> > >> > trove
>>>>>>>>>> > > > >> > >> >> >>> > >> > > has
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > been
>>>>>>>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>>>>> superceded
>>>>>>>>>> > > > >> > >> >
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >> > --
>>>>>>>>>> > > > >> > Gwen Shapira
>>>>>>>>>> > > > >> > Product Manager | Confluent
>>>>>>>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>>>>>>>>> > > > >> > Follow us: Twitter | blog
>>>>>>>>>> > > > >> >
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> --
>>>>>>>>>> > > > >> *Todd Palino*
>>>>>>>>>> > > > >> Staff Site Reliability Engineer
>>>>>>>>>> > > > >> Data Infrastructure Streaming
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >>
>>>>>>>>>> > > > >> linkedin.com/in/toddpalino
>>>>>>>>>> > > > >>
>>>>>>>>>> > > >
>>>>>>>>>> > > >
>>>>>>>>>> > > >
>>>>>>>>>> > > > --
>>>>>>>>>> > > > Gwen Shapira
>>>>>>>>>> > > > Product Manager | Confluent
>>>>>>>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
>>>>>>>>>> > > > Follow us: Twitter | blog
>>>>>>>>>> > > >
>>>>>>>>>> > >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > The information contained in this email is strictly
>>>>> confidential
>>>>>>>>> and for
>>>>>>>>>> > the use of the addressee only, unless otherwise indicated.
>>> If
>>>>> you
>>>>>>>>> are not
>>>>>>>>>> > the intended recipient, please do not read, copy, use or
>>>>> disclose
>>>>>>>>> to others
>>>>>>>>>> > this message or any attachment. Please also notify the
>>> sender
>>>>> by
>>>>>>>>> replying
>>>>>>>>>> > to this email or by telephone (+44(020 7896 0011) and then
>>>>> delete
>>>>>>>>> the email
>>>>>>>>>> > and any copies of it. Opinions, conclusion (etc) that do not
>>>>>>>>> relate to the
>>>>>>>>>> > official business of this company shall be understood as
>>>>> neither
>>>>>>>>> given nor
>>>>>>>>>> > endorsed by it. IG is a trading name of IG Markets Limited
>>> (a
>>>>>>>>> company
>>>>>>>>>> > registered in England and Wales, company number 04008957)
>>> and
>>>>> IG
>>>>>>>>> Index
>>>>>>>>>> > Limited (a company registered in England and Wales, company
>>>>>>> number
>>>>>>>>>> > 01190902). Registered address at Cannon Bridge House, 25
>>>>> Dowgate
>>>>>>>>> Hill,
>>>>>>>>>> > London EC4R 2YA. Both IG Markets Limited (register number
>>>>> 195355)
>>>>>>>>> and IG
>>>>>>>>>> > Index Limited (register number 114059) are authorised and
>>>>>>>>> regulated by the
>>>>>>>>>> > Financial Conduct Authority.
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The information contained in this email is strictly confidential
>>> and
>>>>>>> for
>>>>>>>>> the use of the addressee only, unless otherwise indicated. If you
>>> are
>>>>>>> not
>>>>>>>>> the intended recipient, please do not read, copy, use or disclose
>>> to
>>>>>>> others
>>>>>>>>> this message or any attachment. Please also notify the sender by
>>>>>>> replying
>>>>>>>>> to this email or by telephone (+44(020 7896 0011) and then delete
>>> the
>>>>>>> email
>>>>>>>>> and any copies of it. Opinions, conclusion (etc) that do not relate
>>>>> to
>>>>>>> the
>>>>>>>>> official business of this company shall be understood as neither
>>>>> given
>>>>>>> nor
>>>>>>>>> endorsed by it. IG is a trading name of IG Markets Limited (a
>>> company
>>>>>>>>> registered in England and Wales, company number 04008957) and IG
>>>>> Index
>>>>>>>>> Limited (a company registered in England and Wales, company number
>>>>>>>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>>>>> Hill,
>>>>>>>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
>>>>> and IG
>>>>>>>>> Index Limited (register number 114059) are authorised and regulated
>>>>> by
>>>>>>> the
>>>>>>>>> Financial Conduct Authority.
>>>>>>>>>
>>>>>>>
>>>>>
>>> The information contained in this email is strictly confidential and for
>>> the use of the addressee only, unless otherwise indicated. If you are not
>>> the intended recipient, please do not read, copy, use or disclose to others
>>> this message or any attachment. Please also notify the sender by replying
>>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>>> official business of this company shall be understood as neither given nor
>>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>>> registered in England and Wales, company number 04008957) and IG Index
>>> Limited (a company registered in England and Wales, company number
>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>>> Index Limited (register number 114059) are authorised and regulated by the
>>> Financial Conduct Authority.
>>>
>
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Sean McCauliff <se...@gmail.com>.
Scroll down to "Parse Header, create collection, populate collection
search collection"
https://docs.google.com/document/d/1tfT-6SZdnKOLyWGDH82kS30PnUkmgb7nPLdw6p65pAI/edit#heading=h.66o3tyythvas
If you want string keys this means accepting a header deserialization
time of 1.5-2.5x slower than having int keys. It's not super clear
what performance target one wants to hit but if the user needs to
access the header keys then header processing would be the limiting
CPU performance factor when 1M or so messages needs to be processed
per second and there are header 5-20 keys that need to be accessed.
This would be much lower if the accessing code actually needs to do
something the associated value.
If performance is really a concern then int keys make the most sense.
Compression will probably not make CPU utilization better. Really
only a few people in any organization should care about the value of
these keys and that number may actually be zero if there is a large
third party community that provides solutions that uses the headers.
The only issue is then key namespace which is much better if people
could manage their own namespaces with strings. Otherwise some kind
of namespace scheme with integers (or longs) needs to be created as
others have pointed out.
Preregistering headers means that all producers and consumers need to
see the same mapping. This seems very problematic to get right with a
large Kafka deployment.
The lazy map thing is a good idea but is really a micro optimization.
This can be added later.
Rather than having a new type, like Headers , which involves more
object allocation, just have a method like byte[] header(int
headerKey) in ConsumerRecord that completely hides the implementation
of headers.
Regarding the need for detailed use cases for headers: the use case
for headers is not mixing control (or metadata) with data; that's it.
If I have metadata in my data then it's difficult to reuse code that
wants this data. If I have data in my metadata then it's difficult to
reuse the control code that consumes the metadata. This is a basic
concept that is is implemented in many protocols, serialized data
formats, file systems and more.
Sean
On Wed, Jan 11, 2017 at 9:05 AM, Jay Kreps <ja...@confluent.io> wrote:
> Yeah I think I share a couple of concerns around the details of the
> proposal beyond the use cases:
>
> 1. I think numeric keys are just a non-starter for a user-facing
> interface that is meant to work across third party components (not just
> within an organization). We've gone down this path of making cryptic
> interfaces before based on what's easy to implement (simple consumer!). The
> result is a mess we are still trying to walk away from. I think we are
> solving a compression/normalization problem by having the user implement a
> compression algorithm in their head and share it across their company and
> between components and plugin. This is definitely easier for us, but not
> the right interface! I think we should consider instead (1) just take the
> perf hit and claim it's okay since it's optional (I don't necessarily buy
> that since it makes a lot of common use cases pretty questionable), (2)
> work on getting more efficient compression in place so we can recommend
> that without reservation (e.g. if that facebook lib pans out) or (3) allow
> the client to optionally pre-register headers to get a more compact
> numerical mapping.
> 2. I think we should think about creating the Map lazily to avoid
> parsing out all the headers into little objects. HashMaps themselves are
> kind of expensive and the consumer is very perf sensitive so and making
> gazillions of hashmaps that may or may not get used is probably a bad idea.
> 3. We should think about having our own type, Headers or something
> rather than a Map. I kind of dislike this kind of unnecessary type in
> internal code, but for this kind of public API the ability to get just the
> methods that make sense and document the details of the semantic can be
> worthwhile.
>
> -Jay
>
> On Tue, Jan 10, 2017 at 10:38 PM, Ewen Cheslack-Postava <ew...@confluent.io>
> wrote:
>
>> I'm horribly behind and outdated on this thread, but wanted to mention one
>> other potential issue.
>>
>> The KIP talks about using simple Maps for the headers. For a lot of this
>> discussion, I tend to either think of completely opaque headers (i.e just
>> another byte[] blob, which seems to have been rejected based on
>> interoperability grounds but I think has the nice property of making it
>> easy to blame the implementer of the serde for any problems with the header
>> implementation) or HTTP-style headers (seemingly the most ubiquitous
>> example of headers, though certainly there are many others, and one which
>> we/users may need/want to interoperate with).
>>
>> Strictly speaking, the keys on HTTP headers aren't unique. See
>> https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
>> specifically:
>>
>> > Multiple message-header fields with the same field-name MAY be present in
>> a message if and only if the entire field-value for that header field is
>> defined as a comma-separated list [i.e., #(values)]. It MUST be possible to
>> combine the multiple header fields into one "field-name: field-value" pair,
>> without changing the semantics of the message, by appending each subsequent
>> field-value to the first, each separated by a comma. The order in which
>> header fields with the same field-name are received is therefore
>> significant to the interpretation of the combined field value, and thus a
>> proxy MUST NOT change the order of these field values when a message is
>> forwarded.
>>
>> The spec guarantees that you can collapse repeated headers into a single
>> instance. But in practice there's a ton of stuff that doesn't. And every
>> robust HTTP library I've encountered uses a multimap to represent headers.
>>
>> I'm not necessarily advocating for supporting this, just suggesting that we
>> think about it. As you add these features and get closer to mapping to
>> other systems, people will inevitably try to map them. Headers are an area
>> where, if we're going to add them, it's worth considering compatibility as
>> someone will inevitably come and complain that system X does Y with headers
>> and we should also support Y because any decent system that provides
>> headers will do so.
>>
>> -Ewen
>>
>> On Tue, Jan 10, 2017 at 5:27 PM, Jay Kreps <ja...@confluent.io> wrote:
>>
>> > Hey Gwen, yes, your approach is much better than what I described and
>> > solves one of my objections. I didn't think of doing it in MM rather than
>> > in the original producer. I don't think it adds any overhead beyond the
>> > additional header bytes if I'm not mistaken, since MM already unpacks and
>> > repacks messages (unlike the broker which tries to avoid repacking and
>> > hence can't efficiently add a header). I think my other objection around
>> > size is likely solvable too by some kind of aliasing scheme? If you guys
>> > think this can be made to work then I think we have two pretty credible
>> use
>> > cases with this and the tracing id family of uses, plus a number of
>> things
>> > that are at least moderately improved. If so, I'm convinced.
>> >
>> > -Jay
>> >
>> >
>> > On Tue, Jan 10, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
>> > wrote:
>> >
>> > > Hi Jay et al.
>> > >
>> > > On the replication using cluster id.
>> > >
>> > > Two options depending on how a company wants or needs to implement
>> their
>> > > replication as there are options of doing this as a mesh, star, or
>> ring.
>> > >
>> > > Either as some companies enforce a provider/custom producer/consumer
>> > > wrapper, it can be enforced the interceptors needed to be added, as you
>> > can
>> > > enforce to get support of the platform you must use company x’s
>> internal
>> > > wrappers.
>> > > An alternative approach is to do this at the mirror maker, as Gwen
>> noted
>> > > we can actually make the mirror maker append this, like wise for a more
>> > > complex implementation, where you can have multiple hops/routes this
>> can
>> > > also be the message routing.
>> > >
>> > > Re the size of the clusterId, yes the auto generated one is “cough”
>> > > hideous in size.
>> > >
>> > > Most organisations though will only need a space to uniquely identify
>> 127
>> > > (byte) clusters, or at worst 2 bytes (short) they need to mirror and
>> > > replicate between, and as such can either as platform teams generally
>> > will
>> > > roll out the mirror makers / replicators with the cluster can either
>> > > manually adjust the clusterId on ZK, or use a mapping file all their
>> > mirror
>> > > makers have to map the larger id to a smaller one.
>> > >
>> > > Also as Gwen states this is something you should be able to opt into or
>> > > not, this is the benefit of headers, if you don’t use it, it doesn’t
>> use
>> > up
>> > > the space, its entirely optional.
>> > >
>> > > Regards
>> > > Mike
>> > >
>> > >
>> > > On 09/01/2017, 19:40, "Gwen Shapira" <gw...@confluent.io> wrote:
>> > >
>> > > Mirroring + ClusterIDs:
>> > >
>> > > Kafka clusters already have an automatically generated ID that is
>> > > exposed to clients:
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> > 78%3A+Cluster+Id
>> > >
>> > > It sounds trivial to get MirrorMaker's consumer to grab this ID
>> > (which
>> > > always exists) from the source cluster and add it to the message
>> > > headers.
>> > >
>> > > Obviously adding stuff to the message makes it larger so this
>> should
>> > > be a feature you should be able to turn on/off depending on the
>> > > replication requirements. If you need the feature - you pay the
>> > price.
>> > > I don't think there is an immediate need to put an effort into
>> > > shrinking the Cluster ID, but others can comment on that.
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io>
>> wrote:
>> > > > Hey Michael,
>> > > >
>> > > > Sorry for lagging, I've been out for the holiday break. What
>> would
>> > > help me
>> > > > would be to get to a consolidated list of use cases we've worked
>> > out
>> > > and
>> > > > think add real value. I think getting down to the details really
>> > > matters
>> > > > for these.
>> > > >
>> > > > The two lists I'm aware of are the one in the KIP and the one you
>> > > gave in
>> > > > this email thread. Both seem to be more at the sketch level. Let
>> me
>> > > review
>> > > > the use cases and see if I have it right. If so let's get it in
>> the
>> > > design
>> > > > proposal.
>> > > >
>> > > > Your use cases:
>> > > >
>> > > > 1. Tracing IDs -- I get this use case and think I see how it
>> > > would work.
>> > > > I think because the tracing is from a third party system there
>> > is
>> > > some
>> > > > value to having a header versus a custom format to make this
>> > work
>> > > fully
>> > > > transparently.
>> > > > 2. Mirroring -- I'm a bit unsure how this will work, I'll
>> > explain
>> > > my
>> > > > thinking below
>> > > > 3. End-to-end encryption -- I think what you are saying is
>> that
>> > > if you
>> > > > want "routing' fields that are accessible in transit without
>> > > decrypting you
>> > > > need a scheme to leave this decrypted.I think the idea is to
>> > avoid
>> > > > decrypting to get a header, but I'm not sure if I get how this
>> > is
>> > > better
>> > > > than a custom message format that doesn't encrypt the things
>> you
>> > > don't want
>> > > > encrypted since you have to be aware of the scheme anyway.
>> > > > 4. EOS -- I'm not convinced this is a good approach for EOS.
>> > > >
>> > > > Linked from KIP:
>> > > >
>> > > > 1. Schema-Registry-aware avro serialization -- not clear how
>> > this
>> > > is
>> > > > better than keeping the schema id in the value
>> > > > 2. Polyglot kafka topics -- marginally better than a container
>> > > format,
>> > > > but of course you still have to all agree on the the header
>> and
>> > > > header=>serialization mapping so not sure if it's really
>> better
>> > > than a
>> > > > container format. Also not wild about polyglot topics :-)
>> > > > 3. End-to-End routing trail including mirroring --- I'd put
>> this
>> > > under
>> > > > tracing ids
>> > > > 4. Cross-service process tracing -- ditto
>> > > > 5. Signing -- I like this use case but it seems to depend on a
>> > > feature
>> > > > we don't have and aren't proposing adding, right?
>> > > > 6. Per-record custom compression -- why do you want this?
>> > > compressing
>> > > > per message won't work well compared to batch compression,
>> > right?
>> > > > 7. Large Message Support -- I think we can agree that although
>> > > this
>> > > > might work it's clearly not the right way to support large
>> > > messages in Kafka
>> > > >
>> > > > Matthias
>> > > >
>> > > > 1. Control messages --- Matthias, not sure if I get the
>> > > motivation in
>> > > > streams for this. If we did want to do it, wouldn't it be
>> easier
>> > > to
>> > > > implement and by allowing users to send control messages (in
>> the
>> > > EOS sense)
>> > > > and adding a consumer config to allow either passing these on
>> or
>> > > > suppressing them?
>> > > >
>> > > > So I if I were to summarize I'd list the use cases as follows:
>> > > >
>> > > > - Much better with headers: tracing IDs
>> > > > - Somewhat better: end-to-end encryption, polyglot topics
>> > > > - Don't buy: Mirroring (though perhaps could be made to work
>> > with
>> > > > changes to the proposal), EOS, large messages, compression
>> > > > - Not sure: Control messages
>> > > >
>> > > > Does this match where other people are at? From my point of view
>> > for
>> > > > something like this where we are changing the core data model we
>> > > should
>> > > > ideally get a handful of use cases to the point where they are
>> > > really well
>> > > > thought out and we know they can use the feature. I think we have
>> > > one solid
>> > > > one with tracing IDs, and I think maybe mirroring could be
>> another
>> > > with
>> > > > some work. I'm less sure about the others.
>> > > >
>> > > > Bi-directional Mirroring
>> > > > I think bidirectional mirroring could potentially become a
>> > > compelling use
>> > > > case if we an find a way to make it work. If I understand it, the
>> > > proposal
>> > > > is to add a header like "origin-cluster" with the cluster id of
>> the
>> > > Kafka
>> > > > cluster. Then modify the mirroring tool to disallow mirroring a
>> > > message
>> > > > back to it's own origin cluster. This basic idea is definitely
>> > > workable and
>> > > > useful, but can we really get to a good implementation using
>> > headers?
>> > > > I imagine the approach would be to add this header in the
>> producer
>> > > if some
>> > > > config is set. There are two problems:
>> > > >
>> > > > - You add a pretty big honking cluster ID to each message. For
>> > > small
>> > > > messages this maybe larger than the contents itself.
>> > > > - What happens if someone somewhere forgets to set that config
>> > on
>> > > some
>> > > > client somewhere?
>> > > >
>> > > > If someone forgets to set the config in some client somewhere I
>> > > guess you
>> > > > have various options in mirror maker:
>> > > >
>> > > > 1. drop the data (pretty bad, now your data is out of sync)
>> > > > 2. crash (pretty bad--how do you recover)
>> > > > 3. insert some default (pretty bad since would also lead to
>> data
>> > > loss if
>> > > > that's not right)
>> > > >
>> > > > Basically although I think a cluster tag could be a good way to
>> do
>> > > > bi-directional mirroring, I'm not sure this naive approach I
>> > > described
>> > > > would get us to a usable implementation. This could be fixable. I
>> > > think if
>> > > > there was some scheme to check required headers on the server
>> that
>> > > could
>> > > > alleviate the missing header problem. Not really sure what to do
>> > > about the
>> > > > message bloat problem, but maybe there could be some way to
>> shrink
>> > > the
>> > > > cluster id?
>> > > >
>> > > > Cheers,
>> > > >
>> > > > -Jay
>> > > >
>> > > >
>> > > > On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
>> > > Michael.Pearce@ig.com>
>> > > > wrote:
>> > > >
>> > > >> @Jay, Ismael and a few others.
>> > > >>
>> > > >> Based on the discussion on kip-87, and the additional use case
>> by
>> > > Matthias.
>> > > >>
>> > > >> Just for clarity do we now have consensus on the earlier points
>> > (1)
>> > > >> headers are useful (2) headers should be native in Kafka?
>> > > >>
>> > > >> Cheers
>> > > >> Mike
>> > > >> ________________________________________
>> > > >> From: radai <ra...@gmail.com>
>> > > >> Sent: Monday, December 19, 2016 5:01 PM
>> > > >> To: dev@kafka.apache.org
>> > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>> > > >>
>> > > >> having said that im perfectly fine with something like a "no
>> > > payload" flag
>> > > >> (as opposed to null payload), which would mean such a msg is not
>> > > returned
>> > > >> from poll() unless the user code specifically "opts in" and
>> would
>> > > serve
>> > > >> only to carry headers (making it a control message)
>> > > >>
>> > > >> On Mon, Dec 19, 2016 at 8:58 AM, radai <
>> > radai.rosenblatt@gmail.com>
>> > > wrote:
>> > > >>
>> > > >> > if all you want is to tag "end of stream" wouldnt a header on
>> > the
>> > > last
>> > > >> msg
>> > > >> > in the stream be enough?
>> > > >> >
>> > > >> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
>> > > matthias@confluent.io>
>> > > >> > wrote:
>> > > >> >
>> > > >> >> The details about headers for control messages are still to
>> > > define. But
>> > > >> >> yes, the idea is to have some common default behavior that
>> > > clients would
>> > > >> >> need to implement.
>> > > >> >>
>> > > >> >> The point is, that "regular headers" add meta data to regular
>> > > messages.
>> > > >> >> Thus, those messages will be returned to the user via
>> .poll().
>> > > And after
>> > > >> >> the message is received the user can check if meta data is
>> > > present and
>> > > >> >> read it.
>> > > >> >>
>> > > >> >> For control messages, we do not want those to pop up via
>> > .poll()
>> > > as
>> > > >> >> those are no regular messages. A client would need to opt-in
>> to
>> > > see
>> > > >> >> those messages (either via poll() or maybe a callback). Thus,
>> > we
>> > > need
>> > > >> >> some special (standardized) header IDs that indicate control
>> > > messages
>> > > >> >> that should not be returned to the user via poll() by
>> default.
>> > > >> >>
>> > > >> >>
>> > > >> >> -Matthias
>> > > >> >>
>> > > >> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
>> > > >> >> > Matthias,
>> > > >> >> >
>> > > >> >> > Thanks for your input. I'm +1 on control messages as they
>> > > seem to be
>> > > >> >> the
>> > > >> >> > simplest way to implement watermarks (
>> > > >> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-
>> > > streaming-102),
>> > > >> a
>> > > >> >> > feature that would add a lot of value to Kafka Streams
>> IMHO.
>> > > >> >> >
>> > > >> >> > Your argument that the control-message indicator belongs in
>> > the
>> > > >> >> client-only
>> > > >> >> > section of the record format make sense. Just to make
>> sure I
>> > > >> >> understand,
>> > > >> >> > are you suggesting that control messages would be indicated
>> > by
>> > > a
>> > > >> >> standard
>> > > >> >> > reserved header? By standard, I mean that ALL Kafka
>> > consumers
>> > > would
>> > > >> >> know
>> > > >> >> > to handle these messages differently (possibly just
>> ignoring
>> > > them).
>> > > >> >> This
>> > > >> >> > would need to be added to the specification of the consumer
>> > > protocol
>> > > >> so
>> > > >> >> > that all Kafka clients implement it, right? I think it's a
>> > > good idea
>> > > >> >> but
>> > > >> >> > just checking.
>> > > >> >> >
>> > > >> >> > Cheers,
>> > > >> >> >
>> > > >> >> > Roger
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>> > > >> matthias@confluent.io
>> > > >> >> >
>> > > >> >> > wrote:
>> > > >> >> >
>> > > >> >> >> Yes and no. I did overload the term "control message".
>> > > >> >> >>
>> > > >> >> >> EOS control messages are for client-broker communication
>> and
>> > > thus
>> > > >> never
>> > > >> >> >> exposed to any application. And I think this is a good
>> > design
>> > > because
>> > > >> >> >> broker needs to understand those control messages. Thus,
>> > this
>> > > should
>> > > >> be
>> > > >> >> >> a protocol change.
>> > > >> >> >>
>> > > >> >> >> The type of control messages I have in mind are for
>> > > client-client
>> > > >> >> >> (application-application) communication and the broker is
>> > > agnostic to
>> > > >> >> >> them. Thus, it should not be a protocol change.
>> > > >> >> >>
>> > > >> >> >>
>> > > >> >> >> -Matthias
>> > > >> >> >>
>> > > >> >> >>
>> > > >> >> >>
>> > > >> >> >> On 12/14/16 9:42 AM, radai wrote:
>> > > >> >> >>> arent control messages getting pushed as their own top
>> > level
>> > > >> protocol
>> > > >> >> >>> change (and a fairly massive one) for the transactions
>> KIP
>> > ?
>> > > >> >> >>>
>> > > >> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>> > > >> >> matthias@confluent.io>
>> > > >> >> >>> wrote:
>> > > >> >> >>>
>> > > >> >> >>>> Hi,
>> > > >> >> >>>>
>> > > >> >> >>>> I want to add a completely new angle to this discussion.
>> > > For this,
>> > > >> I
>> > > >> >> >>>> want to propose an extension for the headers feature
>> that
>> > > enables
>> > > >> new
>> > > >> >> >>>> uses cases -- and those new use cases might convince
>> > people
>> > > to
>> > > >> >> support
>> > > >> >> >>>> headers (of course including the larger scoped
>> proposal).
>> > > >> >> >>>>
>> > > >> >> >>>> Extended Proposal:
>> > > >> >> >>>>
>> > > >> >> >>>> Allow messages with a certain header key to be special
>> > > "control
>> > > >> >> >>>> messages" (w/ o w/o payload) that are not exposed to an
>> > > application
>> > > >> >> via
>> > > >> >> >>>> .poll().
>> > > >> >> >>>>
>> > > >> >> >>>> Thus, a consumer client would automatically skip over
>> > those
>> > > >> >> messages. If
>> > > >> >> >>>> an application knows about embedded control messages, it
>> > > can "sing
>> > > >> >> up"
>> > > >> >> >>>> to those messages by the consumer client and either get
>> a
>> > > callback
>> > > >> or
>> > > >> >> >>>> the consumer auto-drop for this messages gets disabled
>> > > (allowing to
>> > > >> >> >>>> consumer those messages via poll()).
>> > > >> >> >>>>
>> > > >> >> >>>> (The details need further considerations/discussion. I
>> > just
>> > > want to
>> > > >> >> >>>> sketch the main idea.)
>> > > >> >> >>>>
>> > > >> >> >>>> Usage:
>> > > >> >> >>>>
>> > > >> >> >>>> There is a shared topic (ie, used by multiple
>> > applications)
>> > > and a
>> > > >> >> >>>> producer application wants to embed a special message in
>> > > the topic
>> > > >> >> for a
>> > > >> >> >>>> dedicated consumer application. Because only one
>> > > application will
>> > > >> >> >>>> understand this message, it cannot be a regular message
>> as
>> > > this
>> > > >> would
>> > > >> >> >>>> break all applications that do not understand this
>> > message.
>> > > The
>> > > >> >> producer
>> > > >> >> >>>> application would set a special metadata key and no
>> > consumer
>> > > >> >> application
>> > > >> >> >>>> would see this control message by default because they
>> did
>> > > not
>> > > >> enable
>> > > >> >> >>>> their consumer client to return this message in poll()
>> > (and
>> > > the
>> > > >> >> client
>> > > >> >> >>>> would just drop this message with special metadata key).
>> > > Only the
>> > > >> >> single
>> > > >> >> >>>> application that should receive this message, will
>> > > subscribe to
>> > > >> this
>> > > >> >> >>>> message on its consumer client and process it.
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> Concrete Use Case: Kafka Streams
>> > > >> >> >>>>
>> > > >> >> >>>> In Kafka Streams, we would like to propagate "control
>> > > messages"
>> > > >> from
>> > > >> >> >>>> subtopology to subtopology. There are multiple scenarios
>> > > for which
>> > > >> >> this
>> > > >> >> >>>> would be useful. For example, currently we do not
>> > guarantee
>> > > a
>> > > >> >> >>>> "consistent shutdown" of an application. By this, I mean
>> > > that input
>> > > >> >> >>>> records might not be completely processed by the whole
>> > > topology
>> > > >> >> because
>> > > >> >> >>>> the application shutdown happens "in between" and an
>> > > intermediate
>> > > >> >> result
>> > > >> >> >>>> topic gets "stock" in an intermediate topic. Thus, a
>> user
>> > > would see
>> > > >> >> an
>> > > >> >> >>>> committed offset of the source topic of the application,
>> > > but no
>> > > >> >> >>>> corresponding result record in the output topic.
>> > > >> >> >>>>
>> > > >> >> >>>> Having "shutdown markers" would allow us, to first stop
>> > the
>> > > >> upstream
>> > > >> >> >>>> subtopology and write this marker into the intermediate
>> > > topic and
>> > > >> the
>> > > >> >> >>>> downstream subtopology would only shut down itself after
>> > is
>> > > sees
>> > > >> the
>> > > >> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown,
>> > that
>> > > no
>> > > >> >> >>>> "in-flight" messages got stuck in intermediate topics.
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> A similar usage would be for KIP-95 (Incremental Batch
>> > > Processing).
>> > > >> >> >>>> There was a discussion about the proposed metadata
>> topic,
>> > > and we
>> > > >> >> could
>> > > >> >> >>>> avoid this metadata topic if we would have "control
>> > > messages".
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> Right now, we cannot insert an "application control
>> > message"
>> > > >> because
>> > > >> >> >>>> Kafka Streams does not own all topics it read/writes and
>> > > thus might
>> > > >> >> >>>> break other consumer application (as described above) if
>> > we
>> > > inject
>> > > >> >> >>>> random messages that are not understood by other apps.
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> Of course, one can work around "embedded control
>> messaged"
>> > > by using
>> > > >> >> an
>> > > >> >> >>>> additional topic to propagate control messaged between
>> > > application
>> > > >> >> (as
>> > > >> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka
>> > > Streams). But
>> > > >> >> there
>> > > >> >> >>>> are major concerns about adding this metadata topic in
>> the
>> > > KIP and
>> > > >> >> this
>> > > >> >> >>>> shows that other application that need a similar pattern
>> > > might
>> > > >> profit
>> > > >> >> >>>> from topic embedded "control messages", too.
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> One last important consideration: those "control
>> messages"
>> > > are used
>> > > >> >> for
>> > > >> >> >>>> client to client communication and are not understood by
>> > the
>> > > >> broker.
>> > > >> >> >>>> Thus, those messages should not be enabled within the
>> > > message
>> > > >> format
>> > > >> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land"
>> > > record
>> > > >> >> headers
>> > > >> >> >>>> would be a nice way to implement them. Because KIP-82
>> did
>> > > consider
>> > > >> >> key
>> > > >> >> >>>> namespaces for metatdata keys, this extension should not
>> > be
>> > > an own
>> > > >> >> KIP
>> > > >> >> >>>> but should be included in KIP-82 to reserve a namespace
>> > for
>> > > >> "control
>> > > >> >> >>>> message" in the first place.
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> Sorry for the long email... Looking forward to your
>> > > feedback.
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> -Matthias
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>> > > >> >> >>>>> Hi Jun
>> > > >> >> >>>>>
>> > > >> >> >>>>> 100) each time a transaction exits a jvm for a remote
>> > > system
>> > > >> >> (HTTP/JMS/
>> > > >> >> >>>> Hopefully one day kafka) the APM tools stich in a unique
>> > id
>> > > >> (though I
>> > > >> >> >>>> believe it contains the end2end uuid embedded in this
>> id),
>> > > on
>> > > >> >> receiving
>> > > >> >> >> the
>> > > >> >> >>>> message at the receiving JVM the apm code takes this
>> out,
>> > > and
>> > > >> >> continues
>> > > >> >> >> its
>> > > >> >> >>>> tracing on the that new thread. Both JVM’s (and other
>> > > languages the
>> > > >> >> APM
>> > > >> >> >>>> tool supports) send this data async back to the central
>> > > controllers
>> > > >> >> >> where
>> > > >> >> >>>> the stiching togeather occurs. For this they need some
>> > > header space
>> > > >> >> for
>> > > >> >> >>>> them to put this id.
>> > > >> >> >>>>>
>> > > >> >> >>>>> 101) Yes indeed we have a business transaction Id in
>> the
>> > > payload.
>> > > >> >> >> Though
>> > > >> >> >>>> this is a system level tracing, that we need to have
>> marry
>> > > up. Also
>> > > >> >> as
>> > > >> >> >> per
>> > > >> >> >>>> note on end2end encryption we’d be unable to prove the
>> > flow
>> > > if the
>> > > >> >> >> payload
>> > > >> >> >>>> is encrypted as we’d not have access to this at certain
>> > > points of
>> > > >> the
>> > > >> >> >> flow
>> > > >> >> >>>> through the infrastructure/platform.
>> > > >> >> >>>>>
>> > > >> >> >>>>>
>> > > >> >> >>>>> 103) As said we use this mechanism in IG very
>> > > successfully, as
>> > > >> >> stated
>> > > >> >> >>>> per key we guarantee the transaction producing app to
>> > > handle the
>> > > >> >> >>>> transaction of a key at one DC unless at point of
>> critical
>> > > failure
>> > > >> >> >> where we
>> > > >> >> >>>> have to flip processing to another. We care about key
>> > > ordering.
>> > > >> >> >>>>> I disagree on the offset comment for the partition
>> > > solution unless
>> > > >> >> you
>> > > >> >> >>>> do full ISR, or expensive full XA transactions even with
>> > > partitions
>> > > >> >> you
>> > > >> >> >>>> cannot fully guarantee offsets would match.
>> > > >> >> >>>>>
>> > > >> >> >>>>> 105) Very much so, I need to have access at the
>> platform
>> > > level to
>> > > >> >> the
>> > > >> >> >>>> other meta data all mentioned, without having to need to
>> > > have
>> > > >> access
>> > > >> >> to
>> > > >> >> >> the
>> > > >> >> >>>> encryption keys of the payload.
>> > > >> >> >>>>>
>> > > >> >> >>>>> 106)
>> > > >> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d
>> need
>> > > to have
>> > > >> a
>> > > >> >> >>>> global producerId register which would be very hard to
>> > > >> >> enforce/ensure is
>> > > >> >> >>>> current and correct, just to understand the message
>> > origins
>> > > of its
>> > > >> >> >>>> region/az/cluster for routing.
>> > > >> >> >>>>> The client wrapper version, producerId can be the same,
>> > as
>> > > >> obviously
>> > > >> >> >> the
>> > > >> >> >>>> producer could upgrade its wrapper, as such we need to
>> > know
>> > > what
>> > > >> >> wrapper
>> > > >> >> >>>> version the message is created with.
>> > > >> >> >>>>> Likewise the IP address, as stated we can have our
>> > > producer move,
>> > > >> >> where
>> > > >> >> >>>> its IP would change.
>> > > >> >> >>>>>
>> > > >> >> >>>>> 107)
>> > > >> >> >>>>> UUID is set on the message by interceptors before
>> actual
>> > > producer
>> > > >> >> >>>> transport send. This is for platform level message
>> dedupe
>> > > >> guarantee,
>> > > >> >> the
>> > > >> >> >>>> business payload should be agnostic to this. Please see
>> > > >> >> >>>> https://activemq.apache.org/
>> artemis/docs/1.5.0/duplicate-
>> > > det
>> > > >> >> ection.html
>> > > >> >> >>>> note this is not touching business payloads.
>> > > >> >> >>>>>
>> > > >> >> >>>>>
>> > > >> >> >>>>>
>> > > >> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io>
>> > wrote:
>> > > >> >> >>>>>
>> > > >> >> >>>>> Hi, Michael,
>> > > >> >> >>>>>
>> > > >> >> >>>>> Thanks for the reply. I find it very helpful.
>> > > >> >> >>>>>
>> > > >> >> >>>>> Data lineage:
>> > > >> >> >>>>> 100. I'd like to understand the APM use case a bit
>> > > more. It
>> > > >> >> sounds
>> > > >> >> >>>> like
>> > > >> >> >>>>> that those APM plugins can generate a transaction
>> id
>> > > that we
>> > > >> >> could
>> > > >> >> >>>>> potentially put in the header of every message. How
>> > > would you
>> > > >> >> >>>> typically
>> > > >> >> >>>>> make use of such transaction ids? Are there other
>> > > metadata
>> > > >> >> >>>> associated with
>> > > >> >> >>>>> the transaction id and if so, how are they
>> propagated
>> > > >> >> downstream?
>> > > >> >> >>>>>
>> > > >> >> >>>>> 101. For the finance use case, if the concept of
>> > > transaction
>> > > >> is
>> > > >> >> >>>> important,
>> > > >> >> >>>>> wouldn't it be typically included in the message
>> > > payload
>> > > >> >> instead of
>> > > >> >> >>>> as an
>> > > >> >> >>>>> optional header field?
>> > > >> >> >>>>>
>> > > >> >> >>>>> 102. The data lineage that Altas and Navigator
>> > support
>> > > seems
>> > > >> to
>> > > >> >> be
>> > > >> >> >>>> at the
>> > > >> >> >>>>> dataset level, not per record level? So, not sure
>> if
>> > > per
>> > > >> message
>> > > >> >> >>>> headers
>> > > >> >> >>>>> are relevant there.
>> > > >> >> >>>>>
>> > > >> >> >>>>> Mirroring:
>> > > >> >> >>>>> 103. The benefit of using separate partitions is
>> that
>> > > it
>> > > >> >> >> potentially
>> > > >> >> >>>> makes
>> > > >> >> >>>>> it easy to preserve offsets during mirroring. This
>> > > will make
>> > > >> it
>> > > >> >> >>>> easier for
>> > > >> >> >>>>> consumer to switch clusters. Currently, the
>> consumers
>> > > can
>> > > >> switch
>> > > >> >> >>>> clusters
>> > > >> >> >>>>> by using the timestampToOffset() api, but it has to
>> > > deal with
>> > > >> >> >>>> duplicates.
>> > > >> >> >>>>> Good point on the issue with log compact and I am
>> not
>> > > sure how
>> > > >> >> to
>> > > >> >> >>>> address
>> > > >> >> >>>>> this. However, even if we mirror into the existing
>> > > partitions,
>> > > >> >> the
>> > > >> >> >>>> ordering
>> > > >> >> >>>>> for messages generated from different clusters
>> seems
>> > > >> >> >>>> non-deterministic
>> > > >> >> >>>>> anyway. So, it seems that the consumers already
>> have
>> > > to deal
>> > > >> >> with
>> > > >> >> >>>> that? If
>> > > >> >> >>>>> a topic is compacted, does that mean which messages
>> > are
>> > > >> >> preserved
>> > > >> >> >> is
>> > > >> >> >>>> also
>> > > >> >> >>>>> non-deterministic across clusters?
>> > > >> >> >>>>>
>> > > >> >> >>>>> 104. Good point on partition key.
>> > > >> >> >>>>>
>> > > >> >> >>>>> End-to-end encryption:
>> > > >> >> >>>>> 105. So, it seems end-to-end encryption is useful.
>> > Are
>> > > headers
>> > > >> >> >>>> useful there?
>> > > >> >> >>>>>
>> > > >> >> >>>>> Auditing:
>> > > >> >> >>>>> 106. It seems other than the UUID, all other
>> metadata
>> > > are per
>> > > >> >> >>>> producer?
>> > > >> >> >>>>>
>> > > >> >> >>>>> EOS:
>> > > >> >> >>>>> 107. How are those UUIDs generated? I am not sure
>> if
>> > > they can
>> > > >> be
>> > > >> >> >>>> generated
>> > > >> >> >>>>> in the producer library. An application may send
>> > > messages
>> > > >> >> through a
>> > > >> >> >>>> load
>> > > >> >> >>>>> balancer and on retry, the same message could be
>> > > routed to a
>> > > >> >> >>>> different
>> > > >> >> >>>>> producer instance. So, it seems that the
>> application
>> > > has to
>> > > >> >> >> generate
>> > > >> >> >>>> the
>> > > >> >> >>>>> UUIDs. In that case, shouldn't the application just
>> > > put the
>> > > >> >> UUID in
>> > > >> >> >>>> the
>> > > >> >> >>>>> payload?
>> > > >> >> >>>>>
>> > > >> >> >>>>> Thanks,
>> > > >> >> >>>>>
>> > > >> >> >>>>> Jun
>> > > >> >> >>>>>
>> > > >> >> >>>>>
>> > > >> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>> > > >> >> >>>> Michael.Pearce@ig.com>
>> > > >> >> >>>>> wrote:
>> > > >> >> >>>>>
>> > > >> >> >>>>> > Hi Jun.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Per Transaction Tracing / Data Lineage.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > As Stated in the KIP this has the first use case
>> of
>> > > how many
>> > > >> >> APM
>> > > >> >> >>>> tools now
>> > > >> >> >>>>> > work.
>> > > >> >> >>>>> > I would find it impossible for any one to argue
>> > this
>> > > is not
>> > > >> >> >>>> important or a
>> > > >> >> >>>>> > niche market as it has its own gartner report for
>> > > this
>> > > >> space.
>> > > >> >> >> Such
>> > > >> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace,
>> > > Hawqular are
>> > > >> >> but a
>> > > >> >> >>>> few.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Likewise these APM tools can help very rapidly
>> > track
>> > > down
>> > > >> >> issues
>> > > >> >> >>>> and
>> > > >> >> >>>>> > automatically capture metrics, perform actions
>> > based
>> > > on
>> > > >> >> >> unexpected
>> > > >> >> >>>> behavior
>> > > >> >> >>>>> > to auto recover services.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Before mentioning looking at aggregated stats, in
>> > > these
>> > > >> cases
>> > > >> >> >> where
>> > > >> >> >>>>> > actually on critical flows we cannot afford to
>> have
>> > > >> aggregated
>> > > >> >> >>>> rolled up
>> > > >> >> >>>>> > stats only.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > With the APM tool we use its actually able to
>> > detect
>> > > a
>> > > >> single
>> > > >> >> >>>> transaction
>> > > >> >> >>>>> > failure and capture the thread traces in the JVM
>> > > where it
>> > > >> >> failed
>> > > >> >> >>>> and
>> > > >> >> >>>>> > everything for us, to the point it sends us
>> alerts
>> > > where we
>> > > >> >> have
>> > > >> >> >>>> this
>> > > >> >> >>>>> > giving the line number of the code that caused
>> it,
>> > > the
>> > > >> >> >> transaction
>> > > >> >> >>>> trace
>> > > >> >> >>>>> > through all the services and endpoints
>> (supported)
>> > > upto the
>> > > >> >> point
>> > > >> >> >>>> of
>> > > >> >> >>>>> > failure, it can also capture the data in and out
>> > (so
>> > > we can
>> > > >> >> >>>> replay).
>> > > >> >> >>>>> > Because atm Kafka doesn’t support us being able
>> to
>> > > stich in
>> > > >> >> these
>> > > >> >> >>>> tracing
>> > > >> >> >>>>> > transaction ids natively, we cannot get these
>> > > benefits as
>> > > >> >> such is
>> > > >> >> >>>> limiting
>> > > >> >> >>>>> > our ability support apps and monitor them to the
>> > same
>> > > >> >> standards
>> > > >> >> >> we
>> > > >> >> >>>> come to
>> > > >> >> >>>>> > expect when on a kafka flow.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > This actually ties in with Data Lineage, as the
>> > same
>> > > tracing
>> > > >> >> can
>> > > >> >> >>>> be used
>> > > >> >> >>>>> > to back stich this. Essentially many times due to
>> > > the sums
>> > > >> of
>> > > >> >> >> money
>> > > >> >> >>>>> > involved there are disputes, and typically as a
>> > > financial
>> > > >> >> >>>> institute the
>> > > >> >> >>>>> > easiest and cleanest way to prove when disputes
>> > > arise is to
>> > > >> >> >>>> present the
>> > > >> >> >>>>> > actual flow and processes involved in a
>> > transaction.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Likewise as Hadoop matures its evident this case
>> is
>> > > >> >> important, as
>> > > >> >> >>>> tools
>> > > >> >> >>>>> > such as Atlas (Hortonworks led) and Navigator
>> > > (cloudera led)
>> > > >> >> are
>> > > >> >> >>>> evident
>> > > >> >> >>>>> > also I believe the importance here is very much
>> NOT
>> > > just a
>> > > >> >> >>>> financial issue.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > From a MDM point of view any company wanting to
>> > care
>> > > about
>> > > >> >> Data
>> > > >> >> >>>> Quality
>> > > >> >> >>>>> > and Data Governance - Data Lineage is a key piece
>> > in
>> > > this
>> > > >> >> puzzle.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > RE Mirroring,
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > As per the KIP in-fact this is exactly what we do
>> > re
>> > > cluster
>> > > >> >> id,
>> > > >> >> >>>> to mirror
>> > > >> >> >>>>> > a network of clusters between AZ’s / Regions. We
>> > > know a
>> > > >> >> >>>> transaction for a
>> > > >> >> >>>>> > key will be done within a AZ/Region, as such we
>> > > know the
>> > > >> >> write
>> > > >> >> >> to
>> > > >> >> >>>> kafka
>> > > >> >> >>>>> > would be ordered per key. But we need eventual
>> view
>> > > of that
>> > > >> >> >> across
>> > > >> >> >>>> in our
>> > > >> >> >>>>> > other regions/az’s. When we have complete AZ or
>> > > Region
>> > > >> >> failure we
>> > > >> >> >>>> know
>> > > >> >> >>>>> > there will be a brief interruption whilst those
>> > > transactions
>> > > >> >> are
>> > > >> >> >>>> moved to
>> > > >> >> >>>>> > another region but we expect after it to
>> continue.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > As mentioned having separate Partions to do this
>> > > starts to
>> > > >> get
>> > > >> >> >>>>> > ugly/complicated for us:
>> > > >> >> >>>>> > how would I do compaction where a key is in two
>> > > partitions?
>> > > >> >> >>>>> > How do we balance consumers so where multiple
>> > > partitions
>> > > >> with
>> > > >> >> the
>> > > >> >> >>>> same key
>> > > >> >> >>>>> > goto the same consumer
>> > > >> >> >>>>> > What do you do if cluster 1 has 5 partitions but
>> > > cluster 20
>> > > >> >> has
>> > > >> >> >> 10
>> > > >> >> >>>> because
>> > > >> >> >>>>> > its larger kit in our more core DC’s, as such key
>> > to
>> > > >> partition
>> > > >> >> >>>> mappings for
>> > > >> >> >>>>> > consumers get even more complicated.
>> > > >> >> >>>>> > What do you do if we add or remove a complete
>> > region
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Where as simple mirror will work we just need to
>> > > ensure we
>> > > >> >> don’t
>> > > >> >> >>>> have a
>> > > >> >> >>>>> > cycle which we can do with clusterId.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > We even have started to look at shortest path
>> > mirror
>> > > routing
>> > > >> >> >> based
>> > > >> >> >>>> on
>> > > >> >> >>>>> > clusterId, if we also had the region and az info
>> on
>> > > the
>> > > >> >> >> originating
>> > > >> >> >>>>> > message, this we have not implemented but some
>> > ideas
>> > > come
>> > > >> from
>> > > >> >> >>>> network
>> > > >> >> >>>>> > routing, and also the dispatcher router in apache
>> > > qpid.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Also we need to have data perimeters e.g. certain
>> > > data
>> > > >> cannot
>> > > >> >> >> leave
>> > > >> >> >>>>> > certain countries borders. We want this all
>> > > automated so
>> > > >> that
>> > > >> >> at
>> > > >> >> >>>> the
>> > > >> >> >>>>> > platform level without having to touch or look at
>> > the
>> > > >> business
>> > > >> >> >>>> data inside
>> > > >> >> >>>>> > we can have headers we can put tags into so that
>> we
>> > > can
>> > > >> ensure
>> > > >> >> >>>> this doesn’t
>> > > >> >> >>>>> > occur when we mirror. (actually links in to data
>> > > lineage /
>> > > >> >> >> tracing
>> > > >> >> >>>> as again
>> > > >> >> >>>>> > we need to tag messages at a platform level)
>> > > Examples are we
>> > > >> >> are
>> > > >> >> >>>> not
>> > > >> >> >>>>> > allowed Private customer details to leave
>> > > Switzerland, yet
>> > > >> we
>> > > >> >> >> need
>> > > >> >> >>>> those
>> > > >> >> >>>>> > systems integrated.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Lastly around mirroring we have a partionKey
>> field,
>> > > as the
>> > > >> key
>> > > >> >> >>>> used for
>> > > >> >> >>>>> > portioning logic != compaction key all the time
>> but
>> > > we want
>> > > >> to
>> > > >> >> >>>> preserve it
>> > > >> >> >>>>> > for when we mirror so that if source cluster
>> > > partition count
>> > > >> >> !=
>> > > >> >> >>>> destination
>> > > >> >> >>>>> > cluster partition count we can honour the same
>> > > partitioning
>> > > >> >> >> logic.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > RE End 2 End encryption
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > As I believe mentioned just before, the solution
>> > you
>> > > mention
>> > > >> >> just
>> > > >> >> >>>> doesn’t
>> > > >> >> >>>>> > cut the mustard these days with many regulators.
>> An
>> > > >> operations
>> > > >> >> >>>> person with
>> > > >> >> >>>>> > access to the box should not be able to have
>> access
>> > > to the
>> > > >> >> data.
>> > > >> >> >>>> Many now
>> > > >> >> >>>>> > actually impose quite literally the
>> implementation
>> > > expected
>> > > >> >> being
>> > > >> >> >>>> end2end
>> > > >> >> >>>>> > encryption for certain data (Singapore for us is
>> > one
>> > > that I
>> > > >> am
>> > > >> >> >>>> most aware
>> > > >> >> >>>>> > of). In fact we’re even now needing encrypt the
>> > data
>> > > and
>> > > >> store
>> > > >> >> >> the
>> > > >> >> >>>> keys in
>> > > >> >> >>>>> > HSM modules.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Likewise the performance penalty on encrypting
>> > > decrypting as
>> > > >> >> you
>> > > >> >> >>>> produce
>> > > >> >> >>>>> > over wire, then again encrypt decrypt as the data
>> > is
>> > > stored
>> > > >> on
>> > > >> >> >> the
>> > > >> >> >>>> brokers
>> > > >> >> >>>>> > disks and back again, then again encrypted and
>> > > decrypted
>> > > >> back
>> > > >> >> >> over
>> > > >> >> >>>> the wire
>> > > >> >> >>>>> > each time for each consumer all adds up, ignoring
>> > > this
>> > > >> >> doubling
>> > > >> >> >>>> with mirror
>> > > >> >> >>>>> > makers etc. simply encrypting the value once on
>> > > write by the
>> > > >> >> >>>> client and
>> > > >> >> >>>>> > again decrypting on consume by the consumer is
>> far
>> > > more
>> > > >> >> >>>> performant, but
>> > > >> >> >>>>> > then the routing and platform meta data needs to
>> be
>> > > separate
>> > > >> >> >> (thus
>> > > >> >> >>>> headers)
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > RE Auditing:
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Our Auditing needs are:
>> > > >> >> >>>>> > Producer Id,
>> > > >> >> >>>>> > Origin Cluster Id that message first produced
>> into
>> > > >> >> >>>>> > Origin AZ – agreed we can derive this if we have
>> > > cluster id,
>> > > >> >> but
>> > > >> >> >>>> it makes
>> > > >> >> >>>>> > resolving this for audit reporting a lot easier.
>> > > >> >> >>>>> > Origin Region – agreed we can derive this if we
>> > have
>> > > cluster
>> > > >> >> id,
>> > > >> >> >>>> but it
>> > > >> >> >>>>> > makes resolving this for audit reporting a lot
>> > > easier.
>> > > >> >> >>>>> > Unique Message Identification (this is not the
>> same
>> > > as
>> > > >> >> >> transaction
>> > > >> >> >>>>> > tracing) – note offset and partition are not the
>> > > same, as
>> > > >> >> when we
>> > > >> >> >>>> mirror or
>> > > >> >> >>>>> > have for what ever system failure duplicate send,
>> > > >> >> >>>>> > Custom Client wrapper version (where
>> organizations
>> > > have to
>> > > >> >> wrap
>> > > >> >> >>>> the kafka
>> > > >> >> >>>>> > client for added features) so we know what
>> version
>> > > of the
>> > > >> >> wrapper
>> > > >> >> >>>> is used
>> > > >> >> >>>>> > Producer IP address (in case of clients being in
>> > our
>> > > vm/open
>> > > >> >> >> stack
>> > > >> >> >>>> infra
>> > > >> >> >>>>> > where they can move around, producer id will stay
>> > > the same
>> > > >> but
>> > > >> >> >>>> this would
>> > > >> >> >>>>> > change)
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > RE Once and only once delivery case
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Using the same Message UUID for auditing we can
>> > > achieve this
>> > > >> >> >> quite
>> > > >> >> >>>> simply.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > As per how some other brokers do this (cough
>> qpid,
>> > > artemis)
>> > > >> >> >>>> message uuid
>> > > >> >> >>>>> > are used to dedupe where message is sent and
>> > > produced but
>> > > >> the
>> > > >> >> >>>> client didn’t
>> > > >> >> >>>>> > receive the ack, and there for replays the send,
>> by
>> > > having a
>> > > >> >> >>>> unique message
>> > > >> >> >>>>> > id per message, this can be filtered out, on
>> > > consumers where
>> > > >> >> >>>> message
>> > > >> >> >>>>> > delivery may occur twice for what ever reasons a
>> > > message
>> > > >> uuid
>> > > >> >> can
>> > > >> >> >>>> be used
>> > > >> >> >>>>> > to remove duplicates being deliverd , like wise
>> we
>> > > can do
>> > > >> >> this in
>> > > >> >> >>>> the
>> > > >> >> >>>>> > mirrormakers so if we detect a dupe message we
>> can
>> > > avoid
>> > > >> >> >>>> replicating it.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Cheers
>> > > >> >> >>>>> > Mike
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <
>> jun@confluent.io>
>> > > wrote:
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Since this KIP affects message format, wire
>> > > protocol,
>> > > >> >> apis, I
>> > > >> >> >>>> think
>> > > >> >> >>>>> > it's
>> > > >> >> >>>>> > worth spending a bit more time to nail down
>> the
>> > > concrete
>> > > >> >> use
>> > > >> >> >>>> cases. It
>> > > >> >> >>>>> > would be bad if we add this feature, but when
>> > > start
>> > > >> >> >>>> implementing it
>> > > >> >> >>>>> > for say
>> > > >> >> >>>>> > mirroring, we then realize that header is not
>> > > the best
>> > > >> >> >>>> approach.
>> > > >> >> >>>>> > Initially,
>> > > >> >> >>>>> > I thought I was convinced of the use cases of
>> > > headers
>> > > >> and
>> > > >> >> was
>> > > >> >> >>>> trying to
>> > > >> >> >>>>> > write down a few use cases to convince
>> others.
>> > > That's
>> > > >> >> when I
>> > > >> >> >>>> became
>> > > >> >> >>>>> > less
>> > > >> >> >>>>> > certain. For me to be convinced, I just want
>> to
>> > > see two
>> > > >> >> >> strong
>> > > >> >> >>>> use
>> > > >> >> >>>>> > cases
>> > > >> >> >>>>> > (instead of 10 maybe use cases) in the
>> > > third-party
>> > > >> space.
>> > > >> >> The
>> > > >> >> >>>> reason is
>> > > >> >> >>>>> > that when we discussed the use cases within a
>> > > company,
>> > > >> >> often
>> > > >> >> >>>> it ends
>> > > >> >> >>>>> > with
>> > > >> >> >>>>> > "we can't force everyone to use this standard
>> > > since we
>> > > >> may
>> > > >> >> >>>> have to
>> > > >> >> >>>>> > integrate with third-party tools".
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > At present, I am not sure why headers are
>> > useful
>> > > for
>> > > >> >> things
>> > > >> >> >>>> like
>> > > >> >> >>>>> > schemaId
>> > > >> >> >>>>> > or encryption. In order to do anything useful
>> > to
>> > > the
>> > > >> >> value,
>> > > >> >> >>>> one needs
>> > > >> >> >>>>> > to
>> > > >> >> >>>>> > know the schemaId or how data is encrypted,
>> but
>> > > header
>> > > >> is
>> > > >> >> >>>> optional.
>> > > >> >> >>>>> > But, I
>> > > >> >> >>>>> > can be convinced if someone (Radai, Sean,
>> > Todd?)
>> > > >> provides
>> > > >> >> >> more
>> > > >> >> >>>> details
>> > > >> >> >>>>> > on
>> > > >> >> >>>>> > the argument.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > I am not very sure header is the best
>> approach
>> > > for
>> > > >> >> mirroring
>> > > >> >> >>>> either. If
>> > > >> >> >>>>> > someone has thought about this more, I'd be
>> > > happy to
>> > > >> hear.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > I can see the data lineage use case. I am
>> just
>> > > not sure
>> > > >> >> how
>> > > >> >> >>>> widely
>> > > >> >> >>>>> > applicable this is. If someone familiar with
>> > > this space
>> > > >> >> can
>> > > >> >> >>>> justify
>> > > >> >> >>>>> > this is
>> > > >> >> >>>>> > a significant use case, say in the finance
>> > > industry,
>> > > >> this
>> > > >> >> >>>> would be a
>> > > >> >> >>>>> > strong
>> > > >> >> >>>>> > use case.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > I can see the auditing use case. I am just
>> not
>> > > sure if a
>> > > >> >> >> native
>> > > >> >> >>>>> > producer id
>> > > >> >> >>>>> > solves that problem. If there are additional
>> > > metadata
>> > > >> >> that's
>> > > >> >> >>>> worth
>> > > >> >> >>>>> > collecting but not covered by the producer
>> id,
>> > > that
>> > > >> would
>> > > >> >> >> make
>> > > >> >> >>>> this a
>> > > >> >> >>>>> > strong use case.
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Thanks,
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > Jun
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>> > > >> >> >>>> radai.rosenblatt@gmail.com>
>> > > >> >> >>>>> > wrote:
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > > this KIP is about enabling headers, nothing
>> > > more
>> > > >> nothing
>> > > >> >> >>>> less - so
>> > > >> >> >>>>> > no,
>> > > >> >> >>>>> > > broker-side use of headers is not in the
>> KIP
>> > > scope.
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> > > obviously though, once you have headers
>> > > potential use
>> > > >> >> cases
>> > > >> >> >>>> could
>> > > >> >> >>>>> > include
>> > > >> >> >>>>> > > broker-side header-aware interceptors
>> (which
>> > > would be
>> > > >> >> the
>> > > >> >> >>>> topic of
>> > > >> >> >>>>> > other
>> > > >> >> >>>>> > > future KIPs).
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> > > a trivially clear use case (to me) would be
>> > > using such
>> > > >> >> >>>> broker-side
>> > > >> >> >>>>> > > interceptors to enforce compliance with
>> > > organizational
>> > > >> >> >>>> policies - it
>> > > >> >> >>>>> > would
>> > > >> >> >>>>> > > make our SREs lives much easier if instead
>> of
>> > > >> >> retroactively
>> > > >> >> >>>>> > discovering
>> > > >> >> >>>>> > > "rogue" topics/users those messages would
>> > have
>> > > been
>> > > >> >> >> rejected
>> > > >> >> >>>>> > up-front.
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> > > the kafka broker code is lacking any such
>> > > >> extensibility
>> > > >> >> >>>> support
>> > > >> >> >>>>> > (beyond
>> > > >> >> >>>>> > > maybe authorizer) which is why these use
>> > cases
>> > > were
>> > > >> left
>> > > >> >> >> out
>> > > >> >> >>>> of the
>> > > >> >> >>>>> > "case
>> > > >> >> >>>>> > > for headers" doc - broker extensibility is
>> a
>> > > separate
>> > > >> >> >>>> discussion.
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen
>> > Shapira <
>> > > >> >> >>>> gwen@confluent.io>
>> > > >> >> >>>>> > wrote:
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> > > > Woah, I wasn't aware this is something
>> > we'll
>> > > do. It
>> > > >> >> >> wasn't
>> > > >> >> >>>> in the
>> > > >> >> >>>>> > KIP,
>> > > >> >> >>>>> > > > right?
>> > > >> >> >>>>> > > >
>> > > >> >> >>>>> > > > I guess we could do it the same way ACLs
>> > > currently
>> > > >> >> work.
>> > > >> >> >>>>> > > > I had in mind something that will allow
>> > > admins to
>> > > >> >> apply
>> > > >> >> >>>> rules to
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > new create/delete/config topic APIs. So
>> > Todd
>> > > can
>> > > >> >> decide
>> > > >> >> >> to
>> > > >> >> >>>> reject
>> > > >> >> >>>>> > > > "create topic" requests that ask for more
>> > > than 40
>> > > >> >> >>>> partitions, or
>> > > >> >> >>>>> > > > require exactly 3 replicas, or no more
>> than
>> > > 50GB
>> > > >> >> >> partition
>> > > >> >> >>>> size,
>> > > >> >> >>>>> > etc.
>> > > >> >> >>>>> > > >
>> > > >> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are
>> > > planning to
>> > > >> >> apply
>> > > >> >> >>>> more
>> > > >> >> >>>>> > rules
>> > > >> >> >>>>> > > > to requests (and I think we should), we
>> may
>> > > want a
>> > > >> bit
>> > > >> >> >>>> more generic
>> > > >> >> >>>>> > > > design around that.
>> > > >> >> >>>>> > > >
>> > > >> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>> > > >> >> >>>> radai.rosenblatt@gmail.com>
>> > > >> >> >>>>> > > wrote:
>> > > >> >> >>>>> > > > > "wouldn't you be in the business of
>> > making
>> > > sure
>> > > >> >> >> everyone
>> > > >> >> >>>> uses
>> > > >> >> >>>>> > them
>> > > >> >> >>>>> > > > > properly?"
>> > > >> >> >>>>> > > > >
>> > > >> >> >>>>> > > > > thats where a broker-side plugin would
>> > > come handy
>> > > >> -
>> > > >> >> any
>> > > >> >> >>>> incoming
>> > > >> >> >>>>> > > message
>> > > >> >> >>>>> > > > > that does not conform to org policy
>> (read
>> > > - does
>> > > >> not
>> > > >> >> >>>> have the
>> > > >> >> >>>>> > proper
>> > > >> >> >>>>> > > > > headers) gets thrown out (with an error
>> > > returned
>> > > >> to
>> > > >> >> >> user)
>> > > >> >> >>>>> > > > >
>> > > >> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
>> > > Palino <
>> > > >> >> >>>> tpalino@gmail.com>
>> > > >> >> >>>>> > wrote:
>> > > >> >> >>>>> > > > >
>> > > >> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on
>> > > this one
>> > > >> :)
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >> Producing counts to a topic is part of
>> > > it, but
>> > > >> >> that’s
>> > > >> >> >>>> only
>> > > >> >> >>>>> > part. So
>> > > >> >> >>>>> > > you
>> > > >> >> >>>>> > > > >> count you have 100 messages in topic
>> A.
>> > > When you
>> > > >> >> >> mirror
>> > > >> >> >>>> topic A
>> > > >> >> >>>>> > to
>> > > >> >> >>>>> > > > another
>> > > >> >> >>>>> > > > >> cluster, you have 99 messages. Where
>> was
>> > > your
>> > > >> >> problem?
>> > > >> >> >>>> Or
>> > > >> >> >>>>> > worse, you
>> > > >> >> >>>>> > > > have
>> > > >> >> >>>>> > > > >> 100 messages, but one producer
>> > duplicated
>> > > >> messages
>> > > >> >> and
>> > > >> >> >>>> another
>> > > >> >> >>>>> > one
>> > > >> >> >>>>> > > lost
>> > > >> >> >>>>> > > > >> messages. You need details about where
>> > the
>> > > >> message
>> > > >> >> >> came
>> > > >> >> >>>> from in
>> > > >> >> >>>>> > order
>> > > >> >> >>>>> > > to
>> > > >> >> >>>>> > > > >> pinpoint problems when they happen.
>> > Source
>> > > >> producer
>> > > >> >> >>>> info, where
>> > > >> >> >>>>> > it was
>> > > >> >> >>>>> > > > >> produced into your infrastructure, and
>> > > when it
>> > > >> was
>> > > >> >> >>>> produced.
>> > > >> >> >>>>> > This
>> > > >> >> >>>>> > > > requires
>> > > >> >> >>>>> > > > >> you to add the information to the
>> > message.
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >> And yes, you still need to maintain
>> your
>> > > clients.
>> > > >> >> So
>> > > >> >> >>>> maybe my
>> > > >> >> >>>>> > original
>> > > >> >> >>>>> > > > >> example was not the best. My thoughts
>> on
>> > > not
>> > > >> >> wanting
>> > > >> >> >> to
>> > > >> >> >>>> be
>> > > >> >> >>>>> > responsible
>> > > >> >> >>>>> > > > for
>> > > >> >> >>>>> > > > >> message formats stands, because that’s
>> > > very much
>> > > >> >> >>>> separate from
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > client.
>> > > >> >> >>>>> > > > >> As you know, we have our own internal
>> > > client
>> > > >> >> library
>> > > >> >> >>>> that can
>> > > >> >> >>>>> > insert
>> > > >> >> >>>>> > > the
>> > > >> >> >>>>> > > > >> right headers, and right now inserts
>> the
>> > > right
>> > > >> >> audit
>> > > >> >> >>>>> > information into
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> message fields. If they exist, and
>> > > assuming the
>> > > >> >> >> message
>> > > >> >> >>>> is Avro
>> > > >> >> >>>>> > > encoded.
>> > > >> >> >>>>> > > > >> What if someone wants to use JSON
>> > instead
>> > > for a
>> > > >> >> good
>> > > >> >> >>>> reason?
>> > > >> >> >>>>> > What if
>> > > >> >> >>>>> > > > user X
>> > > >> >> >>>>> > > > >> wants to encrypt messages, but user Y
>> > > does not?
>> > > >> >> >>>> Maintaining the
>> > > >> >> >>>>> > client
>> > > >> >> >>>>> > > > >> library is still much easier than
>> > > maintaining the
>> > > >> >> >>>> message
>> > > >> >> >>>>> > formats.
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >> -Todd
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
>> > > Shapira <
>> > > >> >> >>>> gwen@confluent.io
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> > > wrote:
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >> > Based on your last sentence,
>> consider
>> > me
>> > > >> >> convinced
>> > > >> >> >> :)
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >> > I get why headers are critical for
>> > > Mirroring
>> > > >> (you
>> > > >> >> >>>> need tags to
>> > > >> >> >>>>> > > prevent
>> > > >> >> >>>>> > > > >> > loops and sometimes to route
>> messages
>> > > to the
>> > > >> >> correct
>> > > >> >> >>>>> > destination).
>> > > >> >> >>>>> > > > >> > But why do you need headers to
>> audit?
>> > > We are
>> > > >> >> >> auditing
>> > > >> >> >>>> by
>> > > >> >> >>>>> > producing
>> > > >> >> >>>>> > > > >> > counts to a side topic (and I was
>> > under
>> > > the
>> > > >> >> >>>> impression you do
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > >> > same), so we never need to modify
>> the
>> > > message.
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >> > Another thing - after we added
>> > headers,
>> > > >> wouldn't
>> > > >> >> you
>> > > >> >> >>>> be in the
>> > > >> >> >>>>> > > > >> > business of making sure everyone
>> uses
>> > > them
>> > > >> >> properly?
>> > > >> >> >>>> Making
>> > > >> >> >>>>> > sure
>> > > >> >> >>>>> > > > >> > everyone includes the right headers
>> > you
>> > > need,
>> > > >> not
>> > > >> >> >>>> using the
>> > > >> >> >>>>> > header
>> > > >> >> >>>>> > > > >> > names you intend to use, etc. I
>> don't
>> > > think the
>> > > >> >> >>>> "policing"
>> > > >> >> >>>>> > business
>> > > >> >> >>>>> > > > >> > will ever go away.
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
>> > > Palino <
>> > > >> >> >>>>> > tpalino@gmail.com>
>> > > >> >> >>>>> > > > wrote:
>> > > >> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not
>> very
>> > > happy
>> > > >> with
>> > > >> >> the
>> > > >> >> >>>>> > workaround.
>> > > >> >> >>>>> > > Avro
>> > > >> >> >>>>> > > > >> means
>> > > >> >> >>>>> > > > >> > > that I have to be concerned with
>> the
>> > > format
>> > > >> of
>> > > >> >> the
>> > > >> >> >>>> messages
>> > > >> >> >>>>> > in
>> > > >> >> >>>>> > > > order to
>> > > >> >> >>>>> > > > >> > run
>> > > >> >> >>>>> > > > >> > > the infrastructure (audit,
>> > mirroring,
>> > > etc.).
>> > > >> >> That
>> > > >> >> >>>> means
>> > > >> >> >>>>> > that I
>> > > >> >> >>>>> > > have
>> > > >> >> >>>>> > > > to
>> > > >> >> >>>>> > > > >> > > handle the schemas, and I have to
>> > > enforce
>> > > >> rules
>> > > >> >> >>>> about good
>> > > >> >> >>>>> > > formats.
>> > > >> >> >>>>> > > > >> This
>> > > >> >> >>>>> > > > >> > is
>> > > >> >> >>>>> > > > >> > > not something I want to be in the
>> > > business
>> > > >> of,
>> > > >> >> >>>> because I
>> > > >> >> >>>>> > should be
>> > > >> >> >>>>> > > > able
>> > > >> >> >>>>> > > > >> > to
>> > > >> >> >>>>> > > > >> > > run a service infrastructure
>> without
>> > > needing
>> > > >> >> to be
>> > > >> >> >>>> in the
>> > > >> >> >>>>> > weeds of
>> > > >> >> >>>>> > > > >> > dealing
>> > > >> >> >>>>> > > > >> > > with customer data formats.
>> > > >> >> >>>>> > > > >> > >
>> > > >> >> >>>>> > > > >> > > Trust me, a sizable portion of my
>> > > support
>> > > >> time
>> > > >> >> is
>> > > >> >> >>>> spent
>> > > >> >> >>>>> > dealing
>> > > >> >> >>>>> > > with
>> > > >> >> >>>>> > > > >> > schema
>> > > >> >> >>>>> > > > >> > > issues. I really would like to get
>> > > away from
>> > > >> >> that.
>> > > >> >> >>>> Maybe
>> > > >> >> >>>>> > I'd have
>> > > >> >> >>>>> > > > more
>> > > >> >> >>>>> > > > >> > time
>> > > >> >> >>>>> > > > >> > > for other hobbies. Like writing.
>> ;)
>> > > >> >> >>>>> > > > >> > >
>> > > >> >> >>>>> > > > >> > > -Todd
>> > > >> >> >>>>> > > > >> > >
>> > > >> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM
>> Gwen
>> > > Shapira <
>> > > >> >> >>>>> > gwen@confluent.io>
>> > > >> >> >>>>> > > > wrote:
>> > > >> >> >>>>> > > > >> > >
>> > > >> >> >>>>> > > > >> > >> I'm pretty satisfied with the
>> > current
>> > > >> >> workarounds
>> > > >> >> >>>> (Avro
>> > > >> >> >>>>> > container
>> > > >> >> >>>>> > > > >> > >> format), so I'm not too excited
>> > > about the
>> > > >> >> extra
>> > > >> >> >>>> work
>> > > >> >> >>>>> > required to
>> > > >> >> >>>>> > > do
>> > > >> >> >>>>> > > > >> > >> headers in Kafka. I absolutely
>> > don't
>> > > mind it
>> > > >> >> if
>> > > >> >> >>>> you do
>> > > >> >> >>>>> > it...
>> > > >> >> >>>>> > > > >> > >> I think the Apache convention for
>> > > "good
>> > > >> idea,
>> > > >> >> but
>> > > >> >> >>>> not
>> > > >> >> >>>>> > willing to
>> > > >> >> >>>>> > > > put
>> > > >> >> >>>>> > > > >> > >> any work toward it" is +0.5?
>> > anyway,
>> > > that's
>> > > >> >> what
>> > > >> >> >> I
>> > > >> >> >>>> was
>> > > >> >> >>>>> > trying to
>> > > >> >> >>>>> > > > >> > >> convey :)
>> > > >> >> >>>>> > > > >> > >>
>> > > >> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM,
>> > Todd
>> > > Palino
>> > > >> <
>> > > >> >> >>>>> > tpalino@gmail.com>
>> > > >> >> >>>>> > > > >> wrote:
>> > > >> >> >>>>> > > > >> > >> > Well I guess my question for
>> you,
>> > > then, is
>> > > >> >> what
>> > > >> >> >>>> is
>> > > >> >> >>>>> > holding you
>> > > >> >> >>>>> > > > back
>> > > >> >> >>>>> > > > >> > from
>> > > >> >> >>>>> > > > >> > >> > full support for headers?
>> What’s
>> > > the bit
>> > > >> >> that
>> > > >> >> >>>> you’re
>> > > >> >> >>>>> > missing
>> > > >> >> >>>>> > > that
>> > > >> >> >>>>> > > > >> has
>> > > >> >> >>>>> > > > >> > you
>> > > >> >> >>>>> > > > >> > >> > under a full +1?
>> > > >> >> >>>>> > > > >> > >> >
>> > > >> >> >>>>> > > > >> > >> > -Todd
>> > > >> >> >>>>> > > > >> > >> >
>> > > >> >> >>>>> > > > >> > >> >
>> > > >> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
>> > > Gwen
>> > > >> >> Shapira <
>> > > >> >> >>>>> > > gwen@confluent.io>
>> > > >> >> >>>>> > > > >> > wrote:
>> > > >> >> >>>>> > > > >> > >> >
>> > > >> >> >>>>> > > > >> > >> >> I know why people who support
>> > > headers
>> > > >> >> support
>> > > >> >> >>>> them, and
>> > > >> >> >>>>> > I've
>> > > >> >> >>>>> > > > seen
>> > > >> >> >>>>> > > > >> > what
>> > > >> >> >>>>> > > > >> > >> >> the discussion is like.
>> > > >> >> >>>>> > > > >> > >> >>
>> > > >> >> >>>>> > > > >> > >> >> This is why I'm asking people
>> > who
>> > > are
>> > > >> >> against
>> > > >> >> >>>> headers
>> > > >> >> >>>>> > > > (especially
>> > > >> >> >>>>> > > > >> > >> >> committers) what will make
>> them
>> > > change
>> > > >> >> their
>> > > >> >> >>>> mind - so
>> > > >> >> >>>>> > we can
>> > > >> >> >>>>> > > > get
>> > > >> >> >>>>> > > > >> > this
>> > > >> >> >>>>> > > > >> > >> >> part over one way or another.
>> > > >> >> >>>>> > > > >> > >> >>
>> > > >> >> >>>>> > > > >> > >> >> If I sound frustrated it is
>> not
>> > > at Radai,
>> > > >> >> Jun
>> > > >> >> >>>> or you
>> > > >> >> >>>>> > (Todd)...
>> > > >> >> >>>>> > > > I am
>> > > >> >> >>>>> > > > >> > >> >> just looking for something
>> > > concrete we
>> > > >> can
>> > > >> >> do
>> > > >> >> >>>> to move
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > >> discussion
>> > > >> >> >>>>> > > > >> > >> >> along to the yummy design
>> > details
>> > > (which
>> > > >> is
>> > > >> >> >> the
>> > > >> >> >>>>> > argument I
>> > > >> >> >>>>> > > > really
>> > > >> >> >>>>> > > > >> am
>> > > >> >> >>>>> > > > >> > >> >> looking forward to).
>> > > >> >> >>>>> > > > >> > >> >>
>> > > >> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53
>> PM,
>> > > Todd
>> > > >> >> Palino <
>> > > >> >> >>>>> > > tpalino@gmail.com>
>> > > >> >> >>>>> > > > >> > wrote:
>> > > >> >> >>>>> > > > >> > >> >> > So, Gwen, to your question
>> > > (even though
>> > > >> >> I’m
>> > > >> >> >>>> not a
>> > > >> >> >>>>> > > > committer)...
>> > > >> >> >>>>> > > > >> > >> >> >
>> > > >> >> >>>>> > > > >> > >> >> > I have always been a strong
>> > > supporter
>> > > >> of
>> > > >> >> >>>> introducing
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > concept
>> > > >> >> >>>>> > > > >> > of an
>> > > >> >> >>>>> > > > >> > >> >> > envelope to messages, which
>> > > headers
>> > > >> >> >>>> accomplishes. The
>> > > >> >> >>>>> > > message
>> > > >> >> >>>>> > > > key
>> > > >> >> >>>>> > > > >> > is
>> > > >> >> >>>>> > > > >> > >> >> > already an example of a
>> piece
>> > of
>> > > >> envelope
>> > > >> >> >>>>> > information. By
>> > > >> >> >>>>> > > > >> > providing a
>> > > >> >> >>>>> > > > >> > >> >> means
>> > > >> >> >>>>> > > > >> > >> >> > to do this within Kafka
>> > itself,
>> > > and not
>> > > >> >> >>>> relying on
>> > > >> >> >>>>> > use-case
>> > > >> >> >>>>> > > > >> > specific
>> > > >> >> >>>>> > > > >> > >> >> > implementations, you make it
>> > > much
>> > > >> easier
>> > > >> >> for
>> > > >> >> >>>>> > components to
>> > > >> >> >>>>> > > > >> > >> interoperate.
>> > > >> >> >>>>> > > > >> > >> >> It
>> > > >> >> >>>>> > > > >> > >> >> > simplifies development of
>> all
>> > > these
>> > > >> >> things
>> > > >> >> >>>> (message
>> > > >> >> >>>>> > routing,
>> > > >> >> >>>>> > > > >> > auditing,
>> > > >> >> >>>>> > > > >> > >> >> > encryption, etc.) because
>> each
>> > > one does
>> > > >> >> not
>> > > >> >> >>>> have to
>> > > >> >> >>>>> > reinvent
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> wheel.
>> > > >> >> >>>>> > > > >> > >> >> >
>> > > >> >> >>>>> > > > >> > >> >> > It also makes it much easier
>> > > from a
>> > > >> >> client
>> > > >> >> >>>> point of
>> > > >> >> >>>>> > view if
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> headers
>> > > >> >> >>>>> > > > >> > >> >> are
>> > > >> >> >>>>> > > > >> > >> >> > defined as part of the
>> > protocol
>> > > and/or
>> > > >> >> >>>> message format
>> > > >> >> >>>>> > in
>> > > >> >> >>>>> > > > general
>> > > >> >> >>>>> > > > >> > >> because
>> > > >> >> >>>>> > > > >> > >> >> > you can easily produce and
>> > > consume
>> > > >> >> messages
>> > > >> >> >>>> without
>> > > >> >> >>>>> > having
>> > > >> >> >>>>> > > to
>> > > >> >> >>>>> > > > >> take
>> > > >> >> >>>>> > > > >> > >> into
>> > > >> >> >>>>> > > > >> > >> >> > account specific cases. For
>> > > example, I
>> > > >> >> want
>> > > >> >> >>>> to route
>> > > >> >> >>>>> > > messages,
>> > > >> >> >>>>> > > > >> but
>> > > >> >> >>>>> > > > >> > >> >> client A
>> > > >> >> >>>>> > > > >> > >> >> > doesn’t support the way
>> audit
>> > > >> implemented
>> > > >> >> >>>> headers, and
>> > > >> >> >>>>> > > client
>> > > >> >> >>>>> > > > B
>> > > >> >> >>>>> > > > >> > >> doesn’t
>> > > >> >> >>>>> > > > >> > >> >> > support the way encryption
>> or
>> > > routing
>> > > >> >> >>>> implemented
>> > > >> >> >>>>> > headers,
>> > > >> >> >>>>> > > so
>> > > >> >> >>>>> > > > now
>> > > >> >> >>>>> > > > >> > my
>> > > >> >> >>>>> > > > >> > >> >> > application has to create
>> some
>> > > really
>> > > >> >> >> fragile
>> > > >> >> >>>> (my
>> > > >> >> >>>>> > > autocorrect
>> > > >> >> >>>>> > > > >> just
>> > > >> >> >>>>> > > > >> > >> tried
>> > > >> >> >>>>> > > > >> > >> >> to
>> > > >> >> >>>>> > > > >> > >> >> > make that “tragic”, which is
>> > > probably
>> > > >> >> >>>> appropriate
>> > > >> >> >>>>> > too) code
>> > > >> >> >>>>> > > to
>> > > >> >> >>>>> > > > >> > strip
>> > > >> >> >>>>> > > > >> > >> >> > everything off, rather than
>> > just
>> > > >> >> consuming
>> > > >> >> >> the
>> > > >> >> >>>>> > messages,
>> > > >> >> >>>>> > > > picking
>> > > >> >> >>>>> > > > >> > out
>> > > >> >> >>>>> > > > >> > >> the
>> > > >> >> >>>>> > > > >> > >> >> 1
>> > > >> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested
>> > > in, and
>> > > >> >> >>>> performing its
>> > > >> >> >>>>> > > function.
>> > > >> >> >>>>> > > > >> > >> >> >
>> > > >> >> >>>>> > > > >> > >> >> > Honestly, this discussion
>> has
>> > > been
>> > > >> going
>> > > >> >> on
>> > > >> >> >>>> for a
>> > > >> >> >>>>> > long time,
>> > > >> >> >>>>> > > > and
>> > > >> >> >>>>> > > > >> > it’s
>> > > >> >> >>>>> > > > >> > >> >> > always “Oh, you came up
>> with 2
>> > > use
>> > > >> cases,
>> > > >> >> >> and
>> > > >> >> >>>> yeah,
>> > > >> >> >>>>> > those
>> > > >> >> >>>>> > > use
>> > > >> >> >>>>> > > > >> cases
>> > > >> >> >>>>> > > > >> > >> are
>> > > >> >> >>>>> > > > >> > >> >> > real things that someone
>> would
>> > > want to
>> > > >> >> do.
>> > > >> >> >>>> Here’s an
>> > > >> >> >>>>> > > alternate
>> > > >> >> >>>>> > > > >> way
>> > > >> >> >>>>> > > > >> > to
>> > > >> >> >>>>> > > > >> > >> >> > implement them so let’s not
>> do
>> > > >> headers.”
>> > > >> >> If
>> > > >> >> >>>> we have a
>> > > >> >> >>>>> > few
>> > > >> >> >>>>> > > use
>> > > >> >> >>>>> > > > >> cases
>> > > >> >> >>>>> > > > >> > >> that
>> > > >> >> >>>>> > > > >> > >> >> we
>> > > >> >> >>>>> > > > >> > >> >> > actually came up with, you
>> can
>> > > be sure
>> > > >> >> that
>> > > >> >> >>>> over the
>> > > >> >> >>>>> > next
>> > > >> >> >>>>> > > year
>> > > >> >> >>>>> > > > >> > >> there’s a
>> > > >> >> >>>>> > > > >> > >> >> > dozen others that we didn’t
>> > > think of
>> > > >> that
>> > > >> >> >>>> someone
>> > > >> >> >>>>> > would like
>> > > >> >> >>>>> > > > to
>> > > >> >> >>>>> > > > >> > do. I
>> > > >> >> >>>>> > > > >> > >> >> > really think it’s time to
>> stop
>> > > >> rehashing
>> > > >> >> >> this
>> > > >> >> >>>>> > discussion and
>> > > >> >> >>>>> > > > >> > instead
>> > > >> >> >>>>> > > > >> > >> >> focus
>> > > >> >> >>>>> > > > >> > >> >> > on a workable standard that
>> we
>> > > can
>> > > >> adopt.
>> > > >> >> >>>>> > > > >> > >> >> >
>> > > >> >> >>>>> > > > >> > >> >> > -Todd
>> > > >> >> >>>>> > > > >> > >> >> >
>> > > >> >> >>>>> > > > >> > >> >> >
>> > > >> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39
>> > PM,
>> > > Todd
>> > > >> >> Palino
>> > > >> >> >> <
>> > > >> >> >>>>> > > > tpalino@gmail.com>
>> > > >> >> >>>>> > > > >> > >> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >
>> > > >> >> >>>>> > > > >> > >> >> >> C. per message encryption
>> > > >> >> >>>>> > > > >> > >> >> >>> One drawback of this
>> > approach
>> > > is that
>> > > >> >> this
>> > > >> >> >>>>> > significantly
>> > > >> >> >>>>> > > > reduce
>> > > >> >> >>>>> > > > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> effectiveness of
>> > compression,
>> > > which
>> > > >> >> >> happens
>> > > >> >> >>>> on a
>> > > >> >> >>>>> > set of
>> > > >> >> >>>>> > > > >> > serialized
>> > > >> >> >>>>> > > > >> > >> >> >>> messages. An alternative
>> is
>> > > to enable
>> > > >> >> SSL
>> > > >> >> >>>> for wire
>> > > >> >> >>>>> > > > encryption
>> > > >> >> >>>>> > > > >> and
>> > > >> >> >>>>> > > > >> > >> rely
>> > > >> >> >>>>> > > > >> > >> >> on
>> > > >> >> >>>>> > > > >> > >> >> >>> the storage system (e.g.
>> > > LUKS) for at
>> > > >> >> rest
>> > > >> >> >>>>> > encryption.
>> > > >> >> >>>>> > > > >> > >> >> >>
>> > > >> >> >>>>> > > > >> > >> >> >>
>> > > >> >> >>>>> > > > >> > >> >> >> Jun, this is not
>> sufficient.
>> > > While
>> > > >> this
>> > > >> >> >> does
>> > > >> >> >>>> cover
>> > > >> >> >>>>> > the case
>> > > >> >> >>>>> > > > of
>> > > >> >> >>>>> > > > >> > >> removing
>> > > >> >> >>>>> > > > >> > >> >> a
>> > > >> >> >>>>> > > > >> > >> >> >> drive from the system, it
>> > will
>> > > not
>> > > >> >> satisfy
>> > > >> >> >>>> most
>> > > >> >> >>>>> > compliance
>> > > >> >> >>>>> > > > >> > >> requirements
>> > > >> >> >>>>> > > > >> > >> >> for
>> > > >> >> >>>>> > > > >> > >> >> >> encryption of data as
>> whoever
>> > > has
>> > > >> >> access to
>> > > >> >> >>>> the
>> > > >> >> >>>>> > broker
>> > > >> >> >>>>> > > itself
>> > > >> >> >>>>> > > > >> > still
>> > > >> >> >>>>> > > > >> > >> has
>> > > >> >> >>>>> > > > >> > >> >> >> access to the unencrypted
>> > > data. For
>> > > >> >> >>>> end-to-end
>> > > >> >> >>>>> > encryption
>> > > >> >> >>>>> > > you
>> > > >> >> >>>>> > > > >> > need to
>> > > >> >> >>>>> > > > >> > >> >> >> encrypt at the producer,
>> > > before it
>> > > >> >> enters
>> > > >> >> >> the
>> > > >> >> >>>>> > system, and
>> > > >> >> >>>>> > > > >> decrypt
>> > > >> >> >>>>> > > > >> > at
>> > > >> >> >>>>> > > > >> > >> the
>> > > >> >> >>>>> > > > >> > >> >> >> consumer, after it exits
>> the
>> > > system.
>> > > >> >> >>>>> > > > >> > >> >> >>
>> > > >> >> >>>>> > > > >> > >> >> >> -Todd
>> > > >> >> >>>>> > > > >> > >> >> >>
>> > > >> >> >>>>> > > > >> > >> >> >>
>> > > >> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03
>> > > PM, radai
>> > > >> <
>> > > >> >> >>>>> > > > >> radai.rosenblatt@gmail.com
>> > > >> >> >>>>> > > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>
>> > > >> >> >>>>> > > > >> > >> >> >>> another big plus of
>> headers
>> > > in the
>> > > >> >> >> protocol
>> > > >> >> >>>> is that
>> > > >> >> >>>>> > it
>> > > >> >> >>>>> > > would
>> > > >> >> >>>>> > > > >> > enable
>> > > >> >> >>>>> > > > >> > >> >> rapid
>> > > >> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside
>> > of
>> > > core
>> > > >> >> kafka
>> > > >> >> >>>> and would
>> > > >> >> >>>>> > reduce
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> number of
>> > > >> >> >>>>> > > > >> > >> >> >>> future wire format changes
>> > > required.
>> > > >> >> >>>>> > > > >> > >> >> >>>
>> > > >> >> >>>>> > > > >> > >> >> >>> a lot of what is
>> currently a
>> > > KIP
>> > > >> >> >> represents
>> > > >> >> >>>> use
>> > > >> >> >>>>> > cases that
>> > > >> >> >>>>> > > > are
>> > > >> >> >>>>> > > > >> > not
>> > > >> >> >>>>> > > > >> > >> 100%
>> > > >> >> >>>>> > > > >> > >> >> >>> relevant to all users, and
>> > > some of
>> > > >> them
>> > > >> >> >>>> require
>> > > >> >> >>>>> > rather
>> > > >> >> >>>>> > > > invasive
>> > > >> >> >>>>> > > > >> > wire
>> > > >> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing
>> a
>> > > good
>> > > >> recent
>> > > >> >> >>>> example of
>> > > >> >> >>>>> > this is
>> > > >> >> >>>>> > > > >> > kip-98.
>> > > >> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is
>> > > expected to
>> > > >> be
>> > > >> >> a
>> > > >> >> >>>> very small
>> > > >> >> >>>>> > > > fraction of
>> > > >> >> >>>>> > > > >> > >> total
>> > > >> >> >>>>> > > > >> > >> >> >>> traffic and yet the
>> changes
>> > > are
>> > > >> >> invasive.
>> > > >> >> >>>>> > > > >> > >> >> >>>
>> > > >> >> >>>>> > > > >> > >> >> >>> every such wire format
>> > change
>> > > >> >> translates
>> > > >> >> >>>> into
>> > > >> >> >>>>> > painful and
>> > > >> >> >>>>> > > > slow
>> > > >> >> >>>>> > > > >> > >> >> adoption of
>> > > >> >> >>>>> > > > >> > >> >> >>> new versions.
>> > > >> >> >>>>> > > > >> > >> >> >>>
>> > > >> >> >>>>> > > > >> > >> >> >>> i think a lot of
>> > functionality
>> > > >> >> currently
>> > > >> >> >> in
>> > > >> >> >>>> KIPs
>> > > >> >> >>>>> > could be
>> > > >> >> >>>>> > > > "spun
>> > > >> >> >>>>> > > > >> > out"
>> > > >> >> >>>>> > > > >> > >> >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> implemented as opt-in
>> > plugins
>> > > >> >> transmitting
>> > > >> >> >>>> data over
>> > > >> >> >>>>> > > > headers.
>> > > >> >> >>>>> > > > >> > this
>> > > >> >> >>>>> > > > >> > >> >> would
>> > > >> >> >>>>> > > > >> > >> >> >>> keep the core wire format
>> > > stable(r),
>> > > >> >> core
>> > > >> >> >>>> codebase
>> > > >> >> >>>>> > > smaller,
>> > > >> >> >>>>> > > > and
>> > > >> >> >>>>> > > > >> > >> avoid
>> > > >> >> >>>>> > > > >> > >> >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats
>> > > sometimes
>> > > >> >> required
>> > > >> >> >>>> to prove
>> > > >> >> >>>>> > a
>> > > >> >> >>>>> > > > certain
>> > > >> >> >>>>> > > > >> > >> feature
>> > > >> >> >>>>> > > > >> > >> >> is
>> > > >> >> >>>>> > > > >> > >> >> >>> useful enough for a
>> > > wide-enough
>> > > >> >> audience
>> > > >> >> >> to
>> > > >> >> >>>> warrant
>> > > >> >> >>>>> > a wire
>> > > >> >> >>>>> > > > >> format
>> > > >> >> >>>>> > > > >> > >> >> change
>> > > >> >> >>>>> > > > >> > >> >> >>> and code complexity
>> > additions.
>> > > >> >> >>>>> > > > >> > >> >> >>>
>> > > >> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
>> > > beyond
>> > > >> >> "mere"
>> > > >> >> >>>> wire format
>> > > >> >> >>>>> > > changes
>> > > >> >> >>>>> > > > >> and
>> > > >> >> >>>>> > > > >> > im
>> > > >> >> >>>>> > > > >> > >> not
>> > > >> >> >>>>> > > > >> > >> >> >>> saying it could have been
>> > > completely
>> > > >> >> done
>> > > >> >> >>>> with
>> > > >> >> >>>>> > headers,
>> > > >> >> >>>>> > > but
>> > > >> >> >>>>> > > > >> > >> >> exactly-once
>> > > >> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
>> > > >> >> >>>>> > > > >> > >> >> >>>
>> > > >> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at
>> 11:20
>> > > AM, Gwen
>> > > >> >> >>>> Shapira <
>> > > >> >> >>>>> > > > >> gwen@confluent.io
>> > > >> >> >>>>> > > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>>
>> > > >> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
>> > > 10:24 AM,
>> > > >> >> radai <
>> > > >> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > > "For use cases within
>> an
>> > > >> >> organization,
>> > > >> >> >>>> one could
>> > > >> >> >>>>> > > always
>> > > >> >> >>>>> > > > use
>> > > >> >> >>>>> > > > >> > >> other
>> > > >> >> >>>>> > > > >> > >> >> >>> > > approaches such as
>> > > company-wise
>> > > >> >> >>>> containers"
>> > > >> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin
>> > has
>> > > >> >> >> traditionally
>> > > >> >> >>>> done
>> > > >> >> >>>>> > but there
>> > > >> >> >>>>> > > > are
>> > > >> >> >>>>> > > > >> > now
>> > > >> >> >>>>> > > > >> > >> >> cases
>> > > >> >> >>>>> > > > >> > >> >> >>> > (read
>> > > >> >> >>>>> > > > >> > >> >> >>> > > - topics) where this
>> is
>> > > not
>> > > >> >> >> acceptable.
>> > > >> >> >>>> this
>> > > >> >> >>>>> > makes
>> > > >> >> >>>>> > > > headers
>> > > >> >> >>>>> > > > >> > >> useful
>> > > >> >> >>>>> > > > >> > >> >> even
>> > > >> >> >>>>> > > > >> > >> >> >>> > > within single orgs for
>> > > cases
>> > > >> where
>> > > >> >> >>>>> > > > one-container-fits-all
>> > > >> >> >>>>> > > > >> > cannot
>> > > >> >> >>>>> > > > >> > >> >> >>> apply.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > > as for the particular
>> > use
>> > > cases
>> > > >> >> >> listed,
>> > > >> >> >>>> i dont
>> > > >> >> >>>>> > want
>> > > >> >> >>>>> > > > this to
>> > > >> >> >>>>> > > > >> > >> devolve
>> > > >> >> >>>>> > > > >> > >> >> >>> to a
>> > > >> >> >>>>> > > > >> > >> >> >>> > > discussion of
>> particular
>> > > use
>> > > >> cases
>> > > >> >> - i
>> > > >> >> >>>> think its
>> > > >> >> >>>>> > > enough
>> > > >> >> >>>>> > > > >> that
>> > > >> >> >>>>> > > > >> > >> some
>> > > >> >> >>>>> > > > >> > >> >> of
>> > > >> >> >>>>> > > > >> > >> >> >>> them
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > I think a main point of
>> > > contention
>> > > >> is
>> > > >> >> >>>> that: We
>> > > >> >> >>>>> > > identified
>> > > >> >> >>>>> > > > few
>> > > >> >> >>>>> > > > >> > >> >> >>> > use-cases where headers
>> > are
>> > > useful,
>> > > >> >> do
>> > > >> >> >> we
>> > > >> >> >>>> want
>> > > >> >> >>>>> > Kafka to
>> > > >> >> >>>>> > > > be a
>> > > >> >> >>>>> > > > >> > >> system
>> > > >> >> >>>>> > > > >> > >> >> >>> > that supports those
>> > > use-cases?
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
>> > > >> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely
>> > useful
>> > > >> >> record-level
>> > > >> >> >>>> lineage
>> > > >> >> >>>>> > is
>> > > >> >> >>>>> > > though
>> > > >> >> >>>>> > > > >> > since
>> > > >> >> >>>>> > > > >> > >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > overhead could
>> > > >> >> >>>>> > > > >> > >> >> >>> > be significant."
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports
>> > > record level
>> > > >> >> >>>> lineage. I
>> > > >> >> >>>>> > don't
>> > > >> >> >>>>> > > think
>> > > >> >> >>>>> > > > it
>> > > >> >> >>>>> > > > >> > was
>> > > >> >> >>>>> > > > >> > >> >> >>> > developed for lols, I
>> > think
>> > > it is
>> > > >> >> safe
>> > > >> >> >> to
>> > > >> >> >>>> assume
>> > > >> >> >>>>> > that
>> > > >> >> >>>>> > > the
>> > > >> >> >>>>> > > > NSA
>> > > >> >> >>>>> > > > >> > >> needed
>> > > >> >> >>>>> > > > >> > >> >> >>> > that functionality. We
>> > also
>> > > know
>> > > >> that
>> > > >> >> >>>> certain
>> > > >> >> >>>>> > financial
>> > > >> >> >>>>> > > > >> > institutes
>> > > >> >> >>>>> > > > >> > >> >> >>> > need to track tampering
>> > with
>> > > >> records
>> > > >> >> at
>> > > >> >> >> a
>> > > >> >> >>>> record
>> > > >> >> >>>>> > level
>> > > >> >> >>>>> > > and
>> > > >> >> >>>>> > > > >> > there
>> > > >> >> >>>>> > > > >> > >> are
>> > > >> >> >>>>> > > > >> > >> >> >>> > federal regulations that
>> > > absolutely
>> > > >> >> >>>> require
>> > > >> >> >>>>> > this. They
>> > > >> >> >>>>> > > > also
>> > > >> >> >>>>> > > > >> > need
>> > > >> >> >>>>> > > > >> > >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> > prove that routing apps
>> > that
>> > > >> >> "touches"
>> > > >> >> >> the
>> > > >> >> >>>>> > messages and
>> > > >> >> >>>>> > > > >> either
>> > > >> >> >>>>> > > > >> > >> reads
>> > > >> >> >>>>> > > > >> > >> >> >>> > or updates headers
>> > couldn't
>> > > have
>> > > >> >> >> possibly
>> > > >> >> >>>>> > modified the
>> > > >> >> >>>>> > > > >> payload
>> > > >> >> >>>>> > > > >> > >> >> itself.
>> > > >> >> >>>>> > > > >> > >> >> >>> > They use record level
>> > > encryption to
>> > > >> >> do
>> > > >> >> >>>> that -
>> > > >> >> >>>>> > apps can
>> > > >> >> >>>>> > > > read
>> > > >> >> >>>>> > > > >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify
>> headers
>> > > but
>> > > >> can't
>> > > >> >> >>>> touch the
>> > > >> >> >>>>> > payload.
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > We can totally say
>> "those
>> > > are
>> > > >> corner
>> > > >> >> >>>> cases and
>> > > >> >> >>>>> > not worth
>> > > >> >> >>>>> > > > >> adding
>> > > >> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for",
>> > they
>> > > should
>> > > >> >> use a
>> > > >> >> >>>> different
>> > > >> >> >>>>> > > pubsub
>> > > >> >> >>>>> > > > >> > message
>> > > >> >> >>>>> > > > >> > >> for
>> > > >> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the
>> > > other 1000
>> > > >> >> that
>> > > >> >> >>>> cater
>> > > >> >> >>>>> > > > specifically
>> > > >> >> >>>>> > > > >> to
>> > > >> >> >>>>> > > > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > financial industry).
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > But this gets us into a
>> > > catch 22:
>> > > >> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific
>> > > use-case,
>> > > >> >> >>>> someone can
>> > > >> >> >>>>> > always
>> > > >> >> >>>>> > > say
>> > > >> >> >>>>> > > > it
>> > > >> >> >>>>> > > > >> > isn't
>> > > >> >> >>>>> > > > >> > >> >> >>> > interesting enough for
>> > > Kafka. If we
>> > > >> >> >>>> discuss more
>> > > >> >> >>>>> > general
>> > > >> >> >>>>> > > > >> > trends,
>> > > >> >> >>>>> > > > >> > >> >> >>> > others can say "well, we
>> > > are not
>> > > >> sure
>> > > >> >> >> any
>> > > >> >> >>>> of them
>> > > >> >> >>>>> > really
>> > > >> >> >>>>> > > > >> needs
>> > > >> >> >>>>> > > > >> > >> >> headers
>> > > >> >> >>>>> > > > >> > >> >> >>> > specifically. This is
>> just
>> > > hand
>> > > >> >> waving
>> > > >> >> >>>> and not
>> > > >> >> >>>>> > > > interesting.".
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > I think discussing
>> > > use-cases in
>> > > >> >> >> specifics
>> > > >> >> >>>> is super
>> > > >> >> >>>>> > > > important
>> > > >> >> >>>>> > > > >> to
>> > > >> >> >>>>> > > > >> > >> >> decide
>> > > >> >> >>>>> > > > >> > >> >> >>> > implementation details
>> for
>> > > headers
>> > > >> >> (my
>> > > >> >> >>>> use-cases
>> > > >> >> >>>>> > lean
>> > > >> >> >>>>> > > > toward
>> > > >> >> >>>>> > > > >> > >> >> numerical
>> > > >> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and
>> > > object
>> > > >> >> values,
>> > > >> >> >>>> others
>> > > >> >> >>>>> > differ),
>> > > >> >> >>>>> > > > but I
>> > > >> >> >>>>> > > > >> > >> think
>> > > >> >> >>>>> > > > >> > >> >> we
>> > > >> >> >>>>> > > > >> > >> >> >>> > need to answer the
>> general
>> > > "Are we
>> > > >> >> going
>> > > >> >> >>>> to have
>> > > >> >> >>>>> > > headers"
>> > > >> >> >>>>> > > > >> > question
>> > > >> >> >>>>> > > > >> > >> >> >>> > first.
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from
>> the
>> > > other
>> > > >> >> >>>> committers in the
>> > > >> >> >>>>> > > > discussion:
>> > > >> >> >>>>> > > > >> > >> >> >>> > What would it take to
>> > > convince you
>> > > >> >> that
>> > > >> >> >>>> headers
>> > > >> >> >>>>> > in Kafka
>> > > >> >> >>>>> > > > are
>> > > >> >> >>>>> > > > >> a
>> > > >> >> >>>>> > > > >> > >> good
>> > > >> >> >>>>> > > > >> > >> >> >>> > idea in general, so we
>> can
>> > > move
>> > > >> ahead
>> > > >> >> >> and
>> > > >> >> >>>> try to
>> > > >> >> >>>>> > agree
>> > > >> >> >>>>> > > on
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> >> details?
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > I feel like we keep
>> moving
>> > > the goal
>> > > >> >> >> posts
>> > > >> >> >>>> and
>> > > >> >> >>>>> > this is
>> > > >> >> >>>>> > > > truly
>> > > >> >> >>>>> > > > >> > >> >> exhausting.
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly
>> > > support
>> > > >> >> adding
>> > > >> >> >>>> headers
>> > > >> >> >>>>> > to Kafka
>> > > >> >> >>>>> > > > >> > (+0.5?).
>> > > >> >> >>>>> > > > >> > >> >> >>> > The community can
>> continue
>> > > to find
>> > > >> >> >>>> workarounds to
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > issue
>> > > >> >> >>>>> > > > >> and
>> > > >> >> >>>>> > > > >> > >> there
>> > > >> >> >>>>> > > > >> > >> >> >>> > are some benefits to
>> > > keeping the
>> > > >> >> message
>> > > >> >> >>>> format
>> > > >> >> >>>>> > and
>> > > >> >> >>>>> > > > clients
>> > > >> >> >>>>> > > > >> > >> simpler.
>> > > >> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness
>> > of
>> > > headers
>> > > >> >> to
>> > > >> >> >>>> many
>> > > >> >> >>>>> > use-cases
>> > > >> >> >>>>> > > and
>> > > >> >> >>>>> > > > if
>> > > >> >> >>>>> > > > >> we
>> > > >> >> >>>>> > > > >> > >> can
>> > > >> >> >>>>> > > > >> > >> >> >>> > find a good and
>> generally
>> > > useful
>> > > >> way
>> > > >> >> to
>> > > >> >> >>>> add it to
>> > > >> >> >>>>> > Kafka,
>> > > >> >> >>>>> > > > it
>> > > >> >> >>>>> > > > >> > will
>> > > >> >> >>>>> > > > >> > >> make
>> > > >> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for
>> > > many -
>> > > >> worthy
>> > > >> >> >>>> goal in my
>> > > >> >> >>>>> > eyes.
>> > > >> >> >>>>> > > > >> > >> >> >>> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > > are
>> > interesting/feasible,
>> > > but:
>> > > >> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are
>> > > use cases
>> > > >> >> for
>> > > >> >> >>>> polyglot
>> > > >> >> >>>>> > topics.
>> > > >> >> >>>>> > > > >> > >> especially if
>> > > >> >> >>>>> > > > >> > >> >> >>> kafka
>> > > >> >> >>>>> > > > >> > >> >> >>> > > is being used to
>> "trunk"
>> > > >> something
>> > > >> >> >> else.
>> > > >> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics
>> would
>> > > make it
>> > > >> >> >> harder
>> > > >> >> >>>> to write
>> > > >> >> >>>>> > > > portable
>> > > >> >> >>>>> > > > >> > >> consumer
>> > > >> >> >>>>> > > > >> > >> >> >>> code.
>> > > >> >> >>>>> > > > >> > >> >> >>> > > partition remapping
>> > would
>> > > mess
>> > > >> with
>> > > >> >> >>>> locality of
>> > > >> >> >>>>> > > > consumption
>> > > >> >> >>>>> > > > >> > >> >> >>> guarantees.
>> > > >> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see
>> > for
>> > > >> >> >>>> lineage/metadata is
>> > > >> >> >>>>> > > > >> > >> billing/chargeback.
>> > > >> >> >>>>> > > > >> > >> >> for
>> > > >> >> >>>>> > > > >> > >> >> >>> > that
>> > > >> >> >>>>> > > > >> > >> >> >>> > > use case it is not
>> > enough
>> > > to
>> > > >> simply
>> > > >> >> >>>> record the
>> > > >> >> >>>>> > point
>> > > >> >> >>>>> > > of
>> > > >> >> >>>>> > > > >> > origin,
>> > > >> >> >>>>> > > > >> > >> but
>> > > >> >> >>>>> > > > >> > >> >> >>> every
>> > > >> >> >>>>> > > > >> > >> >> >>> > > replication stop
>> (think
>> > > mirror
>> > > >> >> maker)
>> > > >> >> >>>> must also
>> > > >> >> >>>>> > add a
>> > > >> >> >>>>> > > > >> record
>> > > >> >> >>>>> > > > >> > to
>> > > >> >> >>>>> > > > >> > >> >> form a
>> > > >> >> >>>>> > > > >> > >> >> >>> > > "transit log".
>> > > >> >> >>>>> > > > >> > >> >> >>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > > as for stream
>> processing
>> > > on top
>> > > >> of
>> > > >> >> >>>> kafka - i
>> > > >> >> >>>>> > know
>> > > >> >> >>>>> > > samza
>> > > >> >> >>>>> > > > >> has a
>> > > >> >> >>>>> > > > >> > >> >> metadata
>> > > >> >> >>>>> > > > >> > >> >> >>> > map
>> > > >> >> >>>>> > > > >> > >> >> >>> > > which they carry
>> around
>> > in
>> > > >> >> addition to
>> > > >> >> >>>> user
>> > > >> >> >>>>> > values.
>> > > >> >> >>>>> > > > headers
>> > > >> >> >>>>> > > > >> > are
>> > > >> >> >>>>> > > > >> > >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > perfect
>> > > >> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016
>> at
>> > > 6:50 PM,
>> > > >> >> Jun
>> > > >> >> >>>> Rao <
>> > > >> >> >>>>> > > > jun@confluent.io
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >> > >> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> In order to answer
>> the
>> > > first two
>> > > >> >> >>>> questions, it
>> > > >> >> >>>>> > would
>> > > >> >> >>>>> > > be
>> > > >> >> >>>>> > > > >> > helpful
>> > > >> >> >>>>> > > > >> > >> >> if we
>> > > >> >> >>>>> > > > >> > >> >> >>> > could
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2
>> strong
>> > > use cases
>> > > >> >> for
>> > > >> >> >>>> headers
>> > > >> >> >>>>> > in the
>> > > >> >> >>>>> > > > space
>> > > >> >> >>>>> > > > >> > for
>> > > >> >> >>>>> > > > >> > >> >> >>> > third-party
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use
>> cases
>> > > within an
>> > > >> >> >>>> organization,
>> > > >> >> >>>>> > one
>> > > >> >> >>>>> > > > could
>> > > >> >> >>>>> > > > >> > always
>> > > >> >> >>>>> > > > >> > >> >> use
>> > > >> >> >>>>> > > > >> > >> >> >>> > other
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> approaches such as
>> > > company-wise
>> > > >> >> >>>> containers to
>> > > >> >> >>>>> > get
>> > > >> >> >>>>> > > > around
>> > > >> >> >>>>> > > > >> w/o
>> > > >> >> >>>>> > > > >> > >> >> >>> headers. I
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> went through the use
>> > > cases in
>> > > >> the
>> > > >> >> KIP
>> > > >> >> >>>> and in
>> > > >> >> >>>>> > Radai's
>> > > >> >> >>>>> > > > wiki
>> > > >> >> >>>>> > > > >> (
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > https://cwiki.apache.org/confl
>> > > >> >> >>>>> > uence/display/KAFKA/A+
>> > > >> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> ).
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> The following are the
>> > > ones that
>> > > >> >> that
>> > > >> >> >> I
>> > > >> >> >>>>> > understand and
>> > > >> >> >>>>> > > > >> could
>> > > >> >> >>>>> > > > >> > be
>> > > >> >> >>>>> > > > >> > >> in
>> > > >> >> >>>>> > > > >> > >> >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> third-party use case
>> > > category.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> It seems that in
>> > general,
>> > > >> >> >> content-type
>> > > >> >> >>>> should
>> > > >> >> >>>>> > be set
>> > > >> >> >>>>> > > at
>> > > >> >> >>>>> > > > >> the
>> > > >> >> >>>>> > > > >> > >> topic
>> > > >> >> >>>>> > > > >> > >> >> >>> level.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing
>> > > messages with
>> > > >> >> >>>> different
>> > > >> >> >>>>> > content
>> > > >> >> >>>>> > > > types
>> > > >> >> >>>>> > > > >> > >> should be
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Since the value is
>> > mostly
>> > > >> useless
>> > > >> >> >>>> without
>> > > >> >> >>>>> > schema id,
>> > > >> >> >>>>> > > it
>> > > >> >> >>>>> > > > >> > seems
>> > > >> >> >>>>> > > > >> > >> that
>> > > >> >> >>>>> > > > >> > >> >> >>> > storing
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> the schema id
>> together
>> > > with
>> > > >> >> >> serialized
>> > > >> >> >>>> bytes
>> > > >> >> >>>>> > in the
>> > > >> >> >>>>> > > > value
>> > > >> >> >>>>> > > > >> is
>> > > >> >> >>>>> > > > >> > >> >> better?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> C. per message
>> > encryption
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this
>> > > approach is
>> > > >> >> that
>> > > >> >> >>>> this
>> > > >> >> >>>>> > > > significantly
>> > > >> >> >>>>> > > > >> > reduce
>> > > >> >> >>>>> > > > >> > >> >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of
>> > > compression,
>> > > >> >> which
>> > > >> >> >>>> happens on
>> > > >> >> >>>>> > a set
>> > > >> >> >>>>> > > of
>> > > >> >> >>>>> > > > >> > >> serialized
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> messages. An
>> > alternative
>> > > is to
>> > > >> >> enable
>> > > >> >> >>>> SSL for
>> > > >> >> >>>>> > wire
>> > > >> >> >>>>> > > > >> > encryption
>> > > >> >> >>>>> > > > >> > >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> rely
>> > > >> >> >>>>> > > > >> > >> >> >>> > on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> the storage system
>> > (e.g.
>> > > LUKS)
>> > > >> >> for at
>> > > >> >> >>>> rest
>> > > >> >> >>>>> > > encryption.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for
>> > > mirroring
>> > > >> across
>> > > >> >> >>>> Kafka
>> > > >> >> >>>>> > clusters
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> This is actually
>> > > interesting.
>> > > >> >> Today,
>> > > >> >> >>>> to avoid
>> > > >> >> >>>>> > > > introducing
>> > > >> >> >>>>> > > > >> > >> cycles
>> > > >> >> >>>>> > > > >> > >> >> when
>> > > >> >> >>>>> > > > >> > >> >> >>> > doing
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data
>> > > centers,
>> > > >> one
>> > > >> >> >>>> would
>> > > >> >> >>>>> > either have
>> > > >> >> >>>>> > > to
>> > > >> >> >>>>> > > > >> set
>> > > >> >> >>>>> > > > >> > up
>> > > >> >> >>>>> > > > >> > >> two
>> > > >> >> >>>>> > > > >> > >> >> >>> Kafka
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and
>> > an
>> > > >> >> aggregate)
>> > > >> >> >>>> per data
>> > > >> >> >>>>> > center
>> > > >> >> >>>>> > > or
>> > > >> >> >>>>> > > > >> > rename
>> > > >> >> >>>>> > > > >> > >> >> topics.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal.
>> With
>> > > headers,
>> > > >> >> the
>> > > >> >> >>>> producer
>> > > >> >> >>>>> > could
>> > > >> >> >>>>> > > tag
>> > > >> >> >>>>> > > > >> each
>> > > >> >> >>>>> > > > >> > >> >> message
>> > > >> >> >>>>> > > > >> > >> >> >>> with
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster
>> > ID
>> > > in the
>> > > >> >> >> header.
>> > > >> >> >>>>> > MirrorMaker
>> > > >> >> >>>>> > > > could
>> > > >> >> >>>>> > > > >> > then
>> > > >> >> >>>>> > > > >> > >> >> avoid
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages
>> to a
>> > > cluster
>> > > >> if
>> > > >> >> >>>> they are
>> > > >> >> >>>>> > tagged
>> > > >> >> >>>>> > > with
>> > > >> >> >>>>> > > > >> the
>> > > >> >> >>>>> > > > >> > >> same
>> > > >> >> >>>>> > > > >> > >> >> >>> cluster
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> id.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> However, an
>> alternative
>> > > approach
>> > > >> >> is
>> > > >> >> >> to
>> > > >> >> >>>>> > introduce sth
>> > > >> >> >>>>> > > > like
>> > > >> >> >>>>> > > > >> > >> >> >>> hierarchical
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> topic and store
>> > messages
>> > > from
>> > > >> >> >> different
>> > > >> >> >>>>> > clusters in
>> > > >> >> >>>>> > > > >> > different
>> > > >> >> >>>>> > > > >> > >> >> >>> partitions
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> under the same topic.
>> > > This
>> > > >> >> approach
>> > > >> >> >>>> avoids
>> > > >> >> >>>>> > filtering
>> > > >> >> >>>>> > > > out
>> > > >> >> >>>>> > > > >> > >> unneeded
>> > > >> >> >>>>> > > > >> > >> >> >>> data
>> > > >> >> >>>>> > > > >> > >> >> >>> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> makes offset
>> preserving
>> > > easier
>> > > >> to
>> > > >> >> >>>> support. It
>> > > >> >> >>>>> > may
>> > > >> >> >>>>> > > make
>> > > >> >> >>>>> > > > >> > >> compaction
>> > > >> >> >>>>> > > > >> > >> >> >>> > trickier
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> though since the same
>> > > key may
>> > > >> >> show up
>> > > >> >> >>>> in
>> > > >> >> >>>>> > different
>> > > >> >> >>>>> > > > >> > partitions.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> E. record-level
>> lineage
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> For example, a source
>> > > connector
>> > > >> >> could
>> > > >> >> >>>> store in
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > message
>> > > >> >> >>>>> > > > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> metadata
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the
>> > source
>> > > >> record.
>> > > >> >> >>>> Similarly,
>> > > >> >> >>>>> > if a
>> > > >> >> >>>>> > > > stream
>> > > >> >> >>>>> > > > >> job
>> > > >> >> >>>>> > > > >> > >> >> >>> transforms
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A
>> > to
>> > > topic
>> > > >> B,
>> > > >> >> the
>> > > >> >> >>>> library
>> > > >> >> >>>>> > could
>> > > >> >> >>>>> > > > >> include
>> > > >> >> >>>>> > > > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> source
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> message offset in
>> each
>> > > of the
>> > > >> >> >>>> transformed
>> > > >> >> >>>>> > message in
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> header.
>> > > >> >> >>>>> > > > >> > >> >> Not
>> > > >> >> >>>>> > > > >> > >> >> >>> > sure
>> > > >> >> >>>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> how widely useful
>> > > record-level
>> > > >> >> >> lineage
>> > > >> >> >>>> is
>> > > >> >> >>>>> > though
>> > > >> >> >>>>> > > since
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> >> overhead
>> > > >> >> >>>>> > > > >> > >> >> >>> > could
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> be significant.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> We could put things
>> > like
>> > > >> >> >>>> clientId/host/user in
>> > > >> >> >>>>> > the
>> > > >> >> >>>>> > > > header
>> > > >> >> >>>>> > > > >> in
>> > > >> >> >>>>> > > > >> > >> each
>> > > >> >> >>>>> > > > >> > >> >> >>> > message
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These
>> > > metadata are
>> > > >> >> >>>> really at the
>> > > >> >> >>>>> > > producer
>> > > >> >> >>>>> > > > >> > level
>> > > >> >> >>>>> > > > >> > >> >> though.
>> > > >> >> >>>>> > > > >> > >> >> >>> > So, a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is
>> > to
>> > > only
>> > > >> >> >> include a
>> > > >> >> >>>>> > "producerId"
>> > > >> >> >>>>> > > > per
>> > > >> >> >>>>> > > > >> > >> message
>> > > >> >> >>>>> > > > >> > >> >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> > send
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> the producerId ->
>> > > metadata
>> > > >> mapping
>> > > >> >> >>>>> > independently.
>> > > >> >> >>>>> > > > KIP-98
>> > > >> >> >>>>> > > > >> is
>> > > >> >> >>>>> > > > >> > >> >> actually
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> proposing including
>> > such
>> > > a
>> > > >> >> producerId
>> > > >> >> >>>> natively
>> > > >> >> >>>>> > in the
>> > > >> >> >>>>> > > > >> > message.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not
>> sure
>> > > that I
>> > > >> am
>> > > >> >> >> fully
>> > > >> >> >>>>> > convinced of
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > strong
>> > > >> >> >>>>> > > > >> > >> >> >>> > third-party
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers
>> > yet.
>> > > >> Perhaps
>> > > >> >> we
>> > > >> >> >>>> could
>> > > >> >> >>>>> > discuss a
>> > > >> >> >>>>> > > > bit
>> > > >> >> >>>>> > > > >> > more
>> > > >> >> >>>>> > > > >> > >> to
>> > > >> >> >>>>> > > > >> > >> >> make
>> > > >> >> >>>>> > > > >> > >> >> >>> > one
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> or two really
>> > convincing
>> > > use
>> > > >> >> cases.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal
>> > > question is
>> > > >> >> >>>> whether header
>> > > >> >> >>>>> > should
>> > > >> >> >>>>> > > > be
>> > > >> >> >>>>> > > > >> > >> exposed
>> > > >> >> >>>>> > > > >> > >> >> in
>> > > >> >> >>>>> > > > >> > >> >> >>> > stream
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> processing systems
>> such
>> > > Kafka
>> > > >> >> stream,
>> > > >> >> >>>> Samza,
>> > > >> >> >>>>> > and
>> > > >> >> >>>>> > > Spark
>> > > >> >> >>>>> > > > >> > >> streaming.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Currently, those
>> > systems
>> > > just
>> > > >> deal
>> > > >> >> >> with
>> > > >> >> >>>>> > key/value
>> > > >> >> >>>>> > > > pairs.
>> > > >> >> >>>>> > > > >> > >> Should we
>> > > >> >> >>>>> > > > >> > >> >> >>> > expose a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> third thing header
>> > there
>> > > too or
>> > > >> >> >>>> somehow map
>> > > >> >> >>>>> > header to
>> > > >> >> >>>>> > > > key
>> > > >> >> >>>>> > > > >> or
>> > > >> >> >>>>> > > > >> > >> >> value?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Jun
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016
>> at
>> > > 3:35 AM,
>> > > >> >> >>>> Michael
>> > > >> >> >>>>> > Pearce <
>> > > >> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that
>> after
>> > a
>> > > period
>> > > >> >> of a
>> > > >> >> >>>> week,
>> > > >> >> >>>>> > that there
>> > > >> >> >>>>> > > > is
>> > > >> >> >>>>> > > > >> no
>> > > >> >> >>>>> > > > >> > >> >> concerns
>> > > >> >> >>>>> > > > >> > >> >> >>> now
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and
>> 2
>> > > and now
>> > > >> we
>> > > >> >> >> have
>> > > >> >> >>>>> > agreement that
>> > > >> >> >>>>> > > > >> > headers
>> > > >> >> >>>>> > > > >> > >> are
>> > > >> >> >>>>> > > > >> > >> >> >>> useful
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
>> > > such if
>> > > >> put
>> > > >> >> to
>> > > >> >> >> a
>> > > >> >> >>>> KIP
>> > > >> >> >>>>> > vote, this
>> > > >> >> >>>>> > > > >> > wouldn’t
>> > > >> >> >>>>> > > > >> > >> be
>> > > >> >> >>>>> > > > >> > >> >> a
>> > > >> >> >>>>> > > > >> > >> >> >>> > reason
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > reject.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > @
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point
>> 4).
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose
>> > of
>> > > getting
>> > > >> >> this
>> > > >> >> >>>> KIP
>> > > >> >> >>>>> > moving past
>> > > >> >> >>>>> > > > >> this,
>> > > >> >> >>>>> > > > >> > we
>> > > >> >> >>>>> > > > >> > >> can
>> > > >> >> >>>>> > > > >> > >> >> >>> state
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4
>> bytes
>> > > space
>> > > >> that
>> > > >> >> >> can
>> > > >> >> >>>> will be
>> > > >> >> >>>>> > > > naturally
>> > > >> >> >>>>> > > > >> > >> >> interpreted
>> > > >> >> >>>>> > > > >> > >> >> >>> as
>> > > >> >> >>>>> > > > >> > >> >> >>> > an
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if
>> namespacing
>> > > is later
>> > > >> >> >>>> wanted you can
>> > > >> >> >>>>> > > easily
>> > > >> >> >>>>> > > > >> split
>> > > >> >> >>>>> > > > >> > >> this
>> > > >> >> >>>>> > > > >> > >> >> >>> into
>> > > >> >> >>>>> > > > >> > >> >> >>> > two
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from
>> > > the wire
>> > > >> >> >> protocol
>> > > >> >> >>>>> > > implementation
>> > > >> >> >>>>> > > > >> this
>> > > >> >> >>>>> > > > >> > >> makes
>> > > >> >> >>>>> > > > >> > >> >> no
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t
>> > > believe. Is
>> > > >> >> this
>> > > >> >> >>>>> > reasonable to
>> > > >> >> >>>>> > > > all?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point
>> 4
>> > > therefor
>> > > >> >> happy
>> > > >> >> >>>> we keep
>> > > >> >> >>>>> > with 32
>> > > >> >> >>>>> > > > >> bits.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016,
>> > 20:34, "
>> > > >> >> >>>>> > ignacio.solis@gmail.com on
>> > > >> >> >>>>> > > > >> behalf
>> > > >> >> >>>>> > > > >> > of
>> > > >> >> >>>>> > > > >> > >> >> >>> Ignacio
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
>> > > >> ignacio.solis@gmail.com
>> > > >> >> on
>> > > >> >> >>>> behalf of
>> > > >> >> >>>>> > > > >> > isolis@igso.net
>> > > >> >> >>>>> > > > >> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
>> > > value as
>> > > >> >> byte[]
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int -
>> No
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String -
>> > > Reluctant
>> > > >> maybe
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe
>> the
>> > > header
>> > > >> >> system
>> > > >> >> >>>> should
>> > > >> >> >>>>> > take a
>> > > >> >> >>>>> > > > single
>> > > >> >> >>>>> > > > >> > >> int. I
>> > > >> >> >>>>> > > > >> > >> >> >>> think
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if
>> > > you want
>> > > >> to
>> > > >> >> >>>> interpret
>> > > >> >> >>>>> > this as
>> > > >> >> >>>>> > > to
>> > > >> >> >>>>> > > > >> 16bit
>> > > >> >> >>>>> > > > >> > >> >> numbers
>> > > >> >> >>>>> > > > >> > >> >> >>> in
>> > > >> >> >>>>> > > > >> > >> >> >>> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > layer
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > above go right
>> > > ahead. If
>> > > >> >> >>>> somebody wants
>> > > >> >> >>>>> > to
>> > > >> >> >>>>> > > argue
>> > > >> >> >>>>> > > > >> for
>> > > >> >> >>>>> > > > >> > 16
>> > > >> >> >>>>> > > > >> > >> >> bits
>> > > >> >> >>>>> > > > >> > >> >> >>> or
>> > > >> >> >>>>> > > > >> > >> >> >>> > 64
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > bits of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > header key
>> space
>> > I
>> > > would
>> > > >> >> >> listen.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the
>> key
>> > > space
>> > > >> into
>> > > >> >> >>>> sub_key_1 and
>> > > >> >> >>>>> > > > sub_key_2
>> > > >> >> >>>>> > > > >> > >> makes no
>> > > >> >> >>>>> > > > >> > >> >> >>> > sense to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > me at
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this layer.
>> Are
>> > > we going
>> > > >> to
>> > > >> >> >>>> start
>> > > >> >> >>>>> > providing
>> > > >> >> >>>>> > > > APIs to
>> > > >> >> >>>>> > > > >> > get
>> > > >> >> >>>>> > > > >> > >> all
>> > > >> >> >>>>> > > > >> > >> >> >>> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or
>> > all
>> > > the
>> > > >> >> >>>> sub_key_2s? If
>> > > >> >> >>>>> > there is
>> > > >> >> >>>>> > > > no
>> > > >> >> >>>>> > > > >> > >> >> >>> distinguishing
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > functions
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > that are
>> applied
>> > > to each
>> > > >> one
>> > > >> >> >>>> then they
>> > > >> >> >>>>> > should
>> > > >> >> >>>>> > > be
>> > > >> >> >>>>> > > > a
>> > > >> >> >>>>> > > > >> > single
>> > > >> >> >>>>> > > > >> > >> >> >>> value.
>> > > >> >> >>>>> > > > >> > >> >> >>> > At
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're
>> > > doing is
>> > > >> >> >>>> equality.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > If the above
>> > layer
>> > > wants
>> > > >> to
>> > > >> >> >>>> interpret
>> > > >> >> >>>>> > this as
>> > > >> >> >>>>> > > 2,
>> > > >> >> >>>>> > > > 3
>> > > >> >> >>>>> > > > >> or
>> > > >> >> >>>>> > > > >> > >> more
>> > > >> >> >>>>> > > > >> > >> >> >>> values
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > that's a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > different
>> > > question. I
>> > > >> >> >>>> personally think
>> > > >> >> >>>>> > it's
>> > > >> >> >>>>> > > all
>> > > >> >> >>>>> > > > one
>> > > >> >> >>>>> > > > >> > >> >> keyspace
>> > > >> >> >>>>> > > > >> > >> >> >>> > that is
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > getting
>> assigned
>> > > using
>> > > >> some
>> > > >> >> >>>> structure,
>> > > >> >> >>>>> > but if
>> > > >> >> >>>>> > > you
>> > > >> >> >>>>> > > > >> > want to
>> > > >> >> >>>>> > > > >> > >> >> >>> > sub-assign
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > parts
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > of it then
>> that's
>> > > fine.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > The same
>> > discussion
>> > > >> applies
>> > > >> >> to
>> > > >> >> >>>> strings.
>> > > >> >> >>>>> > If
>> > > >> >> >>>>> > > > somebody
>> > > >> >> >>>>> > > > >> > >> argued
>> > > >> >> >>>>> > > > >> > >> >> for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > strings,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > would we be
>> > > arguing to
>> > > >> >> divide
>> > > >> >> >> the
>> > > >> >> >>>>> > strings with
>> > > >> >> >>>>> > > > dots
>> > > >> >> >>>>> > > > >> > ('.')
>> > > >> >> >>>>> > > > >> > >> >> as a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we want
>> > them
>> > > to give
>> > > >> >> us
>> > > >> >> >> the
>> > > >> >> >>>>> > different
>> > > >> >> >>>>> > > name
>> > > >> >> >>>>> > > > >> > segments
>> > > >> >> >>>>> > > > >> > >> >> >>> > separately?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we be
>> > > performing any
>> > > >> >> >>>> actions on
>> > > >> >> >>>>> > this key
>> > > >> >> >>>>> > > > other
>> > > >> >> >>>>> > > > >> > than
>> > > >> >> >>>>> > > > >> > >> >> >>> > matching?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
>> > > 2016 at
>> > > >> 9:30
>> > > >> >> >> AM,
>> > > >> >> >>>> Michael
>> > > >> >> >>>>> > > Pearce <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >>
>> Michael.Pearce@ig.com
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
>> > > concerns
>> > > >> on
>> > > >> >> 1
>> > > >> >> >>>> and 2
>> > > >> >> >>>>> > still?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > @all
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > To get this
>> > > moving
>> > > >> along a
>> > > >> >> >> bit
>> > > >> >> >>>> more
>> > > >> >> >>>>> > I'd also
>> > > >> >> >>>>> > > > like
>> > > >> >> >>>>> > > > >> to
>> > > >> >> >>>>> > > > >> > >> ask
>> > > >> >> >>>>> > > > >> > >> >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> get
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the below
>> last
>> > > points:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe
>> > > we're all
>> > > >> >> >> roughly
>> > > >> >> >>>> happy
>> > > >> >> >>>>> > with the
>> > > >> >> >>>>> > > > >> header
>> > > >> >> >>>>> > > > >> > >> value
>> > > >> >> >>>>> > > > >> > >> >> >>> > being a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe
>> > > consensus
>> > > >> has
>> > > >> >> >>>> been for an
>> > > >> >> >>>>> > > > namespace
>> > > >> >> >>>>> > > > >> > based
>> > > >> >> >>>>> > > > >> > >> int
>> > > >> >> >>>>> > > > >> > >> >> >>> > approach
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for
>> > > the key.
>> > > >> Any
>> > > >> >> >>>> objections
>> > > >> >> >>>>> > if this
>> > > >> >> >>>>> > > > is
>> > > >> >> >>>>> > > > >> > what
>> > > >> >> >>>>> > > > >> > >> we
>> > > >> >> >>>>> > > > >> > >> >> go
>> > > >> >> >>>>> > > > >> > >> >> >>> > with?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have
>> > if
>> > > >> >> assumption
>> > > >> >> >> in
>> > > >> >> >>>> (4) is
>> > > >> >> >>>>> > > correct,
>> > > >> >> >>>>> > > > >> > >> {int,int}
>> > > >> >> >>>>> > > > >> > >> >> >>> keys.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Should both
>> > > int's be
>> > > >> >> int16 or
>> > > >> >> >>>> int32?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them
>> > > being
>> > > >> int16(2
>> > > >> >> >>>> bytes) as
>> > > >> >> >>>>> > combined
>> > > >> >> >>>>> > > > is
>> > > >> >> >>>>> > > > >> > space
>> > > >> >> >>>>> > > > >> > >> of
>> > > >> >> >>>>> > > > >> > >> >> >>> > 4bytes as
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > per
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > original and
>> > > gives
>> > > >> plenty
>> > > >> >> of
>> > > >> >> >>>>> > combinations for
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> foreseeable,
>> > > >> >> >>>>> > > > >> > >> >> >>> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead
>> > > small.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any
>> > > benefit in
>> > > >> >> >>>> another kip
>> > > >> >> >>>>> > call to
>> > > >> >> >>>>> > > > >> discuss
>> > > >> >> >>>>> > > > >> > >> >> these at
>> > > >> >> >>>>> > > > >> > >> >> >>> > all?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >> ______________________________
>> > > >> >> >>>>> > __________
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > From: K
>> > Burstev <
>> > > >> >> >>>> k.burstev@yandex.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
>> > > November
>> > > >> 18,
>> > > >> >> >> 2016
>> > > >> >> >>>>> > 7:07:07 AM
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > To:
>> > > >> dev@kafka.apache.org
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
>> > > [DISCUSS]
>> > > >> >> KIP-82
>> > > >> >> >>>> - Add
>> > > >> >> >>>>> > Record
>> > > >> >> >>>>> > > > Headers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > For what it
>> is
>> > > worth
>> > > >> also
>> > > >> >> i
>> > > >> >> >>>> agree. As
>> > > >> >> >>>>> > a user:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
>> > > Headers are
>> > > >> >> >>>> worthwhile
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes -
>> > > Headers should
>> > > >> >> be a
>> > > >> >> >>>> top level
>> > > >> >> >>>>> > > option
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
>> > > 21:15,
>> > > >> >> "Ignacio
>> > > >> >> >>>> Solis" <
>> > > >> >> >>>>> > > > >> isolis@igso.net
>> > > >> >> >>>>> > > > >> > >:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
>> > > Headers are
>> > > >> >> >>>> worthwhile
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
>> > > Headers
>> > > >> should
>> > > >> >> be
>> > > >> >> >> a
>> > > >> >> >>>> top
>> > > >> >> >>>>> > level
>> > > >> >> >>>>> > > option
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov
>> > > 14, 2016
>> > > >> at
>> > > >> >> >> 9:16
>> > > >> >> >>>> AM,
>> > > >> >> >>>>> > Michael
>> > > >> >> >>>>> > > > Pearce
>> > > >> >> >>>>> > > > >> <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > Michael.Pearce@ig.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
>> > > >> >> details/examples
>> > > >> >> >>>> the
>> > > >> >> >>>>> > original
>> > > >> >> >>>>> > > > proposal
>> > > >> >> >>>>> > > > >> > for
>> > > >> >> >>>>> > > > >> > >> key
>> > > >> >> >>>>> > > > >> > >> >> >>> > spacing
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> ,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > not
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> new
>> > > mentioned as per
>> > > >> >> >>>> discussion
>> > > >> >> >>>>> > namespace
>> > > >> >> >>>>> > > > >> idea.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will
>> > need
>> > > to
>> > > >> update
>> > > >> >> >> the
>> > > >> >> >>>> kip,
>> > > >> >> >>>>> > when we
>> > > >> >> >>>>> > > get
>> > > >> >> >>>>> > > > >> > >> agreement
>> > > >> >> >>>>> > > > >> > >> >> >>> this
>> > > >> >> >>>>> > > > >> > >> >> >>> > is a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > better
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach
>> > > (which
>> > > >> seems
>> > > >> >> to
>> > > >> >> >>>> be the
>> > > >> >> >>>>> > case if I
>> > > >> >> >>>>> > > > have
>> > > >> >> >>>>> > > > >> > >> >> understood
>> > > >> >> >>>>> > > > >> > >> >> >>> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > general
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling
>> in
>> > > the
>> > > >> >> >>>> conversation)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the
>> > > variable
>> > > >> ints,
>> > > >> >> at
>> > > >> >> >>>> very
>> > > >> >> >>>>> > early stage
>> > > >> >> >>>>> > > > we
>> > > >> >> >>>>> > > > >> did
>> > > >> >> >>>>> > > > >> > >> think
>> > > >> >> >>>>> > > > >> > >> >> >>> about
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this. I
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added
>> > > complexity
>> > > >> >> for
>> > > >> >> >>>> the
>> > > >> >> >>>>> > saving isn't
>> > > >> >> >>>>> > > > >> worth
>> > > >> >> >>>>> > > > >> > it.
>> > > >> >> >>>>> > > > >> > >> >> I'd
>> > > >> >> >>>>> > > > >> > >> >> >>> > rather
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> go
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want
>> to
>> > > reduce
>> > > >> >> >>>> overheads and
>> > > >> >> >>>>> > size
>> > > >> >> >>>>> > > int16
>> > > >> >> >>>>> > > > >> > (2bytes)
>> > > >> >> >>>>> > > > >> > >> >> keys
>> > > >> >> >>>>> > > > >> > >> >> >>> as
>> > > >> >> >>>>> > > > >> > >> >> >>> > it
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the
>> note
>> > > of no
>> > > >> >> >> headers,
>> > > >> >> >>>> there
>> > > >> >> >>>>> > is as
>> > > >> >> >>>>> > > per
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > kip
>> > > >> >> >>>>> > > > >> > >> as
>> > > >> >> >>>>> > > > >> > >> >> we
>> > > >> >> >>>>> > > > >> > >> >> >>> > use an
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to
>> > > denote if
>> > > >> >> headers
>> > > >> >> >>>> are
>> > > >> >> >>>>> > present or
>> > > >> >> >>>>> > > > not as
>> > > >> >> >>>>> > > > >> > such
>> > > >> >> >>>>> > > > >> > >> >> >>> > provides a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > zero
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
>> > > currently
>> > > >> if
>> > > >> >> >>>> headers are
>> > > >> >> >>>>> > not
>> > > >> >> >>>>> > > used.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think
>> as
>> > > radai
>> > > >> >> mentions
>> > > >> >> >>>> would be
>> > > >> >> >>>>> > good
>> > > >> >> >>>>> > > > first
>> > > >> >> >>>>> > > > >> > if we
>> > > >> >> >>>>> > > > >> > >> >> can
>> > > >> >> >>>>> > > > >> > >> >> >>> get
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > do
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now
>> have
>> > > general
>> > > >> >> >>>> consensus that
>> > > >> >> >>>>> > (1)
>> > > >> >> >>>>> > > > headers
>> > > >> >> >>>>> > > > >> > are
>> > > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2)
>> we
>> > > want it
>> > > >> as
>> > > >> >> a
>> > > >> >> >>>> top level
>> > > >> >> >>>>> > entity.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to
>> > > state the
>> > > >> >> >> obvious i
>> > > >> >> >>>>> > believe (1)
>> > > >> >> >>>>> > > > >> headers
>> > > >> >> >>>>> > > > >> > are
>> > > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as
>> a
>> > > top level
>> > > >> >> >>>> entity.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>> ______________________________
>> > > >> >> >>>>> > __________
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> From:
>> Roger
>> > > Hoover <
>> > > >> >> >>>>> > > roger.hoover@gmail.com
>> > > >> >> >>>>> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent:
>> > > Wednesday,
>> > > >> >> November
>> > > >> >> >>>> 9, 2016
>> > > >> >> >>>>> > 9:10:47
>> > > >> >> >>>>> > > > PM
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
>> > > >> >> dev@kafka.apache.org
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject:
>> > Re:
>> > > >> [DISCUSS]
>> > > >> >> >>>> KIP-82 - Add
>> > > >> >> >>>>> > > Record
>> > > >> >> >>>>> > > > >> > Headers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
>> > > going a
>> > > >> >> little
>> > > >> >> >>>> in the
>> > > >> >> >>>>> > weeds but
>> > > >> >> >>>>> > > > >> thanks
>> > > >> >> >>>>> > > > >> > >> for
>> > > >> >> >>>>> > > > >> > >> >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> replies
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed
>> that
>> > > a prefix
>> > > >> >> and
>> > > >> >> >>>> {int,
>> > > >> >> >>>>> > int} can
>> > > >> >> >>>>> > > be
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> same.
>> > > >> >> >>>>> > > > >> > >> >> It
>> > > >> >> >>>>> > > > >> > >> >> >>> > doesn't
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > look
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > like
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's
>> what
>> > > the KIP
>> > > >> is
>> > > >> >> >>>> saying the
>> > > >> >> >>>>> > "Open"
>> > > >> >> >>>>> > > > >> > section.
>> > > >> >> >>>>> > > > >> > >> The
>> > > >> >> >>>>> > > > >> > >> >> >>> > example
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > shows
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New
>> > > Relic and
>> > > >> >> 210002
>> > > >> >> >>>> for App
>> > > >> >> >>>>> > Dynamics
>> > > >> >> >>>>> > > > >> > implying
>> > > >> >> >>>>> > > > >> > >> >> that
>> > > >> >> >>>>> > > > >> > >> >> >>> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > New
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Relic
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > organization
>> > > will
>> > > >> have
>> > > >> >> >>>> only a
>> > > >> >> >>>>> > single
>> > > >> >> >>>>> > > > header id
>> > > >> >> >>>>> > > > >> > to
>> > > >> >> >>>>> > > > >> > >> work
>> > > >> >> >>>>> > > > >> > >> >> >>> > with. Or
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > is
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix?
>> > > The main
>> > > >> >> point
>> > > >> >> >>>> of a
>> > > >> >> >>>>> > namespace
>> > > >> >> >>>>> > > or
>> > > >> >> >>>>> > > > >> > prefix
>> > > >> >> >>>>> > > > >> > >> is
>> > > >> >> >>>>> > > > >> > >> >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> > reduce
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
>> of
>> > > config
>> > > >> >> >> mapping
>> > > >> >> >>>> or
>> > > >> >> >>>>> > > registration
>> > > >> >> >>>>> > > > >> > >> depending
>> > > >> >> >>>>> > > > >> > >> >> on
>> > > >> >> >>>>> > > > >> > >> >> >>> how
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > namespaces/prefixes
>> > > >> >> are
>> > > >> >> >>>> managed.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would
>> love
>> > > to hear
>> > > >> >> more
>> > > >> >> >>>> feedback
>> > > >> >> >>>>> > on the
>> > > >> >> >>>>> > > > >> > >> higher-level
>> > > >> >> >>>>> > > > >> > >> >> >>> > questions
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > though...
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed,
>> Nov
>> > > 9, 2016
>> > > >> at
>> > > >> >> >>>> 11:38 AM,
>> > > >> >> >>>>> > radai <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > radai.rosenblatt@gmail.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think
>> > > this
>> > > >> >> discussion
>> > > >> >> >>>> is
>> > > >> >> >>>>> > getting a
>> > > >> >> >>>>> > > bit
>> > > >> >> >>>>> > > > >> into
>> > > >> >> >>>>> > > > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> weeds on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > technical
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > implementation
>> > > >> >> details.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd
>> liek
>> > > to step
>> > > >> >> back a
>> > > >> >> >>>> minute
>> > > >> >> >>>>> > and try
>> > > >> >> >>>>> > > > and
>> > > >> >> >>>>> > > > >> > >> establish
>> > > >> >> >>>>> > > > >> > >> >> >>> > where we
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > are in
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger
>> > > picture:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > (re-wording
>> > > >> nacho's
>> > > >> >> >> last
>> > > >> >> >>>>> > paragraph)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are
>> we
>> > > all in
>> > > >> >> >>>> agreement that
>> > > >> >> >>>>> > headers
>> > > >> >> >>>>> > > > are
>> > > >> >> >>>>> > > > >> a
>> > > >> >> >>>>> > > > >> > >> >> >>> worthwhile
>> > > >> >> >>>>> > > > >> > >> >> >>> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > useful
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> addition
>> > > to have?
>> > > >> >> this
>> > > >> >> >>>> was
>> > > >> >> >>>>> > contested
>> > > >> >> >>>>> > > > early
>> > > >> >> >>>>> > > > >> on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are
>> we
>> > > all in
>> > > >> >> >>>> agreement on
>> > > >> >> >>>>> > headers
>> > > >> >> >>>>> > > as
>> > > >> >> >>>>> > > > top
>> > > >> >> >>>>> > > > >> > >> level
>> > > >> >> >>>>> > > > >> > >> >> >>> entity
>> > > >> >> >>>>> > > > >> > >> >> >>> > vs
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > headers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > squirreled-away in
>> > > >> >> V?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if
>> there
>> > > are still
>> > > >> >> >>>> concerns
>> > > >> >> >>>>> > around
>> > > >> >> >>>>> > > these
>> > > >> >> >>>>> > > > #2
>> > > >> >> >>>>> > > > >> > >> points
>> > > >> >> >>>>> > > > >> > >> >> >>> (#jay?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and
>> now
>> > > back to
>> > > >> our
>> > > >> >> >>>> normal
>> > > >> >> >>>>> > programming
>> > > >> >> >>>>> > > > ...)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints
>> > > are nice.
>> > > >> >> >> having
>> > > >> >> >>>> said
>> > > >> >> >>>>> > that, its
>> > > >> >> >>>>> > > > >> adding
>> > > >> >> >>>>> > > > >> > >> >> >>> complexity
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> (see
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>> https://github.com/addthis/
>> > > >> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>> main/java/com/clearspring/
>> > > >> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
>> > > google
>> > > >> >> result)
>> > > >> >> >>>> and would
>> > > >> >> >>>>> > require
>> > > >> >> >>>>> > > > >> anyone
>> > > >> >> >>>>> > > > >> > >> >> writing
>> > > >> >> >>>>> > > > >> > >> >> >>> > other
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clients
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python?
>> > > Go? Bash?
>> > > >> >> ;-) )
>> > > >> >> >>>> to
>> > > >> >> >>>>> > > get/implement
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > >> same,
>> > > >> >> >>>>> > > > >> > >> >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > relatively
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little
>> > > gain (int
>> > > >> vs
>> > > >> >> >>>> string is
>> > > >> >> >>>>> > order of
>> > > >> >> >>>>> > > > >> > magnitude,
>> > > >> >> >>>>> > > > >> > >> >> this
>> > > >> >> >>>>> > > > >> > >> >> >>> > isnt).
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int
>> > > namespacing vs
>> > > >> >> >> {int,
>> > > >> >> >>>> int}
>> > > >> >> >>>>> > > namespacing
>> > > >> >> >>>>> > > > >> are
>> > > >> >> >>>>> > > > >> > >> >> basically
>> > > >> >> >>>>> > > > >> > >> >> >>> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > same
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre
>> > just
>> > > >> >> namespacing
>> > > >> >> >>>> an int64
>> > > >> >> >>>>> > and
>> > > >> >> >>>>> > > > giving
>> > > >> >> >>>>> > > > >> > people
>> > > >> >> >>>>> > > > >> > >> >> while
>> > > >> >> >>>>> > > > >> > >> >> >>> > 2^32
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > ranges
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > at a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time.
>> the
>> > > part i
>> > > >> >> like
>> > > >> >> >>>> about this
>> > > >> >> >>>>> > is
>> > > >> >> >>>>> > > > letting
>> > > >> >> >>>>> > > > >> > >> people
>> > > >> >> >>>>> > > > >> > >> >> >>> have a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> large
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers
>> > > with one
>> > > >> >> >>>> registration so
>> > > >> >> >>>>> > they
>> > > >> >> >>>>> > > > dont
>> > > >> >> >>>>> > > > >> > have
>> > > >> >> >>>>> > > > >> > >> to
>> > > >> >> >>>>> > > > >> > >> >> come
>> > > >> >> >>>>> > > > >> > >> >> >>> > back
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > every
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
>> > > >> plugin/header
>> > > >> >> >>>> they want to
>> > > >> >> >>>>> > > > "reserve".
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
>> > > Nov 9,
>> > > >> 2016
>> > > >> >> at
>> > > >> >> >>>> 11:01 AM,
>> > > >> >> >>>>> > Roger
>> > > >> >> >>>>> > > > >> Hoover
>> > > >> >> >>>>> > > > >> > <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > roger.hoover@gmail.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since
>> > > some of
>> > > >> the
>> > > >> >> >>>> debate has
>> > > >> >> >>>>> > been
>> > > >> >> >>>>> > > about
>> > > >> >> >>>>> > > > >> > >> overhead +
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > wondering if we
>> > > >> >> have
>> > > >> >> >>>>> > considered a
>> > > >> >> >>>>> > > > varint
>> > > >> >> >>>>> > > > >> > >> encoding
>> > > >> >> >>>>> > > > >> > >> >> (
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>> https://developers.google.com/
>> > > >> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > encoding#varints)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the
>> > > header
>> > > >> length
>> > > >> >> >>>> field (int32
>> > > >> >> >>>>> > in the
>> > > >> >> >>>>> > > > >> > proposal)
>> > > >> >> >>>>> > > > >> > >> >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > header
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't
>> > use
>> > > >> headers,
>> > > >> >> >> the
>> > > >> >> >>>>> > overhead would
>> > > >> >> >>>>> > > > be a
>> > > >> >> >>>>> > > > >> > >> single
>> > > >> >> >>>>> > > > >> > >> >> >>> byte
>> > > >> >> >>>>> > > > >> > >> >> >>> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for each
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id <
>> > 128
>> > > would
>> > > >> >> also
>> > > >> >> >>>> need only a
>> > > >> >> >>>>> > > single
>> > > >> >> >>>>> > > > >> byte?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On
>> Wed,
>> > > Nov 9,
>> > > >> >> 2016
>> > > >> >> >> at
>> > > >> >> >>>> 6:43 AM,
>> > > >> >> >>>>> > > radai <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
>> > > radai.rosenblatt@gmail.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > @magnus - and
>> > > >> >> very
>> > > >> >> >>>> dangerous
>> > > >> >> >>>>> > (youre
>> > > >> >> >>>>> > > > >> > >> essentially
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> executing
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > arbitrary code
>> > > >> >> off
>> > > >> >> >>>> the
>> > > >> >> >>>>> > internet on
>> > > >> >> >>>>> > > > your
>> > > >> >> >>>>> > > > >> > >> servers
>> > > >> >> >>>>> > > > >> > >> >> ...
>> > > >> >> >>>>> > > > >> > >> >> >>> > bad
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > idea
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > without
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > sandbox, even
>> > > >> >> with)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as
>> > for
>> > > it
>> > > >> being
>> > > >> >> a
>> > > >> >> >>>> purely
>> > > >> >> >>>>> > > > administrative
>> > > >> >> >>>>> > > > >> > task
>> > > >> >> >>>>> > > > >> > >> - i
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> disagree.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i
>> > wish
>> > > it
>> > > >> would,
>> > > >> >> >>>> really,
>> > > >> >> >>>>> > because
>> > > >> >> >>>>> > > > then my
>> > > >> >> >>>>> > > > >> > >> earlier
>> > > >> >> >>>>> > > > >> > >> >> >>> > point on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > complexity
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>> > > remapping
>> > > >> >> >>>> process would
>> > > >> >> >>>>> > be
>> > > >> >> >>>>> > > > invalid,
>> > > >> >> >>>>> > > > >> > but
>> > > >> >> >>>>> > > > >> > >> at
>> > > >> >> >>>>> > > > >> > >> >> >>> > linkedin,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > example,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> (the
>> > > team im
>> > > >> in)
>> > > >> >> >> run
>> > > >> >> >>>> kafka
>> > > >> >> >>>>> > as a
>> > > >> >> >>>>> > > > service.
>> > > >> >> >>>>> > > > >> > we
>> > > >> >> >>>>> > > > >> > >> dont
>> > > >> >> >>>>> > > > >> > >> >> >>> > really
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > know
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > what our
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > (developing
>> > > >> >> >>>> applications
>> > > >> >> >>>>> > that use
>> > > >> >> >>>>> > > > kafka)
>> > > >> >> >>>>> > > > >> > are
>> > > >> >> >>>>> > > > >> > >> up
>> > > >> >> >>>>> > > > >> > >> >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> at
>> > > >> >> >>>>> > > > >> > >> >> >>> > any
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > given
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is
>> > very
>> > > >> possible
>> > > >> >> >>>> (given the
>> > > >> >> >>>>> > > > existance of
>> > > >> >> >>>>> > > > >> > >> headers
>> > > >> >> >>>>> > > > >> > >> >> >>> and a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> plugin
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > ecosystem) for
>> > > >> >> some
>> > > >> >> >>>>> > application to
>> > > >> >> >>>>> > > > >> "equip"
>> > > >> >> >>>>> > > > >> > >> their
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> producers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> consumers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> with
>> > > the
>> > > >> >> required
>> > > >> >> >>>> plugin
>> > > >> >> >>>>> > without us
>> > > >> >> >>>>> > > > >> > knowing.
>> > > >> >> >>>>> > > > >> > >> i
>> > > >> >> >>>>> > > > >> > >> >> dont
>> > > >> >> >>>>> > > > >> > >> >> >>> > mean
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > to imply
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> bad,
>> > i
>> > > just
>> > > >> >> want to
>> > > >> >> >>>> make the
>> > > >> >> >>>>> > point
>> > > >> >> >>>>> > > > that
>> > > >> >> >>>>> > > > >> > its
>> > > >> >> >>>>> > > > >> > >> not
>> > > >> >> >>>>> > > > >> > >> >> as
>> > > >> >> >>>>> > > > >> > >> >> >>> > simple
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> sync
>> > > across a
>> > > >> >> >>>> large-enough
>> > > >> >> >>>>> > > > organization.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On
>> > > Wed, Nov 9,
>> > > >> >> 2016
>> > > >> >> >>>> at 6:17
>> > > >> >> >>>>> > AM,
>> > > >> >> >>>>> > > > Magnus
>> > > >> >> >>>>> > > > >> > >> Edenhill
>> > > >> >> >>>>> > > > >> > >> >> <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > magnus@edenhill.se>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
>> > > think
>> > > >> there
>> > > >> >> is
>> > > >> >> >> a
>> > > >> >> >>>> piece
>> > > >> >> >>>>> > missing
>> > > >> >> >>>>> > > in
>> > > >> >> >>>>> > > > >> the
>> > > >> >> >>>>> > > > >> > >> >> Strings
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > where
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> pro-Stringers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > reason that
>> > > >> by
>> > > >> >> >>>> providing
>> > > >> >> >>>>> > unique
>> > > >> >> >>>>> > > > string
>> > > >> >> >>>>> > > > >> > >> >> >>> identifiers
>> > > >> >> >>>>> > > > >> > >> >> >>> > for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > each
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > everything
>> > > >> >> will
>> > > >> >> >>>> just
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > magically
>> > > >> work
>> > > >> >> >> for
>> > > >> >> >>>> all
>> > > >> >> >>>>> > parts of
>> > > >> >> >>>>> > > the
>> > > >> >> >>>>> > > > >> > stream
>> > > >> >> >>>>> > > > >> > >> >> >>> pipeline.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> But
>> > > the
>> > > >> >> strings
>> > > >> >> >>>> dont mean
>> > > >> >> >>>>> > > anything
>> > > >> >> >>>>> > > > by
>> > > >> >> >>>>> > > > >> > >> >> themselves,
>> > > >> >> >>>>> > > > >> > >> >> >>> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > while we
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > probably
>> > > >> >> envision
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > some
>> > > auto
>> > > >> >> plugin
>> > > >> >> >>>> loader
>> > > >> >> >>>>> > that
>> > > >> >> >>>>> > > > >> downloads,
>> > > >> >> >>>>> > > > >> > >> >> compiles,
>> > > >> >> >>>>> > > > >> > >> >> >>> > links
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > runs
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> plugins
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > on-demand
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> as
>> > > soon as
>> > > >> >> >> they're
>> > > >> >> >>>> seen by
>> > > >> >> >>>>> > a
>> > > >> >> >>>>> > > > >> consumer, I
>> > > >> >> >>>>> > > > >> > >> dont
>> > > >> >> >>>>> > > > >> > >> >> >>> really
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> see
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > something
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> so
>> > > dynamic
>> > > >> >> (and
>> > > >> >> >>>> fragile) in
>> > > >> >> >>>>> > > > practice.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> In
>> > > the real
>> > > >> >> world
>> > > >> >> >>>> an
>> > > >> >> >>>>> > application
>> > > >> >> >>>>> > > > will
>> > > >> >> >>>>> > > > >> be
>> > > >> >> >>>>> > > > >> > >> >> >>> configured
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> with
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a set
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> plugins
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> to
>> > > either
>> > > >> add
>> > > >> >> >>>> (producer)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> or
>> > > read
>> > > >> >> >> (consumer)
>> > > >> >> >>>> headers.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > This
>> > > is an
>> > > >> >> >>>> administrative
>> > > >> >> >>>>> > task
>> > > >> >> >>>>> > > > based
>> > > >> >> >>>>> > > > >> on
>> > > >> >> >>>>> > > > >> > >> what
>> > > >> >> >>>>> > > > >> > >> >> >>> > features a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > client
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> needs/provides
>> > > >> >> >> and
>> > > >> >> >>>> results
>> > > >> >> >>>>> > in
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > some
>> > > sort of
>> > > >> >> >>>> configuration
>> > > >> >> >>>>> > to
>> > > >> >> >>>>> > > > enable
>> > > >> >> >>>>> > > > >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> configure
>> > > >> >> >>>>> > > > >> > >> >> >>> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > desired
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> plugins.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > Since this
>> > > >> >> needs
>> > > >> >> >>>> to be kept
>> > > >> >> >>>>> > > > somewhat
>> > > >> >> >>>>> > > > >> in
>> > > >> >> >>>>> > > > >> > >> sync
>> > > >> >> >>>>> > > > >> > >> >> >>> across
>> > > >> >> >>>>> > > > >> > >> >> >>> > an
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > (there
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> is
>> > > no point
>> > > >> in
>> > > >> >> >>>> having
>> > > >> >> >>>>> > producers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> add
>> > > headers
>> > > >> no
>> > > >> >> >>>> consumers
>> > > >> >> >>>>> > will
>> > > >> >> >>>>> > > read,
>> > > >> >> >>>>> > > > >> and
>> > > >> >> >>>>> > > > >> > >> vice
>> > > >> >> >>>>> > > > >> > >> >> >>> versa),
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > added
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > complexity
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> of
>> > > assigning
>> > > >> >> an
>> > > >> >> >> id
>> > > >> >> >>>>> > namespace
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> for
>> > > each
>> > > >> >> plugin
>> > > >> >> >> as
>> > > >> >> >>>> it is
>> > > >> >> >>>>> > being
>> > > >> >> >>>>> > > > >> > configured
>> > > >> >> >>>>> > > > >> > >> >> should
>> > > >> >> >>>>> > > > >> > >> >> >>> be
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > /Magnus
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > 2016-11-09
>> > > >> >> 13:06
>> > > >> >> >>>> GMT+01:00
>> > > >> >> >>>>> > > Michael
>> > > >> >> >>>>> > > > >> > Pearce <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> > > Michael.Pearce@ig.com>:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > Just
>> > > >> >> >>>> following/catching
>> > > >> >> >>>>> > up on
>> > > >> >> >>>>> > > > what
>> > > >> >> >>>>> > > > >> > seems
>> > > >> >> >>>>> > > > >> > >> to
>> > > >> >> >>>>> > > > >> > >> >> be
>> > > >> >> >>>>> > > > >> > >> >> >>> an
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > active
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > @Radai
>> > > >> >> sorry if
>> > > >> >> >>>> it may
>> > > >> >> >>>>> > seem
>> > > >> >> >>>>> > > > obvious
>> > > >> >> >>>>> > > > >> > but
>> > > >> >> >>>>> > > > >> > >> what
>> > > >> >> >>>>> > > > >> > >> >> >>> does
>> > > >> >> >>>>> > > > >> > >> >> >>> > MD
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > stand
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > for?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > My
>> > > take on
>> > > >> >> >>>> String vs Int:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > I
>> > > will
>> > > >> state
>> > > >> >> >>>> first I am
>> > > >> >> >>>>> > pro Int
>> > > >> >> >>>>> > > > (16
>> > > >> >> >>>>> > > > >> or
>> > > >> >> >>>>> > > > >> > >> 32).
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > I
>> > > do
>> > > >> though
>> > > >> >> >>>> playing
>> > > >> >> >>>>> > devils
>> > > >> >> >>>>> > > > advocate
>> > > >> >> >>>>> > > > >> > see a
>> > > >> >> >>>>> > > > >> > >> >> big
>> > > >> >> >>>>> > > > >> > >> >> >>> plus
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > with the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> argument
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > String
>> > > >> keys,
>> > > >> >> >>>> this is
>> > > >> >> >>>>> > around
>> > > >> >> >>>>> > > > >> > integrating
>> > > >> >> >>>>> > > > >> > >> >> into an
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > existing
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > eco-system.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > As
>> > > many
>> > > >> >> other
>> > > >> >> >>>> systems use
>> > > >> >> >>>>> > > String
>> > > >> >> >>>>> > > > >> based
>> > > >> >> >>>>> > > > >> > >> >> headers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > it
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > much
>> > > >> easier
>> > > >> >> for
>> > > >> >> >>>> these to
>> > > >> >> >>>>> > be
>> > > >> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > into.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > How with
>> > > >> Int
>> > > >> >> >>>> based
>> > > >> >> >>>>> > headers
>> > > >> >> >>>>> > > could
>> > > >> >> >>>>> > > > we
>> > > >> >> >>>>> > > > >> > >> provide
>> > > >> >> >>>>> > > > >> > >> >> a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> integration
>> > > >> >> >>>> simple /
>> > > >> >> >>>>> > easy with
>> > > >> >> >>>>> > > > >> > transition
>> > > >> >> >>>>> > > > >> > >> >> flows
>> > > >> >> >>>>> > > > >> > >> >> >>> > over
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > *
>> > > tough
>> > > >> luck
>> > > >> >> >>>> buddy
>> > > >> >> >>>>> > you're on
>> > > >> >> >>>>> > > your
>> > > >> >> >>>>> > > > >> own
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > *
>> > > simply
>> > > >> >> hash
>> > > >> >> >>>> the string
>> > > >> >> >>>>> > into
>> > > >> >> >>>>> > > int
>> > > >> >> >>>>> > > > >> code
>> > > >> >> >>>>> > > > >> > >> and
>> > > >> >> >>>>> > > > >> > >> >> hope
>> > > >> >> >>>>> > > > >> > >> >> >>> > for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> no
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > convert
>> > > >> back
>> > > >> >> >>>> though?)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > *
>> > > http2
>> > > >> >> style
>> > > >> >> >> as
>> > > >> >> >>>>> > mentioned by
>> > > >> >> >>>>> > > > nacho.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > cheers,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > Mike
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > ______________________________
>> > > >> >> >>>>> > > > >> > __________
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > From:
>> > > >> radai
>> > > >> >> <
>> > > >> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > Sent:
>> > > >> >> >> Wednesday,
>> > > >> >> >>>>> > November 9,
>> > > >> >> >>>>> > > 2016
>> > > >> >> >>>>> > > > >> > 8:12 AM
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > To:
>> > > >> >> >>>> dev@kafka.apache.org
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > Subject:
>> > > >> Re:
>> > > >> >> >>>> [DISCUSS]
>> > > >> >> >>>>> > KIP-82 -
>> > > >> >> >>>>> > > > Add
>> > > >> >> >>>>> > > > >> > >> Record
>> > > >> >> >>>>> > > > >> > >> >> >>> Headers
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > thinking
>> > > >> >> about
>> > > >> >> >>>> it some
>> > > >> >> >>>>> > more,
>> > > >> >> >>>>> > > the
>> > > >> >> >>>>> > > > >> best
>> > > >> >> >>>>> > > > >> > >> way to
>> > > >> >> >>>>> > > > >> > >> >> >>> > transmit
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > remapping
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > data to
>> > > >> >> >>>> consumers would
>> > > >> >> >>>>> > be to
>> > > >> >> >>>>> > > > put it
>> > > >> >> >>>>> > > > >> > in
>> > > >> >> >>>>> > > > >> > >> the
>> > > >> >> >>>>> > > > >> > >> >> MD
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> response
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> maybe
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > it
>> > > should
>> > > >> be
>> > > >> >> >>>> discussed
>> > > >> >> >>>>> > now.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > On
>> > > Wed,
>> > > >> Nov
>> > > >> >> 9,
>> > > >> >> >>>> 2016 at
>> > > >> >> >>>>> > 12:09
>> > > >> >> >>>>> > > AM,
>> > > >> >> >>>>> > > > >> > radai <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > >> >> >> radai.rosenblatt@gmail.com
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > im not
>> > > >> >> >> opposed
>> > > >> >> >>>> to the
>> > > >> >> >>>>> > idea of
>> > > >> >> >>>>> > > > >> > namespace
>> > > >> >> >>>>> > > > >> > >> >> >>> mapping.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> all
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > im
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > saying
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> that
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > its
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > not part
>> > > >> >> of
>> > > >> >> >>>> the "mvp"
>> > > >> >> >>>>> > and,
>> > > >> >> >>>>> > > > since
>> > > >> >> >>>>> > > > >> it
>> > > >> >> >>>>> > > > >> > >> >> requires
>> > > >> >> >>>>> > > > >> > >> >> >>> no
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> wire
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > format
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > always
>> > > >> be
>> > > >> >> >>>> added later.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > also,
>> > > >> its
>> > > >> >> not
>> > > >> >> >>>> as
>> > > >> >> >>>>> > simple as
>> > > >> >> >>>>> > > just
>> > > >> >> >>>>> > > > >> > >> >> configuring
>> > > >> >> >>>>> > > > >> > >> >> >>> MM
>> > > >> >> >>>>> > > > >> > >> >> >>> > to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> do
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > transform:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > lets
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > say i've
>> > > >> >> >>>> implemented
>> > > >> >> >>>>> > large
>> > > >> >> >>>>> > > > message
>> > > >> >> >>>>> > > > >> > >> >> support as
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > mirror
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > target
>> > > >> >> >> cluster
>> > > >> >> >>>> its been
>> > > >> >> >>>>> > > > remapped
>> > > >> >> >>>>> > > > >> to
>> > > >> >> >>>>> > > > >> > >> >> {999,1}.
>> > > >> >> >>>>> > > > >> > >> >> >>> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > consumer
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > would
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > also
>> > > >> need
>> > > >> >> to
>> > > >> >> >>>> be told
>> > > >> >> >>>>> > to look
>> > > >> >> >>>>> > > > for
>> > > >> >> >>>>> > > > >> the
>> > > >> >> >>>>> > > > >> > >> large
>> > > >> >> >>>>> > > > >> > >> >> >>> > message
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > "part X
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > header
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > under
>> > > >> >> >> {999,1}.
>> > > >> >> >>>> doable,
>> > > >> >> >>>>> > but
>> > > >> >> >>>>> > > > tricky.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > On Tue,
>> > > >> >> Nov
>> > > >> >> >> 8,
>> > > >> >> >>>> 2016 at
>> > > >> >> >>>>> > 10:29
>> > > >> >> >>>>> > > > PM,
>> > > >> >> >>>>> > > > >> > Gwen
>> > > >> >> >>>>> > > > >> > >> >> >>> Shapira <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> > > gwen@confluent.io
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > While
>> > > >> you
>> > > >> >> >> can
>> > > >> >> >>>> do
>> > > >> >> >>>>> > whatever
>> > > >> >> >>>>> > > you
>> > > >> >> >>>>> > > > >> want
>> > > >> >> >>>>> > > > >> > >> with a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> namespace
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > your
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > what
>> > > >> I'd
>> > > >> >> >>>> expect is
>> > > >> >> >>>>> > for each
>> > > >> >> >>>>> > > > app
>> > > >> >> >>>>> > > > >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> namespaces
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
>> configurable...
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > So if I
>> > > >> >> >>>> accidentally
>> > > >> >> >>>>> > used
>> > > >> >> >>>>> > > 666
>> > > >> >> >>>>> > > > for
>> > > >> >> >>>>> > > > >> > my
>> > > >> >> >>>>> > > > >> > >> HR
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> department,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > still
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > run
>> > > >> >> >> RadaiApp,
>> > > >> >> >>>> I can
>> > > >> >> >>>>> > config
>> > > >> >> >>>>> > > > >> > >> "namespace=42"
>> > > >> >> >>>>> > > > >> > >> >> >>> for
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > everything
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > will
>> > > >> look
>> > > >> >> >>>> normal.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > This
>> > > >> >> means
>> > > >> >> >>>> you only
>> > > >> >> >>>>> > need to
>> > > >> >> >>>>> > > > sync
>> > > >> >> >>>>> > > > >> > usage
>> > > >> >> >>>>> > > > >> > >> >> >>> inside
>> > > >> >> >>>>> > > > >> > >> >> >>> > your
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > own
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > organization.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > Still
>> > > >> >> hard,
>> > > >> >> >>>> but
>> > > >> >> >>>>> > somewhat
>> > > >> >> >>>>> > > > easier
>> > > >> >> >>>>> > > > >> > than
>> > > >> >> >>>>> > > > >> > >> >> syncing
>> > > >> >> >>>>> > > > >> > >> >> >>> > with
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > entire
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> world.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > On Tue,
>> > > >> >> Nov
>> > > >> >> >>>> 8, 2016
>> > > >> >> >>>>> > at 10:07
>> > > >> >> >>>>> > > > PM,
>> > > >> >> >>>>> > > > >> > >> radai <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> >>>> radai.rosenblatt@gmail.com>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > and
>> > > >> we
>> > > >> >> can
>> > > >> >> >>>> start
>> > > >> >> >>>>> > with
>> > > >> >> >>>>> > > > >> {namespace,
>> > > >> >> >>>>> > > > >> > >> id}
>> > > >> >> >>>>> > > > >> > >> >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> no
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > always
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > add
>> > > >> it
>> > > >> >> >>>> later on
>> > > >> >> >>>>> > if/when
>> > > >> >> >>>>> > > > >> > collisions
>> > > >> >> >>>>> > > > >> > >> >> >>> actually
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > dont
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > they'd
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > be
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > a
>> > > >> >> >> problem).
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > every
>> > > >> >> >>>> interested
>> > > >> >> >>>>> > party (so
>> > > >> >> >>>>> > > > orgs
>> > > >> >> >>>>> > > > >> > or
>> > > >> >> >>>>> > > > >> > >> >> >>> > individuals)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > could
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > then
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > register
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >
>> > > >> prefix
>> > > >> >> (0
>> > > >> >> >> =
>> > > >> >> >>>>> > reserved, 1 =
>> > > >> >> >>>>> > > > >> > confluent
>> > > >> >> >>>>> > > > >> > >> ...
>> > > >> >> >>>>> > > > >> > >> >> >>> 666
>> > > >> >> >>>>> > > > >> > >> >> >>> > = me
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > whatever
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > with
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > the
>> > > >> >> 2nd ID
>> > > >> >> >>>> - so once
>> > > >> >> >>>>> > > > linkedin
>> > > >> >> >>>>> > > > >> > >> >> registers,
>> > > >> >> >>>>> > > > >> > >> >> >>> say
>> > > >> >> >>>>> > > > >> > >> >> >>> > 3,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > then
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> are
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > free
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > to
>> > > >> use
>> > > >> >> {3,
>> > > >> >> >>>> *} with a
>> > > >> >> >>>>> > > > reasonable
>> > > >> >> >>>>> > > > >> > >> >> >>> expectation
>> > > >> >> >>>>> > > > >> > >> >> >>> > to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > collide
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > anything
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > else.
>> > > >> >> >>>> further
>> > > >> >> >>>>> > partitioning
>> > > >> >> >>>>> > > > of
>> > > >> >> >>>>> > > > >> > that *
>> > > >> >> >>>>> > > > >> > >> >> >>> becomes
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> problem,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >
>> > > >> >> "upstream
>> > > >> >> >>>>> > registration"
>> > > >> >> >>>>> > > of a
>> > > >> >> >>>>> > > > >> > >> namespace
>> > > >> >> >>>>> > > > >> > >> >> >>> only
>> > > >> >> >>>>> > > > >> > >> >> >>> > has
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > happen
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > > On
>> > > >> Tue,
>> > > >> >> >> Nov
>> > > >> >> >>>> 8, 2016
>> > > >> >> >>>>> > at
>> > > >> >> >>>>> > > 9:03
>> > > >> >> >>>>> > > > PM,
>> > > >> >> >>>>> > > > >> > >> James
>> > > >> >> >>>>> > > > >> > >> >> >>> Cheng <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > > >> >> wushujames@gmail.com
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> > On
>> > > >> >> Nov
>> > > >> >> >>>> 8, 2016,
>> > > >> >> >>>>> > at 5:54
>> > > >> >> >>>>> > > > PM,
>> > > >> >> >>>>> > > > >> > Gwen
>> > > >> >> >>>>> > > > >> > >> >> >>> Shapira <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> > > gwen@confluent.io
>> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> Thank
>> > > >> >> >>>> you so
>> > > >> >> >>>>> > much for
>> > > >> >> >>>>> > > > this
>> > > >> >> >>>>> > > > >> > clear
>> > > >> >> >>>>> > > > >> > >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> fair
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > summary of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > arguments.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> I'm
>> > > >> >> in
>> > > >> >> >>>> favor of
>> > > >> >> >>>>> > ints.
>> > > >> >> >>>>> > > > Not a
>> > > >> >> >>>>> > > > >> > >> >> >>> deal-breaker,
>> > > >> >> >>>>> > > > >> > >> >> >>> > but
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > in
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> Even
>> > > >> >> >>>> more in
>> > > >> >> >>>>> > favor of
>> > > >> >> >>>>> > > > >> Magnus's
>> > > >> >> >>>>> > > > >> > >> >> >>> > decentralized
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > Roger's
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> tweak:
>> > > >> >> >>>> add a
>> > > >> >> >>>>> > namespace
>> > > >> >> >>>>> > > > for
>> > > >> >> >>>>> > > > >> > >> headers.
>> > > >> >> >>>>> > > > >> > >> >> >>> This
>> > > >> >> >>>>> > > > >> > >> >> >>> > will
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > allow
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > each
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > just
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> use
>> > > >> >> >>>> whatever IDs
>> > > >> >> >>>>> > it
>> > > >> >> >>>>> > > wants
>> > > >> >> >>>>> > > > >> > >> >> internally,
>> > > >> >> >>>>> > > > >> > >> >> >>> and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> then
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > let
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> deploying
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> the
>> > > >> >> app
>> > > >> >> >>>> figure
>> > > >> >> >>>>> > out an
>> > > >> >> >>>>> > > > >> > available
>> > > >> >> >>>>> > > > >> > >> >> >>> namespace
>> > > >> >> >>>>> > > > >> > >> >> >>> > ID
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> live
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > in.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> > So
>> > > >> >> >>>>> > > > >> > io.confluent.schema-registry
>> > > >> >> >>>>> > > > >> > >> can
>> > > >> >> >>>>> > > > >> > >> >> be
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > namespace
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >> deployment
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> and
>> > > >> >> >> 0x57
>> > > >> >> >>>> on
>> > > >> >> >>>>> > yours, and
>> > > >> >> >>>>> > > > the
>> > > >> >> >>>>> > > > >> > poor
>> > > >> >> >>>>> > > > >> > >> guys
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > developing the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > need
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > to
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> worry
>> > > >> >> >>>> about that.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> Gwen,
>> > > >> >> if
>> > > >> >> >> I
>> > > >> >> >>>>> > understand
>> > > >> >> >>>>> > > your
>> > > >> >> >>>>> > > > >> > example
>> > > >> >> >>>>> > > > >> > >> >> >>> right, an
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > application
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> > deployer
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > might
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> decide to
>> > > >> >> >>>> use 0x01
>> > > >> >> >>>>> > in one
>> > > >> >> >>>>> > > > >> > >> deployment,
>> > > >> >> >>>>> > > > >> > >> >> and
>> > > >> >> >>>>> > > > >> > >> >> >>> > that
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > means
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > that
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > message
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> is
>> > > >> >> >> written
>> > > >> >> >>>> into the
>> > > >> >> >>>>> > > > broker, it
>> > > >> >> >>>>> > > > >> > >> will be
>> > > >> >> >>>>> > > > >> > >> >> >>> > saved on
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > broker
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > that
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> specific
>> > > >> >> >>>> namespace
>> > > >> >> >>>>> > > (0x01).
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> If
>> > > >> you
>> > > >> >> >>>> were to
>> > > >> >> >>>>> > mirror
>> > > >> >> >>>>> > > that
>> > > >> >> >>>>> > > > >> > message
>> > > >> >> >>>>> > > > >> > >> >> into
>> > > >> >> >>>>> > > > >> > >> >> >>> > another
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > would
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> accompany
>> > > >> >> >>>> the
>> > > >> >> >>>>> > message,
>> > > >> >> >>>>> > > > right?
>> > > >> >> >>>>> > > > >> > What
>> > > >> >> >>>>> > > > >> > >> if
>> > > >> >> >>>>> > > > >> > >> >> the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > in
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> other
>> > > >> >> >>>> cluster uses
>> > > >> >> >>>>> > 0x57?
>> > > >> >> >>>>> > > > They
>> > > >> >> >>>>> > > > >> > won't
>> > > >> >> >>>>> > > > >> > >> >> >>> > understand
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > each
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > other?
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> I'm
>> > > >> >> not
>> > > >> >> >>>> sure
>> > > >> >> >>>>> > that's an
>> > > >> >> >>>>> > > > >> avoidable
>> > > >> >> >>>>> > > > >> > >> >> >>> problem. I
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > think it
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > simply
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > that
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > in
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> order
>> > > >> >> to
>> > > >> >> >>>> share
>> > > >> >> >>>>> > data, you
>> > > >> >> >>>>> > > > have
>> > > >> >> >>>>> > > > >> to
>> > > >> >> >>>>> > > > >> > >> also
>> > > >> >> >>>>> > > > >> > >> >> >>> have a
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > shared
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>> understanding of
>> > > >> >> >>>>> > what the
>> > > >> >> >>>>> > > > >> > >> namespaces
>> > > >> >> >>>>> > > > >> > >> >> >>> mean.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> Which
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > sense,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> because
>> > > >> >> >> the
>> > > >> >> >>>>> > alternate
>> > > >> >> >>>>> > > > (sharing
>> > > >> >> >>>>> > > > >> > >> >> *nothing*
>> > > >> >> >>>>> > > > >> > >> >> >>> at
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> all)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > would
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > mean
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > > there
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> would
>> > > >> >> be
>> > > >> >> >>>> no way to
>> > > >> >> >>>>> > > > understand
>> > > >> >> >>>>> > > > >> > each
>> > > >> >> >>>>> > > > >> > >> >> other.
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> -James
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> Gwen
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> > On
>> > > >> >> Tue,
>> > > >> >> >>>> Nov 8,
>> > > >> >> >>>>> > 2016 at
>> > > >> >> >>>>> > > > 4:23
>> > > >> >> >>>>> > > > >> > PM,
>> > > >> >> >>>>> > > > >> > >> >> radai <
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> > > >> >> >>>> radai.rosenblatt@gmail.com
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >>
>> > > >> wrote:
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >>
>> > > >> +1
>> > > >> >> for
>> > > >> >> >>>> sean's
>> > > >> >> >>>>> > > document.
>> > > >> >> >>>>> > > > it
>> > > >> >> >>>>> > > > >> > >> covers
>> > > >> >> >>>>> > > > >> > >> >> >>> pretty
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> much
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > all
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> > > trade-offs
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > and
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >>
>> > > >> >> >> provides
>> > > >> >> >>>>> > concrete
>> > > >> >> >>>>> > > > figures
>> > > >> >> >>>>> > > > >> to
>> > > >> >> >>>>> > > > >> > >> argue
>> > > >> >> >>>>> > > > >> > >> >> >>> about
>> > > >> >> >>>>> > > > >> > >> >> >>> > :-)
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >>
>> > > >> >> >>>> (nit-picking -
>> > > >> >> >>>>> > used
>> > > >> >> >>>>> > > the
>> > > >> >> >>>>> > > > >> same
>> > > >> >> >>>>> > > > >> > >> xkcd
>> > > >> >> >>>>> > > > >> > >> >> >>> twice,
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> also
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > trove
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > has
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
>> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> > >>
>> > > >> >> superceded
>> > > >> >> >>>>> > > > >> > >> >
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >> > --
>> > > >> >> >>>>> > > > >> > Gwen Shapira
>> > > >> >> >>>>> > > > >> > Product Manager | Confluent
>> > > >> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
>> > > @gwenshap
>> > > >> >> >>>>> > > > >> > Follow us: Twitter | blog
>> > > >> >> >>>>> > > > >> >
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >> --
>> > > >> >> >>>>> > > > >> *Todd Palino*
>> > > >> >> >>>>> > > > >> Staff Site Reliability Engineer
>> > > >> >> >>>>> > > > >> Data Infrastructure Streaming
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > > >> linkedin.com/in/toddpalino
>> > > >> >> >>>>> > > > >>
>> > > >> >> >>>>> > > >
>> > > >> >> >>>>> > > >
>> > > >> >> >>>>> > > >
>> > > >> >> >>>>> > > > --
>> > > >> >> >>>>> > > > Gwen Shapira
>> > > >> >> >>>>> > > > Product Manager | Confluent
>> > > >> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> |
>> @gwenshap
>> > > >> >> >>>>> > > > Follow us: Twitter | blog
>> > > >> >> >>>>> > > >
>> > > >> >> >>>>> > >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> >
>> > > >> >> >>>>> > The information contained in this email is
>> strictly
>> > > >> >> confidential
>> > > >> >> >>>> and for
>> > > >> >> >>>>> > the use of the addressee only, unless otherwise
>> > > indicated.
>> > > >> If
>> > > >> >> you
>> > > >> >> >>>> are not
>> > > >> >> >>>>> > the intended recipient, please do not read, copy,
>> > > use or
>> > > >> >> disclose
>> > > >> >> >>>> to others
>> > > >> >> >>>>> > this message or any attachment. Please also
>> notify
>> > > the
>> > > >> sender
>> > > >> >> by
>> > > >> >> >>>> replying
>> > > >> >> >>>>> > to this email or by telephone (+44(020 7896 0011)
>> > > and then
>> > > >> >> delete
>> > > >> >> >>>> the email
>> > > >> >> >>>>> > and any copies of it. Opinions, conclusion (etc)
>> > > that do not
>> > > >> >> >>>> relate to the
>> > > >> >> >>>>> > official business of this company shall be
>> > > understood as
>> > > >> >> neither
>> > > >> >> >>>> given nor
>> > > >> >> >>>>> > endorsed by it. IG is a trading name of IG
>> Markets
>> > > Limited
>> > > >> (a
>> > > >> >> >>>> company
>> > > >> >> >>>>> > registered in England and Wales, company number
>> > > 04008957)
>> > > >> and
>> > > >> >> IG
>> > > >> >> >>>> Index
>> > > >> >> >>>>> > Limited (a company registered in England and
>> Wales,
>> > > company
>> > > >> >> >> number
>> > > >> >> >>>>> > 01190902). Registered address at Cannon Bridge
>> > > House, 25
>> > > >> >> Dowgate
>> > > >> >> >>>> Hill,
>> > > >> >> >>>>> > London EC4R 2YA. Both IG Markets Limited
>> (register
>> > > number
>> > > >> >> 195355)
>> > > >> >> >>>> and IG
>> > > >> >> >>>>> > Index Limited (register number 114059) are
>> > > authorised and
>> > > >> >> >>>> regulated by the
>> > > >> >> >>>>> > Financial Conduct Authority.
>> > > >> >> >>>>> >
>> > > >> >> >>>>>
>> > > >> >> >>>>>
>> > > >> >> >>>>> The information contained in this email is strictly
>> > > confidential
>> > > >> and
>> > > >> >> >> for
>> > > >> >> >>>> the use of the addressee only, unless otherwise
>> indicated.
>> > > If you
>> > > >> are
>> > > >> >> >> not
>> > > >> >> >>>> the intended recipient, please do not read, copy, use or
>> > > disclose
>> > > >> to
>> > > >> >> >> others
>> > > >> >> >>>> this message or any attachment. Please also notify the
>> > > sender by
>> > > >> >> >> replying
>> > > >> >> >>>> to this email or by telephone (+44(020 7896 0011) and
>> then
>> > > delete
>> > > >> the
>> > > >> >> >> email
>> > > >> >> >>>> and any copies of it. Opinions, conclusion (etc) that do
>> > > not relate
>> > > >> >> to
>> > > >> >> >> the
>> > > >> >> >>>> official business of this company shall be understood as
>> > > neither
>> > > >> >> given
>> > > >> >> >> nor
>> > > >> >> >>>> endorsed by it. IG is a trading name of IG Markets
>> Limited
>> > > (a
>> > > >> company
>> > > >> >> >>>> registered in England and Wales, company number
>> 04008957)
>> > > and IG
>> > > >> >> Index
>> > > >> >> >>>> Limited (a company registered in England and Wales,
>> > company
>> > > number
>> > > >> >> >>>> 01190902). Registered address at Cannon Bridge House, 25
>> > > Dowgate
>> > > >> >> Hill,
>> > > >> >> >>>> London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > 195355)
>> > > >> >> and IG
>> > > >> >> >>>> Index Limited (register number 114059) are authorised
>> and
>> > > regulated
>> > > >> >> by
>> > > >> >> >> the
>> > > >> >> >>>> Financial Conduct Authority.
>> > > >> >> >>>>>
>> > > >> >> >>>>
>> > > >> >> >>>>
>> > > >> >> >>>
>> > > >> >> >>
>> > > >> >> >>
>> > > >> >> >
>> > > >> >>
>> > > >> >>
>> > > >> >
>> > > >> The information contained in this email is strictly confidential
>> > > and for
>> > > >> the use of the addressee only, unless otherwise indicated. If
>> you
>> > > are not
>> > > >> the intended recipient, please do not read, copy, use or
>> disclose
>> > > to others
>> > > >> this message or any attachment. Please also notify the sender by
>> > > replying
>> > > >> to this email or by telephone (+44(020 7896 0011) and then
>> delete
>> > > the email
>> > > >> and any copies of it. Opinions, conclusion (etc) that do not
>> > relate
>> > > to the
>> > > >> official business of this company shall be understood as neither
>> > > given nor
>> > > >> endorsed by it. IG is a trading name of IG Markets Limited (a
>> > > company
>> > > >> registered in England and Wales, company number 04008957) and IG
>> > > Index
>> > > >> Limited (a company registered in England and Wales, company
>> number
>> > > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> > > Hill,
>> > > >> London EC4R 2YA. Both IG Markets Limited (register number
>> 195355)
>> > > and IG
>> > > >> Index Limited (register number 114059) are authorised and
>> > regulated
>> > > by the
>> > > >> Financial Conduct Authority.
>> > > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Gwen Shapira
>> > > Product Manager | Confluent
>> > > 650.450.2760 | @gwenshap
>> > > Follow us: Twitter | blog
>> > >
>> > >
>> > > The information contained in this email is strictly confidential and
>> for
>> > > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > > the intended recipient, please do not read, copy, use or disclose to
>> > others
>> > > this message or any attachment. Please also notify the sender by
>> replying
>> > > to this email or by telephone (+44(020 7896 0011) and then delete the
>> > email
>> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> > the
>> > > official business of this company shall be understood as neither given
>> > nor
>> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > > registered in England and Wales, company number 04008957) and IG Index
>> > > Limited (a company registered in England and Wales, company number
>> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
>> IG
>> > > Index Limited (register number 114059) are authorised and regulated by
>> > the
>> > > Financial Conduct Authority.
>> > >
>> >
>>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
☺ Good we’re back onto the fun stuff and the protocol and api detail ☺
On 1)
From a heavy JMS/AMQP background I like the way JMS Properties lean on this there is a host of ‘blessed’ properties (JMS, JMSX and Vendor specific prefix). And marries with you suggested option 3 in your reponse, it allows the use of String key interface, but have a precompiled dictionary headers, so if in the dictionary we can use an integer is to reduce binary wire space, else it’s not in the dictionary it is stored in binary as string. This can be easily handled with either a lead byte in the key to define if string or int, or by having two sets of header vector sets.
On 2)
This I agree, this is an implementation detail but very much important one we should note.
On 3)
Again id take inspiration from JMS, maybe we should expose methods to setStringProperty, setIntegerProperty, setBytesProperty etc.., instead of expose the internal map implementation, also for primitives/base types (larger majority of headers will be these in my experience, custom serialisation but common understanding can be done, but support more complex binary options for advanced cases. To support this think TypedProperties (first byte defines the type (int, short, long, double, binary), and then subsequent bytes are that to be de-serialised/read and returned
Cheers,
Mike
On 11/01/2017, 17:05, "Jay Kreps" <ja...@confluent.io> wrote:
Yeah I think I share a couple of concerns around the details of the
proposal beyond the use cases:
1. I think numeric keys are just a non-starter for a user-facing
interface that is meant to work across third party components (not just
within an organization). We've gone down this path of making cryptic
interfaces before based on what's easy to implement (simple consumer!). The
result is a mess we are still trying to walk away from. I think we are
solving a compression/normalization problem by having the user implement a
compression algorithm in their head and share it across their company and
between components and plugin. This is definitely easier for us, but not
the right interface! I think we should consider instead (1) just take the
perf hit and claim it's okay since it's optional (I don't necessarily buy
that since it makes a lot of common use cases pretty questionable), (2)
work on getting more efficient compression in place so we can recommend
that without reservation (e.g. if that facebook lib pans out) or (3) allow
the client to optionally pre-register headers to get a more compact
numerical mapping.
2. I think we should think about creating the Map lazily to avoid
parsing out all the headers into little objects. HashMaps themselves are
kind of expensive and the consumer is very perf sensitive so and making
gazillions of hashmaps that may or may not get used is probably a bad idea.
3. We should think about having our own type, Headers or something
rather than a Map. I kind of dislike this kind of unnecessary type in
internal code, but for this kind of public API the ability to get just the
methods that make sense and document the details of the semantic can be
worthwhile.
-Jay
On Tue, Jan 10, 2017 at 10:38 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> I'm horribly behind and outdated on this thread, but wanted to mention one
> other potential issue.
>
> The KIP talks about using simple Maps for the headers. For a lot of this
> discussion, I tend to either think of completely opaque headers (i.e just
> another byte[] blob, which seems to have been rejected based on
> interoperability grounds but I think has the nice property of making it
> easy to blame the implementer of the serde for any problems with the header
> implementation) or HTTP-style headers (seemingly the most ubiquitous
> example of headers, though certainly there are many others, and one which
> we/users may need/want to interoperate with).
>
> Strictly speaking, the keys on HTTP headers aren't unique. See
> https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
> specifically:
>
> > Multiple message-header fields with the same field-name MAY be present in
> a message if and only if the entire field-value for that header field is
> defined as a comma-separated list [i.e., #(values)]. It MUST be possible to
> combine the multiple header fields into one "field-name: field-value" pair,
> without changing the semantics of the message, by appending each subsequent
> field-value to the first, each separated by a comma. The order in which
> header fields with the same field-name are received is therefore
> significant to the interpretation of the combined field value, and thus a
> proxy MUST NOT change the order of these field values when a message is
> forwarded.
>
> The spec guarantees that you can collapse repeated headers into a single
> instance. But in practice there's a ton of stuff that doesn't. And every
> robust HTTP library I've encountered uses a multimap to represent headers.
>
> I'm not necessarily advocating for supporting this, just suggesting that we
> think about it. As you add these features and get closer to mapping to
> other systems, people will inevitably try to map them. Headers are an area
> where, if we're going to add them, it's worth considering compatibility as
> someone will inevitably come and complain that system X does Y with headers
> and we should also support Y because any decent system that provides
> headers will do so.
>
> -Ewen
>
> On Tue, Jan 10, 2017 at 5:27 PM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Hey Gwen, yes, your approach is much better than what I described and
> > solves one of my objections. I didn't think of doing it in MM rather than
> > in the original producer. I don't think it adds any overhead beyond the
> > additional header bytes if I'm not mistaken, since MM already unpacks and
> > repacks messages (unlike the broker which tries to avoid repacking and
> > hence can't efficiently add a header). I think my other objection around
> > size is likely solvable too by some kind of aliasing scheme? If you guys
> > think this can be made to work then I think we have two pretty credible
> use
> > cases with this and the tracing id family of uses, plus a number of
> things
> > that are at least moderately improved. If so, I'm convinced.
> >
> > -Jay
> >
> >
> > On Tue, Jan 10, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jay et al.
> > >
> > > On the replication using cluster id.
> > >
> > > Two options depending on how a company wants or needs to implement
> their
> > > replication as there are options of doing this as a mesh, star, or
> ring.
> > >
> > > Either as some companies enforce a provider/custom producer/consumer
> > > wrapper, it can be enforced the interceptors needed to be added, as you
> > can
> > > enforce to get support of the platform you must use company x’s
> internal
> > > wrappers.
> > > An alternative approach is to do this at the mirror maker, as Gwen
> noted
> > > we can actually make the mirror maker append this, like wise for a more
> > > complex implementation, where you can have multiple hops/routes this
> can
> > > also be the message routing.
> > >
> > > Re the size of the clusterId, yes the auto generated one is “cough”
> > > hideous in size.
> > >
> > > Most organisations though will only need a space to uniquely identify
> 127
> > > (byte) clusters, or at worst 2 bytes (short) they need to mirror and
> > > replicate between, and as such can either as platform teams generally
> > will
> > > roll out the mirror makers / replicators with the cluster can either
> > > manually adjust the clusterId on ZK, or use a mapping file all their
> > mirror
> > > makers have to map the larger id to a smaller one.
> > >
> > > Also as Gwen states this is something you should be able to opt into or
> > > not, this is the benefit of headers, if you don’t use it, it doesn’t
> use
> > up
> > > the space, its entirely optional.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > > On 09/01/2017, 19:40, "Gwen Shapira" <gw...@confluent.io> wrote:
> > >
> > > Mirroring + ClusterIDs:
> > >
> > > Kafka clusters already have an automatically generated ID that is
> > > exposed to clients:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 78%3A+Cluster+Id
> > >
> > > It sounds trivial to get MirrorMaker's consumer to grab this ID
> > (which
> > > always exists) from the source cluster and add it to the message
> > > headers.
> > >
> > > Obviously adding stuff to the message makes it larger so this
> should
> > > be a feature you should be able to turn on/off depending on the
> > > replication requirements. If you need the feature - you pay the
> > price.
> > > I don't think there is an immediate need to put an effort into
> > > shrinking the Cluster ID, but others can comment on that.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io>
> wrote:
> > > > Hey Michael,
> > > >
> > > > Sorry for lagging, I've been out for the holiday break. What
> would
> > > help me
> > > > would be to get to a consolidated list of use cases we've worked
> > out
> > > and
> > > > think add real value. I think getting down to the details really
> > > matters
> > > > for these.
> > > >
> > > > The two lists I'm aware of are the one in the KIP and the one you
> > > gave in
> > > > this email thread. Both seem to be more at the sketch level. Let
> me
> > > review
> > > > the use cases and see if I have it right. If so let's get it in
> the
> > > design
> > > > proposal.
> > > >
> > > > Your use cases:
> > > >
> > > > 1. Tracing IDs -- I get this use case and think I see how it
> > > would work.
> > > > I think because the tracing is from a third party system there
> > is
> > > some
> > > > value to having a header versus a custom format to make this
> > work
> > > fully
> > > > transparently.
> > > > 2. Mirroring -- I'm a bit unsure how this will work, I'll
> > explain
> > > my
> > > > thinking below
> > > > 3. End-to-end encryption -- I think what you are saying is
> that
> > > if you
> > > > want "routing' fields that are accessible in transit without
> > > decrypting you
> > > > need a scheme to leave this decrypted.I think the idea is to
> > avoid
> > > > decrypting to get a header, but I'm not sure if I get how this
> > is
> > > better
> > > > than a custom message format that doesn't encrypt the things
> you
> > > don't want
> > > > encrypted since you have to be aware of the scheme anyway.
> > > > 4. EOS -- I'm not convinced this is a good approach for EOS.
> > > >
> > > > Linked from KIP:
> > > >
> > > > 1. Schema-Registry-aware avro serialization -- not clear how
> > this
> > > is
> > > > better than keeping the schema id in the value
> > > > 2. Polyglot kafka topics -- marginally better than a container
> > > format,
> > > > but of course you still have to all agree on the the header
> and
> > > > header=>serialization mapping so not sure if it's really
> better
> > > than a
> > > > container format. Also not wild about polyglot topics :-)
> > > > 3. End-to-End routing trail including mirroring --- I'd put
> this
> > > under
> > > > tracing ids
> > > > 4. Cross-service process tracing -- ditto
> > > > 5. Signing -- I like this use case but it seems to depend on a
> > > feature
> > > > we don't have and aren't proposing adding, right?
> > > > 6. Per-record custom compression -- why do you want this?
> > > compressing
> > > > per message won't work well compared to batch compression,
> > right?
> > > > 7. Large Message Support -- I think we can agree that although
> > > this
> > > > might work it's clearly not the right way to support large
> > > messages in Kafka
> > > >
> > > > Matthias
> > > >
> > > > 1. Control messages --- Matthias, not sure if I get the
> > > motivation in
> > > > streams for this. If we did want to do it, wouldn't it be
> easier
> > > to
> > > > implement and by allowing users to send control messages (in
> the
> > > EOS sense)
> > > > and adding a consumer config to allow either passing these on
> or
> > > > suppressing them?
> > > >
> > > > So I if I were to summarize I'd list the use cases as follows:
> > > >
> > > > - Much better with headers: tracing IDs
> > > > - Somewhat better: end-to-end encryption, polyglot topics
> > > > - Don't buy: Mirroring (though perhaps could be made to work
> > with
> > > > changes to the proposal), EOS, large messages, compression
> > > > - Not sure: Control messages
> > > >
> > > > Does this match where other people are at? From my point of view
> > for
> > > > something like this where we are changing the core data model we
> > > should
> > > > ideally get a handful of use cases to the point where they are
> > > really well
> > > > thought out and we know they can use the feature. I think we have
> > > one solid
> > > > one with tracing IDs, and I think maybe mirroring could be
> another
> > > with
> > > > some work. I'm less sure about the others.
> > > >
> > > > Bi-directional Mirroring
> > > > I think bidirectional mirroring could potentially become a
> > > compelling use
> > > > case if we an find a way to make it work. If I understand it, the
> > > proposal
> > > > is to add a header like "origin-cluster" with the cluster id of
> the
> > > Kafka
> > > > cluster. Then modify the mirroring tool to disallow mirroring a
> > > message
> > > > back to it's own origin cluster. This basic idea is definitely
> > > workable and
> > > > useful, but can we really get to a good implementation using
> > headers?
> > > > I imagine the approach would be to add this header in the
> producer
> > > if some
> > > > config is set. There are two problems:
> > > >
> > > > - You add a pretty big honking cluster ID to each message. For
> > > small
> > > > messages this maybe larger than the contents itself.
> > > > - What happens if someone somewhere forgets to set that config
> > on
> > > some
> > > > client somewhere?
> > > >
> > > > If someone forgets to set the config in some client somewhere I
> > > guess you
> > > > have various options in mirror maker:
> > > >
> > > > 1. drop the data (pretty bad, now your data is out of sync)
> > > > 2. crash (pretty bad--how do you recover)
> > > > 3. insert some default (pretty bad since would also lead to
> data
> > > loss if
> > > > that's not right)
> > > >
> > > > Basically although I think a cluster tag could be a good way to
> do
> > > > bi-directional mirroring, I'm not sure this naive approach I
> > > described
> > > > would get us to a usable implementation. This could be fixable. I
> > > think if
> > > > there was some scheme to check required headers on the server
> that
> > > could
> > > > alleviate the missing header problem. Not really sure what to do
> > > about the
> > > > message bloat problem, but maybe there could be some way to
> shrink
> > > the
> > > > cluster id?
> > > >
> > > > Cheers,
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > >> @Jay, Ismael and a few others.
> > > >>
> > > >> Based on the discussion on kip-87, and the additional use case
> by
> > > Matthias.
> > > >>
> > > >> Just for clarity do we now have consensus on the earlier points
> > (1)
> > > >> headers are useful (2) headers should be native in Kafka?
> > > >>
> > > >> Cheers
> > > >> Mike
> > > >> ________________________________________
> > > >> From: radai <ra...@gmail.com>
> > > >> Sent: Monday, December 19, 2016 5:01 PM
> > > >> To: dev@kafka.apache.org
> > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >>
> > > >> having said that im perfectly fine with something like a "no
> > > payload" flag
> > > >> (as opposed to null payload), which would mean such a msg is not
> > > returned
> > > >> from poll() unless the user code specifically "opts in" and
> would
> > > serve
> > > >> only to carry headers (making it a control message)
> > > >>
> > > >> On Mon, Dec 19, 2016 at 8:58 AM, radai <
> > radai.rosenblatt@gmail.com>
> > > wrote:
> > > >>
> > > >> > if all you want is to tag "end of stream" wouldnt a header on
> > the
> > > last
> > > >> msg
> > > >> > in the stream be enough?
> > > >> >
> > > >> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> > > matthias@confluent.io>
> > > >> > wrote:
> > > >> >
> > > >> >> The details about headers for control messages are still to
> > > define. But
> > > >> >> yes, the idea is to have some common default behavior that
> > > clients would
> > > >> >> need to implement.
> > > >> >>
> > > >> >> The point is, that "regular headers" add meta data to regular
> > > messages.
> > > >> >> Thus, those messages will be returned to the user via
> .poll().
> > > And after
> > > >> >> the message is received the user can check if meta data is
> > > present and
> > > >> >> read it.
> > > >> >>
> > > >> >> For control messages, we do not want those to pop up via
> > .poll()
> > > as
> > > >> >> those are no regular messages. A client would need to opt-in
> to
> > > see
> > > >> >> those messages (either via poll() or maybe a callback). Thus,
> > we
> > > need
> > > >> >> some special (standardized) header IDs that indicate control
> > > messages
> > > >> >> that should not be returned to the user via poll() by
> default.
> > > >> >>
> > > >> >>
> > > >> >> -Matthias
> > > >> >>
> > > >> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
> > > >> >> > Matthias,
> > > >> >> >
> > > >> >> > Thanks for your input. I'm +1 on control messages as they
> > > seem to be
> > > >> >> the
> > > >> >> > simplest way to implement watermarks (
> > > >> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-
> > > streaming-102),
> > > >> a
> > > >> >> > feature that would add a lot of value to Kafka Streams
> IMHO.
> > > >> >> >
> > > >> >> > Your argument that the control-message indicator belongs in
> > the
> > > >> >> client-only
> > > >> >> > section of the record format make sense. Just to make
> sure I
> > > >> >> understand,
> > > >> >> > are you suggesting that control messages would be indicated
> > by
> > > a
> > > >> >> standard
> > > >> >> > reserved header? By standard, I mean that ALL Kafka
> > consumers
> > > would
> > > >> >> know
> > > >> >> > to handle these messages differently (possibly just
> ignoring
> > > them).
> > > >> >> This
> > > >> >> > would need to be added to the specification of the consumer
> > > protocol
> > > >> so
> > > >> >> > that all Kafka clients implement it, right? I think it's a
> > > good idea
> > > >> >> but
> > > >> >> > just checking.
> > > >> >> >
> > > >> >> > Cheers,
> > > >> >> >
> > > >> >> > Roger
> > > >> >> >
> > > >> >> >
> > > >> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> > > >> matthias@confluent.io
> > > >> >> >
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> Yes and no. I did overload the term "control message".
> > > >> >> >>
> > > >> >> >> EOS control messages are for client-broker communication
> and
> > > thus
> > > >> never
> > > >> >> >> exposed to any application. And I think this is a good
> > design
> > > because
> > > >> >> >> broker needs to understand those control messages. Thus,
> > this
> > > should
> > > >> be
> > > >> >> >> a protocol change.
> > > >> >> >>
> > > >> >> >> The type of control messages I have in mind are for
> > > client-client
> > > >> >> >> (application-application) communication and the broker is
> > > agnostic to
> > > >> >> >> them. Thus, it should not be a protocol change.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> -Matthias
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On 12/14/16 9:42 AM, radai wrote:
> > > >> >> >>> arent control messages getting pushed as their own top
> > level
> > > >> protocol
> > > >> >> >>> change (and a fairly massive one) for the transactions
> KIP
> > ?
> > > >> >> >>>
> > > >> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> > > >> >> matthias@confluent.io>
> > > >> >> >>> wrote:
> > > >> >> >>>
> > > >> >> >>>> Hi,
> > > >> >> >>>>
> > > >> >> >>>> I want to add a completely new angle to this discussion.
> > > For this,
> > > >> I
> > > >> >> >>>> want to propose an extension for the headers feature
> that
> > > enables
> > > >> new
> > > >> >> >>>> uses cases -- and those new use cases might convince
> > people
> > > to
> > > >> >> support
> > > >> >> >>>> headers (of course including the larger scoped
> proposal).
> > > >> >> >>>>
> > > >> >> >>>> Extended Proposal:
> > > >> >> >>>>
> > > >> >> >>>> Allow messages with a certain header key to be special
> > > "control
> > > >> >> >>>> messages" (w/ o w/o payload) that are not exposed to an
> > > application
> > > >> >> via
> > > >> >> >>>> .poll().
> > > >> >> >>>>
> > > >> >> >>>> Thus, a consumer client would automatically skip over
> > those
> > > >> >> messages. If
> > > >> >> >>>> an application knows about embedded control messages, it
> > > can "sing
> > > >> >> up"
> > > >> >> >>>> to those messages by the consumer client and either get
> a
> > > callback
> > > >> or
> > > >> >> >>>> the consumer auto-drop for this messages gets disabled
> > > (allowing to
> > > >> >> >>>> consumer those messages via poll()).
> > > >> >> >>>>
> > > >> >> >>>> (The details need further considerations/discussion. I
> > just
> > > want to
> > > >> >> >>>> sketch the main idea.)
> > > >> >> >>>>
> > > >> >> >>>> Usage:
> > > >> >> >>>>
> > > >> >> >>>> There is a shared topic (ie, used by multiple
> > applications)
> > > and a
> > > >> >> >>>> producer application wants to embed a special message in
> > > the topic
> > > >> >> for a
> > > >> >> >>>> dedicated consumer application. Because only one
> > > application will
> > > >> >> >>>> understand this message, it cannot be a regular message
> as
> > > this
> > > >> would
> > > >> >> >>>> break all applications that do not understand this
> > message.
> > > The
> > > >> >> producer
> > > >> >> >>>> application would set a special metadata key and no
> > consumer
> > > >> >> application
> > > >> >> >>>> would see this control message by default because they
> did
> > > not
> > > >> enable
> > > >> >> >>>> their consumer client to return this message in poll()
> > (and
> > > the
> > > >> >> client
> > > >> >> >>>> would just drop this message with special metadata key).
> > > Only the
> > > >> >> single
> > > >> >> >>>> application that should receive this message, will
> > > subscribe to
> > > >> this
> > > >> >> >>>> message on its consumer client and process it.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Concrete Use Case: Kafka Streams
> > > >> >> >>>>
> > > >> >> >>>> In Kafka Streams, we would like to propagate "control
> > > messages"
> > > >> from
> > > >> >> >>>> subtopology to subtopology. There are multiple scenarios
> > > for which
> > > >> >> this
> > > >> >> >>>> would be useful. For example, currently we do not
> > guarantee
> > > a
> > > >> >> >>>> "consistent shutdown" of an application. By this, I mean
> > > that input
> > > >> >> >>>> records might not be completely processed by the whole
> > > topology
> > > >> >> because
> > > >> >> >>>> the application shutdown happens "in between" and an
> > > intermediate
> > > >> >> result
> > > >> >> >>>> topic gets "stock" in an intermediate topic. Thus, a
> user
> > > would see
> > > >> >> an
> > > >> >> >>>> committed offset of the source topic of the application,
> > > but no
> > > >> >> >>>> corresponding result record in the output topic.
> > > >> >> >>>>
> > > >> >> >>>> Having "shutdown markers" would allow us, to first stop
> > the
> > > >> upstream
> > > >> >> >>>> subtopology and write this marker into the intermediate
> > > topic and
> > > >> the
> > > >> >> >>>> downstream subtopology would only shut down itself after
> > is
> > > sees
> > > >> the
> > > >> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown,
> > that
> > > no
> > > >> >> >>>> "in-flight" messages got stuck in intermediate topics.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> A similar usage would be for KIP-95 (Incremental Batch
> > > Processing).
> > > >> >> >>>> There was a discussion about the proposed metadata
> topic,
> > > and we
> > > >> >> could
> > > >> >> >>>> avoid this metadata topic if we would have "control
> > > messages".
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Right now, we cannot insert an "application control
> > message"
> > > >> because
> > > >> >> >>>> Kafka Streams does not own all topics it read/writes and
> > > thus might
> > > >> >> >>>> break other consumer application (as described above) if
> > we
> > > inject
> > > >> >> >>>> random messages that are not understood by other apps.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Of course, one can work around "embedded control
> messaged"
> > > by using
> > > >> >> an
> > > >> >> >>>> additional topic to propagate control messaged between
> > > application
> > > >> >> (as
> > > >> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka
> > > Streams). But
> > > >> >> there
> > > >> >> >>>> are major concerns about adding this metadata topic in
> the
> > > KIP and
> > > >> >> this
> > > >> >> >>>> shows that other application that need a similar pattern
> > > might
> > > >> profit
> > > >> >> >>>> from topic embedded "control messages", too.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> One last important consideration: those "control
> messages"
> > > are used
> > > >> >> for
> > > >> >> >>>> client to client communication and are not understood by
> > the
> > > >> broker.
> > > >> >> >>>> Thus, those messages should not be enabled within the
> > > message
> > > >> format
> > > >> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> > > record
> > > >> >> headers
> > > >> >> >>>> would be a nice way to implement them. Because KIP-82
> did
> > > consider
> > > >> >> key
> > > >> >> >>>> namespaces for metatdata keys, this extension should not
> > be
> > > an own
> > > >> >> KIP
> > > >> >> >>>> but should be included in KIP-82 to reserve a namespace
> > for
> > > >> "control
> > > >> >> >>>> message" in the first place.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Sorry for the long email... Looking forward to your
> > > feedback.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> -Matthias
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> > > >> >> >>>>> Hi Jun
> > > >> >> >>>>>
> > > >> >> >>>>> 100) each time a transaction exits a jvm for a remote
> > > system
> > > >> >> (HTTP/JMS/
> > > >> >> >>>> Hopefully one day kafka) the APM tools stich in a unique
> > id
> > > >> (though I
> > > >> >> >>>> believe it contains the end2end uuid embedded in this
> id),
> > > on
> > > >> >> receiving
> > > >> >> >> the
> > > >> >> >>>> message at the receiving JVM the apm code takes this
> out,
> > > and
> > > >> >> continues
> > > >> >> >> its
> > > >> >> >>>> tracing on the that new thread. Both JVM’s (and other
> > > languages the
> > > >> >> APM
> > > >> >> >>>> tool supports) send this data async back to the central
> > > controllers
> > > >> >> >> where
> > > >> >> >>>> the stiching togeather occurs. For this they need some
> > > header space
> > > >> >> for
> > > >> >> >>>> them to put this id.
> > > >> >> >>>>>
> > > >> >> >>>>> 101) Yes indeed we have a business transaction Id in
> the
> > > payload.
> > > >> >> >> Though
> > > >> >> >>>> this is a system level tracing, that we need to have
> marry
> > > up. Also
> > > >> >> as
> > > >> >> >> per
> > > >> >> >>>> note on end2end encryption we’d be unable to prove the
> > flow
> > > if the
> > > >> >> >> payload
> > > >> >> >>>> is encrypted as we’d not have access to this at certain
> > > points of
> > > >> the
> > > >> >> >> flow
> > > >> >> >>>> through the infrastructure/platform.
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> 103) As said we use this mechanism in IG very
> > > successfully, as
> > > >> >> stated
> > > >> >> >>>> per key we guarantee the transaction producing app to
> > > handle the
> > > >> >> >>>> transaction of a key at one DC unless at point of
> critical
> > > failure
> > > >> >> >> where we
> > > >> >> >>>> have to flip processing to another. We care about key
> > > ordering.
> > > >> >> >>>>> I disagree on the offset comment for the partition
> > > solution unless
> > > >> >> you
> > > >> >> >>>> do full ISR, or expensive full XA transactions even with
> > > partitions
> > > >> >> you
> > > >> >> >>>> cannot fully guarantee offsets would match.
> > > >> >> >>>>>
> > > >> >> >>>>> 105) Very much so, I need to have access at the
> platform
> > > level to
> > > >> >> the
> > > >> >> >>>> other meta data all mentioned, without having to need to
> > > have
> > > >> access
> > > >> >> to
> > > >> >> >> the
> > > >> >> >>>> encryption keys of the payload.
> > > >> >> >>>>>
> > > >> >> >>>>> 106)
> > > >> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d
> need
> > > to have
> > > >> a
> > > >> >> >>>> global producerId register which would be very hard to
> > > >> >> enforce/ensure is
> > > >> >> >>>> current and correct, just to understand the message
> > origins
> > > of its
> > > >> >> >>>> region/az/cluster for routing.
> > > >> >> >>>>> The client wrapper version, producerId can be the same,
> > as
> > > >> obviously
> > > >> >> >> the
> > > >> >> >>>> producer could upgrade its wrapper, as such we need to
> > know
> > > what
> > > >> >> wrapper
> > > >> >> >>>> version the message is created with.
> > > >> >> >>>>> Likewise the IP address, as stated we can have our
> > > producer move,
> > > >> >> where
> > > >> >> >>>> its IP would change.
> > > >> >> >>>>>
> > > >> >> >>>>> 107)
> > > >> >> >>>>> UUID is set on the message by interceptors before
> actual
> > > producer
> > > >> >> >>>> transport send. This is for platform level message
> dedupe
> > > >> guarantee,
> > > >> >> the
> > > >> >> >>>> business payload should be agnostic to this. Please see
> > > >> >> >>>> https://activemq.apache.org/
> artemis/docs/1.5.0/duplicate-
> > > det
> > > >> >> ection.html
> > > >> >> >>>> note this is not touching business payloads.
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io>
> > wrote:
> > > >> >> >>>>>
> > > >> >> >>>>> Hi, Michael,
> > > >> >> >>>>>
> > > >> >> >>>>> Thanks for the reply. I find it very helpful.
> > > >> >> >>>>>
> > > >> >> >>>>> Data lineage:
> > > >> >> >>>>> 100. I'd like to understand the APM use case a bit
> > > more. It
> > > >> >> sounds
> > > >> >> >>>> like
> > > >> >> >>>>> that those APM plugins can generate a transaction
> id
> > > that we
> > > >> >> could
> > > >> >> >>>>> potentially put in the header of every message. How
> > > would you
> > > >> >> >>>> typically
> > > >> >> >>>>> make use of such transaction ids? Are there other
> > > metadata
> > > >> >> >>>> associated with
> > > >> >> >>>>> the transaction id and if so, how are they
> propagated
> > > >> >> downstream?
> > > >> >> >>>>>
> > > >> >> >>>>> 101. For the finance use case, if the concept of
> > > transaction
> > > >> is
> > > >> >> >>>> important,
> > > >> >> >>>>> wouldn't it be typically included in the message
> > > payload
> > > >> >> instead of
> > > >> >> >>>> as an
> > > >> >> >>>>> optional header field?
> > > >> >> >>>>>
> > > >> >> >>>>> 102. The data lineage that Altas and Navigator
> > support
> > > seems
> > > >> to
> > > >> >> be
> > > >> >> >>>> at the
> > > >> >> >>>>> dataset level, not per record level? So, not sure
> if
> > > per
> > > >> message
> > > >> >> >>>> headers
> > > >> >> >>>>> are relevant there.
> > > >> >> >>>>>
> > > >> >> >>>>> Mirroring:
> > > >> >> >>>>> 103. The benefit of using separate partitions is
> that
> > > it
> > > >> >> >> potentially
> > > >> >> >>>> makes
> > > >> >> >>>>> it easy to preserve offsets during mirroring. This
> > > will make
> > > >> it
> > > >> >> >>>> easier for
> > > >> >> >>>>> consumer to switch clusters. Currently, the
> consumers
> > > can
> > > >> switch
> > > >> >> >>>> clusters
> > > >> >> >>>>> by using the timestampToOffset() api, but it has to
> > > deal with
> > > >> >> >>>> duplicates.
> > > >> >> >>>>> Good point on the issue with log compact and I am
> not
> > > sure how
> > > >> >> to
> > > >> >> >>>> address
> > > >> >> >>>>> this. However, even if we mirror into the existing
> > > partitions,
> > > >> >> the
> > > >> >> >>>> ordering
> > > >> >> >>>>> for messages generated from different clusters
> seems
> > > >> >> >>>> non-deterministic
> > > >> >> >>>>> anyway. So, it seems that the consumers already
> have
> > > to deal
> > > >> >> with
> > > >> >> >>>> that? If
> > > >> >> >>>>> a topic is compacted, does that mean which messages
> > are
> > > >> >> preserved
> > > >> >> >> is
> > > >> >> >>>> also
> > > >> >> >>>>> non-deterministic across clusters?
> > > >> >> >>>>>
> > > >> >> >>>>> 104. Good point on partition key.
> > > >> >> >>>>>
> > > >> >> >>>>> End-to-end encryption:
> > > >> >> >>>>> 105. So, it seems end-to-end encryption is useful.
> > Are
> > > headers
> > > >> >> >>>> useful there?
> > > >> >> >>>>>
> > > >> >> >>>>> Auditing:
> > > >> >> >>>>> 106. It seems other than the UUID, all other
> metadata
> > > are per
> > > >> >> >>>> producer?
> > > >> >> >>>>>
> > > >> >> >>>>> EOS:
> > > >> >> >>>>> 107. How are those UUIDs generated? I am not sure
> if
> > > they can
> > > >> be
> > > >> >> >>>> generated
> > > >> >> >>>>> in the producer library. An application may send
> > > messages
> > > >> >> through a
> > > >> >> >>>> load
> > > >> >> >>>>> balancer and on retry, the same message could be
> > > routed to a
> > > >> >> >>>> different
> > > >> >> >>>>> producer instance. So, it seems that the
> application
> > > has to
> > > >> >> >> generate
> > > >> >> >>>> the
> > > >> >> >>>>> UUIDs. In that case, shouldn't the application just
> > > put the
> > > >> >> UUID in
> > > >> >> >>>> the
> > > >> >> >>>>> payload?
> > > >> >> >>>>>
> > > >> >> >>>>> Thanks,
> > > >> >> >>>>>
> > > >> >> >>>>> Jun
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> > > >> >> >>>> Michael.Pearce@ig.com>
> > > >> >> >>>>> wrote:
> > > >> >> >>>>>
> > > >> >> >>>>> > Hi Jun.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Per Transaction Tracing / Data Lineage.
> > > >> >> >>>>> >
> > > >> >> >>>>> > As Stated in the KIP this has the first use case
> of
> > > how many
> > > >> >> APM
> > > >> >> >>>> tools now
> > > >> >> >>>>> > work.
> > > >> >> >>>>> > I would find it impossible for any one to argue
> > this
> > > is not
> > > >> >> >>>> important or a
> > > >> >> >>>>> > niche market as it has its own gartner report for
> > > this
> > > >> space.
> > > >> >> >> Such
> > > >> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace,
> > > Hawqular are
> > > >> >> but a
> > > >> >> >>>> few.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Likewise these APM tools can help very rapidly
> > track
> > > down
> > > >> >> issues
> > > >> >> >>>> and
> > > >> >> >>>>> > automatically capture metrics, perform actions
> > based
> > > on
> > > >> >> >> unexpected
> > > >> >> >>>> behavior
> > > >> >> >>>>> > to auto recover services.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Before mentioning looking at aggregated stats, in
> > > these
> > > >> cases
> > > >> >> >> where
> > > >> >> >>>>> > actually on critical flows we cannot afford to
> have
> > > >> aggregated
> > > >> >> >>>> rolled up
> > > >> >> >>>>> > stats only.
> > > >> >> >>>>> >
> > > >> >> >>>>> > With the APM tool we use its actually able to
> > detect
> > > a
> > > >> single
> > > >> >> >>>> transaction
> > > >> >> >>>>> > failure and capture the thread traces in the JVM
> > > where it
> > > >> >> failed
> > > >> >> >>>> and
> > > >> >> >>>>> > everything for us, to the point it sends us
> alerts
> > > where we
> > > >> >> have
> > > >> >> >>>> this
> > > >> >> >>>>> > giving the line number of the code that caused
> it,
> > > the
> > > >> >> >> transaction
> > > >> >> >>>> trace
> > > >> >> >>>>> > through all the services and endpoints
> (supported)
> > > upto the
> > > >> >> point
> > > >> >> >>>> of
> > > >> >> >>>>> > failure, it can also capture the data in and out
> > (so
> > > we can
> > > >> >> >>>> replay).
> > > >> >> >>>>> > Because atm Kafka doesn’t support us being able
> to
> > > stich in
> > > >> >> these
> > > >> >> >>>> tracing
> > > >> >> >>>>> > transaction ids natively, we cannot get these
> > > benefits as
> > > >> >> such is
> > > >> >> >>>> limiting
> > > >> >> >>>>> > our ability support apps and monitor them to the
> > same
> > > >> >> standards
> > > >> >> >> we
> > > >> >> >>>> come to
> > > >> >> >>>>> > expect when on a kafka flow.
> > > >> >> >>>>> >
> > > >> >> >>>>> > This actually ties in with Data Lineage, as the
> > same
> > > tracing
> > > >> >> can
> > > >> >> >>>> be used
> > > >> >> >>>>> > to back stich this. Essentially many times due to
> > > the sums
> > > >> of
> > > >> >> >> money
> > > >> >> >>>>> > involved there are disputes, and typically as a
> > > financial
> > > >> >> >>>> institute the
> > > >> >> >>>>> > easiest and cleanest way to prove when disputes
> > > arise is to
> > > >> >> >>>> present the
> > > >> >> >>>>> > actual flow and processes involved in a
> > transaction.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Likewise as Hadoop matures its evident this case
> is
> > > >> >> important, as
> > > >> >> >>>> tools
> > > >> >> >>>>> > such as Atlas (Hortonworks led) and Navigator
> > > (cloudera led)
> > > >> >> are
> > > >> >> >>>> evident
> > > >> >> >>>>> > also I believe the importance here is very much
> NOT
> > > just a
> > > >> >> >>>> financial issue.
> > > >> >> >>>>> >
> > > >> >> >>>>> > From a MDM point of view any company wanting to
> > care
> > > about
> > > >> >> Data
> > > >> >> >>>> Quality
> > > >> >> >>>>> > and Data Governance - Data Lineage is a key piece
> > in
> > > this
> > > >> >> puzzle.
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE Mirroring,
> > > >> >> >>>>> >
> > > >> >> >>>>> > As per the KIP in-fact this is exactly what we do
> > re
> > > cluster
> > > >> >> id,
> > > >> >> >>>> to mirror
> > > >> >> >>>>> > a network of clusters between AZ’s / Regions. We
> > > know a
> > > >> >> >>>> transaction for a
> > > >> >> >>>>> > key will be done within a AZ/Region, as such we
> > > know the
> > > >> >> write
> > > >> >> >> to
> > > >> >> >>>> kafka
> > > >> >> >>>>> > would be ordered per key. But we need eventual
> view
> > > of that
> > > >> >> >> across
> > > >> >> >>>> in our
> > > >> >> >>>>> > other regions/az’s. When we have complete AZ or
> > > Region
> > > >> >> failure we
> > > >> >> >>>> know
> > > >> >> >>>>> > there will be a brief interruption whilst those
> > > transactions
> > > >> >> are
> > > >> >> >>>> moved to
> > > >> >> >>>>> > another region but we expect after it to
> continue.
> > > >> >> >>>>> >
> > > >> >> >>>>> > As mentioned having separate Partions to do this
> > > starts to
> > > >> get
> > > >> >> >>>>> > ugly/complicated for us:
> > > >> >> >>>>> > how would I do compaction where a key is in two
> > > partitions?
> > > >> >> >>>>> > How do we balance consumers so where multiple
> > > partitions
> > > >> with
> > > >> >> the
> > > >> >> >>>> same key
> > > >> >> >>>>> > goto the same consumer
> > > >> >> >>>>> > What do you do if cluster 1 has 5 partitions but
> > > cluster 20
> > > >> >> has
> > > >> >> >> 10
> > > >> >> >>>> because
> > > >> >> >>>>> > its larger kit in our more core DC’s, as such key
> > to
> > > >> partition
> > > >> >> >>>> mappings for
> > > >> >> >>>>> > consumers get even more complicated.
> > > >> >> >>>>> > What do you do if we add or remove a complete
> > region
> > > >> >> >>>>> >
> > > >> >> >>>>> > Where as simple mirror will work we just need to
> > > ensure we
> > > >> >> don’t
> > > >> >> >>>> have a
> > > >> >> >>>>> > cycle which we can do with clusterId.
> > > >> >> >>>>> >
> > > >> >> >>>>> > We even have started to look at shortest path
> > mirror
> > > routing
> > > >> >> >> based
> > > >> >> >>>> on
> > > >> >> >>>>> > clusterId, if we also had the region and az info
> on
> > > the
> > > >> >> >> originating
> > > >> >> >>>>> > message, this we have not implemented but some
> > ideas
> > > come
> > > >> from
> > > >> >> >>>> network
> > > >> >> >>>>> > routing, and also the dispatcher router in apache
> > > qpid.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Also we need to have data perimeters e.g. certain
> > > data
> > > >> cannot
> > > >> >> >> leave
> > > >> >> >>>>> > certain countries borders. We want this all
> > > automated so
> > > >> that
> > > >> >> at
> > > >> >> >>>> the
> > > >> >> >>>>> > platform level without having to touch or look at
> > the
> > > >> business
> > > >> >> >>>> data inside
> > > >> >> >>>>> > we can have headers we can put tags into so that
> we
> > > can
> > > >> ensure
> > > >> >> >>>> this doesn’t
> > > >> >> >>>>> > occur when we mirror. (actually links in to data
> > > lineage /
> > > >> >> >> tracing
> > > >> >> >>>> as again
> > > >> >> >>>>> > we need to tag messages at a platform level)
> > > Examples are we
> > > >> >> are
> > > >> >> >>>> not
> > > >> >> >>>>> > allowed Private customer details to leave
> > > Switzerland, yet
> > > >> we
> > > >> >> >> need
> > > >> >> >>>> those
> > > >> >> >>>>> > systems integrated.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Lastly around mirroring we have a partionKey
> field,
> > > as the
> > > >> key
> > > >> >> >>>> used for
> > > >> >> >>>>> > portioning logic != compaction key all the time
> but
> > > we want
> > > >> to
> > > >> >> >>>> preserve it
> > > >> >> >>>>> > for when we mirror so that if source cluster
> > > partition count
> > > >> >> !=
> > > >> >> >>>> destination
> > > >> >> >>>>> > cluster partition count we can honour the same
> > > partitioning
> > > >> >> >> logic.
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE End 2 End encryption
> > > >> >> >>>>> >
> > > >> >> >>>>> > As I believe mentioned just before, the solution
> > you
> > > mention
> > > >> >> just
> > > >> >> >>>> doesn’t
> > > >> >> >>>>> > cut the mustard these days with many regulators.
> An
> > > >> operations
> > > >> >> >>>> person with
> > > >> >> >>>>> > access to the box should not be able to have
> access
> > > to the
> > > >> >> data.
> > > >> >> >>>> Many now
> > > >> >> >>>>> > actually impose quite literally the
> implementation
> > > expected
> > > >> >> being
> > > >> >> >>>> end2end
> > > >> >> >>>>> > encryption for certain data (Singapore for us is
> > one
> > > that I
> > > >> am
> > > >> >> >>>> most aware
> > > >> >> >>>>> > of). In fact we’re even now needing encrypt the
> > data
> > > and
> > > >> store
> > > >> >> >> the
> > > >> >> >>>> keys in
> > > >> >> >>>>> > HSM modules.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Likewise the performance penalty on encrypting
> > > decrypting as
> > > >> >> you
> > > >> >> >>>> produce
> > > >> >> >>>>> > over wire, then again encrypt decrypt as the data
> > is
> > > stored
> > > >> on
> > > >> >> >> the
> > > >> >> >>>> brokers
> > > >> >> >>>>> > disks and back again, then again encrypted and
> > > decrypted
> > > >> back
> > > >> >> >> over
> > > >> >> >>>> the wire
> > > >> >> >>>>> > each time for each consumer all adds up, ignoring
> > > this
> > > >> >> doubling
> > > >> >> >>>> with mirror
> > > >> >> >>>>> > makers etc. simply encrypting the value once on
> > > write by the
> > > >> >> >>>> client and
> > > >> >> >>>>> > again decrypting on consume by the consumer is
> far
> > > more
> > > >> >> >>>> performant, but
> > > >> >> >>>>> > then the routing and platform meta data needs to
> be
> > > separate
> > > >> >> >> (thus
> > > >> >> >>>> headers)
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE Auditing:
> > > >> >> >>>>> >
> > > >> >> >>>>> > Our Auditing needs are:
> > > >> >> >>>>> > Producer Id,
> > > >> >> >>>>> > Origin Cluster Id that message first produced
> into
> > > >> >> >>>>> > Origin AZ – agreed we can derive this if we have
> > > cluster id,
> > > >> >> but
> > > >> >> >>>> it makes
> > > >> >> >>>>> > resolving this for audit reporting a lot easier.
> > > >> >> >>>>> > Origin Region – agreed we can derive this if we
> > have
> > > cluster
> > > >> >> id,
> > > >> >> >>>> but it
> > > >> >> >>>>> > makes resolving this for audit reporting a lot
> > > easier.
> > > >> >> >>>>> > Unique Message Identification (this is not the
> same
> > > as
> > > >> >> >> transaction
> > > >> >> >>>>> > tracing) – note offset and partition are not the
> > > same, as
> > > >> >> when we
> > > >> >> >>>> mirror or
> > > >> >> >>>>> > have for what ever system failure duplicate send,
> > > >> >> >>>>> > Custom Client wrapper version (where
> organizations
> > > have to
> > > >> >> wrap
> > > >> >> >>>> the kafka
> > > >> >> >>>>> > client for added features) so we know what
> version
> > > of the
> > > >> >> wrapper
> > > >> >> >>>> is used
> > > >> >> >>>>> > Producer IP address (in case of clients being in
> > our
> > > vm/open
> > > >> >> >> stack
> > > >> >> >>>> infra
> > > >> >> >>>>> > where they can move around, producer id will stay
> > > the same
> > > >> but
> > > >> >> >>>> this would
> > > >> >> >>>>> > change)
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE Once and only once delivery case
> > > >> >> >>>>> >
> > > >> >> >>>>> > Using the same Message UUID for auditing we can
> > > achieve this
> > > >> >> >> quite
> > > >> >> >>>> simply.
> > > >> >> >>>>> >
> > > >> >> >>>>> > As per how some other brokers do this (cough
> qpid,
> > > artemis)
> > > >> >> >>>> message uuid
> > > >> >> >>>>> > are used to dedupe where message is sent and
> > > produced but
> > > >> the
> > > >> >> >>>> client didn’t
> > > >> >> >>>>> > receive the ack, and there for replays the send,
> by
> > > having a
> > > >> >> >>>> unique message
> > > >> >> >>>>> > id per message, this can be filtered out, on
> > > consumers where
> > > >> >> >>>> message
> > > >> >> >>>>> > delivery may occur twice for what ever reasons a
> > > message
> > > >> uuid
> > > >> >> can
> > > >> >> >>>> be used
> > > >> >> >>>>> > to remove duplicates being deliverd , like wise
> we
> > > can do
> > > >> >> this in
> > > >> >> >>>> the
> > > >> >> >>>>> > mirrormakers so if we detect a dupe message we
> can
> > > avoid
> > > >> >> >>>> replicating it.
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > Cheers
> > > >> >> >>>>> > Mike
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <
> jun@confluent.io>
> > > wrote:
> > > >> >> >>>>> >
> > > >> >> >>>>> > Since this KIP affects message format, wire
> > > protocol,
> > > >> >> apis, I
> > > >> >> >>>> think
> > > >> >> >>>>> > it's
> > > >> >> >>>>> > worth spending a bit more time to nail down
> the
> > > concrete
> > > >> >> use
> > > >> >> >>>> cases. It
> > > >> >> >>>>> > would be bad if we add this feature, but when
> > > start
> > > >> >> >>>> implementing it
> > > >> >> >>>>> > for say
> > > >> >> >>>>> > mirroring, we then realize that header is not
> > > the best
> > > >> >> >>>> approach.
> > > >> >> >>>>> > Initially,
> > > >> >> >>>>> > I thought I was convinced of the use cases of
> > > headers
> > > >> and
> > > >> >> was
> > > >> >> >>>> trying to
> > > >> >> >>>>> > write down a few use cases to convince
> others.
> > > That's
> > > >> >> when I
> > > >> >> >>>> became
> > > >> >> >>>>> > less
> > > >> >> >>>>> > certain. For me to be convinced, I just want
> to
> > > see two
> > > >> >> >> strong
> > > >> >> >>>> use
> > > >> >> >>>>> > cases
> > > >> >> >>>>> > (instead of 10 maybe use cases) in the
> > > third-party
> > > >> space.
> > > >> >> The
> > > >> >> >>>> reason is
> > > >> >> >>>>> > that when we discussed the use cases within a
> > > company,
> > > >> >> often
> > > >> >> >>>> it ends
> > > >> >> >>>>> > with
> > > >> >> >>>>> > "we can't force everyone to use this standard
> > > since we
> > > >> may
> > > >> >> >>>> have to
> > > >> >> >>>>> > integrate with third-party tools".
> > > >> >> >>>>> >
> > > >> >> >>>>> > At present, I am not sure why headers are
> > useful
> > > for
> > > >> >> things
> > > >> >> >>>> like
> > > >> >> >>>>> > schemaId
> > > >> >> >>>>> > or encryption. In order to do anything useful
> > to
> > > the
> > > >> >> value,
> > > >> >> >>>> one needs
> > > >> >> >>>>> > to
> > > >> >> >>>>> > know the schemaId or how data is encrypted,
> but
> > > header
> > > >> is
> > > >> >> >>>> optional.
> > > >> >> >>>>> > But, I
> > > >> >> >>>>> > can be convinced if someone (Radai, Sean,
> > Todd?)
> > > >> provides
> > > >> >> >> more
> > > >> >> >>>> details
> > > >> >> >>>>> > on
> > > >> >> >>>>> > the argument.
> > > >> >> >>>>> >
> > > >> >> >>>>> > I am not very sure header is the best
> approach
> > > for
> > > >> >> mirroring
> > > >> >> >>>> either. If
> > > >> >> >>>>> > someone has thought about this more, I'd be
> > > happy to
> > > >> hear.
> > > >> >> >>>>> >
> > > >> >> >>>>> > I can see the data lineage use case. I am
> just
> > > not sure
> > > >> >> how
> > > >> >> >>>> widely
> > > >> >> >>>>> > applicable this is. If someone familiar with
> > > this space
> > > >> >> can
> > > >> >> >>>> justify
> > > >> >> >>>>> > this is
> > > >> >> >>>>> > a significant use case, say in the finance
> > > industry,
> > > >> this
> > > >> >> >>>> would be a
> > > >> >> >>>>> > strong
> > > >> >> >>>>> > use case.
> > > >> >> >>>>> >
> > > >> >> >>>>> > I can see the auditing use case. I am just
> not
> > > sure if a
> > > >> >> >> native
> > > >> >> >>>>> > producer id
> > > >> >> >>>>> > solves that problem. If there are additional
> > > metadata
> > > >> >> that's
> > > >> >> >>>> worth
> > > >> >> >>>>> > collecting but not covered by the producer
> id,
> > > that
> > > >> would
> > > >> >> >> make
> > > >> >> >>>> this a
> > > >> >> >>>>> > strong use case.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Thanks,
> > > >> >> >>>>> >
> > > >> >> >>>>> > Jun
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> > > >> >> >>>> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > wrote:
> > > >> >> >>>>> >
> > > >> >> >>>>> > > this KIP is about enabling headers, nothing
> > > more
> > > >> nothing
> > > >> >> >>>> less - so
> > > >> >> >>>>> > no,
> > > >> >> >>>>> > > broker-side use of headers is not in the
> KIP
> > > scope.
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > obviously though, once you have headers
> > > potential use
> > > >> >> cases
> > > >> >> >>>> could
> > > >> >> >>>>> > include
> > > >> >> >>>>> > > broker-side header-aware interceptors
> (which
> > > would be
> > > >> >> the
> > > >> >> >>>> topic of
> > > >> >> >>>>> > other
> > > >> >> >>>>> > > future KIPs).
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > a trivially clear use case (to me) would be
> > > using such
> > > >> >> >>>> broker-side
> > > >> >> >>>>> > > interceptors to enforce compliance with
> > > organizational
> > > >> >> >>>> policies - it
> > > >> >> >>>>> > would
> > > >> >> >>>>> > > make our SREs lives much easier if instead
> of
> > > >> >> retroactively
> > > >> >> >>>>> > discovering
> > > >> >> >>>>> > > "rogue" topics/users those messages would
> > have
> > > been
> > > >> >> >> rejected
> > > >> >> >>>>> > up-front.
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > the kafka broker code is lacking any such
> > > >> extensibility
> > > >> >> >>>> support
> > > >> >> >>>>> > (beyond
> > > >> >> >>>>> > > maybe authorizer) which is why these use
> > cases
> > > were
> > > >> left
> > > >> >> >> out
> > > >> >> >>>> of the
> > > >> >> >>>>> > "case
> > > >> >> >>>>> > > for headers" doc - broker extensibility is
> a
> > > separate
> > > >> >> >>>> discussion.
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen
> > Shapira <
> > > >> >> >>>> gwen@confluent.io>
> > > >> >> >>>>> > wrote:
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > > Woah, I wasn't aware this is something
> > we'll
> > > do. It
> > > >> >> >> wasn't
> > > >> >> >>>> in the
> > > >> >> >>>>> > KIP,
> > > >> >> >>>>> > > > right?
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > I guess we could do it the same way ACLs
> > > currently
> > > >> >> work.
> > > >> >> >>>>> > > > I had in mind something that will allow
> > > admins to
> > > >> >> apply
> > > >> >> >>>> rules to
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > new create/delete/config topic APIs. So
> > Todd
> > > can
> > > >> >> decide
> > > >> >> >> to
> > > >> >> >>>> reject
> > > >> >> >>>>> > > > "create topic" requests that ask for more
> > > than 40
> > > >> >> >>>> partitions, or
> > > >> >> >>>>> > > > require exactly 3 replicas, or no more
> than
> > > 50GB
> > > >> >> >> partition
> > > >> >> >>>> size,
> > > >> >> >>>>> > etc.
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are
> > > planning to
> > > >> >> apply
> > > >> >> >>>> more
> > > >> >> >>>>> > rules
> > > >> >> >>>>> > > > to requests (and I think we should), we
> may
> > > want a
> > > >> bit
> > > >> >> >>>> more generic
> > > >> >> >>>>> > > > design around that.
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> > > >> >> >>>> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > wrote:
> > > >> >> >>>>> > > > > "wouldn't you be in the business of
> > making
> > > sure
> > > >> >> >> everyone
> > > >> >> >>>> uses
> > > >> >> >>>>> > them
> > > >> >> >>>>> > > > > properly?"
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > > thats where a broker-side plugin would
> > > come handy
> > > >> -
> > > >> >> any
> > > >> >> >>>> incoming
> > > >> >> >>>>> > > message
> > > >> >> >>>>> > > > > that does not conform to org policy
> (read
> > > - does
> > > >> not
> > > >> >> >>>> have the
> > > >> >> >>>>> > proper
> > > >> >> >>>>> > > > > headers) gets thrown out (with an error
> > > returned
> > > >> to
> > > >> >> >> user)
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> > > Palino <
> > > >> >> >>>> tpalino@gmail.com>
> > > >> >> >>>>> > wrote:
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on
> > > this one
> > > >> :)
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> Producing counts to a topic is part of
> > > it, but
> > > >> >> that’s
> > > >> >> >>>> only
> > > >> >> >>>>> > part. So
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> count you have 100 messages in topic
> A.
> > > When you
> > > >> >> >> mirror
> > > >> >> >>>> topic A
> > > >> >> >>>>> > to
> > > >> >> >>>>> > > > another
> > > >> >> >>>>> > > > >> cluster, you have 99 messages. Where
> was
> > > your
> > > >> >> problem?
> > > >> >> >>>> Or
> > > >> >> >>>>> > worse, you
> > > >> >> >>>>> > > > have
> > > >> >> >>>>> > > > >> 100 messages, but one producer
> > duplicated
> > > >> messages
> > > >> >> and
> > > >> >> >>>> another
> > > >> >> >>>>> > one
> > > >> >> >>>>> > > lost
> > > >> >> >>>>> > > > >> messages. You need details about where
> > the
> > > >> message
> > > >> >> >> came
> > > >> >> >>>> from in
> > > >> >> >>>>> > order
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> pinpoint problems when they happen.
> > Source
> > > >> producer
> > > >> >> >>>> info, where
> > > >> >> >>>>> > it was
> > > >> >> >>>>> > > > >> produced into your infrastructure, and
> > > when it
> > > >> was
> > > >> >> >>>> produced.
> > > >> >> >>>>> > This
> > > >> >> >>>>> > > > requires
> > > >> >> >>>>> > > > >> you to add the information to the
> > message.
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> And yes, you still need to maintain
> your
> > > clients.
> > > >> >> So
> > > >> >> >>>> maybe my
> > > >> >> >>>>> > original
> > > >> >> >>>>> > > > >> example was not the best. My thoughts
> on
> > > not
> > > >> >> wanting
> > > >> >> >> to
> > > >> >> >>>> be
> > > >> >> >>>>> > responsible
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> message formats stands, because that’s
> > > very much
> > > >> >> >>>> separate from
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > client.
> > > >> >> >>>>> > > > >> As you know, we have our own internal
> > > client
> > > >> >> library
> > > >> >> >>>> that can
> > > >> >> >>>>> > insert
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> right headers, and right now inserts
> the
> > > right
> > > >> >> audit
> > > >> >> >>>>> > information into
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> message fields. If they exist, and
> > > assuming the
> > > >> >> >> message
> > > >> >> >>>> is Avro
> > > >> >> >>>>> > > encoded.
> > > >> >> >>>>> > > > >> What if someone wants to use JSON
> > instead
> > > for a
> > > >> >> good
> > > >> >> >>>> reason?
> > > >> >> >>>>> > What if
> > > >> >> >>>>> > > > user X
> > > >> >> >>>>> > > > >> wants to encrypt messages, but user Y
> > > does not?
> > > >> >> >>>> Maintaining the
> > > >> >> >>>>> > client
> > > >> >> >>>>> > > > >> library is still much easier than
> > > maintaining the
> > > >> >> >>>> message
> > > >> >> >>>>> > formats.
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> -Todd
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> > > Shapira <
> > > >> >> >>>> gwen@confluent.io
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > wrote:
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> > Based on your last sentence,
> consider
> > me
> > > >> >> convinced
> > > >> >> >> :)
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > I get why headers are critical for
> > > Mirroring
> > > >> (you
> > > >> >> >>>> need tags to
> > > >> >> >>>>> > > prevent
> > > >> >> >>>>> > > > >> > loops and sometimes to route
> messages
> > > to the
> > > >> >> correct
> > > >> >> >>>>> > destination).
> > > >> >> >>>>> > > > >> > But why do you need headers to
> audit?
> > > We are
> > > >> >> >> auditing
> > > >> >> >>>> by
> > > >> >> >>>>> > producing
> > > >> >> >>>>> > > > >> > counts to a side topic (and I was
> > under
> > > the
> > > >> >> >>>> impression you do
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > >> > same), so we never need to modify
> the
> > > message.
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > Another thing - after we added
> > headers,
> > > >> wouldn't
> > > >> >> you
> > > >> >> >>>> be in the
> > > >> >> >>>>> > > > >> > business of making sure everyone
> uses
> > > them
> > > >> >> properly?
> > > >> >> >>>> Making
> > > >> >> >>>>> > sure
> > > >> >> >>>>> > > > >> > everyone includes the right headers
> > you
> > > need,
> > > >> not
> > > >> >> >>>> using the
> > > >> >> >>>>> > header
> > > >> >> >>>>> > > > >> > names you intend to use, etc. I
> don't
> > > think the
> > > >> >> >>>> "policing"
> > > >> >> >>>>> > business
> > > >> >> >>>>> > > > >> > will ever go away.
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> > > Palino <
> > > >> >> >>>>> > tpalino@gmail.com>
> > > >> >> >>>>> > > > wrote:
> > > >> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not
> very
> > > happy
> > > >> with
> > > >> >> the
> > > >> >> >>>>> > workaround.
> > > >> >> >>>>> > > Avro
> > > >> >> >>>>> > > > >> means
> > > >> >> >>>>> > > > >> > > that I have to be concerned with
> the
> > > format
> > > >> of
> > > >> >> the
> > > >> >> >>>> messages
> > > >> >> >>>>> > in
> > > >> >> >>>>> > > > order to
> > > >> >> >>>>> > > > >> > run
> > > >> >> >>>>> > > > >> > > the infrastructure (audit,
> > mirroring,
> > > etc.).
> > > >> >> That
> > > >> >> >>>> means
> > > >> >> >>>>> > that I
> > > >> >> >>>>> > > have
> > > >> >> >>>>> > > > to
> > > >> >> >>>>> > > > >> > > handle the schemas, and I have to
> > > enforce
> > > >> rules
> > > >> >> >>>> about good
> > > >> >> >>>>> > > formats.
> > > >> >> >>>>> > > > >> This
> > > >> >> >>>>> > > > >> > is
> > > >> >> >>>>> > > > >> > > not something I want to be in the
> > > business
> > > >> of,
> > > >> >> >>>> because I
> > > >> >> >>>>> > should be
> > > >> >> >>>>> > > > able
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > > run a service infrastructure
> without
> > > needing
> > > >> >> to be
> > > >> >> >>>> in the
> > > >> >> >>>>> > weeds of
> > > >> >> >>>>> > > > >> > dealing
> > > >> >> >>>>> > > > >> > > with customer data formats.
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > > Trust me, a sizable portion of my
> > > support
> > > >> time
> > > >> >> is
> > > >> >> >>>> spent
> > > >> >> >>>>> > dealing
> > > >> >> >>>>> > > with
> > > >> >> >>>>> > > > >> > schema
> > > >> >> >>>>> > > > >> > > issues. I really would like to get
> > > away from
> > > >> >> that.
> > > >> >> >>>> Maybe
> > > >> >> >>>>> > I'd have
> > > >> >> >>>>> > > > more
> > > >> >> >>>>> > > > >> > time
> > > >> >> >>>>> > > > >> > > for other hobbies. Like writing.
> ;)
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > > -Todd
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM
> Gwen
> > > Shapira <
> > > >> >> >>>>> > gwen@confluent.io>
> > > >> >> >>>>> > > > wrote:
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > >> I'm pretty satisfied with the
> > current
> > > >> >> workarounds
> > > >> >> >>>> (Avro
> > > >> >> >>>>> > container
> > > >> >> >>>>> > > > >> > >> format), so I'm not too excited
> > > about the
> > > >> >> extra
> > > >> >> >>>> work
> > > >> >> >>>>> > required to
> > > >> >> >>>>> > > do
> > > >> >> >>>>> > > > >> > >> headers in Kafka. I absolutely
> > don't
> > > mind it
> > > >> >> if
> > > >> >> >>>> you do
> > > >> >> >>>>> > it...
> > > >> >> >>>>> > > > >> > >> I think the Apache convention for
> > > "good
> > > >> idea,
> > > >> >> but
> > > >> >> >>>> not
> > > >> >> >>>>> > willing to
> > > >> >> >>>>> > > > put
> > > >> >> >>>>> > > > >> > >> any work toward it" is +0.5?
> > anyway,
> > > that's
> > > >> >> what
> > > >> >> >> I
> > > >> >> >>>> was
> > > >> >> >>>>> > trying to
> > > >> >> >>>>> > > > >> > >> convey :)
> > > >> >> >>>>> > > > >> > >>
> > > >> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM,
> > Todd
> > > Palino
> > > >> <
> > > >> >> >>>>> > tpalino@gmail.com>
> > > >> >> >>>>> > > > >> wrote:
> > > >> >> >>>>> > > > >> > >> > Well I guess my question for
> you,
> > > then, is
> > > >> >> what
> > > >> >> >>>> is
> > > >> >> >>>>> > holding you
> > > >> >> >>>>> > > > back
> > > >> >> >>>>> > > > >> > from
> > > >> >> >>>>> > > > >> > >> > full support for headers?
> What’s
> > > the bit
> > > >> >> that
> > > >> >> >>>> you’re
> > > >> >> >>>>> > missing
> > > >> >> >>>>> > > that
> > > >> >> >>>>> > > > >> has
> > > >> >> >>>>> > > > >> > you
> > > >> >> >>>>> > > > >> > >> > under a full +1?
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> > -Todd
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> > > Gwen
> > > >> >> Shapira <
> > > >> >> >>>>> > > gwen@confluent.io>
> > > >> >> >>>>> > > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> >> I know why people who support
> > > headers
> > > >> >> support
> > > >> >> >>>> them, and
> > > >> >> >>>>> > I've
> > > >> >> >>>>> > > > seen
> > > >> >> >>>>> > > > >> > what
> > > >> >> >>>>> > > > >> > >> >> the discussion is like.
> > > >> >> >>>>> > > > >> > >> >>
> > > >> >> >>>>> > > > >> > >> >> This is why I'm asking people
> > who
> > > are
> > > >> >> against
> > > >> >> >>>> headers
> > > >> >> >>>>> > > > (especially
> > > >> >> >>>>> > > > >> > >> >> committers) what will make
> them
> > > change
> > > >> >> their
> > > >> >> >>>> mind - so
> > > >> >> >>>>> > we can
> > > >> >> >>>>> > > > get
> > > >> >> >>>>> > > > >> > this
> > > >> >> >>>>> > > > >> > >> >> part over one way or another.
> > > >> >> >>>>> > > > >> > >> >>
> > > >> >> >>>>> > > > >> > >> >> If I sound frustrated it is
> not
> > > at Radai,
> > > >> >> Jun
> > > >> >> >>>> or you
> > > >> >> >>>>> > (Todd)...
> > > >> >> >>>>> > > > I am
> > > >> >> >>>>> > > > >> > >> >> just looking for something
> > > concrete we
> > > >> can
> > > >> >> do
> > > >> >> >>>> to move
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > >> discussion
> > > >> >> >>>>> > > > >> > >> >> along to the yummy design
> > details
> > > (which
> > > >> is
> > > >> >> >> the
> > > >> >> >>>>> > argument I
> > > >> >> >>>>> > > > really
> > > >> >> >>>>> > > > >> am
> > > >> >> >>>>> > > > >> > >> >> looking forward to).
> > > >> >> >>>>> > > > >> > >> >>
> > > >> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53
> PM,
> > > Todd
> > > >> >> Palino <
> > > >> >> >>>>> > > tpalino@gmail.com>
> > > >> >> >>>>> > > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> > So, Gwen, to your question
> > > (even though
> > > >> >> I’m
> > > >> >> >>>> not a
> > > >> >> >>>>> > > > committer)...
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > I have always been a strong
> > > supporter
> > > >> of
> > > >> >> >>>> introducing
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > concept
> > > >> >> >>>>> > > > >> > of an
> > > >> >> >>>>> > > > >> > >> >> > envelope to messages, which
> > > headers
> > > >> >> >>>> accomplishes. The
> > > >> >> >>>>> > > message
> > > >> >> >>>>> > > > key
> > > >> >> >>>>> > > > >> > is
> > > >> >> >>>>> > > > >> > >> >> > already an example of a
> piece
> > of
> > > >> envelope
> > > >> >> >>>>> > information. By
> > > >> >> >>>>> > > > >> > providing a
> > > >> >> >>>>> > > > >> > >> >> means
> > > >> >> >>>>> > > > >> > >> >> > to do this within Kafka
> > itself,
> > > and not
> > > >> >> >>>> relying on
> > > >> >> >>>>> > use-case
> > > >> >> >>>>> > > > >> > specific
> > > >> >> >>>>> > > > >> > >> >> > implementations, you make it
> > > much
> > > >> easier
> > > >> >> for
> > > >> >> >>>>> > components to
> > > >> >> >>>>> > > > >> > >> interoperate.
> > > >> >> >>>>> > > > >> > >> >> It
> > > >> >> >>>>> > > > >> > >> >> > simplifies development of
> all
> > > these
> > > >> >> things
> > > >> >> >>>> (message
> > > >> >> >>>>> > routing,
> > > >> >> >>>>> > > > >> > auditing,
> > > >> >> >>>>> > > > >> > >> >> > encryption, etc.) because
> each
> > > one does
> > > >> >> not
> > > >> >> >>>> have to
> > > >> >> >>>>> > reinvent
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> wheel.
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > It also makes it much easier
> > > from a
> > > >> >> client
> > > >> >> >>>> point of
> > > >> >> >>>>> > view if
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> headers
> > > >> >> >>>>> > > > >> > >> >> are
> > > >> >> >>>>> > > > >> > >> >> > defined as part of the
> > protocol
> > > and/or
> > > >> >> >>>> message format
> > > >> >> >>>>> > in
> > > >> >> >>>>> > > > general
> > > >> >> >>>>> > > > >> > >> because
> > > >> >> >>>>> > > > >> > >> >> > you can easily produce and
> > > consume
> > > >> >> messages
> > > >> >> >>>> without
> > > >> >> >>>>> > having
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> take
> > > >> >> >>>>> > > > >> > >> into
> > > >> >> >>>>> > > > >> > >> >> > account specific cases. For
> > > example, I
> > > >> >> want
> > > >> >> >>>> to route
> > > >> >> >>>>> > > messages,
> > > >> >> >>>>> > > > >> but
> > > >> >> >>>>> > > > >> > >> >> client A
> > > >> >> >>>>> > > > >> > >> >> > doesn’t support the way
> audit
> > > >> implemented
> > > >> >> >>>> headers, and
> > > >> >> >>>>> > > client
> > > >> >> >>>>> > > > B
> > > >> >> >>>>> > > > >> > >> doesn’t
> > > >> >> >>>>> > > > >> > >> >> > support the way encryption
> or
> > > routing
> > > >> >> >>>> implemented
> > > >> >> >>>>> > headers,
> > > >> >> >>>>> > > so
> > > >> >> >>>>> > > > now
> > > >> >> >>>>> > > > >> > my
> > > >> >> >>>>> > > > >> > >> >> > application has to create
> some
> > > really
> > > >> >> >> fragile
> > > >> >> >>>> (my
> > > >> >> >>>>> > > autocorrect
> > > >> >> >>>>> > > > >> just
> > > >> >> >>>>> > > > >> > >> tried
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> > make that “tragic”, which is
> > > probably
> > > >> >> >>>> appropriate
> > > >> >> >>>>> > too) code
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> > strip
> > > >> >> >>>>> > > > >> > >> >> > everything off, rather than
> > just
> > > >> >> consuming
> > > >> >> >> the
> > > >> >> >>>>> > messages,
> > > >> >> >>>>> > > > picking
> > > >> >> >>>>> > > > >> > out
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> 1
> > > >> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested
> > > in, and
> > > >> >> >>>> performing its
> > > >> >> >>>>> > > function.
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > Honestly, this discussion
> has
> > > been
> > > >> going
> > > >> >> on
> > > >> >> >>>> for a
> > > >> >> >>>>> > long time,
> > > >> >> >>>>> > > > and
> > > >> >> >>>>> > > > >> > it’s
> > > >> >> >>>>> > > > >> > >> >> > always “Oh, you came up
> with 2
> > > use
> > > >> cases,
> > > >> >> >> and
> > > >> >> >>>> yeah,
> > > >> >> >>>>> > those
> > > >> >> >>>>> > > use
> > > >> >> >>>>> > > > >> cases
> > > >> >> >>>>> > > > >> > >> are
> > > >> >> >>>>> > > > >> > >> >> > real things that someone
> would
> > > want to
> > > >> >> do.
> > > >> >> >>>> Here’s an
> > > >> >> >>>>> > > alternate
> > > >> >> >>>>> > > > >> way
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > >> >> > implement them so let’s not
> do
> > > >> headers.”
> > > >> >> If
> > > >> >> >>>> we have a
> > > >> >> >>>>> > few
> > > >> >> >>>>> > > use
> > > >> >> >>>>> > > > >> cases
> > > >> >> >>>>> > > > >> > >> that
> > > >> >> >>>>> > > > >> > >> >> we
> > > >> >> >>>>> > > > >> > >> >> > actually came up with, you
> can
> > > be sure
> > > >> >> that
> > > >> >> >>>> over the
> > > >> >> >>>>> > next
> > > >> >> >>>>> > > year
> > > >> >> >>>>> > > > >> > >> there’s a
> > > >> >> >>>>> > > > >> > >> >> > dozen others that we didn’t
> > > think of
> > > >> that
> > > >> >> >>>> someone
> > > >> >> >>>>> > would like
> > > >> >> >>>>> > > > to
> > > >> >> >>>>> > > > >> > do. I
> > > >> >> >>>>> > > > >> > >> >> > really think it’s time to
> stop
> > > >> rehashing
> > > >> >> >> this
> > > >> >> >>>>> > discussion and
> > > >> >> >>>>> > > > >> > instead
> > > >> >> >>>>> > > > >> > >> >> focus
> > > >> >> >>>>> > > > >> > >> >> > on a workable standard that
> we
> > > can
> > > >> adopt.
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > -Todd
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39
> > PM,
> > > Todd
> > > >> >> Palino
> > > >> >> >> <
> > > >> >> >>>>> > > > tpalino@gmail.com>
> > > >> >> >>>>> > > > >> > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> >> C. per message encryption
> > > >> >> >>>>> > > > >> > >> >> >>> One drawback of this
> > approach
> > > is that
> > > >> >> this
> > > >> >> >>>>> > significantly
> > > >> >> >>>>> > > > reduce
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> effectiveness of
> > compression,
> > > which
> > > >> >> >> happens
> > > >> >> >>>> on a
> > > >> >> >>>>> > set of
> > > >> >> >>>>> > > > >> > serialized
> > > >> >> >>>>> > > > >> > >> >> >>> messages. An alternative
> is
> > > to enable
> > > >> >> SSL
> > > >> >> >>>> for wire
> > > >> >> >>>>> > > > encryption
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> rely
> > > >> >> >>>>> > > > >> > >> >> on
> > > >> >> >>>>> > > > >> > >> >> >>> the storage system (e.g.
> > > LUKS) for at
> > > >> >> rest
> > > >> >> >>>>> > encryption.
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >> Jun, this is not
> sufficient.
> > > While
> > > >> this
> > > >> >> >> does
> > > >> >> >>>> cover
> > > >> >> >>>>> > the case
> > > >> >> >>>>> > > > of
> > > >> >> >>>>> > > > >> > >> removing
> > > >> >> >>>>> > > > >> > >> >> a
> > > >> >> >>>>> > > > >> > >> >> >> drive from the system, it
> > will
> > > not
> > > >> >> satisfy
> > > >> >> >>>> most
> > > >> >> >>>>> > compliance
> > > >> >> >>>>> > > > >> > >> requirements
> > > >> >> >>>>> > > > >> > >> >> for
> > > >> >> >>>>> > > > >> > >> >> >> encryption of data as
> whoever
> > > has
> > > >> >> access to
> > > >> >> >>>> the
> > > >> >> >>>>> > broker
> > > >> >> >>>>> > > itself
> > > >> >> >>>>> > > > >> > still
> > > >> >> >>>>> > > > >> > >> has
> > > >> >> >>>>> > > > >> > >> >> >> access to the unencrypted
> > > data. For
> > > >> >> >>>> end-to-end
> > > >> >> >>>>> > encryption
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> > need to
> > > >> >> >>>>> > > > >> > >> >> >> encrypt at the producer,
> > > before it
> > > >> >> enters
> > > >> >> >> the
> > > >> >> >>>>> > system, and
> > > >> >> >>>>> > > > >> decrypt
> > > >> >> >>>>> > > > >> > at
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> >> consumer, after it exits
> the
> > > system.
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >> -Todd
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03
> > > PM, radai
> > > >> <
> > > >> >> >>>>> > > > >> radai.rosenblatt@gmail.com
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > >> >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >>> another big plus of
> headers
> > > in the
> > > >> >> >> protocol
> > > >> >> >>>> is that
> > > >> >> >>>>> > it
> > > >> >> >>>>> > > would
> > > >> >> >>>>> > > > >> > enable
> > > >> >> >>>>> > > > >> > >> >> rapid
> > > >> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside
> > of
> > > core
> > > >> >> kafka
> > > >> >> >>>> and would
> > > >> >> >>>>> > reduce
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> number of
> > > >> >> >>>>> > > > >> > >> >> >>> future wire format changes
> > > required.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> a lot of what is
> currently a
> > > KIP
> > > >> >> >> represents
> > > >> >> >>>> use
> > > >> >> >>>>> > cases that
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> > not
> > > >> >> >>>>> > > > >> > >> 100%
> > > >> >> >>>>> > > > >> > >> >> >>> relevant to all users, and
> > > some of
> > > >> them
> > > >> >> >>>> require
> > > >> >> >>>>> > rather
> > > >> >> >>>>> > > > invasive
> > > >> >> >>>>> > > > >> > wire
> > > >> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing
> a
> > > good
> > > >> recent
> > > >> >> >>>> example of
> > > >> >> >>>>> > this is
> > > >> >> >>>>> > > > >> > kip-98.
> > > >> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is
> > > expected to
> > > >> be
> > > >> >> a
> > > >> >> >>>> very small
> > > >> >> >>>>> > > > fraction of
> > > >> >> >>>>> > > > >> > >> total
> > > >> >> >>>>> > > > >> > >> >> >>> traffic and yet the
> changes
> > > are
> > > >> >> invasive.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> every such wire format
> > change
> > > >> >> translates
> > > >> >> >>>> into
> > > >> >> >>>>> > painful and
> > > >> >> >>>>> > > > slow
> > > >> >> >>>>> > > > >> > >> >> adoption of
> > > >> >> >>>>> > > > >> > >> >> >>> new versions.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> i think a lot of
> > functionality
> > > >> >> currently
> > > >> >> >> in
> > > >> >> >>>> KIPs
> > > >> >> >>>>> > could be
> > > >> >> >>>>> > > > "spun
> > > >> >> >>>>> > > > >> > out"
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> implemented as opt-in
> > plugins
> > > >> >> transmitting
> > > >> >> >>>> data over
> > > >> >> >>>>> > > > headers.
> > > >> >> >>>>> > > > >> > this
> > > >> >> >>>>> > > > >> > >> >> would
> > > >> >> >>>>> > > > >> > >> >> >>> keep the core wire format
> > > stable(r),
> > > >> >> core
> > > >> >> >>>> codebase
> > > >> >> >>>>> > > smaller,
> > > >> >> >>>>> > > > and
> > > >> >> >>>>> > > > >> > >> avoid
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats
> > > sometimes
> > > >> >> required
> > > >> >> >>>> to prove
> > > >> >> >>>>> > a
> > > >> >> >>>>> > > > certain
> > > >> >> >>>>> > > > >> > >> feature
> > > >> >> >>>>> > > > >> > >> >> is
> > > >> >> >>>>> > > > >> > >> >> >>> useful enough for a
> > > wide-enough
> > > >> >> audience
> > > >> >> >> to
> > > >> >> >>>> warrant
> > > >> >> >>>>> > a wire
> > > >> >> >>>>> > > > >> format
> > > >> >> >>>>> > > > >> > >> >> change
> > > >> >> >>>>> > > > >> > >> >> >>> and code complexity
> > additions.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
> > > beyond
> > > >> >> "mere"
> > > >> >> >>>> wire format
> > > >> >> >>>>> > > changes
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > im
> > > >> >> >>>>> > > > >> > >> not
> > > >> >> >>>>> > > > >> > >> >> >>> saying it could have been
> > > completely
> > > >> >> done
> > > >> >> >>>> with
> > > >> >> >>>>> > headers,
> > > >> >> >>>>> > > but
> > > >> >> >>>>> > > > >> > >> >> exactly-once
> > > >> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at
> 11:20
> > > AM, Gwen
> > > >> >> >>>> Shapira <
> > > >> >> >>>>> > > > >> gwen@confluent.io
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > >> >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> > > 10:24 AM,
> > > >> >> radai <
> > > >> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > > "For use cases within
> an
> > > >> >> organization,
> > > >> >> >>>> one could
> > > >> >> >>>>> > > always
> > > >> >> >>>>> > > > use
> > > >> >> >>>>> > > > >> > >> other
> > > >> >> >>>>> > > > >> > >> >> >>> > > approaches such as
> > > company-wise
> > > >> >> >>>> containers"
> > > >> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin
> > has
> > > >> >> >> traditionally
> > > >> >> >>>> done
> > > >> >> >>>>> > but there
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> > now
> > > >> >> >>>>> > > > >> > >> >> cases
> > > >> >> >>>>> > > > >> > >> >> >>> > (read
> > > >> >> >>>>> > > > >> > >> >> >>> > > - topics) where this
> is
> > > not
> > > >> >> >> acceptable.
> > > >> >> >>>> this
> > > >> >> >>>>> > makes
> > > >> >> >>>>> > > > headers
> > > >> >> >>>>> > > > >> > >> useful
> > > >> >> >>>>> > > > >> > >> >> even
> > > >> >> >>>>> > > > >> > >> >> >>> > > within single orgs for
> > > cases
> > > >> where
> > > >> >> >>>>> > > > one-container-fits-all
> > > >> >> >>>>> > > > >> > cannot
> > > >> >> >>>>> > > > >> > >> >> >>> apply.
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > > as for the particular
> > use
> > > cases
> > > >> >> >> listed,
> > > >> >> >>>> i dont
> > > >> >> >>>>> > want
> > > >> >> >>>>> > > > this to
> > > >> >> >>>>> > > > >> > >> devolve
> > > >> >> >>>>> > > > >> > >> >> >>> to a
> > > >> >> >>>>> > > > >> > >> >> >>> > > discussion of
> particular
> > > use
> > > >> cases
> > > >> >> - i
> > > >> >> >>>> think its
> > > >> >> >>>>> > > enough
> > > >> >> >>>>> > > > >> that
> > > >> >> >>>>> > > > >> > >> some
> > > >> >> >>>>> > > > >> > >> >> of
> > > >> >> >>>>> > > > >> > >> >> >>> them
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I think a main point of
> > > contention
> > > >> is
> > > >> >> >>>> that: We
> > > >> >> >>>>> > > identified
> > > >> >> >>>>> > > > few
> > > >> >> >>>>> > > > >> > >> >> >>> > use-cases where headers
> > are
> > > useful,
> > > >> >> do
> > > >> >> >> we
> > > >> >> >>>> want
> > > >> >> >>>>> > Kafka to
> > > >> >> >>>>> > > > be a
> > > >> >> >>>>> > > > >> > >> system
> > > >> >> >>>>> > > > >> > >> >> >>> > that supports those
> > > use-cases?
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
> > > >> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely
> > useful
> > > >> >> record-level
> > > >> >> >>>> lineage
> > > >> >> >>>>> > is
> > > >> >> >>>>> > > though
> > > >> >> >>>>> > > > >> > since
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > overhead could
> > > >> >> >>>>> > > > >> > >> >> >>> > be significant."
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports
> > > record level
> > > >> >> >>>> lineage. I
> > > >> >> >>>>> > don't
> > > >> >> >>>>> > > think
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > was
> > > >> >> >>>>> > > > >> > >> >> >>> > developed for lols, I
> > think
> > > it is
> > > >> >> safe
> > > >> >> >> to
> > > >> >> >>>> assume
> > > >> >> >>>>> > that
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > NSA
> > > >> >> >>>>> > > > >> > >> needed
> > > >> >> >>>>> > > > >> > >> >> >>> > that functionality. We
> > also
> > > know
> > > >> that
> > > >> >> >>>> certain
> > > >> >> >>>>> > financial
> > > >> >> >>>>> > > > >> > institutes
> > > >> >> >>>>> > > > >> > >> >> >>> > need to track tampering
> > with
> > > >> records
> > > >> >> at
> > > >> >> >> a
> > > >> >> >>>> record
> > > >> >> >>>>> > level
> > > >> >> >>>>> > > and
> > > >> >> >>>>> > > > >> > there
> > > >> >> >>>>> > > > >> > >> are
> > > >> >> >>>>> > > > >> > >> >> >>> > federal regulations that
> > > absolutely
> > > >> >> >>>> require
> > > >> >> >>>>> > this. They
> > > >> >> >>>>> > > > also
> > > >> >> >>>>> > > > >> > need
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > prove that routing apps
> > that
> > > >> >> "touches"
> > > >> >> >> the
> > > >> >> >>>>> > messages and
> > > >> >> >>>>> > > > >> either
> > > >> >> >>>>> > > > >> > >> reads
> > > >> >> >>>>> > > > >> > >> >> >>> > or updates headers
> > couldn't
> > > have
> > > >> >> >> possibly
> > > >> >> >>>>> > modified the
> > > >> >> >>>>> > > > >> payload
> > > >> >> >>>>> > > > >> > >> >> itself.
> > > >> >> >>>>> > > > >> > >> >> >>> > They use record level
> > > encryption to
> > > >> >> do
> > > >> >> >>>> that -
> > > >> >> >>>>> > apps can
> > > >> >> >>>>> > > > read
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify
> headers
> > > but
> > > >> can't
> > > >> >> >>>> touch the
> > > >> >> >>>>> > payload.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > We can totally say
> "those
> > > are
> > > >> corner
> > > >> >> >>>> cases and
> > > >> >> >>>>> > not worth
> > > >> >> >>>>> > > > >> adding
> > > >> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for",
> > they
> > > should
> > > >> >> use a
> > > >> >> >>>> different
> > > >> >> >>>>> > > pubsub
> > > >> >> >>>>> > > > >> > message
> > > >> >> >>>>> > > > >> > >> for
> > > >> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the
> > > other 1000
> > > >> >> that
> > > >> >> >>>> cater
> > > >> >> >>>>> > > > specifically
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > financial industry).
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > But this gets us into a
> > > catch 22:
> > > >> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific
> > > use-case,
> > > >> >> >>>> someone can
> > > >> >> >>>>> > always
> > > >> >> >>>>> > > say
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > isn't
> > > >> >> >>>>> > > > >> > >> >> >>> > interesting enough for
> > > Kafka. If we
> > > >> >> >>>> discuss more
> > > >> >> >>>>> > general
> > > >> >> >>>>> > > > >> > trends,
> > > >> >> >>>>> > > > >> > >> >> >>> > others can say "well, we
> > > are not
> > > >> sure
> > > >> >> >> any
> > > >> >> >>>> of them
> > > >> >> >>>>> > really
> > > >> >> >>>>> > > > >> needs
> > > >> >> >>>>> > > > >> > >> >> headers
> > > >> >> >>>>> > > > >> > >> >> >>> > specifically. This is
> just
> > > hand
> > > >> >> waving
> > > >> >> >>>> and not
> > > >> >> >>>>> > > > interesting.".
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I think discussing
> > > use-cases in
> > > >> >> >> specifics
> > > >> >> >>>> is super
> > > >> >> >>>>> > > > important
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> >> decide
> > > >> >> >>>>> > > > >> > >> >> >>> > implementation details
> for
> > > headers
> > > >> >> (my
> > > >> >> >>>> use-cases
> > > >> >> >>>>> > lean
> > > >> >> >>>>> > > > toward
> > > >> >> >>>>> > > > >> > >> >> numerical
> > > >> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and
> > > object
> > > >> >> values,
> > > >> >> >>>> others
> > > >> >> >>>>> > differ),
> > > >> >> >>>>> > > > but I
> > > >> >> >>>>> > > > >> > >> think
> > > >> >> >>>>> > > > >> > >> >> we
> > > >> >> >>>>> > > > >> > >> >> >>> > need to answer the
> general
> > > "Are we
> > > >> >> going
> > > >> >> >>>> to have
> > > >> >> >>>>> > > headers"
> > > >> >> >>>>> > > > >> > question
> > > >> >> >>>>> > > > >> > >> >> >>> > first.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from
> the
> > > other
> > > >> >> >>>> committers in the
> > > >> >> >>>>> > > > discussion:
> > > >> >> >>>>> > > > >> > >> >> >>> > What would it take to
> > > convince you
> > > >> >> that
> > > >> >> >>>> headers
> > > >> >> >>>>> > in Kafka
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> a
> > > >> >> >>>>> > > > >> > >> good
> > > >> >> >>>>> > > > >> > >> >> >>> > idea in general, so we
> can
> > > move
> > > >> ahead
> > > >> >> >> and
> > > >> >> >>>> try to
> > > >> >> >>>>> > agree
> > > >> >> >>>>> > > on
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> >> details?
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I feel like we keep
> moving
> > > the goal
> > > >> >> >> posts
> > > >> >> >>>> and
> > > >> >> >>>>> > this is
> > > >> >> >>>>> > > > truly
> > > >> >> >>>>> > > > >> > >> >> exhausting.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly
> > > support
> > > >> >> adding
> > > >> >> >>>> headers
> > > >> >> >>>>> > to Kafka
> > > >> >> >>>>> > > > >> > (+0.5?).
> > > >> >> >>>>> > > > >> > >> >> >>> > The community can
> continue
> > > to find
> > > >> >> >>>> workarounds to
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > issue
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> there
> > > >> >> >>>>> > > > >> > >> >> >>> > are some benefits to
> > > keeping the
> > > >> >> message
> > > >> >> >>>> format
> > > >> >> >>>>> > and
> > > >> >> >>>>> > > > clients
> > > >> >> >>>>> > > > >> > >> simpler.
> > > >> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness
> > of
> > > headers
> > > >> >> to
> > > >> >> >>>> many
> > > >> >> >>>>> > use-cases
> > > >> >> >>>>> > > and
> > > >> >> >>>>> > > > if
> > > >> >> >>>>> > > > >> we
> > > >> >> >>>>> > > > >> > >> can
> > > >> >> >>>>> > > > >> > >> >> >>> > find a good and
> generally
> > > useful
> > > >> way
> > > >> >> to
> > > >> >> >>>> add it to
> > > >> >> >>>>> > Kafka,
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > will
> > > >> >> >>>>> > > > >> > >> make
> > > >> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for
> > > many -
> > > >> worthy
> > > >> >> >>>> goal in my
> > > >> >> >>>>> > eyes.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > > are
> > interesting/feasible,
> > > but:
> > > >> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are
> > > use cases
> > > >> >> for
> > > >> >> >>>> polyglot
> > > >> >> >>>>> > topics.
> > > >> >> >>>>> > > > >> > >> especially if
> > > >> >> >>>>> > > > >> > >> >> >>> kafka
> > > >> >> >>>>> > > > >> > >> >> >>> > > is being used to
> "trunk"
> > > >> something
> > > >> >> >> else.
> > > >> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics
> would
> > > make it
> > > >> >> >> harder
> > > >> >> >>>> to write
> > > >> >> >>>>> > > > portable
> > > >> >> >>>>> > > > >> > >> consumer
> > > >> >> >>>>> > > > >> > >> >> >>> code.
> > > >> >> >>>>> > > > >> > >> >> >>> > > partition remapping
> > would
> > > mess
> > > >> with
> > > >> >> >>>> locality of
> > > >> >> >>>>> > > > consumption
> > > >> >> >>>>> > > > >> > >> >> >>> guarantees.
> > > >> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see
> > for
> > > >> >> >>>> lineage/metadata is
> > > >> >> >>>>> > > > >> > >> billing/chargeback.
> > > >> >> >>>>> > > > >> > >> >> for
> > > >> >> >>>>> > > > >> > >> >> >>> > that
> > > >> >> >>>>> > > > >> > >> >> >>> > > use case it is not
> > enough
> > > to
> > > >> simply
> > > >> >> >>>> record the
> > > >> >> >>>>> > point
> > > >> >> >>>>> > > of
> > > >> >> >>>>> > > > >> > origin,
> > > >> >> >>>>> > > > >> > >> but
> > > >> >> >>>>> > > > >> > >> >> >>> every
> > > >> >> >>>>> > > > >> > >> >> >>> > > replication stop
> (think
> > > mirror
> > > >> >> maker)
> > > >> >> >>>> must also
> > > >> >> >>>>> > add a
> > > >> >> >>>>> > > > >> record
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > >> >> form a
> > > >> >> >>>>> > > > >> > >> >> >>> > > "transit log".
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > > as for stream
> processing
> > > on top
> > > >> of
> > > >> >> >>>> kafka - i
> > > >> >> >>>>> > know
> > > >> >> >>>>> > > samza
> > > >> >> >>>>> > > > >> has a
> > > >> >> >>>>> > > > >> > >> >> metadata
> > > >> >> >>>>> > > > >> > >> >> >>> > map
> > > >> >> >>>>> > > > >> > >> >> >>> > > which they carry
> around
> > in
> > > >> >> addition to
> > > >> >> >>>> user
> > > >> >> >>>>> > values.
> > > >> >> >>>>> > > > headers
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > perfect
> > > >> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016
> at
> > > 6:50 PM,
> > > >> >> Jun
> > > >> >> >>>> Rao <
> > > >> >> >>>>> > > > jun@confluent.io
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> In order to answer
> the
> > > first two
> > > >> >> >>>> questions, it
> > > >> >> >>>>> > would
> > > >> >> >>>>> > > be
> > > >> >> >>>>> > > > >> > helpful
> > > >> >> >>>>> > > > >> > >> >> if we
> > > >> >> >>>>> > > > >> > >> >> >>> > could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2
> strong
> > > use cases
> > > >> >> for
> > > >> >> >>>> headers
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > space
> > > >> >> >>>>> > > > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > third-party
> > > >> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use
> cases
> > > within an
> > > >> >> >>>> organization,
> > > >> >> >>>>> > one
> > > >> >> >>>>> > > > could
> > > >> >> >>>>> > > > >> > always
> > > >> >> >>>>> > > > >> > >> >> use
> > > >> >> >>>>> > > > >> > >> >> >>> > other
> > > >> >> >>>>> > > > >> > >> >> >>> > >> approaches such as
> > > company-wise
> > > >> >> >>>> containers to
> > > >> >> >>>>> > get
> > > >> >> >>>>> > > > around
> > > >> >> >>>>> > > > >> w/o
> > > >> >> >>>>> > > > >> > >> >> >>> headers. I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> went through the use
> > > cases in
> > > >> the
> > > >> >> KIP
> > > >> >> >>>> and in
> > > >> >> >>>>> > Radai's
> > > >> >> >>>>> > > > wiki
> > > >> >> >>>>> > > > >> (
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > https://cwiki.apache.org/confl
> > > >> >> >>>>> > uence/display/KAFKA/A+
> > > >> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> ).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> The following are the
> > > ones that
> > > >> >> that
> > > >> >> >> I
> > > >> >> >>>>> > understand and
> > > >> >> >>>>> > > > >> could
> > > >> >> >>>>> > > > >> > be
> > > >> >> >>>>> > > > >> > >> in
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> third-party use case
> > > category.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
> > > >> >> >>>>> > > > >> > >> >> >>> > >> It seems that in
> > general,
> > > >> >> >> content-type
> > > >> >> >>>> should
> > > >> >> >>>>> > be set
> > > >> >> >>>>> > > at
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> topic
> > > >> >> >>>>> > > > >> > >> >> >>> level.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing
> > > messages with
> > > >> >> >>>> different
> > > >> >> >>>>> > content
> > > >> >> >>>>> > > > types
> > > >> >> >>>>> > > > >> > >> should be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Since the value is
> > mostly
> > > >> useless
> > > >> >> >>>> without
> > > >> >> >>>>> > schema id,
> > > >> >> >>>>> > > it
> > > >> >> >>>>> > > > >> > seems
> > > >> >> >>>>> > > > >> > >> that
> > > >> >> >>>>> > > > >> > >> >> >>> > storing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the schema id
> together
> > > with
> > > >> >> >> serialized
> > > >> >> >>>> bytes
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > value
> > > >> >> >>>>> > > > >> is
> > > >> >> >>>>> > > > >> > >> >> better?
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> C. per message
> > encryption
> > > >> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this
> > > approach is
> > > >> >> that
> > > >> >> >>>> this
> > > >> >> >>>>> > > > significantly
> > > >> >> >>>>> > > > >> > reduce
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of
> > > compression,
> > > >> >> which
> > > >> >> >>>> happens on
> > > >> >> >>>>> > a set
> > > >> >> >>>>> > > of
> > > >> >> >>>>> > > > >> > >> serialized
> > > >> >> >>>>> > > > >> > >> >> >>> > >> messages. An
> > alternative
> > > is to
> > > >> >> enable
> > > >> >> >>>> SSL for
> > > >> >> >>>>> > wire
> > > >> >> >>>>> > > > >> > encryption
> > > >> >> >>>>> > > > >> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> rely
> > > >> >> >>>>> > > > >> > >> >> >>> > on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the storage system
> > (e.g.
> > > LUKS)
> > > >> >> for at
> > > >> >> >>>> rest
> > > >> >> >>>>> > > encryption.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for
> > > mirroring
> > > >> across
> > > >> >> >>>> Kafka
> > > >> >> >>>>> > clusters
> > > >> >> >>>>> > > > >> > >> >> >>> > >> This is actually
> > > interesting.
> > > >> >> Today,
> > > >> >> >>>> to avoid
> > > >> >> >>>>> > > > introducing
> > > >> >> >>>>> > > > >> > >> cycles
> > > >> >> >>>>> > > > >> > >> >> when
> > > >> >> >>>>> > > > >> > >> >> >>> > doing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data
> > > centers,
> > > >> one
> > > >> >> >>>> would
> > > >> >> >>>>> > either have
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> set
> > > >> >> >>>>> > > > >> > up
> > > >> >> >>>>> > > > >> > >> two
> > > >> >> >>>>> > > > >> > >> >> >>> Kafka
> > > >> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and
> > an
> > > >> >> aggregate)
> > > >> >> >>>> per data
> > > >> >> >>>>> > center
> > > >> >> >>>>> > > or
> > > >> >> >>>>> > > > >> > rename
> > > >> >> >>>>> > > > >> > >> >> topics.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal.
> With
> > > headers,
> > > >> >> the
> > > >> >> >>>> producer
> > > >> >> >>>>> > could
> > > >> >> >>>>> > > tag
> > > >> >> >>>>> > > > >> each
> > > >> >> >>>>> > > > >> > >> >> message
> > > >> >> >>>>> > > > >> > >> >> >>> with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster
> > ID
> > > in the
> > > >> >> >> header.
> > > >> >> >>>>> > MirrorMaker
> > > >> >> >>>>> > > > could
> > > >> >> >>>>> > > > >> > then
> > > >> >> >>>>> > > > >> > >> >> avoid
> > > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages
> to a
> > > cluster
> > > >> if
> > > >> >> >>>> they are
> > > >> >> >>>>> > tagged
> > > >> >> >>>>> > > with
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> same
> > > >> >> >>>>> > > > >> > >> >> >>> cluster
> > > >> >> >>>>> > > > >> > >> >> >>> > >> id.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> However, an
> alternative
> > > approach
> > > >> >> is
> > > >> >> >> to
> > > >> >> >>>>> > introduce sth
> > > >> >> >>>>> > > > like
> > > >> >> >>>>> > > > >> > >> >> >>> hierarchical
> > > >> >> >>>>> > > > >> > >> >> >>> > >> topic and store
> > messages
> > > from
> > > >> >> >> different
> > > >> >> >>>>> > clusters in
> > > >> >> >>>>> > > > >> > different
> > > >> >> >>>>> > > > >> > >> >> >>> partitions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> under the same topic.
> > > This
> > > >> >> approach
> > > >> >> >>>> avoids
> > > >> >> >>>>> > filtering
> > > >> >> >>>>> > > > out
> > > >> >> >>>>> > > > >> > >> unneeded
> > > >> >> >>>>> > > > >> > >> >> >>> data
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> makes offset
> preserving
> > > easier
> > > >> to
> > > >> >> >>>> support. It
> > > >> >> >>>>> > may
> > > >> >> >>>>> > > make
> > > >> >> >>>>> > > > >> > >> compaction
> > > >> >> >>>>> > > > >> > >> >> >>> > trickier
> > > >> >> >>>>> > > > >> > >> >> >>> > >> though since the same
> > > key may
> > > >> >> show up
> > > >> >> >>>> in
> > > >> >> >>>>> > different
> > > >> >> >>>>> > > > >> > partitions.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> E. record-level
> lineage
> > > >> >> >>>>> > > > >> > >> >> >>> > >> For example, a source
> > > connector
> > > >> >> could
> > > >> >> >>>> store in
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > message
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> metadata
> > > >> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the
> > source
> > > >> record.
> > > >> >> >>>> Similarly,
> > > >> >> >>>>> > if a
> > > >> >> >>>>> > > > stream
> > > >> >> >>>>> > > > >> job
> > > >> >> >>>>> > > > >> > >> >> >>> transforms
> > > >> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A
> > to
> > > topic
> > > >> B,
> > > >> >> the
> > > >> >> >>>> library
> > > >> >> >>>>> > could
> > > >> >> >>>>> > > > >> include
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> source
> > > >> >> >>>>> > > > >> > >> >> >>> > >> message offset in
> each
> > > of the
> > > >> >> >>>> transformed
> > > >> >> >>>>> > message in
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> header.
> > > >> >> >>>>> > > > >> > >> >> Not
> > > >> >> >>>>> > > > >> > >> >> >>> > sure
> > > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> how widely useful
> > > record-level
> > > >> >> >> lineage
> > > >> >> >>>> is
> > > >> >> >>>>> > though
> > > >> >> >>>>> > > since
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> >> overhead
> > > >> >> >>>>> > > > >> > >> >> >>> > could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> be significant.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> > > >> >> >>>>> > > > >> > >> >> >>> > >> We could put things
> > like
> > > >> >> >>>> clientId/host/user in
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > header
> > > >> >> >>>>> > > > >> in
> > > >> >> >>>>> > > > >> > >> each
> > > >> >> >>>>> > > > >> > >> >> >>> > message
> > > >> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These
> > > metadata are
> > > >> >> >>>> really at the
> > > >> >> >>>>> > > producer
> > > >> >> >>>>> > > > >> > level
> > > >> >> >>>>> > > > >> > >> >> though.
> > > >> >> >>>>> > > > >> > >> >> >>> > So, a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is
> > to
> > > only
> > > >> >> >> include a
> > > >> >> >>>>> > "producerId"
> > > >> >> >>>>> > > > per
> > > >> >> >>>>> > > > >> > >> message
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > send
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the producerId ->
> > > metadata
> > > >> mapping
> > > >> >> >>>>> > independently.
> > > >> >> >>>>> > > > KIP-98
> > > >> >> >>>>> > > > >> is
> > > >> >> >>>>> > > > >> > >> >> actually
> > > >> >> >>>>> > > > >> > >> >> >>> > >> proposing including
> > such
> > > a
> > > >> >> producerId
> > > >> >> >>>> natively
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > >> > message.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not
> sure
> > > that I
> > > >> am
> > > >> >> >> fully
> > > >> >> >>>>> > convinced of
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > strong
> > > >> >> >>>>> > > > >> > >> >> >>> > third-party
> > > >> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers
> > yet.
> > > >> Perhaps
> > > >> >> we
> > > >> >> >>>> could
> > > >> >> >>>>> > discuss a
> > > >> >> >>>>> > > > bit
> > > >> >> >>>>> > > > >> > more
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> make
> > > >> >> >>>>> > > > >> > >> >> >>> > one
> > > >> >> >>>>> > > > >> > >> >> >>> > >> or two really
> > convincing
> > > use
> > > >> >> cases.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal
> > > question is
> > > >> >> >>>> whether header
> > > >> >> >>>>> > should
> > > >> >> >>>>> > > > be
> > > >> >> >>>>> > > > >> > >> exposed
> > > >> >> >>>>> > > > >> > >> >> in
> > > >> >> >>>>> > > > >> > >> >> >>> > stream
> > > >> >> >>>>> > > > >> > >> >> >>> > >> processing systems
> such
> > > Kafka
> > > >> >> stream,
> > > >> >> >>>> Samza,
> > > >> >> >>>>> > and
> > > >> >> >>>>> > > Spark
> > > >> >> >>>>> > > > >> > >> streaming.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Currently, those
> > systems
> > > just
> > > >> deal
> > > >> >> >> with
> > > >> >> >>>>> > key/value
> > > >> >> >>>>> > > > pairs.
> > > >> >> >>>>> > > > >> > >> Should we
> > > >> >> >>>>> > > > >> > >> >> >>> > expose a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> third thing header
> > there
> > > too or
> > > >> >> >>>> somehow map
> > > >> >> >>>>> > header to
> > > >> >> >>>>> > > > key
> > > >> >> >>>>> > > > >> or
> > > >> >> >>>>> > > > >> > >> >> value?
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Jun
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016
> at
> > > 3:35 AM,
> > > >> >> >>>> Michael
> > > >> >> >>>>> > Pearce <
> > > >> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that
> after
> > a
> > > period
> > > >> >> of a
> > > >> >> >>>> week,
> > > >> >> >>>>> > that there
> > > >> >> >>>>> > > > is
> > > >> >> >>>>> > > > >> no
> > > >> >> >>>>> > > > >> > >> >> concerns
> > > >> >> >>>>> > > > >> > >> >> >>> now
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and
> 2
> > > and now
> > > >> we
> > > >> >> >> have
> > > >> >> >>>>> > agreement that
> > > >> >> >>>>> > > > >> > headers
> > > >> >> >>>>> > > > >> > >> are
> > > >> >> >>>>> > > > >> > >> >> >>> useful
> > > >> >> >>>>> > > > >> > >> >> >>> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
> > > such if
> > > >> put
> > > >> >> to
> > > >> >> >> a
> > > >> >> >>>> KIP
> > > >> >> >>>>> > vote, this
> > > >> >> >>>>> > > > >> > wouldn’t
> > > >> >> >>>>> > > > >> > >> be
> > > >> >> >>>>> > > > >> > >> >> a
> > > >> >> >>>>> > > > >> > >> >> >>> > reason
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > reject.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > @
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point
> 4).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose
> > of
> > > getting
> > > >> >> this
> > > >> >> >>>> KIP
> > > >> >> >>>>> > moving past
> > > >> >> >>>>> > > > >> this,
> > > >> >> >>>>> > > > >> > we
> > > >> >> >>>>> > > > >> > >> can
> > > >> >> >>>>> > > > >> > >> >> >>> state
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4
> bytes
> > > space
> > > >> that
> > > >> >> >> can
> > > >> >> >>>> will be
> > > >> >> >>>>> > > > naturally
> > > >> >> >>>>> > > > >> > >> >> interpreted
> > > >> >> >>>>> > > > >> > >> >> >>> as
> > > >> >> >>>>> > > > >> > >> >> >>> > an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if
> namespacing
> > > is later
> > > >> >> >>>> wanted you can
> > > >> >> >>>>> > > easily
> > > >> >> >>>>> > > > >> split
> > > >> >> >>>>> > > > >> > >> this
> > > >> >> >>>>> > > > >> > >> >> >>> into
> > > >> >> >>>>> > > > >> > >> >> >>> > two
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from
> > > the wire
> > > >> >> >> protocol
> > > >> >> >>>>> > > implementation
> > > >> >> >>>>> > > > >> this
> > > >> >> >>>>> > > > >> > >> makes
> > > >> >> >>>>> > > > >> > >> >> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t
> > > believe. Is
> > > >> >> this
> > > >> >> >>>>> > reasonable to
> > > >> >> >>>>> > > > all?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point
> 4
> > > therefor
> > > >> >> happy
> > > >> >> >>>> we keep
> > > >> >> >>>>> > with 32
> > > >> >> >>>>> > > > >> bits.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016,
> > 20:34, "
> > > >> >> >>>>> > ignacio.solis@gmail.com on
> > > >> >> >>>>> > > > >> behalf
> > > >> >> >>>>> > > > >> > of
> > > >> >> >>>>> > > > >> > >> >> >>> Ignacio
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
> > > >> ignacio.solis@gmail.com
> > > >> >> on
> > > >> >> >>>> behalf of
> > > >> >> >>>>> > > > >> > isolis@igso.net
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
> > > value as
> > > >> >> byte[]
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int -
> No
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String -
> > > Reluctant
> > > >> maybe
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe
> the
> > > header
> > > >> >> system
> > > >> >> >>>> should
> > > >> >> >>>>> > take a
> > > >> >> >>>>> > > > single
> > > >> >> >>>>> > > > >> > >> int. I
> > > >> >> >>>>> > > > >> > >> >> >>> think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if
> > > you want
> > > >> to
> > > >> >> >>>> interpret
> > > >> >> >>>>> > this as
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> 16bit
> > > >> >> >>>>> > > > >> > >> >> numbers
> > > >> >> >>>>> > > > >> > >> >> >>> in
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > layer
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > above go right
> > > ahead. If
> > > >> >> >>>> somebody wants
> > > >> >> >>>>> > to
> > > >> >> >>>>> > > argue
> > > >> >> >>>>> > > > >> for
> > > >> >> >>>>> > > > >> > 16
> > > >> >> >>>>> > > > >> > >> >> bits
> > > >> >> >>>>> > > > >> > >> >> >>> or
> > > >> >> >>>>> > > > >> > >> >> >>> > 64
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > bits of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > header key
> space
> > I
> > > would
> > > >> >> >> listen.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the
> key
> > > space
> > > >> into
> > > >> >> >>>> sub_key_1 and
> > > >> >> >>>>> > > > sub_key_2
> > > >> >> >>>>> > > > >> > >> makes no
> > > >> >> >>>>> > > > >> > >> >> >>> > sense to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > me at
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this layer.
> Are
> > > we going
> > > >> to
> > > >> >> >>>> start
> > > >> >> >>>>> > providing
> > > >> >> >>>>> > > > APIs to
> > > >> >> >>>>> > > > >> > get
> > > >> >> >>>>> > > > >> > >> all
> > > >> >> >>>>> > > > >> > >> >> >>> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or
> > all
> > > the
> > > >> >> >>>> sub_key_2s? If
> > > >> >> >>>>> > there is
> > > >> >> >>>>> > > > no
> > > >> >> >>>>> > > > >> > >> >> >>> distinguishing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > functions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > that are
> applied
> > > to each
> > > >> one
> > > >> >> >>>> then they
> > > >> >> >>>>> > should
> > > >> >> >>>>> > > be
> > > >> >> >>>>> > > > a
> > > >> >> >>>>> > > > >> > single
> > > >> >> >>>>> > > > >> > >> >> >>> value.
> > > >> >> >>>>> > > > >> > >> >> >>> > At
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're
> > > doing is
> > > >> >> >>>> equality.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > If the above
> > layer
> > > wants
> > > >> to
> > > >> >> >>>> interpret
> > > >> >> >>>>> > this as
> > > >> >> >>>>> > > 2,
> > > >> >> >>>>> > > > 3
> > > >> >> >>>>> > > > >> or
> > > >> >> >>>>> > > > >> > >> more
> > > >> >> >>>>> > > > >> > >> >> >>> values
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > that's a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > different
> > > question. I
> > > >> >> >>>> personally think
> > > >> >> >>>>> > it's
> > > >> >> >>>>> > > all
> > > >> >> >>>>> > > > one
> > > >> >> >>>>> > > > >> > >> >> keyspace
> > > >> >> >>>>> > > > >> > >> >> >>> > that is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > getting
> assigned
> > > using
> > > >> some
> > > >> >> >>>> structure,
> > > >> >> >>>>> > but if
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> > want to
> > > >> >> >>>>> > > > >> > >> >> >>> > sub-assign
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > parts
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > of it then
> that's
> > > fine.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > The same
> > discussion
> > > >> applies
> > > >> >> to
> > > >> >> >>>> strings.
> > > >> >> >>>>> > If
> > > >> >> >>>>> > > > somebody
> > > >> >> >>>>> > > > >> > >> argued
> > > >> >> >>>>> > > > >> > >> >> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > strings,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > would we be
> > > arguing to
> > > >> >> divide
> > > >> >> >> the
> > > >> >> >>>>> > strings with
> > > >> >> >>>>> > > > dots
> > > >> >> >>>>> > > > >> > ('.')
> > > >> >> >>>>> > > > >> > >> >> as a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we want
> > them
> > > to give
> > > >> >> us
> > > >> >> >> the
> > > >> >> >>>>> > different
> > > >> >> >>>>> > > name
> > > >> >> >>>>> > > > >> > segments
> > > >> >> >>>>> > > > >> > >> >> >>> > separately?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we be
> > > performing any
> > > >> >> >>>> actions on
> > > >> >> >>>>> > this key
> > > >> >> >>>>> > > > other
> > > >> >> >>>>> > > > >> > than
> > > >> >> >>>>> > > > >> > >> >> >>> > matching?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
> > > 2016 at
> > > >> 9:30
> > > >> >> >> AM,
> > > >> >> >>>> Michael
> > > >> >> >>>>> > > Pearce <
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> Michael.Pearce@ig.com
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> > > concerns
> > > >> on
> > > >> >> 1
> > > >> >> >>>> and 2
> > > >> >> >>>>> > still?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > @all
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > To get this
> > > moving
> > > >> along a
> > > >> >> >> bit
> > > >> >> >>>> more
> > > >> >> >>>>> > I'd also
> > > >> >> >>>>> > > > like
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> ask
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> >>> get
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the below
> last
> > > points:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe
> > > we're all
> > > >> >> >> roughly
> > > >> >> >>>> happy
> > > >> >> >>>>> > with the
> > > >> >> >>>>> > > > >> header
> > > >> >> >>>>> > > > >> > >> value
> > > >> >> >>>>> > > > >> > >> >> >>> > being a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> > > consensus
> > > >> has
> > > >> >> >>>> been for an
> > > >> >> >>>>> > > > namespace
> > > >> >> >>>>> > > > >> > based
> > > >> >> >>>>> > > > >> > >> int
> > > >> >> >>>>> > > > >> > >> >> >>> > approach
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for
> > > the key.
> > > >> Any
> > > >> >> >>>> objections
> > > >> >> >>>>> > if this
> > > >> >> >>>>> > > > is
> > > >> >> >>>>> > > > >> > what
> > > >> >> >>>>> > > > >> > >> we
> > > >> >> >>>>> > > > >> > >> >> go
> > > >> >> >>>>> > > > >> > >> >> >>> > with?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have
> > if
> > > >> >> assumption
> > > >> >> >> in
> > > >> >> >>>> (4) is
> > > >> >> >>>>> > > correct,
> > > >> >> >>>>> > > > >> > >> {int,int}
> > > >> >> >>>>> > > > >> > >> >> >>> keys.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Should both
> > > int's be
> > > >> >> int16 or
> > > >> >> >>>> int32?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them
> > > being
> > > >> int16(2
> > > >> >> >>>> bytes) as
> > > >> >> >>>>> > combined
> > > >> >> >>>>> > > > is
> > > >> >> >>>>> > > > >> > space
> > > >> >> >>>>> > > > >> > >> of
> > > >> >> >>>>> > > > >> > >> >> >>> > 4bytes as
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > per
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > original and
> > > gives
> > > >> plenty
> > > >> >> of
> > > >> >> >>>>> > combinations for
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> >> >>> foreseeable,
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead
> > > small.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any
> > > benefit in
> > > >> >> >>>> another kip
> > > >> >> >>>>> > call to
> > > >> >> >>>>> > > > >> discuss
> > > >> >> >>>>> > > > >> > >> >> these at
> > > >> >> >>>>> > > > >> > >> >> >>> > all?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >> ______________________________
> > > >> >> >>>>> > __________
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > From: K
> > Burstev <
> > > >> >> >>>> k.burstev@yandex.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> > > November
> > > >> 18,
> > > >> >> >> 2016
> > > >> >> >>>>> > 7:07:07 AM
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > To:
> > > >> dev@kafka.apache.org
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> > > [DISCUSS]
> > > >> >> KIP-82
> > > >> >> >>>> - Add
> > > >> >> >>>>> > Record
> > > >> >> >>>>> > > > Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > For what it
> is
> > > worth
> > > >> also
> > > >> >> i
> > > >> >> >>>> agree. As
> > > >> >> >>>>> > a user:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
> > > Headers are
> > > >> >> >>>> worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes -
> > > Headers should
> > > >> >> be a
> > > >> >> >>>> top level
> > > >> >> >>>>> > > option
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
> > > 21:15,
> > > >> >> "Ignacio
> > > >> >> >>>> Solis" <
> > > >> >> >>>>> > > > >> isolis@igso.net
> > > >> >> >>>>> > > > >> > >:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
> > > Headers are
> > > >> >> >>>> worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
> > > Headers
> > > >> should
> > > >> >> be
> > > >> >> >> a
> > > >> >> >>>> top
> > > >> >> >>>>> > level
> > > >> >> >>>>> > > option
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov
> > > 14, 2016
> > > >> at
> > > >> >> >> 9:16
> > > >> >> >>>> AM,
> > > >> >> >>>>> > Michael
> > > >> >> >>>>> > > > Pearce
> > > >> >> >>>>> > > > >> <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > Michael.Pearce@ig.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
> > > >> >> details/examples
> > > >> >> >>>> the
> > > >> >> >>>>> > original
> > > >> >> >>>>> > > > proposal
> > > >> >> >>>>> > > > >> > for
> > > >> >> >>>>> > > > >> > >> key
> > > >> >> >>>>> > > > >> > >> >> >>> > spacing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> ,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > not
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> new
> > > mentioned as per
> > > >> >> >>>> discussion
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > > >> idea.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will
> > need
> > > to
> > > >> update
> > > >> >> >> the
> > > >> >> >>>> kip,
> > > >> >> >>>>> > when we
> > > >> >> >>>>> > > get
> > > >> >> >>>>> > > > >> > >> agreement
> > > >> >> >>>>> > > > >> > >> >> >>> this
> > > >> >> >>>>> > > > >> > >> >> >>> > is a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > better
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach
> > > (which
> > > >> seems
> > > >> >> to
> > > >> >> >>>> be the
> > > >> >> >>>>> > case if I
> > > >> >> >>>>> > > > have
> > > >> >> >>>>> > > > >> > >> >> understood
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > general
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling
> in
> > > the
> > > >> >> >>>> conversation)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the
> > > variable
> > > >> ints,
> > > >> >> at
> > > >> >> >>>> very
> > > >> >> >>>>> > early stage
> > > >> >> >>>>> > > > we
> > > >> >> >>>>> > > > >> did
> > > >> >> >>>>> > > > >> > >> think
> > > >> >> >>>>> > > > >> > >> >> >>> about
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this. I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added
> > > complexity
> > > >> >> for
> > > >> >> >>>> the
> > > >> >> >>>>> > saving isn't
> > > >> >> >>>>> > > > >> worth
> > > >> >> >>>>> > > > >> > it.
> > > >> >> >>>>> > > > >> > >> >> I'd
> > > >> >> >>>>> > > > >> > >> >> >>> > rather
> > > >> >> >>>>> > > > >> > >> >> >>> > >> go
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want
> to
> > > reduce
> > > >> >> >>>> overheads and
> > > >> >> >>>>> > size
> > > >> >> >>>>> > > int16
> > > >> >> >>>>> > > > >> > (2bytes)
> > > >> >> >>>>> > > > >> > >> >> keys
> > > >> >> >>>>> > > > >> > >> >> >>> as
> > > >> >> >>>>> > > > >> > >> >> >>> > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the
> note
> > > of no
> > > >> >> >> headers,
> > > >> >> >>>> there
> > > >> >> >>>>> > is as
> > > >> >> >>>>> > > per
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > kip
> > > >> >> >>>>> > > > >> > >> as
> > > >> >> >>>>> > > > >> > >> >> we
> > > >> >> >>>>> > > > >> > >> >> >>> > use an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to
> > > denote if
> > > >> >> headers
> > > >> >> >>>> are
> > > >> >> >>>>> > present or
> > > >> >> >>>>> > > > not as
> > > >> >> >>>>> > > > >> > such
> > > >> >> >>>>> > > > >> > >> >> >>> > provides a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > zero
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
> > > currently
> > > >> if
> > > >> >> >>>> headers are
> > > >> >> >>>>> > not
> > > >> >> >>>>> > > used.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think
> as
> > > radai
> > > >> >> mentions
> > > >> >> >>>> would be
> > > >> >> >>>>> > good
> > > >> >> >>>>> > > > first
> > > >> >> >>>>> > > > >> > if we
> > > >> >> >>>>> > > > >> > >> >> can
> > > >> >> >>>>> > > > >> > >> >> >>> get
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > do
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now
> have
> > > general
> > > >> >> >>>> consensus that
> > > >> >> >>>>> > (1)
> > > >> >> >>>>> > > > headers
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2)
> we
> > > want it
> > > >> as
> > > >> >> a
> > > >> >> >>>> top level
> > > >> >> >>>>> > entity.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to
> > > state the
> > > >> >> >> obvious i
> > > >> >> >>>>> > believe (1)
> > > >> >> >>>>> > > > >> headers
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as
> a
> > > top level
> > > >> >> >>>> entity.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>> ______________________________
> > > >> >> >>>>> > __________
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> From:
> Roger
> > > Hoover <
> > > >> >> >>>>> > > roger.hoover@gmail.com
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent:
> > > Wednesday,
> > > >> >> November
> > > >> >> >>>> 9, 2016
> > > >> >> >>>>> > 9:10:47
> > > >> >> >>>>> > > > PM
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
> > > >> >> dev@kafka.apache.org
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject:
> > Re:
> > > >> [DISCUSS]
> > > >> >> >>>> KIP-82 - Add
> > > >> >> >>>>> > > Record
> > > >> >> >>>>> > > > >> > Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
> > > going a
> > > >> >> little
> > > >> >> >>>> in the
> > > >> >> >>>>> > weeds but
> > > >> >> >>>>> > > > >> thanks
> > > >> >> >>>>> > > > >> > >> for
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> replies
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed
> that
> > > a prefix
> > > >> >> and
> > > >> >> >>>> {int,
> > > >> >> >>>>> > int} can
> > > >> >> >>>>> > > be
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> same.
> > > >> >> >>>>> > > > >> > >> >> It
> > > >> >> >>>>> > > > >> > >> >> >>> > doesn't
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > look
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > like
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's
> what
> > > the KIP
> > > >> is
> > > >> >> >>>> saying the
> > > >> >> >>>>> > "Open"
> > > >> >> >>>>> > > > >> > section.
> > > >> >> >>>>> > > > >> > >> The
> > > >> >> >>>>> > > > >> > >> >> >>> > example
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > shows
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New
> > > Relic and
> > > >> >> 210002
> > > >> >> >>>> for App
> > > >> >> >>>>> > Dynamics
> > > >> >> >>>>> > > > >> > implying
> > > >> >> >>>>> > > > >> > >> >> that
> > > >> >> >>>>> > > > >> > >> >> >>> the
> > > >> >> >>>>> > > > >> > >> >> >>> > New
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Relic
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > organization
> > > will
> > > >> have
> > > >> >> >>>> only a
> > > >> >> >>>>> > single
> > > >> >> >>>>> > > > header id
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > >> work
> > > >> >> >>>>> > > > >> > >> >> >>> > with. Or
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix?
> > > The main
> > > >> >> point
> > > >> >> >>>> of a
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > or
> > > >> >> >>>>> > > > >> > prefix
> > > >> >> >>>>> > > > >> > >> is
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > reduce
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
> of
> > > config
> > > >> >> >> mapping
> > > >> >> >>>> or
> > > >> >> >>>>> > > registration
> > > >> >> >>>>> > > > >> > >> depending
> > > >> >> >>>>> > > > >> > >> >> on
> > > >> >> >>>>> > > > >> > >> >> >>> how
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > namespaces/prefixes
> > > >> >> are
> > > >> >> >>>> managed.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would
> love
> > > to hear
> > > >> >> more
> > > >> >> >>>> feedback
> > > >> >> >>>>> > on the
> > > >> >> >>>>> > > > >> > >> higher-level
> > > >> >> >>>>> > > > >> > >> >> >>> > questions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > though...
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed,
> Nov
> > > 9, 2016
> > > >> at
> > > >> >> >>>> 11:38 AM,
> > > >> >> >>>>> > radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think
> > > this
> > > >> >> discussion
> > > >> >> >>>> is
> > > >> >> >>>>> > getting a
> > > >> >> >>>>> > > bit
> > > >> >> >>>>> > > > >> into
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> weeds on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > technical
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > implementation
> > > >> >> details.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd
> liek
> > > to step
> > > >> >> back a
> > > >> >> >>>> minute
> > > >> >> >>>>> > and try
> > > >> >> >>>>> > > > and
> > > >> >> >>>>> > > > >> > >> establish
> > > >> >> >>>>> > > > >> > >> >> >>> > where we
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > are in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger
> > > picture:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > (re-wording
> > > >> nacho's
> > > >> >> >> last
> > > >> >> >>>>> > paragraph)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are
> we
> > > all in
> > > >> >> >>>> agreement that
> > > >> >> >>>>> > headers
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> a
> > > >> >> >>>>> > > > >> > >> >> >>> worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > useful
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> addition
> > > to have?
> > > >> >> this
> > > >> >> >>>> was
> > > >> >> >>>>> > contested
> > > >> >> >>>>> > > > early
> > > >> >> >>>>> > > > >> on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are
> we
> > > all in
> > > >> >> >>>> agreement on
> > > >> >> >>>>> > headers
> > > >> >> >>>>> > > as
> > > >> >> >>>>> > > > top
> > > >> >> >>>>> > > > >> > >> level
> > > >> >> >>>>> > > > >> > >> >> >>> entity
> > > >> >> >>>>> > > > >> > >> >> >>> > vs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > squirreled-away in
> > > >> >> V?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if
> there
> > > are still
> > > >> >> >>>> concerns
> > > >> >> >>>>> > around
> > > >> >> >>>>> > > these
> > > >> >> >>>>> > > > #2
> > > >> >> >>>>> > > > >> > >> points
> > > >> >> >>>>> > > > >> > >> >> >>> (#jay?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and
> now
> > > back to
> > > >> our
> > > >> >> >>>> normal
> > > >> >> >>>>> > programming
> > > >> >> >>>>> > > > ...)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints
> > > are nice.
> > > >> >> >> having
> > > >> >> >>>> said
> > > >> >> >>>>> > that, its
> > > >> >> >>>>> > > > >> adding
> > > >> >> >>>>> > > > >> > >> >> >>> complexity
> > > >> >> >>>>> > > > >> > >> >> >>> > >> (see
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>> https://github.com/addthis/
> > > >> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>> main/java/com/clearspring/
> > > >> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
> > > google
> > > >> >> result)
> > > >> >> >>>> and would
> > > >> >> >>>>> > require
> > > >> >> >>>>> > > > >> anyone
> > > >> >> >>>>> > > > >> > >> >> writing
> > > >> >> >>>>> > > > >> > >> >> >>> > other
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clients
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python?
> > > Go? Bash?
> > > >> >> ;-) )
> > > >> >> >>>> to
> > > >> >> >>>>> > > get/implement
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> same,
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > relatively
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little
> > > gain (int
> > > >> vs
> > > >> >> >>>> string is
> > > >> >> >>>>> > order of
> > > >> >> >>>>> > > > >> > magnitude,
> > > >> >> >>>>> > > > >> > >> >> this
> > > >> >> >>>>> > > > >> > >> >> >>> > isnt).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int
> > > namespacing vs
> > > >> >> >> {int,
> > > >> >> >>>> int}
> > > >> >> >>>>> > > namespacing
> > > >> >> >>>>> > > > >> are
> > > >> >> >>>>> > > > >> > >> >> basically
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > same
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre
> > just
> > > >> >> namespacing
> > > >> >> >>>> an int64
> > > >> >> >>>>> > and
> > > >> >> >>>>> > > > giving
> > > >> >> >>>>> > > > >> > people
> > > >> >> >>>>> > > > >> > >> >> while
> > > >> >> >>>>> > > > >> > >> >> >>> > 2^32
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > ranges
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > at a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time.
> the
> > > part i
> > > >> >> like
> > > >> >> >>>> about this
> > > >> >> >>>>> > is
> > > >> >> >>>>> > > > letting
> > > >> >> >>>>> > > > >> > >> people
> > > >> >> >>>>> > > > >> > >> >> >>> have a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> large
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers
> > > with one
> > > >> >> >>>> registration so
> > > >> >> >>>>> > they
> > > >> >> >>>>> > > > dont
> > > >> >> >>>>> > > > >> > have
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> come
> > > >> >> >>>>> > > > >> > >> >> >>> > back
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > every
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
> > > >> plugin/header
> > > >> >> >>>> they want to
> > > >> >> >>>>> > > > "reserve".
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
> > > Nov 9,
> > > >> 2016
> > > >> >> at
> > > >> >> >>>> 11:01 AM,
> > > >> >> >>>>> > Roger
> > > >> >> >>>>> > > > >> Hoover
> > > >> >> >>>>> > > > >> > <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > roger.hoover@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since
> > > some of
> > > >> the
> > > >> >> >>>> debate has
> > > >> >> >>>>> > been
> > > >> >> >>>>> > > about
> > > >> >> >>>>> > > > >> > >> overhead +
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > wondering if we
> > > >> >> have
> > > >> >> >>>>> > considered a
> > > >> >> >>>>> > > > varint
> > > >> >> >>>>> > > > >> > >> encoding
> > > >> >> >>>>> > > > >> > >> >> (
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>> https://developers.google.com/
> > > >> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > encoding#varints)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the
> > > header
> > > >> length
> > > >> >> >>>> field (int32
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > >> > proposal)
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't
> > use
> > > >> headers,
> > > >> >> >> the
> > > >> >> >>>>> > overhead would
> > > >> >> >>>>> > > > be a
> > > >> >> >>>>> > > > >> > >> single
> > > >> >> >>>>> > > > >> > >> >> >>> byte
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id <
> > 128
> > > would
> > > >> >> also
> > > >> >> >>>> need only a
> > > >> >> >>>>> > > single
> > > >> >> >>>>> > > > >> byte?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On
> Wed,
> > > Nov 9,
> > > >> >> 2016
> > > >> >> >> at
> > > >> >> >>>> 6:43 AM,
> > > >> >> >>>>> > > radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > @magnus - and
> > > >> >> very
> > > >> >> >>>> dangerous
> > > >> >> >>>>> > (youre
> > > >> >> >>>>> > > > >> > >> essentially
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> executing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > arbitrary code
> > > >> >> off
> > > >> >> >>>> the
> > > >> >> >>>>> > internet on
> > > >> >> >>>>> > > > your
> > > >> >> >>>>> > > > >> > >> servers
> > > >> >> >>>>> > > > >> > >> >> ...
> > > >> >> >>>>> > > > >> > >> >> >>> > bad
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > idea
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > without
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > sandbox, even
> > > >> >> with)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as
> > for
> > > it
> > > >> being
> > > >> >> a
> > > >> >> >>>> purely
> > > >> >> >>>>> > > > administrative
> > > >> >> >>>>> > > > >> > task
> > > >> >> >>>>> > > > >> > >> - i
> > > >> >> >>>>> > > > >> > >> >> >>> > >> disagree.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i
> > wish
> > > it
> > > >> would,
> > > >> >> >>>> really,
> > > >> >> >>>>> > because
> > > >> >> >>>>> > > > then my
> > > >> >> >>>>> > > > >> > >> earlier
> > > >> >> >>>>> > > > >> > >> >> >>> > point on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > complexity
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> > > remapping
> > > >> >> >>>> process would
> > > >> >> >>>>> > be
> > > >> >> >>>>> > > > invalid,
> > > >> >> >>>>> > > > >> > but
> > > >> >> >>>>> > > > >> > >> at
> > > >> >> >>>>> > > > >> > >> >> >>> > linkedin,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > example,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> (the
> > > team im
> > > >> in)
> > > >> >> >> run
> > > >> >> >>>> kafka
> > > >> >> >>>>> > as a
> > > >> >> >>>>> > > > service.
> > > >> >> >>>>> > > > >> > we
> > > >> >> >>>>> > > > >> > >> dont
> > > >> >> >>>>> > > > >> > >> >> >>> > really
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > know
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > what our
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > (developing
> > > >> >> >>>> applications
> > > >> >> >>>>> > that use
> > > >> >> >>>>> > > > kafka)
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> up
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> >>> at
> > > >> >> >>>>> > > > >> > >> >> >>> > any
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > given
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is
> > very
> > > >> possible
> > > >> >> >>>> (given the
> > > >> >> >>>>> > > > existance of
> > > >> >> >>>>> > > > >> > >> headers
> > > >> >> >>>>> > > > >> > >> >> >>> and a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> plugin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > ecosystem) for
> > > >> >> some
> > > >> >> >>>>> > application to
> > > >> >> >>>>> > > > >> "equip"
> > > >> >> >>>>> > > > >> > >> their
> > > >> >> >>>>> > > > >> > >> >> >>> > >> producers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> consumers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> with
> > > the
> > > >> >> required
> > > >> >> >>>> plugin
> > > >> >> >>>>> > without us
> > > >> >> >>>>> > > > >> > knowing.
> > > >> >> >>>>> > > > >> > >> i
> > > >> >> >>>>> > > > >> > >> >> dont
> > > >> >> >>>>> > > > >> > >> >> >>> > mean
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > to imply
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> bad,
> > i
> > > just
> > > >> >> want to
> > > >> >> >>>> make the
> > > >> >> >>>>> > point
> > > >> >> >>>>> > > > that
> > > >> >> >>>>> > > > >> > its
> > > >> >> >>>>> > > > >> > >> not
> > > >> >> >>>>> > > > >> > >> >> as
> > > >> >> >>>>> > > > >> > >> >> >>> > simple
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> sync
> > > across a
> > > >> >> >>>> large-enough
> > > >> >> >>>>> > > > organization.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On
> > > Wed, Nov 9,
> > > >> >> 2016
> > > >> >> >>>> at 6:17
> > > >> >> >>>>> > AM,
> > > >> >> >>>>> > > > Magnus
> > > >> >> >>>>> > > > >> > >> Edenhill
> > > >> >> >>>>> > > > >> > >> >> <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > magnus@edenhill.se>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
> > > think
> > > >> there
> > > >> >> is
> > > >> >> >> a
> > > >> >> >>>> piece
> > > >> >> >>>>> > missing
> > > >> >> >>>>> > > in
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> >> Strings
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > where
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> pro-Stringers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > reason that
> > > >> by
> > > >> >> >>>> providing
> > > >> >> >>>>> > unique
> > > >> >> >>>>> > > > string
> > > >> >> >>>>> > > > >> > >> >> >>> identifiers
> > > >> >> >>>>> > > > >> > >> >> >>> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > everything
> > > >> >> will
> > > >> >> >>>> just
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > magically
> > > >> work
> > > >> >> >> for
> > > >> >> >>>> all
> > > >> >> >>>>> > parts of
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> > stream
> > > >> >> >>>>> > > > >> > >> >> >>> pipeline.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> But
> > > the
> > > >> >> strings
> > > >> >> >>>> dont mean
> > > >> >> >>>>> > > anything
> > > >> >> >>>>> > > > by
> > > >> >> >>>>> > > > >> > >> >> themselves,
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > while we
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > probably
> > > >> >> envision
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > some
> > > auto
> > > >> >> plugin
> > > >> >> >>>> loader
> > > >> >> >>>>> > that
> > > >> >> >>>>> > > > >> downloads,
> > > >> >> >>>>> > > > >> > >> >> compiles,
> > > >> >> >>>>> > > > >> > >> >> >>> > links
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > runs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> plugins
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > on-demand
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> as
> > > soon as
> > > >> >> >> they're
> > > >> >> >>>> seen by
> > > >> >> >>>>> > a
> > > >> >> >>>>> > > > >> consumer, I
> > > >> >> >>>>> > > > >> > >> dont
> > > >> >> >>>>> > > > >> > >> >> >>> really
> > > >> >> >>>>> > > > >> > >> >> >>> > >> see
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > something
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> so
> > > dynamic
> > > >> >> (and
> > > >> >> >>>> fragile) in
> > > >> >> >>>>> > > > practice.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> In
> > > the real
> > > >> >> world
> > > >> >> >>>> an
> > > >> >> >>>>> > application
> > > >> >> >>>>> > > > will
> > > >> >> >>>>> > > > >> be
> > > >> >> >>>>> > > > >> > >> >> >>> configured
> > > >> >> >>>>> > > > >> > >> >> >>> > >> with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a set
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> plugins
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> to
> > > either
> > > >> add
> > > >> >> >>>> (producer)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> or
> > > read
> > > >> >> >> (consumer)
> > > >> >> >>>> headers.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > This
> > > is an
> > > >> >> >>>> administrative
> > > >> >> >>>>> > task
> > > >> >> >>>>> > > > based
> > > >> >> >>>>> > > > >> on
> > > >> >> >>>>> > > > >> > >> what
> > > >> >> >>>>> > > > >> > >> >> >>> > features a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > client
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> needs/provides
> > > >> >> >> and
> > > >> >> >>>> results
> > > >> >> >>>>> > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > some
> > > sort of
> > > >> >> >>>> configuration
> > > >> >> >>>>> > to
> > > >> >> >>>>> > > > enable
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> configure
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > desired
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> plugins.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > Since this
> > > >> >> needs
> > > >> >> >>>> to be kept
> > > >> >> >>>>> > > > somewhat
> > > >> >> >>>>> > > > >> in
> > > >> >> >>>>> > > > >> > >> sync
> > > >> >> >>>>> > > > >> > >> >> >>> across
> > > >> >> >>>>> > > > >> > >> >> >>> > an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > (there
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> is
> > > no point
> > > >> in
> > > >> >> >>>> having
> > > >> >> >>>>> > producers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> add
> > > headers
> > > >> no
> > > >> >> >>>> consumers
> > > >> >> >>>>> > will
> > > >> >> >>>>> > > read,
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> vice
> > > >> >> >>>>> > > > >> > >> >> >>> versa),
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > added
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > complexity
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> of
> > > assigning
> > > >> >> an
> > > >> >> >> id
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> for
> > > each
> > > >> >> plugin
> > > >> >> >> as
> > > >> >> >>>> it is
> > > >> >> >>>>> > being
> > > >> >> >>>>> > > > >> > configured
> > > >> >> >>>>> > > > >> > >> >> should
> > > >> >> >>>>> > > > >> > >> >> >>> be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > /Magnus
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > 2016-11-09
> > > >> >> 13:06
> > > >> >> >>>> GMT+01:00
> > > >> >> >>>>> > > Michael
> > > >> >> >>>>> > > > >> > Pearce <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > Michael.Pearce@ig.com>:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Just
> > > >> >> >>>> following/catching
> > > >> >> >>>>> > up on
> > > >> >> >>>>> > > > what
> > > >> >> >>>>> > > > >> > seems
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> be
> > > >> >> >>>>> > > > >> > >> >> >>> an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > active
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > @Radai
> > > >> >> sorry if
> > > >> >> >>>> it may
> > > >> >> >>>>> > seem
> > > >> >> >>>>> > > > obvious
> > > >> >> >>>>> > > > >> > but
> > > >> >> >>>>> > > > >> > >> what
> > > >> >> >>>>> > > > >> > >> >> >>> does
> > > >> >> >>>>> > > > >> > >> >> >>> > MD
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > stand
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > for?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > My
> > > take on
> > > >> >> >>>> String vs Int:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > I
> > > will
> > > >> state
> > > >> >> >>>> first I am
> > > >> >> >>>>> > pro Int
> > > >> >> >>>>> > > > (16
> > > >> >> >>>>> > > > >> or
> > > >> >> >>>>> > > > >> > >> 32).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > I
> > > do
> > > >> though
> > > >> >> >>>> playing
> > > >> >> >>>>> > devils
> > > >> >> >>>>> > > > advocate
> > > >> >> >>>>> > > > >> > see a
> > > >> >> >>>>> > > > >> > >> >> big
> > > >> >> >>>>> > > > >> > >> >> >>> plus
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > with the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> argument
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > String
> > > >> keys,
> > > >> >> >>>> this is
> > > >> >> >>>>> > around
> > > >> >> >>>>> > > > >> > integrating
> > > >> >> >>>>> > > > >> > >> >> into an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > existing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > eco-system.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > As
> > > many
> > > >> >> other
> > > >> >> >>>> systems use
> > > >> >> >>>>> > > String
> > > >> >> >>>>> > > > >> based
> > > >> >> >>>>> > > > >> > >> >> headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > much
> > > >> easier
> > > >> >> for
> > > >> >> >>>> these to
> > > >> >> >>>>> > be
> > > >> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > into.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > How with
> > > >> Int
> > > >> >> >>>> based
> > > >> >> >>>>> > headers
> > > >> >> >>>>> > > could
> > > >> >> >>>>> > > > we
> > > >> >> >>>>> > > > >> > >> provide
> > > >> >> >>>>> > > > >> > >> >> a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> integration
> > > >> >> >>>> simple /
> > > >> >> >>>>> > easy with
> > > >> >> >>>>> > > > >> > transition
> > > >> >> >>>>> > > > >> > >> >> flows
> > > >> >> >>>>> > > > >> > >> >> >>> > over
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > *
> > > tough
> > > >> luck
> > > >> >> >>>> buddy
> > > >> >> >>>>> > you're on
> > > >> >> >>>>> > > your
> > > >> >> >>>>> > > > >> own
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > *
> > > simply
> > > >> >> hash
> > > >> >> >>>> the string
> > > >> >> >>>>> > into
> > > >> >> >>>>> > > int
> > > >> >> >>>>> > > > >> code
> > > >> >> >>>>> > > > >> > >> and
> > > >> >> >>>>> > > > >> > >> >> hope
> > > >> >> >>>>> > > > >> > >> >> >>> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > convert
> > > >> back
> > > >> >> >>>> though?)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > *
> > > http2
> > > >> >> style
> > > >> >> >> as
> > > >> >> >>>>> > mentioned by
> > > >> >> >>>>> > > > nacho.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > cheers,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Mike
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > ______________________________
> > > >> >> >>>>> > > > >> > __________
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > From:
> > > >> radai
> > > >> >> <
> > > >> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Sent:
> > > >> >> >> Wednesday,
> > > >> >> >>>>> > November 9,
> > > >> >> >>>>> > > 2016
> > > >> >> >>>>> > > > >> > 8:12 AM
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > To:
> > > >> >> >>>> dev@kafka.apache.org
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Subject:
> > > >> Re:
> > > >> >> >>>> [DISCUSS]
> > > >> >> >>>>> > KIP-82 -
> > > >> >> >>>>> > > > Add
> > > >> >> >>>>> > > > >> > >> Record
> > > >> >> >>>>> > > > >> > >> >> >>> Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > thinking
> > > >> >> about
> > > >> >> >>>> it some
> > > >> >> >>>>> > more,
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> best
> > > >> >> >>>>> > > > >> > >> way to
> > > >> >> >>>>> > > > >> > >> >> >>> > transmit
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > remapping
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > data to
> > > >> >> >>>> consumers would
> > > >> >> >>>>> > be to
> > > >> >> >>>>> > > > put it
> > > >> >> >>>>> > > > >> > in
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> MD
> > > >> >> >>>>> > > > >> > >> >> >>> > >> response
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> maybe
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > it
> > > should
> > > >> be
> > > >> >> >>>> discussed
> > > >> >> >>>>> > now.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > On
> > > Wed,
> > > >> Nov
> > > >> >> 9,
> > > >> >> >>>> 2016 at
> > > >> >> >>>>> > 12:09
> > > >> >> >>>>> > > AM,
> > > >> >> >>>>> > > > >> > radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >> radai.rosenblatt@gmail.com
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > im not
> > > >> >> >> opposed
> > > >> >> >>>> to the
> > > >> >> >>>>> > idea of
> > > >> >> >>>>> > > > >> > namespace
> > > >> >> >>>>> > > > >> > >> >> >>> mapping.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> all
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > im
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > saying
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > its
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > not part
> > > >> >> of
> > > >> >> >>>> the "mvp"
> > > >> >> >>>>> > and,
> > > >> >> >>>>> > > > since
> > > >> >> >>>>> > > > >> it
> > > >> >> >>>>> > > > >> > >> >> requires
> > > >> >> >>>>> > > > >> > >> >> >>> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> wire
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > format
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > always
> > > >> be
> > > >> >> >>>> added later.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > also,
> > > >> its
> > > >> >> not
> > > >> >> >>>> as
> > > >> >> >>>>> > simple as
> > > >> >> >>>>> > > just
> > > >> >> >>>>> > > > >> > >> >> configuring
> > > >> >> >>>>> > > > >> > >> >> >>> MM
> > > >> >> >>>>> > > > >> > >> >> >>> > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> do
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > transform:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > lets
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > say i've
> > > >> >> >>>> implemented
> > > >> >> >>>>> > large
> > > >> >> >>>>> > > > message
> > > >> >> >>>>> > > > >> > >> >> support as
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > mirror
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > target
> > > >> >> >> cluster
> > > >> >> >>>> its been
> > > >> >> >>>>> > > > remapped
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> >> {999,1}.
> > > >> >> >>>>> > > > >> > >> >> >>> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > consumer
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > would
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > also
> > > >> need
> > > >> >> to
> > > >> >> >>>> be told
> > > >> >> >>>>> > to look
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> large
> > > >> >> >>>>> > > > >> > >> >> >>> > message
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > "part X
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > under
> > > >> >> >> {999,1}.
> > > >> >> >>>> doable,
> > > >> >> >>>>> > but
> > > >> >> >>>>> > > > tricky.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > On Tue,
> > > >> >> Nov
> > > >> >> >> 8,
> > > >> >> >>>> 2016 at
> > > >> >> >>>>> > 10:29
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > Gwen
> > > >> >> >>>>> > > > >> > >> >> >>> Shapira <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > gwen@confluent.io
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > While
> > > >> you
> > > >> >> >> can
> > > >> >> >>>> do
> > > >> >> >>>>> > whatever
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> want
> > > >> >> >>>>> > > > >> > >> with a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > your
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > what
> > > >> I'd
> > > >> >> >>>> expect is
> > > >> >> >>>>> > for each
> > > >> >> >>>>> > > > app
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> namespaces
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> configurable...
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > So if I
> > > >> >> >>>> accidentally
> > > >> >> >>>>> > used
> > > >> >> >>>>> > > 666
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> > my
> > > >> >> >>>>> > > > >> > >> HR
> > > >> >> >>>>> > > > >> > >> >> >>> > >> department,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > still
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > run
> > > >> >> >> RadaiApp,
> > > >> >> >>>> I can
> > > >> >> >>>>> > config
> > > >> >> >>>>> > > > >> > >> "namespace=42"
> > > >> >> >>>>> > > > >> > >> >> >>> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > everything
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > will
> > > >> look
> > > >> >> >>>> normal.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > This
> > > >> >> means
> > > >> >> >>>> you only
> > > >> >> >>>>> > need to
> > > >> >> >>>>> > > > sync
> > > >> >> >>>>> > > > >> > usage
> > > >> >> >>>>> > > > >> > >> >> >>> inside
> > > >> >> >>>>> > > > >> > >> >> >>> > your
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > own
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > organization.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > Still
> > > >> >> hard,
> > > >> >> >>>> but
> > > >> >> >>>>> > somewhat
> > > >> >> >>>>> > > > easier
> > > >> >> >>>>> > > > >> > than
> > > >> >> >>>>> > > > >> > >> >> syncing
> > > >> >> >>>>> > > > >> > >> >> >>> > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > entire
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> world.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > On Tue,
> > > >> >> Nov
> > > >> >> >>>> 8, 2016
> > > >> >> >>>>> > at 10:07
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > >> radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > and
> > > >> we
> > > >> >> can
> > > >> >> >>>> start
> > > >> >> >>>>> > with
> > > >> >> >>>>> > > > >> {namespace,
> > > >> >> >>>>> > > > >> > >> id}
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > always
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > add
> > > >> it
> > > >> >> >>>> later on
> > > >> >> >>>>> > if/when
> > > >> >> >>>>> > > > >> > collisions
> > > >> >> >>>>> > > > >> > >> >> >>> actually
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > dont
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > they'd
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > a
> > > >> >> >> problem).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > every
> > > >> >> >>>> interested
> > > >> >> >>>>> > party (so
> > > >> >> >>>>> > > > orgs
> > > >> >> >>>>> > > > >> > or
> > > >> >> >>>>> > > > >> > >> >> >>> > individuals)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > then
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > register
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> prefix
> > > >> >> (0
> > > >> >> >> =
> > > >> >> >>>>> > reserved, 1 =
> > > >> >> >>>>> > > > >> > confluent
> > > >> >> >>>>> > > > >> > >> ...
> > > >> >> >>>>> > > > >> > >> >> >>> 666
> > > >> >> >>>>> > > > >> > >> >> >>> > = me
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > whatever
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > the
> > > >> >> 2nd ID
> > > >> >> >>>> - so once
> > > >> >> >>>>> > > > linkedin
> > > >> >> >>>>> > > > >> > >> >> registers,
> > > >> >> >>>>> > > > >> > >> >> >>> say
> > > >> >> >>>>> > > > >> > >> >> >>> > 3,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > then
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> are
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > free
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > to
> > > >> use
> > > >> >> {3,
> > > >> >> >>>> *} with a
> > > >> >> >>>>> > > > reasonable
> > > >> >> >>>>> > > > >> > >> >> >>> expectation
> > > >> >> >>>>> > > > >> > >> >> >>> > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > collide
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > anything
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > else.
> > > >> >> >>>> further
> > > >> >> >>>>> > partitioning
> > > >> >> >>>>> > > > of
> > > >> >> >>>>> > > > >> > that *
> > > >> >> >>>>> > > > >> > >> >> >>> becomes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> problem,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> "upstream
> > > >> >> >>>>> > registration"
> > > >> >> >>>>> > > of a
> > > >> >> >>>>> > > > >> > >> namespace
> > > >> >> >>>>> > > > >> > >> >> >>> only
> > > >> >> >>>>> > > > >> > >> >> >>> > has
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > happen
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > On
> > > >> Tue,
> > > >> >> >> Nov
> > > >> >> >>>> 8, 2016
> > > >> >> >>>>> > at
> > > >> >> >>>>> > > 9:03
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > >> James
> > > >> >> >>>>> > > > >> > >> >> >>> Cheng <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> wushujames@gmail.com
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> > On
> > > >> >> Nov
> > > >> >> >>>> 8, 2016,
> > > >> >> >>>>> > at 5:54
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > Gwen
> > > >> >> >>>>> > > > >> > >> >> >>> Shapira <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > gwen@confluent.io
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> Thank
> > > >> >> >>>> you so
> > > >> >> >>>>> > much for
> > > >> >> >>>>> > > > this
> > > >> >> >>>>> > > > >> > clear
> > > >> >> >>>>> > > > >> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> fair
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > summary of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > arguments.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> I'm
> > > >> >> in
> > > >> >> >>>> favor of
> > > >> >> >>>>> > ints.
> > > >> >> >>>>> > > > Not a
> > > >> >> >>>>> > > > >> > >> >> >>> deal-breaker,
> > > >> >> >>>>> > > > >> > >> >> >>> > but
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> Even
> > > >> >> >>>> more in
> > > >> >> >>>>> > favor of
> > > >> >> >>>>> > > > >> Magnus's
> > > >> >> >>>>> > > > >> > >> >> >>> > decentralized
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Roger's
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> tweak:
> > > >> >> >>>> add a
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> > >> headers.
> > > >> >> >>>>> > > > >> > >> >> >>> This
> > > >> >> >>>>> > > > >> > >> >> >>> > will
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > allow
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > just
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> use
> > > >> >> >>>> whatever IDs
> > > >> >> >>>>> > it
> > > >> >> >>>>> > > wants
> > > >> >> >>>>> > > > >> > >> >> internally,
> > > >> >> >>>>> > > > >> > >> >> >>> and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> then
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > let
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> deploying
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> the
> > > >> >> app
> > > >> >> >>>> figure
> > > >> >> >>>>> > out an
> > > >> >> >>>>> > > > >> > available
> > > >> >> >>>>> > > > >> > >> >> >>> namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > ID
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> live
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > in.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> > So
> > > >> >> >>>>> > > > >> > io.confluent.schema-registry
> > > >> >> >>>>> > > > >> > >> can
> > > >> >> >>>>> > > > >> > >> >> be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> deployment
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> and
> > > >> >> >> 0x57
> > > >> >> >>>> on
> > > >> >> >>>>> > yours, and
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > poor
> > > >> >> >>>>> > > > >> > >> guys
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > developing the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > need
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> worry
> > > >> >> >>>> about that.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> Gwen,
> > > >> >> if
> > > >> >> >> I
> > > >> >> >>>>> > understand
> > > >> >> >>>>> > > your
> > > >> >> >>>>> > > > >> > example
> > > >> >> >>>>> > > > >> > >> >> >>> right, an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > application
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > deployer
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > might
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> decide to
> > > >> >> >>>> use 0x01
> > > >> >> >>>>> > in one
> > > >> >> >>>>> > > > >> > >> deployment,
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > means
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > message
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> is
> > > >> >> >> written
> > > >> >> >>>> into the
> > > >> >> >>>>> > > > broker, it
> > > >> >> >>>>> > > > >> > >> will be
> > > >> >> >>>>> > > > >> > >> >> >>> > saved on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > broker
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> specific
> > > >> >> >>>> namespace
> > > >> >> >>>>> > > (0x01).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> If
> > > >> you
> > > >> >> >>>> were to
> > > >> >> >>>>> > mirror
> > > >> >> >>>>> > > that
> > > >> >> >>>>> > > > >> > message
> > > >> >> >>>>> > > > >> > >> >> into
> > > >> >> >>>>> > > > >> > >> >> >>> > another
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > would
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> accompany
> > > >> >> >>>> the
> > > >> >> >>>>> > message,
> > > >> >> >>>>> > > > right?
> > > >> >> >>>>> > > > >> > What
> > > >> >> >>>>> > > > >> > >> if
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> other
> > > >> >> >>>> cluster uses
> > > >> >> >>>>> > 0x57?
> > > >> >> >>>>> > > > They
> > > >> >> >>>>> > > > >> > won't
> > > >> >> >>>>> > > > >> > >> >> >>> > understand
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > other?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> I'm
> > > >> >> not
> > > >> >> >>>> sure
> > > >> >> >>>>> > that's an
> > > >> >> >>>>> > > > >> avoidable
> > > >> >> >>>>> > > > >> > >> >> >>> problem. I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > think it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > simply
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> order
> > > >> >> to
> > > >> >> >>>> share
> > > >> >> >>>>> > data, you
> > > >> >> >>>>> > > > have
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> also
> > > >> >> >>>>> > > > >> > >> >> >>> have a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > shared
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>> understanding of
> > > >> >> >>>>> > what the
> > > >> >> >>>>> > > > >> > >> namespaces
> > > >> >> >>>>> > > > >> > >> >> >>> mean.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Which
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > sense,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> because
> > > >> >> >> the
> > > >> >> >>>>> > alternate
> > > >> >> >>>>> > > > (sharing
> > > >> >> >>>>> > > > >> > >> >> *nothing*
> > > >> >> >>>>> > > > >> > >> >> >>> at
> > > >> >> >>>>> > > > >> > >> >> >>> > >> all)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > would
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > mean
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > there
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> would
> > > >> >> be
> > > >> >> >>>> no way to
> > > >> >> >>>>> > > > understand
> > > >> >> >>>>> > > > >> > each
> > > >> >> >>>>> > > > >> > >> >> other.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> -James
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> Gwen
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> > On
> > > >> >> Tue,
> > > >> >> >>>> Nov 8,
> > > >> >> >>>>> > 2016 at
> > > >> >> >>>>> > > > 4:23
> > > >> >> >>>>> > > > >> > PM,
> > > >> >> >>>>> > > > >> > >> >> radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>> radai.rosenblatt@gmail.com
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >>
> > > >> +1
> > > >> >> for
> > > >> >> >>>> sean's
> > > >> >> >>>>> > > document.
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > >> covers
> > > >> >> >>>>> > > > >> > >> >> >>> pretty
> > > >> >> >>>>> > > > >> > >> >> >>> > >> much
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > all
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > trade-offs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >>
> > > >> >> >> provides
> > > >> >> >>>>> > concrete
> > > >> >> >>>>> > > > figures
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> argue
> > > >> >> >>>>> > > > >> > >> >> >>> about
> > > >> >> >>>>> > > > >> > >> >> >>> > :-)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >>
> > > >> >> >>>> (nit-picking -
> > > >> >> >>>>> > used
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> same
> > > >> >> >>>>> > > > >> > >> xkcd
> > > >> >> >>>>> > > > >> > >> >> >>> twice,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> also
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > trove
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > has
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> superceded
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > --
> > > >> >> >>>>> > > > >> > Gwen Shapira
> > > >> >> >>>>> > > > >> > Product Manager | Confluent
> > > >> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
> > > @gwenshap
> > > >> >> >>>>> > > > >> > Follow us: Twitter | blog
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> --
> > > >> >> >>>>> > > > >> *Todd Palino*
> > > >> >> >>>>> > > > >> Staff Site Reliability Engineer
> > > >> >> >>>>> > > > >> Data Infrastructure Streaming
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> linkedin.com/in/toddpalino
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > --
> > > >> >> >>>>> > > > Gwen Shapira
> > > >> >> >>>>> > > > Product Manager | Confluent
> > > >> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> |
> @gwenshap
> > > >> >> >>>>> > > > Follow us: Twitter | blog
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > The information contained in this email is
> strictly
> > > >> >> confidential
> > > >> >> >>>> and for
> > > >> >> >>>>> > the use of the addressee only, unless otherwise
> > > indicated.
> > > >> If
> > > >> >> you
> > > >> >> >>>> are not
> > > >> >> >>>>> > the intended recipient, please do not read, copy,
> > > use or
> > > >> >> disclose
> > > >> >> >>>> to others
> > > >> >> >>>>> > this message or any attachment. Please also
> notify
> > > the
> > > >> sender
> > > >> >> by
> > > >> >> >>>> replying
> > > >> >> >>>>> > to this email or by telephone (+44(020 7896 0011)
> > > and then
> > > >> >> delete
> > > >> >> >>>> the email
> > > >> >> >>>>> > and any copies of it. Opinions, conclusion (etc)
> > > that do not
> > > >> >> >>>> relate to the
> > > >> >> >>>>> > official business of this company shall be
> > > understood as
> > > >> >> neither
> > > >> >> >>>> given nor
> > > >> >> >>>>> > endorsed by it. IG is a trading name of IG
> Markets
> > > Limited
> > > >> (a
> > > >> >> >>>> company
> > > >> >> >>>>> > registered in England and Wales, company number
> > > 04008957)
> > > >> and
> > > >> >> IG
> > > >> >> >>>> Index
> > > >> >> >>>>> > Limited (a company registered in England and
> Wales,
> > > company
> > > >> >> >> number
> > > >> >> >>>>> > 01190902). Registered address at Cannon Bridge
> > > House, 25
> > > >> >> Dowgate
> > > >> >> >>>> Hill,
> > > >> >> >>>>> > London EC4R 2YA. Both IG Markets Limited
> (register
> > > number
> > > >> >> 195355)
> > > >> >> >>>> and IG
> > > >> >> >>>>> > Index Limited (register number 114059) are
> > > authorised and
> > > >> >> >>>> regulated by the
> > > >> >> >>>>> > Financial Conduct Authority.
> > > >> >> >>>>> >
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> The information contained in this email is strictly
> > > confidential
> > > >> and
> > > >> >> >> for
> > > >> >> >>>> the use of the addressee only, unless otherwise
> indicated.
> > > If you
> > > >> are
> > > >> >> >> not
> > > >> >> >>>> the intended recipient, please do not read, copy, use or
> > > disclose
> > > >> to
> > > >> >> >> others
> > > >> >> >>>> this message or any attachment. Please also notify the
> > > sender by
> > > >> >> >> replying
> > > >> >> >>>> to this email or by telephone (+44(020 7896 0011) and
> then
> > > delete
> > > >> the
> > > >> >> >> email
> > > >> >> >>>> and any copies of it. Opinions, conclusion (etc) that do
> > > not relate
> > > >> >> to
> > > >> >> >> the
> > > >> >> >>>> official business of this company shall be understood as
> > > neither
> > > >> >> given
> > > >> >> >> nor
> > > >> >> >>>> endorsed by it. IG is a trading name of IG Markets
> Limited
> > > (a
> > > >> company
> > > >> >> >>>> registered in England and Wales, company number
> 04008957)
> > > and IG
> > > >> >> Index
> > > >> >> >>>> Limited (a company registered in England and Wales,
> > company
> > > number
> > > >> >> >>>> 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > >> >> Hill,
> > > >> >> >>>> London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355)
> > > >> >> and IG
> > > >> >> >>>> Index Limited (register number 114059) are authorised
> and
> > > regulated
> > > >> >> by
> > > >> >> >> the
> > > >> >> >>>> Financial Conduct Authority.
> > > >> >> >>>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >
> > > >> The information contained in this email is strictly confidential
> > > and for
> > > >> the use of the addressee only, unless otherwise indicated. If
> you
> > > are not
> > > >> the intended recipient, please do not read, copy, use or
> disclose
> > > to others
> > > >> this message or any attachment. Please also notify the sender by
> > > replying
> > > >> to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > the email
> > > >> and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > to the
> > > >> official business of this company shall be understood as neither
> > > given nor
> > > >> endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > > >> registered in England and Wales, company number 04008957) and IG
> > > Index
> > > >> Limited (a company registered in England and Wales, company
> number
> > > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > >> London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > and IG
> > > >> Index Limited (register number 114059) are authorised and
> > regulated
> > > by the
> > > >> Financial Conduct Authority.
> > > >>
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jay Kreps <ja...@confluent.io>.
Yeah I think I share a couple of concerns around the details of the
proposal beyond the use cases:
1. I think numeric keys are just a non-starter for a user-facing
interface that is meant to work across third party components (not just
within an organization). We've gone down this path of making cryptic
interfaces before based on what's easy to implement (simple consumer!). The
result is a mess we are still trying to walk away from. I think we are
solving a compression/normalization problem by having the user implement a
compression algorithm in their head and share it across their company and
between components and plugin. This is definitely easier for us, but not
the right interface! I think we should consider instead (1) just take the
perf hit and claim it's okay since it's optional (I don't necessarily buy
that since it makes a lot of common use cases pretty questionable), (2)
work on getting more efficient compression in place so we can recommend
that without reservation (e.g. if that facebook lib pans out) or (3) allow
the client to optionally pre-register headers to get a more compact
numerical mapping.
2. I think we should think about creating the Map lazily to avoid
parsing out all the headers into little objects. HashMaps themselves are
kind of expensive and the consumer is very perf sensitive so and making
gazillions of hashmaps that may or may not get used is probably a bad idea.
3. We should think about having our own type, Headers or something
rather than a Map. I kind of dislike this kind of unnecessary type in
internal code, but for this kind of public API the ability to get just the
methods that make sense and document the details of the semantic can be
worthwhile.
-Jay
On Tue, Jan 10, 2017 at 10:38 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> I'm horribly behind and outdated on this thread, but wanted to mention one
> other potential issue.
>
> The KIP talks about using simple Maps for the headers. For a lot of this
> discussion, I tend to either think of completely opaque headers (i.e just
> another byte[] blob, which seems to have been rejected based on
> interoperability grounds but I think has the nice property of making it
> easy to blame the implementer of the serde for any problems with the header
> implementation) or HTTP-style headers (seemingly the most ubiquitous
> example of headers, though certainly there are many others, and one which
> we/users may need/want to interoperate with).
>
> Strictly speaking, the keys on HTTP headers aren't unique. See
> https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2
> specifically:
>
> > Multiple message-header fields with the same field-name MAY be present in
> a message if and only if the entire field-value for that header field is
> defined as a comma-separated list [i.e., #(values)]. It MUST be possible to
> combine the multiple header fields into one "field-name: field-value" pair,
> without changing the semantics of the message, by appending each subsequent
> field-value to the first, each separated by a comma. The order in which
> header fields with the same field-name are received is therefore
> significant to the interpretation of the combined field value, and thus a
> proxy MUST NOT change the order of these field values when a message is
> forwarded.
>
> The spec guarantees that you can collapse repeated headers into a single
> instance. But in practice there's a ton of stuff that doesn't. And every
> robust HTTP library I've encountered uses a multimap to represent headers.
>
> I'm not necessarily advocating for supporting this, just suggesting that we
> think about it. As you add these features and get closer to mapping to
> other systems, people will inevitably try to map them. Headers are an area
> where, if we're going to add them, it's worth considering compatibility as
> someone will inevitably come and complain that system X does Y with headers
> and we should also support Y because any decent system that provides
> headers will do so.
>
> -Ewen
>
> On Tue, Jan 10, 2017 at 5:27 PM, Jay Kreps <ja...@confluent.io> wrote:
>
> > Hey Gwen, yes, your approach is much better than what I described and
> > solves one of my objections. I didn't think of doing it in MM rather than
> > in the original producer. I don't think it adds any overhead beyond the
> > additional header bytes if I'm not mistaken, since MM already unpacks and
> > repacks messages (unlike the broker which tries to avoid repacking and
> > hence can't efficiently add a header). I think my other objection around
> > size is likely solvable too by some kind of aliasing scheme? If you guys
> > think this can be made to work then I think we have two pretty credible
> use
> > cases with this and the tracing id family of uses, plus a number of
> things
> > that are at least moderately improved. If so, I'm convinced.
> >
> > -Jay
> >
> >
> > On Tue, Jan 10, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Hi Jay et al.
> > >
> > > On the replication using cluster id.
> > >
> > > Two options depending on how a company wants or needs to implement
> their
> > > replication as there are options of doing this as a mesh, star, or
> ring.
> > >
> > > Either as some companies enforce a provider/custom producer/consumer
> > > wrapper, it can be enforced the interceptors needed to be added, as you
> > can
> > > enforce to get support of the platform you must use company x’s
> internal
> > > wrappers.
> > > An alternative approach is to do this at the mirror maker, as Gwen
> noted
> > > we can actually make the mirror maker append this, like wise for a more
> > > complex implementation, where you can have multiple hops/routes this
> can
> > > also be the message routing.
> > >
> > > Re the size of the clusterId, yes the auto generated one is “cough”
> > > hideous in size.
> > >
> > > Most organisations though will only need a space to uniquely identify
> 127
> > > (byte) clusters, or at worst 2 bytes (short) they need to mirror and
> > > replicate between, and as such can either as platform teams generally
> > will
> > > roll out the mirror makers / replicators with the cluster can either
> > > manually adjust the clusterId on ZK, or use a mapping file all their
> > mirror
> > > makers have to map the larger id to a smaller one.
> > >
> > > Also as Gwen states this is something you should be able to opt into or
> > > not, this is the benefit of headers, if you don’t use it, it doesn’t
> use
> > up
> > > the space, its entirely optional.
> > >
> > > Regards
> > > Mike
> > >
> > >
> > > On 09/01/2017, 19:40, "Gwen Shapira" <gw...@confluent.io> wrote:
> > >
> > > Mirroring + ClusterIDs:
> > >
> > > Kafka clusters already have an automatically generated ID that is
> > > exposed to clients:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 78%3A+Cluster+Id
> > >
> > > It sounds trivial to get MirrorMaker's consumer to grab this ID
> > (which
> > > always exists) from the source cluster and add it to the message
> > > headers.
> > >
> > > Obviously adding stuff to the message makes it larger so this
> should
> > > be a feature you should be able to turn on/off depending on the
> > > replication requirements. If you need the feature - you pay the
> > price.
> > > I don't think there is an immediate need to put an effort into
> > > shrinking the Cluster ID, but others can comment on that.
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io>
> wrote:
> > > > Hey Michael,
> > > >
> > > > Sorry for lagging, I've been out for the holiday break. What
> would
> > > help me
> > > > would be to get to a consolidated list of use cases we've worked
> > out
> > > and
> > > > think add real value. I think getting down to the details really
> > > matters
> > > > for these.
> > > >
> > > > The two lists I'm aware of are the one in the KIP and the one you
> > > gave in
> > > > this email thread. Both seem to be more at the sketch level. Let
> me
> > > review
> > > > the use cases and see if I have it right. If so let's get it in
> the
> > > design
> > > > proposal.
> > > >
> > > > Your use cases:
> > > >
> > > > 1. Tracing IDs -- I get this use case and think I see how it
> > > would work.
> > > > I think because the tracing is from a third party system there
> > is
> > > some
> > > > value to having a header versus a custom format to make this
> > work
> > > fully
> > > > transparently.
> > > > 2. Mirroring -- I'm a bit unsure how this will work, I'll
> > explain
> > > my
> > > > thinking below
> > > > 3. End-to-end encryption -- I think what you are saying is
> that
> > > if you
> > > > want "routing' fields that are accessible in transit without
> > > decrypting you
> > > > need a scheme to leave this decrypted.I think the idea is to
> > avoid
> > > > decrypting to get a header, but I'm not sure if I get how this
> > is
> > > better
> > > > than a custom message format that doesn't encrypt the things
> you
> > > don't want
> > > > encrypted since you have to be aware of the scheme anyway.
> > > > 4. EOS -- I'm not convinced this is a good approach for EOS.
> > > >
> > > > Linked from KIP:
> > > >
> > > > 1. Schema-Registry-aware avro serialization -- not clear how
> > this
> > > is
> > > > better than keeping the schema id in the value
> > > > 2. Polyglot kafka topics -- marginally better than a container
> > > format,
> > > > but of course you still have to all agree on the the header
> and
> > > > header=>serialization mapping so not sure if it's really
> better
> > > than a
> > > > container format. Also not wild about polyglot topics :-)
> > > > 3. End-to-End routing trail including mirroring --- I'd put
> this
> > > under
> > > > tracing ids
> > > > 4. Cross-service process tracing -- ditto
> > > > 5. Signing -- I like this use case but it seems to depend on a
> > > feature
> > > > we don't have and aren't proposing adding, right?
> > > > 6. Per-record custom compression -- why do you want this?
> > > compressing
> > > > per message won't work well compared to batch compression,
> > right?
> > > > 7. Large Message Support -- I think we can agree that although
> > > this
> > > > might work it's clearly not the right way to support large
> > > messages in Kafka
> > > >
> > > > Matthias
> > > >
> > > > 1. Control messages --- Matthias, not sure if I get the
> > > motivation in
> > > > streams for this. If we did want to do it, wouldn't it be
> easier
> > > to
> > > > implement and by allowing users to send control messages (in
> the
> > > EOS sense)
> > > > and adding a consumer config to allow either passing these on
> or
> > > > suppressing them?
> > > >
> > > > So I if I were to summarize I'd list the use cases as follows:
> > > >
> > > > - Much better with headers: tracing IDs
> > > > - Somewhat better: end-to-end encryption, polyglot topics
> > > > - Don't buy: Mirroring (though perhaps could be made to work
> > with
> > > > changes to the proposal), EOS, large messages, compression
> > > > - Not sure: Control messages
> > > >
> > > > Does this match where other people are at? From my point of view
> > for
> > > > something like this where we are changing the core data model we
> > > should
> > > > ideally get a handful of use cases to the point where they are
> > > really well
> > > > thought out and we know they can use the feature. I think we have
> > > one solid
> > > > one with tracing IDs, and I think maybe mirroring could be
> another
> > > with
> > > > some work. I'm less sure about the others.
> > > >
> > > > Bi-directional Mirroring
> > > > I think bidirectional mirroring could potentially become a
> > > compelling use
> > > > case if we an find a way to make it work. If I understand it, the
> > > proposal
> > > > is to add a header like "origin-cluster" with the cluster id of
> the
> > > Kafka
> > > > cluster. Then modify the mirroring tool to disallow mirroring a
> > > message
> > > > back to it's own origin cluster. This basic idea is definitely
> > > workable and
> > > > useful, but can we really get to a good implementation using
> > headers?
> > > > I imagine the approach would be to add this header in the
> producer
> > > if some
> > > > config is set. There are two problems:
> > > >
> > > > - You add a pretty big honking cluster ID to each message. For
> > > small
> > > > messages this maybe larger than the contents itself.
> > > > - What happens if someone somewhere forgets to set that config
> > on
> > > some
> > > > client somewhere?
> > > >
> > > > If someone forgets to set the config in some client somewhere I
> > > guess you
> > > > have various options in mirror maker:
> > > >
> > > > 1. drop the data (pretty bad, now your data is out of sync)
> > > > 2. crash (pretty bad--how do you recover)
> > > > 3. insert some default (pretty bad since would also lead to
> data
> > > loss if
> > > > that's not right)
> > > >
> > > > Basically although I think a cluster tag could be a good way to
> do
> > > > bi-directional mirroring, I'm not sure this naive approach I
> > > described
> > > > would get us to a usable implementation. This could be fixable. I
> > > think if
> > > > there was some scheme to check required headers on the server
> that
> > > could
> > > > alleviate the missing header problem. Not really sure what to do
> > > about the
> > > > message bloat problem, but maybe there could be some way to
> shrink
> > > the
> > > > cluster id?
> > > >
> > > > Cheers,
> > > >
> > > > -Jay
> > > >
> > > >
> > > > On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> > > Michael.Pearce@ig.com>
> > > > wrote:
> > > >
> > > >> @Jay, Ismael and a few others.
> > > >>
> > > >> Based on the discussion on kip-87, and the additional use case
> by
> > > Matthias.
> > > >>
> > > >> Just for clarity do we now have consensus on the earlier points
> > (1)
> > > >> headers are useful (2) headers should be native in Kafka?
> > > >>
> > > >> Cheers
> > > >> Mike
> > > >> ________________________________________
> > > >> From: radai <ra...@gmail.com>
> > > >> Sent: Monday, December 19, 2016 5:01 PM
> > > >> To: dev@kafka.apache.org
> > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > > >>
> > > >> having said that im perfectly fine with something like a "no
> > > payload" flag
> > > >> (as opposed to null payload), which would mean such a msg is not
> > > returned
> > > >> from poll() unless the user code specifically "opts in" and
> would
> > > serve
> > > >> only to carry headers (making it a control message)
> > > >>
> > > >> On Mon, Dec 19, 2016 at 8:58 AM, radai <
> > radai.rosenblatt@gmail.com>
> > > wrote:
> > > >>
> > > >> > if all you want is to tag "end of stream" wouldnt a header on
> > the
> > > last
> > > >> msg
> > > >> > in the stream be enough?
> > > >> >
> > > >> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> > > matthias@confluent.io>
> > > >> > wrote:
> > > >> >
> > > >> >> The details about headers for control messages are still to
> > > define. But
> > > >> >> yes, the idea is to have some common default behavior that
> > > clients would
> > > >> >> need to implement.
> > > >> >>
> > > >> >> The point is, that "regular headers" add meta data to regular
> > > messages.
> > > >> >> Thus, those messages will be returned to the user via
> .poll().
> > > And after
> > > >> >> the message is received the user can check if meta data is
> > > present and
> > > >> >> read it.
> > > >> >>
> > > >> >> For control messages, we do not want those to pop up via
> > .poll()
> > > as
> > > >> >> those are no regular messages. A client would need to opt-in
> to
> > > see
> > > >> >> those messages (either via poll() or maybe a callback). Thus,
> > we
> > > need
> > > >> >> some special (standardized) header IDs that indicate control
> > > messages
> > > >> >> that should not be returned to the user via poll() by
> default.
> > > >> >>
> > > >> >>
> > > >> >> -Matthias
> > > >> >>
> > > >> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
> > > >> >> > Matthias,
> > > >> >> >
> > > >> >> > Thanks for your input. I'm +1 on control messages as they
> > > seem to be
> > > >> >> the
> > > >> >> > simplest way to implement watermarks (
> > > >> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-
> > > streaming-102),
> > > >> a
> > > >> >> > feature that would add a lot of value to Kafka Streams
> IMHO.
> > > >> >> >
> > > >> >> > Your argument that the control-message indicator belongs in
> > the
> > > >> >> client-only
> > > >> >> > section of the record format make sense. Just to make
> sure I
> > > >> >> understand,
> > > >> >> > are you suggesting that control messages would be indicated
> > by
> > > a
> > > >> >> standard
> > > >> >> > reserved header? By standard, I mean that ALL Kafka
> > consumers
> > > would
> > > >> >> know
> > > >> >> > to handle these messages differently (possibly just
> ignoring
> > > them).
> > > >> >> This
> > > >> >> > would need to be added to the specification of the consumer
> > > protocol
> > > >> so
> > > >> >> > that all Kafka clients implement it, right? I think it's a
> > > good idea
> > > >> >> but
> > > >> >> > just checking.
> > > >> >> >
> > > >> >> > Cheers,
> > > >> >> >
> > > >> >> > Roger
> > > >> >> >
> > > >> >> >
> > > >> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> > > >> matthias@confluent.io
> > > >> >> >
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> Yes and no. I did overload the term "control message".
> > > >> >> >>
> > > >> >> >> EOS control messages are for client-broker communication
> and
> > > thus
> > > >> never
> > > >> >> >> exposed to any application. And I think this is a good
> > design
> > > because
> > > >> >> >> broker needs to understand those control messages. Thus,
> > this
> > > should
> > > >> be
> > > >> >> >> a protocol change.
> > > >> >> >>
> > > >> >> >> The type of control messages I have in mind are for
> > > client-client
> > > >> >> >> (application-application) communication and the broker is
> > > agnostic to
> > > >> >> >> them. Thus, it should not be a protocol change.
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> -Matthias
> > > >> >> >>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On 12/14/16 9:42 AM, radai wrote:
> > > >> >> >>> arent control messages getting pushed as their own top
> > level
> > > >> protocol
> > > >> >> >>> change (and a fairly massive one) for the transactions
> KIP
> > ?
> > > >> >> >>>
> > > >> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> > > >> >> matthias@confluent.io>
> > > >> >> >>> wrote:
> > > >> >> >>>
> > > >> >> >>>> Hi,
> > > >> >> >>>>
> > > >> >> >>>> I want to add a completely new angle to this discussion.
> > > For this,
> > > >> I
> > > >> >> >>>> want to propose an extension for the headers feature
> that
> > > enables
> > > >> new
> > > >> >> >>>> uses cases -- and those new use cases might convince
> > people
> > > to
> > > >> >> support
> > > >> >> >>>> headers (of course including the larger scoped
> proposal).
> > > >> >> >>>>
> > > >> >> >>>> Extended Proposal:
> > > >> >> >>>>
> > > >> >> >>>> Allow messages with a certain header key to be special
> > > "control
> > > >> >> >>>> messages" (w/ o w/o payload) that are not exposed to an
> > > application
> > > >> >> via
> > > >> >> >>>> .poll().
> > > >> >> >>>>
> > > >> >> >>>> Thus, a consumer client would automatically skip over
> > those
> > > >> >> messages. If
> > > >> >> >>>> an application knows about embedded control messages, it
> > > can "sing
> > > >> >> up"
> > > >> >> >>>> to those messages by the consumer client and either get
> a
> > > callback
> > > >> or
> > > >> >> >>>> the consumer auto-drop for this messages gets disabled
> > > (allowing to
> > > >> >> >>>> consumer those messages via poll()).
> > > >> >> >>>>
> > > >> >> >>>> (The details need further considerations/discussion. I
> > just
> > > want to
> > > >> >> >>>> sketch the main idea.)
> > > >> >> >>>>
> > > >> >> >>>> Usage:
> > > >> >> >>>>
> > > >> >> >>>> There is a shared topic (ie, used by multiple
> > applications)
> > > and a
> > > >> >> >>>> producer application wants to embed a special message in
> > > the topic
> > > >> >> for a
> > > >> >> >>>> dedicated consumer application. Because only one
> > > application will
> > > >> >> >>>> understand this message, it cannot be a regular message
> as
> > > this
> > > >> would
> > > >> >> >>>> break all applications that do not understand this
> > message.
> > > The
> > > >> >> producer
> > > >> >> >>>> application would set a special metadata key and no
> > consumer
> > > >> >> application
> > > >> >> >>>> would see this control message by default because they
> did
> > > not
> > > >> enable
> > > >> >> >>>> their consumer client to return this message in poll()
> > (and
> > > the
> > > >> >> client
> > > >> >> >>>> would just drop this message with special metadata key).
> > > Only the
> > > >> >> single
> > > >> >> >>>> application that should receive this message, will
> > > subscribe to
> > > >> this
> > > >> >> >>>> message on its consumer client and process it.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Concrete Use Case: Kafka Streams
> > > >> >> >>>>
> > > >> >> >>>> In Kafka Streams, we would like to propagate "control
> > > messages"
> > > >> from
> > > >> >> >>>> subtopology to subtopology. There are multiple scenarios
> > > for which
> > > >> >> this
> > > >> >> >>>> would be useful. For example, currently we do not
> > guarantee
> > > a
> > > >> >> >>>> "consistent shutdown" of an application. By this, I mean
> > > that input
> > > >> >> >>>> records might not be completely processed by the whole
> > > topology
> > > >> >> because
> > > >> >> >>>> the application shutdown happens "in between" and an
> > > intermediate
> > > >> >> result
> > > >> >> >>>> topic gets "stock" in an intermediate topic. Thus, a
> user
> > > would see
> > > >> >> an
> > > >> >> >>>> committed offset of the source topic of the application,
> > > but no
> > > >> >> >>>> corresponding result record in the output topic.
> > > >> >> >>>>
> > > >> >> >>>> Having "shutdown markers" would allow us, to first stop
> > the
> > > >> upstream
> > > >> >> >>>> subtopology and write this marker into the intermediate
> > > topic and
> > > >> the
> > > >> >> >>>> downstream subtopology would only shut down itself after
> > is
> > > sees
> > > >> the
> > > >> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown,
> > that
> > > no
> > > >> >> >>>> "in-flight" messages got stuck in intermediate topics.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> A similar usage would be for KIP-95 (Incremental Batch
> > > Processing).
> > > >> >> >>>> There was a discussion about the proposed metadata
> topic,
> > > and we
> > > >> >> could
> > > >> >> >>>> avoid this metadata topic if we would have "control
> > > messages".
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Right now, we cannot insert an "application control
> > message"
> > > >> because
> > > >> >> >>>> Kafka Streams does not own all topics it read/writes and
> > > thus might
> > > >> >> >>>> break other consumer application (as described above) if
> > we
> > > inject
> > > >> >> >>>> random messages that are not understood by other apps.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Of course, one can work around "embedded control
> messaged"
> > > by using
> > > >> >> an
> > > >> >> >>>> additional topic to propagate control messaged between
> > > application
> > > >> >> (as
> > > >> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka
> > > Streams). But
> > > >> >> there
> > > >> >> >>>> are major concerns about adding this metadata topic in
> the
> > > KIP and
> > > >> >> this
> > > >> >> >>>> shows that other application that need a similar pattern
> > > might
> > > >> profit
> > > >> >> >>>> from topic embedded "control messages", too.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> One last important consideration: those "control
> messages"
> > > are used
> > > >> >> for
> > > >> >> >>>> client to client communication and are not understood by
> > the
> > > >> broker.
> > > >> >> >>>> Thus, those messages should not be enabled within the
> > > message
> > > >> format
> > > >> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> > > record
> > > >> >> headers
> > > >> >> >>>> would be a nice way to implement them. Because KIP-82
> did
> > > consider
> > > >> >> key
> > > >> >> >>>> namespaces for metatdata keys, this extension should not
> > be
> > > an own
> > > >> >> KIP
> > > >> >> >>>> but should be included in KIP-82 to reserve a namespace
> > for
> > > >> "control
> > > >> >> >>>> message" in the first place.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Sorry for the long email... Looking forward to your
> > > feedback.
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> -Matthias
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> > > >> >> >>>>> Hi Jun
> > > >> >> >>>>>
> > > >> >> >>>>> 100) each time a transaction exits a jvm for a remote
> > > system
> > > >> >> (HTTP/JMS/
> > > >> >> >>>> Hopefully one day kafka) the APM tools stich in a unique
> > id
> > > >> (though I
> > > >> >> >>>> believe it contains the end2end uuid embedded in this
> id),
> > > on
> > > >> >> receiving
> > > >> >> >> the
> > > >> >> >>>> message at the receiving JVM the apm code takes this
> out,
> > > and
> > > >> >> continues
> > > >> >> >> its
> > > >> >> >>>> tracing on the that new thread. Both JVM’s (and other
> > > languages the
> > > >> >> APM
> > > >> >> >>>> tool supports) send this data async back to the central
> > > controllers
> > > >> >> >> where
> > > >> >> >>>> the stiching togeather occurs. For this they need some
> > > header space
> > > >> >> for
> > > >> >> >>>> them to put this id.
> > > >> >> >>>>>
> > > >> >> >>>>> 101) Yes indeed we have a business transaction Id in
> the
> > > payload.
> > > >> >> >> Though
> > > >> >> >>>> this is a system level tracing, that we need to have
> marry
> > > up. Also
> > > >> >> as
> > > >> >> >> per
> > > >> >> >>>> note on end2end encryption we’d be unable to prove the
> > flow
> > > if the
> > > >> >> >> payload
> > > >> >> >>>> is encrypted as we’d not have access to this at certain
> > > points of
> > > >> the
> > > >> >> >> flow
> > > >> >> >>>> through the infrastructure/platform.
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> 103) As said we use this mechanism in IG very
> > > successfully, as
> > > >> >> stated
> > > >> >> >>>> per key we guarantee the transaction producing app to
> > > handle the
> > > >> >> >>>> transaction of a key at one DC unless at point of
> critical
> > > failure
> > > >> >> >> where we
> > > >> >> >>>> have to flip processing to another. We care about key
> > > ordering.
> > > >> >> >>>>> I disagree on the offset comment for the partition
> > > solution unless
> > > >> >> you
> > > >> >> >>>> do full ISR, or expensive full XA transactions even with
> > > partitions
> > > >> >> you
> > > >> >> >>>> cannot fully guarantee offsets would match.
> > > >> >> >>>>>
> > > >> >> >>>>> 105) Very much so, I need to have access at the
> platform
> > > level to
> > > >> >> the
> > > >> >> >>>> other meta data all mentioned, without having to need to
> > > have
> > > >> access
> > > >> >> to
> > > >> >> >> the
> > > >> >> >>>> encryption keys of the payload.
> > > >> >> >>>>>
> > > >> >> >>>>> 106)
> > > >> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d
> need
> > > to have
> > > >> a
> > > >> >> >>>> global producerId register which would be very hard to
> > > >> >> enforce/ensure is
> > > >> >> >>>> current and correct, just to understand the message
> > origins
> > > of its
> > > >> >> >>>> region/az/cluster for routing.
> > > >> >> >>>>> The client wrapper version, producerId can be the same,
> > as
> > > >> obviously
> > > >> >> >> the
> > > >> >> >>>> producer could upgrade its wrapper, as such we need to
> > know
> > > what
> > > >> >> wrapper
> > > >> >> >>>> version the message is created with.
> > > >> >> >>>>> Likewise the IP address, as stated we can have our
> > > producer move,
> > > >> >> where
> > > >> >> >>>> its IP would change.
> > > >> >> >>>>>
> > > >> >> >>>>> 107)
> > > >> >> >>>>> UUID is set on the message by interceptors before
> actual
> > > producer
> > > >> >> >>>> transport send. This is for platform level message
> dedupe
> > > >> guarantee,
> > > >> >> the
> > > >> >> >>>> business payload should be agnostic to this. Please see
> > > >> >> >>>> https://activemq.apache.org/
> artemis/docs/1.5.0/duplicate-
> > > det
> > > >> >> ection.html
> > > >> >> >>>> note this is not touching business payloads.
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io>
> > wrote:
> > > >> >> >>>>>
> > > >> >> >>>>> Hi, Michael,
> > > >> >> >>>>>
> > > >> >> >>>>> Thanks for the reply. I find it very helpful.
> > > >> >> >>>>>
> > > >> >> >>>>> Data lineage:
> > > >> >> >>>>> 100. I'd like to understand the APM use case a bit
> > > more. It
> > > >> >> sounds
> > > >> >> >>>> like
> > > >> >> >>>>> that those APM plugins can generate a transaction
> id
> > > that we
> > > >> >> could
> > > >> >> >>>>> potentially put in the header of every message. How
> > > would you
> > > >> >> >>>> typically
> > > >> >> >>>>> make use of such transaction ids? Are there other
> > > metadata
> > > >> >> >>>> associated with
> > > >> >> >>>>> the transaction id and if so, how are they
> propagated
> > > >> >> downstream?
> > > >> >> >>>>>
> > > >> >> >>>>> 101. For the finance use case, if the concept of
> > > transaction
> > > >> is
> > > >> >> >>>> important,
> > > >> >> >>>>> wouldn't it be typically included in the message
> > > payload
> > > >> >> instead of
> > > >> >> >>>> as an
> > > >> >> >>>>> optional header field?
> > > >> >> >>>>>
> > > >> >> >>>>> 102. The data lineage that Altas and Navigator
> > support
> > > seems
> > > >> to
> > > >> >> be
> > > >> >> >>>> at the
> > > >> >> >>>>> dataset level, not per record level? So, not sure
> if
> > > per
> > > >> message
> > > >> >> >>>> headers
> > > >> >> >>>>> are relevant there.
> > > >> >> >>>>>
> > > >> >> >>>>> Mirroring:
> > > >> >> >>>>> 103. The benefit of using separate partitions is
> that
> > > it
> > > >> >> >> potentially
> > > >> >> >>>> makes
> > > >> >> >>>>> it easy to preserve offsets during mirroring. This
> > > will make
> > > >> it
> > > >> >> >>>> easier for
> > > >> >> >>>>> consumer to switch clusters. Currently, the
> consumers
> > > can
> > > >> switch
> > > >> >> >>>> clusters
> > > >> >> >>>>> by using the timestampToOffset() api, but it has to
> > > deal with
> > > >> >> >>>> duplicates.
> > > >> >> >>>>> Good point on the issue with log compact and I am
> not
> > > sure how
> > > >> >> to
> > > >> >> >>>> address
> > > >> >> >>>>> this. However, even if we mirror into the existing
> > > partitions,
> > > >> >> the
> > > >> >> >>>> ordering
> > > >> >> >>>>> for messages generated from different clusters
> seems
> > > >> >> >>>> non-deterministic
> > > >> >> >>>>> anyway. So, it seems that the consumers already
> have
> > > to deal
> > > >> >> with
> > > >> >> >>>> that? If
> > > >> >> >>>>> a topic is compacted, does that mean which messages
> > are
> > > >> >> preserved
> > > >> >> >> is
> > > >> >> >>>> also
> > > >> >> >>>>> non-deterministic across clusters?
> > > >> >> >>>>>
> > > >> >> >>>>> 104. Good point on partition key.
> > > >> >> >>>>>
> > > >> >> >>>>> End-to-end encryption:
> > > >> >> >>>>> 105. So, it seems end-to-end encryption is useful.
> > Are
> > > headers
> > > >> >> >>>> useful there?
> > > >> >> >>>>>
> > > >> >> >>>>> Auditing:
> > > >> >> >>>>> 106. It seems other than the UUID, all other
> metadata
> > > are per
> > > >> >> >>>> producer?
> > > >> >> >>>>>
> > > >> >> >>>>> EOS:
> > > >> >> >>>>> 107. How are those UUIDs generated? I am not sure
> if
> > > they can
> > > >> be
> > > >> >> >>>> generated
> > > >> >> >>>>> in the producer library. An application may send
> > > messages
> > > >> >> through a
> > > >> >> >>>> load
> > > >> >> >>>>> balancer and on retry, the same message could be
> > > routed to a
> > > >> >> >>>> different
> > > >> >> >>>>> producer instance. So, it seems that the
> application
> > > has to
> > > >> >> >> generate
> > > >> >> >>>> the
> > > >> >> >>>>> UUIDs. In that case, shouldn't the application just
> > > put the
> > > >> >> UUID in
> > > >> >> >>>> the
> > > >> >> >>>>> payload?
> > > >> >> >>>>>
> > > >> >> >>>>> Thanks,
> > > >> >> >>>>>
> > > >> >> >>>>> Jun
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> > > >> >> >>>> Michael.Pearce@ig.com>
> > > >> >> >>>>> wrote:
> > > >> >> >>>>>
> > > >> >> >>>>> > Hi Jun.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Per Transaction Tracing / Data Lineage.
> > > >> >> >>>>> >
> > > >> >> >>>>> > As Stated in the KIP this has the first use case
> of
> > > how many
> > > >> >> APM
> > > >> >> >>>> tools now
> > > >> >> >>>>> > work.
> > > >> >> >>>>> > I would find it impossible for any one to argue
> > this
> > > is not
> > > >> >> >>>> important or a
> > > >> >> >>>>> > niche market as it has its own gartner report for
> > > this
> > > >> space.
> > > >> >> >> Such
> > > >> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace,
> > > Hawqular are
> > > >> >> but a
> > > >> >> >>>> few.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Likewise these APM tools can help very rapidly
> > track
> > > down
> > > >> >> issues
> > > >> >> >>>> and
> > > >> >> >>>>> > automatically capture metrics, perform actions
> > based
> > > on
> > > >> >> >> unexpected
> > > >> >> >>>> behavior
> > > >> >> >>>>> > to auto recover services.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Before mentioning looking at aggregated stats, in
> > > these
> > > >> cases
> > > >> >> >> where
> > > >> >> >>>>> > actually on critical flows we cannot afford to
> have
> > > >> aggregated
> > > >> >> >>>> rolled up
> > > >> >> >>>>> > stats only.
> > > >> >> >>>>> >
> > > >> >> >>>>> > With the APM tool we use its actually able to
> > detect
> > > a
> > > >> single
> > > >> >> >>>> transaction
> > > >> >> >>>>> > failure and capture the thread traces in the JVM
> > > where it
> > > >> >> failed
> > > >> >> >>>> and
> > > >> >> >>>>> > everything for us, to the point it sends us
> alerts
> > > where we
> > > >> >> have
> > > >> >> >>>> this
> > > >> >> >>>>> > giving the line number of the code that caused
> it,
> > > the
> > > >> >> >> transaction
> > > >> >> >>>> trace
> > > >> >> >>>>> > through all the services and endpoints
> (supported)
> > > upto the
> > > >> >> point
> > > >> >> >>>> of
> > > >> >> >>>>> > failure, it can also capture the data in and out
> > (so
> > > we can
> > > >> >> >>>> replay).
> > > >> >> >>>>> > Because atm Kafka doesn’t support us being able
> to
> > > stich in
> > > >> >> these
> > > >> >> >>>> tracing
> > > >> >> >>>>> > transaction ids natively, we cannot get these
> > > benefits as
> > > >> >> such is
> > > >> >> >>>> limiting
> > > >> >> >>>>> > our ability support apps and monitor them to the
> > same
> > > >> >> standards
> > > >> >> >> we
> > > >> >> >>>> come to
> > > >> >> >>>>> > expect when on a kafka flow.
> > > >> >> >>>>> >
> > > >> >> >>>>> > This actually ties in with Data Lineage, as the
> > same
> > > tracing
> > > >> >> can
> > > >> >> >>>> be used
> > > >> >> >>>>> > to back stich this. Essentially many times due to
> > > the sums
> > > >> of
> > > >> >> >> money
> > > >> >> >>>>> > involved there are disputes, and typically as a
> > > financial
> > > >> >> >>>> institute the
> > > >> >> >>>>> > easiest and cleanest way to prove when disputes
> > > arise is to
> > > >> >> >>>> present the
> > > >> >> >>>>> > actual flow and processes involved in a
> > transaction.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Likewise as Hadoop matures its evident this case
> is
> > > >> >> important, as
> > > >> >> >>>> tools
> > > >> >> >>>>> > such as Atlas (Hortonworks led) and Navigator
> > > (cloudera led)
> > > >> >> are
> > > >> >> >>>> evident
> > > >> >> >>>>> > also I believe the importance here is very much
> NOT
> > > just a
> > > >> >> >>>> financial issue.
> > > >> >> >>>>> >
> > > >> >> >>>>> > From a MDM point of view any company wanting to
> > care
> > > about
> > > >> >> Data
> > > >> >> >>>> Quality
> > > >> >> >>>>> > and Data Governance - Data Lineage is a key piece
> > in
> > > this
> > > >> >> puzzle.
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE Mirroring,
> > > >> >> >>>>> >
> > > >> >> >>>>> > As per the KIP in-fact this is exactly what we do
> > re
> > > cluster
> > > >> >> id,
> > > >> >> >>>> to mirror
> > > >> >> >>>>> > a network of clusters between AZ’s / Regions. We
> > > know a
> > > >> >> >>>> transaction for a
> > > >> >> >>>>> > key will be done within a AZ/Region, as such we
> > > know the
> > > >> >> write
> > > >> >> >> to
> > > >> >> >>>> kafka
> > > >> >> >>>>> > would be ordered per key. But we need eventual
> view
> > > of that
> > > >> >> >> across
> > > >> >> >>>> in our
> > > >> >> >>>>> > other regions/az’s. When we have complete AZ or
> > > Region
> > > >> >> failure we
> > > >> >> >>>> know
> > > >> >> >>>>> > there will be a brief interruption whilst those
> > > transactions
> > > >> >> are
> > > >> >> >>>> moved to
> > > >> >> >>>>> > another region but we expect after it to
> continue.
> > > >> >> >>>>> >
> > > >> >> >>>>> > As mentioned having separate Partions to do this
> > > starts to
> > > >> get
> > > >> >> >>>>> > ugly/complicated for us:
> > > >> >> >>>>> > how would I do compaction where a key is in two
> > > partitions?
> > > >> >> >>>>> > How do we balance consumers so where multiple
> > > partitions
> > > >> with
> > > >> >> the
> > > >> >> >>>> same key
> > > >> >> >>>>> > goto the same consumer
> > > >> >> >>>>> > What do you do if cluster 1 has 5 partitions but
> > > cluster 20
> > > >> >> has
> > > >> >> >> 10
> > > >> >> >>>> because
> > > >> >> >>>>> > its larger kit in our more core DC’s, as such key
> > to
> > > >> partition
> > > >> >> >>>> mappings for
> > > >> >> >>>>> > consumers get even more complicated.
> > > >> >> >>>>> > What do you do if we add or remove a complete
> > region
> > > >> >> >>>>> >
> > > >> >> >>>>> > Where as simple mirror will work we just need to
> > > ensure we
> > > >> >> don’t
> > > >> >> >>>> have a
> > > >> >> >>>>> > cycle which we can do with clusterId.
> > > >> >> >>>>> >
> > > >> >> >>>>> > We even have started to look at shortest path
> > mirror
> > > routing
> > > >> >> >> based
> > > >> >> >>>> on
> > > >> >> >>>>> > clusterId, if we also had the region and az info
> on
> > > the
> > > >> >> >> originating
> > > >> >> >>>>> > message, this we have not implemented but some
> > ideas
> > > come
> > > >> from
> > > >> >> >>>> network
> > > >> >> >>>>> > routing, and also the dispatcher router in apache
> > > qpid.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Also we need to have data perimeters e.g. certain
> > > data
> > > >> cannot
> > > >> >> >> leave
> > > >> >> >>>>> > certain countries borders. We want this all
> > > automated so
> > > >> that
> > > >> >> at
> > > >> >> >>>> the
> > > >> >> >>>>> > platform level without having to touch or look at
> > the
> > > >> business
> > > >> >> >>>> data inside
> > > >> >> >>>>> > we can have headers we can put tags into so that
> we
> > > can
> > > >> ensure
> > > >> >> >>>> this doesn’t
> > > >> >> >>>>> > occur when we mirror. (actually links in to data
> > > lineage /
> > > >> >> >> tracing
> > > >> >> >>>> as again
> > > >> >> >>>>> > we need to tag messages at a platform level)
> > > Examples are we
> > > >> >> are
> > > >> >> >>>> not
> > > >> >> >>>>> > allowed Private customer details to leave
> > > Switzerland, yet
> > > >> we
> > > >> >> >> need
> > > >> >> >>>> those
> > > >> >> >>>>> > systems integrated.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Lastly around mirroring we have a partionKey
> field,
> > > as the
> > > >> key
> > > >> >> >>>> used for
> > > >> >> >>>>> > portioning logic != compaction key all the time
> but
> > > we want
> > > >> to
> > > >> >> >>>> preserve it
> > > >> >> >>>>> > for when we mirror so that if source cluster
> > > partition count
> > > >> >> !=
> > > >> >> >>>> destination
> > > >> >> >>>>> > cluster partition count we can honour the same
> > > partitioning
> > > >> >> >> logic.
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE End 2 End encryption
> > > >> >> >>>>> >
> > > >> >> >>>>> > As I believe mentioned just before, the solution
> > you
> > > mention
> > > >> >> just
> > > >> >> >>>> doesn’t
> > > >> >> >>>>> > cut the mustard these days with many regulators.
> An
> > > >> operations
> > > >> >> >>>> person with
> > > >> >> >>>>> > access to the box should not be able to have
> access
> > > to the
> > > >> >> data.
> > > >> >> >>>> Many now
> > > >> >> >>>>> > actually impose quite literally the
> implementation
> > > expected
> > > >> >> being
> > > >> >> >>>> end2end
> > > >> >> >>>>> > encryption for certain data (Singapore for us is
> > one
> > > that I
> > > >> am
> > > >> >> >>>> most aware
> > > >> >> >>>>> > of). In fact we’re even now needing encrypt the
> > data
> > > and
> > > >> store
> > > >> >> >> the
> > > >> >> >>>> keys in
> > > >> >> >>>>> > HSM modules.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Likewise the performance penalty on encrypting
> > > decrypting as
> > > >> >> you
> > > >> >> >>>> produce
> > > >> >> >>>>> > over wire, then again encrypt decrypt as the data
> > is
> > > stored
> > > >> on
> > > >> >> >> the
> > > >> >> >>>> brokers
> > > >> >> >>>>> > disks and back again, then again encrypted and
> > > decrypted
> > > >> back
> > > >> >> >> over
> > > >> >> >>>> the wire
> > > >> >> >>>>> > each time for each consumer all adds up, ignoring
> > > this
> > > >> >> doubling
> > > >> >> >>>> with mirror
> > > >> >> >>>>> > makers etc. simply encrypting the value once on
> > > write by the
> > > >> >> >>>> client and
> > > >> >> >>>>> > again decrypting on consume by the consumer is
> far
> > > more
> > > >> >> >>>> performant, but
> > > >> >> >>>>> > then the routing and platform meta data needs to
> be
> > > separate
> > > >> >> >> (thus
> > > >> >> >>>> headers)
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE Auditing:
> > > >> >> >>>>> >
> > > >> >> >>>>> > Our Auditing needs are:
> > > >> >> >>>>> > Producer Id,
> > > >> >> >>>>> > Origin Cluster Id that message first produced
> into
> > > >> >> >>>>> > Origin AZ – agreed we can derive this if we have
> > > cluster id,
> > > >> >> but
> > > >> >> >>>> it makes
> > > >> >> >>>>> > resolving this for audit reporting a lot easier.
> > > >> >> >>>>> > Origin Region – agreed we can derive this if we
> > have
> > > cluster
> > > >> >> id,
> > > >> >> >>>> but it
> > > >> >> >>>>> > makes resolving this for audit reporting a lot
> > > easier.
> > > >> >> >>>>> > Unique Message Identification (this is not the
> same
> > > as
> > > >> >> >> transaction
> > > >> >> >>>>> > tracing) – note offset and partition are not the
> > > same, as
> > > >> >> when we
> > > >> >> >>>> mirror or
> > > >> >> >>>>> > have for what ever system failure duplicate send,
> > > >> >> >>>>> > Custom Client wrapper version (where
> organizations
> > > have to
> > > >> >> wrap
> > > >> >> >>>> the kafka
> > > >> >> >>>>> > client for added features) so we know what
> version
> > > of the
> > > >> >> wrapper
> > > >> >> >>>> is used
> > > >> >> >>>>> > Producer IP address (in case of clients being in
> > our
> > > vm/open
> > > >> >> >> stack
> > > >> >> >>>> infra
> > > >> >> >>>>> > where they can move around, producer id will stay
> > > the same
> > > >> but
> > > >> >> >>>> this would
> > > >> >> >>>>> > change)
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > RE Once and only once delivery case
> > > >> >> >>>>> >
> > > >> >> >>>>> > Using the same Message UUID for auditing we can
> > > achieve this
> > > >> >> >> quite
> > > >> >> >>>> simply.
> > > >> >> >>>>> >
> > > >> >> >>>>> > As per how some other brokers do this (cough
> qpid,
> > > artemis)
> > > >> >> >>>> message uuid
> > > >> >> >>>>> > are used to dedupe where message is sent and
> > > produced but
> > > >> the
> > > >> >> >>>> client didn’t
> > > >> >> >>>>> > receive the ack, and there for replays the send,
> by
> > > having a
> > > >> >> >>>> unique message
> > > >> >> >>>>> > id per message, this can be filtered out, on
> > > consumers where
> > > >> >> >>>> message
> > > >> >> >>>>> > delivery may occur twice for what ever reasons a
> > > message
> > > >> uuid
> > > >> >> can
> > > >> >> >>>> be used
> > > >> >> >>>>> > to remove duplicates being deliverd , like wise
> we
> > > can do
> > > >> >> this in
> > > >> >> >>>> the
> > > >> >> >>>>> > mirrormakers so if we detect a dupe message we
> can
> > > avoid
> > > >> >> >>>> replicating it.
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > Cheers
> > > >> >> >>>>> > Mike
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <
> jun@confluent.io>
> > > wrote:
> > > >> >> >>>>> >
> > > >> >> >>>>> > Since this KIP affects message format, wire
> > > protocol,
> > > >> >> apis, I
> > > >> >> >>>> think
> > > >> >> >>>>> > it's
> > > >> >> >>>>> > worth spending a bit more time to nail down
> the
> > > concrete
> > > >> >> use
> > > >> >> >>>> cases. It
> > > >> >> >>>>> > would be bad if we add this feature, but when
> > > start
> > > >> >> >>>> implementing it
> > > >> >> >>>>> > for say
> > > >> >> >>>>> > mirroring, we then realize that header is not
> > > the best
> > > >> >> >>>> approach.
> > > >> >> >>>>> > Initially,
> > > >> >> >>>>> > I thought I was convinced of the use cases of
> > > headers
> > > >> and
> > > >> >> was
> > > >> >> >>>> trying to
> > > >> >> >>>>> > write down a few use cases to convince
> others.
> > > That's
> > > >> >> when I
> > > >> >> >>>> became
> > > >> >> >>>>> > less
> > > >> >> >>>>> > certain. For me to be convinced, I just want
> to
> > > see two
> > > >> >> >> strong
> > > >> >> >>>> use
> > > >> >> >>>>> > cases
> > > >> >> >>>>> > (instead of 10 maybe use cases) in the
> > > third-party
> > > >> space.
> > > >> >> The
> > > >> >> >>>> reason is
> > > >> >> >>>>> > that when we discussed the use cases within a
> > > company,
> > > >> >> often
> > > >> >> >>>> it ends
> > > >> >> >>>>> > with
> > > >> >> >>>>> > "we can't force everyone to use this standard
> > > since we
> > > >> may
> > > >> >> >>>> have to
> > > >> >> >>>>> > integrate with third-party tools".
> > > >> >> >>>>> >
> > > >> >> >>>>> > At present, I am not sure why headers are
> > useful
> > > for
> > > >> >> things
> > > >> >> >>>> like
> > > >> >> >>>>> > schemaId
> > > >> >> >>>>> > or encryption. In order to do anything useful
> > to
> > > the
> > > >> >> value,
> > > >> >> >>>> one needs
> > > >> >> >>>>> > to
> > > >> >> >>>>> > know the schemaId or how data is encrypted,
> but
> > > header
> > > >> is
> > > >> >> >>>> optional.
> > > >> >> >>>>> > But, I
> > > >> >> >>>>> > can be convinced if someone (Radai, Sean,
> > Todd?)
> > > >> provides
> > > >> >> >> more
> > > >> >> >>>> details
> > > >> >> >>>>> > on
> > > >> >> >>>>> > the argument.
> > > >> >> >>>>> >
> > > >> >> >>>>> > I am not very sure header is the best
> approach
> > > for
> > > >> >> mirroring
> > > >> >> >>>> either. If
> > > >> >> >>>>> > someone has thought about this more, I'd be
> > > happy to
> > > >> hear.
> > > >> >> >>>>> >
> > > >> >> >>>>> > I can see the data lineage use case. I am
> just
> > > not sure
> > > >> >> how
> > > >> >> >>>> widely
> > > >> >> >>>>> > applicable this is. If someone familiar with
> > > this space
> > > >> >> can
> > > >> >> >>>> justify
> > > >> >> >>>>> > this is
> > > >> >> >>>>> > a significant use case, say in the finance
> > > industry,
> > > >> this
> > > >> >> >>>> would be a
> > > >> >> >>>>> > strong
> > > >> >> >>>>> > use case.
> > > >> >> >>>>> >
> > > >> >> >>>>> > I can see the auditing use case. I am just
> not
> > > sure if a
> > > >> >> >> native
> > > >> >> >>>>> > producer id
> > > >> >> >>>>> > solves that problem. If there are additional
> > > metadata
> > > >> >> that's
> > > >> >> >>>> worth
> > > >> >> >>>>> > collecting but not covered by the producer
> id,
> > > that
> > > >> would
> > > >> >> >> make
> > > >> >> >>>> this a
> > > >> >> >>>>> > strong use case.
> > > >> >> >>>>> >
> > > >> >> >>>>> > Thanks,
> > > >> >> >>>>> >
> > > >> >> >>>>> > Jun
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> > > >> >> >>>> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > wrote:
> > > >> >> >>>>> >
> > > >> >> >>>>> > > this KIP is about enabling headers, nothing
> > > more
> > > >> nothing
> > > >> >> >>>> less - so
> > > >> >> >>>>> > no,
> > > >> >> >>>>> > > broker-side use of headers is not in the
> KIP
> > > scope.
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > obviously though, once you have headers
> > > potential use
> > > >> >> cases
> > > >> >> >>>> could
> > > >> >> >>>>> > include
> > > >> >> >>>>> > > broker-side header-aware interceptors
> (which
> > > would be
> > > >> >> the
> > > >> >> >>>> topic of
> > > >> >> >>>>> > other
> > > >> >> >>>>> > > future KIPs).
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > a trivially clear use case (to me) would be
> > > using such
> > > >> >> >>>> broker-side
> > > >> >> >>>>> > > interceptors to enforce compliance with
> > > organizational
> > > >> >> >>>> policies - it
> > > >> >> >>>>> > would
> > > >> >> >>>>> > > make our SREs lives much easier if instead
> of
> > > >> >> retroactively
> > > >> >> >>>>> > discovering
> > > >> >> >>>>> > > "rogue" topics/users those messages would
> > have
> > > been
> > > >> >> >> rejected
> > > >> >> >>>>> > up-front.
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > the kafka broker code is lacking any such
> > > >> extensibility
> > > >> >> >>>> support
> > > >> >> >>>>> > (beyond
> > > >> >> >>>>> > > maybe authorizer) which is why these use
> > cases
> > > were
> > > >> left
> > > >> >> >> out
> > > >> >> >>>> of the
> > > >> >> >>>>> > "case
> > > >> >> >>>>> > > for headers" doc - broker extensibility is
> a
> > > separate
> > > >> >> >>>> discussion.
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen
> > Shapira <
> > > >> >> >>>> gwen@confluent.io>
> > > >> >> >>>>> > wrote:
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > > Woah, I wasn't aware this is something
> > we'll
> > > do. It
> > > >> >> >> wasn't
> > > >> >> >>>> in the
> > > >> >> >>>>> > KIP,
> > > >> >> >>>>> > > > right?
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > I guess we could do it the same way ACLs
> > > currently
> > > >> >> work.
> > > >> >> >>>>> > > > I had in mind something that will allow
> > > admins to
> > > >> >> apply
> > > >> >> >>>> rules to
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > new create/delete/config topic APIs. So
> > Todd
> > > can
> > > >> >> decide
> > > >> >> >> to
> > > >> >> >>>> reject
> > > >> >> >>>>> > > > "create topic" requests that ask for more
> > > than 40
> > > >> >> >>>> partitions, or
> > > >> >> >>>>> > > > require exactly 3 replicas, or no more
> than
> > > 50GB
> > > >> >> >> partition
> > > >> >> >>>> size,
> > > >> >> >>>>> > etc.
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are
> > > planning to
> > > >> >> apply
> > > >> >> >>>> more
> > > >> >> >>>>> > rules
> > > >> >> >>>>> > > > to requests (and I think we should), we
> may
> > > want a
> > > >> bit
> > > >> >> >>>> more generic
> > > >> >> >>>>> > > > design around that.
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> > > >> >> >>>> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > wrote:
> > > >> >> >>>>> > > > > "wouldn't you be in the business of
> > making
> > > sure
> > > >> >> >> everyone
> > > >> >> >>>> uses
> > > >> >> >>>>> > them
> > > >> >> >>>>> > > > > properly?"
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > > thats where a broker-side plugin would
> > > come handy
> > > >> -
> > > >> >> any
> > > >> >> >>>> incoming
> > > >> >> >>>>> > > message
> > > >> >> >>>>> > > > > that does not conform to org policy
> (read
> > > - does
> > > >> not
> > > >> >> >>>> have the
> > > >> >> >>>>> > proper
> > > >> >> >>>>> > > > > headers) gets thrown out (with an error
> > > returned
> > > >> to
> > > >> >> >> user)
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> > > Palino <
> > > >> >> >>>> tpalino@gmail.com>
> > > >> >> >>>>> > wrote:
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on
> > > this one
> > > >> :)
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> Producing counts to a topic is part of
> > > it, but
> > > >> >> that’s
> > > >> >> >>>> only
> > > >> >> >>>>> > part. So
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> count you have 100 messages in topic
> A.
> > > When you
> > > >> >> >> mirror
> > > >> >> >>>> topic A
> > > >> >> >>>>> > to
> > > >> >> >>>>> > > > another
> > > >> >> >>>>> > > > >> cluster, you have 99 messages. Where
> was
> > > your
> > > >> >> problem?
> > > >> >> >>>> Or
> > > >> >> >>>>> > worse, you
> > > >> >> >>>>> > > > have
> > > >> >> >>>>> > > > >> 100 messages, but one producer
> > duplicated
> > > >> messages
> > > >> >> and
> > > >> >> >>>> another
> > > >> >> >>>>> > one
> > > >> >> >>>>> > > lost
> > > >> >> >>>>> > > > >> messages. You need details about where
> > the
> > > >> message
> > > >> >> >> came
> > > >> >> >>>> from in
> > > >> >> >>>>> > order
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> pinpoint problems when they happen.
> > Source
> > > >> producer
> > > >> >> >>>> info, where
> > > >> >> >>>>> > it was
> > > >> >> >>>>> > > > >> produced into your infrastructure, and
> > > when it
> > > >> was
> > > >> >> >>>> produced.
> > > >> >> >>>>> > This
> > > >> >> >>>>> > > > requires
> > > >> >> >>>>> > > > >> you to add the information to the
> > message.
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> And yes, you still need to maintain
> your
> > > clients.
> > > >> >> So
> > > >> >> >>>> maybe my
> > > >> >> >>>>> > original
> > > >> >> >>>>> > > > >> example was not the best. My thoughts
> on
> > > not
> > > >> >> wanting
> > > >> >> >> to
> > > >> >> >>>> be
> > > >> >> >>>>> > responsible
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> message formats stands, because that’s
> > > very much
> > > >> >> >>>> separate from
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > client.
> > > >> >> >>>>> > > > >> As you know, we have our own internal
> > > client
> > > >> >> library
> > > >> >> >>>> that can
> > > >> >> >>>>> > insert
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> right headers, and right now inserts
> the
> > > right
> > > >> >> audit
> > > >> >> >>>>> > information into
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> message fields. If they exist, and
> > > assuming the
> > > >> >> >> message
> > > >> >> >>>> is Avro
> > > >> >> >>>>> > > encoded.
> > > >> >> >>>>> > > > >> What if someone wants to use JSON
> > instead
> > > for a
> > > >> >> good
> > > >> >> >>>> reason?
> > > >> >> >>>>> > What if
> > > >> >> >>>>> > > > user X
> > > >> >> >>>>> > > > >> wants to encrypt messages, but user Y
> > > does not?
> > > >> >> >>>> Maintaining the
> > > >> >> >>>>> > client
> > > >> >> >>>>> > > > >> library is still much easier than
> > > maintaining the
> > > >> >> >>>> message
> > > >> >> >>>>> > formats.
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> -Todd
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> > > Shapira <
> > > >> >> >>>> gwen@confluent.io
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > wrote:
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> > Based on your last sentence,
> consider
> > me
> > > >> >> convinced
> > > >> >> >> :)
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > I get why headers are critical for
> > > Mirroring
> > > >> (you
> > > >> >> >>>> need tags to
> > > >> >> >>>>> > > prevent
> > > >> >> >>>>> > > > >> > loops and sometimes to route
> messages
> > > to the
> > > >> >> correct
> > > >> >> >>>>> > destination).
> > > >> >> >>>>> > > > >> > But why do you need headers to
> audit?
> > > We are
> > > >> >> >> auditing
> > > >> >> >>>> by
> > > >> >> >>>>> > producing
> > > >> >> >>>>> > > > >> > counts to a side topic (and I was
> > under
> > > the
> > > >> >> >>>> impression you do
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > >> > same), so we never need to modify
> the
> > > message.
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > Another thing - after we added
> > headers,
> > > >> wouldn't
> > > >> >> you
> > > >> >> >>>> be in the
> > > >> >> >>>>> > > > >> > business of making sure everyone
> uses
> > > them
> > > >> >> properly?
> > > >> >> >>>> Making
> > > >> >> >>>>> > sure
> > > >> >> >>>>> > > > >> > everyone includes the right headers
> > you
> > > need,
> > > >> not
> > > >> >> >>>> using the
> > > >> >> >>>>> > header
> > > >> >> >>>>> > > > >> > names you intend to use, etc. I
> don't
> > > think the
> > > >> >> >>>> "policing"
> > > >> >> >>>>> > business
> > > >> >> >>>>> > > > >> > will ever go away.
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> > > Palino <
> > > >> >> >>>>> > tpalino@gmail.com>
> > > >> >> >>>>> > > > wrote:
> > > >> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not
> very
> > > happy
> > > >> with
> > > >> >> the
> > > >> >> >>>>> > workaround.
> > > >> >> >>>>> > > Avro
> > > >> >> >>>>> > > > >> means
> > > >> >> >>>>> > > > >> > > that I have to be concerned with
> the
> > > format
> > > >> of
> > > >> >> the
> > > >> >> >>>> messages
> > > >> >> >>>>> > in
> > > >> >> >>>>> > > > order to
> > > >> >> >>>>> > > > >> > run
> > > >> >> >>>>> > > > >> > > the infrastructure (audit,
> > mirroring,
> > > etc.).
> > > >> >> That
> > > >> >> >>>> means
> > > >> >> >>>>> > that I
> > > >> >> >>>>> > > have
> > > >> >> >>>>> > > > to
> > > >> >> >>>>> > > > >> > > handle the schemas, and I have to
> > > enforce
> > > >> rules
> > > >> >> >>>> about good
> > > >> >> >>>>> > > formats.
> > > >> >> >>>>> > > > >> This
> > > >> >> >>>>> > > > >> > is
> > > >> >> >>>>> > > > >> > > not something I want to be in the
> > > business
> > > >> of,
> > > >> >> >>>> because I
> > > >> >> >>>>> > should be
> > > >> >> >>>>> > > > able
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > > run a service infrastructure
> without
> > > needing
> > > >> >> to be
> > > >> >> >>>> in the
> > > >> >> >>>>> > weeds of
> > > >> >> >>>>> > > > >> > dealing
> > > >> >> >>>>> > > > >> > > with customer data formats.
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > > Trust me, a sizable portion of my
> > > support
> > > >> time
> > > >> >> is
> > > >> >> >>>> spent
> > > >> >> >>>>> > dealing
> > > >> >> >>>>> > > with
> > > >> >> >>>>> > > > >> > schema
> > > >> >> >>>>> > > > >> > > issues. I really would like to get
> > > away from
> > > >> >> that.
> > > >> >> >>>> Maybe
> > > >> >> >>>>> > I'd have
> > > >> >> >>>>> > > > more
> > > >> >> >>>>> > > > >> > time
> > > >> >> >>>>> > > > >> > > for other hobbies. Like writing.
> ;)
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > > -Todd
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM
> Gwen
> > > Shapira <
> > > >> >> >>>>> > gwen@confluent.io>
> > > >> >> >>>>> > > > wrote:
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > >> I'm pretty satisfied with the
> > current
> > > >> >> workarounds
> > > >> >> >>>> (Avro
> > > >> >> >>>>> > container
> > > >> >> >>>>> > > > >> > >> format), so I'm not too excited
> > > about the
> > > >> >> extra
> > > >> >> >>>> work
> > > >> >> >>>>> > required to
> > > >> >> >>>>> > > do
> > > >> >> >>>>> > > > >> > >> headers in Kafka. I absolutely
> > don't
> > > mind it
> > > >> >> if
> > > >> >> >>>> you do
> > > >> >> >>>>> > it...
> > > >> >> >>>>> > > > >> > >> I think the Apache convention for
> > > "good
> > > >> idea,
> > > >> >> but
> > > >> >> >>>> not
> > > >> >> >>>>> > willing to
> > > >> >> >>>>> > > > put
> > > >> >> >>>>> > > > >> > >> any work toward it" is +0.5?
> > anyway,
> > > that's
> > > >> >> what
> > > >> >> >> I
> > > >> >> >>>> was
> > > >> >> >>>>> > trying to
> > > >> >> >>>>> > > > >> > >> convey :)
> > > >> >> >>>>> > > > >> > >>
> > > >> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM,
> > Todd
> > > Palino
> > > >> <
> > > >> >> >>>>> > tpalino@gmail.com>
> > > >> >> >>>>> > > > >> wrote:
> > > >> >> >>>>> > > > >> > >> > Well I guess my question for
> you,
> > > then, is
> > > >> >> what
> > > >> >> >>>> is
> > > >> >> >>>>> > holding you
> > > >> >> >>>>> > > > back
> > > >> >> >>>>> > > > >> > from
> > > >> >> >>>>> > > > >> > >> > full support for headers?
> What’s
> > > the bit
> > > >> >> that
> > > >> >> >>>> you’re
> > > >> >> >>>>> > missing
> > > >> >> >>>>> > > that
> > > >> >> >>>>> > > > >> has
> > > >> >> >>>>> > > > >> > you
> > > >> >> >>>>> > > > >> > >> > under a full +1?
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> > -Todd
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> > > Gwen
> > > >> >> Shapira <
> > > >> >> >>>>> > > gwen@confluent.io>
> > > >> >> >>>>> > > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> >> I know why people who support
> > > headers
> > > >> >> support
> > > >> >> >>>> them, and
> > > >> >> >>>>> > I've
> > > >> >> >>>>> > > > seen
> > > >> >> >>>>> > > > >> > what
> > > >> >> >>>>> > > > >> > >> >> the discussion is like.
> > > >> >> >>>>> > > > >> > >> >>
> > > >> >> >>>>> > > > >> > >> >> This is why I'm asking people
> > who
> > > are
> > > >> >> against
> > > >> >> >>>> headers
> > > >> >> >>>>> > > > (especially
> > > >> >> >>>>> > > > >> > >> >> committers) what will make
> them
> > > change
> > > >> >> their
> > > >> >> >>>> mind - so
> > > >> >> >>>>> > we can
> > > >> >> >>>>> > > > get
> > > >> >> >>>>> > > > >> > this
> > > >> >> >>>>> > > > >> > >> >> part over one way or another.
> > > >> >> >>>>> > > > >> > >> >>
> > > >> >> >>>>> > > > >> > >> >> If I sound frustrated it is
> not
> > > at Radai,
> > > >> >> Jun
> > > >> >> >>>> or you
> > > >> >> >>>>> > (Todd)...
> > > >> >> >>>>> > > > I am
> > > >> >> >>>>> > > > >> > >> >> just looking for something
> > > concrete we
> > > >> can
> > > >> >> do
> > > >> >> >>>> to move
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > >> discussion
> > > >> >> >>>>> > > > >> > >> >> along to the yummy design
> > details
> > > (which
> > > >> is
> > > >> >> >> the
> > > >> >> >>>>> > argument I
> > > >> >> >>>>> > > > really
> > > >> >> >>>>> > > > >> am
> > > >> >> >>>>> > > > >> > >> >> looking forward to).
> > > >> >> >>>>> > > > >> > >> >>
> > > >> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53
> PM,
> > > Todd
> > > >> >> Palino <
> > > >> >> >>>>> > > tpalino@gmail.com>
> > > >> >> >>>>> > > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> > So, Gwen, to your question
> > > (even though
> > > >> >> I’m
> > > >> >> >>>> not a
> > > >> >> >>>>> > > > committer)...
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > I have always been a strong
> > > supporter
> > > >> of
> > > >> >> >>>> introducing
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > concept
> > > >> >> >>>>> > > > >> > of an
> > > >> >> >>>>> > > > >> > >> >> > envelope to messages, which
> > > headers
> > > >> >> >>>> accomplishes. The
> > > >> >> >>>>> > > message
> > > >> >> >>>>> > > > key
> > > >> >> >>>>> > > > >> > is
> > > >> >> >>>>> > > > >> > >> >> > already an example of a
> piece
> > of
> > > >> envelope
> > > >> >> >>>>> > information. By
> > > >> >> >>>>> > > > >> > providing a
> > > >> >> >>>>> > > > >> > >> >> means
> > > >> >> >>>>> > > > >> > >> >> > to do this within Kafka
> > itself,
> > > and not
> > > >> >> >>>> relying on
> > > >> >> >>>>> > use-case
> > > >> >> >>>>> > > > >> > specific
> > > >> >> >>>>> > > > >> > >> >> > implementations, you make it
> > > much
> > > >> easier
> > > >> >> for
> > > >> >> >>>>> > components to
> > > >> >> >>>>> > > > >> > >> interoperate.
> > > >> >> >>>>> > > > >> > >> >> It
> > > >> >> >>>>> > > > >> > >> >> > simplifies development of
> all
> > > these
> > > >> >> things
> > > >> >> >>>> (message
> > > >> >> >>>>> > routing,
> > > >> >> >>>>> > > > >> > auditing,
> > > >> >> >>>>> > > > >> > >> >> > encryption, etc.) because
> each
> > > one does
> > > >> >> not
> > > >> >> >>>> have to
> > > >> >> >>>>> > reinvent
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> wheel.
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > It also makes it much easier
> > > from a
> > > >> >> client
> > > >> >> >>>> point of
> > > >> >> >>>>> > view if
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> headers
> > > >> >> >>>>> > > > >> > >> >> are
> > > >> >> >>>>> > > > >> > >> >> > defined as part of the
> > protocol
> > > and/or
> > > >> >> >>>> message format
> > > >> >> >>>>> > in
> > > >> >> >>>>> > > > general
> > > >> >> >>>>> > > > >> > >> because
> > > >> >> >>>>> > > > >> > >> >> > you can easily produce and
> > > consume
> > > >> >> messages
> > > >> >> >>>> without
> > > >> >> >>>>> > having
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> take
> > > >> >> >>>>> > > > >> > >> into
> > > >> >> >>>>> > > > >> > >> >> > account specific cases. For
> > > example, I
> > > >> >> want
> > > >> >> >>>> to route
> > > >> >> >>>>> > > messages,
> > > >> >> >>>>> > > > >> but
> > > >> >> >>>>> > > > >> > >> >> client A
> > > >> >> >>>>> > > > >> > >> >> > doesn’t support the way
> audit
> > > >> implemented
> > > >> >> >>>> headers, and
> > > >> >> >>>>> > > client
> > > >> >> >>>>> > > > B
> > > >> >> >>>>> > > > >> > >> doesn’t
> > > >> >> >>>>> > > > >> > >> >> > support the way encryption
> or
> > > routing
> > > >> >> >>>> implemented
> > > >> >> >>>>> > headers,
> > > >> >> >>>>> > > so
> > > >> >> >>>>> > > > now
> > > >> >> >>>>> > > > >> > my
> > > >> >> >>>>> > > > >> > >> >> > application has to create
> some
> > > really
> > > >> >> >> fragile
> > > >> >> >>>> (my
> > > >> >> >>>>> > > autocorrect
> > > >> >> >>>>> > > > >> just
> > > >> >> >>>>> > > > >> > >> tried
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> > make that “tragic”, which is
> > > probably
> > > >> >> >>>> appropriate
> > > >> >> >>>>> > too) code
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> > strip
> > > >> >> >>>>> > > > >> > >> >> > everything off, rather than
> > just
> > > >> >> consuming
> > > >> >> >> the
> > > >> >> >>>>> > messages,
> > > >> >> >>>>> > > > picking
> > > >> >> >>>>> > > > >> > out
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> 1
> > > >> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested
> > > in, and
> > > >> >> >>>> performing its
> > > >> >> >>>>> > > function.
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > Honestly, this discussion
> has
> > > been
> > > >> going
> > > >> >> on
> > > >> >> >>>> for a
> > > >> >> >>>>> > long time,
> > > >> >> >>>>> > > > and
> > > >> >> >>>>> > > > >> > it’s
> > > >> >> >>>>> > > > >> > >> >> > always “Oh, you came up
> with 2
> > > use
> > > >> cases,
> > > >> >> >> and
> > > >> >> >>>> yeah,
> > > >> >> >>>>> > those
> > > >> >> >>>>> > > use
> > > >> >> >>>>> > > > >> cases
> > > >> >> >>>>> > > > >> > >> are
> > > >> >> >>>>> > > > >> > >> >> > real things that someone
> would
> > > want to
> > > >> >> do.
> > > >> >> >>>> Here’s an
> > > >> >> >>>>> > > alternate
> > > >> >> >>>>> > > > >> way
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > >> >> > implement them so let’s not
> do
> > > >> headers.”
> > > >> >> If
> > > >> >> >>>> we have a
> > > >> >> >>>>> > few
> > > >> >> >>>>> > > use
> > > >> >> >>>>> > > > >> cases
> > > >> >> >>>>> > > > >> > >> that
> > > >> >> >>>>> > > > >> > >> >> we
> > > >> >> >>>>> > > > >> > >> >> > actually came up with, you
> can
> > > be sure
> > > >> >> that
> > > >> >> >>>> over the
> > > >> >> >>>>> > next
> > > >> >> >>>>> > > year
> > > >> >> >>>>> > > > >> > >> there’s a
> > > >> >> >>>>> > > > >> > >> >> > dozen others that we didn’t
> > > think of
> > > >> that
> > > >> >> >>>> someone
> > > >> >> >>>>> > would like
> > > >> >> >>>>> > > > to
> > > >> >> >>>>> > > > >> > do. I
> > > >> >> >>>>> > > > >> > >> >> > really think it’s time to
> stop
> > > >> rehashing
> > > >> >> >> this
> > > >> >> >>>>> > discussion and
> > > >> >> >>>>> > > > >> > instead
> > > >> >> >>>>> > > > >> > >> >> focus
> > > >> >> >>>>> > > > >> > >> >> > on a workable standard that
> we
> > > can
> > > >> adopt.
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > -Todd
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39
> > PM,
> > > Todd
> > > >> >> Palino
> > > >> >> >> <
> > > >> >> >>>>> > > > tpalino@gmail.com>
> > > >> >> >>>>> > > > >> > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >
> > > >> >> >>>>> > > > >> > >> >> >> C. per message encryption
> > > >> >> >>>>> > > > >> > >> >> >>> One drawback of this
> > approach
> > > is that
> > > >> >> this
> > > >> >> >>>>> > significantly
> > > >> >> >>>>> > > > reduce
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> effectiveness of
> > compression,
> > > which
> > > >> >> >> happens
> > > >> >> >>>> on a
> > > >> >> >>>>> > set of
> > > >> >> >>>>> > > > >> > serialized
> > > >> >> >>>>> > > > >> > >> >> >>> messages. An alternative
> is
> > > to enable
> > > >> >> SSL
> > > >> >> >>>> for wire
> > > >> >> >>>>> > > > encryption
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> rely
> > > >> >> >>>>> > > > >> > >> >> on
> > > >> >> >>>>> > > > >> > >> >> >>> the storage system (e.g.
> > > LUKS) for at
> > > >> >> rest
> > > >> >> >>>>> > encryption.
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >> Jun, this is not
> sufficient.
> > > While
> > > >> this
> > > >> >> >> does
> > > >> >> >>>> cover
> > > >> >> >>>>> > the case
> > > >> >> >>>>> > > > of
> > > >> >> >>>>> > > > >> > >> removing
> > > >> >> >>>>> > > > >> > >> >> a
> > > >> >> >>>>> > > > >> > >> >> >> drive from the system, it
> > will
> > > not
> > > >> >> satisfy
> > > >> >> >>>> most
> > > >> >> >>>>> > compliance
> > > >> >> >>>>> > > > >> > >> requirements
> > > >> >> >>>>> > > > >> > >> >> for
> > > >> >> >>>>> > > > >> > >> >> >> encryption of data as
> whoever
> > > has
> > > >> >> access to
> > > >> >> >>>> the
> > > >> >> >>>>> > broker
> > > >> >> >>>>> > > itself
> > > >> >> >>>>> > > > >> > still
> > > >> >> >>>>> > > > >> > >> has
> > > >> >> >>>>> > > > >> > >> >> >> access to the unencrypted
> > > data. For
> > > >> >> >>>> end-to-end
> > > >> >> >>>>> > encryption
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> > need to
> > > >> >> >>>>> > > > >> > >> >> >> encrypt at the producer,
> > > before it
> > > >> >> enters
> > > >> >> >> the
> > > >> >> >>>>> > system, and
> > > >> >> >>>>> > > > >> decrypt
> > > >> >> >>>>> > > > >> > at
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> >> consumer, after it exits
> the
> > > system.
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >> -Todd
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03
> > > PM, radai
> > > >> <
> > > >> >> >>>>> > > > >> radai.rosenblatt@gmail.com
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > >> >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>
> > > >> >> >>>>> > > > >> > >> >> >>> another big plus of
> headers
> > > in the
> > > >> >> >> protocol
> > > >> >> >>>> is that
> > > >> >> >>>>> > it
> > > >> >> >>>>> > > would
> > > >> >> >>>>> > > > >> > enable
> > > >> >> >>>>> > > > >> > >> >> rapid
> > > >> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside
> > of
> > > core
> > > >> >> kafka
> > > >> >> >>>> and would
> > > >> >> >>>>> > reduce
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> number of
> > > >> >> >>>>> > > > >> > >> >> >>> future wire format changes
> > > required.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> a lot of what is
> currently a
> > > KIP
> > > >> >> >> represents
> > > >> >> >>>> use
> > > >> >> >>>>> > cases that
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> > not
> > > >> >> >>>>> > > > >> > >> 100%
> > > >> >> >>>>> > > > >> > >> >> >>> relevant to all users, and
> > > some of
> > > >> them
> > > >> >> >>>> require
> > > >> >> >>>>> > rather
> > > >> >> >>>>> > > > invasive
> > > >> >> >>>>> > > > >> > wire
> > > >> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing
> a
> > > good
> > > >> recent
> > > >> >> >>>> example of
> > > >> >> >>>>> > this is
> > > >> >> >>>>> > > > >> > kip-98.
> > > >> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is
> > > expected to
> > > >> be
> > > >> >> a
> > > >> >> >>>> very small
> > > >> >> >>>>> > > > fraction of
> > > >> >> >>>>> > > > >> > >> total
> > > >> >> >>>>> > > > >> > >> >> >>> traffic and yet the
> changes
> > > are
> > > >> >> invasive.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> every such wire format
> > change
> > > >> >> translates
> > > >> >> >>>> into
> > > >> >> >>>>> > painful and
> > > >> >> >>>>> > > > slow
> > > >> >> >>>>> > > > >> > >> >> adoption of
> > > >> >> >>>>> > > > >> > >> >> >>> new versions.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> i think a lot of
> > functionality
> > > >> >> currently
> > > >> >> >> in
> > > >> >> >>>> KIPs
> > > >> >> >>>>> > could be
> > > >> >> >>>>> > > > "spun
> > > >> >> >>>>> > > > >> > out"
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> implemented as opt-in
> > plugins
> > > >> >> transmitting
> > > >> >> >>>> data over
> > > >> >> >>>>> > > > headers.
> > > >> >> >>>>> > > > >> > this
> > > >> >> >>>>> > > > >> > >> >> would
> > > >> >> >>>>> > > > >> > >> >> >>> keep the core wire format
> > > stable(r),
> > > >> >> core
> > > >> >> >>>> codebase
> > > >> >> >>>>> > > smaller,
> > > >> >> >>>>> > > > and
> > > >> >> >>>>> > > > >> > >> avoid
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats
> > > sometimes
> > > >> >> required
> > > >> >> >>>> to prove
> > > >> >> >>>>> > a
> > > >> >> >>>>> > > > certain
> > > >> >> >>>>> > > > >> > >> feature
> > > >> >> >>>>> > > > >> > >> >> is
> > > >> >> >>>>> > > > >> > >> >> >>> useful enough for a
> > > wide-enough
> > > >> >> audience
> > > >> >> >> to
> > > >> >> >>>> warrant
> > > >> >> >>>>> > a wire
> > > >> >> >>>>> > > > >> format
> > > >> >> >>>>> > > > >> > >> >> change
> > > >> >> >>>>> > > > >> > >> >> >>> and code complexity
> > additions.
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
> > > beyond
> > > >> >> "mere"
> > > >> >> >>>> wire format
> > > >> >> >>>>> > > changes
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > im
> > > >> >> >>>>> > > > >> > >> not
> > > >> >> >>>>> > > > >> > >> >> >>> saying it could have been
> > > completely
> > > >> >> done
> > > >> >> >>>> with
> > > >> >> >>>>> > headers,
> > > >> >> >>>>> > > but
> > > >> >> >>>>> > > > >> > >> >> exactly-once
> > > >> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at
> 11:20
> > > AM, Gwen
> > > >> >> >>>> Shapira <
> > > >> >> >>>>> > > > >> gwen@confluent.io
> > > >> >> >>>>> > > > >> > >
> > > >> >> >>>>> > > > >> > >> >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> > > 10:24 AM,
> > > >> >> radai <
> > > >> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > > "For use cases within
> an
> > > >> >> organization,
> > > >> >> >>>> one could
> > > >> >> >>>>> > > always
> > > >> >> >>>>> > > > use
> > > >> >> >>>>> > > > >> > >> other
> > > >> >> >>>>> > > > >> > >> >> >>> > > approaches such as
> > > company-wise
> > > >> >> >>>> containers"
> > > >> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin
> > has
> > > >> >> >> traditionally
> > > >> >> >>>> done
> > > >> >> >>>>> > but there
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> > now
> > > >> >> >>>>> > > > >> > >> >> cases
> > > >> >> >>>>> > > > >> > >> >> >>> > (read
> > > >> >> >>>>> > > > >> > >> >> >>> > > - topics) where this
> is
> > > not
> > > >> >> >> acceptable.
> > > >> >> >>>> this
> > > >> >> >>>>> > makes
> > > >> >> >>>>> > > > headers
> > > >> >> >>>>> > > > >> > >> useful
> > > >> >> >>>>> > > > >> > >> >> even
> > > >> >> >>>>> > > > >> > >> >> >>> > > within single orgs for
> > > cases
> > > >> where
> > > >> >> >>>>> > > > one-container-fits-all
> > > >> >> >>>>> > > > >> > cannot
> > > >> >> >>>>> > > > >> > >> >> >>> apply.
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > > as for the particular
> > use
> > > cases
> > > >> >> >> listed,
> > > >> >> >>>> i dont
> > > >> >> >>>>> > want
> > > >> >> >>>>> > > > this to
> > > >> >> >>>>> > > > >> > >> devolve
> > > >> >> >>>>> > > > >> > >> >> >>> to a
> > > >> >> >>>>> > > > >> > >> >> >>> > > discussion of
> particular
> > > use
> > > >> cases
> > > >> >> - i
> > > >> >> >>>> think its
> > > >> >> >>>>> > > enough
> > > >> >> >>>>> > > > >> that
> > > >> >> >>>>> > > > >> > >> some
> > > >> >> >>>>> > > > >> > >> >> of
> > > >> >> >>>>> > > > >> > >> >> >>> them
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I think a main point of
> > > contention
> > > >> is
> > > >> >> >>>> that: We
> > > >> >> >>>>> > > identified
> > > >> >> >>>>> > > > few
> > > >> >> >>>>> > > > >> > >> >> >>> > use-cases where headers
> > are
> > > useful,
> > > >> >> do
> > > >> >> >> we
> > > >> >> >>>> want
> > > >> >> >>>>> > Kafka to
> > > >> >> >>>>> > > > be a
> > > >> >> >>>>> > > > >> > >> system
> > > >> >> >>>>> > > > >> > >> >> >>> > that supports those
> > > use-cases?
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
> > > >> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely
> > useful
> > > >> >> record-level
> > > >> >> >>>> lineage
> > > >> >> >>>>> > is
> > > >> >> >>>>> > > though
> > > >> >> >>>>> > > > >> > since
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > overhead could
> > > >> >> >>>>> > > > >> > >> >> >>> > be significant."
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports
> > > record level
> > > >> >> >>>> lineage. I
> > > >> >> >>>>> > don't
> > > >> >> >>>>> > > think
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > was
> > > >> >> >>>>> > > > >> > >> >> >>> > developed for lols, I
> > think
> > > it is
> > > >> >> safe
> > > >> >> >> to
> > > >> >> >>>> assume
> > > >> >> >>>>> > that
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > NSA
> > > >> >> >>>>> > > > >> > >> needed
> > > >> >> >>>>> > > > >> > >> >> >>> > that functionality. We
> > also
> > > know
> > > >> that
> > > >> >> >>>> certain
> > > >> >> >>>>> > financial
> > > >> >> >>>>> > > > >> > institutes
> > > >> >> >>>>> > > > >> > >> >> >>> > need to track tampering
> > with
> > > >> records
> > > >> >> at
> > > >> >> >> a
> > > >> >> >>>> record
> > > >> >> >>>>> > level
> > > >> >> >>>>> > > and
> > > >> >> >>>>> > > > >> > there
> > > >> >> >>>>> > > > >> > >> are
> > > >> >> >>>>> > > > >> > >> >> >>> > federal regulations that
> > > absolutely
> > > >> >> >>>> require
> > > >> >> >>>>> > this. They
> > > >> >> >>>>> > > > also
> > > >> >> >>>>> > > > >> > need
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > prove that routing apps
> > that
> > > >> >> "touches"
> > > >> >> >> the
> > > >> >> >>>>> > messages and
> > > >> >> >>>>> > > > >> either
> > > >> >> >>>>> > > > >> > >> reads
> > > >> >> >>>>> > > > >> > >> >> >>> > or updates headers
> > couldn't
> > > have
> > > >> >> >> possibly
> > > >> >> >>>>> > modified the
> > > >> >> >>>>> > > > >> payload
> > > >> >> >>>>> > > > >> > >> >> itself.
> > > >> >> >>>>> > > > >> > >> >> >>> > They use record level
> > > encryption to
> > > >> >> do
> > > >> >> >>>> that -
> > > >> >> >>>>> > apps can
> > > >> >> >>>>> > > > read
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify
> headers
> > > but
> > > >> can't
> > > >> >> >>>> touch the
> > > >> >> >>>>> > payload.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > We can totally say
> "those
> > > are
> > > >> corner
> > > >> >> >>>> cases and
> > > >> >> >>>>> > not worth
> > > >> >> >>>>> > > > >> adding
> > > >> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for",
> > they
> > > should
> > > >> >> use a
> > > >> >> >>>> different
> > > >> >> >>>>> > > pubsub
> > > >> >> >>>>> > > > >> > message
> > > >> >> >>>>> > > > >> > >> for
> > > >> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the
> > > other 1000
> > > >> >> that
> > > >> >> >>>> cater
> > > >> >> >>>>> > > > specifically
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > financial industry).
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > But this gets us into a
> > > catch 22:
> > > >> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific
> > > use-case,
> > > >> >> >>>> someone can
> > > >> >> >>>>> > always
> > > >> >> >>>>> > > say
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > isn't
> > > >> >> >>>>> > > > >> > >> >> >>> > interesting enough for
> > > Kafka. If we
> > > >> >> >>>> discuss more
> > > >> >> >>>>> > general
> > > >> >> >>>>> > > > >> > trends,
> > > >> >> >>>>> > > > >> > >> >> >>> > others can say "well, we
> > > are not
> > > >> sure
> > > >> >> >> any
> > > >> >> >>>> of them
> > > >> >> >>>>> > really
> > > >> >> >>>>> > > > >> needs
> > > >> >> >>>>> > > > >> > >> >> headers
> > > >> >> >>>>> > > > >> > >> >> >>> > specifically. This is
> just
> > > hand
> > > >> >> waving
> > > >> >> >>>> and not
> > > >> >> >>>>> > > > interesting.".
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I think discussing
> > > use-cases in
> > > >> >> >> specifics
> > > >> >> >>>> is super
> > > >> >> >>>>> > > > important
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> >> decide
> > > >> >> >>>>> > > > >> > >> >> >>> > implementation details
> for
> > > headers
> > > >> >> (my
> > > >> >> >>>> use-cases
> > > >> >> >>>>> > lean
> > > >> >> >>>>> > > > toward
> > > >> >> >>>>> > > > >> > >> >> numerical
> > > >> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and
> > > object
> > > >> >> values,
> > > >> >> >>>> others
> > > >> >> >>>>> > differ),
> > > >> >> >>>>> > > > but I
> > > >> >> >>>>> > > > >> > >> think
> > > >> >> >>>>> > > > >> > >> >> we
> > > >> >> >>>>> > > > >> > >> >> >>> > need to answer the
> general
> > > "Are we
> > > >> >> going
> > > >> >> >>>> to have
> > > >> >> >>>>> > > headers"
> > > >> >> >>>>> > > > >> > question
> > > >> >> >>>>> > > > >> > >> >> >>> > first.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from
> the
> > > other
> > > >> >> >>>> committers in the
> > > >> >> >>>>> > > > discussion:
> > > >> >> >>>>> > > > >> > >> >> >>> > What would it take to
> > > convince you
> > > >> >> that
> > > >> >> >>>> headers
> > > >> >> >>>>> > in Kafka
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> a
> > > >> >> >>>>> > > > >> > >> good
> > > >> >> >>>>> > > > >> > >> >> >>> > idea in general, so we
> can
> > > move
> > > >> ahead
> > > >> >> >> and
> > > >> >> >>>> try to
> > > >> >> >>>>> > agree
> > > >> >> >>>>> > > on
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> >> details?
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > I feel like we keep
> moving
> > > the goal
> > > >> >> >> posts
> > > >> >> >>>> and
> > > >> >> >>>>> > this is
> > > >> >> >>>>> > > > truly
> > > >> >> >>>>> > > > >> > >> >> exhausting.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly
> > > support
> > > >> >> adding
> > > >> >> >>>> headers
> > > >> >> >>>>> > to Kafka
> > > >> >> >>>>> > > > >> > (+0.5?).
> > > >> >> >>>>> > > > >> > >> >> >>> > The community can
> continue
> > > to find
> > > >> >> >>>> workarounds to
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > issue
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> there
> > > >> >> >>>>> > > > >> > >> >> >>> > are some benefits to
> > > keeping the
> > > >> >> message
> > > >> >> >>>> format
> > > >> >> >>>>> > and
> > > >> >> >>>>> > > > clients
> > > >> >> >>>>> > > > >> > >> simpler.
> > > >> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness
> > of
> > > headers
> > > >> >> to
> > > >> >> >>>> many
> > > >> >> >>>>> > use-cases
> > > >> >> >>>>> > > and
> > > >> >> >>>>> > > > if
> > > >> >> >>>>> > > > >> we
> > > >> >> >>>>> > > > >> > >> can
> > > >> >> >>>>> > > > >> > >> >> >>> > find a good and
> generally
> > > useful
> > > >> way
> > > >> >> to
> > > >> >> >>>> add it to
> > > >> >> >>>>> > Kafka,
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > will
> > > >> >> >>>>> > > > >> > >> make
> > > >> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for
> > > many -
> > > >> worthy
> > > >> >> >>>> goal in my
> > > >> >> >>>>> > eyes.
> > > >> >> >>>>> > > > >> > >> >> >>> >
> > > >> >> >>>>> > > > >> > >> >> >>> > > are
> > interesting/feasible,
> > > but:
> > > >> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are
> > > use cases
> > > >> >> for
> > > >> >> >>>> polyglot
> > > >> >> >>>>> > topics.
> > > >> >> >>>>> > > > >> > >> especially if
> > > >> >> >>>>> > > > >> > >> >> >>> kafka
> > > >> >> >>>>> > > > >> > >> >> >>> > > is being used to
> "trunk"
> > > >> something
> > > >> >> >> else.
> > > >> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics
> would
> > > make it
> > > >> >> >> harder
> > > >> >> >>>> to write
> > > >> >> >>>>> > > > portable
> > > >> >> >>>>> > > > >> > >> consumer
> > > >> >> >>>>> > > > >> > >> >> >>> code.
> > > >> >> >>>>> > > > >> > >> >> >>> > > partition remapping
> > would
> > > mess
> > > >> with
> > > >> >> >>>> locality of
> > > >> >> >>>>> > > > consumption
> > > >> >> >>>>> > > > >> > >> >> >>> guarantees.
> > > >> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see
> > for
> > > >> >> >>>> lineage/metadata is
> > > >> >> >>>>> > > > >> > >> billing/chargeback.
> > > >> >> >>>>> > > > >> > >> >> for
> > > >> >> >>>>> > > > >> > >> >> >>> > that
> > > >> >> >>>>> > > > >> > >> >> >>> > > use case it is not
> > enough
> > > to
> > > >> simply
> > > >> >> >>>> record the
> > > >> >> >>>>> > point
> > > >> >> >>>>> > > of
> > > >> >> >>>>> > > > >> > origin,
> > > >> >> >>>>> > > > >> > >> but
> > > >> >> >>>>> > > > >> > >> >> >>> every
> > > >> >> >>>>> > > > >> > >> >> >>> > > replication stop
> (think
> > > mirror
> > > >> >> maker)
> > > >> >> >>>> must also
> > > >> >> >>>>> > add a
> > > >> >> >>>>> > > > >> record
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > >> >> form a
> > > >> >> >>>>> > > > >> > >> >> >>> > > "transit log".
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > > as for stream
> processing
> > > on top
> > > >> of
> > > >> >> >>>> kafka - i
> > > >> >> >>>>> > know
> > > >> >> >>>>> > > samza
> > > >> >> >>>>> > > > >> has a
> > > >> >> >>>>> > > > >> > >> >> metadata
> > > >> >> >>>>> > > > >> > >> >> >>> > map
> > > >> >> >>>>> > > > >> > >> >> >>> > > which they carry
> around
> > in
> > > >> >> addition to
> > > >> >> >>>> user
> > > >> >> >>>>> > values.
> > > >> >> >>>>> > > > headers
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > perfect
> > > >> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016
> at
> > > 6:50 PM,
> > > >> >> Jun
> > > >> >> >>>> Rao <
> > > >> >> >>>>> > > > jun@confluent.io
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> In order to answer
> the
> > > first two
> > > >> >> >>>> questions, it
> > > >> >> >>>>> > would
> > > >> >> >>>>> > > be
> > > >> >> >>>>> > > > >> > helpful
> > > >> >> >>>>> > > > >> > >> >> if we
> > > >> >> >>>>> > > > >> > >> >> >>> > could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2
> strong
> > > use cases
> > > >> >> for
> > > >> >> >>>> headers
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > space
> > > >> >> >>>>> > > > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > third-party
> > > >> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use
> cases
> > > within an
> > > >> >> >>>> organization,
> > > >> >> >>>>> > one
> > > >> >> >>>>> > > > could
> > > >> >> >>>>> > > > >> > always
> > > >> >> >>>>> > > > >> > >> >> use
> > > >> >> >>>>> > > > >> > >> >> >>> > other
> > > >> >> >>>>> > > > >> > >> >> >>> > >> approaches such as
> > > company-wise
> > > >> >> >>>> containers to
> > > >> >> >>>>> > get
> > > >> >> >>>>> > > > around
> > > >> >> >>>>> > > > >> w/o
> > > >> >> >>>>> > > > >> > >> >> >>> headers. I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> went through the use
> > > cases in
> > > >> the
> > > >> >> KIP
> > > >> >> >>>> and in
> > > >> >> >>>>> > Radai's
> > > >> >> >>>>> > > > wiki
> > > >> >> >>>>> > > > >> (
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > https://cwiki.apache.org/confl
> > > >> >> >>>>> > uence/display/KAFKA/A+
> > > >> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> ).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> The following are the
> > > ones that
> > > >> >> that
> > > >> >> >> I
> > > >> >> >>>>> > understand and
> > > >> >> >>>>> > > > >> could
> > > >> >> >>>>> > > > >> > be
> > > >> >> >>>>> > > > >> > >> in
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> third-party use case
> > > category.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
> > > >> >> >>>>> > > > >> > >> >> >>> > >> It seems that in
> > general,
> > > >> >> >> content-type
> > > >> >> >>>> should
> > > >> >> >>>>> > be set
> > > >> >> >>>>> > > at
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> topic
> > > >> >> >>>>> > > > >> > >> >> >>> level.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing
> > > messages with
> > > >> >> >>>> different
> > > >> >> >>>>> > content
> > > >> >> >>>>> > > > types
> > > >> >> >>>>> > > > >> > >> should be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Since the value is
> > mostly
> > > >> useless
> > > >> >> >>>> without
> > > >> >> >>>>> > schema id,
> > > >> >> >>>>> > > it
> > > >> >> >>>>> > > > >> > seems
> > > >> >> >>>>> > > > >> > >> that
> > > >> >> >>>>> > > > >> > >> >> >>> > storing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the schema id
> together
> > > with
> > > >> >> >> serialized
> > > >> >> >>>> bytes
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > value
> > > >> >> >>>>> > > > >> is
> > > >> >> >>>>> > > > >> > >> >> better?
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> C. per message
> > encryption
> > > >> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this
> > > approach is
> > > >> >> that
> > > >> >> >>>> this
> > > >> >> >>>>> > > > significantly
> > > >> >> >>>>> > > > >> > reduce
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of
> > > compression,
> > > >> >> which
> > > >> >> >>>> happens on
> > > >> >> >>>>> > a set
> > > >> >> >>>>> > > of
> > > >> >> >>>>> > > > >> > >> serialized
> > > >> >> >>>>> > > > >> > >> >> >>> > >> messages. An
> > alternative
> > > is to
> > > >> >> enable
> > > >> >> >>>> SSL for
> > > >> >> >>>>> > wire
> > > >> >> >>>>> > > > >> > encryption
> > > >> >> >>>>> > > > >> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> rely
> > > >> >> >>>>> > > > >> > >> >> >>> > on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the storage system
> > (e.g.
> > > LUKS)
> > > >> >> for at
> > > >> >> >>>> rest
> > > >> >> >>>>> > > encryption.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for
> > > mirroring
> > > >> across
> > > >> >> >>>> Kafka
> > > >> >> >>>>> > clusters
> > > >> >> >>>>> > > > >> > >> >> >>> > >> This is actually
> > > interesting.
> > > >> >> Today,
> > > >> >> >>>> to avoid
> > > >> >> >>>>> > > > introducing
> > > >> >> >>>>> > > > >> > >> cycles
> > > >> >> >>>>> > > > >> > >> >> when
> > > >> >> >>>>> > > > >> > >> >> >>> > doing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data
> > > centers,
> > > >> one
> > > >> >> >>>> would
> > > >> >> >>>>> > either have
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> set
> > > >> >> >>>>> > > > >> > up
> > > >> >> >>>>> > > > >> > >> two
> > > >> >> >>>>> > > > >> > >> >> >>> Kafka
> > > >> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and
> > an
> > > >> >> aggregate)
> > > >> >> >>>> per data
> > > >> >> >>>>> > center
> > > >> >> >>>>> > > or
> > > >> >> >>>>> > > > >> > rename
> > > >> >> >>>>> > > > >> > >> >> topics.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal.
> With
> > > headers,
> > > >> >> the
> > > >> >> >>>> producer
> > > >> >> >>>>> > could
> > > >> >> >>>>> > > tag
> > > >> >> >>>>> > > > >> each
> > > >> >> >>>>> > > > >> > >> >> message
> > > >> >> >>>>> > > > >> > >> >> >>> with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster
> > ID
> > > in the
> > > >> >> >> header.
> > > >> >> >>>>> > MirrorMaker
> > > >> >> >>>>> > > > could
> > > >> >> >>>>> > > > >> > then
> > > >> >> >>>>> > > > >> > >> >> avoid
> > > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages
> to a
> > > cluster
> > > >> if
> > > >> >> >>>> they are
> > > >> >> >>>>> > tagged
> > > >> >> >>>>> > > with
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> same
> > > >> >> >>>>> > > > >> > >> >> >>> cluster
> > > >> >> >>>>> > > > >> > >> >> >>> > >> id.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> However, an
> alternative
> > > approach
> > > >> >> is
> > > >> >> >> to
> > > >> >> >>>>> > introduce sth
> > > >> >> >>>>> > > > like
> > > >> >> >>>>> > > > >> > >> >> >>> hierarchical
> > > >> >> >>>>> > > > >> > >> >> >>> > >> topic and store
> > messages
> > > from
> > > >> >> >> different
> > > >> >> >>>>> > clusters in
> > > >> >> >>>>> > > > >> > different
> > > >> >> >>>>> > > > >> > >> >> >>> partitions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> under the same topic.
> > > This
> > > >> >> approach
> > > >> >> >>>> avoids
> > > >> >> >>>>> > filtering
> > > >> >> >>>>> > > > out
> > > >> >> >>>>> > > > >> > >> unneeded
> > > >> >> >>>>> > > > >> > >> >> >>> data
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> makes offset
> preserving
> > > easier
> > > >> to
> > > >> >> >>>> support. It
> > > >> >> >>>>> > may
> > > >> >> >>>>> > > make
> > > >> >> >>>>> > > > >> > >> compaction
> > > >> >> >>>>> > > > >> > >> >> >>> > trickier
> > > >> >> >>>>> > > > >> > >> >> >>> > >> though since the same
> > > key may
> > > >> >> show up
> > > >> >> >>>> in
> > > >> >> >>>>> > different
> > > >> >> >>>>> > > > >> > partitions.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> E. record-level
> lineage
> > > >> >> >>>>> > > > >> > >> >> >>> > >> For example, a source
> > > connector
> > > >> >> could
> > > >> >> >>>> store in
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > message
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> metadata
> > > >> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the
> > source
> > > >> record.
> > > >> >> >>>> Similarly,
> > > >> >> >>>>> > if a
> > > >> >> >>>>> > > > stream
> > > >> >> >>>>> > > > >> job
> > > >> >> >>>>> > > > >> > >> >> >>> transforms
> > > >> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A
> > to
> > > topic
> > > >> B,
> > > >> >> the
> > > >> >> >>>> library
> > > >> >> >>>>> > could
> > > >> >> >>>>> > > > >> include
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> source
> > > >> >> >>>>> > > > >> > >> >> >>> > >> message offset in
> each
> > > of the
> > > >> >> >>>> transformed
> > > >> >> >>>>> > message in
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> header.
> > > >> >> >>>>> > > > >> > >> >> Not
> > > >> >> >>>>> > > > >> > >> >> >>> > sure
> > > >> >> >>>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> how widely useful
> > > record-level
> > > >> >> >> lineage
> > > >> >> >>>> is
> > > >> >> >>>>> > though
> > > >> >> >>>>> > > since
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> >> overhead
> > > >> >> >>>>> > > > >> > >> >> >>> > could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> be significant.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> > > >> >> >>>>> > > > >> > >> >> >>> > >> We could put things
> > like
> > > >> >> >>>> clientId/host/user in
> > > >> >> >>>>> > the
> > > >> >> >>>>> > > > header
> > > >> >> >>>>> > > > >> in
> > > >> >> >>>>> > > > >> > >> each
> > > >> >> >>>>> > > > >> > >> >> >>> > message
> > > >> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These
> > > metadata are
> > > >> >> >>>> really at the
> > > >> >> >>>>> > > producer
> > > >> >> >>>>> > > > >> > level
> > > >> >> >>>>> > > > >> > >> >> though.
> > > >> >> >>>>> > > > >> > >> >> >>> > So, a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is
> > to
> > > only
> > > >> >> >> include a
> > > >> >> >>>>> > "producerId"
> > > >> >> >>>>> > > > per
> > > >> >> >>>>> > > > >> > >> message
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > send
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the producerId ->
> > > metadata
> > > >> mapping
> > > >> >> >>>>> > independently.
> > > >> >> >>>>> > > > KIP-98
> > > >> >> >>>>> > > > >> is
> > > >> >> >>>>> > > > >> > >> >> actually
> > > >> >> >>>>> > > > >> > >> >> >>> > >> proposing including
> > such
> > > a
> > > >> >> producerId
> > > >> >> >>>> natively
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > >> > message.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not
> sure
> > > that I
> > > >> am
> > > >> >> >> fully
> > > >> >> >>>>> > convinced of
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > strong
> > > >> >> >>>>> > > > >> > >> >> >>> > third-party
> > > >> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers
> > yet.
> > > >> Perhaps
> > > >> >> we
> > > >> >> >>>> could
> > > >> >> >>>>> > discuss a
> > > >> >> >>>>> > > > bit
> > > >> >> >>>>> > > > >> > more
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> make
> > > >> >> >>>>> > > > >> > >> >> >>> > one
> > > >> >> >>>>> > > > >> > >> >> >>> > >> or two really
> > convincing
> > > use
> > > >> >> cases.
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal
> > > question is
> > > >> >> >>>> whether header
> > > >> >> >>>>> > should
> > > >> >> >>>>> > > > be
> > > >> >> >>>>> > > > >> > >> exposed
> > > >> >> >>>>> > > > >> > >> >> in
> > > >> >> >>>>> > > > >> > >> >> >>> > stream
> > > >> >> >>>>> > > > >> > >> >> >>> > >> processing systems
> such
> > > Kafka
> > > >> >> stream,
> > > >> >> >>>> Samza,
> > > >> >> >>>>> > and
> > > >> >> >>>>> > > Spark
> > > >> >> >>>>> > > > >> > >> streaming.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Currently, those
> > systems
> > > just
> > > >> deal
> > > >> >> >> with
> > > >> >> >>>>> > key/value
> > > >> >> >>>>> > > > pairs.
> > > >> >> >>>>> > > > >> > >> Should we
> > > >> >> >>>>> > > > >> > >> >> >>> > expose a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> third thing header
> > there
> > > too or
> > > >> >> >>>> somehow map
> > > >> >> >>>>> > header to
> > > >> >> >>>>> > > > key
> > > >> >> >>>>> > > > >> or
> > > >> >> >>>>> > > > >> > >> >> value?
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Jun
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016
> at
> > > 3:35 AM,
> > > >> >> >>>> Michael
> > > >> >> >>>>> > Pearce <
> > > >> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that
> after
> > a
> > > period
> > > >> >> of a
> > > >> >> >>>> week,
> > > >> >> >>>>> > that there
> > > >> >> >>>>> > > > is
> > > >> >> >>>>> > > > >> no
> > > >> >> >>>>> > > > >> > >> >> concerns
> > > >> >> >>>>> > > > >> > >> >> >>> now
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and
> 2
> > > and now
> > > >> we
> > > >> >> >> have
> > > >> >> >>>>> > agreement that
> > > >> >> >>>>> > > > >> > headers
> > > >> >> >>>>> > > > >> > >> are
> > > >> >> >>>>> > > > >> > >> >> >>> useful
> > > >> >> >>>>> > > > >> > >> >> >>> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
> > > such if
> > > >> put
> > > >> >> to
> > > >> >> >> a
> > > >> >> >>>> KIP
> > > >> >> >>>>> > vote, this
> > > >> >> >>>>> > > > >> > wouldn’t
> > > >> >> >>>>> > > > >> > >> be
> > > >> >> >>>>> > > > >> > >> >> a
> > > >> >> >>>>> > > > >> > >> >> >>> > reason
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > reject.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > @
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point
> 4).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose
> > of
> > > getting
> > > >> >> this
> > > >> >> >>>> KIP
> > > >> >> >>>>> > moving past
> > > >> >> >>>>> > > > >> this,
> > > >> >> >>>>> > > > >> > we
> > > >> >> >>>>> > > > >> > >> can
> > > >> >> >>>>> > > > >> > >> >> >>> state
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4
> bytes
> > > space
> > > >> that
> > > >> >> >> can
> > > >> >> >>>> will be
> > > >> >> >>>>> > > > naturally
> > > >> >> >>>>> > > > >> > >> >> interpreted
> > > >> >> >>>>> > > > >> > >> >> >>> as
> > > >> >> >>>>> > > > >> > >> >> >>> > an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if
> namespacing
> > > is later
> > > >> >> >>>> wanted you can
> > > >> >> >>>>> > > easily
> > > >> >> >>>>> > > > >> split
> > > >> >> >>>>> > > > >> > >> this
> > > >> >> >>>>> > > > >> > >> >> >>> into
> > > >> >> >>>>> > > > >> > >> >> >>> > two
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from
> > > the wire
> > > >> >> >> protocol
> > > >> >> >>>>> > > implementation
> > > >> >> >>>>> > > > >> this
> > > >> >> >>>>> > > > >> > >> makes
> > > >> >> >>>>> > > > >> > >> >> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t
> > > believe. Is
> > > >> >> this
> > > >> >> >>>>> > reasonable to
> > > >> >> >>>>> > > > all?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point
> 4
> > > therefor
> > > >> >> happy
> > > >> >> >>>> we keep
> > > >> >> >>>>> > with 32
> > > >> >> >>>>> > > > >> bits.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016,
> > 20:34, "
> > > >> >> >>>>> > ignacio.solis@gmail.com on
> > > >> >> >>>>> > > > >> behalf
> > > >> >> >>>>> > > > >> > of
> > > >> >> >>>>> > > > >> > >> >> >>> Ignacio
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
> > > >> ignacio.solis@gmail.com
> > > >> >> on
> > > >> >> >>>> behalf of
> > > >> >> >>>>> > > > >> > isolis@igso.net
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
> > > value as
> > > >> >> byte[]
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int -
> No
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String -
> > > Reluctant
> > > >> maybe
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe
> the
> > > header
> > > >> >> system
> > > >> >> >>>> should
> > > >> >> >>>>> > take a
> > > >> >> >>>>> > > > single
> > > >> >> >>>>> > > > >> > >> int. I
> > > >> >> >>>>> > > > >> > >> >> >>> think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if
> > > you want
> > > >> to
> > > >> >> >>>> interpret
> > > >> >> >>>>> > this as
> > > >> >> >>>>> > > to
> > > >> >> >>>>> > > > >> 16bit
> > > >> >> >>>>> > > > >> > >> >> numbers
> > > >> >> >>>>> > > > >> > >> >> >>> in
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > layer
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > above go right
> > > ahead. If
> > > >> >> >>>> somebody wants
> > > >> >> >>>>> > to
> > > >> >> >>>>> > > argue
> > > >> >> >>>>> > > > >> for
> > > >> >> >>>>> > > > >> > 16
> > > >> >> >>>>> > > > >> > >> >> bits
> > > >> >> >>>>> > > > >> > >> >> >>> or
> > > >> >> >>>>> > > > >> > >> >> >>> > 64
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > bits of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > header key
> space
> > I
> > > would
> > > >> >> >> listen.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the
> key
> > > space
> > > >> into
> > > >> >> >>>> sub_key_1 and
> > > >> >> >>>>> > > > sub_key_2
> > > >> >> >>>>> > > > >> > >> makes no
> > > >> >> >>>>> > > > >> > >> >> >>> > sense to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > me at
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this layer.
> Are
> > > we going
> > > >> to
> > > >> >> >>>> start
> > > >> >> >>>>> > providing
> > > >> >> >>>>> > > > APIs to
> > > >> >> >>>>> > > > >> > get
> > > >> >> >>>>> > > > >> > >> all
> > > >> >> >>>>> > > > >> > >> >> >>> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or
> > all
> > > the
> > > >> >> >>>> sub_key_2s? If
> > > >> >> >>>>> > there is
> > > >> >> >>>>> > > > no
> > > >> >> >>>>> > > > >> > >> >> >>> distinguishing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > functions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > that are
> applied
> > > to each
> > > >> one
> > > >> >> >>>> then they
> > > >> >> >>>>> > should
> > > >> >> >>>>> > > be
> > > >> >> >>>>> > > > a
> > > >> >> >>>>> > > > >> > single
> > > >> >> >>>>> > > > >> > >> >> >>> value.
> > > >> >> >>>>> > > > >> > >> >> >>> > At
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're
> > > doing is
> > > >> >> >>>> equality.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > If the above
> > layer
> > > wants
> > > >> to
> > > >> >> >>>> interpret
> > > >> >> >>>>> > this as
> > > >> >> >>>>> > > 2,
> > > >> >> >>>>> > > > 3
> > > >> >> >>>>> > > > >> or
> > > >> >> >>>>> > > > >> > >> more
> > > >> >> >>>>> > > > >> > >> >> >>> values
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > that's a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > different
> > > question. I
> > > >> >> >>>> personally think
> > > >> >> >>>>> > it's
> > > >> >> >>>>> > > all
> > > >> >> >>>>> > > > one
> > > >> >> >>>>> > > > >> > >> >> keyspace
> > > >> >> >>>>> > > > >> > >> >> >>> > that is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > getting
> assigned
> > > using
> > > >> some
> > > >> >> >>>> structure,
> > > >> >> >>>>> > but if
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> > want to
> > > >> >> >>>>> > > > >> > >> >> >>> > sub-assign
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > parts
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > of it then
> that's
> > > fine.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > The same
> > discussion
> > > >> applies
> > > >> >> to
> > > >> >> >>>> strings.
> > > >> >> >>>>> > If
> > > >> >> >>>>> > > > somebody
> > > >> >> >>>>> > > > >> > >> argued
> > > >> >> >>>>> > > > >> > >> >> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > strings,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > would we be
> > > arguing to
> > > >> >> divide
> > > >> >> >> the
> > > >> >> >>>>> > strings with
> > > >> >> >>>>> > > > dots
> > > >> >> >>>>> > > > >> > ('.')
> > > >> >> >>>>> > > > >> > >> >> as a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we want
> > them
> > > to give
> > > >> >> us
> > > >> >> >> the
> > > >> >> >>>>> > different
> > > >> >> >>>>> > > name
> > > >> >> >>>>> > > > >> > segments
> > > >> >> >>>>> > > > >> > >> >> >>> > separately?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we be
> > > performing any
> > > >> >> >>>> actions on
> > > >> >> >>>>> > this key
> > > >> >> >>>>> > > > other
> > > >> >> >>>>> > > > >> > than
> > > >> >> >>>>> > > > >> > >> >> >>> > matching?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
> > > 2016 at
> > > >> 9:30
> > > >> >> >> AM,
> > > >> >> >>>> Michael
> > > >> >> >>>>> > > Pearce <
> > > >> >> >>>>> > > > >> > >> >> >>> > >>
> Michael.Pearce@ig.com
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> > > concerns
> > > >> on
> > > >> >> 1
> > > >> >> >>>> and 2
> > > >> >> >>>>> > still?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > @all
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > To get this
> > > moving
> > > >> along a
> > > >> >> >> bit
> > > >> >> >>>> more
> > > >> >> >>>>> > I'd also
> > > >> >> >>>>> > > > like
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> ask
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> >>> get
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the below
> last
> > > points:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe
> > > we're all
> > > >> >> >> roughly
> > > >> >> >>>> happy
> > > >> >> >>>>> > with the
> > > >> >> >>>>> > > > >> header
> > > >> >> >>>>> > > > >> > >> value
> > > >> >> >>>>> > > > >> > >> >> >>> > being a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> > > consensus
> > > >> has
> > > >> >> >>>> been for an
> > > >> >> >>>>> > > > namespace
> > > >> >> >>>>> > > > >> > based
> > > >> >> >>>>> > > > >> > >> int
> > > >> >> >>>>> > > > >> > >> >> >>> > approach
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for
> > > the key.
> > > >> Any
> > > >> >> >>>> objections
> > > >> >> >>>>> > if this
> > > >> >> >>>>> > > > is
> > > >> >> >>>>> > > > >> > what
> > > >> >> >>>>> > > > >> > >> we
> > > >> >> >>>>> > > > >> > >> >> go
> > > >> >> >>>>> > > > >> > >> >> >>> > with?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have
> > if
> > > >> >> assumption
> > > >> >> >> in
> > > >> >> >>>> (4) is
> > > >> >> >>>>> > > correct,
> > > >> >> >>>>> > > > >> > >> {int,int}
> > > >> >> >>>>> > > > >> > >> >> >>> keys.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Should both
> > > int's be
> > > >> >> int16 or
> > > >> >> >>>> int32?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them
> > > being
> > > >> int16(2
> > > >> >> >>>> bytes) as
> > > >> >> >>>>> > combined
> > > >> >> >>>>> > > > is
> > > >> >> >>>>> > > > >> > space
> > > >> >> >>>>> > > > >> > >> of
> > > >> >> >>>>> > > > >> > >> >> >>> > 4bytes as
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > per
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > original and
> > > gives
> > > >> plenty
> > > >> >> of
> > > >> >> >>>>> > combinations for
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> >> >>> foreseeable,
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead
> > > small.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any
> > > benefit in
> > > >> >> >>>> another kip
> > > >> >> >>>>> > call to
> > > >> >> >>>>> > > > >> discuss
> > > >> >> >>>>> > > > >> > >> >> these at
> > > >> >> >>>>> > > > >> > >> >> >>> > all?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >> ______________________________
> > > >> >> >>>>> > __________
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > From: K
> > Burstev <
> > > >> >> >>>> k.burstev@yandex.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> > > November
> > > >> 18,
> > > >> >> >> 2016
> > > >> >> >>>>> > 7:07:07 AM
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > To:
> > > >> dev@kafka.apache.org
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> > > [DISCUSS]
> > > >> >> KIP-82
> > > >> >> >>>> - Add
> > > >> >> >>>>> > Record
> > > >> >> >>>>> > > > Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > For what it
> is
> > > worth
> > > >> also
> > > >> >> i
> > > >> >> >>>> agree. As
> > > >> >> >>>>> > a user:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
> > > Headers are
> > > >> >> >>>> worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes -
> > > Headers should
> > > >> >> be a
> > > >> >> >>>> top level
> > > >> >> >>>>> > > option
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
> > > 21:15,
> > > >> >> "Ignacio
> > > >> >> >>>> Solis" <
> > > >> >> >>>>> > > > >> isolis@igso.net
> > > >> >> >>>>> > > > >> > >:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
> > > Headers are
> > > >> >> >>>> worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
> > > Headers
> > > >> should
> > > >> >> be
> > > >> >> >> a
> > > >> >> >>>> top
> > > >> >> >>>>> > level
> > > >> >> >>>>> > > option
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov
> > > 14, 2016
> > > >> at
> > > >> >> >> 9:16
> > > >> >> >>>> AM,
> > > >> >> >>>>> > Michael
> > > >> >> >>>>> > > > Pearce
> > > >> >> >>>>> > > > >> <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > Michael.Pearce@ig.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
> > > >> >> details/examples
> > > >> >> >>>> the
> > > >> >> >>>>> > original
> > > >> >> >>>>> > > > proposal
> > > >> >> >>>>> > > > >> > for
> > > >> >> >>>>> > > > >> > >> key
> > > >> >> >>>>> > > > >> > >> >> >>> > spacing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> ,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > not
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> new
> > > mentioned as per
> > > >> >> >>>> discussion
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > > >> idea.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will
> > need
> > > to
> > > >> update
> > > >> >> >> the
> > > >> >> >>>> kip,
> > > >> >> >>>>> > when we
> > > >> >> >>>>> > > get
> > > >> >> >>>>> > > > >> > >> agreement
> > > >> >> >>>>> > > > >> > >> >> >>> this
> > > >> >> >>>>> > > > >> > >> >> >>> > is a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > better
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach
> > > (which
> > > >> seems
> > > >> >> to
> > > >> >> >>>> be the
> > > >> >> >>>>> > case if I
> > > >> >> >>>>> > > > have
> > > >> >> >>>>> > > > >> > >> >> understood
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > general
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling
> in
> > > the
> > > >> >> >>>> conversation)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the
> > > variable
> > > >> ints,
> > > >> >> at
> > > >> >> >>>> very
> > > >> >> >>>>> > early stage
> > > >> >> >>>>> > > > we
> > > >> >> >>>>> > > > >> did
> > > >> >> >>>>> > > > >> > >> think
> > > >> >> >>>>> > > > >> > >> >> >>> about
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > this. I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added
> > > complexity
> > > >> >> for
> > > >> >> >>>> the
> > > >> >> >>>>> > saving isn't
> > > >> >> >>>>> > > > >> worth
> > > >> >> >>>>> > > > >> > it.
> > > >> >> >>>>> > > > >> > >> >> I'd
> > > >> >> >>>>> > > > >> > >> >> >>> > rather
> > > >> >> >>>>> > > > >> > >> >> >>> > >> go
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want
> to
> > > reduce
> > > >> >> >>>> overheads and
> > > >> >> >>>>> > size
> > > >> >> >>>>> > > int16
> > > >> >> >>>>> > > > >> > (2bytes)
> > > >> >> >>>>> > > > >> > >> >> keys
> > > >> >> >>>>> > > > >> > >> >> >>> as
> > > >> >> >>>>> > > > >> > >> >> >>> > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the
> note
> > > of no
> > > >> >> >> headers,
> > > >> >> >>>> there
> > > >> >> >>>>> > is as
> > > >> >> >>>>> > > per
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > kip
> > > >> >> >>>>> > > > >> > >> as
> > > >> >> >>>>> > > > >> > >> >> we
> > > >> >> >>>>> > > > >> > >> >> >>> > use an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to
> > > denote if
> > > >> >> headers
> > > >> >> >>>> are
> > > >> >> >>>>> > present or
> > > >> >> >>>>> > > > not as
> > > >> >> >>>>> > > > >> > such
> > > >> >> >>>>> > > > >> > >> >> >>> > provides a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > zero
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
> > > currently
> > > >> if
> > > >> >> >>>> headers are
> > > >> >> >>>>> > not
> > > >> >> >>>>> > > used.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think
> as
> > > radai
> > > >> >> mentions
> > > >> >> >>>> would be
> > > >> >> >>>>> > good
> > > >> >> >>>>> > > > first
> > > >> >> >>>>> > > > >> > if we
> > > >> >> >>>>> > > > >> > >> >> can
> > > >> >> >>>>> > > > >> > >> >> >>> get
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > do
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now
> have
> > > general
> > > >> >> >>>> consensus that
> > > >> >> >>>>> > (1)
> > > >> >> >>>>> > > > headers
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2)
> we
> > > want it
> > > >> as
> > > >> >> a
> > > >> >> >>>> top level
> > > >> >> >>>>> > entity.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to
> > > state the
> > > >> >> >> obvious i
> > > >> >> >>>>> > believe (1)
> > > >> >> >>>>> > > > >> headers
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as
> a
> > > top level
> > > >> >> >>>> entity.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>> ______________________________
> > > >> >> >>>>> > __________
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> From:
> Roger
> > > Hoover <
> > > >> >> >>>>> > > roger.hoover@gmail.com
> > > >> >> >>>>> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent:
> > > Wednesday,
> > > >> >> November
> > > >> >> >>>> 9, 2016
> > > >> >> >>>>> > 9:10:47
> > > >> >> >>>>> > > > PM
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
> > > >> >> dev@kafka.apache.org
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject:
> > Re:
> > > >> [DISCUSS]
> > > >> >> >>>> KIP-82 - Add
> > > >> >> >>>>> > > Record
> > > >> >> >>>>> > > > >> > Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
> > > going a
> > > >> >> little
> > > >> >> >>>> in the
> > > >> >> >>>>> > weeds but
> > > >> >> >>>>> > > > >> thanks
> > > >> >> >>>>> > > > >> > >> for
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> replies
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed
> that
> > > a prefix
> > > >> >> and
> > > >> >> >>>> {int,
> > > >> >> >>>>> > int} can
> > > >> >> >>>>> > > be
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> same.
> > > >> >> >>>>> > > > >> > >> >> It
> > > >> >> >>>>> > > > >> > >> >> >>> > doesn't
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > look
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > like
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's
> what
> > > the KIP
> > > >> is
> > > >> >> >>>> saying the
> > > >> >> >>>>> > "Open"
> > > >> >> >>>>> > > > >> > section.
> > > >> >> >>>>> > > > >> > >> The
> > > >> >> >>>>> > > > >> > >> >> >>> > example
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > shows
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New
> > > Relic and
> > > >> >> 210002
> > > >> >> >>>> for App
> > > >> >> >>>>> > Dynamics
> > > >> >> >>>>> > > > >> > implying
> > > >> >> >>>>> > > > >> > >> >> that
> > > >> >> >>>>> > > > >> > >> >> >>> the
> > > >> >> >>>>> > > > >> > >> >> >>> > New
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > Relic
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > organization
> > > will
> > > >> have
> > > >> >> >>>> only a
> > > >> >> >>>>> > single
> > > >> >> >>>>> > > > header id
> > > >> >> >>>>> > > > >> > to
> > > >> >> >>>>> > > > >> > >> work
> > > >> >> >>>>> > > > >> > >> >> >>> > with. Or
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix?
> > > The main
> > > >> >> point
> > > >> >> >>>> of a
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > or
> > > >> >> >>>>> > > > >> > prefix
> > > >> >> >>>>> > > > >> > >> is
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > reduce
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
> of
> > > config
> > > >> >> >> mapping
> > > >> >> >>>> or
> > > >> >> >>>>> > > registration
> > > >> >> >>>>> > > > >> > >> depending
> > > >> >> >>>>> > > > >> > >> >> on
> > > >> >> >>>>> > > > >> > >> >> >>> how
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > namespaces/prefixes
> > > >> >> are
> > > >> >> >>>> managed.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would
> love
> > > to hear
> > > >> >> more
> > > >> >> >>>> feedback
> > > >> >> >>>>> > on the
> > > >> >> >>>>> > > > >> > >> higher-level
> > > >> >> >>>>> > > > >> > >> >> >>> > questions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > though...
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed,
> Nov
> > > 9, 2016
> > > >> at
> > > >> >> >>>> 11:38 AM,
> > > >> >> >>>>> > radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think
> > > this
> > > >> >> discussion
> > > >> >> >>>> is
> > > >> >> >>>>> > getting a
> > > >> >> >>>>> > > bit
> > > >> >> >>>>> > > > >> into
> > > >> >> >>>>> > > > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> weeds on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > technical
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > implementation
> > > >> >> details.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd
> liek
> > > to step
> > > >> >> back a
> > > >> >> >>>> minute
> > > >> >> >>>>> > and try
> > > >> >> >>>>> > > > and
> > > >> >> >>>>> > > > >> > >> establish
> > > >> >> >>>>> > > > >> > >> >> >>> > where we
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > are in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger
> > > picture:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > (re-wording
> > > >> nacho's
> > > >> >> >> last
> > > >> >> >>>>> > paragraph)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are
> we
> > > all in
> > > >> >> >>>> agreement that
> > > >> >> >>>>> > headers
> > > >> >> >>>>> > > > are
> > > >> >> >>>>> > > > >> a
> > > >> >> >>>>> > > > >> > >> >> >>> worthwhile
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > useful
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> addition
> > > to have?
> > > >> >> this
> > > >> >> >>>> was
> > > >> >> >>>>> > contested
> > > >> >> >>>>> > > > early
> > > >> >> >>>>> > > > >> on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are
> we
> > > all in
> > > >> >> >>>> agreement on
> > > >> >> >>>>> > headers
> > > >> >> >>>>> > > as
> > > >> >> >>>>> > > > top
> > > >> >> >>>>> > > > >> > >> level
> > > >> >> >>>>> > > > >> > >> >> >>> entity
> > > >> >> >>>>> > > > >> > >> >> >>> > vs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > squirreled-away in
> > > >> >> V?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if
> there
> > > are still
> > > >> >> >>>> concerns
> > > >> >> >>>>> > around
> > > >> >> >>>>> > > these
> > > >> >> >>>>> > > > #2
> > > >> >> >>>>> > > > >> > >> points
> > > >> >> >>>>> > > > >> > >> >> >>> (#jay?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and
> now
> > > back to
> > > >> our
> > > >> >> >>>> normal
> > > >> >> >>>>> > programming
> > > >> >> >>>>> > > > ...)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints
> > > are nice.
> > > >> >> >> having
> > > >> >> >>>> said
> > > >> >> >>>>> > that, its
> > > >> >> >>>>> > > > >> adding
> > > >> >> >>>>> > > > >> > >> >> >>> complexity
> > > >> >> >>>>> > > > >> > >> >> >>> > >> (see
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>> https://github.com/addthis/
> > > >> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>> main/java/com/clearspring/
> > > >> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
> > > google
> > > >> >> result)
> > > >> >> >>>> and would
> > > >> >> >>>>> > require
> > > >> >> >>>>> > > > >> anyone
> > > >> >> >>>>> > > > >> > >> >> writing
> > > >> >> >>>>> > > > >> > >> >> >>> > other
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > clients
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python?
> > > Go? Bash?
> > > >> >> ;-) )
> > > >> >> >>>> to
> > > >> >> >>>>> > > get/implement
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > >> same,
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > relatively
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little
> > > gain (int
> > > >> vs
> > > >> >> >>>> string is
> > > >> >> >>>>> > order of
> > > >> >> >>>>> > > > >> > magnitude,
> > > >> >> >>>>> > > > >> > >> >> this
> > > >> >> >>>>> > > > >> > >> >> >>> > isnt).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int
> > > namespacing vs
> > > >> >> >> {int,
> > > >> >> >>>> int}
> > > >> >> >>>>> > > namespacing
> > > >> >> >>>>> > > > >> are
> > > >> >> >>>>> > > > >> > >> >> basically
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > same
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre
> > just
> > > >> >> namespacing
> > > >> >> >>>> an int64
> > > >> >> >>>>> > and
> > > >> >> >>>>> > > > giving
> > > >> >> >>>>> > > > >> > people
> > > >> >> >>>>> > > > >> > >> >> while
> > > >> >> >>>>> > > > >> > >> >> >>> > 2^32
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > ranges
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > at a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time.
> the
> > > part i
> > > >> >> like
> > > >> >> >>>> about this
> > > >> >> >>>>> > is
> > > >> >> >>>>> > > > letting
> > > >> >> >>>>> > > > >> > >> people
> > > >> >> >>>>> > > > >> > >> >> >>> have a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> large
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers
> > > with one
> > > >> >> >>>> registration so
> > > >> >> >>>>> > they
> > > >> >> >>>>> > > > dont
> > > >> >> >>>>> > > > >> > have
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> come
> > > >> >> >>>>> > > > >> > >> >> >>> > back
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > every
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
> > > >> plugin/header
> > > >> >> >>>> they want to
> > > >> >> >>>>> > > > "reserve".
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
> > > Nov 9,
> > > >> 2016
> > > >> >> at
> > > >> >> >>>> 11:01 AM,
> > > >> >> >>>>> > Roger
> > > >> >> >>>>> > > > >> Hoover
> > > >> >> >>>>> > > > >> > <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > roger.hoover@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since
> > > some of
> > > >> the
> > > >> >> >>>> debate has
> > > >> >> >>>>> > been
> > > >> >> >>>>> > > about
> > > >> >> >>>>> > > > >> > >> overhead +
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > wondering if we
> > > >> >> have
> > > >> >> >>>>> > considered a
> > > >> >> >>>>> > > > varint
> > > >> >> >>>>> > > > >> > >> encoding
> > > >> >> >>>>> > > > >> > >> >> (
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>> https://developers.google.com/
> > > >> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > encoding#varints)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the
> > > header
> > > >> length
> > > >> >> >>>> field (int32
> > > >> >> >>>>> > in the
> > > >> >> >>>>> > > > >> > proposal)
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't
> > use
> > > >> headers,
> > > >> >> >> the
> > > >> >> >>>>> > overhead would
> > > >> >> >>>>> > > > be a
> > > >> >> >>>>> > > > >> > >> single
> > > >> >> >>>>> > > > >> > >> >> >>> byte
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id <
> > 128
> > > would
> > > >> >> also
> > > >> >> >>>> need only a
> > > >> >> >>>>> > > single
> > > >> >> >>>>> > > > >> byte?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On
> Wed,
> > > Nov 9,
> > > >> >> 2016
> > > >> >> >> at
> > > >> >> >>>> 6:43 AM,
> > > >> >> >>>>> > > radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > > radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > @magnus - and
> > > >> >> very
> > > >> >> >>>> dangerous
> > > >> >> >>>>> > (youre
> > > >> >> >>>>> > > > >> > >> essentially
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> executing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > arbitrary code
> > > >> >> off
> > > >> >> >>>> the
> > > >> >> >>>>> > internet on
> > > >> >> >>>>> > > > your
> > > >> >> >>>>> > > > >> > >> servers
> > > >> >> >>>>> > > > >> > >> >> ...
> > > >> >> >>>>> > > > >> > >> >> >>> > bad
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > idea
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > without
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > sandbox, even
> > > >> >> with)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as
> > for
> > > it
> > > >> being
> > > >> >> a
> > > >> >> >>>> purely
> > > >> >> >>>>> > > > administrative
> > > >> >> >>>>> > > > >> > task
> > > >> >> >>>>> > > > >> > >> - i
> > > >> >> >>>>> > > > >> > >> >> >>> > >> disagree.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i
> > wish
> > > it
> > > >> would,
> > > >> >> >>>> really,
> > > >> >> >>>>> > because
> > > >> >> >>>>> > > > then my
> > > >> >> >>>>> > > > >> > >> earlier
> > > >> >> >>>>> > > > >> > >> >> >>> > point on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > complexity
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> > > remapping
> > > >> >> >>>> process would
> > > >> >> >>>>> > be
> > > >> >> >>>>> > > > invalid,
> > > >> >> >>>>> > > > >> > but
> > > >> >> >>>>> > > > >> > >> at
> > > >> >> >>>>> > > > >> > >> >> >>> > linkedin,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > example,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> (the
> > > team im
> > > >> in)
> > > >> >> >> run
> > > >> >> >>>> kafka
> > > >> >> >>>>> > as a
> > > >> >> >>>>> > > > service.
> > > >> >> >>>>> > > > >> > we
> > > >> >> >>>>> > > > >> > >> dont
> > > >> >> >>>>> > > > >> > >> >> >>> > really
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > know
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > what our
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > (developing
> > > >> >> >>>> applications
> > > >> >> >>>>> > that use
> > > >> >> >>>>> > > > kafka)
> > > >> >> >>>>> > > > >> > are
> > > >> >> >>>>> > > > >> > >> up
> > > >> >> >>>>> > > > >> > >> >> to
> > > >> >> >>>>> > > > >> > >> >> >>> at
> > > >> >> >>>>> > > > >> > >> >> >>> > any
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > given
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is
> > very
> > > >> possible
> > > >> >> >>>> (given the
> > > >> >> >>>>> > > > existance of
> > > >> >> >>>>> > > > >> > >> headers
> > > >> >> >>>>> > > > >> > >> >> >>> and a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> plugin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > ecosystem) for
> > > >> >> some
> > > >> >> >>>>> > application to
> > > >> >> >>>>> > > > >> "equip"
> > > >> >> >>>>> > > > >> > >> their
> > > >> >> >>>>> > > > >> > >> >> >>> > >> producers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> consumers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> with
> > > the
> > > >> >> required
> > > >> >> >>>> plugin
> > > >> >> >>>>> > without us
> > > >> >> >>>>> > > > >> > knowing.
> > > >> >> >>>>> > > > >> > >> i
> > > >> >> >>>>> > > > >> > >> >> dont
> > > >> >> >>>>> > > > >> > >> >> >>> > mean
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > to imply
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> bad,
> > i
> > > just
> > > >> >> want to
> > > >> >> >>>> make the
> > > >> >> >>>>> > point
> > > >> >> >>>>> > > > that
> > > >> >> >>>>> > > > >> > its
> > > >> >> >>>>> > > > >> > >> not
> > > >> >> >>>>> > > > >> > >> >> as
> > > >> >> >>>>> > > > >> > >> >> >>> > simple
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> sync
> > > across a
> > > >> >> >>>> large-enough
> > > >> >> >>>>> > > > organization.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On
> > > Wed, Nov 9,
> > > >> >> 2016
> > > >> >> >>>> at 6:17
> > > >> >> >>>>> > AM,
> > > >> >> >>>>> > > > Magnus
> > > >> >> >>>>> > > > >> > >> Edenhill
> > > >> >> >>>>> > > > >> > >> >> <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > magnus@edenhill.se>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
> > > think
> > > >> there
> > > >> >> is
> > > >> >> >> a
> > > >> >> >>>> piece
> > > >> >> >>>>> > missing
> > > >> >> >>>>> > > in
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> >> Strings
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > where
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> pro-Stringers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > reason that
> > > >> by
> > > >> >> >>>> providing
> > > >> >> >>>>> > unique
> > > >> >> >>>>> > > > string
> > > >> >> >>>>> > > > >> > >> >> >>> identifiers
> > > >> >> >>>>> > > > >> > >> >> >>> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > everything
> > > >> >> will
> > > >> >> >>>> just
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > magically
> > > >> work
> > > >> >> >> for
> > > >> >> >>>> all
> > > >> >> >>>>> > parts of
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> > stream
> > > >> >> >>>>> > > > >> > >> >> >>> pipeline.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> But
> > > the
> > > >> >> strings
> > > >> >> >>>> dont mean
> > > >> >> >>>>> > > anything
> > > >> >> >>>>> > > > by
> > > >> >> >>>>> > > > >> > >> >> themselves,
> > > >> >> >>>>> > > > >> > >> >> >>> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > while we
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > probably
> > > >> >> envision
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > some
> > > auto
> > > >> >> plugin
> > > >> >> >>>> loader
> > > >> >> >>>>> > that
> > > >> >> >>>>> > > > >> downloads,
> > > >> >> >>>>> > > > >> > >> >> compiles,
> > > >> >> >>>>> > > > >> > >> >> >>> > links
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > runs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> plugins
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > on-demand
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> as
> > > soon as
> > > >> >> >> they're
> > > >> >> >>>> seen by
> > > >> >> >>>>> > a
> > > >> >> >>>>> > > > >> consumer, I
> > > >> >> >>>>> > > > >> > >> dont
> > > >> >> >>>>> > > > >> > >> >> >>> really
> > > >> >> >>>>> > > > >> > >> >> >>> > >> see
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > something
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> so
> > > dynamic
> > > >> >> (and
> > > >> >> >>>> fragile) in
> > > >> >> >>>>> > > > practice.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> In
> > > the real
> > > >> >> world
> > > >> >> >>>> an
> > > >> >> >>>>> > application
> > > >> >> >>>>> > > > will
> > > >> >> >>>>> > > > >> be
> > > >> >> >>>>> > > > >> > >> >> >>> configured
> > > >> >> >>>>> > > > >> > >> >> >>> > >> with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > a set
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> plugins
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> to
> > > either
> > > >> add
> > > >> >> >>>> (producer)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> or
> > > read
> > > >> >> >> (consumer)
> > > >> >> >>>> headers.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > This
> > > is an
> > > >> >> >>>> administrative
> > > >> >> >>>>> > task
> > > >> >> >>>>> > > > based
> > > >> >> >>>>> > > > >> on
> > > >> >> >>>>> > > > >> > >> what
> > > >> >> >>>>> > > > >> > >> >> >>> > features a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > client
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> needs/provides
> > > >> >> >> and
> > > >> >> >>>> results
> > > >> >> >>>>> > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > some
> > > sort of
> > > >> >> >>>> configuration
> > > >> >> >>>>> > to
> > > >> >> >>>>> > > > enable
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> configure
> > > >> >> >>>>> > > > >> > >> >> >>> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > desired
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> plugins.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > Since this
> > > >> >> needs
> > > >> >> >>>> to be kept
> > > >> >> >>>>> > > > somewhat
> > > >> >> >>>>> > > > >> in
> > > >> >> >>>>> > > > >> > >> sync
> > > >> >> >>>>> > > > >> > >> >> >>> across
> > > >> >> >>>>> > > > >> > >> >> >>> > an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > (there
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> is
> > > no point
> > > >> in
> > > >> >> >>>> having
> > > >> >> >>>>> > producers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> add
> > > headers
> > > >> no
> > > >> >> >>>> consumers
> > > >> >> >>>>> > will
> > > >> >> >>>>> > > read,
> > > >> >> >>>>> > > > >> and
> > > >> >> >>>>> > > > >> > >> vice
> > > >> >> >>>>> > > > >> > >> >> >>> versa),
> > > >> >> >>>>> > > > >> > >> >> >>> > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > added
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > complexity
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> of
> > > assigning
> > > >> >> an
> > > >> >> >> id
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> for
> > > each
> > > >> >> plugin
> > > >> >> >> as
> > > >> >> >>>> it is
> > > >> >> >>>>> > being
> > > >> >> >>>>> > > > >> > configured
> > > >> >> >>>>> > > > >> > >> >> should
> > > >> >> >>>>> > > > >> > >> >> >>> be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > /Magnus
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > 2016-11-09
> > > >> >> 13:06
> > > >> >> >>>> GMT+01:00
> > > >> >> >>>>> > > Michael
> > > >> >> >>>>> > > > >> > Pearce <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > > Michael.Pearce@ig.com>:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Just
> > > >> >> >>>> following/catching
> > > >> >> >>>>> > up on
> > > >> >> >>>>> > > > what
> > > >> >> >>>>> > > > >> > seems
> > > >> >> >>>>> > > > >> > >> to
> > > >> >> >>>>> > > > >> > >> >> be
> > > >> >> >>>>> > > > >> > >> >> >>> an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > active
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > @Radai
> > > >> >> sorry if
> > > >> >> >>>> it may
> > > >> >> >>>>> > seem
> > > >> >> >>>>> > > > obvious
> > > >> >> >>>>> > > > >> > but
> > > >> >> >>>>> > > > >> > >> what
> > > >> >> >>>>> > > > >> > >> >> >>> does
> > > >> >> >>>>> > > > >> > >> >> >>> > MD
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > stand
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > for?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > My
> > > take on
> > > >> >> >>>> String vs Int:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > I
> > > will
> > > >> state
> > > >> >> >>>> first I am
> > > >> >> >>>>> > pro Int
> > > >> >> >>>>> > > > (16
> > > >> >> >>>>> > > > >> or
> > > >> >> >>>>> > > > >> > >> 32).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > I
> > > do
> > > >> though
> > > >> >> >>>> playing
> > > >> >> >>>>> > devils
> > > >> >> >>>>> > > > advocate
> > > >> >> >>>>> > > > >> > see a
> > > >> >> >>>>> > > > >> > >> >> big
> > > >> >> >>>>> > > > >> > >> >> >>> plus
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > with the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> argument
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > String
> > > >> keys,
> > > >> >> >>>> this is
> > > >> >> >>>>> > around
> > > >> >> >>>>> > > > >> > integrating
> > > >> >> >>>>> > > > >> > >> >> into an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > existing
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > eco-system.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > As
> > > many
> > > >> >> other
> > > >> >> >>>> systems use
> > > >> >> >>>>> > > String
> > > >> >> >>>>> > > > >> based
> > > >> >> >>>>> > > > >> > >> >> headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > much
> > > >> easier
> > > >> >> for
> > > >> >> >>>> these to
> > > >> >> >>>>> > be
> > > >> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > into.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > How with
> > > >> Int
> > > >> >> >>>> based
> > > >> >> >>>>> > headers
> > > >> >> >>>>> > > could
> > > >> >> >>>>> > > > we
> > > >> >> >>>>> > > > >> > >> provide
> > > >> >> >>>>> > > > >> > >> >> a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> integration
> > > >> >> >>>> simple /
> > > >> >> >>>>> > easy with
> > > >> >> >>>>> > > > >> > transition
> > > >> >> >>>>> > > > >> > >> >> flows
> > > >> >> >>>>> > > > >> > >> >> >>> > over
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > *
> > > tough
> > > >> luck
> > > >> >> >>>> buddy
> > > >> >> >>>>> > you're on
> > > >> >> >>>>> > > your
> > > >> >> >>>>> > > > >> own
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > *
> > > simply
> > > >> >> hash
> > > >> >> >>>> the string
> > > >> >> >>>>> > into
> > > >> >> >>>>> > > int
> > > >> >> >>>>> > > > >> code
> > > >> >> >>>>> > > > >> > >> and
> > > >> >> >>>>> > > > >> > >> >> hope
> > > >> >> >>>>> > > > >> > >> >> >>> > for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > convert
> > > >> back
> > > >> >> >>>> though?)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > *
> > > http2
> > > >> >> style
> > > >> >> >> as
> > > >> >> >>>>> > mentioned by
> > > >> >> >>>>> > > > nacho.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > cheers,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Mike
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > ______________________________
> > > >> >> >>>>> > > > >> > __________
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > From:
> > > >> radai
> > > >> >> <
> > > >> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Sent:
> > > >> >> >> Wednesday,
> > > >> >> >>>>> > November 9,
> > > >> >> >>>>> > > 2016
> > > >> >> >>>>> > > > >> > 8:12 AM
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > To:
> > > >> >> >>>> dev@kafka.apache.org
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Subject:
> > > >> Re:
> > > >> >> >>>> [DISCUSS]
> > > >> >> >>>>> > KIP-82 -
> > > >> >> >>>>> > > > Add
> > > >> >> >>>>> > > > >> > >> Record
> > > >> >> >>>>> > > > >> > >> >> >>> Headers
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > thinking
> > > >> >> about
> > > >> >> >>>> it some
> > > >> >> >>>>> > more,
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> best
> > > >> >> >>>>> > > > >> > >> way to
> > > >> >> >>>>> > > > >> > >> >> >>> > transmit
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > remapping
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > data to
> > > >> >> >>>> consumers would
> > > >> >> >>>>> > be to
> > > >> >> >>>>> > > > put it
> > > >> >> >>>>> > > > >> > in
> > > >> >> >>>>> > > > >> > >> the
> > > >> >> >>>>> > > > >> > >> >> MD
> > > >> >> >>>>> > > > >> > >> >> >>> > >> response
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> maybe
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > it
> > > should
> > > >> be
> > > >> >> >>>> discussed
> > > >> >> >>>>> > now.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > On
> > > Wed,
> > > >> Nov
> > > >> >> 9,
> > > >> >> >>>> 2016 at
> > > >> >> >>>>> > 12:09
> > > >> >> >>>>> > > AM,
> > > >> >> >>>>> > > > >> > radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > >> >> >> radai.rosenblatt@gmail.com
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > im not
> > > >> >> >> opposed
> > > >> >> >>>> to the
> > > >> >> >>>>> > idea of
> > > >> >> >>>>> > > > >> > namespace
> > > >> >> >>>>> > > > >> > >> >> >>> mapping.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> all
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > im
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > saying
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > its
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > not part
> > > >> >> of
> > > >> >> >>>> the "mvp"
> > > >> >> >>>>> > and,
> > > >> >> >>>>> > > > since
> > > >> >> >>>>> > > > >> it
> > > >> >> >>>>> > > > >> > >> >> requires
> > > >> >> >>>>> > > > >> > >> >> >>> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> wire
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > format
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > always
> > > >> be
> > > >> >> >>>> added later.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > also,
> > > >> its
> > > >> >> not
> > > >> >> >>>> as
> > > >> >> >>>>> > simple as
> > > >> >> >>>>> > > just
> > > >> >> >>>>> > > > >> > >> >> configuring
> > > >> >> >>>>> > > > >> > >> >> >>> MM
> > > >> >> >>>>> > > > >> > >> >> >>> > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> do
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > transform:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > lets
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > say i've
> > > >> >> >>>> implemented
> > > >> >> >>>>> > large
> > > >> >> >>>>> > > > message
> > > >> >> >>>>> > > > >> > >> >> support as
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > mirror
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > target
> > > >> >> >> cluster
> > > >> >> >>>> its been
> > > >> >> >>>>> > > > remapped
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> >> {999,1}.
> > > >> >> >>>>> > > > >> > >> >> >>> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > consumer
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > would
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > also
> > > >> need
> > > >> >> to
> > > >> >> >>>> be told
> > > >> >> >>>>> > to look
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> the
> > > >> >> >>>>> > > > >> > >> large
> > > >> >> >>>>> > > > >> > >> >> >>> > message
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > "part X
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > header
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > under
> > > >> >> >> {999,1}.
> > > >> >> >>>> doable,
> > > >> >> >>>>> > but
> > > >> >> >>>>> > > > tricky.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > On Tue,
> > > >> >> Nov
> > > >> >> >> 8,
> > > >> >> >>>> 2016 at
> > > >> >> >>>>> > 10:29
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > Gwen
> > > >> >> >>>>> > > > >> > >> >> >>> Shapira <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > > gwen@confluent.io
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > While
> > > >> you
> > > >> >> >> can
> > > >> >> >>>> do
> > > >> >> >>>>> > whatever
> > > >> >> >>>>> > > you
> > > >> >> >>>>> > > > >> want
> > > >> >> >>>>> > > > >> > >> with a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > your
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > what
> > > >> I'd
> > > >> >> >>>> expect is
> > > >> >> >>>>> > for each
> > > >> >> >>>>> > > > app
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> namespaces
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> configurable...
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > So if I
> > > >> >> >>>> accidentally
> > > >> >> >>>>> > used
> > > >> >> >>>>> > > 666
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> > my
> > > >> >> >>>>> > > > >> > >> HR
> > > >> >> >>>>> > > > >> > >> >> >>> > >> department,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > still
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > run
> > > >> >> >> RadaiApp,
> > > >> >> >>>> I can
> > > >> >> >>>>> > config
> > > >> >> >>>>> > > > >> > >> "namespace=42"
> > > >> >> >>>>> > > > >> > >> >> >>> for
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > everything
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > will
> > > >> look
> > > >> >> >>>> normal.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > This
> > > >> >> means
> > > >> >> >>>> you only
> > > >> >> >>>>> > need to
> > > >> >> >>>>> > > > sync
> > > >> >> >>>>> > > > >> > usage
> > > >> >> >>>>> > > > >> > >> >> >>> inside
> > > >> >> >>>>> > > > >> > >> >> >>> > your
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > own
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > organization.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > Still
> > > >> >> hard,
> > > >> >> >>>> but
> > > >> >> >>>>> > somewhat
> > > >> >> >>>>> > > > easier
> > > >> >> >>>>> > > > >> > than
> > > >> >> >>>>> > > > >> > >> >> syncing
> > > >> >> >>>>> > > > >> > >> >> >>> > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > entire
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> world.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > On Tue,
> > > >> >> Nov
> > > >> >> >>>> 8, 2016
> > > >> >> >>>>> > at 10:07
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > >> radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> >>>> radai.rosenblatt@gmail.com>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > and
> > > >> we
> > > >> >> can
> > > >> >> >>>> start
> > > >> >> >>>>> > with
> > > >> >> >>>>> > > > >> {namespace,
> > > >> >> >>>>> > > > >> > >> id}
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> no
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > always
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > add
> > > >> it
> > > >> >> >>>> later on
> > > >> >> >>>>> > if/when
> > > >> >> >>>>> > > > >> > collisions
> > > >> >> >>>>> > > > >> > >> >> >>> actually
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > dont
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > they'd
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > a
> > > >> >> >> problem).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > every
> > > >> >> >>>> interested
> > > >> >> >>>>> > party (so
> > > >> >> >>>>> > > > orgs
> > > >> >> >>>>> > > > >> > or
> > > >> >> >>>>> > > > >> > >> >> >>> > individuals)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > could
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > then
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > register
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> prefix
> > > >> >> (0
> > > >> >> >> =
> > > >> >> >>>>> > reserved, 1 =
> > > >> >> >>>>> > > > >> > confluent
> > > >> >> >>>>> > > > >> > >> ...
> > > >> >> >>>>> > > > >> > >> >> >>> 666
> > > >> >> >>>>> > > > >> > >> >> >>> > = me
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > whatever
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > the
> > > >> >> 2nd ID
> > > >> >> >>>> - so once
> > > >> >> >>>>> > > > linkedin
> > > >> >> >>>>> > > > >> > >> >> registers,
> > > >> >> >>>>> > > > >> > >> >> >>> say
> > > >> >> >>>>> > > > >> > >> >> >>> > 3,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > then
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> are
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > free
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > to
> > > >> use
> > > >> >> {3,
> > > >> >> >>>> *} with a
> > > >> >> >>>>> > > > reasonable
> > > >> >> >>>>> > > > >> > >> >> >>> expectation
> > > >> >> >>>>> > > > >> > >> >> >>> > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > collide
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > anything
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > else.
> > > >> >> >>>> further
> > > >> >> >>>>> > partitioning
> > > >> >> >>>>> > > > of
> > > >> >> >>>>> > > > >> > that *
> > > >> >> >>>>> > > > >> > >> >> >>> becomes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> problem,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> "upstream
> > > >> >> >>>>> > registration"
> > > >> >> >>>>> > > of a
> > > >> >> >>>>> > > > >> > >> namespace
> > > >> >> >>>>> > > > >> > >> >> >>> only
> > > >> >> >>>>> > > > >> > >> >> >>> > has
> > > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > happen
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > > On
> > > >> Tue,
> > > >> >> >> Nov
> > > >> >> >>>> 8, 2016
> > > >> >> >>>>> > at
> > > >> >> >>>>> > > 9:03
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > >> James
> > > >> >> >>>>> > > > >> > >> >> >>> Cheng <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > > >> >> wushujames@gmail.com
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> > On
> > > >> >> Nov
> > > >> >> >>>> 8, 2016,
> > > >> >> >>>>> > at 5:54
> > > >> >> >>>>> > > > PM,
> > > >> >> >>>>> > > > >> > Gwen
> > > >> >> >>>>> > > > >> > >> >> >>> Shapira <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > > gwen@confluent.io
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> Thank
> > > >> >> >>>> you so
> > > >> >> >>>>> > much for
> > > >> >> >>>>> > > > this
> > > >> >> >>>>> > > > >> > clear
> > > >> >> >>>>> > > > >> > >> and
> > > >> >> >>>>> > > > >> > >> >> >>> fair
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > summary of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > arguments.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> I'm
> > > >> >> in
> > > >> >> >>>> favor of
> > > >> >> >>>>> > ints.
> > > >> >> >>>>> > > > Not a
> > > >> >> >>>>> > > > >> > >> >> >>> deal-breaker,
> > > >> >> >>>>> > > > >> > >> >> >>> > but
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> Even
> > > >> >> >>>> more in
> > > >> >> >>>>> > favor of
> > > >> >> >>>>> > > > >> Magnus's
> > > >> >> >>>>> > > > >> > >> >> >>> > decentralized
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > Roger's
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> tweak:
> > > >> >> >>>> add a
> > > >> >> >>>>> > namespace
> > > >> >> >>>>> > > > for
> > > >> >> >>>>> > > > >> > >> headers.
> > > >> >> >>>>> > > > >> > >> >> >>> This
> > > >> >> >>>>> > > > >> > >> >> >>> > will
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > allow
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > just
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> use
> > > >> >> >>>> whatever IDs
> > > >> >> >>>>> > it
> > > >> >> >>>>> > > wants
> > > >> >> >>>>> > > > >> > >> >> internally,
> > > >> >> >>>>> > > > >> > >> >> >>> and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> then
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > let
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> deploying
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> the
> > > >> >> app
> > > >> >> >>>> figure
> > > >> >> >>>>> > out an
> > > >> >> >>>>> > > > >> > available
> > > >> >> >>>>> > > > >> > >> >> >>> namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > ID
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > for the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> live
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > in.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> > So
> > > >> >> >>>>> > > > >> > io.confluent.schema-registry
> > > >> >> >>>>> > > > >> > >> can
> > > >> >> >>>>> > > > >> > >> >> be
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > namespace
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> deployment
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> and
> > > >> >> >> 0x57
> > > >> >> >>>> on
> > > >> >> >>>>> > yours, and
> > > >> >> >>>>> > > > the
> > > >> >> >>>>> > > > >> > poor
> > > >> >> >>>>> > > > >> > >> guys
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > developing the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > need
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > to
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> worry
> > > >> >> >>>> about that.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> Gwen,
> > > >> >> if
> > > >> >> >> I
> > > >> >> >>>>> > understand
> > > >> >> >>>>> > > your
> > > >> >> >>>>> > > > >> > example
> > > >> >> >>>>> > > > >> > >> >> >>> right, an
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > application
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > deployer
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > might
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> decide to
> > > >> >> >>>> use 0x01
> > > >> >> >>>>> > in one
> > > >> >> >>>>> > > > >> > >> deployment,
> > > >> >> >>>>> > > > >> > >> >> and
> > > >> >> >>>>> > > > >> > >> >> >>> > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > means
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > message
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> is
> > > >> >> >> written
> > > >> >> >>>> into the
> > > >> >> >>>>> > > > broker, it
> > > >> >> >>>>> > > > >> > >> will be
> > > >> >> >>>>> > > > >> > >> >> >>> > saved on
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > broker
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> specific
> > > >> >> >>>> namespace
> > > >> >> >>>>> > > (0x01).
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> If
> > > >> you
> > > >> >> >>>> were to
> > > >> >> >>>>> > mirror
> > > >> >> >>>>> > > that
> > > >> >> >>>>> > > > >> > message
> > > >> >> >>>>> > > > >> > >> >> into
> > > >> >> >>>>> > > > >> > >> >> >>> > another
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > would
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> accompany
> > > >> >> >>>> the
> > > >> >> >>>>> > message,
> > > >> >> >>>>> > > > right?
> > > >> >> >>>>> > > > >> > What
> > > >> >> >>>>> > > > >> > >> if
> > > >> >> >>>>> > > > >> > >> >> the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> other
> > > >> >> >>>> cluster uses
> > > >> >> >>>>> > 0x57?
> > > >> >> >>>>> > > > They
> > > >> >> >>>>> > > > >> > won't
> > > >> >> >>>>> > > > >> > >> >> >>> > understand
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > each
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > other?
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> I'm
> > > >> >> not
> > > >> >> >>>> sure
> > > >> >> >>>>> > that's an
> > > >> >> >>>>> > > > >> avoidable
> > > >> >> >>>>> > > > >> > >> >> >>> problem. I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > think it
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > simply
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > in
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> order
> > > >> >> to
> > > >> >> >>>> share
> > > >> >> >>>>> > data, you
> > > >> >> >>>>> > > > have
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> also
> > > >> >> >>>>> > > > >> > >> >> >>> have a
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > shared
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>> understanding of
> > > >> >> >>>>> > what the
> > > >> >> >>>>> > > > >> > >> namespaces
> > > >> >> >>>>> > > > >> > >> >> >>> mean.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> Which
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > I
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > sense,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> because
> > > >> >> >> the
> > > >> >> >>>>> > alternate
> > > >> >> >>>>> > > > (sharing
> > > >> >> >>>>> > > > >> > >> >> *nothing*
> > > >> >> >>>>> > > > >> > >> >> >>> at
> > > >> >> >>>>> > > > >> > >> >> >>> > >> all)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > would
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > mean
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > > there
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> would
> > > >> >> be
> > > >> >> >>>> no way to
> > > >> >> >>>>> > > > understand
> > > >> >> >>>>> > > > >> > each
> > > >> >> >>>>> > > > >> > >> >> other.
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> -James
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> Gwen
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> > On
> > > >> >> Tue,
> > > >> >> >>>> Nov 8,
> > > >> >> >>>>> > 2016 at
> > > >> >> >>>>> > > > 4:23
> > > >> >> >>>>> > > > >> > PM,
> > > >> >> >>>>> > > > >> > >> >> radai <
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > > >> >> >>>> radai.rosenblatt@gmail.com
> > > >> >> >>>>> > >
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >>
> > > >> wrote:
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >>
> > > >> +1
> > > >> >> for
> > > >> >> >>>> sean's
> > > >> >> >>>>> > > document.
> > > >> >> >>>>> > > > it
> > > >> >> >>>>> > > > >> > >> covers
> > > >> >> >>>>> > > > >> > >> >> >>> pretty
> > > >> >> >>>>> > > > >> > >> >> >>> > >> much
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > all
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > > trade-offs
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > and
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >>
> > > >> >> >> provides
> > > >> >> >>>>> > concrete
> > > >> >> >>>>> > > > figures
> > > >> >> >>>>> > > > >> to
> > > >> >> >>>>> > > > >> > >> argue
> > > >> >> >>>>> > > > >> > >> >> >>> about
> > > >> >> >>>>> > > > >> > >> >> >>> > :-)
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >>
> > > >> >> >>>> (nit-picking -
> > > >> >> >>>>> > used
> > > >> >> >>>>> > > the
> > > >> >> >>>>> > > > >> same
> > > >> >> >>>>> > > > >> > >> xkcd
> > > >> >> >>>>> > > > >> > >> >> >>> twice,
> > > >> >> >>>>> > > > >> > >> >> >>> > >> also
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > trove
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > has
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
> > > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >>
> > > >> >> superceded
> > > >> >> >>>>> > > > >> > >> >
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >> > --
> > > >> >> >>>>> > > > >> > Gwen Shapira
> > > >> >> >>>>> > > > >> > Product Manager | Confluent
> > > >> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
> > > @gwenshap
> > > >> >> >>>>> > > > >> > Follow us: Twitter | blog
> > > >> >> >>>>> > > > >> >
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> --
> > > >> >> >>>>> > > > >> *Todd Palino*
> > > >> >> >>>>> > > > >> Staff Site Reliability Engineer
> > > >> >> >>>>> > > > >> Data Infrastructure Streaming
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > > >> linkedin.com/in/toddpalino
> > > >> >> >>>>> > > > >>
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > > > --
> > > >> >> >>>>> > > > Gwen Shapira
> > > >> >> >>>>> > > > Product Manager | Confluent
> > > >> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> |
> @gwenshap
> > > >> >> >>>>> > > > Follow us: Twitter | blog
> > > >> >> >>>>> > > >
> > > >> >> >>>>> > >
> > > >> >> >>>>> >
> > > >> >> >>>>> >
> > > >> >> >>>>> > The information contained in this email is
> strictly
> > > >> >> confidential
> > > >> >> >>>> and for
> > > >> >> >>>>> > the use of the addressee only, unless otherwise
> > > indicated.
> > > >> If
> > > >> >> you
> > > >> >> >>>> are not
> > > >> >> >>>>> > the intended recipient, please do not read, copy,
> > > use or
> > > >> >> disclose
> > > >> >> >>>> to others
> > > >> >> >>>>> > this message or any attachment. Please also
> notify
> > > the
> > > >> sender
> > > >> >> by
> > > >> >> >>>> replying
> > > >> >> >>>>> > to this email or by telephone (+44(020 7896 0011)
> > > and then
> > > >> >> delete
> > > >> >> >>>> the email
> > > >> >> >>>>> > and any copies of it. Opinions, conclusion (etc)
> > > that do not
> > > >> >> >>>> relate to the
> > > >> >> >>>>> > official business of this company shall be
> > > understood as
> > > >> >> neither
> > > >> >> >>>> given nor
> > > >> >> >>>>> > endorsed by it. IG is a trading name of IG
> Markets
> > > Limited
> > > >> (a
> > > >> >> >>>> company
> > > >> >> >>>>> > registered in England and Wales, company number
> > > 04008957)
> > > >> and
> > > >> >> IG
> > > >> >> >>>> Index
> > > >> >> >>>>> > Limited (a company registered in England and
> Wales,
> > > company
> > > >> >> >> number
> > > >> >> >>>>> > 01190902). Registered address at Cannon Bridge
> > > House, 25
> > > >> >> Dowgate
> > > >> >> >>>> Hill,
> > > >> >> >>>>> > London EC4R 2YA. Both IG Markets Limited
> (register
> > > number
> > > >> >> 195355)
> > > >> >> >>>> and IG
> > > >> >> >>>>> > Index Limited (register number 114059) are
> > > authorised and
> > > >> >> >>>> regulated by the
> > > >> >> >>>>> > Financial Conduct Authority.
> > > >> >> >>>>> >
> > > >> >> >>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> The information contained in this email is strictly
> > > confidential
> > > >> and
> > > >> >> >> for
> > > >> >> >>>> the use of the addressee only, unless otherwise
> indicated.
> > > If you
> > > >> are
> > > >> >> >> not
> > > >> >> >>>> the intended recipient, please do not read, copy, use or
> > > disclose
> > > >> to
> > > >> >> >> others
> > > >> >> >>>> this message or any attachment. Please also notify the
> > > sender by
> > > >> >> >> replying
> > > >> >> >>>> to this email or by telephone (+44(020 7896 0011) and
> then
> > > delete
> > > >> the
> > > >> >> >> email
> > > >> >> >>>> and any copies of it. Opinions, conclusion (etc) that do
> > > not relate
> > > >> >> to
> > > >> >> >> the
> > > >> >> >>>> official business of this company shall be understood as
> > > neither
> > > >> >> given
> > > >> >> >> nor
> > > >> >> >>>> endorsed by it. IG is a trading name of IG Markets
> Limited
> > > (a
> > > >> company
> > > >> >> >>>> registered in England and Wales, company number
> 04008957)
> > > and IG
> > > >> >> Index
> > > >> >> >>>> Limited (a company registered in England and Wales,
> > company
> > > number
> > > >> >> >>>> 01190902). Registered address at Cannon Bridge House, 25
> > > Dowgate
> > > >> >> Hill,
> > > >> >> >>>> London EC4R 2YA. Both IG Markets Limited (register
> number
> > > 195355)
> > > >> >> and IG
> > > >> >> >>>> Index Limited (register number 114059) are authorised
> and
> > > regulated
> > > >> >> by
> > > >> >> >> the
> > > >> >> >>>> Financial Conduct Authority.
> > > >> >> >>>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>
> > > >> >> >>
> > > >> >> >>
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >
> > > >> The information contained in this email is strictly confidential
> > > and for
> > > >> the use of the addressee only, unless otherwise indicated. If
> you
> > > are not
> > > >> the intended recipient, please do not read, copy, use or
> disclose
> > > to others
> > > >> this message or any attachment. Please also notify the sender by
> > > replying
> > > >> to this email or by telephone (+44(020 7896 0011) and then
> delete
> > > the email
> > > >> and any copies of it. Opinions, conclusion (etc) that do not
> > relate
> > > to the
> > > >> official business of this company shall be understood as neither
> > > given nor
> > > >> endorsed by it. IG is a trading name of IG Markets Limited (a
> > > company
> > > >> registered in England and Wales, company number 04008957) and IG
> > > Index
> > > >> Limited (a company registered in England and Wales, company
> number
> > > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > > Hill,
> > > >> London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> > > and IG
> > > >> Index Limited (register number 114059) are authorised and
> > regulated
> > > by the
> > > >> Financial Conduct Authority.
> > > >>
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 | @gwenshap
> > > Follow us: Twitter | blog
> > >
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by radai <ra...@gmail.com>.
while HTTP-style (string, string) are the most common and most familiar,
there is a very significant impact on msg size, especially given that some
payloads are literally a few integers (think stock quotes) and would be
dwarfed by an http-like header segment.
I think we're ok with not allowing for repeatable headers because:
1. either theyre set by multiple parties that are clashing by mistake - in
this case downstream parsers will fail no matter what we do as the payloads
in those headers are presumably different
2. or theyre set by the same "component"/system/plugin in which case the
owner of the code can read-modify-write the same single header.
On Tue, Jan 10, 2017 at 10:38 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> not necessarily advocating for supporting this, just suggesting that we
> think about it. As you add these features and get closer to mapping to
> other systems, people will inevitably try to map them. Headers are an area
> where, if we're going to add them, it's worth considering compatibility as
> someone will inevitably come and complain that system X does Y with headers
> and we should also support Y because any decent system that provides
> headers will do so.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
I'm horribly behind and outdated on this thread, but wanted to mention one
other potential issue.
The KIP talks about using simple Maps for the headers. For a lot of this
discussion, I tend to either think of completely opaque headers (i.e just
another byte[] blob, which seems to have been rejected based on
interoperability grounds but I think has the nice property of making it
easy to blame the implementer of the serde for any problems with the header
implementation) or HTTP-style headers (seemingly the most ubiquitous
example of headers, though certainly there are many others, and one which
we/users may need/want to interoperate with).
Strictly speaking, the keys on HTTP headers aren't unique. See
https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2 specifically:
> Multiple message-header fields with the same field-name MAY be present in
a message if and only if the entire field-value for that header field is
defined as a comma-separated list [i.e., #(values)]. It MUST be possible to
combine the multiple header fields into one "field-name: field-value" pair,
without changing the semantics of the message, by appending each subsequent
field-value to the first, each separated by a comma. The order in which
header fields with the same field-name are received is therefore
significant to the interpretation of the combined field value, and thus a
proxy MUST NOT change the order of these field values when a message is
forwarded.
The spec guarantees that you can collapse repeated headers into a single
instance. But in practice there's a ton of stuff that doesn't. And every
robust HTTP library I've encountered uses a multimap to represent headers.
I'm not necessarily advocating for supporting this, just suggesting that we
think about it. As you add these features and get closer to mapping to
other systems, people will inevitably try to map them. Headers are an area
where, if we're going to add them, it's worth considering compatibility as
someone will inevitably come and complain that system X does Y with headers
and we should also support Y because any decent system that provides
headers will do so.
-Ewen
On Tue, Jan 10, 2017 at 5:27 PM, Jay Kreps <ja...@confluent.io> wrote:
> Hey Gwen, yes, your approach is much better than what I described and
> solves one of my objections. I didn't think of doing it in MM rather than
> in the original producer. I don't think it adds any overhead beyond the
> additional header bytes if I'm not mistaken, since MM already unpacks and
> repacks messages (unlike the broker which tries to avoid repacking and
> hence can't efficiently add a header). I think my other objection around
> size is likely solvable too by some kind of aliasing scheme? If you guys
> think this can be made to work then I think we have two pretty credible use
> cases with this and the tracing id family of uses, plus a number of things
> that are at least moderately improved. If so, I'm convinced.
>
> -Jay
>
>
> On Tue, Jan 10, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Hi Jay et al.
> >
> > On the replication using cluster id.
> >
> > Two options depending on how a company wants or needs to implement their
> > replication as there are options of doing this as a mesh, star, or ring.
> >
> > Either as some companies enforce a provider/custom producer/consumer
> > wrapper, it can be enforced the interceptors needed to be added, as you
> can
> > enforce to get support of the platform you must use company x’s internal
> > wrappers.
> > An alternative approach is to do this at the mirror maker, as Gwen noted
> > we can actually make the mirror maker append this, like wise for a more
> > complex implementation, where you can have multiple hops/routes this can
> > also be the message routing.
> >
> > Re the size of the clusterId, yes the auto generated one is “cough”
> > hideous in size.
> >
> > Most organisations though will only need a space to uniquely identify 127
> > (byte) clusters, or at worst 2 bytes (short) they need to mirror and
> > replicate between, and as such can either as platform teams generally
> will
> > roll out the mirror makers / replicators with the cluster can either
> > manually adjust the clusterId on ZK, or use a mapping file all their
> mirror
> > makers have to map the larger id to a smaller one.
> >
> > Also as Gwen states this is something you should be able to opt into or
> > not, this is the benefit of headers, if you don’t use it, it doesn’t use
> up
> > the space, its entirely optional.
> >
> > Regards
> > Mike
> >
> >
> > On 09/01/2017, 19:40, "Gwen Shapira" <gw...@confluent.io> wrote:
> >
> > Mirroring + ClusterIDs:
> >
> > Kafka clusters already have an automatically generated ID that is
> > exposed to clients:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 78%3A+Cluster+Id
> >
> > It sounds trivial to get MirrorMaker's consumer to grab this ID
> (which
> > always exists) from the source cluster and add it to the message
> > headers.
> >
> > Obviously adding stuff to the message makes it larger so this should
> > be a feature you should be able to turn on/off depending on the
> > replication requirements. If you need the feature - you pay the
> price.
> > I don't think there is an immediate need to put an effort into
> > shrinking the Cluster ID, but others can comment on that.
> >
> >
> >
> >
> >
> > On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> > > Hey Michael,
> > >
> > > Sorry for lagging, I've been out for the holiday break. What would
> > help me
> > > would be to get to a consolidated list of use cases we've worked
> out
> > and
> > > think add real value. I think getting down to the details really
> > matters
> > > for these.
> > >
> > > The two lists I'm aware of are the one in the KIP and the one you
> > gave in
> > > this email thread. Both seem to be more at the sketch level. Let me
> > review
> > > the use cases and see if I have it right. If so let's get it in the
> > design
> > > proposal.
> > >
> > > Your use cases:
> > >
> > > 1. Tracing IDs -- I get this use case and think I see how it
> > would work.
> > > I think because the tracing is from a third party system there
> is
> > some
> > > value to having a header versus a custom format to make this
> work
> > fully
> > > transparently.
> > > 2. Mirroring -- I'm a bit unsure how this will work, I'll
> explain
> > my
> > > thinking below
> > > 3. End-to-end encryption -- I think what you are saying is that
> > if you
> > > want "routing' fields that are accessible in transit without
> > decrypting you
> > > need a scheme to leave this decrypted.I think the idea is to
> avoid
> > > decrypting to get a header, but I'm not sure if I get how this
> is
> > better
> > > than a custom message format that doesn't encrypt the things you
> > don't want
> > > encrypted since you have to be aware of the scheme anyway.
> > > 4. EOS -- I'm not convinced this is a good approach for EOS.
> > >
> > > Linked from KIP:
> > >
> > > 1. Schema-Registry-aware avro serialization -- not clear how
> this
> > is
> > > better than keeping the schema id in the value
> > > 2. Polyglot kafka topics -- marginally better than a container
> > format,
> > > but of course you still have to all agree on the the header and
> > > header=>serialization mapping so not sure if it's really better
> > than a
> > > container format. Also not wild about polyglot topics :-)
> > > 3. End-to-End routing trail including mirroring --- I'd put this
> > under
> > > tracing ids
> > > 4. Cross-service process tracing -- ditto
> > > 5. Signing -- I like this use case but it seems to depend on a
> > feature
> > > we don't have and aren't proposing adding, right?
> > > 6. Per-record custom compression -- why do you want this?
> > compressing
> > > per message won't work well compared to batch compression,
> right?
> > > 7. Large Message Support -- I think we can agree that although
> > this
> > > might work it's clearly not the right way to support large
> > messages in Kafka
> > >
> > > Matthias
> > >
> > > 1. Control messages --- Matthias, not sure if I get the
> > motivation in
> > > streams for this. If we did want to do it, wouldn't it be easier
> > to
> > > implement and by allowing users to send control messages (in the
> > EOS sense)
> > > and adding a consumer config to allow either passing these on or
> > > suppressing them?
> > >
> > > So I if I were to summarize I'd list the use cases as follows:
> > >
> > > - Much better with headers: tracing IDs
> > > - Somewhat better: end-to-end encryption, polyglot topics
> > > - Don't buy: Mirroring (though perhaps could be made to work
> with
> > > changes to the proposal), EOS, large messages, compression
> > > - Not sure: Control messages
> > >
> > > Does this match where other people are at? From my point of view
> for
> > > something like this where we are changing the core data model we
> > should
> > > ideally get a handful of use cases to the point where they are
> > really well
> > > thought out and we know they can use the feature. I think we have
> > one solid
> > > one with tracing IDs, and I think maybe mirroring could be another
> > with
> > > some work. I'm less sure about the others.
> > >
> > > Bi-directional Mirroring
> > > I think bidirectional mirroring could potentially become a
> > compelling use
> > > case if we an find a way to make it work. If I understand it, the
> > proposal
> > > is to add a header like "origin-cluster" with the cluster id of the
> > Kafka
> > > cluster. Then modify the mirroring tool to disallow mirroring a
> > message
> > > back to it's own origin cluster. This basic idea is definitely
> > workable and
> > > useful, but can we really get to a good implementation using
> headers?
> > > I imagine the approach would be to add this header in the producer
> > if some
> > > config is set. There are two problems:
> > >
> > > - You add a pretty big honking cluster ID to each message. For
> > small
> > > messages this maybe larger than the contents itself.
> > > - What happens if someone somewhere forgets to set that config
> on
> > some
> > > client somewhere?
> > >
> > > If someone forgets to set the config in some client somewhere I
> > guess you
> > > have various options in mirror maker:
> > >
> > > 1. drop the data (pretty bad, now your data is out of sync)
> > > 2. crash (pretty bad--how do you recover)
> > > 3. insert some default (pretty bad since would also lead to data
> > loss if
> > > that's not right)
> > >
> > > Basically although I think a cluster tag could be a good way to do
> > > bi-directional mirroring, I'm not sure this naive approach I
> > described
> > > would get us to a usable implementation. This could be fixable. I
> > think if
> > > there was some scheme to check required headers on the server that
> > could
> > > alleviate the missing header problem. Not really sure what to do
> > about the
> > > message bloat problem, but maybe there could be some way to shrink
> > the
> > > cluster id?
> > >
> > > Cheers,
> > >
> > > -Jay
> > >
> > >
> > > On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> > Michael.Pearce@ig.com>
> > > wrote:
> > >
> > >> @Jay, Ismael and a few others.
> > >>
> > >> Based on the discussion on kip-87, and the additional use case by
> > Matthias.
> > >>
> > >> Just for clarity do we now have consensus on the earlier points
> (1)
> > >> headers are useful (2) headers should be native in Kafka?
> > >>
> > >> Cheers
> > >> Mike
> > >> ________________________________________
> > >> From: radai <ra...@gmail.com>
> > >> Sent: Monday, December 19, 2016 5:01 PM
> > >> To: dev@kafka.apache.org
> > >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> > >>
> > >> having said that im perfectly fine with something like a "no
> > payload" flag
> > >> (as opposed to null payload), which would mean such a msg is not
> > returned
> > >> from poll() unless the user code specifically "opts in" and would
> > serve
> > >> only to carry headers (making it a control message)
> > >>
> > >> On Mon, Dec 19, 2016 at 8:58 AM, radai <
> radai.rosenblatt@gmail.com>
> > wrote:
> > >>
> > >> > if all you want is to tag "end of stream" wouldnt a header on
> the
> > last
> > >> msg
> > >> > in the stream be enough?
> > >> >
> > >> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> > matthias@confluent.io>
> > >> > wrote:
> > >> >
> > >> >> The details about headers for control messages are still to
> > define. But
> > >> >> yes, the idea is to have some common default behavior that
> > clients would
> > >> >> need to implement.
> > >> >>
> > >> >> The point is, that "regular headers" add meta data to regular
> > messages.
> > >> >> Thus, those messages will be returned to the user via .poll().
> > And after
> > >> >> the message is received the user can check if meta data is
> > present and
> > >> >> read it.
> > >> >>
> > >> >> For control messages, we do not want those to pop up via
> .poll()
> > as
> > >> >> those are no regular messages. A client would need to opt-in to
> > see
> > >> >> those messages (either via poll() or maybe a callback). Thus,
> we
> > need
> > >> >> some special (standardized) header IDs that indicate control
> > messages
> > >> >> that should not be returned to the user via poll() by default.
> > >> >>
> > >> >>
> > >> >> -Matthias
> > >> >>
> > >> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
> > >> >> > Matthias,
> > >> >> >
> > >> >> > Thanks for your input. I'm +1 on control messages as they
> > seem to be
> > >> >> the
> > >> >> > simplest way to implement watermarks (
> > >> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-
> > streaming-102),
> > >> a
> > >> >> > feature that would add a lot of value to Kafka Streams IMHO.
> > >> >> >
> > >> >> > Your argument that the control-message indicator belongs in
> the
> > >> >> client-only
> > >> >> > section of the record format make sense. Just to make sure I
> > >> >> understand,
> > >> >> > are you suggesting that control messages would be indicated
> by
> > a
> > >> >> standard
> > >> >> > reserved header? By standard, I mean that ALL Kafka
> consumers
> > would
> > >> >> know
> > >> >> > to handle these messages differently (possibly just ignoring
> > them).
> > >> >> This
> > >> >> > would need to be added to the specification of the consumer
> > protocol
> > >> so
> > >> >> > that all Kafka clients implement it, right? I think it's a
> > good idea
> > >> >> but
> > >> >> > just checking.
> > >> >> >
> > >> >> > Cheers,
> > >> >> >
> > >> >> > Roger
> > >> >> >
> > >> >> >
> > >> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> > >> matthias@confluent.io
> > >> >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> >> Yes and no. I did overload the term "control message".
> > >> >> >>
> > >> >> >> EOS control messages are for client-broker communication and
> > thus
> > >> never
> > >> >> >> exposed to any application. And I think this is a good
> design
> > because
> > >> >> >> broker needs to understand those control messages. Thus,
> this
> > should
> > >> be
> > >> >> >> a protocol change.
> > >> >> >>
> > >> >> >> The type of control messages I have in mind are for
> > client-client
> > >> >> >> (application-application) communication and the broker is
> > agnostic to
> > >> >> >> them. Thus, it should not be a protocol change.
> > >> >> >>
> > >> >> >>
> > >> >> >> -Matthias
> > >> >> >>
> > >> >> >>
> > >> >> >>
> > >> >> >> On 12/14/16 9:42 AM, radai wrote:
> > >> >> >>> arent control messages getting pushed as their own top
> level
> > >> protocol
> > >> >> >>> change (and a fairly massive one) for the transactions KIP
> ?
> > >> >> >>>
> > >> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> > >> >> matthias@confluent.io>
> > >> >> >>> wrote:
> > >> >> >>>
> > >> >> >>>> Hi,
> > >> >> >>>>
> > >> >> >>>> I want to add a completely new angle to this discussion.
> > For this,
> > >> I
> > >> >> >>>> want to propose an extension for the headers feature that
> > enables
> > >> new
> > >> >> >>>> uses cases -- and those new use cases might convince
> people
> > to
> > >> >> support
> > >> >> >>>> headers (of course including the larger scoped proposal).
> > >> >> >>>>
> > >> >> >>>> Extended Proposal:
> > >> >> >>>>
> > >> >> >>>> Allow messages with a certain header key to be special
> > "control
> > >> >> >>>> messages" (w/ o w/o payload) that are not exposed to an
> > application
> > >> >> via
> > >> >> >>>> .poll().
> > >> >> >>>>
> > >> >> >>>> Thus, a consumer client would automatically skip over
> those
> > >> >> messages. If
> > >> >> >>>> an application knows about embedded control messages, it
> > can "sing
> > >> >> up"
> > >> >> >>>> to those messages by the consumer client and either get a
> > callback
> > >> or
> > >> >> >>>> the consumer auto-drop for this messages gets disabled
> > (allowing to
> > >> >> >>>> consumer those messages via poll()).
> > >> >> >>>>
> > >> >> >>>> (The details need further considerations/discussion. I
> just
> > want to
> > >> >> >>>> sketch the main idea.)
> > >> >> >>>>
> > >> >> >>>> Usage:
> > >> >> >>>>
> > >> >> >>>> There is a shared topic (ie, used by multiple
> applications)
> > and a
> > >> >> >>>> producer application wants to embed a special message in
> > the topic
> > >> >> for a
> > >> >> >>>> dedicated consumer application. Because only one
> > application will
> > >> >> >>>> understand this message, it cannot be a regular message as
> > this
> > >> would
> > >> >> >>>> break all applications that do not understand this
> message.
> > The
> > >> >> producer
> > >> >> >>>> application would set a special metadata key and no
> consumer
> > >> >> application
> > >> >> >>>> would see this control message by default because they did
> > not
> > >> enable
> > >> >> >>>> their consumer client to return this message in poll()
> (and
> > the
> > >> >> client
> > >> >> >>>> would just drop this message with special metadata key).
> > Only the
> > >> >> single
> > >> >> >>>> application that should receive this message, will
> > subscribe to
> > >> this
> > >> >> >>>> message on its consumer client and process it.
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> Concrete Use Case: Kafka Streams
> > >> >> >>>>
> > >> >> >>>> In Kafka Streams, we would like to propagate "control
> > messages"
> > >> from
> > >> >> >>>> subtopology to subtopology. There are multiple scenarios
> > for which
> > >> >> this
> > >> >> >>>> would be useful. For example, currently we do not
> guarantee
> > a
> > >> >> >>>> "consistent shutdown" of an application. By this, I mean
> > that input
> > >> >> >>>> records might not be completely processed by the whole
> > topology
> > >> >> because
> > >> >> >>>> the application shutdown happens "in between" and an
> > intermediate
> > >> >> result
> > >> >> >>>> topic gets "stock" in an intermediate topic. Thus, a user
> > would see
> > >> >> an
> > >> >> >>>> committed offset of the source topic of the application,
> > but no
> > >> >> >>>> corresponding result record in the output topic.
> > >> >> >>>>
> > >> >> >>>> Having "shutdown markers" would allow us, to first stop
> the
> > >> upstream
> > >> >> >>>> subtopology and write this marker into the intermediate
> > topic and
> > >> the
> > >> >> >>>> downstream subtopology would only shut down itself after
> is
> > sees
> > >> the
> > >> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown,
> that
> > no
> > >> >> >>>> "in-flight" messages got stuck in intermediate topics.
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> A similar usage would be for KIP-95 (Incremental Batch
> > Processing).
> > >> >> >>>> There was a discussion about the proposed metadata topic,
> > and we
> > >> >> could
> > >> >> >>>> avoid this metadata topic if we would have "control
> > messages".
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> Right now, we cannot insert an "application control
> message"
> > >> because
> > >> >> >>>> Kafka Streams does not own all topics it read/writes and
> > thus might
> > >> >> >>>> break other consumer application (as described above) if
> we
> > inject
> > >> >> >>>> random messages that are not understood by other apps.
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> Of course, one can work around "embedded control messaged"
> > by using
> > >> >> an
> > >> >> >>>> additional topic to propagate control messaged between
> > application
> > >> >> (as
> > >> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka
> > Streams). But
> > >> >> there
> > >> >> >>>> are major concerns about adding this metadata topic in the
> > KIP and
> > >> >> this
> > >> >> >>>> shows that other application that need a similar pattern
> > might
> > >> profit
> > >> >> >>>> from topic embedded "control messages", too.
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> One last important consideration: those "control messages"
> > are used
> > >> >> for
> > >> >> >>>> client to client communication and are not understood by
> the
> > >> broker.
> > >> >> >>>> Thus, those messages should not be enabled within the
> > message
> > >> format
> > >> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> > record
> > >> >> headers
> > >> >> >>>> would be a nice way to implement them. Because KIP-82 did
> > consider
> > >> >> key
> > >> >> >>>> namespaces for metatdata keys, this extension should not
> be
> > an own
> > >> >> KIP
> > >> >> >>>> but should be included in KIP-82 to reserve a namespace
> for
> > >> "control
> > >> >> >>>> message" in the first place.
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> Sorry for the long email... Looking forward to your
> > feedback.
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> -Matthias
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> > >> >> >>>>> Hi Jun
> > >> >> >>>>>
> > >> >> >>>>> 100) each time a transaction exits a jvm for a remote
> > system
> > >> >> (HTTP/JMS/
> > >> >> >>>> Hopefully one day kafka) the APM tools stich in a unique
> id
> > >> (though I
> > >> >> >>>> believe it contains the end2end uuid embedded in this id),
> > on
> > >> >> receiving
> > >> >> >> the
> > >> >> >>>> message at the receiving JVM the apm code takes this out,
> > and
> > >> >> continues
> > >> >> >> its
> > >> >> >>>> tracing on the that new thread. Both JVM’s (and other
> > languages the
> > >> >> APM
> > >> >> >>>> tool supports) send this data async back to the central
> > controllers
> > >> >> >> where
> > >> >> >>>> the stiching togeather occurs. For this they need some
> > header space
> > >> >> for
> > >> >> >>>> them to put this id.
> > >> >> >>>>>
> > >> >> >>>>> 101) Yes indeed we have a business transaction Id in the
> > payload.
> > >> >> >> Though
> > >> >> >>>> this is a system level tracing, that we need to have marry
> > up. Also
> > >> >> as
> > >> >> >> per
> > >> >> >>>> note on end2end encryption we’d be unable to prove the
> flow
> > if the
> > >> >> >> payload
> > >> >> >>>> is encrypted as we’d not have access to this at certain
> > points of
> > >> the
> > >> >> >> flow
> > >> >> >>>> through the infrastructure/platform.
> > >> >> >>>>>
> > >> >> >>>>>
> > >> >> >>>>> 103) As said we use this mechanism in IG very
> > successfully, as
> > >> >> stated
> > >> >> >>>> per key we guarantee the transaction producing app to
> > handle the
> > >> >> >>>> transaction of a key at one DC unless at point of critical
> > failure
> > >> >> >> where we
> > >> >> >>>> have to flip processing to another. We care about key
> > ordering.
> > >> >> >>>>> I disagree on the offset comment for the partition
> > solution unless
> > >> >> you
> > >> >> >>>> do full ISR, or expensive full XA transactions even with
> > partitions
> > >> >> you
> > >> >> >>>> cannot fully guarantee offsets would match.
> > >> >> >>>>>
> > >> >> >>>>> 105) Very much so, I need to have access at the platform
> > level to
> > >> >> the
> > >> >> >>>> other meta data all mentioned, without having to need to
> > have
> > >> access
> > >> >> to
> > >> >> >> the
> > >> >> >>>> encryption keys of the payload.
> > >> >> >>>>>
> > >> >> >>>>> 106)
> > >> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d need
> > to have
> > >> a
> > >> >> >>>> global producerId register which would be very hard to
> > >> >> enforce/ensure is
> > >> >> >>>> current and correct, just to understand the message
> origins
> > of its
> > >> >> >>>> region/az/cluster for routing.
> > >> >> >>>>> The client wrapper version, producerId can be the same,
> as
> > >> obviously
> > >> >> >> the
> > >> >> >>>> producer could upgrade its wrapper, as such we need to
> know
> > what
> > >> >> wrapper
> > >> >> >>>> version the message is created with.
> > >> >> >>>>> Likewise the IP address, as stated we can have our
> > producer move,
> > >> >> where
> > >> >> >>>> its IP would change.
> > >> >> >>>>>
> > >> >> >>>>> 107)
> > >> >> >>>>> UUID is set on the message by interceptors before actual
> > producer
> > >> >> >>>> transport send. This is for platform level message dedupe
> > >> guarantee,
> > >> >> the
> > >> >> >>>> business payload should be agnostic to this. Please see
> > >> >> >>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-
> > det
> > >> >> ection.html
> > >> >> >>>> note this is not touching business payloads.
> > >> >> >>>>>
> > >> >> >>>>>
> > >> >> >>>>>
> > >> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io>
> wrote:
> > >> >> >>>>>
> > >> >> >>>>> Hi, Michael,
> > >> >> >>>>>
> > >> >> >>>>> Thanks for the reply. I find it very helpful.
> > >> >> >>>>>
> > >> >> >>>>> Data lineage:
> > >> >> >>>>> 100. I'd like to understand the APM use case a bit
> > more. It
> > >> >> sounds
> > >> >> >>>> like
> > >> >> >>>>> that those APM plugins can generate a transaction id
> > that we
> > >> >> could
> > >> >> >>>>> potentially put in the header of every message. How
> > would you
> > >> >> >>>> typically
> > >> >> >>>>> make use of such transaction ids? Are there other
> > metadata
> > >> >> >>>> associated with
> > >> >> >>>>> the transaction id and if so, how are they propagated
> > >> >> downstream?
> > >> >> >>>>>
> > >> >> >>>>> 101. For the finance use case, if the concept of
> > transaction
> > >> is
> > >> >> >>>> important,
> > >> >> >>>>> wouldn't it be typically included in the message
> > payload
> > >> >> instead of
> > >> >> >>>> as an
> > >> >> >>>>> optional header field?
> > >> >> >>>>>
> > >> >> >>>>> 102. The data lineage that Altas and Navigator
> support
> > seems
> > >> to
> > >> >> be
> > >> >> >>>> at the
> > >> >> >>>>> dataset level, not per record level? So, not sure if
> > per
> > >> message
> > >> >> >>>> headers
> > >> >> >>>>> are relevant there.
> > >> >> >>>>>
> > >> >> >>>>> Mirroring:
> > >> >> >>>>> 103. The benefit of using separate partitions is that
> > it
> > >> >> >> potentially
> > >> >> >>>> makes
> > >> >> >>>>> it easy to preserve offsets during mirroring. This
> > will make
> > >> it
> > >> >> >>>> easier for
> > >> >> >>>>> consumer to switch clusters. Currently, the consumers
> > can
> > >> switch
> > >> >> >>>> clusters
> > >> >> >>>>> by using the timestampToOffset() api, but it has to
> > deal with
> > >> >> >>>> duplicates.
> > >> >> >>>>> Good point on the issue with log compact and I am not
> > sure how
> > >> >> to
> > >> >> >>>> address
> > >> >> >>>>> this. However, even if we mirror into the existing
> > partitions,
> > >> >> the
> > >> >> >>>> ordering
> > >> >> >>>>> for messages generated from different clusters seems
> > >> >> >>>> non-deterministic
> > >> >> >>>>> anyway. So, it seems that the consumers already have
> > to deal
> > >> >> with
> > >> >> >>>> that? If
> > >> >> >>>>> a topic is compacted, does that mean which messages
> are
> > >> >> preserved
> > >> >> >> is
> > >> >> >>>> also
> > >> >> >>>>> non-deterministic across clusters?
> > >> >> >>>>>
> > >> >> >>>>> 104. Good point on partition key.
> > >> >> >>>>>
> > >> >> >>>>> End-to-end encryption:
> > >> >> >>>>> 105. So, it seems end-to-end encryption is useful.
> Are
> > headers
> > >> >> >>>> useful there?
> > >> >> >>>>>
> > >> >> >>>>> Auditing:
> > >> >> >>>>> 106. It seems other than the UUID, all other metadata
> > are per
> > >> >> >>>> producer?
> > >> >> >>>>>
> > >> >> >>>>> EOS:
> > >> >> >>>>> 107. How are those UUIDs generated? I am not sure if
> > they can
> > >> be
> > >> >> >>>> generated
> > >> >> >>>>> in the producer library. An application may send
> > messages
> > >> >> through a
> > >> >> >>>> load
> > >> >> >>>>> balancer and on retry, the same message could be
> > routed to a
> > >> >> >>>> different
> > >> >> >>>>> producer instance. So, it seems that the application
> > has to
> > >> >> >> generate
> > >> >> >>>> the
> > >> >> >>>>> UUIDs. In that case, shouldn't the application just
> > put the
> > >> >> UUID in
> > >> >> >>>> the
> > >> >> >>>>> payload?
> > >> >> >>>>>
> > >> >> >>>>> Thanks,
> > >> >> >>>>>
> > >> >> >>>>> Jun
> > >> >> >>>>>
> > >> >> >>>>>
> > >> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> > >> >> >>>> Michael.Pearce@ig.com>
> > >> >> >>>>> wrote:
> > >> >> >>>>>
> > >> >> >>>>> > Hi Jun.
> > >> >> >>>>> >
> > >> >> >>>>> > Per Transaction Tracing / Data Lineage.
> > >> >> >>>>> >
> > >> >> >>>>> > As Stated in the KIP this has the first use case of
> > how many
> > >> >> APM
> > >> >> >>>> tools now
> > >> >> >>>>> > work.
> > >> >> >>>>> > I would find it impossible for any one to argue
> this
> > is not
> > >> >> >>>> important or a
> > >> >> >>>>> > niche market as it has its own gartner report for
> > this
> > >> space.
> > >> >> >> Such
> > >> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace,
> > Hawqular are
> > >> >> but a
> > >> >> >>>> few.
> > >> >> >>>>> >
> > >> >> >>>>> > Likewise these APM tools can help very rapidly
> track
> > down
> > >> >> issues
> > >> >> >>>> and
> > >> >> >>>>> > automatically capture metrics, perform actions
> based
> > on
> > >> >> >> unexpected
> > >> >> >>>> behavior
> > >> >> >>>>> > to auto recover services.
> > >> >> >>>>> >
> > >> >> >>>>> > Before mentioning looking at aggregated stats, in
> > these
> > >> cases
> > >> >> >> where
> > >> >> >>>>> > actually on critical flows we cannot afford to have
> > >> aggregated
> > >> >> >>>> rolled up
> > >> >> >>>>> > stats only.
> > >> >> >>>>> >
> > >> >> >>>>> > With the APM tool we use its actually able to
> detect
> > a
> > >> single
> > >> >> >>>> transaction
> > >> >> >>>>> > failure and capture the thread traces in the JVM
> > where it
> > >> >> failed
> > >> >> >>>> and
> > >> >> >>>>> > everything for us, to the point it sends us alerts
> > where we
> > >> >> have
> > >> >> >>>> this
> > >> >> >>>>> > giving the line number of the code that caused it,
> > the
> > >> >> >> transaction
> > >> >> >>>> trace
> > >> >> >>>>> > through all the services and endpoints (supported)
> > upto the
> > >> >> point
> > >> >> >>>> of
> > >> >> >>>>> > failure, it can also capture the data in and out
> (so
> > we can
> > >> >> >>>> replay).
> > >> >> >>>>> > Because atm Kafka doesn’t support us being able to
> > stich in
> > >> >> these
> > >> >> >>>> tracing
> > >> >> >>>>> > transaction ids natively, we cannot get these
> > benefits as
> > >> >> such is
> > >> >> >>>> limiting
> > >> >> >>>>> > our ability support apps and monitor them to the
> same
> > >> >> standards
> > >> >> >> we
> > >> >> >>>> come to
> > >> >> >>>>> > expect when on a kafka flow.
> > >> >> >>>>> >
> > >> >> >>>>> > This actually ties in with Data Lineage, as the
> same
> > tracing
> > >> >> can
> > >> >> >>>> be used
> > >> >> >>>>> > to back stich this. Essentially many times due to
> > the sums
> > >> of
> > >> >> >> money
> > >> >> >>>>> > involved there are disputes, and typically as a
> > financial
> > >> >> >>>> institute the
> > >> >> >>>>> > easiest and cleanest way to prove when disputes
> > arise is to
> > >> >> >>>> present the
> > >> >> >>>>> > actual flow and processes involved in a
> transaction.
> > >> >> >>>>> >
> > >> >> >>>>> > Likewise as Hadoop matures its evident this case is
> > >> >> important, as
> > >> >> >>>> tools
> > >> >> >>>>> > such as Atlas (Hortonworks led) and Navigator
> > (cloudera led)
> > >> >> are
> > >> >> >>>> evident
> > >> >> >>>>> > also I believe the importance here is very much NOT
> > just a
> > >> >> >>>> financial issue.
> > >> >> >>>>> >
> > >> >> >>>>> > From a MDM point of view any company wanting to
> care
> > about
> > >> >> Data
> > >> >> >>>> Quality
> > >> >> >>>>> > and Data Governance - Data Lineage is a key piece
> in
> > this
> > >> >> puzzle.
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > RE Mirroring,
> > >> >> >>>>> >
> > >> >> >>>>> > As per the KIP in-fact this is exactly what we do
> re
> > cluster
> > >> >> id,
> > >> >> >>>> to mirror
> > >> >> >>>>> > a network of clusters between AZ’s / Regions. We
> > know a
> > >> >> >>>> transaction for a
> > >> >> >>>>> > key will be done within a AZ/Region, as such we
> > know the
> > >> >> write
> > >> >> >> to
> > >> >> >>>> kafka
> > >> >> >>>>> > would be ordered per key. But we need eventual view
> > of that
> > >> >> >> across
> > >> >> >>>> in our
> > >> >> >>>>> > other regions/az’s. When we have complete AZ or
> > Region
> > >> >> failure we
> > >> >> >>>> know
> > >> >> >>>>> > there will be a brief interruption whilst those
> > transactions
> > >> >> are
> > >> >> >>>> moved to
> > >> >> >>>>> > another region but we expect after it to continue.
> > >> >> >>>>> >
> > >> >> >>>>> > As mentioned having separate Partions to do this
> > starts to
> > >> get
> > >> >> >>>>> > ugly/complicated for us:
> > >> >> >>>>> > how would I do compaction where a key is in two
> > partitions?
> > >> >> >>>>> > How do we balance consumers so where multiple
> > partitions
> > >> with
> > >> >> the
> > >> >> >>>> same key
> > >> >> >>>>> > goto the same consumer
> > >> >> >>>>> > What do you do if cluster 1 has 5 partitions but
> > cluster 20
> > >> >> has
> > >> >> >> 10
> > >> >> >>>> because
> > >> >> >>>>> > its larger kit in our more core DC’s, as such key
> to
> > >> partition
> > >> >> >>>> mappings for
> > >> >> >>>>> > consumers get even more complicated.
> > >> >> >>>>> > What do you do if we add or remove a complete
> region
> > >> >> >>>>> >
> > >> >> >>>>> > Where as simple mirror will work we just need to
> > ensure we
> > >> >> don’t
> > >> >> >>>> have a
> > >> >> >>>>> > cycle which we can do with clusterId.
> > >> >> >>>>> >
> > >> >> >>>>> > We even have started to look at shortest path
> mirror
> > routing
> > >> >> >> based
> > >> >> >>>> on
> > >> >> >>>>> > clusterId, if we also had the region and az info on
> > the
> > >> >> >> originating
> > >> >> >>>>> > message, this we have not implemented but some
> ideas
> > come
> > >> from
> > >> >> >>>> network
> > >> >> >>>>> > routing, and also the dispatcher router in apache
> > qpid.
> > >> >> >>>>> >
> > >> >> >>>>> > Also we need to have data perimeters e.g. certain
> > data
> > >> cannot
> > >> >> >> leave
> > >> >> >>>>> > certain countries borders. We want this all
> > automated so
> > >> that
> > >> >> at
> > >> >> >>>> the
> > >> >> >>>>> > platform level without having to touch or look at
> the
> > >> business
> > >> >> >>>> data inside
> > >> >> >>>>> > we can have headers we can put tags into so that we
> > can
> > >> ensure
> > >> >> >>>> this doesn’t
> > >> >> >>>>> > occur when we mirror. (actually links in to data
> > lineage /
> > >> >> >> tracing
> > >> >> >>>> as again
> > >> >> >>>>> > we need to tag messages at a platform level)
> > Examples are we
> > >> >> are
> > >> >> >>>> not
> > >> >> >>>>> > allowed Private customer details to leave
> > Switzerland, yet
> > >> we
> > >> >> >> need
> > >> >> >>>> those
> > >> >> >>>>> > systems integrated.
> > >> >> >>>>> >
> > >> >> >>>>> > Lastly around mirroring we have a partionKey field,
> > as the
> > >> key
> > >> >> >>>> used for
> > >> >> >>>>> > portioning logic != compaction key all the time but
> > we want
> > >> to
> > >> >> >>>> preserve it
> > >> >> >>>>> > for when we mirror so that if source cluster
> > partition count
> > >> >> !=
> > >> >> >>>> destination
> > >> >> >>>>> > cluster partition count we can honour the same
> > partitioning
> > >> >> >> logic.
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > RE End 2 End encryption
> > >> >> >>>>> >
> > >> >> >>>>> > As I believe mentioned just before, the solution
> you
> > mention
> > >> >> just
> > >> >> >>>> doesn’t
> > >> >> >>>>> > cut the mustard these days with many regulators. An
> > >> operations
> > >> >> >>>> person with
> > >> >> >>>>> > access to the box should not be able to have access
> > to the
> > >> >> data.
> > >> >> >>>> Many now
> > >> >> >>>>> > actually impose quite literally the implementation
> > expected
> > >> >> being
> > >> >> >>>> end2end
> > >> >> >>>>> > encryption for certain data (Singapore for us is
> one
> > that I
> > >> am
> > >> >> >>>> most aware
> > >> >> >>>>> > of). In fact we’re even now needing encrypt the
> data
> > and
> > >> store
> > >> >> >> the
> > >> >> >>>> keys in
> > >> >> >>>>> > HSM modules.
> > >> >> >>>>> >
> > >> >> >>>>> > Likewise the performance penalty on encrypting
> > decrypting as
> > >> >> you
> > >> >> >>>> produce
> > >> >> >>>>> > over wire, then again encrypt decrypt as the data
> is
> > stored
> > >> on
> > >> >> >> the
> > >> >> >>>> brokers
> > >> >> >>>>> > disks and back again, then again encrypted and
> > decrypted
> > >> back
> > >> >> >> over
> > >> >> >>>> the wire
> > >> >> >>>>> > each time for each consumer all adds up, ignoring
> > this
> > >> >> doubling
> > >> >> >>>> with mirror
> > >> >> >>>>> > makers etc. simply encrypting the value once on
> > write by the
> > >> >> >>>> client and
> > >> >> >>>>> > again decrypting on consume by the consumer is far
> > more
> > >> >> >>>> performant, but
> > >> >> >>>>> > then the routing and platform meta data needs to be
> > separate
> > >> >> >> (thus
> > >> >> >>>> headers)
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > RE Auditing:
> > >> >> >>>>> >
> > >> >> >>>>> > Our Auditing needs are:
> > >> >> >>>>> > Producer Id,
> > >> >> >>>>> > Origin Cluster Id that message first produced into
> > >> >> >>>>> > Origin AZ – agreed we can derive this if we have
> > cluster id,
> > >> >> but
> > >> >> >>>> it makes
> > >> >> >>>>> > resolving this for audit reporting a lot easier.
> > >> >> >>>>> > Origin Region – agreed we can derive this if we
> have
> > cluster
> > >> >> id,
> > >> >> >>>> but it
> > >> >> >>>>> > makes resolving this for audit reporting a lot
> > easier.
> > >> >> >>>>> > Unique Message Identification (this is not the same
> > as
> > >> >> >> transaction
> > >> >> >>>>> > tracing) – note offset and partition are not the
> > same, as
> > >> >> when we
> > >> >> >>>> mirror or
> > >> >> >>>>> > have for what ever system failure duplicate send,
> > >> >> >>>>> > Custom Client wrapper version (where organizations
> > have to
> > >> >> wrap
> > >> >> >>>> the kafka
> > >> >> >>>>> > client for added features) so we know what version
> > of the
> > >> >> wrapper
> > >> >> >>>> is used
> > >> >> >>>>> > Producer IP address (in case of clients being in
> our
> > vm/open
> > >> >> >> stack
> > >> >> >>>> infra
> > >> >> >>>>> > where they can move around, producer id will stay
> > the same
> > >> but
> > >> >> >>>> this would
> > >> >> >>>>> > change)
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > RE Once and only once delivery case
> > >> >> >>>>> >
> > >> >> >>>>> > Using the same Message UUID for auditing we can
> > achieve this
> > >> >> >> quite
> > >> >> >>>> simply.
> > >> >> >>>>> >
> > >> >> >>>>> > As per how some other brokers do this (cough qpid,
> > artemis)
> > >> >> >>>> message uuid
> > >> >> >>>>> > are used to dedupe where message is sent and
> > produced but
> > >> the
> > >> >> >>>> client didn’t
> > >> >> >>>>> > receive the ack, and there for replays the send, by
> > having a
> > >> >> >>>> unique message
> > >> >> >>>>> > id per message, this can be filtered out, on
> > consumers where
> > >> >> >>>> message
> > >> >> >>>>> > delivery may occur twice for what ever reasons a
> > message
> > >> uuid
> > >> >> can
> > >> >> >>>> be used
> > >> >> >>>>> > to remove duplicates being deliverd , like wise we
> > can do
> > >> >> this in
> > >> >> >>>> the
> > >> >> >>>>> > mirrormakers so if we detect a dupe message we can
> > avoid
> > >> >> >>>> replicating it.
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > Cheers
> > >> >> >>>>> > Mike
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
> > wrote:
> > >> >> >>>>> >
> > >> >> >>>>> > Since this KIP affects message format, wire
> > protocol,
> > >> >> apis, I
> > >> >> >>>> think
> > >> >> >>>>> > it's
> > >> >> >>>>> > worth spending a bit more time to nail down the
> > concrete
> > >> >> use
> > >> >> >>>> cases. It
> > >> >> >>>>> > would be bad if we add this feature, but when
> > start
> > >> >> >>>> implementing it
> > >> >> >>>>> > for say
> > >> >> >>>>> > mirroring, we then realize that header is not
> > the best
> > >> >> >>>> approach.
> > >> >> >>>>> > Initially,
> > >> >> >>>>> > I thought I was convinced of the use cases of
> > headers
> > >> and
> > >> >> was
> > >> >> >>>> trying to
> > >> >> >>>>> > write down a few use cases to convince others.
> > That's
> > >> >> when I
> > >> >> >>>> became
> > >> >> >>>>> > less
> > >> >> >>>>> > certain. For me to be convinced, I just want to
> > see two
> > >> >> >> strong
> > >> >> >>>> use
> > >> >> >>>>> > cases
> > >> >> >>>>> > (instead of 10 maybe use cases) in the
> > third-party
> > >> space.
> > >> >> The
> > >> >> >>>> reason is
> > >> >> >>>>> > that when we discussed the use cases within a
> > company,
> > >> >> often
> > >> >> >>>> it ends
> > >> >> >>>>> > with
> > >> >> >>>>> > "we can't force everyone to use this standard
> > since we
> > >> may
> > >> >> >>>> have to
> > >> >> >>>>> > integrate with third-party tools".
> > >> >> >>>>> >
> > >> >> >>>>> > At present, I am not sure why headers are
> useful
> > for
> > >> >> things
> > >> >> >>>> like
> > >> >> >>>>> > schemaId
> > >> >> >>>>> > or encryption. In order to do anything useful
> to
> > the
> > >> >> value,
> > >> >> >>>> one needs
> > >> >> >>>>> > to
> > >> >> >>>>> > know the schemaId or how data is encrypted, but
> > header
> > >> is
> > >> >> >>>> optional.
> > >> >> >>>>> > But, I
> > >> >> >>>>> > can be convinced if someone (Radai, Sean,
> Todd?)
> > >> provides
> > >> >> >> more
> > >> >> >>>> details
> > >> >> >>>>> > on
> > >> >> >>>>> > the argument.
> > >> >> >>>>> >
> > >> >> >>>>> > I am not very sure header is the best approach
> > for
> > >> >> mirroring
> > >> >> >>>> either. If
> > >> >> >>>>> > someone has thought about this more, I'd be
> > happy to
> > >> hear.
> > >> >> >>>>> >
> > >> >> >>>>> > I can see the data lineage use case. I am just
> > not sure
> > >> >> how
> > >> >> >>>> widely
> > >> >> >>>>> > applicable this is. If someone familiar with
> > this space
> > >> >> can
> > >> >> >>>> justify
> > >> >> >>>>> > this is
> > >> >> >>>>> > a significant use case, say in the finance
> > industry,
> > >> this
> > >> >> >>>> would be a
> > >> >> >>>>> > strong
> > >> >> >>>>> > use case.
> > >> >> >>>>> >
> > >> >> >>>>> > I can see the auditing use case. I am just not
> > sure if a
> > >> >> >> native
> > >> >> >>>>> > producer id
> > >> >> >>>>> > solves that problem. If there are additional
> > metadata
> > >> >> that's
> > >> >> >>>> worth
> > >> >> >>>>> > collecting but not covered by the producer id,
> > that
> > >> would
> > >> >> >> make
> > >> >> >>>> this a
> > >> >> >>>>> > strong use case.
> > >> >> >>>>> >
> > >> >> >>>>> > Thanks,
> > >> >> >>>>> >
> > >> >> >>>>> > Jun
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> > >> >> >>>> radai.rosenblatt@gmail.com>
> > >> >> >>>>> > wrote:
> > >> >> >>>>> >
> > >> >> >>>>> > > this KIP is about enabling headers, nothing
> > more
> > >> nothing
> > >> >> >>>> less - so
> > >> >> >>>>> > no,
> > >> >> >>>>> > > broker-side use of headers is not in the KIP
> > scope.
> > >> >> >>>>> > >
> > >> >> >>>>> > > obviously though, once you have headers
> > potential use
> > >> >> cases
> > >> >> >>>> could
> > >> >> >>>>> > include
> > >> >> >>>>> > > broker-side header-aware interceptors (which
> > would be
> > >> >> the
> > >> >> >>>> topic of
> > >> >> >>>>> > other
> > >> >> >>>>> > > future KIPs).
> > >> >> >>>>> > >
> > >> >> >>>>> > > a trivially clear use case (to me) would be
> > using such
> > >> >> >>>> broker-side
> > >> >> >>>>> > > interceptors to enforce compliance with
> > organizational
> > >> >> >>>> policies - it
> > >> >> >>>>> > would
> > >> >> >>>>> > > make our SREs lives much easier if instead of
> > >> >> retroactively
> > >> >> >>>>> > discovering
> > >> >> >>>>> > > "rogue" topics/users those messages would
> have
> > been
> > >> >> >> rejected
> > >> >> >>>>> > up-front.
> > >> >> >>>>> > >
> > >> >> >>>>> > > the kafka broker code is lacking any such
> > >> extensibility
> > >> >> >>>> support
> > >> >> >>>>> > (beyond
> > >> >> >>>>> > > maybe authorizer) which is why these use
> cases
> > were
> > >> left
> > >> >> >> out
> > >> >> >>>> of the
> > >> >> >>>>> > "case
> > >> >> >>>>> > > for headers" doc - broker extensibility is a
> > separate
> > >> >> >>>> discussion.
> > >> >> >>>>> > >
> > >> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen
> Shapira <
> > >> >> >>>> gwen@confluent.io>
> > >> >> >>>>> > wrote:
> > >> >> >>>>> > >
> > >> >> >>>>> > > > Woah, I wasn't aware this is something
> we'll
> > do. It
> > >> >> >> wasn't
> > >> >> >>>> in the
> > >> >> >>>>> > KIP,
> > >> >> >>>>> > > > right?
> > >> >> >>>>> > > >
> > >> >> >>>>> > > > I guess we could do it the same way ACLs
> > currently
> > >> >> work.
> > >> >> >>>>> > > > I had in mind something that will allow
> > admins to
> > >> >> apply
> > >> >> >>>> rules to
> > >> >> >>>>> > the
> > >> >> >>>>> > > > new create/delete/config topic APIs. So
> Todd
> > can
> > >> >> decide
> > >> >> >> to
> > >> >> >>>> reject
> > >> >> >>>>> > > > "create topic" requests that ask for more
> > than 40
> > >> >> >>>> partitions, or
> > >> >> >>>>> > > > require exactly 3 replicas, or no more than
> > 50GB
> > >> >> >> partition
> > >> >> >>>> size,
> > >> >> >>>>> > etc.
> > >> >> >>>>> > > >
> > >> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are
> > planning to
> > >> >> apply
> > >> >> >>>> more
> > >> >> >>>>> > rules
> > >> >> >>>>> > > > to requests (and I think we should), we may
> > want a
> > >> bit
> > >> >> >>>> more generic
> > >> >> >>>>> > > > design around that.
> > >> >> >>>>> > > >
> > >> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> > >> >> >>>> radai.rosenblatt@gmail.com>
> > >> >> >>>>> > > wrote:
> > >> >> >>>>> > > > > "wouldn't you be in the business of
> making
> > sure
> > >> >> >> everyone
> > >> >> >>>> uses
> > >> >> >>>>> > them
> > >> >> >>>>> > > > > properly?"
> > >> >> >>>>> > > > >
> > >> >> >>>>> > > > > thats where a broker-side plugin would
> > come handy
> > >> -
> > >> >> any
> > >> >> >>>> incoming
> > >> >> >>>>> > > message
> > >> >> >>>>> > > > > that does not conform to org policy (read
> > - does
> > >> not
> > >> >> >>>> have the
> > >> >> >>>>> > proper
> > >> >> >>>>> > > > > headers) gets thrown out (with an error
> > returned
> > >> to
> > >> >> >> user)
> > >> >> >>>>> > > > >
> > >> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> > Palino <
> > >> >> >>>> tpalino@gmail.com>
> > >> >> >>>>> > wrote:
> > >> >> >>>>> > > > >
> > >> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on
> > this one
> > >> :)
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >> Producing counts to a topic is part of
> > it, but
> > >> >> that’s
> > >> >> >>>> only
> > >> >> >>>>> > part. So
> > >> >> >>>>> > > you
> > >> >> >>>>> > > > >> count you have 100 messages in topic A.
> > When you
> > >> >> >> mirror
> > >> >> >>>> topic A
> > >> >> >>>>> > to
> > >> >> >>>>> > > > another
> > >> >> >>>>> > > > >> cluster, you have 99 messages. Where was
> > your
> > >> >> problem?
> > >> >> >>>> Or
> > >> >> >>>>> > worse, you
> > >> >> >>>>> > > > have
> > >> >> >>>>> > > > >> 100 messages, but one producer
> duplicated
> > >> messages
> > >> >> and
> > >> >> >>>> another
> > >> >> >>>>> > one
> > >> >> >>>>> > > lost
> > >> >> >>>>> > > > >> messages. You need details about where
> the
> > >> message
> > >> >> >> came
> > >> >> >>>> from in
> > >> >> >>>>> > order
> > >> >> >>>>> > > to
> > >> >> >>>>> > > > >> pinpoint problems when they happen.
> Source
> > >> producer
> > >> >> >>>> info, where
> > >> >> >>>>> > it was
> > >> >> >>>>> > > > >> produced into your infrastructure, and
> > when it
> > >> was
> > >> >> >>>> produced.
> > >> >> >>>>> > This
> > >> >> >>>>> > > > requires
> > >> >> >>>>> > > > >> you to add the information to the
> message.
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >> And yes, you still need to maintain your
> > clients.
> > >> >> So
> > >> >> >>>> maybe my
> > >> >> >>>>> > original
> > >> >> >>>>> > > > >> example was not the best. My thoughts on
> > not
> > >> >> wanting
> > >> >> >> to
> > >> >> >>>> be
> > >> >> >>>>> > responsible
> > >> >> >>>>> > > > for
> > >> >> >>>>> > > > >> message formats stands, because that’s
> > very much
> > >> >> >>>> separate from
> > >> >> >>>>> > the
> > >> >> >>>>> > > > client.
> > >> >> >>>>> > > > >> As you know, we have our own internal
> > client
> > >> >> library
> > >> >> >>>> that can
> > >> >> >>>>> > insert
> > >> >> >>>>> > > the
> > >> >> >>>>> > > > >> right headers, and right now inserts the
> > right
> > >> >> audit
> > >> >> >>>>> > information into
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> message fields. If they exist, and
> > assuming the
> > >> >> >> message
> > >> >> >>>> is Avro
> > >> >> >>>>> > > encoded.
> > >> >> >>>>> > > > >> What if someone wants to use JSON
> instead
> > for a
> > >> >> good
> > >> >> >>>> reason?
> > >> >> >>>>> > What if
> > >> >> >>>>> > > > user X
> > >> >> >>>>> > > > >> wants to encrypt messages, but user Y
> > does not?
> > >> >> >>>> Maintaining the
> > >> >> >>>>> > client
> > >> >> >>>>> > > > >> library is still much easier than
> > maintaining the
> > >> >> >>>> message
> > >> >> >>>>> > formats.
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >> -Todd
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> > Shapira <
> > >> >> >>>> gwen@confluent.io
> > >> >> >>>>> > >
> > >> >> >>>>> > > wrote:
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >> > Based on your last sentence, consider
> me
> > >> >> convinced
> > >> >> >> :)
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >> > I get why headers are critical for
> > Mirroring
> > >> (you
> > >> >> >>>> need tags to
> > >> >> >>>>> > > prevent
> > >> >> >>>>> > > > >> > loops and sometimes to route messages
> > to the
> > >> >> correct
> > >> >> >>>>> > destination).
> > >> >> >>>>> > > > >> > But why do you need headers to audit?
> > We are
> > >> >> >> auditing
> > >> >> >>>> by
> > >> >> >>>>> > producing
> > >> >> >>>>> > > > >> > counts to a side topic (and I was
> under
> > the
> > >> >> >>>> impression you do
> > >> >> >>>>> > the
> > >> >> >>>>> > > > >> > same), so we never need to modify the
> > message.
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >> > Another thing - after we added
> headers,
> > >> wouldn't
> > >> >> you
> > >> >> >>>> be in the
> > >> >> >>>>> > > > >> > business of making sure everyone uses
> > them
> > >> >> properly?
> > >> >> >>>> Making
> > >> >> >>>>> > sure
> > >> >> >>>>> > > > >> > everyone includes the right headers
> you
> > need,
> > >> not
> > >> >> >>>> using the
> > >> >> >>>>> > header
> > >> >> >>>>> > > > >> > names you intend to use, etc. I don't
> > think the
> > >> >> >>>> "policing"
> > >> >> >>>>> > business
> > >> >> >>>>> > > > >> > will ever go away.
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> > Palino <
> > >> >> >>>>> > tpalino@gmail.com>
> > >> >> >>>>> > > > wrote:
> > >> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not very
> > happy
> > >> with
> > >> >> the
> > >> >> >>>>> > workaround.
> > >> >> >>>>> > > Avro
> > >> >> >>>>> > > > >> means
> > >> >> >>>>> > > > >> > > that I have to be concerned with the
> > format
> > >> of
> > >> >> the
> > >> >> >>>> messages
> > >> >> >>>>> > in
> > >> >> >>>>> > > > order to
> > >> >> >>>>> > > > >> > run
> > >> >> >>>>> > > > >> > > the infrastructure (audit,
> mirroring,
> > etc.).
> > >> >> That
> > >> >> >>>> means
> > >> >> >>>>> > that I
> > >> >> >>>>> > > have
> > >> >> >>>>> > > > to
> > >> >> >>>>> > > > >> > > handle the schemas, and I have to
> > enforce
> > >> rules
> > >> >> >>>> about good
> > >> >> >>>>> > > formats.
> > >> >> >>>>> > > > >> This
> > >> >> >>>>> > > > >> > is
> > >> >> >>>>> > > > >> > > not something I want to be in the
> > business
> > >> of,
> > >> >> >>>> because I
> > >> >> >>>>> > should be
> > >> >> >>>>> > > > able
> > >> >> >>>>> > > > >> > to
> > >> >> >>>>> > > > >> > > run a service infrastructure without
> > needing
> > >> >> to be
> > >> >> >>>> in the
> > >> >> >>>>> > weeds of
> > >> >> >>>>> > > > >> > dealing
> > >> >> >>>>> > > > >> > > with customer data formats.
> > >> >> >>>>> > > > >> > >
> > >> >> >>>>> > > > >> > > Trust me, a sizable portion of my
> > support
> > >> time
> > >> >> is
> > >> >> >>>> spent
> > >> >> >>>>> > dealing
> > >> >> >>>>> > > with
> > >> >> >>>>> > > > >> > schema
> > >> >> >>>>> > > > >> > > issues. I really would like to get
> > away from
> > >> >> that.
> > >> >> >>>> Maybe
> > >> >> >>>>> > I'd have
> > >> >> >>>>> > > > more
> > >> >> >>>>> > > > >> > time
> > >> >> >>>>> > > > >> > > for other hobbies. Like writing. ;)
> > >> >> >>>>> > > > >> > >
> > >> >> >>>>> > > > >> > > -Todd
> > >> >> >>>>> > > > >> > >
> > >> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
> > Shapira <
> > >> >> >>>>> > gwen@confluent.io>
> > >> >> >>>>> > > > wrote:
> > >> >> >>>>> > > > >> > >
> > >> >> >>>>> > > > >> > >> I'm pretty satisfied with the
> current
> > >> >> workarounds
> > >> >> >>>> (Avro
> > >> >> >>>>> > container
> > >> >> >>>>> > > > >> > >> format), so I'm not too excited
> > about the
> > >> >> extra
> > >> >> >>>> work
> > >> >> >>>>> > required to
> > >> >> >>>>> > > do
> > >> >> >>>>> > > > >> > >> headers in Kafka. I absolutely
> don't
> > mind it
> > >> >> if
> > >> >> >>>> you do
> > >> >> >>>>> > it...
> > >> >> >>>>> > > > >> > >> I think the Apache convention for
> > "good
> > >> idea,
> > >> >> but
> > >> >> >>>> not
> > >> >> >>>>> > willing to
> > >> >> >>>>> > > > put
> > >> >> >>>>> > > > >> > >> any work toward it" is +0.5?
> anyway,
> > that's
> > >> >> what
> > >> >> >> I
> > >> >> >>>> was
> > >> >> >>>>> > trying to
> > >> >> >>>>> > > > >> > >> convey :)
> > >> >> >>>>> > > > >> > >>
> > >> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM,
> Todd
> > Palino
> > >> <
> > >> >> >>>>> > tpalino@gmail.com>
> > >> >> >>>>> > > > >> wrote:
> > >> >> >>>>> > > > >> > >> > Well I guess my question for you,
> > then, is
> > >> >> what
> > >> >> >>>> is
> > >> >> >>>>> > holding you
> > >> >> >>>>> > > > back
> > >> >> >>>>> > > > >> > from
> > >> >> >>>>> > > > >> > >> > full support for headers? What’s
> > the bit
> > >> >> that
> > >> >> >>>> you’re
> > >> >> >>>>> > missing
> > >> >> >>>>> > > that
> > >> >> >>>>> > > > >> has
> > >> >> >>>>> > > > >> > you
> > >> >> >>>>> > > > >> > >> > under a full +1?
> > >> >> >>>>> > > > >> > >> >
> > >> >> >>>>> > > > >> > >> > -Todd
> > >> >> >>>>> > > > >> > >> >
> > >> >> >>>>> > > > >> > >> >
> > >> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> > Gwen
> > >> >> Shapira <
> > >> >> >>>>> > > gwen@confluent.io>
> > >> >> >>>>> > > > >> > wrote:
> > >> >> >>>>> > > > >> > >> >
> > >> >> >>>>> > > > >> > >> >> I know why people who support
> > headers
> > >> >> support
> > >> >> >>>> them, and
> > >> >> >>>>> > I've
> > >> >> >>>>> > > > seen
> > >> >> >>>>> > > > >> > what
> > >> >> >>>>> > > > >> > >> >> the discussion is like.
> > >> >> >>>>> > > > >> > >> >>
> > >> >> >>>>> > > > >> > >> >> This is why I'm asking people
> who
> > are
> > >> >> against
> > >> >> >>>> headers
> > >> >> >>>>> > > > (especially
> > >> >> >>>>> > > > >> > >> >> committers) what will make them
> > change
> > >> >> their
> > >> >> >>>> mind - so
> > >> >> >>>>> > we can
> > >> >> >>>>> > > > get
> > >> >> >>>>> > > > >> > this
> > >> >> >>>>> > > > >> > >> >> part over one way or another.
> > >> >> >>>>> > > > >> > >> >>
> > >> >> >>>>> > > > >> > >> >> If I sound frustrated it is not
> > at Radai,
> > >> >> Jun
> > >> >> >>>> or you
> > >> >> >>>>> > (Todd)...
> > >> >> >>>>> > > > I am
> > >> >> >>>>> > > > >> > >> >> just looking for something
> > concrete we
> > >> can
> > >> >> do
> > >> >> >>>> to move
> > >> >> >>>>> > the
> > >> >> >>>>> > > > >> discussion
> > >> >> >>>>> > > > >> > >> >> along to the yummy design
> details
> > (which
> > >> is
> > >> >> >> the
> > >> >> >>>>> > argument I
> > >> >> >>>>> > > > really
> > >> >> >>>>> > > > >> am
> > >> >> >>>>> > > > >> > >> >> looking forward to).
> > >> >> >>>>> > > > >> > >> >>
> > >> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM,
> > Todd
> > >> >> Palino <
> > >> >> >>>>> > > tpalino@gmail.com>
> > >> >> >>>>> > > > >> > wrote:
> > >> >> >>>>> > > > >> > >> >> > So, Gwen, to your question
> > (even though
> > >> >> I’m
> > >> >> >>>> not a
> > >> >> >>>>> > > > committer)...
> > >> >> >>>>> > > > >> > >> >> >
> > >> >> >>>>> > > > >> > >> >> > I have always been a strong
> > supporter
> > >> of
> > >> >> >>>> introducing
> > >> >> >>>>> > the
> > >> >> >>>>> > > > concept
> > >> >> >>>>> > > > >> > of an
> > >> >> >>>>> > > > >> > >> >> > envelope to messages, which
> > headers
> > >> >> >>>> accomplishes. The
> > >> >> >>>>> > > message
> > >> >> >>>>> > > > key
> > >> >> >>>>> > > > >> > is
> > >> >> >>>>> > > > >> > >> >> > already an example of a piece
> of
> > >> envelope
> > >> >> >>>>> > information. By
> > >> >> >>>>> > > > >> > providing a
> > >> >> >>>>> > > > >> > >> >> means
> > >> >> >>>>> > > > >> > >> >> > to do this within Kafka
> itself,
> > and not
> > >> >> >>>> relying on
> > >> >> >>>>> > use-case
> > >> >> >>>>> > > > >> > specific
> > >> >> >>>>> > > > >> > >> >> > implementations, you make it
> > much
> > >> easier
> > >> >> for
> > >> >> >>>>> > components to
> > >> >> >>>>> > > > >> > >> interoperate.
> > >> >> >>>>> > > > >> > >> >> It
> > >> >> >>>>> > > > >> > >> >> > simplifies development of all
> > these
> > >> >> things
> > >> >> >>>> (message
> > >> >> >>>>> > routing,
> > >> >> >>>>> > > > >> > auditing,
> > >> >> >>>>> > > > >> > >> >> > encryption, etc.) because each
> > one does
> > >> >> not
> > >> >> >>>> have to
> > >> >> >>>>> > reinvent
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> wheel.
> > >> >> >>>>> > > > >> > >> >> >
> > >> >> >>>>> > > > >> > >> >> > It also makes it much easier
> > from a
> > >> >> client
> > >> >> >>>> point of
> > >> >> >>>>> > view if
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> headers
> > >> >> >>>>> > > > >> > >> >> are
> > >> >> >>>>> > > > >> > >> >> > defined as part of the
> protocol
> > and/or
> > >> >> >>>> message format
> > >> >> >>>>> > in
> > >> >> >>>>> > > > general
> > >> >> >>>>> > > > >> > >> because
> > >> >> >>>>> > > > >> > >> >> > you can easily produce and
> > consume
> > >> >> messages
> > >> >> >>>> without
> > >> >> >>>>> > having
> > >> >> >>>>> > > to
> > >> >> >>>>> > > > >> take
> > >> >> >>>>> > > > >> > >> into
> > >> >> >>>>> > > > >> > >> >> > account specific cases. For
> > example, I
> > >> >> want
> > >> >> >>>> to route
> > >> >> >>>>> > > messages,
> > >> >> >>>>> > > > >> but
> > >> >> >>>>> > > > >> > >> >> client A
> > >> >> >>>>> > > > >> > >> >> > doesn’t support the way audit
> > >> implemented
> > >> >> >>>> headers, and
> > >> >> >>>>> > > client
> > >> >> >>>>> > > > B
> > >> >> >>>>> > > > >> > >> doesn’t
> > >> >> >>>>> > > > >> > >> >> > support the way encryption or
> > routing
> > >> >> >>>> implemented
> > >> >> >>>>> > headers,
> > >> >> >>>>> > > so
> > >> >> >>>>> > > > now
> > >> >> >>>>> > > > >> > my
> > >> >> >>>>> > > > >> > >> >> > application has to create some
> > really
> > >> >> >> fragile
> > >> >> >>>> (my
> > >> >> >>>>> > > autocorrect
> > >> >> >>>>> > > > >> just
> > >> >> >>>>> > > > >> > >> tried
> > >> >> >>>>> > > > >> > >> >> to
> > >> >> >>>>> > > > >> > >> >> > make that “tragic”, which is
> > probably
> > >> >> >>>> appropriate
> > >> >> >>>>> > too) code
> > >> >> >>>>> > > to
> > >> >> >>>>> > > > >> > strip
> > >> >> >>>>> > > > >> > >> >> > everything off, rather than
> just
> > >> >> consuming
> > >> >> >> the
> > >> >> >>>>> > messages,
> > >> >> >>>>> > > > picking
> > >> >> >>>>> > > > >> > out
> > >> >> >>>>> > > > >> > >> the
> > >> >> >>>>> > > > >> > >> >> 1
> > >> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested
> > in, and
> > >> >> >>>> performing its
> > >> >> >>>>> > > function.
> > >> >> >>>>> > > > >> > >> >> >
> > >> >> >>>>> > > > >> > >> >> > Honestly, this discussion has
> > been
> > >> going
> > >> >> on
> > >> >> >>>> for a
> > >> >> >>>>> > long time,
> > >> >> >>>>> > > > and
> > >> >> >>>>> > > > >> > it’s
> > >> >> >>>>> > > > >> > >> >> > always “Oh, you came up with 2
> > use
> > >> cases,
> > >> >> >> and
> > >> >> >>>> yeah,
> > >> >> >>>>> > those
> > >> >> >>>>> > > use
> > >> >> >>>>> > > > >> cases
> > >> >> >>>>> > > > >> > >> are
> > >> >> >>>>> > > > >> > >> >> > real things that someone would
> > want to
> > >> >> do.
> > >> >> >>>> Here’s an
> > >> >> >>>>> > > alternate
> > >> >> >>>>> > > > >> way
> > >> >> >>>>> > > > >> > to
> > >> >> >>>>> > > > >> > >> >> > implement them so let’s not do
> > >> headers.”
> > >> >> If
> > >> >> >>>> we have a
> > >> >> >>>>> > few
> > >> >> >>>>> > > use
> > >> >> >>>>> > > > >> cases
> > >> >> >>>>> > > > >> > >> that
> > >> >> >>>>> > > > >> > >> >> we
> > >> >> >>>>> > > > >> > >> >> > actually came up with, you can
> > be sure
> > >> >> that
> > >> >> >>>> over the
> > >> >> >>>>> > next
> > >> >> >>>>> > > year
> > >> >> >>>>> > > > >> > >> there’s a
> > >> >> >>>>> > > > >> > >> >> > dozen others that we didn’t
> > think of
> > >> that
> > >> >> >>>> someone
> > >> >> >>>>> > would like
> > >> >> >>>>> > > > to
> > >> >> >>>>> > > > >> > do. I
> > >> >> >>>>> > > > >> > >> >> > really think it’s time to stop
> > >> rehashing
> > >> >> >> this
> > >> >> >>>>> > discussion and
> > >> >> >>>>> > > > >> > instead
> > >> >> >>>>> > > > >> > >> >> focus
> > >> >> >>>>> > > > >> > >> >> > on a workable standard that we
> > can
> > >> adopt.
> > >> >> >>>>> > > > >> > >> >> >
> > >> >> >>>>> > > > >> > >> >> > -Todd
> > >> >> >>>>> > > > >> > >> >> >
> > >> >> >>>>> > > > >> > >> >> >
> > >> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39
> PM,
> > Todd
> > >> >> Palino
> > >> >> >> <
> > >> >> >>>>> > > > tpalino@gmail.com>
> > >> >> >>>>> > > > >> > >> wrote:
> > >> >> >>>>> > > > >> > >> >> >
> > >> >> >>>>> > > > >> > >> >> >> C. per message encryption
> > >> >> >>>>> > > > >> > >> >> >>> One drawback of this
> approach
> > is that
> > >> >> this
> > >> >> >>>>> > significantly
> > >> >> >>>>> > > > reduce
> > >> >> >>>>> > > > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> effectiveness of
> compression,
> > which
> > >> >> >> happens
> > >> >> >>>> on a
> > >> >> >>>>> > set of
> > >> >> >>>>> > > > >> > serialized
> > >> >> >>>>> > > > >> > >> >> >>> messages. An alternative is
> > to enable
> > >> >> SSL
> > >> >> >>>> for wire
> > >> >> >>>>> > > > encryption
> > >> >> >>>>> > > > >> and
> > >> >> >>>>> > > > >> > >> rely
> > >> >> >>>>> > > > >> > >> >> on
> > >> >> >>>>> > > > >> > >> >> >>> the storage system (e.g.
> > LUKS) for at
> > >> >> rest
> > >> >> >>>>> > encryption.
> > >> >> >>>>> > > > >> > >> >> >>
> > >> >> >>>>> > > > >> > >> >> >>
> > >> >> >>>>> > > > >> > >> >> >> Jun, this is not sufficient.
> > While
> > >> this
> > >> >> >> does
> > >> >> >>>> cover
> > >> >> >>>>> > the case
> > >> >> >>>>> > > > of
> > >> >> >>>>> > > > >> > >> removing
> > >> >> >>>>> > > > >> > >> >> a
> > >> >> >>>>> > > > >> > >> >> >> drive from the system, it
> will
> > not
> > >> >> satisfy
> > >> >> >>>> most
> > >> >> >>>>> > compliance
> > >> >> >>>>> > > > >> > >> requirements
> > >> >> >>>>> > > > >> > >> >> for
> > >> >> >>>>> > > > >> > >> >> >> encryption of data as whoever
> > has
> > >> >> access to
> > >> >> >>>> the
> > >> >> >>>>> > broker
> > >> >> >>>>> > > itself
> > >> >> >>>>> > > > >> > still
> > >> >> >>>>> > > > >> > >> has
> > >> >> >>>>> > > > >> > >> >> >> access to the unencrypted
> > data. For
> > >> >> >>>> end-to-end
> > >> >> >>>>> > encryption
> > >> >> >>>>> > > you
> > >> >> >>>>> > > > >> > need to
> > >> >> >>>>> > > > >> > >> >> >> encrypt at the producer,
> > before it
> > >> >> enters
> > >> >> >> the
> > >> >> >>>>> > system, and
> > >> >> >>>>> > > > >> decrypt
> > >> >> >>>>> > > > >> > at
> > >> >> >>>>> > > > >> > >> the
> > >> >> >>>>> > > > >> > >> >> >> consumer, after it exits the
> > system.
> > >> >> >>>>> > > > >> > >> >> >>
> > >> >> >>>>> > > > >> > >> >> >> -Todd
> > >> >> >>>>> > > > >> > >> >> >>
> > >> >> >>>>> > > > >> > >> >> >>
> > >> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03
> > PM, radai
> > >> <
> > >> >> >>>>> > > > >> radai.rosenblatt@gmail.com
> > >> >> >>>>> > > > >> > >
> > >> >> >>>>> > > > >> > >> >> wrote:
> > >> >> >>>>> > > > >> > >> >> >>
> > >> >> >>>>> > > > >> > >> >> >>> another big plus of headers
> > in the
> > >> >> >> protocol
> > >> >> >>>> is that
> > >> >> >>>>> > it
> > >> >> >>>>> > > would
> > >> >> >>>>> > > > >> > enable
> > >> >> >>>>> > > > >> > >> >> rapid
> > >> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside
> of
> > core
> > >> >> kafka
> > >> >> >>>> and would
> > >> >> >>>>> > reduce
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> number of
> > >> >> >>>>> > > > >> > >> >> >>> future wire format changes
> > required.
> > >> >> >>>>> > > > >> > >> >> >>>
> > >> >> >>>>> > > > >> > >> >> >>> a lot of what is currently a
> > KIP
> > >> >> >> represents
> > >> >> >>>> use
> > >> >> >>>>> > cases that
> > >> >> >>>>> > > > are
> > >> >> >>>>> > > > >> > not
> > >> >> >>>>> > > > >> > >> 100%
> > >> >> >>>>> > > > >> > >> >> >>> relevant to all users, and
> > some of
> > >> them
> > >> >> >>>> require
> > >> >> >>>>> > rather
> > >> >> >>>>> > > > invasive
> > >> >> >>>>> > > > >> > wire
> > >> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing a
> > good
> > >> recent
> > >> >> >>>> example of
> > >> >> >>>>> > this is
> > >> >> >>>>> > > > >> > kip-98.
> > >> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is
> > expected to
> > >> be
> > >> >> a
> > >> >> >>>> very small
> > >> >> >>>>> > > > fraction of
> > >> >> >>>>> > > > >> > >> total
> > >> >> >>>>> > > > >> > >> >> >>> traffic and yet the changes
> > are
> > >> >> invasive.
> > >> >> >>>>> > > > >> > >> >> >>>
> > >> >> >>>>> > > > >> > >> >> >>> every such wire format
> change
> > >> >> translates
> > >> >> >>>> into
> > >> >> >>>>> > painful and
> > >> >> >>>>> > > > slow
> > >> >> >>>>> > > > >> > >> >> adoption of
> > >> >> >>>>> > > > >> > >> >> >>> new versions.
> > >> >> >>>>> > > > >> > >> >> >>>
> > >> >> >>>>> > > > >> > >> >> >>> i think a lot of
> functionality
> > >> >> currently
> > >> >> >> in
> > >> >> >>>> KIPs
> > >> >> >>>>> > could be
> > >> >> >>>>> > > > "spun
> > >> >> >>>>> > > > >> > out"
> > >> >> >>>>> > > > >> > >> >> and
> > >> >> >>>>> > > > >> > >> >> >>> implemented as opt-in
> plugins
> > >> >> transmitting
> > >> >> >>>> data over
> > >> >> >>>>> > > > headers.
> > >> >> >>>>> > > > >> > this
> > >> >> >>>>> > > > >> > >> >> would
> > >> >> >>>>> > > > >> > >> >> >>> keep the core wire format
> > stable(r),
> > >> >> core
> > >> >> >>>> codebase
> > >> >> >>>>> > > smaller,
> > >> >> >>>>> > > > and
> > >> >> >>>>> > > > >> > >> avoid
> > >> >> >>>>> > > > >> > >> >> the
> > >> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats
> > sometimes
> > >> >> required
> > >> >> >>>> to prove
> > >> >> >>>>> > a
> > >> >> >>>>> > > > certain
> > >> >> >>>>> > > > >> > >> feature
> > >> >> >>>>> > > > >> > >> >> is
> > >> >> >>>>> > > > >> > >> >> >>> useful enough for a
> > wide-enough
> > >> >> audience
> > >> >> >> to
> > >> >> >>>> warrant
> > >> >> >>>>> > a wire
> > >> >> >>>>> > > > >> format
> > >> >> >>>>> > > > >> > >> >> change
> > >> >> >>>>> > > > >> > >> >> >>> and code complexity
> additions.
> > >> >> >>>>> > > > >> > >> >> >>>
> > >> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
> > beyond
> > >> >> "mere"
> > >> >> >>>> wire format
> > >> >> >>>>> > > changes
> > >> >> >>>>> > > > >> and
> > >> >> >>>>> > > > >> > im
> > >> >> >>>>> > > > >> > >> not
> > >> >> >>>>> > > > >> > >> >> >>> saying it could have been
> > completely
> > >> >> done
> > >> >> >>>> with
> > >> >> >>>>> > headers,
> > >> >> >>>>> > > but
> > >> >> >>>>> > > > >> > >> >> exactly-once
> > >> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
> > >> >> >>>>> > > > >> > >> >> >>>
> > >> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20
> > AM, Gwen
> > >> >> >>>> Shapira <
> > >> >> >>>>> > > > >> gwen@confluent.io
> > >> >> >>>>> > > > >> > >
> > >> >> >>>>> > > > >> > >> >> wrote:
> > >> >> >>>>> > > > >> > >> >> >>>
> > >> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> > 10:24 AM,
> > >> >> radai <
> > >> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> > >> >> >>>>> > > > >> > >> >> >>> wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > > "For use cases within an
> > >> >> organization,
> > >> >> >>>> one could
> > >> >> >>>>> > > always
> > >> >> >>>>> > > > use
> > >> >> >>>>> > > > >> > >> other
> > >> >> >>>>> > > > >> > >> >> >>> > > approaches such as
> > company-wise
> > >> >> >>>> containers"
> > >> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin
> has
> > >> >> >> traditionally
> > >> >> >>>> done
> > >> >> >>>>> > but there
> > >> >> >>>>> > > > are
> > >> >> >>>>> > > > >> > now
> > >> >> >>>>> > > > >> > >> >> cases
> > >> >> >>>>> > > > >> > >> >> >>> > (read
> > >> >> >>>>> > > > >> > >> >> >>> > > - topics) where this is
> > not
> > >> >> >> acceptable.
> > >> >> >>>> this
> > >> >> >>>>> > makes
> > >> >> >>>>> > > > headers
> > >> >> >>>>> > > > >> > >> useful
> > >> >> >>>>> > > > >> > >> >> even
> > >> >> >>>>> > > > >> > >> >> >>> > > within single orgs for
> > cases
> > >> where
> > >> >> >>>>> > > > one-container-fits-all
> > >> >> >>>>> > > > >> > cannot
> > >> >> >>>>> > > > >> > >> >> >>> apply.
> > >> >> >>>>> > > > >> > >> >> >>> > >
> > >> >> >>>>> > > > >> > >> >> >>> > > as for the particular
> use
> > cases
> > >> >> >> listed,
> > >> >> >>>> i dont
> > >> >> >>>>> > want
> > >> >> >>>>> > > > this to
> > >> >> >>>>> > > > >> > >> devolve
> > >> >> >>>>> > > > >> > >> >> >>> to a
> > >> >> >>>>> > > > >> > >> >> >>> > > discussion of particular
> > use
> > >> cases
> > >> >> - i
> > >> >> >>>> think its
> > >> >> >>>>> > > enough
> > >> >> >>>>> > > > >> that
> > >> >> >>>>> > > > >> > >> some
> > >> >> >>>>> > > > >> > >> >> of
> > >> >> >>>>> > > > >> > >> >> >>> them
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > I think a main point of
> > contention
> > >> is
> > >> >> >>>> that: We
> > >> >> >>>>> > > identified
> > >> >> >>>>> > > > few
> > >> >> >>>>> > > > >> > >> >> >>> > use-cases where headers
> are
> > useful,
> > >> >> do
> > >> >> >> we
> > >> >> >>>> want
> > >> >> >>>>> > Kafka to
> > >> >> >>>>> > > > be a
> > >> >> >>>>> > > > >> > >> system
> > >> >> >>>>> > > > >> > >> >> >>> > that supports those
> > use-cases?
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
> > >> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely
> useful
> > >> >> record-level
> > >> >> >>>> lineage
> > >> >> >>>>> > is
> > >> >> >>>>> > > though
> > >> >> >>>>> > > > >> > since
> > >> >> >>>>> > > > >> > >> the
> > >> >> >>>>> > > > >> > >> >> >>> > overhead could
> > >> >> >>>>> > > > >> > >> >> >>> > be significant."
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports
> > record level
> > >> >> >>>> lineage. I
> > >> >> >>>>> > don't
> > >> >> >>>>> > > think
> > >> >> >>>>> > > > it
> > >> >> >>>>> > > > >> > was
> > >> >> >>>>> > > > >> > >> >> >>> > developed for lols, I
> think
> > it is
> > >> >> safe
> > >> >> >> to
> > >> >> >>>> assume
> > >> >> >>>>> > that
> > >> >> >>>>> > > the
> > >> >> >>>>> > > > NSA
> > >> >> >>>>> > > > >> > >> needed
> > >> >> >>>>> > > > >> > >> >> >>> > that functionality. We
> also
> > know
> > >> that
> > >> >> >>>> certain
> > >> >> >>>>> > financial
> > >> >> >>>>> > > > >> > institutes
> > >> >> >>>>> > > > >> > >> >> >>> > need to track tampering
> with
> > >> records
> > >> >> at
> > >> >> >> a
> > >> >> >>>> record
> > >> >> >>>>> > level
> > >> >> >>>>> > > and
> > >> >> >>>>> > > > >> > there
> > >> >> >>>>> > > > >> > >> are
> > >> >> >>>>> > > > >> > >> >> >>> > federal regulations that
> > absolutely
> > >> >> >>>> require
> > >> >> >>>>> > this. They
> > >> >> >>>>> > > > also
> > >> >> >>>>> > > > >> > need
> > >> >> >>>>> > > > >> > >> to
> > >> >> >>>>> > > > >> > >> >> >>> > prove that routing apps
> that
> > >> >> "touches"
> > >> >> >> the
> > >> >> >>>>> > messages and
> > >> >> >>>>> > > > >> either
> > >> >> >>>>> > > > >> > >> reads
> > >> >> >>>>> > > > >> > >> >> >>> > or updates headers
> couldn't
> > have
> > >> >> >> possibly
> > >> >> >>>>> > modified the
> > >> >> >>>>> > > > >> payload
> > >> >> >>>>> > > > >> > >> >> itself.
> > >> >> >>>>> > > > >> > >> >> >>> > They use record level
> > encryption to
> > >> >> do
> > >> >> >>>> that -
> > >> >> >>>>> > apps can
> > >> >> >>>>> > > > read
> > >> >> >>>>> > > > >> and
> > >> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify headers
> > but
> > >> can't
> > >> >> >>>> touch the
> > >> >> >>>>> > payload.
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > We can totally say "those
> > are
> > >> corner
> > >> >> >>>> cases and
> > >> >> >>>>> > not worth
> > >> >> >>>>> > > > >> adding
> > >> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for",
> they
> > should
> > >> >> use a
> > >> >> >>>> different
> > >> >> >>>>> > > pubsub
> > >> >> >>>>> > > > >> > message
> > >> >> >>>>> > > > >> > >> for
> > >> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the
> > other 1000
> > >> >> that
> > >> >> >>>> cater
> > >> >> >>>>> > > > specifically
> > >> >> >>>>> > > > >> to
> > >> >> >>>>> > > > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> > financial industry).
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > But this gets us into a
> > catch 22:
> > >> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific
> > use-case,
> > >> >> >>>> someone can
> > >> >> >>>>> > always
> > >> >> >>>>> > > say
> > >> >> >>>>> > > > it
> > >> >> >>>>> > > > >> > isn't
> > >> >> >>>>> > > > >> > >> >> >>> > interesting enough for
> > Kafka. If we
> > >> >> >>>> discuss more
> > >> >> >>>>> > general
> > >> >> >>>>> > > > >> > trends,
> > >> >> >>>>> > > > >> > >> >> >>> > others can say "well, we
> > are not
> > >> sure
> > >> >> >> any
> > >> >> >>>> of them
> > >> >> >>>>> > really
> > >> >> >>>>> > > > >> needs
> > >> >> >>>>> > > > >> > >> >> headers
> > >> >> >>>>> > > > >> > >> >> >>> > specifically. This is just
> > hand
> > >> >> waving
> > >> >> >>>> and not
> > >> >> >>>>> > > > interesting.".
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > I think discussing
> > use-cases in
> > >> >> >> specifics
> > >> >> >>>> is super
> > >> >> >>>>> > > > important
> > >> >> >>>>> > > > >> to
> > >> >> >>>>> > > > >> > >> >> decide
> > >> >> >>>>> > > > >> > >> >> >>> > implementation details for
> > headers
> > >> >> (my
> > >> >> >>>> use-cases
> > >> >> >>>>> > lean
> > >> >> >>>>> > > > toward
> > >> >> >>>>> > > > >> > >> >> numerical
> > >> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and
> > object
> > >> >> values,
> > >> >> >>>> others
> > >> >> >>>>> > differ),
> > >> >> >>>>> > > > but I
> > >> >> >>>>> > > > >> > >> think
> > >> >> >>>>> > > > >> > >> >> we
> > >> >> >>>>> > > > >> > >> >> >>> > need to answer the general
> > "Are we
> > >> >> going
> > >> >> >>>> to have
> > >> >> >>>>> > > headers"
> > >> >> >>>>> > > > >> > question
> > >> >> >>>>> > > > >> > >> >> >>> > first.
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from the
> > other
> > >> >> >>>> committers in the
> > >> >> >>>>> > > > discussion:
> > >> >> >>>>> > > > >> > >> >> >>> > What would it take to
> > convince you
> > >> >> that
> > >> >> >>>> headers
> > >> >> >>>>> > in Kafka
> > >> >> >>>>> > > > are
> > >> >> >>>>> > > > >> a
> > >> >> >>>>> > > > >> > >> good
> > >> >> >>>>> > > > >> > >> >> >>> > idea in general, so we can
> > move
> > >> ahead
> > >> >> >> and
> > >> >> >>>> try to
> > >> >> >>>>> > agree
> > >> >> >>>>> > > on
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> >> details?
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > I feel like we keep moving
> > the goal
> > >> >> >> posts
> > >> >> >>>> and
> > >> >> >>>>> > this is
> > >> >> >>>>> > > > truly
> > >> >> >>>>> > > > >> > >> >> exhausting.
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly
> > support
> > >> >> adding
> > >> >> >>>> headers
> > >> >> >>>>> > to Kafka
> > >> >> >>>>> > > > >> > (+0.5?).
> > >> >> >>>>> > > > >> > >> >> >>> > The community can continue
> > to find
> > >> >> >>>> workarounds to
> > >> >> >>>>> > the
> > >> >> >>>>> > > > issue
> > >> >> >>>>> > > > >> and
> > >> >> >>>>> > > > >> > >> there
> > >> >> >>>>> > > > >> > >> >> >>> > are some benefits to
> > keeping the
> > >> >> message
> > >> >> >>>> format
> > >> >> >>>>> > and
> > >> >> >>>>> > > > clients
> > >> >> >>>>> > > > >> > >> simpler.
> > >> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness
> of
> > headers
> > >> >> to
> > >> >> >>>> many
> > >> >> >>>>> > use-cases
> > >> >> >>>>> > > and
> > >> >> >>>>> > > > if
> > >> >> >>>>> > > > >> we
> > >> >> >>>>> > > > >> > >> can
> > >> >> >>>>> > > > >> > >> >> >>> > find a good and generally
> > useful
> > >> way
> > >> >> to
> > >> >> >>>> add it to
> > >> >> >>>>> > Kafka,
> > >> >> >>>>> > > > it
> > >> >> >>>>> > > > >> > will
> > >> >> >>>>> > > > >> > >> make
> > >> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for
> > many -
> > >> worthy
> > >> >> >>>> goal in my
> > >> >> >>>>> > eyes.
> > >> >> >>>>> > > > >> > >> >> >>> >
> > >> >> >>>>> > > > >> > >> >> >>> > > are
> interesting/feasible,
> > but:
> > >> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are
> > use cases
> > >> >> for
> > >> >> >>>> polyglot
> > >> >> >>>>> > topics.
> > >> >> >>>>> > > > >> > >> especially if
> > >> >> >>>>> > > > >> > >> >> >>> kafka
> > >> >> >>>>> > > > >> > >> >> >>> > > is being used to "trunk"
> > >> something
> > >> >> >> else.
> > >> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics would
> > make it
> > >> >> >> harder
> > >> >> >>>> to write
> > >> >> >>>>> > > > portable
> > >> >> >>>>> > > > >> > >> consumer
> > >> >> >>>>> > > > >> > >> >> >>> code.
> > >> >> >>>>> > > > >> > >> >> >>> > > partition remapping
> would
> > mess
> > >> with
> > >> >> >>>> locality of
> > >> >> >>>>> > > > consumption
> > >> >> >>>>> > > > >> > >> >> >>> guarantees.
> > >> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see
> for
> > >> >> >>>> lineage/metadata is
> > >> >> >>>>> > > > >> > >> billing/chargeback.
> > >> >> >>>>> > > > >> > >> >> for
> > >> >> >>>>> > > > >> > >> >> >>> > that
> > >> >> >>>>> > > > >> > >> >> >>> > > use case it is not
> enough
> > to
> > >> simply
> > >> >> >>>> record the
> > >> >> >>>>> > point
> > >> >> >>>>> > > of
> > >> >> >>>>> > > > >> > origin,
> > >> >> >>>>> > > > >> > >> but
> > >> >> >>>>> > > > >> > >> >> >>> every
> > >> >> >>>>> > > > >> > >> >> >>> > > replication stop (think
> > mirror
> > >> >> maker)
> > >> >> >>>> must also
> > >> >> >>>>> > add a
> > >> >> >>>>> > > > >> record
> > >> >> >>>>> > > > >> > to
> > >> >> >>>>> > > > >> > >> >> form a
> > >> >> >>>>> > > > >> > >> >> >>> > > "transit log".
> > >> >> >>>>> > > > >> > >> >> >>> > >
> > >> >> >>>>> > > > >> > >> >> >>> > > as for stream processing
> > on top
> > >> of
> > >> >> >>>> kafka - i
> > >> >> >>>>> > know
> > >> >> >>>>> > > samza
> > >> >> >>>>> > > > >> has a
> > >> >> >>>>> > > > >> > >> >> metadata
> > >> >> >>>>> > > > >> > >> >> >>> > map
> > >> >> >>>>> > > > >> > >> >> >>> > > which they carry around
> in
> > >> >> addition to
> > >> >> >>>> user
> > >> >> >>>>> > values.
> > >> >> >>>>> > > > headers
> > >> >> >>>>> > > > >> > are
> > >> >> >>>>> > > > >> > >> the
> > >> >> >>>>> > > > >> > >> >> >>> > perfect
> > >> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
> > >> >> >>>>> > > > >> > >> >> >>> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >
> > >> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at
> > 6:50 PM,
> > >> >> Jun
> > >> >> >>>> Rao <
> > >> >> >>>>> > > > jun@confluent.io
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >> > >> wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> In order to answer the
> > first two
> > >> >> >>>> questions, it
> > >> >> >>>>> > would
> > >> >> >>>>> > > be
> > >> >> >>>>> > > > >> > helpful
> > >> >> >>>>> > > > >> > >> >> if we
> > >> >> >>>>> > > > >> > >> >> >>> > could
> > >> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong
> > use cases
> > >> >> for
> > >> >> >>>> headers
> > >> >> >>>>> > in the
> > >> >> >>>>> > > > space
> > >> >> >>>>> > > > >> > for
> > >> >> >>>>> > > > >> > >> >> >>> > third-party
> > >> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use cases
> > within an
> > >> >> >>>> organization,
> > >> >> >>>>> > one
> > >> >> >>>>> > > > could
> > >> >> >>>>> > > > >> > always
> > >> >> >>>>> > > > >> > >> >> use
> > >> >> >>>>> > > > >> > >> >> >>> > other
> > >> >> >>>>> > > > >> > >> >> >>> > >> approaches such as
> > company-wise
> > >> >> >>>> containers to
> > >> >> >>>>> > get
> > >> >> >>>>> > > > around
> > >> >> >>>>> > > > >> w/o
> > >> >> >>>>> > > > >> > >> >> >>> headers. I
> > >> >> >>>>> > > > >> > >> >> >>> > >> went through the use
> > cases in
> > >> the
> > >> >> KIP
> > >> >> >>>> and in
> > >> >> >>>>> > Radai's
> > >> >> >>>>> > > > wiki
> > >> >> >>>>> > > > >> (
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > https://cwiki.apache.org/confl
> > >> >> >>>>> > uence/display/KAFKA/A+
> > >> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> > >> >> >>>>> > > > >> > >> >> >>> > >> ).
> > >> >> >>>>> > > > >> > >> >> >>> > >> The following are the
> > ones that
> > >> >> that
> > >> >> >> I
> > >> >> >>>>> > understand and
> > >> >> >>>>> > > > >> could
> > >> >> >>>>> > > > >> > be
> > >> >> >>>>> > > > >> > >> in
> > >> >> >>>>> > > > >> > >> >> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> third-party use case
> > category.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
> > >> >> >>>>> > > > >> > >> >> >>> > >> It seems that in
> general,
> > >> >> >> content-type
> > >> >> >>>> should
> > >> >> >>>>> > be set
> > >> >> >>>>> > > at
> > >> >> >>>>> > > > >> the
> > >> >> >>>>> > > > >> > >> topic
> > >> >> >>>>> > > > >> > >> >> >>> level.
> > >> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing
> > messages with
> > >> >> >>>> different
> > >> >> >>>>> > content
> > >> >> >>>>> > > > types
> > >> >> >>>>> > > > >> > >> should be
> > >> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
> > >> >> >>>>> > > > >> > >> >> >>> > >> Since the value is
> mostly
> > >> useless
> > >> >> >>>> without
> > >> >> >>>>> > schema id,
> > >> >> >>>>> > > it
> > >> >> >>>>> > > > >> > seems
> > >> >> >>>>> > > > >> > >> that
> > >> >> >>>>> > > > >> > >> >> >>> > storing
> > >> >> >>>>> > > > >> > >> >> >>> > >> the schema id together
> > with
> > >> >> >> serialized
> > >> >> >>>> bytes
> > >> >> >>>>> > in the
> > >> >> >>>>> > > > value
> > >> >> >>>>> > > > >> is
> > >> >> >>>>> > > > >> > >> >> better?
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> C. per message
> encryption
> > >> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this
> > approach is
> > >> >> that
> > >> >> >>>> this
> > >> >> >>>>> > > > significantly
> > >> >> >>>>> > > > >> > reduce
> > >> >> >>>>> > > > >> > >> >> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of
> > compression,
> > >> >> which
> > >> >> >>>> happens on
> > >> >> >>>>> > a set
> > >> >> >>>>> > > of
> > >> >> >>>>> > > > >> > >> serialized
> > >> >> >>>>> > > > >> > >> >> >>> > >> messages. An
> alternative
> > is to
> > >> >> enable
> > >> >> >>>> SSL for
> > >> >> >>>>> > wire
> > >> >> >>>>> > > > >> > encryption
> > >> >> >>>>> > > > >> > >> and
> > >> >> >>>>> > > > >> > >> >> >>> rely
> > >> >> >>>>> > > > >> > >> >> >>> > on
> > >> >> >>>>> > > > >> > >> >> >>> > >> the storage system
> (e.g.
> > LUKS)
> > >> >> for at
> > >> >> >>>> rest
> > >> >> >>>>> > > encryption.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for
> > mirroring
> > >> across
> > >> >> >>>> Kafka
> > >> >> >>>>> > clusters
> > >> >> >>>>> > > > >> > >> >> >>> > >> This is actually
> > interesting.
> > >> >> Today,
> > >> >> >>>> to avoid
> > >> >> >>>>> > > > introducing
> > >> >> >>>>> > > > >> > >> cycles
> > >> >> >>>>> > > > >> > >> >> when
> > >> >> >>>>> > > > >> > >> >> >>> > doing
> > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data
> > centers,
> > >> one
> > >> >> >>>> would
> > >> >> >>>>> > either have
> > >> >> >>>>> > > to
> > >> >> >>>>> > > > >> set
> > >> >> >>>>> > > > >> > up
> > >> >> >>>>> > > > >> > >> two
> > >> >> >>>>> > > > >> > >> >> >>> Kafka
> > >> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and
> an
> > >> >> aggregate)
> > >> >> >>>> per data
> > >> >> >>>>> > center
> > >> >> >>>>> > > or
> > >> >> >>>>> > > > >> > rename
> > >> >> >>>>> > > > >> > >> >> topics.
> > >> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
> > headers,
> > >> >> the
> > >> >> >>>> producer
> > >> >> >>>>> > could
> > >> >> >>>>> > > tag
> > >> >> >>>>> > > > >> each
> > >> >> >>>>> > > > >> > >> >> message
> > >> >> >>>>> > > > >> > >> >> >>> with
> > >> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster
> ID
> > in the
> > >> >> >> header.
> > >> >> >>>>> > MirrorMaker
> > >> >> >>>>> > > > could
> > >> >> >>>>> > > > >> > then
> > >> >> >>>>> > > > >> > >> >> avoid
> > >> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages to a
> > cluster
> > >> if
> > >> >> >>>> they are
> > >> >> >>>>> > tagged
> > >> >> >>>>> > > with
> > >> >> >>>>> > > > >> the
> > >> >> >>>>> > > > >> > >> same
> > >> >> >>>>> > > > >> > >> >> >>> cluster
> > >> >> >>>>> > > > >> > >> >> >>> > >> id.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> However, an alternative
> > approach
> > >> >> is
> > >> >> >> to
> > >> >> >>>>> > introduce sth
> > >> >> >>>>> > > > like
> > >> >> >>>>> > > > >> > >> >> >>> hierarchical
> > >> >> >>>>> > > > >> > >> >> >>> > >> topic and store
> messages
> > from
> > >> >> >> different
> > >> >> >>>>> > clusters in
> > >> >> >>>>> > > > >> > different
> > >> >> >>>>> > > > >> > >> >> >>> partitions
> > >> >> >>>>> > > > >> > >> >> >>> > >> under the same topic.
> > This
> > >> >> approach
> > >> >> >>>> avoids
> > >> >> >>>>> > filtering
> > >> >> >>>>> > > > out
> > >> >> >>>>> > > > >> > >> unneeded
> > >> >> >>>>> > > > >> > >> >> >>> data
> > >> >> >>>>> > > > >> > >> >> >>> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> makes offset preserving
> > easier
> > >> to
> > >> >> >>>> support. It
> > >> >> >>>>> > may
> > >> >> >>>>> > > make
> > >> >> >>>>> > > > >> > >> compaction
> > >> >> >>>>> > > > >> > >> >> >>> > trickier
> > >> >> >>>>> > > > >> > >> >> >>> > >> though since the same
> > key may
> > >> >> show up
> > >> >> >>>> in
> > >> >> >>>>> > different
> > >> >> >>>>> > > > >> > partitions.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> E. record-level lineage
> > >> >> >>>>> > > > >> > >> >> >>> > >> For example, a source
> > connector
> > >> >> could
> > >> >> >>>> store in
> > >> >> >>>>> > the
> > >> >> >>>>> > > > message
> > >> >> >>>>> > > > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> metadata
> > >> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the
> source
> > >> record.
> > >> >> >>>> Similarly,
> > >> >> >>>>> > if a
> > >> >> >>>>> > > > stream
> > >> >> >>>>> > > > >> job
> > >> >> >>>>> > > > >> > >> >> >>> transforms
> > >> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A
> to
> > topic
> > >> B,
> > >> >> the
> > >> >> >>>> library
> > >> >> >>>>> > could
> > >> >> >>>>> > > > >> include
> > >> >> >>>>> > > > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> source
> > >> >> >>>>> > > > >> > >> >> >>> > >> message offset in each
> > of the
> > >> >> >>>> transformed
> > >> >> >>>>> > message in
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> header.
> > >> >> >>>>> > > > >> > >> >> Not
> > >> >> >>>>> > > > >> > >> >> >>> > sure
> > >> >> >>>
> > >> >> >>>>> > > > >> > >> >> >>> > >> how widely useful
> > record-level
> > >> >> >> lineage
> > >> >> >>>> is
> > >> >> >>>>> > though
> > >> >> >>>>> > > since
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> >> overhead
> > >> >> >>>>> > > > >> > >> >> >>> > could
> > >> >> >>>>> > > > >> > >> >> >>> > >> be significant.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> > >> >> >>>>> > > > >> > >> >> >>> > >> We could put things
> like
> > >> >> >>>> clientId/host/user in
> > >> >> >>>>> > the
> > >> >> >>>>> > > > header
> > >> >> >>>>> > > > >> in
> > >> >> >>>>> > > > >> > >> each
> > >> >> >>>>> > > > >> > >> >> >>> > message
> > >> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These
> > metadata are
> > >> >> >>>> really at the
> > >> >> >>>>> > > producer
> > >> >> >>>>> > > > >> > level
> > >> >> >>>>> > > > >> > >> >> though.
> > >> >> >>>>> > > > >> > >> >> >>> > So, a
> > >> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is
> to
> > only
> > >> >> >> include a
> > >> >> >>>>> > "producerId"
> > >> >> >>>>> > > > per
> > >> >> >>>>> > > > >> > >> message
> > >> >> >>>>> > > > >> > >> >> and
> > >> >> >>>>> > > > >> > >> >> >>> > send
> > >> >> >>>>> > > > >> > >> >> >>> > >> the producerId ->
> > metadata
> > >> mapping
> > >> >> >>>>> > independently.
> > >> >> >>>>> > > > KIP-98
> > >> >> >>>>> > > > >> is
> > >> >> >>>>> > > > >> > >> >> actually
> > >> >> >>>>> > > > >> > >> >> >>> > >> proposing including
> such
> > a
> > >> >> producerId
> > >> >> >>>> natively
> > >> >> >>>>> > in the
> > >> >> >>>>> > > > >> > message.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not sure
> > that I
> > >> am
> > >> >> >> fully
> > >> >> >>>>> > convinced of
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > strong
> > >> >> >>>>> > > > >> > >> >> >>> > third-party
> > >> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers
> yet.
> > >> Perhaps
> > >> >> we
> > >> >> >>>> could
> > >> >> >>>>> > discuss a
> > >> >> >>>>> > > > bit
> > >> >> >>>>> > > > >> > more
> > >> >> >>>>> > > > >> > >> to
> > >> >> >>>>> > > > >> > >> >> make
> > >> >> >>>>> > > > >> > >> >> >>> > one
> > >> >> >>>>> > > > >> > >> >> >>> > >> or two really
> convincing
> > use
> > >> >> cases.
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal
> > question is
> > >> >> >>>> whether header
> > >> >> >>>>> > should
> > >> >> >>>>> > > > be
> > >> >> >>>>> > > > >> > >> exposed
> > >> >> >>>>> > > > >> > >> >> in
> > >> >> >>>>> > > > >> > >> >> >>> > stream
> > >> >> >>>>> > > > >> > >> >> >>> > >> processing systems such
> > Kafka
> > >> >> stream,
> > >> >> >>>> Samza,
> > >> >> >>>>> > and
> > >> >> >>>>> > > Spark
> > >> >> >>>>> > > > >> > >> streaming.
> > >> >> >>>>> > > > >> > >> >> >>> > >> Currently, those
> systems
> > just
> > >> deal
> > >> >> >> with
> > >> >> >>>>> > key/value
> > >> >> >>>>> > > > pairs.
> > >> >> >>>>> > > > >> > >> Should we
> > >> >> >>>>> > > > >> > >> >> >>> > expose a
> > >> >> >>>>> > > > >> > >> >> >>> > >> third thing header
> there
> > too or
> > >> >> >>>> somehow map
> > >> >> >>>>> > header to
> > >> >> >>>>> > > > key
> > >> >> >>>>> > > > >> or
> > >> >> >>>>> > > > >> > >> >> value?
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> Jun
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at
> > 3:35 AM,
> > >> >> >>>> Michael
> > >> >> >>>>> > Pearce <
> > >> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that after
> a
> > period
> > >> >> of a
> > >> >> >>>> week,
> > >> >> >>>>> > that there
> > >> >> >>>>> > > > is
> > >> >> >>>>> > > > >> no
> > >> >> >>>>> > > > >> > >> >> concerns
> > >> >> >>>>> > > > >> > >> >> >>> now
> > >> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and 2
> > and now
> > >> we
> > >> >> >> have
> > >> >> >>>>> > agreement that
> > >> >> >>>>> > > > >> > headers
> > >> >> >>>>> > > > >> > >> are
> > >> >> >>>>> > > > >> > >> >> >>> useful
> > >> >> >>>>> > > > >> > >> >> >>> > >> and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
> > such if
> > >> put
> > >> >> to
> > >> >> >> a
> > >> >> >>>> KIP
> > >> >> >>>>> > vote, this
> > >> >> >>>>> > > > >> > wouldn’t
> > >> >> >>>>> > > > >> > >> be
> > >> >> >>>>> > > > >> > >> >> a
> > >> >> >>>>> > > > >> > >> >> >>> > reason
> > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > reject.
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > @
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
> > >> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose
> of
> > getting
> > >> >> this
> > >> >> >>>> KIP
> > >> >> >>>>> > moving past
> > >> >> >>>>> > > > >> this,
> > >> >> >>>>> > > > >> > we
> > >> >> >>>>> > > > >> > >> can
> > >> >> >>>>> > > > >> > >> >> >>> state
> > >> >> >>>>> > > > >> > >> >> >>> > >> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes
> > space
> > >> that
> > >> >> >> can
> > >> >> >>>> will be
> > >> >> >>>>> > > > naturally
> > >> >> >>>>> > > > >> > >> >> interpreted
> > >> >> >>>>> > > > >> > >> >> >>> as
> > >> >> >>>>> > > > >> > >> >> >>> > an
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing
> > is later
> > >> >> >>>> wanted you can
> > >> >> >>>>> > > easily
> > >> >> >>>>> > > > >> split
> > >> >> >>>>> > > > >> > >> this
> > >> >> >>>>> > > > >> > >> >> >>> into
> > >> >> >>>>> > > > >> > >> >> >>> > two
> > >> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from
> > the wire
> > >> >> >> protocol
> > >> >> >>>>> > > implementation
> > >> >> >>>>> > > > >> this
> > >> >> >>>>> > > > >> > >> makes
> > >> >> >>>>> > > > >> > >> >> no
> > >> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t
> > believe. Is
> > >> >> this
> > >> >> >>>>> > reasonable to
> > >> >> >>>>> > > > all?
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
> > therefor
> > >> >> happy
> > >> >> >>>> we keep
> > >> >> >>>>> > with 32
> > >> >> >>>>> > > > >> bits.
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016,
> 20:34, "
> > >> >> >>>>> > ignacio.solis@gmail.com on
> > >> >> >>>>> > > > >> behalf
> > >> >> >>>>> > > > >> > of
> > >> >> >>>>> > > > >> > >> >> >>> Ignacio
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
> > >> ignacio.solis@gmail.com
> > >> >> on
> > >> >> >>>> behalf of
> > >> >> >>>>> > > > >> > isolis@igso.net
> > >> >> >>>>> > > > >> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
> > value as
> > >> >> byte[]
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
> > >> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> > >> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String -
> > Reluctant
> > >> maybe
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe the
> > header
> > >> >> system
> > >> >> >>>> should
> > >> >> >>>>> > take a
> > >> >> >>>>> > > > single
> > >> >> >>>>> > > > >> > >> int. I
> > >> >> >>>>> > > > >> > >> >> >>> think
> > >> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
> > >> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if
> > you want
> > >> to
> > >> >> >>>> interpret
> > >> >> >>>>> > this as
> > >> >> >>>>> > > to
> > >> >> >>>>> > > > >> 16bit
> > >> >> >>>>> > > > >> > >> >> numbers
> > >> >> >>>>> > > > >> > >> >> >>> in
> > >> >> >>>>> > > > >> > >> >> >>> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > layer
> > >> >> >>>>> > > > >> > >> >> >>> > >> > above go right
> > ahead. If
> > >> >> >>>> somebody wants
> > >> >> >>>>> > to
> > >> >> >>>>> > > argue
> > >> >> >>>>> > > > >> for
> > >> >> >>>>> > > > >> > 16
> > >> >> >>>>> > > > >> > >> >> bits
> > >> >> >>>>> > > > >> > >> >> >>> or
> > >> >> >>>>> > > > >> > >> >> >>> > 64
> > >> >> >>>>> > > > >> > >> >> >>> > >> > bits of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > header key space
> I
> > would
> > >> >> >> listen.
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the key
> > space
> > >> into
> > >> >> >>>> sub_key_1 and
> > >> >> >>>>> > > > sub_key_2
> > >> >> >>>>> > > > >> > >> makes no
> > >> >> >>>>> > > > >> > >> >> >>> > sense to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > me at
> > >> >> >>>>> > > > >> > >> >> >>> > >> > this layer. Are
> > we going
> > >> to
> > >> >> >>>> start
> > >> >> >>>>> > providing
> > >> >> >>>>> > > > APIs to
> > >> >> >>>>> > > > >> > get
> > >> >> >>>>> > > > >> > >> all
> > >> >> >>>>> > > > >> > >> >> >>> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or
> all
> > the
> > >> >> >>>> sub_key_2s? If
> > >> >> >>>>> > there is
> > >> >> >>>>> > > > no
> > >> >> >>>>> > > > >> > >> >> >>> distinguishing
> > >> >> >>>>> > > > >> > >> >> >>> > >> > functions
> > >> >> >>>>> > > > >> > >> >> >>> > >> > that are applied
> > to each
> > >> one
> > >> >> >>>> then they
> > >> >> >>>>> > should
> > >> >> >>>>> > > be
> > >> >> >>>>> > > > a
> > >> >> >>>>> > > > >> > single
> > >> >> >>>>> > > > >> > >> >> >>> value.
> > >> >> >>>>> > > > >> > >> >> >>> > At
> > >> >> >>>>> > > > >> > >> >> >>> > >> > this
> > >> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're
> > doing is
> > >> >> >>>> equality.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > If the above
> layer
> > wants
> > >> to
> > >> >> >>>> interpret
> > >> >> >>>>> > this as
> > >> >> >>>>> > > 2,
> > >> >> >>>>> > > > 3
> > >> >> >>>>> > > > >> or
> > >> >> >>>>> > > > >> > >> more
> > >> >> >>>>> > > > >> > >> >> >>> values
> > >> >> >>>>> > > > >> > >> >> >>> > >> > that's a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > different
> > question. I
> > >> >> >>>> personally think
> > >> >> >>>>> > it's
> > >> >> >>>>> > > all
> > >> >> >>>>> > > > one
> > >> >> >>>>> > > > >> > >> >> keyspace
> > >> >> >>>>> > > > >> > >> >> >>> > that is
> > >> >> >>>>> > > > >> > >> >> >>> > >> > getting assigned
> > using
> > >> some
> > >> >> >>>> structure,
> > >> >> >>>>> > but if
> > >> >> >>>>> > > you
> > >> >> >>>>> > > > >> > want to
> > >> >> >>>>> > > > >> > >> >> >>> > sub-assign
> > >> >> >>>>> > > > >> > >> >> >>> > >> > parts
> > >> >> >>>>> > > > >> > >> >> >>> > >> > of it then that's
> > fine.
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > The same
> discussion
> > >> applies
> > >> >> to
> > >> >> >>>> strings.
> > >> >> >>>>> > If
> > >> >> >>>>> > > > somebody
> > >> >> >>>>> > > > >> > >> argued
> > >> >> >>>>> > > > >> > >> >> for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > strings,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > would we be
> > arguing to
> > >> >> divide
> > >> >> >> the
> > >> >> >>>>> > strings with
> > >> >> >>>>> > > > dots
> > >> >> >>>>> > > > >> > ('.')
> > >> >> >>>>> > > > >> > >> >> as a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we want
> them
> > to give
> > >> >> us
> > >> >> >> the
> > >> >> >>>>> > different
> > >> >> >>>>> > > name
> > >> >> >>>>> > > > >> > segments
> > >> >> >>>>> > > > >> > >> >> >>> > separately?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Would we be
> > performing any
> > >> >> >>>> actions on
> > >> >> >>>>> > this key
> > >> >> >>>>> > > > other
> > >> >> >>>>> > > > >> > than
> > >> >> >>>>> > > > >> > >> >> >>> > matching?
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
> > 2016 at
> > >> 9:30
> > >> >> >> AM,
> > >> >> >>>> Michael
> > >> >> >>>>> > > Pearce <
> > >> >> >>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> > concerns
> > >> on
> > >> >> 1
> > >> >> >>>> and 2
> > >> >> >>>>> > still?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > @all
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > To get this
> > moving
> > >> along a
> > >> >> >> bit
> > >> >> >>>> more
> > >> >> >>>>> > I'd also
> > >> >> >>>>> > > > like
> > >> >> >>>>> > > > >> to
> > >> >> >>>>> > > > >> > >> ask
> > >> >> >>>>> > > > >> > >> >> to
> > >> >> >>>>> > > > >> > >> >> >>> get
> > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the below last
> > points:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe
> > we're all
> > >> >> >> roughly
> > >> >> >>>> happy
> > >> >> >>>>> > with the
> > >> >> >>>>> > > > >> header
> > >> >> >>>>> > > > >> > >> value
> > >> >> >>>>> > > > >> > >> >> >>> > being a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> > consensus
> > >> has
> > >> >> >>>> been for an
> > >> >> >>>>> > > > namespace
> > >> >> >>>>> > > > >> > based
> > >> >> >>>>> > > > >> > >> int
> > >> >> >>>>> > > > >> > >> >> >>> > approach
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for
> > the key.
> > >> Any
> > >> >> >>>> objections
> > >> >> >>>>> > if this
> > >> >> >>>>> > > > is
> > >> >> >>>>> > > > >> > what
> > >> >> >>>>> > > > >> > >> we
> > >> >> >>>>> > > > >> > >> >> go
> > >> >> >>>>> > > > >> > >> >> >>> > with?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have
> if
> > >> >> assumption
> > >> >> >> in
> > >> >> >>>> (4) is
> > >> >> >>>>> > > correct,
> > >> >> >>>>> > > > >> > >> {int,int}
> > >> >> >>>>> > > > >> > >> >> >>> keys.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > Should both
> > int's be
> > >> >> int16 or
> > >> >> >>>> int32?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them
> > being
> > >> int16(2
> > >> >> >>>> bytes) as
> > >> >> >>>>> > combined
> > >> >> >>>>> > > > is
> > >> >> >>>>> > > > >> > space
> > >> >> >>>>> > > > >> > >> of
> > >> >> >>>>> > > > >> > >> >> >>> > 4bytes as
> > >> >> >>>>> > > > >> > >> >> >>> > >> > per
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > original and
> > gives
> > >> plenty
> > >> >> of
> > >> >> >>>>> > combinations for
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> >> >>> foreseeable,
> > >> >> >>>>> > > > >> > >> >> >>> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead
> > small.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any
> > benefit in
> > >> >> >>>> another kip
> > >> >> >>>>> > call to
> > >> >> >>>>> > > > >> discuss
> > >> >> >>>>> > > > >> > >> >> these at
> > >> >> >>>>> > > > >> > >> >> >>> > all?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >> ______________________________
> > >> >> >>>>> > __________
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > From: K
> Burstev <
> > >> >> >>>> k.burstev@yandex.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> > November
> > >> 18,
> > >> >> >> 2016
> > >> >> >>>>> > 7:07:07 AM
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > To:
> > >> dev@kafka.apache.org
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> > [DISCUSS]
> > >> >> KIP-82
> > >> >> >>>> - Add
> > >> >> >>>>> > Record
> > >> >> >>>>> > > > Headers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > For what it is
> > worth
> > >> also
> > >> >> i
> > >> >> >>>> agree. As
> > >> >> >>>>> > a user:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
> > Headers are
> > >> >> >>>> worthwhile
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes -
> > Headers should
> > >> >> be a
> > >> >> >>>> top level
> > >> >> >>>>> > > option
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
> > 21:15,
> > >> >> "Ignacio
> > >> >> >>>> Solis" <
> > >> >> >>>>> > > > >> isolis@igso.net
> > >> >> >>>>> > > > >> > >:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
> > Headers are
> > >> >> >>>> worthwhile
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
> > Headers
> > >> should
> > >> >> be
> > >> >> >> a
> > >> >> >>>> top
> > >> >> >>>>> > level
> > >> >> >>>>> > > option
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov
> > 14, 2016
> > >> at
> > >> >> >> 9:16
> > >> >> >>>> AM,
> > >> >> >>>>> > Michael
> > >> >> >>>>> > > > Pearce
> > >> >> >>>>> > > > >> <
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> Michael.Pearce@ig.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
> > >> >> details/examples
> > >> >> >>>> the
> > >> >> >>>>> > original
> > >> >> >>>>> > > > proposal
> > >> >> >>>>> > > > >> > for
> > >> >> >>>>> > > > >> > >> key
> > >> >> >>>>> > > > >> > >> >> >>> > spacing
> > >> >> >>>>> > > > >> > >> >> >>> > >> ,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > not
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> new
> > mentioned as per
> > >> >> >>>> discussion
> > >> >> >>>>> > namespace
> > >> >> >>>>> > > > >> idea.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will
> need
> > to
> > >> update
> > >> >> >> the
> > >> >> >>>> kip,
> > >> >> >>>>> > when we
> > >> >> >>>>> > > get
> > >> >> >>>>> > > > >> > >> agreement
> > >> >> >>>>> > > > >> > >> >> >>> this
> > >> >> >>>>> > > > >> > >> >> >>> > is a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > better
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach
> > (which
> > >> seems
> > >> >> to
> > >> >> >>>> be the
> > >> >> >>>>> > case if I
> > >> >> >>>>> > > > have
> > >> >> >>>>> > > > >> > >> >> understood
> > >> >> >>>>> > > > >> > >> >> >>> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > general
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling in
> > the
> > >> >> >>>> conversation)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the
> > variable
> > >> ints,
> > >> >> at
> > >> >> >>>> very
> > >> >> >>>>> > early stage
> > >> >> >>>>> > > > we
> > >> >> >>>>> > > > >> did
> > >> >> >>>>> > > > >> > >> think
> > >> >> >>>>> > > > >> > >> >> >>> about
> > >> >> >>>>> > > > >> > >> >> >>> > >> > this. I
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added
> > complexity
> > >> >> for
> > >> >> >>>> the
> > >> >> >>>>> > saving isn't
> > >> >> >>>>> > > > >> worth
> > >> >> >>>>> > > > >> > it.
> > >> >> >>>>> > > > >> > >> >> I'd
> > >> >> >>>>> > > > >> > >> >> >>> > rather
> > >> >> >>>>> > > > >> > >> >> >>> > >> go
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want to
> > reduce
> > >> >> >>>> overheads and
> > >> >> >>>>> > size
> > >> >> >>>>> > > int16
> > >> >> >>>>> > > > >> > (2bytes)
> > >> >> >>>>> > > > >> > >> >> keys
> > >> >> >>>>> > > > >> > >> >> >>> as
> > >> >> >>>>> > > > >> > >> >> >>> > it
> > >> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the note
> > of no
> > >> >> >> headers,
> > >> >> >>>> there
> > >> >> >>>>> > is as
> > >> >> >>>>> > > per
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > kip
> > >> >> >>>>> > > > >> > >> as
> > >> >> >>>>> > > > >> > >> >> we
> > >> >> >>>>> > > > >> > >> >> >>> > use an
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to
> > denote if
> > >> >> headers
> > >> >> >>>> are
> > >> >> >>>>> > present or
> > >> >> >>>>> > > > not as
> > >> >> >>>>> > > > >> > such
> > >> >> >>>>> > > > >> > >> >> >>> > provides a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > zero
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
> > currently
> > >> if
> > >> >> >>>> headers are
> > >> >> >>>>> > not
> > >> >> >>>>> > > used.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think as
> > radai
> > >> >> mentions
> > >> >> >>>> would be
> > >> >> >>>>> > good
> > >> >> >>>>> > > > first
> > >> >> >>>>> > > > >> > if we
> > >> >> >>>>> > > > >> > >> >> can
> > >> >> >>>>> > > > >> > >> >> >>> get
> > >> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > do
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now have
> > general
> > >> >> >>>> consensus that
> > >> >> >>>>> > (1)
> > >> >> >>>>> > > > headers
> > >> >> >>>>> > > > >> > are
> > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> > >> >> >>>>> > > > >> > >> >> >>> > >> and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2) we
> > want it
> > >> as
> > >> >> a
> > >> >> >>>> top level
> > >> >> >>>>> > entity.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to
> > state the
> > >> >> >> obvious i
> > >> >> >>>>> > believe (1)
> > >> >> >>>>> > > > >> headers
> > >> >> >>>>> > > > >> > are
> > >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> > >> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as a
> > top level
> > >> >> >>>> entity.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>> ______________________________
> > >> >> >>>>> > __________
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
> > Hoover <
> > >> >> >>>>> > > roger.hoover@gmail.com
> > >> >> >>>>> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent:
> > Wednesday,
> > >> >> November
> > >> >> >>>> 9, 2016
> > >> >> >>>>> > 9:10:47
> > >> >> >>>>> > > > PM
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
> > >> >> dev@kafka.apache.org
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject:
> Re:
> > >> [DISCUSS]
> > >> >> >>>> KIP-82 - Add
> > >> >> >>>>> > > Record
> > >> >> >>>>> > > > >> > Headers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
> > going a
> > >> >> little
> > >> >> >>>> in the
> > >> >> >>>>> > weeds but
> > >> >> >>>>> > > > >> thanks
> > >> >> >>>>> > > > >> > >> for
> > >> >> >>>>> > > > >> > >> >> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> replies
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed that
> > a prefix
> > >> >> and
> > >> >> >>>> {int,
> > >> >> >>>>> > int} can
> > >> >> >>>>> > > be
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> same.
> > >> >> >>>>> > > > >> > >> >> It
> > >> >> >>>>> > > > >> > >> >> >>> > doesn't
> > >> >> >>>>> > > > >> > >> >> >>> > >> > look
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > like
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's what
> > the KIP
> > >> is
> > >> >> >>>> saying the
> > >> >> >>>>> > "Open"
> > >> >> >>>>> > > > >> > section.
> > >> >> >>>>> > > > >> > >> The
> > >> >> >>>>> > > > >> > >> >> >>> > example
> > >> >> >>>>> > > > >> > >> >> >>> > >> > shows
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New
> > Relic and
> > >> >> 210002
> > >> >> >>>> for App
> > >> >> >>>>> > Dynamics
> > >> >> >>>>> > > > >> > implying
> > >> >> >>>>> > > > >> > >> >> that
> > >> >> >>>>> > > > >> > >> >> >>> the
> > >> >> >>>>> > > > >> > >> >> >>> > New
> > >> >> >>>>> > > > >> > >> >> >>> > >> > Relic
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> organization
> > will
> > >> have
> > >> >> >>>> only a
> > >> >> >>>>> > single
> > >> >> >>>>> > > > header id
> > >> >> >>>>> > > > >> > to
> > >> >> >>>>> > > > >> > >> work
> > >> >> >>>>> > > > >> > >> >> >>> > with. Or
> > >> >> >>>>> > > > >> > >> >> >>> > >> > is
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix?
> > The main
> > >> >> point
> > >> >> >>>> of a
> > >> >> >>>>> > namespace
> > >> >> >>>>> > > or
> > >> >> >>>>> > > > >> > prefix
> > >> >> >>>>> > > > >> > >> is
> > >> >> >>>>> > > > >> > >> >> to
> > >> >> >>>>> > > > >> > >> >> >>> > reduce
> > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead of
> > config
> > >> >> >> mapping
> > >> >> >>>> or
> > >> >> >>>>> > > registration
> > >> >> >>>>> > > > >> > >> depending
> > >> >> >>>>> > > > >> > >> >> on
> > >> >> >>>>> > > > >> > >> >> >>> how
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > namespaces/prefixes
> > >> >> are
> > >> >> >>>> managed.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would love
> > to hear
> > >> >> more
> > >> >> >>>> feedback
> > >> >> >>>>> > on the
> > >> >> >>>>> > > > >> > >> higher-level
> > >> >> >>>>> > > > >> > >> >> >>> > questions
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > though...
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov
> > 9, 2016
> > >> at
> > >> >> >>>> 11:38 AM,
> > >> >> >>>>> > radai <
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > radai.rosenblatt@gmail.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think
> > this
> > >> >> discussion
> > >> >> >>>> is
> > >> >> >>>>> > getting a
> > >> >> >>>>> > > bit
> > >> >> >>>>> > > > >> into
> > >> >> >>>>> > > > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> weeds on
> > >> >> >>>>> > > > >> > >> >> >>> > >> > technical
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > implementation
> > >> >> details.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek
> > to step
> > >> >> back a
> > >> >> >>>> minute
> > >> >> >>>>> > and try
> > >> >> >>>>> > > > and
> > >> >> >>>>> > > > >> > >> establish
> > >> >> >>>>> > > > >> > >> >> >>> > where we
> > >> >> >>>>> > > > >> > >> >> >>> > >> > are in
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger
> > picture:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> (re-wording
> > >> nacho's
> > >> >> >> last
> > >> >> >>>>> > paragraph)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we
> > all in
> > >> >> >>>> agreement that
> > >> >> >>>>> > headers
> > >> >> >>>>> > > > are
> > >> >> >>>>> > > > >> a
> > >> >> >>>>> > > > >> > >> >> >>> worthwhile
> > >> >> >>>>> > > > >> > >> >> >>> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > useful
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > addition
> > to have?
> > >> >> this
> > >> >> >>>> was
> > >> >> >>>>> > contested
> > >> >> >>>>> > > > early
> > >> >> >>>>> > > > >> on
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we
> > all in
> > >> >> >>>> agreement on
> > >> >> >>>>> > headers
> > >> >> >>>>> > > as
> > >> >> >>>>> > > > top
> > >> >> >>>>> > > > >> > >> level
> > >> >> >>>>> > > > >> > >> >> >>> entity
> > >> >> >>>>> > > > >> > >> >> >>> > vs
> > >> >> >>>>> > > > >> > >> >> >>> > >> > headers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > squirreled-away in
> > >> >> V?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if there
> > are still
> > >> >> >>>> concerns
> > >> >> >>>>> > around
> > >> >> >>>>> > > these
> > >> >> >>>>> > > > #2
> > >> >> >>>>> > > > >> > >> points
> > >> >> >>>>> > > > >> > >> >> >>> (#jay?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and now
> > back to
> > >> our
> > >> >> >>>> normal
> > >> >> >>>>> > programming
> > >> >> >>>>> > > > ...)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints
> > are nice.
> > >> >> >> having
> > >> >> >>>> said
> > >> >> >>>>> > that, its
> > >> >> >>>>> > > > >> adding
> > >> >> >>>>> > > > >> > >> >> >>> complexity
> > >> >> >>>>> > > > >> > >> >> >>> > >> (see
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>> https://github.com/addthis/
> > >> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>> main/java/com/clearspring/
> > >> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
> > google
> > >> >> result)
> > >> >> >>>> and would
> > >> >> >>>>> > require
> > >> >> >>>>> > > > >> anyone
> > >> >> >>>>> > > > >> > >> >> writing
> > >> >> >>>>> > > > >> > >> >> >>> > other
> > >> >> >>>>> > > > >> > >> >> >>> > >> > clients
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python?
> > Go? Bash?
> > >> >> ;-) )
> > >> >> >>>> to
> > >> >> >>>>> > > get/implement
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > >> same,
> > >> >> >>>>> > > > >> > >> >> and
> > >> >> >>>>> > > > >> > >> >> >>> for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > relatively
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little
> > gain (int
> > >> vs
> > >> >> >>>> string is
> > >> >> >>>>> > order of
> > >> >> >>>>> > > > >> > magnitude,
> > >> >> >>>>> > > > >> > >> >> this
> > >> >> >>>>> > > > >> > >> >> >>> > isnt).
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int
> > namespacing vs
> > >> >> >> {int,
> > >> >> >>>> int}
> > >> >> >>>>> > > namespacing
> > >> >> >>>>> > > > >> are
> > >> >> >>>>> > > > >> > >> >> basically
> > >> >> >>>>> > > > >> > >> >> >>> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > same
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre
> just
> > >> >> namespacing
> > >> >> >>>> an int64
> > >> >> >>>>> > and
> > >> >> >>>>> > > > giving
> > >> >> >>>>> > > > >> > people
> > >> >> >>>>> > > > >> > >> >> while
> > >> >> >>>>> > > > >> > >> >> >>> > 2^32
> > >> >> >>>>> > > > >> > >> >> >>> > >> > ranges
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > at a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time. the
> > part i
> > >> >> like
> > >> >> >>>> about this
> > >> >> >>>>> > is
> > >> >> >>>>> > > > letting
> > >> >> >>>>> > > > >> > >> people
> > >> >> >>>>> > > > >> > >> >> >>> have a
> > >> >> >>>>> > > > >> > >> >> >>> > >> large
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers
> > with one
> > >> >> >>>> registration so
> > >> >> >>>>> > they
> > >> >> >>>>> > > > dont
> > >> >> >>>>> > > > >> > have
> > >> >> >>>>> > > > >> > >> to
> > >> >> >>>>> > > > >> > >> >> come
> > >> >> >>>>> > > > >> > >> >> >>> > back
> > >> >> >>>>> > > > >> > >> >> >>> > >> > for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > every
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
> > >> plugin/header
> > >> >> >>>> they want to
> > >> >> >>>>> > > > "reserve".
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
> > Nov 9,
> > >> 2016
> > >> >> at
> > >> >> >>>> 11:01 AM,
> > >> >> >>>>> > Roger
> > >> >> >>>>> > > > >> Hoover
> > >> >> >>>>> > > > >> > <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > roger.hoover@gmail.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since
> > some of
> > >> the
> > >> >> >>>> debate has
> > >> >> >>>>> > been
> > >> >> >>>>> > > about
> > >> >> >>>>> > > > >> > >> overhead +
> > >> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > wondering if we
> > >> >> have
> > >> >> >>>>> > considered a
> > >> >> >>>>> > > > varint
> > >> >> >>>>> > > > >> > >> encoding
> > >> >> >>>>> > > > >> > >> >> (
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>> https://developers.google.com/
> > >> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> encoding#varints)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the
> > header
> > >> length
> > >> >> >>>> field (int32
> > >> >> >>>>> > in the
> > >> >> >>>>> > > > >> > proposal)
> > >> >> >>>>> > > > >> > >> >> and
> > >> >> >>>>> > > > >> > >> >> >>> for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > header
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't
> use
> > >> headers,
> > >> >> >> the
> > >> >> >>>>> > overhead would
> > >> >> >>>>> > > > be a
> > >> >> >>>>> > > > >> > >> single
> > >> >> >>>>> > > > >> > >> >> >>> byte
> > >> >> >>>>> > > > >> > >> >> >>> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > for each
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id <
> 128
> > would
> > >> >> also
> > >> >> >>>> need only a
> > >> >> >>>>> > > single
> > >> >> >>>>> > > > >> byte?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed,
> > Nov 9,
> > >> >> 2016
> > >> >> >> at
> > >> >> >>>> 6:43 AM,
> > >> >> >>>>> > > radai <
> > >> >> >>>>> > > > >> > >> >> >>> > >> >
> > radai.rosenblatt@gmail.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > @magnus - and
> > >> >> very
> > >> >> >>>> dangerous
> > >> >> >>>>> > (youre
> > >> >> >>>>> > > > >> > >> essentially
> > >> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > executing
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > arbitrary code
> > >> >> off
> > >> >> >>>> the
> > >> >> >>>>> > internet on
> > >> >> >>>>> > > > your
> > >> >> >>>>> > > > >> > >> servers
> > >> >> >>>>> > > > >> > >> >> ...
> > >> >> >>>>> > > > >> > >> >> >>> > bad
> > >> >> >>>>> > > > >> > >> >> >>> > >> > idea
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > without
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > sandbox, even
> > >> >> with)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as
> for
> > it
> > >> being
> > >> >> a
> > >> >> >>>> purely
> > >> >> >>>>> > > > administrative
> > >> >> >>>>> > > > >> > task
> > >> >> >>>>> > > > >> > >> - i
> > >> >> >>>>> > > > >> > >> >> >>> > >> disagree.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i
> wish
> > it
> > >> would,
> > >> >> >>>> really,
> > >> >> >>>>> > because
> > >> >> >>>>> > > > then my
> > >> >> >>>>> > > > >> > >> earlier
> > >> >> >>>>> > > > >> > >> >> >>> > point on
> > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> complexity
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> > remapping
> > >> >> >>>> process would
> > >> >> >>>>> > be
> > >> >> >>>>> > > > invalid,
> > >> >> >>>>> > > > >> > but
> > >> >> >>>>> > > > >> > >> at
> > >> >> >>>>> > > > >> > >> >> >>> > linkedin,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > example,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (the
> > team im
> > >> in)
> > >> >> >> run
> > >> >> >>>> kafka
> > >> >> >>>>> > as a
> > >> >> >>>>> > > > service.
> > >> >> >>>>> > > > >> > we
> > >> >> >>>>> > > > >> > >> dont
> > >> >> >>>>> > > > >> > >> >> >>> > really
> > >> >> >>>>> > > > >> > >> >> >>> > >> > know
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > what our
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > (developing
> > >> >> >>>> applications
> > >> >> >>>>> > that use
> > >> >> >>>>> > > > kafka)
> > >> >> >>>>> > > > >> > are
> > >> >> >>>>> > > > >> > >> up
> > >> >> >>>>> > > > >> > >> >> to
> > >> >> >>>>> > > > >> > >> >> >>> at
> > >> >> >>>>> > > > >> > >> >> >>> > any
> > >> >> >>>>> > > > >> > >> >> >>> > >> > given
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is
> very
> > >> possible
> > >> >> >>>> (given the
> > >> >> >>>>> > > > existance of
> > >> >> >>>>> > > > >> > >> headers
> > >> >> >>>>> > > > >> > >> >> >>> and a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > ecosystem) for
> > >> >> some
> > >> >> >>>>> > application to
> > >> >> >>>>> > > > >> "equip"
> > >> >> >>>>> > > > >> > >> their
> > >> >> >>>>> > > > >> > >> >> >>> > >> producers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > consumers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > with
> > the
> > >> >> required
> > >> >> >>>> plugin
> > >> >> >>>>> > without us
> > >> >> >>>>> > > > >> > knowing.
> > >> >> >>>>> > > > >> > >> i
> > >> >> >>>>> > > > >> > >> >> dont
> > >> >> >>>>> > > > >> > >> >> >>> > mean
> > >> >> >>>>> > > > >> > >> >> >>> > >> > to imply
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > bad,
> i
> > just
> > >> >> want to
> > >> >> >>>> make the
> > >> >> >>>>> > point
> > >> >> >>>>> > > > that
> > >> >> >>>>> > > > >> > its
> > >> >> >>>>> > > > >> > >> not
> > >> >> >>>>> > > > >> > >> >> as
> > >> >> >>>>> > > > >> > >> >> >>> > simple
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
> > across a
> > >> >> >>>> large-enough
> > >> >> >>>>> > > > organization.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On
> > Wed, Nov 9,
> > >> >> 2016
> > >> >> >>>> at 6:17
> > >> >> >>>>> > AM,
> > >> >> >>>>> > > > Magnus
> > >> >> >>>>> > > > >> > >> Edenhill
> > >> >> >>>>> > > > >> > >> >> <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > magnus@edenhill.se>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
> > think
> > >> there
> > >> >> is
> > >> >> >> a
> > >> >> >>>> piece
> > >> >> >>>>> > missing
> > >> >> >>>>> > > in
> > >> >> >>>>> > > > >> the
> > >> >> >>>>> > > > >> > >> >> Strings
> > >> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > where
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> pro-Stringers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > reason that
> > >> by
> > >> >> >>>> providing
> > >> >> >>>>> > unique
> > >> >> >>>>> > > > string
> > >> >> >>>>> > > > >> > >> >> >>> identifiers
> > >> >> >>>>> > > > >> > >> >> >>> > for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > each
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > everything
> > >> >> will
> > >> >> >>>> just
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > magically
> > >> work
> > >> >> >> for
> > >> >> >>>> all
> > >> >> >>>>> > parts of
> > >> >> >>>>> > > the
> > >> >> >>>>> > > > >> > stream
> > >> >> >>>>> > > > >> > >> >> >>> pipeline.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > But
> > the
> > >> >> strings
> > >> >> >>>> dont mean
> > >> >> >>>>> > > anything
> > >> >> >>>>> > > > by
> > >> >> >>>>> > > > >> > >> >> themselves,
> > >> >> >>>>> > > > >> > >> >> >>> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > while we
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > probably
> > >> >> envision
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> some
> > auto
> > >> >> plugin
> > >> >> >>>> loader
> > >> >> >>>>> > that
> > >> >> >>>>> > > > >> downloads,
> > >> >> >>>>> > > > >> > >> >> compiles,
> > >> >> >>>>> > > > >> > >> >> >>> > links
> > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > runs
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > on-demand
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > as
> > soon as
> > >> >> >> they're
> > >> >> >>>> seen by
> > >> >> >>>>> > a
> > >> >> >>>>> > > > >> consumer, I
> > >> >> >>>>> > > > >> > >> dont
> > >> >> >>>>> > > > >> > >> >> >>> really
> > >> >> >>>>> > > > >> > >> >> >>> > >> see
> > >> >> >>>>> > > > >> > >> >> >>> > >> > a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > something
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
> > dynamic
> > >> >> (and
> > >> >> >>>> fragile) in
> > >> >> >>>>> > > > practice.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > In
> > the real
> > >> >> world
> > >> >> >>>> an
> > >> >> >>>>> > application
> > >> >> >>>>> > > > will
> > >> >> >>>>> > > > >> be
> > >> >> >>>>> > > > >> > >> >> >>> configured
> > >> >> >>>>> > > > >> > >> >> >>> > >> with
> > >> >> >>>>> > > > >> > >> >> >>> > >> > a set
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > to
> > either
> > >> add
> > >> >> >>>> (producer)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > or
> > read
> > >> >> >> (consumer)
> > >> >> >>>> headers.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> This
> > is an
> > >> >> >>>> administrative
> > >> >> >>>>> > task
> > >> >> >>>>> > > > based
> > >> >> >>>>> > > > >> on
> > >> >> >>>>> > > > >> > >> what
> > >> >> >>>>> > > > >> > >> >> >>> > features a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > client
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> needs/provides
> > >> >> >> and
> > >> >> >>>> results
> > >> >> >>>>> > in
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> some
> > sort of
> > >> >> >>>> configuration
> > >> >> >>>>> > to
> > >> >> >>>>> > > > enable
> > >> >> >>>>> > > > >> and
> > >> >> >>>>> > > > >> > >> >> >>> configure
> > >> >> >>>>> > > > >> > >> >> >>> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > desired
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > Since this
> > >> >> needs
> > >> >> >>>> to be kept
> > >> >> >>>>> > > > somewhat
> > >> >> >>>>> > > > >> in
> > >> >> >>>>> > > > >> > >> sync
> > >> >> >>>>> > > > >> > >> >> >>> across
> > >> >> >>>>> > > > >> > >> >> >>> > an
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> (there
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > is
> > no point
> > >> in
> > >> >> >>>> having
> > >> >> >>>>> > producers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
> > headers
> > >> no
> > >> >> >>>> consumers
> > >> >> >>>>> > will
> > >> >> >>>>> > > read,
> > >> >> >>>>> > > > >> and
> > >> >> >>>>> > > > >> > >> vice
> > >> >> >>>>> > > > >> > >> >> >>> versa),
> > >> >> >>>>> > > > >> > >> >> >>> > >> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > added
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > complexity
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
> > assigning
> > >> >> an
> > >> >> >> id
> > >> >> >>>>> > namespace
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > for
> > each
> > >> >> plugin
> > >> >> >> as
> > >> >> >>>> it is
> > >> >> >>>>> > being
> > >> >> >>>>> > > > >> > configured
> > >> >> >>>>> > > > >> > >> >> should
> > >> >> >>>>> > > > >> > >> >> >>> be
> > >> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > /Magnus
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > 2016-11-09
> > >> >> 13:06
> > >> >> >>>> GMT+01:00
> > >> >> >>>>> > > Michael
> > >> >> >>>>> > > > >> > Pearce <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > >
> > Michael.Pearce@ig.com>:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > Just
> > >> >> >>>> following/catching
> > >> >> >>>>> > up on
> > >> >> >>>>> > > > what
> > >> >> >>>>> > > > >> > seems
> > >> >> >>>>> > > > >> > >> to
> > >> >> >>>>> > > > >> > >> >> be
> > >> >> >>>>> > > > >> > >> >> >>> an
> > >> >> >>>>> > > > >> > >> >> >>> > >> > active
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > @Radai
> > >> >> sorry if
> > >> >> >>>> it may
> > >> >> >>>>> > seem
> > >> >> >>>>> > > > obvious
> > >> >> >>>>> > > > >> > but
> > >> >> >>>>> > > > >> > >> what
> > >> >> >>>>> > > > >> > >> >> >>> does
> > >> >> >>>>> > > > >> > >> >> >>> > MD
> > >> >> >>>>> > > > >> > >> >> >>> > >> > stand
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > for?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> My
> > take on
> > >> >> >>>> String vs Int:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> > will
> > >> state
> > >> >> >>>> first I am
> > >> >> >>>>> > pro Int
> > >> >> >>>>> > > > (16
> > >> >> >>>>> > > > >> or
> > >> >> >>>>> > > > >> > >> 32).
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> > do
> > >> though
> > >> >> >>>> playing
> > >> >> >>>>> > devils
> > >> >> >>>>> > > > advocate
> > >> >> >>>>> > > > >> > see a
> > >> >> >>>>> > > > >> > >> >> big
> > >> >> >>>>> > > > >> > >> >> >>> plus
> > >> >> >>>>> > > > >> > >> >> >>> > >> > with the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > argument
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > String
> > >> keys,
> > >> >> >>>> this is
> > >> >> >>>>> > around
> > >> >> >>>>> > > > >> > integrating
> > >> >> >>>>> > > > >> > >> >> into an
> > >> >> >>>>> > > > >> > >> >> >>> > >> > existing
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> eco-system.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> As
> > many
> > >> >> other
> > >> >> >>>> systems use
> > >> >> >>>>> > > String
> > >> >> >>>>> > > > >> based
> > >> >> >>>>> > > > >> > >> >> headers
> > >> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > it
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > much
> > >> easier
> > >> >> for
> > >> >> >>>> these to
> > >> >> >>>>> > be
> > >> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
> > >> >> >>>>> > > > >> > >> >> >>> > >> > into.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > How with
> > >> Int
> > >> >> >>>> based
> > >> >> >>>>> > headers
> > >> >> >>>>> > > could
> > >> >> >>>>> > > > we
> > >> >> >>>>> > > > >> > >> provide
> > >> >> >>>>> > > > >> > >> >> a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> integration
> > >> >> >>>> simple /
> > >> >> >>>>> > easy with
> > >> >> >>>>> > > > >> > transition
> > >> >> >>>>> > > > >> > >> >> flows
> > >> >> >>>>> > > > >> > >> >> >>> > over
> > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> > tough
> > >> luck
> > >> >> >>>> buddy
> > >> >> >>>>> > you're on
> > >> >> >>>>> > > your
> > >> >> >>>>> > > > >> own
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> > simply
> > >> >> hash
> > >> >> >>>> the string
> > >> >> >>>>> > into
> > >> >> >>>>> > > int
> > >> >> >>>>> > > > >> code
> > >> >> >>>>> > > > >> > >> and
> > >> >> >>>>> > > > >> > >> >> hope
> > >> >> >>>>> > > > >> > >> >> >>> > for
> > >> >> >>>>> > > > >> > >> >> >>> > >> no
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > convert
> > >> back
> > >> >> >>>> though?)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> > http2
> > >> >> style
> > >> >> >> as
> > >> >> >>>>> > mentioned by
> > >> >> >>>>> > > > nacho.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > cheers,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > Mike
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > ______________________________
> > >> >> >>>>> > > > >> > __________
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > From:
> > >> radai
> > >> >> <
> > >> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > Sent:
> > >> >> >> Wednesday,
> > >> >> >>>>> > November 9,
> > >> >> >>>>> > > 2016
> > >> >> >>>>> > > > >> > 8:12 AM
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> To:
> > >> >> >>>> dev@kafka.apache.org
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > Subject:
> > >> Re:
> > >> >> >>>> [DISCUSS]
> > >> >> >>>>> > KIP-82 -
> > >> >> >>>>> > > > Add
> > >> >> >>>>> > > > >> > >> Record
> > >> >> >>>>> > > > >> > >> >> >>> Headers
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > thinking
> > >> >> about
> > >> >> >>>> it some
> > >> >> >>>>> > more,
> > >> >> >>>>> > > the
> > >> >> >>>>> > > > >> best
> > >> >> >>>>> > > > >> > >> way to
> > >> >> >>>>> > > > >> > >> >> >>> > transmit
> > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > remapping
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > data to
> > >> >> >>>> consumers would
> > >> >> >>>>> > be to
> > >> >> >>>>> > > > put it
> > >> >> >>>>> > > > >> > in
> > >> >> >>>>> > > > >> > >> the
> > >> >> >>>>> > > > >> > >> >> MD
> > >> >> >>>>> > > > >> > >> >> >>> > >> response
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> it
> > should
> > >> be
> > >> >> >>>> discussed
> > >> >> >>>>> > now.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> On
> > Wed,
> > >> Nov
> > >> >> 9,
> > >> >> >>>> 2016 at
> > >> >> >>>>> > 12:09
> > >> >> >>>>> > > AM,
> > >> >> >>>>> > > > >> > radai <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > >> >> >> radai.rosenblatt@gmail.com
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > im not
> > >> >> >> opposed
> > >> >> >>>> to the
> > >> >> >>>>> > idea of
> > >> >> >>>>> > > > >> > namespace
> > >> >> >>>>> > > > >> > >> >> >>> mapping.
> > >> >> >>>>> > > > >> > >> >> >>> > >> all
> > >> >> >>>>> > > > >> > >> >> >>> > >> > im
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > saying
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > that
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> its
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > not part
> > >> >> of
> > >> >> >>>> the "mvp"
> > >> >> >>>>> > and,
> > >> >> >>>>> > > > since
> > >> >> >>>>> > > > >> it
> > >> >> >>>>> > > > >> > >> >> requires
> > >> >> >>>>> > > > >> > >> >> >>> no
> > >> >> >>>>> > > > >> > >> >> >>> > >> wire
> > >> >> >>>>> > > > >> > >> >> >>> > >> > format
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > always
> > >> be
> > >> >> >>>> added later.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > also,
> > >> its
> > >> >> not
> > >> >> >>>> as
> > >> >> >>>>> > simple as
> > >> >> >>>>> > > just
> > >> >> >>>>> > > > >> > >> >> configuring
> > >> >> >>>>> > > > >> > >> >> >>> MM
> > >> >> >>>>> > > > >> > >> >> >>> > to
> > >> >> >>>>> > > > >> > >> >> >>> > >> do
> > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> transform:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> lets
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > say i've
> > >> >> >>>> implemented
> > >> >> >>>>> > large
> > >> >> >>>>> > > > message
> > >> >> >>>>> > > > >> > >> >> support as
> > >> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > on
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > mirror
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > target
> > >> >> >> cluster
> > >> >> >>>> its been
> > >> >> >>>>> > > > remapped
> > >> >> >>>>> > > > >> to
> > >> >> >>>>> > > > >> > >> >> {999,1}.
> > >> >> >>>>> > > > >> > >> >> >>> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > consumer
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > would
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > also
> > >> need
> > >> >> to
> > >> >> >>>> be told
> > >> >> >>>>> > to look
> > >> >> >>>>> > > > for
> > >> >> >>>>> > > > >> the
> > >> >> >>>>> > > > >> > >> large
> > >> >> >>>>> > > > >> > >> >> >>> > message
> > >> >> >>>>> > > > >> > >> >> >>> > >> > "part X
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> header
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > under
> > >> >> >> {999,1}.
> > >> >> >>>> doable,
> > >> >> >>>>> > but
> > >> >> >>>>> > > > tricky.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > On Tue,
> > >> >> Nov
> > >> >> >> 8,
> > >> >> >>>> 2016 at
> > >> >> >>>>> > 10:29
> > >> >> >>>>> > > > PM,
> > >> >> >>>>> > > > >> > Gwen
> > >> >> >>>>> > > > >> > >> >> >>> Shapira <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> > gwen@confluent.io
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > While
> > >> you
> > >> >> >> can
> > >> >> >>>> do
> > >> >> >>>>> > whatever
> > >> >> >>>>> > > you
> > >> >> >>>>> > > > >> want
> > >> >> >>>>> > > > >> > >> with a
> > >> >> >>>>> > > > >> > >> >> >>> > >> namespace
> > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > your
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > what
> > >> I'd
> > >> >> >>>> expect is
> > >> >> >>>>> > for each
> > >> >> >>>>> > > > app
> > >> >> >>>>> > > > >> to
> > >> >> >>>>> > > > >> > >> >> >>> namespaces
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > configurable...
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > So if I
> > >> >> >>>> accidentally
> > >> >> >>>>> > used
> > >> >> >>>>> > > 666
> > >> >> >>>>> > > > for
> > >> >> >>>>> > > > >> > my
> > >> >> >>>>> > > > >> > >> HR
> > >> >> >>>>> > > > >> > >> >> >>> > >> department,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > still
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > run
> > >> >> >> RadaiApp,
> > >> >> >>>> I can
> > >> >> >>>>> > config
> > >> >> >>>>> > > > >> > >> "namespace=42"
> > >> >> >>>>> > > > >> > >> >> >>> for
> > >> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > everything
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > will
> > >> look
> > >> >> >>>> normal.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > This
> > >> >> means
> > >> >> >>>> you only
> > >> >> >>>>> > need to
> > >> >> >>>>> > > > sync
> > >> >> >>>>> > > > >> > usage
> > >> >> >>>>> > > > >> > >> >> >>> inside
> > >> >> >>>>> > > > >> > >> >> >>> > your
> > >> >> >>>>> > > > >> > >> >> >>> > >> > own
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > organization.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > Still
> > >> >> hard,
> > >> >> >>>> but
> > >> >> >>>>> > somewhat
> > >> >> >>>>> > > > easier
> > >> >> >>>>> > > > >> > than
> > >> >> >>>>> > > > >> > >> >> syncing
> > >> >> >>>>> > > > >> > >> >> >>> > with
> > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > entire
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > world.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > On Tue,
> > >> >> Nov
> > >> >> >>>> 8, 2016
> > >> >> >>>>> > at 10:07
> > >> >> >>>>> > > > PM,
> > >> >> >>>>> > > > >> > >> radai <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> >>>> radai.rosenblatt@gmail.com>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > and
> > >> we
> > >> >> can
> > >> >> >>>> start
> > >> >> >>>>> > with
> > >> >> >>>>> > > > >> {namespace,
> > >> >> >>>>> > > > >> > >> id}
> > >> >> >>>>> > > > >> > >> >> and
> > >> >> >>>>> > > > >> > >> >> >>> no
> > >> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > always
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > add
> > >> it
> > >> >> >>>> later on
> > >> >> >>>>> > if/when
> > >> >> >>>>> > > > >> > collisions
> > >> >> >>>>> > > > >> > >> >> >>> actually
> > >> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > dont
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > they'd
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > be
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > a
> > >> >> >> problem).
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > every
> > >> >> >>>> interested
> > >> >> >>>>> > party (so
> > >> >> >>>>> > > > orgs
> > >> >> >>>>> > > > >> > or
> > >> >> >>>>> > > > >> > >> >> >>> > individuals)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > could
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > then
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > register
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >
> > >> prefix
> > >> >> (0
> > >> >> >> =
> > >> >> >>>>> > reserved, 1 =
> > >> >> >>>>> > > > >> > confluent
> > >> >> >>>>> > > > >> > >> ...
> > >> >> >>>>> > > > >> > >> >> >>> 666
> > >> >> >>>>> > > > >> > >> >> >>> > = me
> > >> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > whatever
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > with
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > the
> > >> >> 2nd ID
> > >> >> >>>> - so once
> > >> >> >>>>> > > > linkedin
> > >> >> >>>>> > > > >> > >> >> registers,
> > >> >> >>>>> > > > >> > >> >> >>> say
> > >> >> >>>>> > > > >> > >> >> >>> > 3,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > then
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > free
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > to
> > >> use
> > >> >> {3,
> > >> >> >>>> *} with a
> > >> >> >>>>> > > > reasonable
> > >> >> >>>>> > > > >> > >> >> >>> expectation
> > >> >> >>>>> > > > >> > >> >> >>> > to
> > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > collide
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > anything
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > else.
> > >> >> >>>> further
> > >> >> >>>>> > partitioning
> > >> >> >>>>> > > > of
> > >> >> >>>>> > > > >> > that *
> > >> >> >>>>> > > > >> > >> >> >>> becomes
> > >> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > problem,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >
> > >> >> "upstream
> > >> >> >>>>> > registration"
> > >> >> >>>>> > > of a
> > >> >> >>>>> > > > >> > >> namespace
> > >> >> >>>>> > > > >> > >> >> >>> only
> > >> >> >>>>> > > > >> > >> >> >>> > has
> > >> >> >>>>> > > > >> > >> >> >>> > >> to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > happen
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > > On
> > >> Tue,
> > >> >> >> Nov
> > >> >> >>>> 8, 2016
> > >> >> >>>>> > at
> > >> >> >>>>> > > 9:03
> > >> >> >>>>> > > > PM,
> > >> >> >>>>> > > > >> > >> James
> > >> >> >>>>> > > > >> > >> >> >>> Cheng <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> > >> >> wushujames@gmail.com
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> > On
> > >> >> Nov
> > >> >> >>>> 8, 2016,
> > >> >> >>>>> > at 5:54
> > >> >> >>>>> > > > PM,
> > >> >> >>>>> > > > >> > Gwen
> > >> >> >>>>> > > > >> > >> >> >>> Shapira <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> > gwen@confluent.io
> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> Thank
> > >> >> >>>> you so
> > >> >> >>>>> > much for
> > >> >> >>>>> > > > this
> > >> >> >>>>> > > > >> > clear
> > >> >> >>>>> > > > >> > >> and
> > >> >> >>>>> > > > >> > >> >> >>> fair
> > >> >> >>>>> > > > >> > >> >> >>> > >> > summary of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > arguments.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> I'm
> > >> >> in
> > >> >> >>>> favor of
> > >> >> >>>>> > ints.
> > >> >> >>>>> > > > Not a
> > >> >> >>>>> > > > >> > >> >> >>> deal-breaker,
> > >> >> >>>>> > > > >> > >> >> >>> > but
> > >> >> >>>>> > > > >> > >> >> >>> > >> > in
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> Even
> > >> >> >>>> more in
> > >> >> >>>>> > favor of
> > >> >> >>>>> > > > >> Magnus's
> > >> >> >>>>> > > > >> > >> >> >>> > decentralized
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > Roger's
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> tweak:
> > >> >> >>>> add a
> > >> >> >>>>> > namespace
> > >> >> >>>>> > > > for
> > >> >> >>>>> > > > >> > >> headers.
> > >> >> >>>>> > > > >> > >> >> >>> This
> > >> >> >>>>> > > > >> > >> >> >>> > will
> > >> >> >>>>> > > > >> > >> >> >>> > >> > allow
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > each
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > just
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> use
> > >> >> >>>> whatever IDs
> > >> >> >>>>> > it
> > >> >> >>>>> > > wants
> > >> >> >>>>> > > > >> > >> >> internally,
> > >> >> >>>>> > > > >> > >> >> >>> and
> > >> >> >>>>> > > > >> > >> >> >>> > >> then
> > >> >> >>>>> > > > >> > >> >> >>> > >> > let
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> deploying
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> the
> > >> >> app
> > >> >> >>>> figure
> > >> >> >>>>> > out an
> > >> >> >>>>> > > > >> > available
> > >> >> >>>>> > > > >> > >> >> >>> namespace
> > >> >> >>>>> > > > >> > >> >> >>> > ID
> > >> >> >>>>> > > > >> > >> >> >>> > >> > for the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > live
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> in.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> > So
> > >> >> >>>>> > > > >> > io.confluent.schema-registry
> > >> >> >>>>> > > > >> > >> can
> > >> >> >>>>> > > > >> > >> >> be
> > >> >> >>>>> > > > >> > >> >> >>> > >> > namespace
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >> deployment
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> and
> > >> >> >> 0x57
> > >> >> >>>> on
> > >> >> >>>>> > yours, and
> > >> >> >>>>> > > > the
> > >> >> >>>>> > > > >> > poor
> > >> >> >>>>> > > > >> > >> guys
> > >> >> >>>>> > > > >> > >> >> >>> > >> > developing the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> need
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> to
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> worry
> > >> >> >>>> about that.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> Gwen,
> > >> >> if
> > >> >> >> I
> > >> >> >>>>> > understand
> > >> >> >>>>> > > your
> > >> >> >>>>> > > > >> > example
> > >> >> >>>>> > > > >> > >> >> >>> right, an
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > application
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> deployer
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > might
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> decide to
> > >> >> >>>> use 0x01
> > >> >> >>>>> > in one
> > >> >> >>>>> > > > >> > >> deployment,
> > >> >> >>>>> > > > >> > >> >> and
> > >> >> >>>>> > > > >> > >> >> >>> > that
> > >> >> >>>>> > > > >> > >> >> >>> > >> > means
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > that
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > message
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> is
> > >> >> >> written
> > >> >> >>>> into the
> > >> >> >>>>> > > > broker, it
> > >> >> >>>>> > > > >> > >> will be
> > >> >> >>>>> > > > >> > >> >> >>> > saved on
> > >> >> >>>>> > > > >> > >> >> >>> > >> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > broker
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> that
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> specific
> > >> >> >>>> namespace
> > >> >> >>>>> > > (0x01).
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> If
> > >> you
> > >> >> >>>> were to
> > >> >> >>>>> > mirror
> > >> >> >>>>> > > that
> > >> >> >>>>> > > > >> > message
> > >> >> >>>>> > > > >> > >> >> into
> > >> >> >>>>> > > > >> > >> >> >>> > another
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > would
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> accompany
> > >> >> >>>> the
> > >> >> >>>>> > message,
> > >> >> >>>>> > > > right?
> > >> >> >>>>> > > > >> > What
> > >> >> >>>>> > > > >> > >> if
> > >> >> >>>>> > > > >> > >> >> the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> in
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> other
> > >> >> >>>> cluster uses
> > >> >> >>>>> > 0x57?
> > >> >> >>>>> > > > They
> > >> >> >>>>> > > > >> > won't
> > >> >> >>>>> > > > >> > >> >> >>> > understand
> > >> >> >>>>> > > > >> > >> >> >>> > >> > each
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > other?
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> I'm
> > >> >> not
> > >> >> >>>> sure
> > >> >> >>>>> > that's an
> > >> >> >>>>> > > > >> avoidable
> > >> >> >>>>> > > > >> > >> >> >>> problem. I
> > >> >> >>>>> > > > >> > >> >> >>> > >> > think it
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > simply
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > that
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > in
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> order
> > >> >> to
> > >> >> >>>> share
> > >> >> >>>>> > data, you
> > >> >> >>>>> > > > have
> > >> >> >>>>> > > > >> to
> > >> >> >>>>> > > > >> > >> also
> > >> >> >>>>> > > > >> > >> >> >>> have a
> > >> >> >>>>> > > > >> > >> >> >>> > >> > shared
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>> understanding of
> > >> >> >>>>> > what the
> > >> >> >>>>> > > > >> > >> namespaces
> > >> >> >>>>> > > > >> > >> >> >>> mean.
> > >> >> >>>>> > > > >> > >> >> >>> > >> Which
> > >> >> >>>>> > > > >> > >> >> >>> > >> > I
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > sense,
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> because
> > >> >> >> the
> > >> >> >>>>> > alternate
> > >> >> >>>>> > > > (sharing
> > >> >> >>>>> > > > >> > >> >> *nothing*
> > >> >> >>>>> > > > >> > >> >> >>> at
> > >> >> >>>>> > > > >> > >> >> >>> > >> all)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > would
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > mean
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> > there
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> would
> > >> >> be
> > >> >> >>>> no way to
> > >> >> >>>>> > > > understand
> > >> >> >>>>> > > > >> > each
> > >> >> >>>>> > > > >> > >> >> other.
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> -James
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> Gwen
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> > On
> > >> >> Tue,
> > >> >> >>>> Nov 8,
> > >> >> >>>>> > 2016 at
> > >> >> >>>>> > > > 4:23
> > >> >> >>>>> > > > >> > PM,
> > >> >> >>>>> > > > >> > >> >> radai <
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> > >> >> >>>> radai.rosenblatt@gmail.com
> > >> >> >>>>> > >
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >>
> > >> wrote:
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >>
> > >> +1
> > >> >> for
> > >> >> >>>> sean's
> > >> >> >>>>> > > document.
> > >> >> >>>>> > > > it
> > >> >> >>>>> > > > >> > >> covers
> > >> >> >>>>> > > > >> > >> >> >>> pretty
> > >> >> >>>>> > > > >> > >> >> >>> > >> much
> > >> >> >>>>> > > > >> > >> >> >>> > >> > all
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> > trade-offs
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> and
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >>
> > >> >> >> provides
> > >> >> >>>>> > concrete
> > >> >> >>>>> > > > figures
> > >> >> >>>>> > > > >> to
> > >> >> >>>>> > > > >> > >> argue
> > >> >> >>>>> > > > >> > >> >> >>> about
> > >> >> >>>>> > > > >> > >> >> >>> > :-)
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >>
> > >> >> >>>> (nit-picking -
> > >> >> >>>>> > used
> > >> >> >>>>> > > the
> > >> >> >>>>> > > > >> same
> > >> >> >>>>> > > > >> > >> xkcd
> > >> >> >>>>> > > > >> > >> >> >>> twice,
> > >> >> >>>>> > > > >> > >> >> >>> > >> also
> > >> >> >>>>> > > > >> > >> >> >>> > >> > trove
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > has
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
> > >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >>
> > >> >> superceded
> > >> >> >>>>> > > > >> > >> >
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >> > --
> > >> >> >>>>> > > > >> > Gwen Shapira
> > >> >> >>>>> > > > >> > Product Manager | Confluent
> > >> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
> > @gwenshap
> > >> >> >>>>> > > > >> > Follow us: Twitter | blog
> > >> >> >>>>> > > > >> >
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >> --
> > >> >> >>>>> > > > >> *Todd Palino*
> > >> >> >>>>> > > > >> Staff Site Reliability Engineer
> > >> >> >>>>> > > > >> Data Infrastructure Streaming
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > > >> linkedin.com/in/toddpalino
> > >> >> >>>>> > > > >>
> > >> >> >>>>> > > >
> > >> >> >>>>> > > >
> > >> >> >>>>> > > >
> > >> >> >>>>> > > > --
> > >> >> >>>>> > > > Gwen Shapira
> > >> >> >>>>> > > > Product Manager | Confluent
> > >> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > >> >> >>>>> > > > Follow us: Twitter | blog
> > >> >> >>>>> > > >
> > >> >> >>>>> > >
> > >> >> >>>>> >
> > >> >> >>>>> >
> > >> >> >>>>> > The information contained in this email is strictly
> > >> >> confidential
> > >> >> >>>> and for
> > >> >> >>>>> > the use of the addressee only, unless otherwise
> > indicated.
> > >> If
> > >> >> you
> > >> >> >>>> are not
> > >> >> >>>>> > the intended recipient, please do not read, copy,
> > use or
> > >> >> disclose
> > >> >> >>>> to others
> > >> >> >>>>> > this message or any attachment. Please also notify
> > the
> > >> sender
> > >> >> by
> > >> >> >>>> replying
> > >> >> >>>>> > to this email or by telephone (+44(020 7896 0011)
> > and then
> > >> >> delete
> > >> >> >>>> the email
> > >> >> >>>>> > and any copies of it. Opinions, conclusion (etc)
> > that do not
> > >> >> >>>> relate to the
> > >> >> >>>>> > official business of this company shall be
> > understood as
> > >> >> neither
> > >> >> >>>> given nor
> > >> >> >>>>> > endorsed by it. IG is a trading name of IG Markets
> > Limited
> > >> (a
> > >> >> >>>> company
> > >> >> >>>>> > registered in England and Wales, company number
> > 04008957)
> > >> and
> > >> >> IG
> > >> >> >>>> Index
> > >> >> >>>>> > Limited (a company registered in England and Wales,
> > company
> > >> >> >> number
> > >> >> >>>>> > 01190902). Registered address at Cannon Bridge
> > House, 25
> > >> >> Dowgate
> > >> >> >>>> Hill,
> > >> >> >>>>> > London EC4R 2YA. Both IG Markets Limited (register
> > number
> > >> >> 195355)
> > >> >> >>>> and IG
> > >> >> >>>>> > Index Limited (register number 114059) are
> > authorised and
> > >> >> >>>> regulated by the
> > >> >> >>>>> > Financial Conduct Authority.
> > >> >> >>>>> >
> > >> >> >>>>>
> > >> >> >>>>>
> > >> >> >>>>> The information contained in this email is strictly
> > confidential
> > >> and
> > >> >> >> for
> > >> >> >>>> the use of the addressee only, unless otherwise indicated.
> > If you
> > >> are
> > >> >> >> not
> > >> >> >>>> the intended recipient, please do not read, copy, use or
> > disclose
> > >> to
> > >> >> >> others
> > >> >> >>>> this message or any attachment. Please also notify the
> > sender by
> > >> >> >> replying
> > >> >> >>>> to this email or by telephone (+44(020 7896 0011) and then
> > delete
> > >> the
> > >> >> >> email
> > >> >> >>>> and any copies of it. Opinions, conclusion (etc) that do
> > not relate
> > >> >> to
> > >> >> >> the
> > >> >> >>>> official business of this company shall be understood as
> > neither
> > >> >> given
> > >> >> >> nor
> > >> >> >>>> endorsed by it. IG is a trading name of IG Markets Limited
> > (a
> > >> company
> > >> >> >>>> registered in England and Wales, company number 04008957)
> > and IG
> > >> >> Index
> > >> >> >>>> Limited (a company registered in England and Wales,
> company
> > number
> > >> >> >>>> 01190902). Registered address at Cannon Bridge House, 25
> > Dowgate
> > >> >> Hill,
> > >> >> >>>> London EC4R 2YA. Both IG Markets Limited (register number
> > 195355)
> > >> >> and IG
> > >> >> >>>> Index Limited (register number 114059) are authorised and
> > regulated
> > >> >> by
> > >> >> >> the
> > >> >> >>>> Financial Conduct Authority.
> > >> >> >>>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>
> > >> >> >>
> > >> >> >>
> > >> >> >
> > >> >>
> > >> >>
> > >> >
> > >> The information contained in this email is strictly confidential
> > and for
> > >> the use of the addressee only, unless otherwise indicated. If you
> > are not
> > >> the intended recipient, please do not read, copy, use or disclose
> > to others
> > >> this message or any attachment. Please also notify the sender by
> > replying
> > >> to this email or by telephone (+44(020 7896 0011) and then delete
> > the email
> > >> and any copies of it. Opinions, conclusion (etc) that do not
> relate
> > to the
> > >> official business of this company shall be understood as neither
> > given nor
> > >> endorsed by it. IG is a trading name of IG Markets Limited (a
> > company
> > >> registered in England and Wales, company number 04008957) and IG
> > Index
> > >> Limited (a company registered in England and Wales, company number
> > >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> > Hill,
> > >> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> > and IG
> > >> Index Limited (register number 114059) are authorised and
> regulated
> > by the
> > >> Financial Conduct Authority.
> > >>
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
> >
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Jay Kreps <ja...@confluent.io>.
Hey Gwen, yes, your approach is much better than what I described and
solves one of my objections. I didn't think of doing it in MM rather than
in the original producer. I don't think it adds any overhead beyond the
additional header bytes if I'm not mistaken, since MM already unpacks and
repacks messages (unlike the broker which tries to avoid repacking and
hence can't efficiently add a header). I think my other objection around
size is likely solvable too by some kind of aliasing scheme? If you guys
think this can be made to work then I think we have two pretty credible use
cases with this and the tracing id family of uses, plus a number of things
that are at least moderately improved. If so, I'm convinced.
-Jay
On Tue, Jan 10, 2017 at 8:48 AM, Michael Pearce <Mi...@ig.com>
wrote:
> Hi Jay et al.
>
> On the replication using cluster id.
>
> Two options depending on how a company wants or needs to implement their
> replication as there are options of doing this as a mesh, star, or ring.
>
> Either as some companies enforce a provider/custom producer/consumer
> wrapper, it can be enforced the interceptors needed to be added, as you can
> enforce to get support of the platform you must use company x’s internal
> wrappers.
> An alternative approach is to do this at the mirror maker, as Gwen noted
> we can actually make the mirror maker append this, like wise for a more
> complex implementation, where you can have multiple hops/routes this can
> also be the message routing.
>
> Re the size of the clusterId, yes the auto generated one is “cough”
> hideous in size.
>
> Most organisations though will only need a space to uniquely identify 127
> (byte) clusters, or at worst 2 bytes (short) they need to mirror and
> replicate between, and as such can either as platform teams generally will
> roll out the mirror makers / replicators with the cluster can either
> manually adjust the clusterId on ZK, or use a mapping file all their mirror
> makers have to map the larger id to a smaller one.
>
> Also as Gwen states this is something you should be able to opt into or
> not, this is the benefit of headers, if you don’t use it, it doesn’t use up
> the space, its entirely optional.
>
> Regards
> Mike
>
>
> On 09/01/2017, 19:40, "Gwen Shapira" <gw...@confluent.io> wrote:
>
> Mirroring + ClusterIDs:
>
> Kafka clusters already have an automatically generated ID that is
> exposed to clients:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
>
> It sounds trivial to get MirrorMaker's consumer to grab this ID (which
> always exists) from the source cluster and add it to the message
> headers.
>
> Obviously adding stuff to the message makes it larger so this should
> be a feature you should be able to turn on/off depending on the
> replication requirements. If you need the feature - you pay the price.
> I don't think there is an immediate need to put an effort into
> shrinking the Cluster ID, but others can comment on that.
>
>
>
>
>
> On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> > Hey Michael,
> >
> > Sorry for lagging, I've been out for the holiday break. What would
> help me
> > would be to get to a consolidated list of use cases we've worked out
> and
> > think add real value. I think getting down to the details really
> matters
> > for these.
> >
> > The two lists I'm aware of are the one in the KIP and the one you
> gave in
> > this email thread. Both seem to be more at the sketch level. Let me
> review
> > the use cases and see if I have it right. If so let's get it in the
> design
> > proposal.
> >
> > Your use cases:
> >
> > 1. Tracing IDs -- I get this use case and think I see how it
> would work.
> > I think because the tracing is from a third party system there is
> some
> > value to having a header versus a custom format to make this work
> fully
> > transparently.
> > 2. Mirroring -- I'm a bit unsure how this will work, I'll explain
> my
> > thinking below
> > 3. End-to-end encryption -- I think what you are saying is that
> if you
> > want "routing' fields that are accessible in transit without
> decrypting you
> > need a scheme to leave this decrypted.I think the idea is to avoid
> > decrypting to get a header, but I'm not sure if I get how this is
> better
> > than a custom message format that doesn't encrypt the things you
> don't want
> > encrypted since you have to be aware of the scheme anyway.
> > 4. EOS -- I'm not convinced this is a good approach for EOS.
> >
> > Linked from KIP:
> >
> > 1. Schema-Registry-aware avro serialization -- not clear how this
> is
> > better than keeping the schema id in the value
> > 2. Polyglot kafka topics -- marginally better than a container
> format,
> > but of course you still have to all agree on the the header and
> > header=>serialization mapping so not sure if it's really better
> than a
> > container format. Also not wild about polyglot topics :-)
> > 3. End-to-End routing trail including mirroring --- I'd put this
> under
> > tracing ids
> > 4. Cross-service process tracing -- ditto
> > 5. Signing -- I like this use case but it seems to depend on a
> feature
> > we don't have and aren't proposing adding, right?
> > 6. Per-record custom compression -- why do you want this?
> compressing
> > per message won't work well compared to batch compression, right?
> > 7. Large Message Support -- I think we can agree that although
> this
> > might work it's clearly not the right way to support large
> messages in Kafka
> >
> > Matthias
> >
> > 1. Control messages --- Matthias, not sure if I get the
> motivation in
> > streams for this. If we did want to do it, wouldn't it be easier
> to
> > implement and by allowing users to send control messages (in the
> EOS sense)
> > and adding a consumer config to allow either passing these on or
> > suppressing them?
> >
> > So I if I were to summarize I'd list the use cases as follows:
> >
> > - Much better with headers: tracing IDs
> > - Somewhat better: end-to-end encryption, polyglot topics
> > - Don't buy: Mirroring (though perhaps could be made to work with
> > changes to the proposal), EOS, large messages, compression
> > - Not sure: Control messages
> >
> > Does this match where other people are at? From my point of view for
> > something like this where we are changing the core data model we
> should
> > ideally get a handful of use cases to the point where they are
> really well
> > thought out and we know they can use the feature. I think we have
> one solid
> > one with tracing IDs, and I think maybe mirroring could be another
> with
> > some work. I'm less sure about the others.
> >
> > Bi-directional Mirroring
> > I think bidirectional mirroring could potentially become a
> compelling use
> > case if we an find a way to make it work. If I understand it, the
> proposal
> > is to add a header like "origin-cluster" with the cluster id of the
> Kafka
> > cluster. Then modify the mirroring tool to disallow mirroring a
> message
> > back to it's own origin cluster. This basic idea is definitely
> workable and
> > useful, but can we really get to a good implementation using headers?
> > I imagine the approach would be to add this header in the producer
> if some
> > config is set. There are two problems:
> >
> > - You add a pretty big honking cluster ID to each message. For
> small
> > messages this maybe larger than the contents itself.
> > - What happens if someone somewhere forgets to set that config on
> some
> > client somewhere?
> >
> > If someone forgets to set the config in some client somewhere I
> guess you
> > have various options in mirror maker:
> >
> > 1. drop the data (pretty bad, now your data is out of sync)
> > 2. crash (pretty bad--how do you recover)
> > 3. insert some default (pretty bad since would also lead to data
> loss if
> > that's not right)
> >
> > Basically although I think a cluster tag could be a good way to do
> > bi-directional mirroring, I'm not sure this naive approach I
> described
> > would get us to a usable implementation. This could be fixable. I
> think if
> > there was some scheme to check required headers on the server that
> could
> > alleviate the missing header problem. Not really sure what to do
> about the
> > message bloat problem, but maybe there could be some way to shrink
> the
> > cluster id?
> >
> > Cheers,
> >
> > -Jay
> >
> >
> > On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <
> Michael.Pearce@ig.com>
> > wrote:
> >
> >> @Jay, Ismael and a few others.
> >>
> >> Based on the discussion on kip-87, and the additional use case by
> Matthias.
> >>
> >> Just for clarity do we now have consensus on the earlier points (1)
> >> headers are useful (2) headers should be native in Kafka?
> >>
> >> Cheers
> >> Mike
> >> ________________________________________
> >> From: radai <ra...@gmail.com>
> >> Sent: Monday, December 19, 2016 5:01 PM
> >> To: dev@kafka.apache.org
> >> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
> >>
> >> having said that im perfectly fine with something like a "no
> payload" flag
> >> (as opposed to null payload), which would mean such a msg is not
> returned
> >> from poll() unless the user code specifically "opts in" and would
> serve
> >> only to carry headers (making it a control message)
> >>
> >> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com>
> wrote:
> >>
> >> > if all you want is to tag "end of stream" wouldnt a header on the
> last
> >> msg
> >> > in the stream be enough?
> >> >
> >> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <
> matthias@confluent.io>
> >> > wrote:
> >> >
> >> >> The details about headers for control messages are still to
> define. But
> >> >> yes, the idea is to have some common default behavior that
> clients would
> >> >> need to implement.
> >> >>
> >> >> The point is, that "regular headers" add meta data to regular
> messages.
> >> >> Thus, those messages will be returned to the user via .poll().
> And after
> >> >> the message is received the user can check if meta data is
> present and
> >> >> read it.
> >> >>
> >> >> For control messages, we do not want those to pop up via .poll()
> as
> >> >> those are no regular messages. A client would need to opt-in to
> see
> >> >> those messages (either via poll() or maybe a callback). Thus, we
> need
> >> >> some special (standardized) header IDs that indicate control
> messages
> >> >> that should not be returned to the user via poll() by default.
> >> >>
> >> >>
> >> >> -Matthias
> >> >>
> >> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
> >> >> > Matthias,
> >> >> >
> >> >> > Thanks for your input. I'm +1 on control messages as they
> seem to be
> >> >> the
> >> >> > simplest way to implement watermarks (
> >> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-
> streaming-102),
> >> a
> >> >> > feature that would add a lot of value to Kafka Streams IMHO.
> >> >> >
> >> >> > Your argument that the control-message indicator belongs in the
> >> >> client-only
> >> >> > section of the record format make sense. Just to make sure I
> >> >> understand,
> >> >> > are you suggesting that control messages would be indicated by
> a
> >> >> standard
> >> >> > reserved header? By standard, I mean that ALL Kafka consumers
> would
> >> >> know
> >> >> > to handle these messages differently (possibly just ignoring
> them).
> >> >> This
> >> >> > would need to be added to the specification of the consumer
> protocol
> >> so
> >> >> > that all Kafka clients implement it, right? I think it's a
> good idea
> >> >> but
> >> >> > just checking.
> >> >> >
> >> >> > Cheers,
> >> >> >
> >> >> > Roger
> >> >> >
> >> >> >
> >> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
> >> matthias@confluent.io
> >> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> Yes and no. I did overload the term "control message".
> >> >> >>
> >> >> >> EOS control messages are for client-broker communication and
> thus
> >> never
> >> >> >> exposed to any application. And I think this is a good design
> because
> >> >> >> broker needs to understand those control messages. Thus, this
> should
> >> be
> >> >> >> a protocol change.
> >> >> >>
> >> >> >> The type of control messages I have in mind are for
> client-client
> >> >> >> (application-application) communication and the broker is
> agnostic to
> >> >> >> them. Thus, it should not be a protocol change.
> >> >> >>
> >> >> >>
> >> >> >> -Matthias
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 12/14/16 9:42 AM, radai wrote:
> >> >> >>> arent control messages getting pushed as their own top level
> >> protocol
> >> >> >>> change (and a fairly massive one) for the transactions KIP ?
> >> >> >>>
> >> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
> >> >> matthias@confluent.io>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>>> Hi,
> >> >> >>>>
> >> >> >>>> I want to add a completely new angle to this discussion.
> For this,
> >> I
> >> >> >>>> want to propose an extension for the headers feature that
> enables
> >> new
> >> >> >>>> uses cases -- and those new use cases might convince people
> to
> >> >> support
> >> >> >>>> headers (of course including the larger scoped proposal).
> >> >> >>>>
> >> >> >>>> Extended Proposal:
> >> >> >>>>
> >> >> >>>> Allow messages with a certain header key to be special
> "control
> >> >> >>>> messages" (w/ o w/o payload) that are not exposed to an
> application
> >> >> via
> >> >> >>>> .poll().
> >> >> >>>>
> >> >> >>>> Thus, a consumer client would automatically skip over those
> >> >> messages. If
> >> >> >>>> an application knows about embedded control messages, it
> can "sing
> >> >> up"
> >> >> >>>> to those messages by the consumer client and either get a
> callback
> >> or
> >> >> >>>> the consumer auto-drop for this messages gets disabled
> (allowing to
> >> >> >>>> consumer those messages via poll()).
> >> >> >>>>
> >> >> >>>> (The details need further considerations/discussion. I just
> want to
> >> >> >>>> sketch the main idea.)
> >> >> >>>>
> >> >> >>>> Usage:
> >> >> >>>>
> >> >> >>>> There is a shared topic (ie, used by multiple applications)
> and a
> >> >> >>>> producer application wants to embed a special message in
> the topic
> >> >> for a
> >> >> >>>> dedicated consumer application. Because only one
> application will
> >> >> >>>> understand this message, it cannot be a regular message as
> this
> >> would
> >> >> >>>> break all applications that do not understand this message.
> The
> >> >> producer
> >> >> >>>> application would set a special metadata key and no consumer
> >> >> application
> >> >> >>>> would see this control message by default because they did
> not
> >> enable
> >> >> >>>> their consumer client to return this message in poll() (and
> the
> >> >> client
> >> >> >>>> would just drop this message with special metadata key).
> Only the
> >> >> single
> >> >> >>>> application that should receive this message, will
> subscribe to
> >> this
> >> >> >>>> message on its consumer client and process it.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> Concrete Use Case: Kafka Streams
> >> >> >>>>
> >> >> >>>> In Kafka Streams, we would like to propagate "control
> messages"
> >> from
> >> >> >>>> subtopology to subtopology. There are multiple scenarios
> for which
> >> >> this
> >> >> >>>> would be useful. For example, currently we do not guarantee
> a
> >> >> >>>> "consistent shutdown" of an application. By this, I mean
> that input
> >> >> >>>> records might not be completely processed by the whole
> topology
> >> >> because
> >> >> >>>> the application shutdown happens "in between" and an
> intermediate
> >> >> result
> >> >> >>>> topic gets "stock" in an intermediate topic. Thus, a user
> would see
> >> >> an
> >> >> >>>> committed offset of the source topic of the application,
> but no
> >> >> >>>> corresponding result record in the output topic.
> >> >> >>>>
> >> >> >>>> Having "shutdown markers" would allow us, to first stop the
> >> upstream
> >> >> >>>> subtopology and write this marker into the intermediate
> topic and
> >> the
> >> >> >>>> downstream subtopology would only shut down itself after is
> sees
> >> the
> >> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown, that
> no
> >> >> >>>> "in-flight" messages got stuck in intermediate topics.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> A similar usage would be for KIP-95 (Incremental Batch
> Processing).
> >> >> >>>> There was a discussion about the proposed metadata topic,
> and we
> >> >> could
> >> >> >>>> avoid this metadata topic if we would have "control
> messages".
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> Right now, we cannot insert an "application control message"
> >> because
> >> >> >>>> Kafka Streams does not own all topics it read/writes and
> thus might
> >> >> >>>> break other consumer application (as described above) if we
> inject
> >> >> >>>> random messages that are not understood by other apps.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> Of course, one can work around "embedded control messaged"
> by using
> >> >> an
> >> >> >>>> additional topic to propagate control messaged between
> application
> >> >> (as
> >> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka
> Streams). But
> >> >> there
> >> >> >>>> are major concerns about adding this metadata topic in the
> KIP and
> >> >> this
> >> >> >>>> shows that other application that need a similar pattern
> might
> >> profit
> >> >> >>>> from topic embedded "control messages", too.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> One last important consideration: those "control messages"
> are used
> >> >> for
> >> >> >>>> client to client communication and are not understood by the
> >> broker.
> >> >> >>>> Thus, those messages should not be enabled within the
> message
> >> format
> >> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land"
> record
> >> >> headers
> >> >> >>>> would be a nice way to implement them. Because KIP-82 did
> consider
> >> >> key
> >> >> >>>> namespaces for metatdata keys, this extension should not be
> an own
> >> >> KIP
> >> >> >>>> but should be included in KIP-82 to reserve a namespace for
> >> "control
> >> >> >>>> message" in the first place.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> Sorry for the long email... Looking forward to your
> feedback.
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> -Matthias
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
> >> >> >>>>> Hi Jun
> >> >> >>>>>
> >> >> >>>>> 100) each time a transaction exits a jvm for a remote
> system
> >> >> (HTTP/JMS/
> >> >> >>>> Hopefully one day kafka) the APM tools stich in a unique id
> >> (though I
> >> >> >>>> believe it contains the end2end uuid embedded in this id),
> on
> >> >> receiving
> >> >> >> the
> >> >> >>>> message at the receiving JVM the apm code takes this out,
> and
> >> >> continues
> >> >> >> its
> >> >> >>>> tracing on the that new thread. Both JVM’s (and other
> languages the
> >> >> APM
> >> >> >>>> tool supports) send this data async back to the central
> controllers
> >> >> >> where
> >> >> >>>> the stiching togeather occurs. For this they need some
> header space
> >> >> for
> >> >> >>>> them to put this id.
> >> >> >>>>>
> >> >> >>>>> 101) Yes indeed we have a business transaction Id in the
> payload.
> >> >> >> Though
> >> >> >>>> this is a system level tracing, that we need to have marry
> up. Also
> >> >> as
> >> >> >> per
> >> >> >>>> note on end2end encryption we’d be unable to prove the flow
> if the
> >> >> >> payload
> >> >> >>>> is encrypted as we’d not have access to this at certain
> points of
> >> the
> >> >> >> flow
> >> >> >>>> through the infrastructure/platform.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> 103) As said we use this mechanism in IG very
> successfully, as
> >> >> stated
> >> >> >>>> per key we guarantee the transaction producing app to
> handle the
> >> >> >>>> transaction of a key at one DC unless at point of critical
> failure
> >> >> >> where we
> >> >> >>>> have to flip processing to another. We care about key
> ordering.
> >> >> >>>>> I disagree on the offset comment for the partition
> solution unless
> >> >> you
> >> >> >>>> do full ISR, or expensive full XA transactions even with
> partitions
> >> >> you
> >> >> >>>> cannot fully guarantee offsets would match.
> >> >> >>>>>
> >> >> >>>>> 105) Very much so, I need to have access at the platform
> level to
> >> >> the
> >> >> >>>> other meta data all mentioned, without having to need to
> have
> >> access
> >> >> to
> >> >> >> the
> >> >> >>>> encryption keys of the payload.
> >> >> >>>>>
> >> >> >>>>> 106)
> >> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d need
> to have
> >> a
> >> >> >>>> global producerId register which would be very hard to
> >> >> enforce/ensure is
> >> >> >>>> current and correct, just to understand the message origins
> of its
> >> >> >>>> region/az/cluster for routing.
> >> >> >>>>> The client wrapper version, producerId can be the same, as
> >> obviously
> >> >> >> the
> >> >> >>>> producer could upgrade its wrapper, as such we need to know
> what
> >> >> wrapper
> >> >> >>>> version the message is created with.
> >> >> >>>>> Likewise the IP address, as stated we can have our
> producer move,
> >> >> where
> >> >> >>>> its IP would change.
> >> >> >>>>>
> >> >> >>>>> 107)
> >> >> >>>>> UUID is set on the message by interceptors before actual
> producer
> >> >> >>>> transport send. This is for platform level message dedupe
> >> guarantee,
> >> >> the
> >> >> >>>> business payload should be agnostic to this. Please see
> >> >> >>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-
> det
> >> >> ection.html
> >> >> >>>> note this is not touching business payloads.
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
> >> >> >>>>>
> >> >> >>>>> Hi, Michael,
> >> >> >>>>>
> >> >> >>>>> Thanks for the reply. I find it very helpful.
> >> >> >>>>>
> >> >> >>>>> Data lineage:
> >> >> >>>>> 100. I'd like to understand the APM use case a bit
> more. It
> >> >> sounds
> >> >> >>>> like
> >> >> >>>>> that those APM plugins can generate a transaction id
> that we
> >> >> could
> >> >> >>>>> potentially put in the header of every message. How
> would you
> >> >> >>>> typically
> >> >> >>>>> make use of such transaction ids? Are there other
> metadata
> >> >> >>>> associated with
> >> >> >>>>> the transaction id and if so, how are they propagated
> >> >> downstream?
> >> >> >>>>>
> >> >> >>>>> 101. For the finance use case, if the concept of
> transaction
> >> is
> >> >> >>>> important,
> >> >> >>>>> wouldn't it be typically included in the message
> payload
> >> >> instead of
> >> >> >>>> as an
> >> >> >>>>> optional header field?
> >> >> >>>>>
> >> >> >>>>> 102. The data lineage that Altas and Navigator support
> seems
> >> to
> >> >> be
> >> >> >>>> at the
> >> >> >>>>> dataset level, not per record level? So, not sure if
> per
> >> message
> >> >> >>>> headers
> >> >> >>>>> are relevant there.
> >> >> >>>>>
> >> >> >>>>> Mirroring:
> >> >> >>>>> 103. The benefit of using separate partitions is that
> it
> >> >> >> potentially
> >> >> >>>> makes
> >> >> >>>>> it easy to preserve offsets during mirroring. This
> will make
> >> it
> >> >> >>>> easier for
> >> >> >>>>> consumer to switch clusters. Currently, the consumers
> can
> >> switch
> >> >> >>>> clusters
> >> >> >>>>> by using the timestampToOffset() api, but it has to
> deal with
> >> >> >>>> duplicates.
> >> >> >>>>> Good point on the issue with log compact and I am not
> sure how
> >> >> to
> >> >> >>>> address
> >> >> >>>>> this. However, even if we mirror into the existing
> partitions,
> >> >> the
> >> >> >>>> ordering
> >> >> >>>>> for messages generated from different clusters seems
> >> >> >>>> non-deterministic
> >> >> >>>>> anyway. So, it seems that the consumers already have
> to deal
> >> >> with
> >> >> >>>> that? If
> >> >> >>>>> a topic is compacted, does that mean which messages are
> >> >> preserved
> >> >> >> is
> >> >> >>>> also
> >> >> >>>>> non-deterministic across clusters?
> >> >> >>>>>
> >> >> >>>>> 104. Good point on partition key.
> >> >> >>>>>
> >> >> >>>>> End-to-end encryption:
> >> >> >>>>> 105. So, it seems end-to-end encryption is useful. Are
> headers
> >> >> >>>> useful there?
> >> >> >>>>>
> >> >> >>>>> Auditing:
> >> >> >>>>> 106. It seems other than the UUID, all other metadata
> are per
> >> >> >>>> producer?
> >> >> >>>>>
> >> >> >>>>> EOS:
> >> >> >>>>> 107. How are those UUIDs generated? I am not sure if
> they can
> >> be
> >> >> >>>> generated
> >> >> >>>>> in the producer library. An application may send
> messages
> >> >> through a
> >> >> >>>> load
> >> >> >>>>> balancer and on retry, the same message could be
> routed to a
> >> >> >>>> different
> >> >> >>>>> producer instance. So, it seems that the application
> has to
> >> >> >> generate
> >> >> >>>> the
> >> >> >>>>> UUIDs. In that case, shouldn't the application just
> put the
> >> >> UUID in
> >> >> >>>> the
> >> >> >>>>> payload?
> >> >> >>>>>
> >> >> >>>>> Thanks,
> >> >> >>>>>
> >> >> >>>>> Jun
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
> >> >> >>>> Michael.Pearce@ig.com>
> >> >> >>>>> wrote:
> >> >> >>>>>
> >> >> >>>>> > Hi Jun.
> >> >> >>>>> >
> >> >> >>>>> > Per Transaction Tracing / Data Lineage.
> >> >> >>>>> >
> >> >> >>>>> > As Stated in the KIP this has the first use case of
> how many
> >> >> APM
> >> >> >>>> tools now
> >> >> >>>>> > work.
> >> >> >>>>> > I would find it impossible for any one to argue this
> is not
> >> >> >>>> important or a
> >> >> >>>>> > niche market as it has its own gartner report for
> this
> >> space.
> >> >> >> Such
> >> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace,
> Hawqular are
> >> >> but a
> >> >> >>>> few.
> >> >> >>>>> >
> >> >> >>>>> > Likewise these APM tools can help very rapidly track
> down
> >> >> issues
> >> >> >>>> and
> >> >> >>>>> > automatically capture metrics, perform actions based
> on
> >> >> >> unexpected
> >> >> >>>> behavior
> >> >> >>>>> > to auto recover services.
> >> >> >>>>> >
> >> >> >>>>> > Before mentioning looking at aggregated stats, in
> these
> >> cases
> >> >> >> where
> >> >> >>>>> > actually on critical flows we cannot afford to have
> >> aggregated
> >> >> >>>> rolled up
> >> >> >>>>> > stats only.
> >> >> >>>>> >
> >> >> >>>>> > With the APM tool we use its actually able to detect
> a
> >> single
> >> >> >>>> transaction
> >> >> >>>>> > failure and capture the thread traces in the JVM
> where it
> >> >> failed
> >> >> >>>> and
> >> >> >>>>> > everything for us, to the point it sends us alerts
> where we
> >> >> have
> >> >> >>>> this
> >> >> >>>>> > giving the line number of the code that caused it,
> the
> >> >> >> transaction
> >> >> >>>> trace
> >> >> >>>>> > through all the services and endpoints (supported)
> upto the
> >> >> point
> >> >> >>>> of
> >> >> >>>>> > failure, it can also capture the data in and out (so
> we can
> >> >> >>>> replay).
> >> >> >>>>> > Because atm Kafka doesn’t support us being able to
> stich in
> >> >> these
> >> >> >>>> tracing
> >> >> >>>>> > transaction ids natively, we cannot get these
> benefits as
> >> >> such is
> >> >> >>>> limiting
> >> >> >>>>> > our ability support apps and monitor them to the same
> >> >> standards
> >> >> >> we
> >> >> >>>> come to
> >> >> >>>>> > expect when on a kafka flow.
> >> >> >>>>> >
> >> >> >>>>> > This actually ties in with Data Lineage, as the same
> tracing
> >> >> can
> >> >> >>>> be used
> >> >> >>>>> > to back stich this. Essentially many times due to
> the sums
> >> of
> >> >> >> money
> >> >> >>>>> > involved there are disputes, and typically as a
> financial
> >> >> >>>> institute the
> >> >> >>>>> > easiest and cleanest way to prove when disputes
> arise is to
> >> >> >>>> present the
> >> >> >>>>> > actual flow and processes involved in a transaction.
> >> >> >>>>> >
> >> >> >>>>> > Likewise as Hadoop matures its evident this case is
> >> >> important, as
> >> >> >>>> tools
> >> >> >>>>> > such as Atlas (Hortonworks led) and Navigator
> (cloudera led)
> >> >> are
> >> >> >>>> evident
> >> >> >>>>> > also I believe the importance here is very much NOT
> just a
> >> >> >>>> financial issue.
> >> >> >>>>> >
> >> >> >>>>> > From a MDM point of view any company wanting to care
> about
> >> >> Data
> >> >> >>>> Quality
> >> >> >>>>> > and Data Governance - Data Lineage is a key piece in
> this
> >> >> puzzle.
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > RE Mirroring,
> >> >> >>>>> >
> >> >> >>>>> > As per the KIP in-fact this is exactly what we do re
> cluster
> >> >> id,
> >> >> >>>> to mirror
> >> >> >>>>> > a network of clusters between AZ’s / Regions. We
> know a
> >> >> >>>> transaction for a
> >> >> >>>>> > key will be done within a AZ/Region, as such we
> know the
> >> >> write
> >> >> >> to
> >> >> >>>> kafka
> >> >> >>>>> > would be ordered per key. But we need eventual view
> of that
> >> >> >> across
> >> >> >>>> in our
> >> >> >>>>> > other regions/az’s. When we have complete AZ or
> Region
> >> >> failure we
> >> >> >>>> know
> >> >> >>>>> > there will be a brief interruption whilst those
> transactions
> >> >> are
> >> >> >>>> moved to
> >> >> >>>>> > another region but we expect after it to continue.
> >> >> >>>>> >
> >> >> >>>>> > As mentioned having separate Partions to do this
> starts to
> >> get
> >> >> >>>>> > ugly/complicated for us:
> >> >> >>>>> > how would I do compaction where a key is in two
> partitions?
> >> >> >>>>> > How do we balance consumers so where multiple
> partitions
> >> with
> >> >> the
> >> >> >>>> same key
> >> >> >>>>> > goto the same consumer
> >> >> >>>>> > What do you do if cluster 1 has 5 partitions but
> cluster 20
> >> >> has
> >> >> >> 10
> >> >> >>>> because
> >> >> >>>>> > its larger kit in our more core DC’s, as such key to
> >> partition
> >> >> >>>> mappings for
> >> >> >>>>> > consumers get even more complicated.
> >> >> >>>>> > What do you do if we add or remove a complete region
> >> >> >>>>> >
> >> >> >>>>> > Where as simple mirror will work we just need to
> ensure we
> >> >> don’t
> >> >> >>>> have a
> >> >> >>>>> > cycle which we can do with clusterId.
> >> >> >>>>> >
> >> >> >>>>> > We even have started to look at shortest path mirror
> routing
> >> >> >> based
> >> >> >>>> on
> >> >> >>>>> > clusterId, if we also had the region and az info on
> the
> >> >> >> originating
> >> >> >>>>> > message, this we have not implemented but some ideas
> come
> >> from
> >> >> >>>> network
> >> >> >>>>> > routing, and also the dispatcher router in apache
> qpid.
> >> >> >>>>> >
> >> >> >>>>> > Also we need to have data perimeters e.g. certain
> data
> >> cannot
> >> >> >> leave
> >> >> >>>>> > certain countries borders. We want this all
> automated so
> >> that
> >> >> at
> >> >> >>>> the
> >> >> >>>>> > platform level without having to touch or look at the
> >> business
> >> >> >>>> data inside
> >> >> >>>>> > we can have headers we can put tags into so that we
> can
> >> ensure
> >> >> >>>> this doesn’t
> >> >> >>>>> > occur when we mirror. (actually links in to data
> lineage /
> >> >> >> tracing
> >> >> >>>> as again
> >> >> >>>>> > we need to tag messages at a platform level)
> Examples are we
> >> >> are
> >> >> >>>> not
> >> >> >>>>> > allowed Private customer details to leave
> Switzerland, yet
> >> we
> >> >> >> need
> >> >> >>>> those
> >> >> >>>>> > systems integrated.
> >> >> >>>>> >
> >> >> >>>>> > Lastly around mirroring we have a partionKey field,
> as the
> >> key
> >> >> >>>> used for
> >> >> >>>>> > portioning logic != compaction key all the time but
> we want
> >> to
> >> >> >>>> preserve it
> >> >> >>>>> > for when we mirror so that if source cluster
> partition count
> >> >> !=
> >> >> >>>> destination
> >> >> >>>>> > cluster partition count we can honour the same
> partitioning
> >> >> >> logic.
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > RE End 2 End encryption
> >> >> >>>>> >
> >> >> >>>>> > As I believe mentioned just before, the solution you
> mention
> >> >> just
> >> >> >>>> doesn’t
> >> >> >>>>> > cut the mustard these days with many regulators. An
> >> operations
> >> >> >>>> person with
> >> >> >>>>> > access to the box should not be able to have access
> to the
> >> >> data.
> >> >> >>>> Many now
> >> >> >>>>> > actually impose quite literally the implementation
> expected
> >> >> being
> >> >> >>>> end2end
> >> >> >>>>> > encryption for certain data (Singapore for us is one
> that I
> >> am
> >> >> >>>> most aware
> >> >> >>>>> > of). In fact we’re even now needing encrypt the data
> and
> >> store
> >> >> >> the
> >> >> >>>> keys in
> >> >> >>>>> > HSM modules.
> >> >> >>>>> >
> >> >> >>>>> > Likewise the performance penalty on encrypting
> decrypting as
> >> >> you
> >> >> >>>> produce
> >> >> >>>>> > over wire, then again encrypt decrypt as the data is
> stored
> >> on
> >> >> >> the
> >> >> >>>> brokers
> >> >> >>>>> > disks and back again, then again encrypted and
> decrypted
> >> back
> >> >> >> over
> >> >> >>>> the wire
> >> >> >>>>> > each time for each consumer all adds up, ignoring
> this
> >> >> doubling
> >> >> >>>> with mirror
> >> >> >>>>> > makers etc. simply encrypting the value once on
> write by the
> >> >> >>>> client and
> >> >> >>>>> > again decrypting on consume by the consumer is far
> more
> >> >> >>>> performant, but
> >> >> >>>>> > then the routing and platform meta data needs to be
> separate
> >> >> >> (thus
> >> >> >>>> headers)
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > RE Auditing:
> >> >> >>>>> >
> >> >> >>>>> > Our Auditing needs are:
> >> >> >>>>> > Producer Id,
> >> >> >>>>> > Origin Cluster Id that message first produced into
> >> >> >>>>> > Origin AZ – agreed we can derive this if we have
> cluster id,
> >> >> but
> >> >> >>>> it makes
> >> >> >>>>> > resolving this for audit reporting a lot easier.
> >> >> >>>>> > Origin Region – agreed we can derive this if we have
> cluster
> >> >> id,
> >> >> >>>> but it
> >> >> >>>>> > makes resolving this for audit reporting a lot
> easier.
> >> >> >>>>> > Unique Message Identification (this is not the same
> as
> >> >> >> transaction
> >> >> >>>>> > tracing) – note offset and partition are not the
> same, as
> >> >> when we
> >> >> >>>> mirror or
> >> >> >>>>> > have for what ever system failure duplicate send,
> >> >> >>>>> > Custom Client wrapper version (where organizations
> have to
> >> >> wrap
> >> >> >>>> the kafka
> >> >> >>>>> > client for added features) so we know what version
> of the
> >> >> wrapper
> >> >> >>>> is used
> >> >> >>>>> > Producer IP address (in case of clients being in our
> vm/open
> >> >> >> stack
> >> >> >>>> infra
> >> >> >>>>> > where they can move around, producer id will stay
> the same
> >> but
> >> >> >>>> this would
> >> >> >>>>> > change)
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > RE Once and only once delivery case
> >> >> >>>>> >
> >> >> >>>>> > Using the same Message UUID for auditing we can
> achieve this
> >> >> >> quite
> >> >> >>>> simply.
> >> >> >>>>> >
> >> >> >>>>> > As per how some other brokers do this (cough qpid,
> artemis)
> >> >> >>>> message uuid
> >> >> >>>>> > are used to dedupe where message is sent and
> produced but
> >> the
> >> >> >>>> client didn’t
> >> >> >>>>> > receive the ack, and there for replays the send, by
> having a
> >> >> >>>> unique message
> >> >> >>>>> > id per message, this can be filtered out, on
> consumers where
> >> >> >>>> message
> >> >> >>>>> > delivery may occur twice for what ever reasons a
> message
> >> uuid
> >> >> can
> >> >> >>>> be used
> >> >> >>>>> > to remove duplicates being deliverd , like wise we
> can do
> >> >> this in
> >> >> >>>> the
> >> >> >>>>> > mirrormakers so if we detect a dupe message we can
> avoid
> >> >> >>>> replicating it.
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > Cheers
> >> >> >>>>> > Mike
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io>
> wrote:
> >> >> >>>>> >
> >> >> >>>>> > Since this KIP affects message format, wire
> protocol,
> >> >> apis, I
> >> >> >>>> think
> >> >> >>>>> > it's
> >> >> >>>>> > worth spending a bit more time to nail down the
> concrete
> >> >> use
> >> >> >>>> cases. It
> >> >> >>>>> > would be bad if we add this feature, but when
> start
> >> >> >>>> implementing it
> >> >> >>>>> > for say
> >> >> >>>>> > mirroring, we then realize that header is not
> the best
> >> >> >>>> approach.
> >> >> >>>>> > Initially,
> >> >> >>>>> > I thought I was convinced of the use cases of
> headers
> >> and
> >> >> was
> >> >> >>>> trying to
> >> >> >>>>> > write down a few use cases to convince others.
> That's
> >> >> when I
> >> >> >>>> became
> >> >> >>>>> > less
> >> >> >>>>> > certain. For me to be convinced, I just want to
> see two
> >> >> >> strong
> >> >> >>>> use
> >> >> >>>>> > cases
> >> >> >>>>> > (instead of 10 maybe use cases) in the
> third-party
> >> space.
> >> >> The
> >> >> >>>> reason is
> >> >> >>>>> > that when we discussed the use cases within a
> company,
> >> >> often
> >> >> >>>> it ends
> >> >> >>>>> > with
> >> >> >>>>> > "we can't force everyone to use this standard
> since we
> >> may
> >> >> >>>> have to
> >> >> >>>>> > integrate with third-party tools".
> >> >> >>>>> >
> >> >> >>>>> > At present, I am not sure why headers are useful
> for
> >> >> things
> >> >> >>>> like
> >> >> >>>>> > schemaId
> >> >> >>>>> > or encryption. In order to do anything useful to
> the
> >> >> value,
> >> >> >>>> one needs
> >> >> >>>>> > to
> >> >> >>>>> > know the schemaId or how data is encrypted, but
> header
> >> is
> >> >> >>>> optional.
> >> >> >>>>> > But, I
> >> >> >>>>> > can be convinced if someone (Radai, Sean, Todd?)
> >> provides
> >> >> >> more
> >> >> >>>> details
> >> >> >>>>> > on
> >> >> >>>>> > the argument.
> >> >> >>>>> >
> >> >> >>>>> > I am not very sure header is the best approach
> for
> >> >> mirroring
> >> >> >>>> either. If
> >> >> >>>>> > someone has thought about this more, I'd be
> happy to
> >> hear.
> >> >> >>>>> >
> >> >> >>>>> > I can see the data lineage use case. I am just
> not sure
> >> >> how
> >> >> >>>> widely
> >> >> >>>>> > applicable this is. If someone familiar with
> this space
> >> >> can
> >> >> >>>> justify
> >> >> >>>>> > this is
> >> >> >>>>> > a significant use case, say in the finance
> industry,
> >> this
> >> >> >>>> would be a
> >> >> >>>>> > strong
> >> >> >>>>> > use case.
> >> >> >>>>> >
> >> >> >>>>> > I can see the auditing use case. I am just not
> sure if a
> >> >> >> native
> >> >> >>>>> > producer id
> >> >> >>>>> > solves that problem. If there are additional
> metadata
> >> >> that's
> >> >> >>>> worth
> >> >> >>>>> > collecting but not covered by the producer id,
> that
> >> would
> >> >> >> make
> >> >> >>>> this a
> >> >> >>>>> > strong use case.
> >> >> >>>>> >
> >> >> >>>>> > Thanks,
> >> >> >>>>> >
> >> >> >>>>> > Jun
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
> >> >> >>>> radai.rosenblatt@gmail.com>
> >> >> >>>>> > wrote:
> >> >> >>>>> >
> >> >> >>>>> > > this KIP is about enabling headers, nothing
> more
> >> nothing
> >> >> >>>> less - so
> >> >> >>>>> > no,
> >> >> >>>>> > > broker-side use of headers is not in the KIP
> scope.
> >> >> >>>>> > >
> >> >> >>>>> > > obviously though, once you have headers
> potential use
> >> >> cases
> >> >> >>>> could
> >> >> >>>>> > include
> >> >> >>>>> > > broker-side header-aware interceptors (which
> would be
> >> >> the
> >> >> >>>> topic of
> >> >> >>>>> > other
> >> >> >>>>> > > future KIPs).
> >> >> >>>>> > >
> >> >> >>>>> > > a trivially clear use case (to me) would be
> using such
> >> >> >>>> broker-side
> >> >> >>>>> > > interceptors to enforce compliance with
> organizational
> >> >> >>>> policies - it
> >> >> >>>>> > would
> >> >> >>>>> > > make our SREs lives much easier if instead of
> >> >> retroactively
> >> >> >>>>> > discovering
> >> >> >>>>> > > "rogue" topics/users those messages would have
> been
> >> >> >> rejected
> >> >> >>>>> > up-front.
> >> >> >>>>> > >
> >> >> >>>>> > > the kafka broker code is lacking any such
> >> extensibility
> >> >> >>>> support
> >> >> >>>>> > (beyond
> >> >> >>>>> > > maybe authorizer) which is why these use cases
> were
> >> left
> >> >> >> out
> >> >> >>>> of the
> >> >> >>>>> > "case
> >> >> >>>>> > > for headers" doc - broker extensibility is a
> separate
> >> >> >>>> discussion.
> >> >> >>>>> > >
> >> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
> >> >> >>>> gwen@confluent.io>
> >> >> >>>>> > wrote:
> >> >> >>>>> > >
> >> >> >>>>> > > > Woah, I wasn't aware this is something we'll
> do. It
> >> >> >> wasn't
> >> >> >>>> in the
> >> >> >>>>> > KIP,
> >> >> >>>>> > > > right?
> >> >> >>>>> > > >
> >> >> >>>>> > > > I guess we could do it the same way ACLs
> currently
> >> >> work.
> >> >> >>>>> > > > I had in mind something that will allow
> admins to
> >> >> apply
> >> >> >>>> rules to
> >> >> >>>>> > the
> >> >> >>>>> > > > new create/delete/config topic APIs. So Todd
> can
> >> >> decide
> >> >> >> to
> >> >> >>>> reject
> >> >> >>>>> > > > "create topic" requests that ask for more
> than 40
> >> >> >>>> partitions, or
> >> >> >>>>> > > > require exactly 3 replicas, or no more than
> 50GB
> >> >> >> partition
> >> >> >>>> size,
> >> >> >>>>> > etc.
> >> >> >>>>> > > >
> >> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are
> planning to
> >> >> apply
> >> >> >>>> more
> >> >> >>>>> > rules
> >> >> >>>>> > > > to requests (and I think we should), we may
> want a
> >> bit
> >> >> >>>> more generic
> >> >> >>>>> > > > design around that.
> >> >> >>>>> > > >
> >> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
> >> >> >>>> radai.rosenblatt@gmail.com>
> >> >> >>>>> > > wrote:
> >> >> >>>>> > > > > "wouldn't you be in the business of making
> sure
> >> >> >> everyone
> >> >> >>>> uses
> >> >> >>>>> > them
> >> >> >>>>> > > > > properly?"
> >> >> >>>>> > > > >
> >> >> >>>>> > > > > thats where a broker-side plugin would
> come handy
> >> -
> >> >> any
> >> >> >>>> incoming
> >> >> >>>>> > > message
> >> >> >>>>> > > > > that does not conform to org policy (read
> - does
> >> not
> >> >> >>>> have the
> >> >> >>>>> > proper
> >> >> >>>>> > > > > headers) gets thrown out (with an error
> returned
> >> to
> >> >> >> user)
> >> >> >>>>> > > > >
> >> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd
> Palino <
> >> >> >>>> tpalino@gmail.com>
> >> >> >>>>> > wrote:
> >> >> >>>>> > > > >
> >> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on
> this one
> >> :)
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >> Producing counts to a topic is part of
> it, but
> >> >> that’s
> >> >> >>>> only
> >> >> >>>>> > part. So
> >> >> >>>>> > > you
> >> >> >>>>> > > > >> count you have 100 messages in topic A.
> When you
> >> >> >> mirror
> >> >> >>>> topic A
> >> >> >>>>> > to
> >> >> >>>>> > > > another
> >> >> >>>>> > > > >> cluster, you have 99 messages. Where was
> your
> >> >> problem?
> >> >> >>>> Or
> >> >> >>>>> > worse, you
> >> >> >>>>> > > > have
> >> >> >>>>> > > > >> 100 messages, but one producer duplicated
> >> messages
> >> >> and
> >> >> >>>> another
> >> >> >>>>> > one
> >> >> >>>>> > > lost
> >> >> >>>>> > > > >> messages. You need details about where the
> >> message
> >> >> >> came
> >> >> >>>> from in
> >> >> >>>>> > order
> >> >> >>>>> > > to
> >> >> >>>>> > > > >> pinpoint problems when they happen. Source
> >> producer
> >> >> >>>> info, where
> >> >> >>>>> > it was
> >> >> >>>>> > > > >> produced into your infrastructure, and
> when it
> >> was
> >> >> >>>> produced.
> >> >> >>>>> > This
> >> >> >>>>> > > > requires
> >> >> >>>>> > > > >> you to add the information to the message.
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >> And yes, you still need to maintain your
> clients.
> >> >> So
> >> >> >>>> maybe my
> >> >> >>>>> > original
> >> >> >>>>> > > > >> example was not the best. My thoughts on
> not
> >> >> wanting
> >> >> >> to
> >> >> >>>> be
> >> >> >>>>> > responsible
> >> >> >>>>> > > > for
> >> >> >>>>> > > > >> message formats stands, because that’s
> very much
> >> >> >>>> separate from
> >> >> >>>>> > the
> >> >> >>>>> > > > client.
> >> >> >>>>> > > > >> As you know, we have our own internal
> client
> >> >> library
> >> >> >>>> that can
> >> >> >>>>> > insert
> >> >> >>>>> > > the
> >> >> >>>>> > > > >> right headers, and right now inserts the
> right
> >> >> audit
> >> >> >>>>> > information into
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> message fields. If they exist, and
> assuming the
> >> >> >> message
> >> >> >>>> is Avro
> >> >> >>>>> > > encoded.
> >> >> >>>>> > > > >> What if someone wants to use JSON instead
> for a
> >> >> good
> >> >> >>>> reason?
> >> >> >>>>> > What if
> >> >> >>>>> > > > user X
> >> >> >>>>> > > > >> wants to encrypt messages, but user Y
> does not?
> >> >> >>>> Maintaining the
> >> >> >>>>> > client
> >> >> >>>>> > > > >> library is still much easier than
> maintaining the
> >> >> >>>> message
> >> >> >>>>> > formats.
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >> -Todd
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen
> Shapira <
> >> >> >>>> gwen@confluent.io
> >> >> >>>>> > >
> >> >> >>>>> > > wrote:
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >> > Based on your last sentence, consider me
> >> >> convinced
> >> >> >> :)
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >> > I get why headers are critical for
> Mirroring
> >> (you
> >> >> >>>> need tags to
> >> >> >>>>> > > prevent
> >> >> >>>>> > > > >> > loops and sometimes to route messages
> to the
> >> >> correct
> >> >> >>>>> > destination).
> >> >> >>>>> > > > >> > But why do you need headers to audit?
> We are
> >> >> >> auditing
> >> >> >>>> by
> >> >> >>>>> > producing
> >> >> >>>>> > > > >> > counts to a side topic (and I was under
> the
> >> >> >>>> impression you do
> >> >> >>>>> > the
> >> >> >>>>> > > > >> > same), so we never need to modify the
> message.
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >> > Another thing - after we added headers,
> >> wouldn't
> >> >> you
> >> >> >>>> be in the
> >> >> >>>>> > > > >> > business of making sure everyone uses
> them
> >> >> properly?
> >> >> >>>> Making
> >> >> >>>>> > sure
> >> >> >>>>> > > > >> > everyone includes the right headers you
> need,
> >> not
> >> >> >>>> using the
> >> >> >>>>> > header
> >> >> >>>>> > > > >> > names you intend to use, etc. I don't
> think the
> >> >> >>>> "policing"
> >> >> >>>>> > business
> >> >> >>>>> > > > >> > will ever go away.
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd
> Palino <
> >> >> >>>>> > tpalino@gmail.com>
> >> >> >>>>> > > > wrote:
> >> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not very
> happy
> >> with
> >> >> the
> >> >> >>>>> > workaround.
> >> >> >>>>> > > Avro
> >> >> >>>>> > > > >> means
> >> >> >>>>> > > > >> > > that I have to be concerned with the
> format
> >> of
> >> >> the
> >> >> >>>> messages
> >> >> >>>>> > in
> >> >> >>>>> > > > order to
> >> >> >>>>> > > > >> > run
> >> >> >>>>> > > > >> > > the infrastructure (audit, mirroring,
> etc.).
> >> >> That
> >> >> >>>> means
> >> >> >>>>> > that I
> >> >> >>>>> > > have
> >> >> >>>>> > > > to
> >> >> >>>>> > > > >> > > handle the schemas, and I have to
> enforce
> >> rules
> >> >> >>>> about good
> >> >> >>>>> > > formats.
> >> >> >>>>> > > > >> This
> >> >> >>>>> > > > >> > is
> >> >> >>>>> > > > >> > > not something I want to be in the
> business
> >> of,
> >> >> >>>> because I
> >> >> >>>>> > should be
> >> >> >>>>> > > > able
> >> >> >>>>> > > > >> > to
> >> >> >>>>> > > > >> > > run a service infrastructure without
> needing
> >> >> to be
> >> >> >>>> in the
> >> >> >>>>> > weeds of
> >> >> >>>>> > > > >> > dealing
> >> >> >>>>> > > > >> > > with customer data formats.
> >> >> >>>>> > > > >> > >
> >> >> >>>>> > > > >> > > Trust me, a sizable portion of my
> support
> >> time
> >> >> is
> >> >> >>>> spent
> >> >> >>>>> > dealing
> >> >> >>>>> > > with
> >> >> >>>>> > > > >> > schema
> >> >> >>>>> > > > >> > > issues. I really would like to get
> away from
> >> >> that.
> >> >> >>>> Maybe
> >> >> >>>>> > I'd have
> >> >> >>>>> > > > more
> >> >> >>>>> > > > >> > time
> >> >> >>>>> > > > >> > > for other hobbies. Like writing. ;)
> >> >> >>>>> > > > >> > >
> >> >> >>>>> > > > >> > > -Todd
> >> >> >>>>> > > > >> > >
> >> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen
> Shapira <
> >> >> >>>>> > gwen@confluent.io>
> >> >> >>>>> > > > wrote:
> >> >> >>>>> > > > >> > >
> >> >> >>>>> > > > >> > >> I'm pretty satisfied with the current
> >> >> workarounds
> >> >> >>>> (Avro
> >> >> >>>>> > container
> >> >> >>>>> > > > >> > >> format), so I'm not too excited
> about the
> >> >> extra
> >> >> >>>> work
> >> >> >>>>> > required to
> >> >> >>>>> > > do
> >> >> >>>>> > > > >> > >> headers in Kafka. I absolutely don't
> mind it
> >> >> if
> >> >> >>>> you do
> >> >> >>>>> > it...
> >> >> >>>>> > > > >> > >> I think the Apache convention for
> "good
> >> idea,
> >> >> but
> >> >> >>>> not
> >> >> >>>>> > willing to
> >> >> >>>>> > > > put
> >> >> >>>>> > > > >> > >> any work toward it" is +0.5? anyway,
> that's
> >> >> what
> >> >> >> I
> >> >> >>>> was
> >> >> >>>>> > trying to
> >> >> >>>>> > > > >> > >> convey :)
> >> >> >>>>> > > > >> > >>
> >> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd
> Palino
> >> <
> >> >> >>>>> > tpalino@gmail.com>
> >> >> >>>>> > > > >> wrote:
> >> >> >>>>> > > > >> > >> > Well I guess my question for you,
> then, is
> >> >> what
> >> >> >>>> is
> >> >> >>>>> > holding you
> >> >> >>>>> > > > back
> >> >> >>>>> > > > >> > from
> >> >> >>>>> > > > >> > >> > full support for headers? What’s
> the bit
> >> >> that
> >> >> >>>> you’re
> >> >> >>>>> > missing
> >> >> >>>>> > > that
> >> >> >>>>> > > > >> has
> >> >> >>>>> > > > >> > you
> >> >> >>>>> > > > >> > >> > under a full +1?
> >> >> >>>>> > > > >> > >> >
> >> >> >>>>> > > > >> > >> > -Todd
> >> >> >>>>> > > > >> > >> >
> >> >> >>>>> > > > >> > >> >
> >> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM,
> Gwen
> >> >> Shapira <
> >> >> >>>>> > > gwen@confluent.io>
> >> >> >>>>> > > > >> > wrote:
> >> >> >>>>> > > > >> > >> >
> >> >> >>>>> > > > >> > >> >> I know why people who support
> headers
> >> >> support
> >> >> >>>> them, and
> >> >> >>>>> > I've
> >> >> >>>>> > > > seen
> >> >> >>>>> > > > >> > what
> >> >> >>>>> > > > >> > >> >> the discussion is like.
> >> >> >>>>> > > > >> > >> >>
> >> >> >>>>> > > > >> > >> >> This is why I'm asking people who
> are
> >> >> against
> >> >> >>>> headers
> >> >> >>>>> > > > (especially
> >> >> >>>>> > > > >> > >> >> committers) what will make them
> change
> >> >> their
> >> >> >>>> mind - so
> >> >> >>>>> > we can
> >> >> >>>>> > > > get
> >> >> >>>>> > > > >> > this
> >> >> >>>>> > > > >> > >> >> part over one way or another.
> >> >> >>>>> > > > >> > >> >>
> >> >> >>>>> > > > >> > >> >> If I sound frustrated it is not
> at Radai,
> >> >> Jun
> >> >> >>>> or you
> >> >> >>>>> > (Todd)...
> >> >> >>>>> > > > I am
> >> >> >>>>> > > > >> > >> >> just looking for something
> concrete we
> >> can
> >> >> do
> >> >> >>>> to move
> >> >> >>>>> > the
> >> >> >>>>> > > > >> discussion
> >> >> >>>>> > > > >> > >> >> along to the yummy design details
> (which
> >> is
> >> >> >> the
> >> >> >>>>> > argument I
> >> >> >>>>> > > > really
> >> >> >>>>> > > > >> am
> >> >> >>>>> > > > >> > >> >> looking forward to).
> >> >> >>>>> > > > >> > >> >>
> >> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM,
> Todd
> >> >> Palino <
> >> >> >>>>> > > tpalino@gmail.com>
> >> >> >>>>> > > > >> > wrote:
> >> >> >>>>> > > > >> > >> >> > So, Gwen, to your question
> (even though
> >> >> I’m
> >> >> >>>> not a
> >> >> >>>>> > > > committer)...
> >> >> >>>>> > > > >> > >> >> >
> >> >> >>>>> > > > >> > >> >> > I have always been a strong
> supporter
> >> of
> >> >> >>>> introducing
> >> >> >>>>> > the
> >> >> >>>>> > > > concept
> >> >> >>>>> > > > >> > of an
> >> >> >>>>> > > > >> > >> >> > envelope to messages, which
> headers
> >> >> >>>> accomplishes. The
> >> >> >>>>> > > message
> >> >> >>>>> > > > key
> >> >> >>>>> > > > >> > is
> >> >> >>>>> > > > >> > >> >> > already an example of a piece of
> >> envelope
> >> >> >>>>> > information. By
> >> >> >>>>> > > > >> > providing a
> >> >> >>>>> > > > >> > >> >> means
> >> >> >>>>> > > > >> > >> >> > to do this within Kafka itself,
> and not
> >> >> >>>> relying on
> >> >> >>>>> > use-case
> >> >> >>>>> > > > >> > specific
> >> >> >>>>> > > > >> > >> >> > implementations, you make it
> much
> >> easier
> >> >> for
> >> >> >>>>> > components to
> >> >> >>>>> > > > >> > >> interoperate.
> >> >> >>>>> > > > >> > >> >> It
> >> >> >>>>> > > > >> > >> >> > simplifies development of all
> these
> >> >> things
> >> >> >>>> (message
> >> >> >>>>> > routing,
> >> >> >>>>> > > > >> > auditing,
> >> >> >>>>> > > > >> > >> >> > encryption, etc.) because each
> one does
> >> >> not
> >> >> >>>> have to
> >> >> >>>>> > reinvent
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> wheel.
> >> >> >>>>> > > > >> > >> >> >
> >> >> >>>>> > > > >> > >> >> > It also makes it much easier
> from a
> >> >> client
> >> >> >>>> point of
> >> >> >>>>> > view if
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> headers
> >> >> >>>>> > > > >> > >> >> are
> >> >> >>>>> > > > >> > >> >> > defined as part of the protocol
> and/or
> >> >> >>>> message format
> >> >> >>>>> > in
> >> >> >>>>> > > > general
> >> >> >>>>> > > > >> > >> because
> >> >> >>>>> > > > >> > >> >> > you can easily produce and
> consume
> >> >> messages
> >> >> >>>> without
> >> >> >>>>> > having
> >> >> >>>>> > > to
> >> >> >>>>> > > > >> take
> >> >> >>>>> > > > >> > >> into
> >> >> >>>>> > > > >> > >> >> > account specific cases. For
> example, I
> >> >> want
> >> >> >>>> to route
> >> >> >>>>> > > messages,
> >> >> >>>>> > > > >> but
> >> >> >>>>> > > > >> > >> >> client A
> >> >> >>>>> > > > >> > >> >> > doesn’t support the way audit
> >> implemented
> >> >> >>>> headers, and
> >> >> >>>>> > > client
> >> >> >>>>> > > > B
> >> >> >>>>> > > > >> > >> doesn’t
> >> >> >>>>> > > > >> > >> >> > support the way encryption or
> routing
> >> >> >>>> implemented
> >> >> >>>>> > headers,
> >> >> >>>>> > > so
> >> >> >>>>> > > > now
> >> >> >>>>> > > > >> > my
> >> >> >>>>> > > > >> > >> >> > application has to create some
> really
> >> >> >> fragile
> >> >> >>>> (my
> >> >> >>>>> > > autocorrect
> >> >> >>>>> > > > >> just
> >> >> >>>>> > > > >> > >> tried
> >> >> >>>>> > > > >> > >> >> to
> >> >> >>>>> > > > >> > >> >> > make that “tragic”, which is
> probably
> >> >> >>>> appropriate
> >> >> >>>>> > too) code
> >> >> >>>>> > > to
> >> >> >>>>> > > > >> > strip
> >> >> >>>>> > > > >> > >> >> > everything off, rather than just
> >> >> consuming
> >> >> >> the
> >> >> >>>>> > messages,
> >> >> >>>>> > > > picking
> >> >> >>>>> > > > >> > out
> >> >> >>>>> > > > >> > >> the
> >> >> >>>>> > > > >> > >> >> 1
> >> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested
> in, and
> >> >> >>>> performing its
> >> >> >>>>> > > function.
> >> >> >>>>> > > > >> > >> >> >
> >> >> >>>>> > > > >> > >> >> > Honestly, this discussion has
> been
> >> going
> >> >> on
> >> >> >>>> for a
> >> >> >>>>> > long time,
> >> >> >>>>> > > > and
> >> >> >>>>> > > > >> > it’s
> >> >> >>>>> > > > >> > >> >> > always “Oh, you came up with 2
> use
> >> cases,
> >> >> >> and
> >> >> >>>> yeah,
> >> >> >>>>> > those
> >> >> >>>>> > > use
> >> >> >>>>> > > > >> cases
> >> >> >>>>> > > > >> > >> are
> >> >> >>>>> > > > >> > >> >> > real things that someone would
> want to
> >> >> do.
> >> >> >>>> Here’s an
> >> >> >>>>> > > alternate
> >> >> >>>>> > > > >> way
> >> >> >>>>> > > > >> > to
> >> >> >>>>> > > > >> > >> >> > implement them so let’s not do
> >> headers.”
> >> >> If
> >> >> >>>> we have a
> >> >> >>>>> > few
> >> >> >>>>> > > use
> >> >> >>>>> > > > >> cases
> >> >> >>>>> > > > >> > >> that
> >> >> >>>>> > > > >> > >> >> we
> >> >> >>>>> > > > >> > >> >> > actually came up with, you can
> be sure
> >> >> that
> >> >> >>>> over the
> >> >> >>>>> > next
> >> >> >>>>> > > year
> >> >> >>>>> > > > >> > >> there’s a
> >> >> >>>>> > > > >> > >> >> > dozen others that we didn’t
> think of
> >> that
> >> >> >>>> someone
> >> >> >>>>> > would like
> >> >> >>>>> > > > to
> >> >> >>>>> > > > >> > do. I
> >> >> >>>>> > > > >> > >> >> > really think it’s time to stop
> >> rehashing
> >> >> >> this
> >> >> >>>>> > discussion and
> >> >> >>>>> > > > >> > instead
> >> >> >>>>> > > > >> > >> >> focus
> >> >> >>>>> > > > >> > >> >> > on a workable standard that we
> can
> >> adopt.
> >> >> >>>>> > > > >> > >> >> >
> >> >> >>>>> > > > >> > >> >> > -Todd
> >> >> >>>>> > > > >> > >> >> >
> >> >> >>>>> > > > >> > >> >> >
> >> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM,
> Todd
> >> >> Palino
> >> >> >> <
> >> >> >>>>> > > > tpalino@gmail.com>
> >> >> >>>>> > > > >> > >> wrote:
> >> >> >>>>> > > > >> > >> >> >
> >> >> >>>>> > > > >> > >> >> >> C. per message encryption
> >> >> >>>>> > > > >> > >> >> >>> One drawback of this approach
> is that
> >> >> this
> >> >> >>>>> > significantly
> >> >> >>>>> > > > reduce
> >> >> >>>>> > > > >> > the
> >> >> >>>>> > > > >> > >> >> >>> effectiveness of compression,
> which
> >> >> >> happens
> >> >> >>>> on a
> >> >> >>>>> > set of
> >> >> >>>>> > > > >> > serialized
> >> >> >>>>> > > > >> > >> >> >>> messages. An alternative is
> to enable
> >> >> SSL
> >> >> >>>> for wire
> >> >> >>>>> > > > encryption
> >> >> >>>>> > > > >> and
> >> >> >>>>> > > > >> > >> rely
> >> >> >>>>> > > > >> > >> >> on
> >> >> >>>>> > > > >> > >> >> >>> the storage system (e.g.
> LUKS) for at
> >> >> rest
> >> >> >>>>> > encryption.
> >> >> >>>>> > > > >> > >> >> >>
> >> >> >>>>> > > > >> > >> >> >>
> >> >> >>>>> > > > >> > >> >> >> Jun, this is not sufficient.
> While
> >> this
> >> >> >> does
> >> >> >>>> cover
> >> >> >>>>> > the case
> >> >> >>>>> > > > of
> >> >> >>>>> > > > >> > >> removing
> >> >> >>>>> > > > >> > >> >> a
> >> >> >>>>> > > > >> > >> >> >> drive from the system, it will
> not
> >> >> satisfy
> >> >> >>>> most
> >> >> >>>>> > compliance
> >> >> >>>>> > > > >> > >> requirements
> >> >> >>>>> > > > >> > >> >> for
> >> >> >>>>> > > > >> > >> >> >> encryption of data as whoever
> has
> >> >> access to
> >> >> >>>> the
> >> >> >>>>> > broker
> >> >> >>>>> > > itself
> >> >> >>>>> > > > >> > still
> >> >> >>>>> > > > >> > >> has
> >> >> >>>>> > > > >> > >> >> >> access to the unencrypted
> data. For
> >> >> >>>> end-to-end
> >> >> >>>>> > encryption
> >> >> >>>>> > > you
> >> >> >>>>> > > > >> > need to
> >> >> >>>>> > > > >> > >> >> >> encrypt at the producer,
> before it
> >> >> enters
> >> >> >> the
> >> >> >>>>> > system, and
> >> >> >>>>> > > > >> decrypt
> >> >> >>>>> > > > >> > at
> >> >> >>>>> > > > >> > >> the
> >> >> >>>>> > > > >> > >> >> >> consumer, after it exits the
> system.
> >> >> >>>>> > > > >> > >> >> >>
> >> >> >>>>> > > > >> > >> >> >> -Todd
> >> >> >>>>> > > > >> > >> >> >>
> >> >> >>>>> > > > >> > >> >> >>
> >> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03
> PM, radai
> >> <
> >> >> >>>>> > > > >> radai.rosenblatt@gmail.com
> >> >> >>>>> > > > >> > >
> >> >> >>>>> > > > >> > >> >> wrote:
> >> >> >>>>> > > > >> > >> >> >>
> >> >> >>>>> > > > >> > >> >> >>> another big plus of headers
> in the
> >> >> >> protocol
> >> >> >>>> is that
> >> >> >>>>> > it
> >> >> >>>>> > > would
> >> >> >>>>> > > > >> > enable
> >> >> >>>>> > > > >> > >> >> rapid
> >> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside of
> core
> >> >> kafka
> >> >> >>>> and would
> >> >> >>>>> > reduce
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> number of
> >> >> >>>>> > > > >> > >> >> >>> future wire format changes
> required.
> >> >> >>>>> > > > >> > >> >> >>>
> >> >> >>>>> > > > >> > >> >> >>> a lot of what is currently a
> KIP
> >> >> >> represents
> >> >> >>>> use
> >> >> >>>>> > cases that
> >> >> >>>>> > > > are
> >> >> >>>>> > > > >> > not
> >> >> >>>>> > > > >> > >> 100%
> >> >> >>>>> > > > >> > >> >> >>> relevant to all users, and
> some of
> >> them
> >> >> >>>> require
> >> >> >>>>> > rather
> >> >> >>>>> > > > invasive
> >> >> >>>>> > > > >> > wire
> >> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing a
> good
> >> recent
> >> >> >>>> example of
> >> >> >>>>> > this is
> >> >> >>>>> > > > >> > kip-98.
> >> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is
> expected to
> >> be
> >> >> a
> >> >> >>>> very small
> >> >> >>>>> > > > fraction of
> >> >> >>>>> > > > >> > >> total
> >> >> >>>>> > > > >> > >> >> >>> traffic and yet the changes
> are
> >> >> invasive.
> >> >> >>>>> > > > >> > >> >> >>>
> >> >> >>>>> > > > >> > >> >> >>> every such wire format change
> >> >> translates
> >> >> >>>> into
> >> >> >>>>> > painful and
> >> >> >>>>> > > > slow
> >> >> >>>>> > > > >> > >> >> adoption of
> >> >> >>>>> > > > >> > >> >> >>> new versions.
> >> >> >>>>> > > > >> > >> >> >>>
> >> >> >>>>> > > > >> > >> >> >>> i think a lot of functionality
> >> >> currently
> >> >> >> in
> >> >> >>>> KIPs
> >> >> >>>>> > could be
> >> >> >>>>> > > > "spun
> >> >> >>>>> > > > >> > out"
> >> >> >>>>> > > > >> > >> >> and
> >> >> >>>>> > > > >> > >> >> >>> implemented as opt-in plugins
> >> >> transmitting
> >> >> >>>> data over
> >> >> >>>>> > > > headers.
> >> >> >>>>> > > > >> > this
> >> >> >>>>> > > > >> > >> >> would
> >> >> >>>>> > > > >> > >> >> >>> keep the core wire format
> stable(r),
> >> >> core
> >> >> >>>> codebase
> >> >> >>>>> > > smaller,
> >> >> >>>>> > > > and
> >> >> >>>>> > > > >> > >> avoid
> >> >> >>>>> > > > >> > >> >> the
> >> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats
> sometimes
> >> >> required
> >> >> >>>> to prove
> >> >> >>>>> > a
> >> >> >>>>> > > > certain
> >> >> >>>>> > > > >> > >> feature
> >> >> >>>>> > > > >> > >> >> is
> >> >> >>>>> > > > >> > >> >> >>> useful enough for a
> wide-enough
> >> >> audience
> >> >> >> to
> >> >> >>>> warrant
> >> >> >>>>> > a wire
> >> >> >>>>> > > > >> format
> >> >> >>>>> > > > >> > >> >> change
> >> >> >>>>> > > > >> > >> >> >>> and code complexity additions.
> >> >> >>>>> > > > >> > >> >> >>>
> >> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes
> beyond
> >> >> "mere"
> >> >> >>>> wire format
> >> >> >>>>> > > changes
> >> >> >>>>> > > > >> and
> >> >> >>>>> > > > >> > im
> >> >> >>>>> > > > >> > >> not
> >> >> >>>>> > > > >> > >> >> >>> saying it could have been
> completely
> >> >> done
> >> >> >>>> with
> >> >> >>>>> > headers,
> >> >> >>>>> > > but
> >> >> >>>>> > > > >> > >> >> exactly-once
> >> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
> >> >> >>>>> > > > >> > >> >> >>>
> >> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20
> AM, Gwen
> >> >> >>>> Shapira <
> >> >> >>>>> > > > >> gwen@confluent.io
> >> >> >>>>> > > > >> > >
> >> >> >>>>> > > > >> > >> >> wrote:
> >> >> >>>>> > > > >> > >> >> >>>
> >> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at
> 10:24 AM,
> >> >> radai <
> >> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
> >> >> >>>>> > > > >> > >> >> >>> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > > "For use cases within an
> >> >> organization,
> >> >> >>>> one could
> >> >> >>>>> > > always
> >> >> >>>>> > > > use
> >> >> >>>>> > > > >> > >> other
> >> >> >>>>> > > > >> > >> >> >>> > > approaches such as
> company-wise
> >> >> >>>> containers"
> >> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin has
> >> >> >> traditionally
> >> >> >>>> done
> >> >> >>>>> > but there
> >> >> >>>>> > > > are
> >> >> >>>>> > > > >> > now
> >> >> >>>>> > > > >> > >> >> cases
> >> >> >>>>> > > > >> > >> >> >>> > (read
> >> >> >>>>> > > > >> > >> >> >>> > > - topics) where this is
> not
> >> >> >> acceptable.
> >> >> >>>> this
> >> >> >>>>> > makes
> >> >> >>>>> > > > headers
> >> >> >>>>> > > > >> > >> useful
> >> >> >>>>> > > > >> > >> >> even
> >> >> >>>>> > > > >> > >> >> >>> > > within single orgs for
> cases
> >> where
> >> >> >>>>> > > > one-container-fits-all
> >> >> >>>>> > > > >> > cannot
> >> >> >>>>> > > > >> > >> >> >>> apply.
> >> >> >>>>> > > > >> > >> >> >>> > >
> >> >> >>>>> > > > >> > >> >> >>> > > as for the particular use
> cases
> >> >> >> listed,
> >> >> >>>> i dont
> >> >> >>>>> > want
> >> >> >>>>> > > > this to
> >> >> >>>>> > > > >> > >> devolve
> >> >> >>>>> > > > >> > >> >> >>> to a
> >> >> >>>>> > > > >> > >> >> >>> > > discussion of particular
> use
> >> cases
> >> >> - i
> >> >> >>>> think its
> >> >> >>>>> > > enough
> >> >> >>>>> > > > >> that
> >> >> >>>>> > > > >> > >> some
> >> >> >>>>> > > > >> > >> >> of
> >> >> >>>>> > > > >> > >> >> >>> them
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > I think a main point of
> contention
> >> is
> >> >> >>>> that: We
> >> >> >>>>> > > identified
> >> >> >>>>> > > > few
> >> >> >>>>> > > > >> > >> >> >>> > use-cases where headers are
> useful,
> >> >> do
> >> >> >> we
> >> >> >>>> want
> >> >> >>>>> > Kafka to
> >> >> >>>>> > > > be a
> >> >> >>>>> > > > >> > >> system
> >> >> >>>>> > > > >> > >> >> >>> > that supports those
> use-cases?
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
> >> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely useful
> >> >> record-level
> >> >> >>>> lineage
> >> >> >>>>> > is
> >> >> >>>>> > > though
> >> >> >>>>> > > > >> > since
> >> >> >>>>> > > > >> > >> the
> >> >> >>>>> > > > >> > >> >> >>> > overhead could
> >> >> >>>>> > > > >> > >> >> >>> > be significant."
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports
> record level
> >> >> >>>> lineage. I
> >> >> >>>>> > don't
> >> >> >>>>> > > think
> >> >> >>>>> > > > it
> >> >> >>>>> > > > >> > was
> >> >> >>>>> > > > >> > >> >> >>> > developed for lols, I think
> it is
> >> >> safe
> >> >> >> to
> >> >> >>>> assume
> >> >> >>>>> > that
> >> >> >>>>> > > the
> >> >> >>>>> > > > NSA
> >> >> >>>>> > > > >> > >> needed
> >> >> >>>>> > > > >> > >> >> >>> > that functionality. We also
> know
> >> that
> >> >> >>>> certain
> >> >> >>>>> > financial
> >> >> >>>>> > > > >> > institutes
> >> >> >>>>> > > > >> > >> >> >>> > need to track tampering with
> >> records
> >> >> at
> >> >> >> a
> >> >> >>>> record
> >> >> >>>>> > level
> >> >> >>>>> > > and
> >> >> >>>>> > > > >> > there
> >> >> >>>>> > > > >> > >> are
> >> >> >>>>> > > > >> > >> >> >>> > federal regulations that
> absolutely
> >> >> >>>> require
> >> >> >>>>> > this. They
> >> >> >>>>> > > > also
> >> >> >>>>> > > > >> > need
> >> >> >>>>> > > > >> > >> to
> >> >> >>>>> > > > >> > >> >> >>> > prove that routing apps that
> >> >> "touches"
> >> >> >> the
> >> >> >>>>> > messages and
> >> >> >>>>> > > > >> either
> >> >> >>>>> > > > >> > >> reads
> >> >> >>>>> > > > >> > >> >> >>> > or updates headers couldn't
> have
> >> >> >> possibly
> >> >> >>>>> > modified the
> >> >> >>>>> > > > >> payload
> >> >> >>>>> > > > >> > >> >> itself.
> >> >> >>>>> > > > >> > >> >> >>> > They use record level
> encryption to
> >> >> do
> >> >> >>>> that -
> >> >> >>>>> > apps can
> >> >> >>>>> > > > read
> >> >> >>>>> > > > >> and
> >> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify headers
> but
> >> can't
> >> >> >>>> touch the
> >> >> >>>>> > payload.
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > We can totally say "those
> are
> >> corner
> >> >> >>>> cases and
> >> >> >>>>> > not worth
> >> >> >>>>> > > > >> adding
> >> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for", they
> should
> >> >> use a
> >> >> >>>> different
> >> >> >>>>> > > pubsub
> >> >> >>>>> > > > >> > message
> >> >> >>>>> > > > >> > >> for
> >> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the
> other 1000
> >> >> that
> >> >> >>>> cater
> >> >> >>>>> > > > specifically
> >> >> >>>>> > > > >> to
> >> >> >>>>> > > > >> > the
> >> >> >>>>> > > > >> > >> >> >>> > financial industry).
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > But this gets us into a
> catch 22:
> >> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific
> use-case,
> >> >> >>>> someone can
> >> >> >>>>> > always
> >> >> >>>>> > > say
> >> >> >>>>> > > > it
> >> >> >>>>> > > > >> > isn't
> >> >> >>>>> > > > >> > >> >> >>> > interesting enough for
> Kafka. If we
> >> >> >>>> discuss more
> >> >> >>>>> > general
> >> >> >>>>> > > > >> > trends,
> >> >> >>>>> > > > >> > >> >> >>> > others can say "well, we
> are not
> >> sure
> >> >> >> any
> >> >> >>>> of them
> >> >> >>>>> > really
> >> >> >>>>> > > > >> needs
> >> >> >>>>> > > > >> > >> >> headers
> >> >> >>>>> > > > >> > >> >> >>> > specifically. This is just
> hand
> >> >> waving
> >> >> >>>> and not
> >> >> >>>>> > > > interesting.".
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > I think discussing
> use-cases in
> >> >> >> specifics
> >> >> >>>> is super
> >> >> >>>>> > > > important
> >> >> >>>>> > > > >> to
> >> >> >>>>> > > > >> > >> >> decide
> >> >> >>>>> > > > >> > >> >> >>> > implementation details for
> headers
> >> >> (my
> >> >> >>>> use-cases
> >> >> >>>>> > lean
> >> >> >>>>> > > > toward
> >> >> >>>>> > > > >> > >> >> numerical
> >> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and
> object
> >> >> values,
> >> >> >>>> others
> >> >> >>>>> > differ),
> >> >> >>>>> > > > but I
> >> >> >>>>> > > > >> > >> think
> >> >> >>>>> > > > >> > >> >> we
> >> >> >>>>> > > > >> > >> >> >>> > need to answer the general
> "Are we
> >> >> going
> >> >> >>>> to have
> >> >> >>>>> > > headers"
> >> >> >>>>> > > > >> > question
> >> >> >>>>> > > > >> > >> >> >>> > first.
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from the
> other
> >> >> >>>> committers in the
> >> >> >>>>> > > > discussion:
> >> >> >>>>> > > > >> > >> >> >>> > What would it take to
> convince you
> >> >> that
> >> >> >>>> headers
> >> >> >>>>> > in Kafka
> >> >> >>>>> > > > are
> >> >> >>>>> > > > >> a
> >> >> >>>>> > > > >> > >> good
> >> >> >>>>> > > > >> > >> >> >>> > idea in general, so we can
> move
> >> ahead
> >> >> >> and
> >> >> >>>> try to
> >> >> >>>>> > agree
> >> >> >>>>> > > on
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> >> details?
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > I feel like we keep moving
> the goal
> >> >> >> posts
> >> >> >>>> and
> >> >> >>>>> > this is
> >> >> >>>>> > > > truly
> >> >> >>>>> > > > >> > >> >> exhausting.
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly
> support
> >> >> adding
> >> >> >>>> headers
> >> >> >>>>> > to Kafka
> >> >> >>>>> > > > >> > (+0.5?).
> >> >> >>>>> > > > >> > >> >> >>> > The community can continue
> to find
> >> >> >>>> workarounds to
> >> >> >>>>> > the
> >> >> >>>>> > > > issue
> >> >> >>>>> > > > >> and
> >> >> >>>>> > > > >> > >> there
> >> >> >>>>> > > > >> > >> >> >>> > are some benefits to
> keeping the
> >> >> message
> >> >> >>>> format
> >> >> >>>>> > and
> >> >> >>>>> > > > clients
> >> >> >>>>> > > > >> > >> simpler.
> >> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness of
> headers
> >> >> to
> >> >> >>>> many
> >> >> >>>>> > use-cases
> >> >> >>>>> > > and
> >> >> >>>>> > > > if
> >> >> >>>>> > > > >> we
> >> >> >>>>> > > > >> > >> can
> >> >> >>>>> > > > >> > >> >> >>> > find a good and generally
> useful
> >> way
> >> >> to
> >> >> >>>> add it to
> >> >> >>>>> > Kafka,
> >> >> >>>>> > > > it
> >> >> >>>>> > > > >> > will
> >> >> >>>>> > > > >> > >> make
> >> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for
> many -
> >> worthy
> >> >> >>>> goal in my
> >> >> >>>>> > eyes.
> >> >> >>>>> > > > >> > >> >> >>> >
> >> >> >>>>> > > > >> > >> >> >>> > > are interesting/feasible,
> but:
> >> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are
> use cases
> >> >> for
> >> >> >>>> polyglot
> >> >> >>>>> > topics.
> >> >> >>>>> > > > >> > >> especially if
> >> >> >>>>> > > > >> > >> >> >>> kafka
> >> >> >>>>> > > > >> > >> >> >>> > > is being used to "trunk"
> >> something
> >> >> >> else.
> >> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics would
> make it
> >> >> >> harder
> >> >> >>>> to write
> >> >> >>>>> > > > portable
> >> >> >>>>> > > > >> > >> consumer
> >> >> >>>>> > > > >> > >> >> >>> code.
> >> >> >>>>> > > > >> > >> >> >>> > > partition remapping would
> mess
> >> with
> >> >> >>>> locality of
> >> >> >>>>> > > > consumption
> >> >> >>>>> > > > >> > >> >> >>> guarantees.
> >> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
> >> >> >>>> lineage/metadata is
> >> >> >>>>> > > > >> > >> billing/chargeback.
> >> >> >>>>> > > > >> > >> >> for
> >> >> >>>>> > > > >> > >> >> >>> > that
> >> >> >>>>> > > > >> > >> >> >>> > > use case it is not enough
> to
> >> simply
> >> >> >>>> record the
> >> >> >>>>> > point
> >> >> >>>>> > > of
> >> >> >>>>> > > > >> > origin,
> >> >> >>>>> > > > >> > >> but
> >> >> >>>>> > > > >> > >> >> >>> every
> >> >> >>>>> > > > >> > >> >> >>> > > replication stop (think
> mirror
> >> >> maker)
> >> >> >>>> must also
> >> >> >>>>> > add a
> >> >> >>>>> > > > >> record
> >> >> >>>>> > > > >> > to
> >> >> >>>>> > > > >> > >> >> form a
> >> >> >>>>> > > > >> > >> >> >>> > > "transit log".
> >> >> >>>>> > > > >> > >> >> >>> > >
> >> >> >>>>> > > > >> > >> >> >>> > > as for stream processing
> on top
> >> of
> >> >> >>>> kafka - i
> >> >> >>>>> > know
> >> >> >>>>> > > samza
> >> >> >>>>> > > > >> has a
> >> >> >>>>> > > > >> > >> >> metadata
> >> >> >>>>> > > > >> > >> >> >>> > map
> >> >> >>>>> > > > >> > >> >> >>> > > which they carry around in
> >> >> addition to
> >> >> >>>> user
> >> >> >>>>> > values.
> >> >> >>>>> > > > headers
> >> >> >>>>> > > > >> > are
> >> >> >>>>> > > > >> > >> the
> >> >> >>>>> > > > >> > >> >> >>> > perfect
> >> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
> >> >> >>>>> > > > >> > >> >> >>> > >
> >> >> >>>>> > > > >> > >> >> >>> > >
> >> >> >>>>> > > > >> > >> >> >>> > >
> >> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at
> 6:50 PM,
> >> >> Jun
> >> >> >>>> Rao <
> >> >> >>>>> > > > jun@confluent.io
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >> > >> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> In order to answer the
> first two
> >> >> >>>> questions, it
> >> >> >>>>> > would
> >> >> >>>>> > > be
> >> >> >>>>> > > > >> > helpful
> >> >> >>>>> > > > >> > >> >> if we
> >> >> >>>>> > > > >> > >> >> >>> > could
> >> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong
> use cases
> >> >> for
> >> >> >>>> headers
> >> >> >>>>> > in the
> >> >> >>>>> > > > space
> >> >> >>>>> > > > >> > for
> >> >> >>>>> > > > >> > >> >> >>> > third-party
> >> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use cases
> within an
> >> >> >>>> organization,
> >> >> >>>>> > one
> >> >> >>>>> > > > could
> >> >> >>>>> > > > >> > always
> >> >> >>>>> > > > >> > >> >> use
> >> >> >>>>> > > > >> > >> >> >>> > other
> >> >> >>>>> > > > >> > >> >> >>> > >> approaches such as
> company-wise
> >> >> >>>> containers to
> >> >> >>>>> > get
> >> >> >>>>> > > > around
> >> >> >>>>> > > > >> w/o
> >> >> >>>>> > > > >> > >> >> >>> headers. I
> >> >> >>>>> > > > >> > >> >> >>> > >> went through the use
> cases in
> >> the
> >> >> KIP
> >> >> >>>> and in
> >> >> >>>>> > Radai's
> >> >> >>>>> > > > wiki
> >> >> >>>>> > > > >> (
> >> >> >>>>> > > > >> > >> >> >>> > >>
> https://cwiki.apache.org/confl
> >> >> >>>>> > uence/display/KAFKA/A+
> >> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
> >> >> >>>>> > > > >> > >> >> >>> > >> ).
> >> >> >>>>> > > > >> > >> >> >>> > >> The following are the
> ones that
> >> >> that
> >> >> >> I
> >> >> >>>>> > understand and
> >> >> >>>>> > > > >> could
> >> >> >>>>> > > > >> > be
> >> >> >>>>> > > > >> > >> in
> >> >> >>>>> > > > >> > >> >> the
> >> >> >>>>> > > > >> > >> >> >>> > >> third-party use case
> category.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
> >> >> >>>>> > > > >> > >> >> >>> > >> It seems that in general,
> >> >> >> content-type
> >> >> >>>> should
> >> >> >>>>> > be set
> >> >> >>>>> > > at
> >> >> >>>>> > > > >> the
> >> >> >>>>> > > > >> > >> topic
> >> >> >>>>> > > > >> > >> >> >>> level.
> >> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing
> messages with
> >> >> >>>> different
> >> >> >>>>> > content
> >> >> >>>>> > > > types
> >> >> >>>>> > > > >> > >> should be
> >> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
> >> >> >>>>> > > > >> > >> >> >>> > >> Since the value is mostly
> >> useless
> >> >> >>>> without
> >> >> >>>>> > schema id,
> >> >> >>>>> > > it
> >> >> >>>>> > > > >> > seems
> >> >> >>>>> > > > >> > >> that
> >> >> >>>>> > > > >> > >> >> >>> > storing
> >> >> >>>>> > > > >> > >> >> >>> > >> the schema id together
> with
> >> >> >> serialized
> >> >> >>>> bytes
> >> >> >>>>> > in the
> >> >> >>>>> > > > value
> >> >> >>>>> > > > >> is
> >> >> >>>>> > > > >> > >> >> better?
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> C. per message encryption
> >> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this
> approach is
> >> >> that
> >> >> >>>> this
> >> >> >>>>> > > > significantly
> >> >> >>>>> > > > >> > reduce
> >> >> >>>>> > > > >> > >> >> the
> >> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of
> compression,
> >> >> which
> >> >> >>>> happens on
> >> >> >>>>> > a set
> >> >> >>>>> > > of
> >> >> >>>>> > > > >> > >> serialized
> >> >> >>>>> > > > >> > >> >> >>> > >> messages. An alternative
> is to
> >> >> enable
> >> >> >>>> SSL for
> >> >> >>>>> > wire
> >> >> >>>>> > > > >> > encryption
> >> >> >>>>> > > > >> > >> and
> >> >> >>>>> > > > >> > >> >> >>> rely
> >> >> >>>>> > > > >> > >> >> >>> > on
> >> >> >>>>> > > > >> > >> >> >>> > >> the storage system (e.g.
> LUKS)
> >> >> for at
> >> >> >>>> rest
> >> >> >>>>> > > encryption.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for
> mirroring
> >> across
> >> >> >>>> Kafka
> >> >> >>>>> > clusters
> >> >> >>>>> > > > >> > >> >> >>> > >> This is actually
> interesting.
> >> >> Today,
> >> >> >>>> to avoid
> >> >> >>>>> > > > introducing
> >> >> >>>>> > > > >> > >> cycles
> >> >> >>>>> > > > >> > >> >> when
> >> >> >>>>> > > > >> > >> >> >>> > doing
> >> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data
> centers,
> >> one
> >> >> >>>> would
> >> >> >>>>> > either have
> >> >> >>>>> > > to
> >> >> >>>>> > > > >> set
> >> >> >>>>> > > > >> > up
> >> >> >>>>> > > > >> > >> two
> >> >> >>>>> > > > >> > >> >> >>> Kafka
> >> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and an
> >> >> aggregate)
> >> >> >>>> per data
> >> >> >>>>> > center
> >> >> >>>>> > > or
> >> >> >>>>> > > > >> > rename
> >> >> >>>>> > > > >> > >> >> topics.
> >> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal. With
> headers,
> >> >> the
> >> >> >>>> producer
> >> >> >>>>> > could
> >> >> >>>>> > > tag
> >> >> >>>>> > > > >> each
> >> >> >>>>> > > > >> > >> >> message
> >> >> >>>>> > > > >> > >> >> >>> with
> >> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster ID
> in the
> >> >> >> header.
> >> >> >>>>> > MirrorMaker
> >> >> >>>>> > > > could
> >> >> >>>>> > > > >> > then
> >> >> >>>>> > > > >> > >> >> avoid
> >> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages to a
> cluster
> >> if
> >> >> >>>> they are
> >> >> >>>>> > tagged
> >> >> >>>>> > > with
> >> >> >>>>> > > > >> the
> >> >> >>>>> > > > >> > >> same
> >> >> >>>>> > > > >> > >> >> >>> cluster
> >> >> >>>>> > > > >> > >> >> >>> > >> id.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> However, an alternative
> approach
> >> >> is
> >> >> >> to
> >> >> >>>>> > introduce sth
> >> >> >>>>> > > > like
> >> >> >>>>> > > > >> > >> >> >>> hierarchical
> >> >> >>>>> > > > >> > >> >> >>> > >> topic and store messages
> from
> >> >> >> different
> >> >> >>>>> > clusters in
> >> >> >>>>> > > > >> > different
> >> >> >>>>> > > > >> > >> >> >>> partitions
> >> >> >>>>> > > > >> > >> >> >>> > >> under the same topic.
> This
> >> >> approach
> >> >> >>>> avoids
> >> >> >>>>> > filtering
> >> >> >>>>> > > > out
> >> >> >>>>> > > > >> > >> unneeded
> >> >> >>>>> > > > >> > >> >> >>> data
> >> >> >>>>> > > > >> > >> >> >>> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> makes offset preserving
> easier
> >> to
> >> >> >>>> support. It
> >> >> >>>>> > may
> >> >> >>>>> > > make
> >> >> >>>>> > > > >> > >> compaction
> >> >> >>>>> > > > >> > >> >> >>> > trickier
> >> >> >>>>> > > > >> > >> >> >>> > >> though since the same
> key may
> >> >> show up
> >> >> >>>> in
> >> >> >>>>> > different
> >> >> >>>>> > > > >> > partitions.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> E. record-level lineage
> >> >> >>>>> > > > >> > >> >> >>> > >> For example, a source
> connector
> >> >> could
> >> >> >>>> store in
> >> >> >>>>> > the
> >> >> >>>>> > > > message
> >> >> >>>>> > > > >> > the
> >> >> >>>>> > > > >> > >> >> >>> metadata
> >> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
> >> record.
> >> >> >>>> Similarly,
> >> >> >>>>> > if a
> >> >> >>>>> > > > stream
> >> >> >>>>> > > > >> job
> >> >> >>>>> > > > >> > >> >> >>> transforms
> >> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A to
> topic
> >> B,
> >> >> the
> >> >> >>>> library
> >> >> >>>>> > could
> >> >> >>>>> > > > >> include
> >> >> >>>>> > > > >> > the
> >> >> >>>>> > > > >> > >> >> >>> source
> >> >> >>>>> > > > >> > >> >> >>> > >> message offset in each
> of the
> >> >> >>>> transformed
> >> >> >>>>> > message in
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> header.
> >> >> >>>>> > > > >> > >> >> Not
> >> >> >>>>> > > > >> > >> >> >>> > sure
> >> >> >>>
> >> >> >>>>> > > > >> > >> >> >>> > >> how widely useful
> record-level
> >> >> >> lineage
> >> >> >>>> is
> >> >> >>>>> > though
> >> >> >>>>> > > since
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> >> overhead
> >> >> >>>>> > > > >> > >> >> >>> > could
> >> >> >>>>> > > > >> > >> >> >>> > >> be significant.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
> >> >> >>>>> > > > >> > >> >> >>> > >> We could put things like
> >> >> >>>> clientId/host/user in
> >> >> >>>>> > the
> >> >> >>>>> > > > header
> >> >> >>>>> > > > >> in
> >> >> >>>>> > > > >> > >> each
> >> >> >>>>> > > > >> > >> >> >>> > message
> >> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These
> metadata are
> >> >> >>>> really at the
> >> >> >>>>> > > producer
> >> >> >>>>> > > > >> > level
> >> >> >>>>> > > > >> > >> >> though.
> >> >> >>>>> > > > >> > >> >> >>> > So, a
> >> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is to
> only
> >> >> >> include a
> >> >> >>>>> > "producerId"
> >> >> >>>>> > > > per
> >> >> >>>>> > > > >> > >> message
> >> >> >>>>> > > > >> > >> >> and
> >> >> >>>>> > > > >> > >> >> >>> > send
> >> >> >>>>> > > > >> > >> >> >>> > >> the producerId ->
> metadata
> >> mapping
> >> >> >>>>> > independently.
> >> >> >>>>> > > > KIP-98
> >> >> >>>>> > > > >> is
> >> >> >>>>> > > > >> > >> >> actually
> >> >> >>>>> > > > >> > >> >> >>> > >> proposing including such
> a
> >> >> producerId
> >> >> >>>> natively
> >> >> >>>>> > in the
> >> >> >>>>> > > > >> > message.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not sure
> that I
> >> am
> >> >> >> fully
> >> >> >>>>> > convinced of
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > strong
> >> >> >>>>> > > > >> > >> >> >>> > third-party
> >> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
> >> Perhaps
> >> >> we
> >> >> >>>> could
> >> >> >>>>> > discuss a
> >> >> >>>>> > > > bit
> >> >> >>>>> > > > >> > more
> >> >> >>>>> > > > >> > >> to
> >> >> >>>>> > > > >> > >> >> make
> >> >> >>>>> > > > >> > >> >> >>> > one
> >> >> >>>>> > > > >> > >> >> >>> > >> or two really convincing
> use
> >> >> cases.
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal
> question is
> >> >> >>>> whether header
> >> >> >>>>> > should
> >> >> >>>>> > > > be
> >> >> >>>>> > > > >> > >> exposed
> >> >> >>>>> > > > >> > >> >> in
> >> >> >>>>> > > > >> > >> >> >>> > stream
> >> >> >>>>> > > > >> > >> >> >>> > >> processing systems such
> Kafka
> >> >> stream,
> >> >> >>>> Samza,
> >> >> >>>>> > and
> >> >> >>>>> > > Spark
> >> >> >>>>> > > > >> > >> streaming.
> >> >> >>>>> > > > >> > >> >> >>> > >> Currently, those systems
> just
> >> deal
> >> >> >> with
> >> >> >>>>> > key/value
> >> >> >>>>> > > > pairs.
> >> >> >>>>> > > > >> > >> Should we
> >> >> >>>>> > > > >> > >> >> >>> > expose a
> >> >> >>>>> > > > >> > >> >> >>> > >> third thing header there
> too or
> >> >> >>>> somehow map
> >> >> >>>>> > header to
> >> >> >>>>> > > > key
> >> >> >>>>> > > > >> or
> >> >> >>>>> > > > >> > >> >> value?
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> Jun
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at
> 3:35 AM,
> >> >> >>>> Michael
> >> >> >>>>> > Pearce <
> >> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that after a
> period
> >> >> of a
> >> >> >>>> week,
> >> >> >>>>> > that there
> >> >> >>>>> > > > is
> >> >> >>>>> > > > >> no
> >> >> >>>>> > > > >> > >> >> concerns
> >> >> >>>>> > > > >> > >> >> >>> now
> >> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and 2
> and now
> >> we
> >> >> >> have
> >> >> >>>>> > agreement that
> >> >> >>>>> > > > >> > headers
> >> >> >>>>> > > > >> > >> are
> >> >> >>>>> > > > >> > >> >> >>> useful
> >> >> >>>>> > > > >> > >> >> >>> > >> and
> >> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As
> such if
> >> put
> >> >> to
> >> >> >> a
> >> >> >>>> KIP
> >> >> >>>>> > vote, this
> >> >> >>>>> > > > >> > wouldn’t
> >> >> >>>>> > > > >> > >> be
> >> >> >>>>> > > > >> > >> >> a
> >> >> >>>>> > > > >> > >> >> >>> > reason
> >> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >> >>>>> > > > >> > >> >> >>> > >> > reject.
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > @
> >> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
> >> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose of
> getting
> >> >> this
> >> >> >>>> KIP
> >> >> >>>>> > moving past
> >> >> >>>>> > > > >> this,
> >> >> >>>>> > > > >> > we
> >> >> >>>>> > > > >> > >> can
> >> >> >>>>> > > > >> > >> >> >>> state
> >> >> >>>>> > > > >> > >> >> >>> > >> the
> >> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes
> space
> >> that
> >> >> >> can
> >> >> >>>> will be
> >> >> >>>>> > > > naturally
> >> >> >>>>> > > > >> > >> >> interpreted
> >> >> >>>>> > > > >> > >> >> >>> as
> >> >> >>>>> > > > >> > >> >> >>> > an
> >> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing
> is later
> >> >> >>>> wanted you can
> >> >> >>>>> > > easily
> >> >> >>>>> > > > >> split
> >> >> >>>>> > > > >> > >> this
> >> >> >>>>> > > > >> > >> >> >>> into
> >> >> >>>>> > > > >> > >> >> >>> > two
> >> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from
> the wire
> >> >> >> protocol
> >> >> >>>>> > > implementation
> >> >> >>>>> > > > >> this
> >> >> >>>>> > > > >> > >> makes
> >> >> >>>>> > > > >> > >> >> no
> >> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t
> believe. Is
> >> >> this
> >> >> >>>>> > reasonable to
> >> >> >>>>> > > > all?
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4
> therefor
> >> >> happy
> >> >> >>>> we keep
> >> >> >>>>> > with 32
> >> >> >>>>> > > > >> bits.
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
> >> >> >>>>> > ignacio.solis@gmail.com on
> >> >> >>>>> > > > >> behalf
> >> >> >>>>> > > > >> > of
> >> >> >>>>> > > > >> > >> >> >>> Ignacio
> >> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
> >> ignacio.solis@gmail.com
> >> >> on
> >> >> >>>> behalf of
> >> >> >>>>> > > > >> > isolis@igso.net
> >> >> >>>>> > > > >> > >> >
> >> >> >>>>> > > > >> > >> >> >>> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header
> value as
> >> >> byte[]
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
> >> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
> >> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String -
> Reluctant
> >> maybe
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe the
> header
> >> >> system
> >> >> >>>> should
> >> >> >>>>> > take a
> >> >> >>>>> > > > single
> >> >> >>>>> > > > >> > >> int. I
> >> >> >>>>> > > > >> > >> >> >>> think
> >> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
> >> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if
> you want
> >> to
> >> >> >>>> interpret
> >> >> >>>>> > this as
> >> >> >>>>> > > to
> >> >> >>>>> > > > >> 16bit
> >> >> >>>>> > > > >> > >> >> numbers
> >> >> >>>>> > > > >> > >> >> >>> in
> >> >> >>>>> > > > >> > >> >> >>> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > layer
> >> >> >>>>> > > > >> > >> >> >>> > >> > above go right
> ahead. If
> >> >> >>>> somebody wants
> >> >> >>>>> > to
> >> >> >>>>> > > argue
> >> >> >>>>> > > > >> for
> >> >> >>>>> > > > >> > 16
> >> >> >>>>> > > > >> > >> >> bits
> >> >> >>>>> > > > >> > >> >> >>> or
> >> >> >>>>> > > > >> > >> >> >>> > 64
> >> >> >>>>> > > > >> > >> >> >>> > >> > bits of
> >> >> >>>>> > > > >> > >> >> >>> > >> > header key space I
> would
> >> >> >> listen.
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
> >> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the key
> space
> >> into
> >> >> >>>> sub_key_1 and
> >> >> >>>>> > > > sub_key_2
> >> >> >>>>> > > > >> > >> makes no
> >> >> >>>>> > > > >> > >> >> >>> > sense to
> >> >> >>>>> > > > >> > >> >> >>> > >> > me at
> >> >> >>>>> > > > >> > >> >> >>> > >> > this layer. Are
> we going
> >> to
> >> >> >>>> start
> >> >> >>>>> > providing
> >> >> >>>>> > > > APIs to
> >> >> >>>>> > > > >> > get
> >> >> >>>>> > > > >> > >> all
> >> >> >>>>> > > > >> > >> >> >>> the
> >> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all
> the
> >> >> >>>> sub_key_2s? If
> >> >> >>>>> > there is
> >> >> >>>>> > > > no
> >> >> >>>>> > > > >> > >> >> >>> distinguishing
> >> >> >>>>> > > > >> > >> >> >>> > >> > functions
> >> >> >>>>> > > > >> > >> >> >>> > >> > that are applied
> to each
> >> one
> >> >> >>>> then they
> >> >> >>>>> > should
> >> >> >>>>> > > be
> >> >> >>>>> > > > a
> >> >> >>>>> > > > >> > single
> >> >> >>>>> > > > >> > >> >> >>> value.
> >> >> >>>>> > > > >> > >> >> >>> > At
> >> >> >>>>> > > > >> > >> >> >>> > >> > this
> >> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're
> doing is
> >> >> >>>> equality.
> >> >> >>>>> > > > >> > >> >> >>> > >> > If the above layer
> wants
> >> to
> >> >> >>>> interpret
> >> >> >>>>> > this as
> >> >> >>>>> > > 2,
> >> >> >>>>> > > > 3
> >> >> >>>>> > > > >> or
> >> >> >>>>> > > > >> > >> more
> >> >> >>>>> > > > >> > >> >> >>> values
> >> >> >>>>> > > > >> > >> >> >>> > >> > that's a
> >> >> >>>>> > > > >> > >> >> >>> > >> > different
> question. I
> >> >> >>>> personally think
> >> >> >>>>> > it's
> >> >> >>>>> > > all
> >> >> >>>>> > > > one
> >> >> >>>>> > > > >> > >> >> keyspace
> >> >> >>>>> > > > >> > >> >> >>> > that is
> >> >> >>>>> > > > >> > >> >> >>> > >> > getting assigned
> using
> >> some
> >> >> >>>> structure,
> >> >> >>>>> > but if
> >> >> >>>>> > > you
> >> >> >>>>> > > > >> > want to
> >> >> >>>>> > > > >> > >> >> >>> > sub-assign
> >> >> >>>>> > > > >> > >> >> >>> > >> > parts
> >> >> >>>>> > > > >> > >> >> >>> > >> > of it then that's
> fine.
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > The same discussion
> >> applies
> >> >> to
> >> >> >>>> strings.
> >> >> >>>>> > If
> >> >> >>>>> > > > somebody
> >> >> >>>>> > > > >> > >> argued
> >> >> >>>>> > > > >> > >> >> for
> >> >> >>>>> > > > >> > >> >> >>> > >> > strings,
> >> >> >>>>> > > > >> > >> >> >>> > >> > would we be
> arguing to
> >> >> divide
> >> >> >> the
> >> >> >>>>> > strings with
> >> >> >>>>> > > > dots
> >> >> >>>>> > > > >> > ('.')
> >> >> >>>>> > > > >> > >> >> as a
> >> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
> >> >> >>>>> > > > >> > >> >> >>> > >> > Would we want them
> to give
> >> >> us
> >> >> >> the
> >> >> >>>>> > different
> >> >> >>>>> > > name
> >> >> >>>>> > > > >> > segments
> >> >> >>>>> > > > >> > >> >> >>> > separately?
> >> >> >>>>> > > > >> > >> >> >>> > >> > Would we be
> performing any
> >> >> >>>> actions on
> >> >> >>>>> > this key
> >> >> >>>>> > > > other
> >> >> >>>>> > > > >> > than
> >> >> >>>>> > > > >> > >> >> >>> > matching?
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18,
> 2016 at
> >> 9:30
> >> >> >> AM,
> >> >> >>>> Michael
> >> >> >>>>> > > Pearce <
> >> >> >>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any
> concerns
> >> on
> >> >> 1
> >> >> >>>> and 2
> >> >> >>>>> > still?
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > @all
> >> >> >>>>> > > > >> > >> >> >>> > >> > > To get this
> moving
> >> along a
> >> >> >> bit
> >> >> >>>> more
> >> >> >>>>> > I'd also
> >> >> >>>>> > > > like
> >> >> >>>>> > > > >> to
> >> >> >>>>> > > > >> > >> ask
> >> >> >>>>> > > > >> > >> >> to
> >> >> >>>>> > > > >> > >> >> >>> get
> >> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the below last
> points:
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe
> we're all
> >> >> >> roughly
> >> >> >>>> happy
> >> >> >>>>> > with the
> >> >> >>>>> > > > >> header
> >> >> >>>>> > > > >> > >> value
> >> >> >>>>> > > > >> > >> >> >>> > being a
> >> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe
> consensus
> >> has
> >> >> >>>> been for an
> >> >> >>>>> > > > namespace
> >> >> >>>>> > > > >> > based
> >> >> >>>>> > > > >> > >> int
> >> >> >>>>> > > > >> > >> >> >>> > approach
> >> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for
> the key.
> >> Any
> >> >> >>>> objections
> >> >> >>>>> > if this
> >> >> >>>>> > > > is
> >> >> >>>>> > > > >> > what
> >> >> >>>>> > > > >> > >> we
> >> >> >>>>> > > > >> > >> >> go
> >> >> >>>>> > > > >> > >> >> >>> > with?
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
> >> >> assumption
> >> >> >> in
> >> >> >>>> (4) is
> >> >> >>>>> > > correct,
> >> >> >>>>> > > > >> > >> {int,int}
> >> >> >>>>> > > > >> > >> >> >>> keys.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > Should both
> int's be
> >> >> int16 or
> >> >> >>>> int32?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them
> being
> >> int16(2
> >> >> >>>> bytes) as
> >> >> >>>>> > combined
> >> >> >>>>> > > > is
> >> >> >>>>> > > > >> > space
> >> >> >>>>> > > > >> > >> of
> >> >> >>>>> > > > >> > >> >> >>> > 4bytes as
> >> >> >>>>> > > > >> > >> >> >>> > >> > per
> >> >> >>>>> > > > >> > >> >> >>> > >> > > original and
> gives
> >> plenty
> >> >> of
> >> >> >>>>> > combinations for
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> >> >>> foreseeable,
> >> >> >>>>> > > > >> > >> >> >>> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > keeps
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead
> small.
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any
> benefit in
> >> >> >>>> another kip
> >> >> >>>>> > call to
> >> >> >>>>> > > > >> discuss
> >> >> >>>>> > > > >> > >> >> these at
> >> >> >>>>> > > > >> > >> >> >>> > all?
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >> ______________________________
> >> >> >>>>> > __________
> >> >> >>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
> >> >> >>>> k.burstev@yandex.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday,
> November
> >> 18,
> >> >> >> 2016
> >> >> >>>>> > 7:07:07 AM
> >> >> >>>>> > > > >> > >> >> >>> > >> > > To:
> >> dev@kafka.apache.org
> >> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re:
> [DISCUSS]
> >> >> KIP-82
> >> >> >>>> - Add
> >> >> >>>>> > Record
> >> >> >>>>> > > > Headers
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > For what it is
> worth
> >> also
> >> >> i
> >> >> >>>> agree. As
> >> >> >>>>> > a user:
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes -
> Headers are
> >> >> >>>> worthwhile
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes -
> Headers should
> >> >> be a
> >> >> >>>> top level
> >> >> >>>>> > > option
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016,
> 21:15,
> >> >> "Ignacio
> >> >> >>>> Solis" <
> >> >> >>>>> > > > >> isolis@igso.net
> >> >> >>>>> > > > >> > >:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes -
> Headers are
> >> >> >>>> worthwhile
> >> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes -
> Headers
> >> should
> >> >> be
> >> >> >> a
> >> >> >>>> top
> >> >> >>>>> > level
> >> >> >>>>> > > option
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov
> 14, 2016
> >> at
> >> >> >> 9:16
> >> >> >>>> AM,
> >> >> >>>>> > Michael
> >> >> >>>>> > > > Pearce
> >> >> >>>>> > > > >> <
> >> >> >>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
> >> >> details/examples
> >> >> >>>> the
> >> >> >>>>> > original
> >> >> >>>>> > > > proposal
> >> >> >>>>> > > > >> > for
> >> >> >>>>> > > > >> > >> key
> >> >> >>>>> > > > >> > >> >> >>> > spacing
> >> >> >>>>> > > > >> > >> >> >>> > >> ,
> >> >> >>>>> > > > >> > >> >> >>> > >> > not
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> new
> mentioned as per
> >> >> >>>> discussion
> >> >> >>>>> > namespace
> >> >> >>>>> > > > >> idea.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will need
> to
> >> update
> >> >> >> the
> >> >> >>>> kip,
> >> >> >>>>> > when we
> >> >> >>>>> > > get
> >> >> >>>>> > > > >> > >> agreement
> >> >> >>>>> > > > >> > >> >> >>> this
> >> >> >>>>> > > > >> > >> >> >>> > is a
> >> >> >>>>> > > > >> > >> >> >>> > >> > better
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach
> (which
> >> seems
> >> >> to
> >> >> >>>> be the
> >> >> >>>>> > case if I
> >> >> >>>>> > > > have
> >> >> >>>>> > > > >> > >> >> understood
> >> >> >>>>> > > > >> > >> >> >>> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > general
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling in
> the
> >> >> >>>> conversation)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the
> variable
> >> ints,
> >> >> at
> >> >> >>>> very
> >> >> >>>>> > early stage
> >> >> >>>>> > > > we
> >> >> >>>>> > > > >> did
> >> >> >>>>> > > > >> > >> think
> >> >> >>>>> > > > >> > >> >> >>> about
> >> >> >>>>> > > > >> > >> >> >>> > >> > this. I
> >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added
> complexity
> >> >> for
> >> >> >>>> the
> >> >> >>>>> > saving isn't
> >> >> >>>>> > > > >> worth
> >> >> >>>>> > > > >> > it.
> >> >> >>>>> > > > >> > >> >> I'd
> >> >> >>>>> > > > >> > >> >> >>> > rather
> >> >> >>>>> > > > >> > >> >> >>> > >> go
> >> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want to
> reduce
> >> >> >>>> overheads and
> >> >> >>>>> > size
> >> >> >>>>> > > int16
> >> >> >>>>> > > > >> > (2bytes)
> >> >> >>>>> > > > >> > >> >> keys
> >> >> >>>>> > > > >> > >> >> >>> as
> >> >> >>>>> > > > >> > >> >> >>> > it
> >> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the note
> of no
> >> >> >> headers,
> >> >> >>>> there
> >> >> >>>>> > is as
> >> >> >>>>> > > per
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > kip
> >> >> >>>>> > > > >> > >> as
> >> >> >>>>> > > > >> > >> >> we
> >> >> >>>>> > > > >> > >> >> >>> > use an
> >> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to
> denote if
> >> >> headers
> >> >> >>>> are
> >> >> >>>>> > present or
> >> >> >>>>> > > > not as
> >> >> >>>>> > > > >> > such
> >> >> >>>>> > > > >> > >> >> >>> > provides a
> >> >> >>>>> > > > >> > >> >> >>> > >> > zero
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead
> currently
> >> if
> >> >> >>>> headers are
> >> >> >>>>> > not
> >> >> >>>>> > > used.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think as
> radai
> >> >> mentions
> >> >> >>>> would be
> >> >> >>>>> > good
> >> >> >>>>> > > > first
> >> >> >>>>> > > > >> > if we
> >> >> >>>>> > > > >> > >> >> can
> >> >> >>>>> > > > >> > >> >> >>> get
> >> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
> >> >> >>>>> > > > >> > >> >> >>> > >> > > do
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now have
> general
> >> >> >>>> consensus that
> >> >> >>>>> > (1)
> >> >> >>>>> > > > headers
> >> >> >>>>> > > > >> > are
> >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> >> >> >>>>> > > > >> > >> >> >>> > >> and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2) we
> want it
> >> as
> >> >> a
> >> >> >>>> top level
> >> >> >>>>> > entity.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to
> state the
> >> >> >> obvious i
> >> >> >>>>> > believe (1)
> >> >> >>>>> > > > >> headers
> >> >> >>>>> > > > >> > are
> >> >> >>>>> > > > >> > >> >> >>> > worthwhile
> >> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as a
> top level
> >> >> >>>> entity.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>> ______________________________
> >> >> >>>>> > __________
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> From: Roger
> Hoover <
> >> >> >>>>> > > roger.hoover@gmail.com
> >> >> >>>>> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent:
> Wednesday,
> >> >> November
> >> >> >>>> 9, 2016
> >> >> >>>>> > 9:10:47
> >> >> >>>>> > > > PM
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
> >> >> dev@kafka.apache.org
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
> >> [DISCUSS]
> >> >> >>>> KIP-82 - Add
> >> >> >>>>> > > Record
> >> >> >>>>> > > > >> > Headers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for
> going a
> >> >> little
> >> >> >>>> in the
> >> >> >>>>> > weeds but
> >> >> >>>>> > > > >> thanks
> >> >> >>>>> > > > >> > >> for
> >> >> >>>>> > > > >> > >> >> the
> >> >> >>>>> > > > >> > >> >> >>> > >> replies
> >> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed that
> a prefix
> >> >> and
> >> >> >>>> {int,
> >> >> >>>>> > int} can
> >> >> >>>>> > > be
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> same.
> >> >> >>>>> > > > >> > >> >> It
> >> >> >>>>> > > > >> > >> >> >>> > doesn't
> >> >> >>>>> > > > >> > >> >> >>> > >> > look
> >> >> >>>>> > > > >> > >> >> >>> > >> > > like
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's what
> the KIP
> >> is
> >> >> >>>> saying the
> >> >> >>>>> > "Open"
> >> >> >>>>> > > > >> > section.
> >> >> >>>>> > > > >> > >> The
> >> >> >>>>> > > > >> > >> >> >>> > example
> >> >> >>>>> > > > >> > >> >> >>> > >> > shows
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New
> Relic and
> >> >> 210002
> >> >> >>>> for App
> >> >> >>>>> > Dynamics
> >> >> >>>>> > > > >> > implying
> >> >> >>>>> > > > >> > >> >> that
> >> >> >>>>> > > > >> > >> >> >>> the
> >> >> >>>>> > > > >> > >> >> >>> > New
> >> >> >>>>> > > > >> > >> >> >>> > >> > Relic
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> organization
> will
> >> have
> >> >> >>>> only a
> >> >> >>>>> > single
> >> >> >>>>> > > > header id
> >> >> >>>>> > > > >> > to
> >> >> >>>>> > > > >> > >> work
> >> >> >>>>> > > > >> > >> >> >>> > with. Or
> >> >> >>>>> > > > >> > >> >> >>> > >> > is
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix?
> The main
> >> >> point
> >> >> >>>> of a
> >> >> >>>>> > namespace
> >> >> >>>>> > > or
> >> >> >>>>> > > > >> > prefix
> >> >> >>>>> > > > >> > >> is
> >> >> >>>>> > > > >> > >> >> to
> >> >> >>>>> > > > >> > >> >> >>> > reduce
> >> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead of
> config
> >> >> >> mapping
> >> >> >>>> or
> >> >> >>>>> > > registration
> >> >> >>>>> > > > >> > >> depending
> >> >> >>>>> > > > >> > >> >> on
> >> >> >>>>> > > > >> > >> >> >>> how
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> namespaces/prefixes
> >> >> are
> >> >> >>>> managed.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would love
> to hear
> >> >> more
> >> >> >>>> feedback
> >> >> >>>>> > on the
> >> >> >>>>> > > > >> > >> higher-level
> >> >> >>>>> > > > >> > >> >> >>> > questions
> >> >> >>>>> > > > >> > >> >> >>> > >> > > though...
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov
> 9, 2016
> >> at
> >> >> >>>> 11:38 AM,
> >> >> >>>>> > radai <
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think
> this
> >> >> discussion
> >> >> >>>> is
> >> >> >>>>> > getting a
> >> >> >>>>> > > bit
> >> >> >>>>> > > > >> into
> >> >> >>>>> > > > >> > the
> >> >> >>>>> > > > >> > >> >> >>> weeds on
> >> >> >>>>> > > > >> > >> >> >>> > >> > technical
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> implementation
> >> >> details.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek
> to step
> >> >> back a
> >> >> >>>> minute
> >> >> >>>>> > and try
> >> >> >>>>> > > > and
> >> >> >>>>> > > > >> > >> establish
> >> >> >>>>> > > > >> > >> >> >>> > where we
> >> >> >>>>> > > > >> > >> >> >>> > >> > are in
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger
> picture:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
> >> nacho's
> >> >> >> last
> >> >> >>>>> > paragraph)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we
> all in
> >> >> >>>> agreement that
> >> >> >>>>> > headers
> >> >> >>>>> > > > are
> >> >> >>>>> > > > >> a
> >> >> >>>>> > > > >> > >> >> >>> worthwhile
> >> >> >>>>> > > > >> > >> >> >>> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > useful
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > addition
> to have?
> >> >> this
> >> >> >>>> was
> >> >> >>>>> > contested
> >> >> >>>>> > > > early
> >> >> >>>>> > > > >> on
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we
> all in
> >> >> >>>> agreement on
> >> >> >>>>> > headers
> >> >> >>>>> > > as
> >> >> >>>>> > > > top
> >> >> >>>>> > > > >> > >> level
> >> >> >>>>> > > > >> > >> >> >>> entity
> >> >> >>>>> > > > >> > >> >> >>> > vs
> >> >> >>>>> > > > >> > >> >> >>> > >> > headers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> squirreled-away in
> >> >> V?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if there
> are still
> >> >> >>>> concerns
> >> >> >>>>> > around
> >> >> >>>>> > > these
> >> >> >>>>> > > > #2
> >> >> >>>>> > > > >> > >> points
> >> >> >>>>> > > > >> > >> >> >>> (#jay?
> >> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and now
> back to
> >> our
> >> >> >>>> normal
> >> >> >>>>> > programming
> >> >> >>>>> > > > ...)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints
> are nice.
> >> >> >> having
> >> >> >>>> said
> >> >> >>>>> > that, its
> >> >> >>>>> > > > >> adding
> >> >> >>>>> > > > >> > >> >> >>> complexity
> >> >> >>>>> > > > >> > >> >> >>> > >> (see
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>> https://github.com/addthis/
> >> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>> main/java/com/clearspring/
> >> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st
> google
> >> >> result)
> >> >> >>>> and would
> >> >> >>>>> > require
> >> >> >>>>> > > > >> anyone
> >> >> >>>>> > > > >> > >> >> writing
> >> >> >>>>> > > > >> > >> >> >>> > other
> >> >> >>>>> > > > >> > >> >> >>> > >> > clients
> >> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python?
> Go? Bash?
> >> >> ;-) )
> >> >> >>>> to
> >> >> >>>>> > > get/implement
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > >> same,
> >> >> >>>>> > > > >> > >> >> and
> >> >> >>>>> > > > >> > >> >> >>> for
> >> >> >>>>> > > > >> > >> >> >>> > >> > relatively
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little
> gain (int
> >> vs
> >> >> >>>> string is
> >> >> >>>>> > order of
> >> >> >>>>> > > > >> > magnitude,
> >> >> >>>>> > > > >> > >> >> this
> >> >> >>>>> > > > >> > >> >> >>> > isnt).
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int
> namespacing vs
> >> >> >> {int,
> >> >> >>>> int}
> >> >> >>>>> > > namespacing
> >> >> >>>>> > > > >> are
> >> >> >>>>> > > > >> > >> >> basically
> >> >> >>>>> > > > >> > >> >> >>> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > same
> >> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre just
> >> >> namespacing
> >> >> >>>> an int64
> >> >> >>>>> > and
> >> >> >>>>> > > > giving
> >> >> >>>>> > > > >> > people
> >> >> >>>>> > > > >> > >> >> while
> >> >> >>>>> > > > >> > >> >> >>> > 2^32
> >> >> >>>>> > > > >> > >> >> >>> > >> > ranges
> >> >> >>>>> > > > >> > >> >> >>> > >> > > at a
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time. the
> part i
> >> >> like
> >> >> >>>> about this
> >> >> >>>>> > is
> >> >> >>>>> > > > letting
> >> >> >>>>> > > > >> > >> people
> >> >> >>>>> > > > >> > >> >> >>> have a
> >> >> >>>>> > > > >> > >> >> >>> > >> large
> >> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers
> with one
> >> >> >>>> registration so
> >> >> >>>>> > they
> >> >> >>>>> > > > dont
> >> >> >>>>> > > > >> > have
> >> >> >>>>> > > > >> > >> to
> >> >> >>>>> > > > >> > >> >> come
> >> >> >>>>> > > > >> > >> >> >>> > back
> >> >> >>>>> > > > >> > >> >> >>> > >> > for
> >> >> >>>>> > > > >> > >> >> >>> > >> > > every
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
> >> plugin/header
> >> >> >>>> they want to
> >> >> >>>>> > > > "reserve".
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed,
> Nov 9,
> >> 2016
> >> >> at
> >> >> >>>> 11:01 AM,
> >> >> >>>>> > Roger
> >> >> >>>>> > > > >> Hoover
> >> >> >>>>> > > > >> > <
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> roger.hoover@gmail.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since
> some of
> >> the
> >> >> >>>> debate has
> >> >> >>>>> > been
> >> >> >>>>> > > about
> >> >> >>>>> > > > >> > >> overhead +
> >> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> wondering if we
> >> >> have
> >> >> >>>>> > considered a
> >> >> >>>>> > > > varint
> >> >> >>>>> > > > >> > >> encoding
> >> >> >>>>> > > > >> > >> >> (
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>> https://developers.google.com/
> >> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
> >> >> >>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the
> header
> >> length
> >> >> >>>> field (int32
> >> >> >>>>> > in the
> >> >> >>>>> > > > >> > proposal)
> >> >> >>>>> > > > >> > >> >> and
> >> >> >>>>> > > > >> > >> >> >>> for
> >> >> >>>>> > > > >> > >> >> >>> > >> > header
> >> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
> >> headers,
> >> >> >> the
> >> >> >>>>> > overhead would
> >> >> >>>>> > > > be a
> >> >> >>>>> > > > >> > >> single
> >> >> >>>>> > > > >> > >> >> >>> byte
> >> >> >>>>> > > > >> > >> >> >>> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > for each
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128
> would
> >> >> also
> >> >> >>>> need only a
> >> >> >>>>> > > single
> >> >> >>>>> > > > >> byte?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed,
> Nov 9,
> >> >> 2016
> >> >> >> at
> >> >> >>>> 6:43 AM,
> >> >> >>>>> > > radai <
> >> >> >>>>> > > > >> > >> >> >>> > >> >
> radai.rosenblatt@gmail.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> @magnus - and
> >> >> very
> >> >> >>>> dangerous
> >> >> >>>>> > (youre
> >> >> >>>>> > > > >> > >> essentially
> >> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > executing
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> arbitrary code
> >> >> off
> >> >> >>>> the
> >> >> >>>>> > internet on
> >> >> >>>>> > > > your
> >> >> >>>>> > > > >> > >> servers
> >> >> >>>>> > > > >> > >> >> ...
> >> >> >>>>> > > > >> > >> >> >>> > bad
> >> >> >>>>> > > > >> > >> >> >>> > >> > idea
> >> >> >>>>> > > > >> > >> >> >>> > >> > > without
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> sandbox, even
> >> >> with)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as for
> it
> >> being
> >> >> a
> >> >> >>>> purely
> >> >> >>>>> > > > administrative
> >> >> >>>>> > > > >> > task
> >> >> >>>>> > > > >> > >> - i
> >> >> >>>>> > > > >> > >> >> >>> > >> disagree.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish
> it
> >> would,
> >> >> >>>> really,
> >> >> >>>>> > because
> >> >> >>>>> > > > then my
> >> >> >>>>> > > > >> > >> earlier
> >> >> >>>>> > > > >> > >> >> >>> > point on
> >> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > complexity
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> remapping
> >> >> >>>> process would
> >> >> >>>>> > be
> >> >> >>>>> > > > invalid,
> >> >> >>>>> > > > >> > but
> >> >> >>>>> > > > >> > >> at
> >> >> >>>>> > > > >> > >> >> >>> > linkedin,
> >> >> >>>>> > > > >> > >> >> >>> > >> > for
> >> >> >>>>> > > > >> > >> >> >>> > >> > > example,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (the
> team im
> >> in)
> >> >> >> run
> >> >> >>>> kafka
> >> >> >>>>> > as a
> >> >> >>>>> > > > service.
> >> >> >>>>> > > > >> > we
> >> >> >>>>> > > > >> > >> dont
> >> >> >>>>> > > > >> > >> >> >>> > really
> >> >> >>>>> > > > >> > >> >> >>> > >> > know
> >> >> >>>>> > > > >> > >> >> >>> > >> > > what our
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> (developing
> >> >> >>>> applications
> >> >> >>>>> > that use
> >> >> >>>>> > > > kafka)
> >> >> >>>>> > > > >> > are
> >> >> >>>>> > > > >> > >> up
> >> >> >>>>> > > > >> > >> >> to
> >> >> >>>>> > > > >> > >> >> >>> at
> >> >> >>>>> > > > >> > >> >> >>> > any
> >> >> >>>>> > > > >> > >> >> >>> > >> > given
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
> >> possible
> >> >> >>>> (given the
> >> >> >>>>> > > > existance of
> >> >> >>>>> > > > >> > >> headers
> >> >> >>>>> > > > >> > >> >> >>> and a
> >> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> ecosystem) for
> >> >> some
> >> >> >>>>> > application to
> >> >> >>>>> > > > >> "equip"
> >> >> >>>>> > > > >> > >> their
> >> >> >>>>> > > > >> > >> >> >>> > >> producers
> >> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > consumers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > with
> the
> >> >> required
> >> >> >>>> plugin
> >> >> >>>>> > without us
> >> >> >>>>> > > > >> > knowing.
> >> >> >>>>> > > > >> > >> i
> >> >> >>>>> > > > >> > >> >> dont
> >> >> >>>>> > > > >> > >> >> >>> > mean
> >> >> >>>>> > > > >> > >> >> >>> > >> > to imply
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i
> just
> >> >> want to
> >> >> >>>> make the
> >> >> >>>>> > point
> >> >> >>>>> > > > that
> >> >> >>>>> > > > >> > its
> >> >> >>>>> > > > >> > >> not
> >> >> >>>>> > > > >> > >> >> as
> >> >> >>>>> > > > >> > >> >> >>> > simple
> >> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sync
> across a
> >> >> >>>> large-enough
> >> >> >>>>> > > > organization.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On
> Wed, Nov 9,
> >> >> 2016
> >> >> >>>> at 6:17
> >> >> >>>>> > AM,
> >> >> >>>>> > > > Magnus
> >> >> >>>>> > > > >> > >> Edenhill
> >> >> >>>>> > > > >> > >> >> <
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> magnus@edenhill.se>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I
> think
> >> there
> >> >> is
> >> >> >> a
> >> >> >>>> piece
> >> >> >>>>> > missing
> >> >> >>>>> > > in
> >> >> >>>>> > > > >> the
> >> >> >>>>> > > > >> > >> >> Strings
> >> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > where
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> pro-Stringers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> reason that
> >> by
> >> >> >>>> providing
> >> >> >>>>> > unique
> >> >> >>>>> > > > string
> >> >> >>>>> > > > >> > >> >> >>> identifiers
> >> >> >>>>> > > > >> > >> >> >>> > for
> >> >> >>>>> > > > >> > >> >> >>> > >> > each
> >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> everything
> >> >> will
> >> >> >>>> just
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> magically
> >> work
> >> >> >> for
> >> >> >>>> all
> >> >> >>>>> > parts of
> >> >> >>>>> > > the
> >> >> >>>>> > > > >> > stream
> >> >> >>>>> > > > >> > >> >> >>> pipeline.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > But
> the
> >> >> strings
> >> >> >>>> dont mean
> >> >> >>>>> > > anything
> >> >> >>>>> > > > by
> >> >> >>>>> > > > >> > >> >> themselves,
> >> >> >>>>> > > > >> > >> >> >>> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > while we
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> probably
> >> >> envision
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> auto
> >> >> plugin
> >> >> >>>> loader
> >> >> >>>>> > that
> >> >> >>>>> > > > >> downloads,
> >> >> >>>>> > > > >> > >> >> compiles,
> >> >> >>>>> > > > >> > >> >> >>> > links
> >> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > runs
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> on-demand
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > as
> soon as
> >> >> >> they're
> >> >> >>>> seen by
> >> >> >>>>> > a
> >> >> >>>>> > > > >> consumer, I
> >> >> >>>>> > > > >> > >> dont
> >> >> >>>>> > > > >> > >> >> >>> really
> >> >> >>>>> > > > >> > >> >> >>> > >> see
> >> >> >>>>> > > > >> > >> >> >>> > >> > a
> >> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> something
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > so
> dynamic
> >> >> (and
> >> >> >>>> fragile) in
> >> >> >>>>> > > > practice.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > In
> the real
> >> >> world
> >> >> >>>> an
> >> >> >>>>> > application
> >> >> >>>>> > > > will
> >> >> >>>>> > > > >> be
> >> >> >>>>> > > > >> > >> >> >>> configured
> >> >> >>>>> > > > >> > >> >> >>> > >> with
> >> >> >>>>> > > > >> > >> >> >>> > >> > a set
> >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > to
> either
> >> add
> >> >> >>>> (producer)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > or
> read
> >> >> >> (consumer)
> >> >> >>>> headers.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > This
> is an
> >> >> >>>> administrative
> >> >> >>>>> > task
> >> >> >>>>> > > > based
> >> >> >>>>> > > > >> on
> >> >> >>>>> > > > >> > >> what
> >> >> >>>>> > > > >> > >> >> >>> > features a
> >> >> >>>>> > > > >> > >> >> >>> > >> > client
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> needs/provides
> >> >> >> and
> >> >> >>>> results
> >> >> >>>>> > in
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some
> sort of
> >> >> >>>> configuration
> >> >> >>>>> > to
> >> >> >>>>> > > > enable
> >> >> >>>>> > > > >> and
> >> >> >>>>> > > > >> > >> >> >>> configure
> >> >> >>>>> > > > >> > >> >> >>> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > desired
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> Since this
> >> >> needs
> >> >> >>>> to be kept
> >> >> >>>>> > > > somewhat
> >> >> >>>>> > > > >> in
> >> >> >>>>> > > > >> > >> sync
> >> >> >>>>> > > > >> > >> >> >>> across
> >> >> >>>>> > > > >> > >> >> >>> > an
> >> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > is
> no point
> >> in
> >> >> >>>> having
> >> >> >>>>> > producers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > add
> headers
> >> no
> >> >> >>>> consumers
> >> >> >>>>> > will
> >> >> >>>>> > > read,
> >> >> >>>>> > > > >> and
> >> >> >>>>> > > > >> > >> vice
> >> >> >>>>> > > > >> > >> >> >>> versa),
> >> >> >>>>> > > > >> > >> >> >>> > >> the
> >> >> >>>>> > > > >> > >> >> >>> > >> > added
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> complexity
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > of
> assigning
> >> >> an
> >> >> >> id
> >> >> >>>>> > namespace
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > for
> each
> >> >> plugin
> >> >> >> as
> >> >> >>>> it is
> >> >> >>>>> > being
> >> >> >>>>> > > > >> > configured
> >> >> >>>>> > > > >> > >> >> should
> >> >> >>>>> > > > >> > >> >> >>> be
> >> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> /Magnus
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> 2016-11-09
> >> >> 13:06
> >> >> >>>> GMT+01:00
> >> >> >>>>> > > Michael
> >> >> >>>>> > > > >> > Pearce <
> >> >> >>>>> > > > >> > >> >> >>> > >> > >
> Michael.Pearce@ig.com>:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Just
> >> >> >>>> following/catching
> >> >> >>>>> > up on
> >> >> >>>>> > > > what
> >> >> >>>>> > > > >> > seems
> >> >> >>>>> > > > >> > >> to
> >> >> >>>>> > > > >> > >> >> be
> >> >> >>>>> > > > >> > >> >> >>> an
> >> >> >>>>> > > > >> > >> >> >>> > >> > active
> >> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> @Radai
> >> >> sorry if
> >> >> >>>> it may
> >> >> >>>>> > seem
> >> >> >>>>> > > > obvious
> >> >> >>>>> > > > >> > but
> >> >> >>>>> > > > >> > >> what
> >> >> >>>>> > > > >> > >> >> >>> does
> >> >> >>>>> > > > >> > >> >> >>> > MD
> >> >> >>>>> > > > >> > >> >> >>> > >> > stand
> >> >> >>>>> > > > >> > >> >> >>> > >> > > for?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My
> take on
> >> >> >>>> String vs Int:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> will
> >> state
> >> >> >>>> first I am
> >> >> >>>>> > pro Int
> >> >> >>>>> > > > (16
> >> >> >>>>> > > > >> or
> >> >> >>>>> > > > >> > >> 32).
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I
> do
> >> though
> >> >> >>>> playing
> >> >> >>>>> > devils
> >> >> >>>>> > > > advocate
> >> >> >>>>> > > > >> > see a
> >> >> >>>>> > > > >> > >> >> big
> >> >> >>>>> > > > >> > >> >> >>> plus
> >> >> >>>>> > > > >> > >> >> >>> > >> > with the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > argument
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> String
> >> keys,
> >> >> >>>> this is
> >> >> >>>>> > around
> >> >> >>>>> > > > >> > integrating
> >> >> >>>>> > > > >> > >> >> into an
> >> >> >>>>> > > > >> > >> >> >>> > >> > existing
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As
> many
> >> >> other
> >> >> >>>> systems use
> >> >> >>>>> > > String
> >> >> >>>>> > > > >> based
> >> >> >>>>> > > > >> > >> >> headers
> >> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
> >> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > it
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> much
> >> easier
> >> >> for
> >> >> >>>> these to
> >> >> >>>>> > be
> >> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
> >> >> >>>>> > > > >> > >> >> >>> > >> > into.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> How with
> >> Int
> >> >> >>>> based
> >> >> >>>>> > headers
> >> >> >>>>> > > could
> >> >> >>>>> > > > we
> >> >> >>>>> > > > >> > >> provide
> >> >> >>>>> > > > >> > >> >> a
> >> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> integration
> >> >> >>>> simple /
> >> >> >>>>> > easy with
> >> >> >>>>> > > > >> > transition
> >> >> >>>>> > > > >> > >> >> flows
> >> >> >>>>> > > > >> > >> >> >>> > over
> >> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> tough
> >> luck
> >> >> >>>> buddy
> >> >> >>>>> > you're on
> >> >> >>>>> > > your
> >> >> >>>>> > > > >> own
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> simply
> >> >> hash
> >> >> >>>> the string
> >> >> >>>>> > into
> >> >> >>>>> > > int
> >> >> >>>>> > > > >> code
> >> >> >>>>> > > > >> > >> and
> >> >> >>>>> > > > >> > >> >> hope
> >> >> >>>>> > > > >> > >> >> >>> > for
> >> >> >>>>> > > > >> > >> >> >>> > >> no
> >> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> convert
> >> back
> >> >> >>>> though?)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > *
> http2
> >> >> style
> >> >> >> as
> >> >> >>>>> > mentioned by
> >> >> >>>>> > > > nacho.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> cheers,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Mike
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > ______________________________
> >> >> >>>>> > > > >> > __________
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> From:
> >> radai
> >> >> <
> >> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Sent:
> >> >> >> Wednesday,
> >> >> >>>>> > November 9,
> >> >> >>>>> > > 2016
> >> >> >>>>> > > > >> > 8:12 AM
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
> >> >> >>>> dev@kafka.apache.org
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Subject:
> >> Re:
> >> >> >>>> [DISCUSS]
> >> >> >>>>> > KIP-82 -
> >> >> >>>>> > > > Add
> >> >> >>>>> > > > >> > >> Record
> >> >> >>>>> > > > >> > >> >> >>> Headers
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> thinking
> >> >> about
> >> >> >>>> it some
> >> >> >>>>> > more,
> >> >> >>>>> > > the
> >> >> >>>>> > > > >> best
> >> >> >>>>> > > > >> > >> way to
> >> >> >>>>> > > > >> > >> >> >>> > transmit
> >> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > header
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> remapping
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> data to
> >> >> >>>> consumers would
> >> >> >>>>> > be to
> >> >> >>>>> > > > put it
> >> >> >>>>> > > > >> > in
> >> >> >>>>> > > > >> > >> the
> >> >> >>>>> > > > >> > >> >> MD
> >> >> >>>>> > > > >> > >> >> >>> > >> response
> >> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it
> should
> >> be
> >> >> >>>> discussed
> >> >> >>>>> > now.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On
> Wed,
> >> Nov
> >> >> 9,
> >> >> >>>> 2016 at
> >> >> >>>>> > 12:09
> >> >> >>>>> > > AM,
> >> >> >>>>> > > > >> > radai <
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> >> >> >> radai.rosenblatt@gmail.com
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> im not
> >> >> >> opposed
> >> >> >>>> to the
> >> >> >>>>> > idea of
> >> >> >>>>> > > > >> > namespace
> >> >> >>>>> > > > >> > >> >> >>> mapping.
> >> >> >>>>> > > > >> > >> >> >>> > >> all
> >> >> >>>>> > > > >> > >> >> >>> > >> > im
> >> >> >>>>> > > > >> > >> >> >>> > >> > > saying
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > that
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> not part
> >> >> of
> >> >> >>>> the "mvp"
> >> >> >>>>> > and,
> >> >> >>>>> > > > since
> >> >> >>>>> > > > >> it
> >> >> >>>>> > > > >> > >> >> requires
> >> >> >>>>> > > > >> > >> >> >>> no
> >> >> >>>>> > > > >> > >> >> >>> > >> wire
> >> >> >>>>> > > > >> > >> >> >>> > >> > format
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> always
> >> be
> >> >> >>>> added later.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also,
> >> its
> >> >> not
> >> >> >>>> as
> >> >> >>>>> > simple as
> >> >> >>>>> > > just
> >> >> >>>>> > > > >> > >> >> configuring
> >> >> >>>>> > > > >> > >> >> >>> MM
> >> >> >>>>> > > > >> > >> >> >>> > to
> >> >> >>>>> > > > >> > >> >> >>> > >> do
> >> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > transform:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> say i've
> >> >> >>>> implemented
> >> >> >>>>> > large
> >> >> >>>>> > > > message
> >> >> >>>>> > > > >> > >> >> support as
> >> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > on
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> mirror
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> target
> >> >> >> cluster
> >> >> >>>> its been
> >> >> >>>>> > > > remapped
> >> >> >>>>> > > > >> to
> >> >> >>>>> > > > >> > >> >> {999,1}.
> >> >> >>>>> > > > >> > >> >> >>> the
> >> >> >>>>> > > > >> > >> >> >>> > >> > consumer
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> also
> >> need
> >> >> to
> >> >> >>>> be told
> >> >> >>>>> > to look
> >> >> >>>>> > > > for
> >> >> >>>>> > > > >> the
> >> >> >>>>> > > > >> > >> large
> >> >> >>>>> > > > >> > >> >> >>> > message
> >> >> >>>>> > > > >> > >> >> >>> > >> > "part X
> >> >> >>>>> > > > >> > >> >> >>> > >> > > of
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > header
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> under
> >> >> >> {999,1}.
> >> >> >>>> doable,
> >> >> >>>>> > but
> >> >> >>>>> > > > tricky.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> On Tue,
> >> >> Nov
> >> >> >> 8,
> >> >> >>>> 2016 at
> >> >> >>>>> > 10:29
> >> >> >>>>> > > > PM,
> >> >> >>>>> > > > >> > Gwen
> >> >> >>>>> > > > >> > >> >> >>> Shapira <
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >>
> gwen@confluent.io
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> While
> >> you
> >> >> >> can
> >> >> >>>> do
> >> >> >>>>> > whatever
> >> >> >>>>> > > you
> >> >> >>>>> > > > >> want
> >> >> >>>>> > > > >> > >> with a
> >> >> >>>>> > > > >> > >> >> >>> > >> namespace
> >> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > your
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> what
> >> I'd
> >> >> >>>> expect is
> >> >> >>>>> > for each
> >> >> >>>>> > > > app
> >> >> >>>>> > > > >> to
> >> >> >>>>> > > > >> > >> >> >>> namespaces
> >> >> >>>>> > > > >> > >> >> >>> > >> > > configurable...
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> So if I
> >> >> >>>> accidentally
> >> >> >>>>> > used
> >> >> >>>>> > > 666
> >> >> >>>>> > > > for
> >> >> >>>>> > > > >> > my
> >> >> >>>>> > > > >> > >> HR
> >> >> >>>>> > > > >> > >> >> >>> > >> department,
> >> >> >>>>> > > > >> > >> >> >>> > >> > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > still
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> run
> >> >> >> RadaiApp,
> >> >> >>>> I can
> >> >> >>>>> > config
> >> >> >>>>> > > > >> > >> "namespace=42"
> >> >> >>>>> > > > >> > >> >> >>> for
> >> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> everything
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> will
> >> look
> >> >> >>>> normal.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> This
> >> >> means
> >> >> >>>> you only
> >> >> >>>>> > need to
> >> >> >>>>> > > > sync
> >> >> >>>>> > > > >> > usage
> >> >> >>>>> > > > >> > >> >> >>> inside
> >> >> >>>>> > > > >> > >> >> >>> > your
> >> >> >>>>> > > > >> > >> >> >>> > >> > own
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> organization.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> Still
> >> >> hard,
> >> >> >>>> but
> >> >> >>>>> > somewhat
> >> >> >>>>> > > > easier
> >> >> >>>>> > > > >> > than
> >> >> >>>>> > > > >> > >> >> syncing
> >> >> >>>>> > > > >> > >> >> >>> > with
> >> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > entire
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > world.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> On Tue,
> >> >> Nov
> >> >> >>>> 8, 2016
> >> >> >>>>> > at 10:07
> >> >> >>>>> > > > PM,
> >> >> >>>>> > > > >> > >> radai <
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> >>>> radai.rosenblatt@gmail.com>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > and
> >> we
> >> >> can
> >> >> >>>> start
> >> >> >>>>> > with
> >> >> >>>>> > > > >> {namespace,
> >> >> >>>>> > > > >> > >> id}
> >> >> >>>>> > > > >> > >> >> and
> >> >> >>>>> > > > >> > >> >> >>> no
> >> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> always
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > add
> >> it
> >> >> >>>> later on
> >> >> >>>>> > if/when
> >> >> >>>>> > > > >> > collisions
> >> >> >>>>> > > > >> > >> >> >>> actually
> >> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
> >> >> >>>>> > > > >> > >> >> >>> > >> > > dont
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> they'd
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> be
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > a
> >> >> >> problem).
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > every
> >> >> >>>> interested
> >> >> >>>>> > party (so
> >> >> >>>>> > > > orgs
> >> >> >>>>> > > > >> > or
> >> >> >>>>> > > > >> > >> >> >>> > individuals)
> >> >> >>>>> > > > >> > >> >> >>> > >> > could
> >> >> >>>>> > > > >> > >> >> >>> > >> > > then
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> register
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >> prefix
> >> >> (0
> >> >> >> =
> >> >> >>>>> > reserved, 1 =
> >> >> >>>>> > > > >> > confluent
> >> >> >>>>> > > > >> > >> ...
> >> >> >>>>> > > > >> > >> >> >>> 666
> >> >> >>>>> > > > >> > >> >> >>> > = me
> >> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
> >> >> >>>>> > > > >> > >> >> >>> > >> > > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> whatever
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> with
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > the
> >> >> 2nd ID
> >> >> >>>> - so once
> >> >> >>>>> > > > linkedin
> >> >> >>>>> > > > >> > >> >> registers,
> >> >> >>>>> > > > >> > >> >> >>> say
> >> >> >>>>> > > > >> > >> >> >>> > 3,
> >> >> >>>>> > > > >> > >> >> >>> > >> > then
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> free
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > to
> >> use
> >> >> {3,
> >> >> >>>> *} with a
> >> >> >>>>> > > > reasonable
> >> >> >>>>> > > > >> > >> >> >>> expectation
> >> >> >>>>> > > > >> > >> >> >>> > to
> >> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > collide
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> anything
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > else.
> >> >> >>>> further
> >> >> >>>>> > partitioning
> >> >> >>>>> > > > of
> >> >> >>>>> > > > >> > that *
> >> >> >>>>> > > > >> > >> >> >>> becomes
> >> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > problem,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >> >> "upstream
> >> >> >>>>> > registration"
> >> >> >>>>> > > of a
> >> >> >>>>> > > > >> > >> namespace
> >> >> >>>>> > > > >> > >> >> >>> only
> >> >> >>>>> > > > >> > >> >> >>> > has
> >> >> >>>>> > > > >> > >> >> >>> > >> to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > happen
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> > On
> >> Tue,
> >> >> >> Nov
> >> >> >>>> 8, 2016
> >> >> >>>>> > at
> >> >> >>>>> > > 9:03
> >> >> >>>>> > > > PM,
> >> >> >>>>> > > > >> > >> James
> >> >> >>>>> > > > >> > >> >> >>> Cheng <
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
> >> >> wushujames@gmail.com
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> > On
> >> >> Nov
> >> >> >>>> 8, 2016,
> >> >> >>>>> > at 5:54
> >> >> >>>>> > > > PM,
> >> >> >>>>> > > > >> > Gwen
> >> >> >>>>> > > > >> > >> >> >>> Shapira <
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
> gwen@confluent.io
> >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> Thank
> >> >> >>>> you so
> >> >> >>>>> > much for
> >> >> >>>>> > > > this
> >> >> >>>>> > > > >> > clear
> >> >> >>>>> > > > >> > >> and
> >> >> >>>>> > > > >> > >> >> >>> fair
> >> >> >>>>> > > > >> > >> >> >>> > >> > summary of
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> arguments.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> I'm
> >> >> in
> >> >> >>>> favor of
> >> >> >>>>> > ints.
> >> >> >>>>> > > > Not a
> >> >> >>>>> > > > >> > >> >> >>> deal-breaker,
> >> >> >>>>> > > > >> > >> >> >>> > but
> >> >> >>>>> > > > >> > >> >> >>> > >> > in
> >> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> Even
> >> >> >>>> more in
> >> >> >>>>> > favor of
> >> >> >>>>> > > > >> Magnus's
> >> >> >>>>> > > > >> > >> >> >>> > decentralized
> >> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> Roger's
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> tweak:
> >> >> >>>> add a
> >> >> >>>>> > namespace
> >> >> >>>>> > > > for
> >> >> >>>>> > > > >> > >> headers.
> >> >> >>>>> > > > >> > >> >> >>> This
> >> >> >>>>> > > > >> > >> >> >>> > will
> >> >> >>>>> > > > >> > >> >> >>> > >> > allow
> >> >> >>>>> > > > >> > >> >> >>> > >> > > each
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> just
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> use
> >> >> >>>> whatever IDs
> >> >> >>>>> > it
> >> >> >>>>> > > wants
> >> >> >>>>> > > > >> > >> >> internally,
> >> >> >>>>> > > > >> > >> >> >>> and
> >> >> >>>>> > > > >> > >> >> >>> > >> then
> >> >> >>>>> > > > >> > >> >> >>> > >> > let
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> deploying
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> the
> >> >> app
> >> >> >>>> figure
> >> >> >>>>> > out an
> >> >> >>>>> > > > >> > available
> >> >> >>>>> > > > >> > >> >> >>> namespace
> >> >> >>>>> > > > >> > >> >> >>> > ID
> >> >> >>>>> > > > >> > >> >> >>> > >> > for the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > live
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> > So
> >> >> >>>>> > > > >> > io.confluent.schema-registry
> >> >> >>>>> > > > >> > >> can
> >> >> >>>>> > > > >> > >> >> be
> >> >> >>>>> > > > >> > >> >> >>> > >> > namespace
> >> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >> deployment
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> and
> >> >> >> 0x57
> >> >> >>>> on
> >> >> >>>>> > yours, and
> >> >> >>>>> > > > the
> >> >> >>>>> > > > >> > poor
> >> >> >>>>> > > > >> > >> guys
> >> >> >>>>> > > > >> > >> >> >>> > >> > developing the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > app
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> worry
> >> >> >>>> about that.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> Gwen,
> >> >> if
> >> >> >> I
> >> >> >>>>> > understand
> >> >> >>>>> > > your
> >> >> >>>>> > > > >> > example
> >> >> >>>>> > > > >> > >> >> >>> right, an
> >> >> >>>>> > > > >> > >> >> >>> > >> > > application
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> might
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> decide to
> >> >> >>>> use 0x01
> >> >> >>>>> > in one
> >> >> >>>>> > > > >> > >> deployment,
> >> >> >>>>> > > > >> > >> >> and
> >> >> >>>>> > > > >> > >> >> >>> > that
> >> >> >>>>> > > > >> > >> >> >>> > >> > means
> >> >> >>>>> > > > >> > >> >> >>> > >> > > that
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> message
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> is
> >> >> >> written
> >> >> >>>> into the
> >> >> >>>>> > > > broker, it
> >> >> >>>>> > > > >> > >> will be
> >> >> >>>>> > > > >> > >> >> >>> > saved on
> >> >> >>>>> > > > >> > >> >> >>> > >> > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > broker
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> specific
> >> >> >>>> namespace
> >> >> >>>>> > > (0x01).
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> If
> >> you
> >> >> >>>> were to
> >> >> >>>>> > mirror
> >> >> >>>>> > > that
> >> >> >>>>> > > > >> > message
> >> >> >>>>> > > > >> > >> >> into
> >> >> >>>>> > > > >> > >> >> >>> > another
> >> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> would
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> accompany
> >> >> >>>> the
> >> >> >>>>> > message,
> >> >> >>>>> > > > right?
> >> >> >>>>> > > > >> > What
> >> >> >>>>> > > > >> > >> if
> >> >> >>>>> > > > >> > >> >> the
> >> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> other
> >> >> >>>> cluster uses
> >> >> >>>>> > 0x57?
> >> >> >>>>> > > > They
> >> >> >>>>> > > > >> > won't
> >> >> >>>>> > > > >> > >> >> >>> > understand
> >> >> >>>>> > > > >> > >> >> >>> > >> > each
> >> >> >>>>> > > > >> > >> >> >>> > >> > > other?
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> I'm
> >> >> not
> >> >> >>>> sure
> >> >> >>>>> > that's an
> >> >> >>>>> > > > >> avoidable
> >> >> >>>>> > > > >> > >> >> >>> problem. I
> >> >> >>>>> > > > >> > >> >> >>> > >> > think it
> >> >> >>>>> > > > >> > >> >> >>> > >> > > simply
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> that
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> in
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> order
> >> >> to
> >> >> >>>> share
> >> >> >>>>> > data, you
> >> >> >>>>> > > > have
> >> >> >>>>> > > > >> to
> >> >> >>>>> > > > >> > >> also
> >> >> >>>>> > > > >> > >> >> >>> have a
> >> >> >>>>> > > > >> > >> >> >>> > >> > shared
> >> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>> understanding of
> >> >> >>>>> > what the
> >> >> >>>>> > > > >> > >> namespaces
> >> >> >>>>> > > > >> > >> >> >>> mean.
> >> >> >>>>> > > > >> > >> >> >>> > >> Which
> >> >> >>>>> > > > >> > >> >> >>> > >> > I
> >> >> >>>>> > > > >> > >> >> >>> > >> > > think
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> sense,
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> because
> >> >> >> the
> >> >> >>>>> > alternate
> >> >> >>>>> > > > (sharing
> >> >> >>>>> > > > >> > >> >> *nothing*
> >> >> >>>>> > > > >> > >> >> >>> at
> >> >> >>>>> > > > >> > >> >> >>> > >> all)
> >> >> >>>>> > > > >> > >> >> >>> > >> > would
> >> >> >>>>> > > > >> > >> >> >>> > >> > > mean
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
> there
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> would
> >> >> be
> >> >> >>>> no way to
> >> >> >>>>> > > > understand
> >> >> >>>>> > > > >> > each
> >> >> >>>>> > > > >> > >> >> other.
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> -James
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> Gwen
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> > On
> >> >> Tue,
> >> >> >>>> Nov 8,
> >> >> >>>>> > 2016 at
> >> >> >>>>> > > > 4:23
> >> >> >>>>> > > > >> > PM,
> >> >> >>>>> > > > >> > >> >> radai <
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
> >> >> >>>> radai.rosenblatt@gmail.com
> >> >> >>>>> > >
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >>
> >> wrote:
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >>
> >> +1
> >> >> for
> >> >> >>>> sean's
> >> >> >>>>> > > document.
> >> >> >>>>> > > > it
> >> >> >>>>> > > > >> > >> covers
> >> >> >>>>> > > > >> > >> >> >>> pretty
> >> >> >>>>> > > > >> > >> >> >>> > >> much
> >> >> >>>>> > > > >> > >> >> >>> > >> > all
> >> >> >>>>> > > > >> > >> >> >>> > >> > > the
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
> trade-offs
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >>
> >> >> >> provides
> >> >> >>>>> > concrete
> >> >> >>>>> > > > figures
> >> >> >>>>> > > > >> to
> >> >> >>>>> > > > >> > >> argue
> >> >> >>>>> > > > >> > >> >> >>> about
> >> >> >>>>> > > > >> > >> >> >>> > :-)
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >>
> >> >> >>>> (nit-picking -
> >> >> >>>>> > used
> >> >> >>>>> > > the
> >> >> >>>>> > > > >> same
> >> >> >>>>> > > > >> > >> xkcd
> >> >> >>>>> > > > >> > >> >> >>> twice,
> >> >> >>>>> > > > >> > >> >> >>> > >> also
> >> >> >>>>> > > > >> > >> >> >>> > >> > trove
> >> >> >>>>> > > > >> > >> >> >>> > >> > > has
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
> >> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
> >> >> superceded
> >> >> >>>>> > > > >> > >> >
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >> > --
> >> >> >>>>> > > > >> > Gwen Shapira
> >> >> >>>>> > > > >> > Product Manager | Confluent
> >> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> |
> @gwenshap
> >> >> >>>>> > > > >> > Follow us: Twitter | blog
> >> >> >>>>> > > > >> >
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >> --
> >> >> >>>>> > > > >> *Todd Palino*
> >> >> >>>>> > > > >> Staff Site Reliability Engineer
> >> >> >>>>> > > > >> Data Infrastructure Streaming
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >>
> >> >> >>>>> > > > >> linkedin.com/in/toddpalino
> >> >> >>>>> > > > >>
> >> >> >>>>> > > >
> >> >> >>>>> > > >
> >> >> >>>>> > > >
> >> >> >>>>> > > > --
> >> >> >>>>> > > > Gwen Shapira
> >> >> >>>>> > > > Product Manager | Confluent
> >> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >> >> >>>>> > > > Follow us: Twitter | blog
> >> >> >>>>> > > >
> >> >> >>>>> > >
> >> >> >>>>> >
> >> >> >>>>> >
> >> >> >>>>> > The information contained in this email is strictly
> >> >> confidential
> >> >> >>>> and for
> >> >> >>>>> > the use of the addressee only, unless otherwise
> indicated.
> >> If
> >> >> you
> >> >> >>>> are not
> >> >> >>>>> > the intended recipient, please do not read, copy,
> use or
> >> >> disclose
> >> >> >>>> to others
> >> >> >>>>> > this message or any attachment. Please also notify
> the
> >> sender
> >> >> by
> >> >> >>>> replying
> >> >> >>>>> > to this email or by telephone (+44(020 7896 0011)
> and then
> >> >> delete
> >> >> >>>> the email
> >> >> >>>>> > and any copies of it. Opinions, conclusion (etc)
> that do not
> >> >> >>>> relate to the
> >> >> >>>>> > official business of this company shall be
> understood as
> >> >> neither
> >> >> >>>> given nor
> >> >> >>>>> > endorsed by it. IG is a trading name of IG Markets
> Limited
> >> (a
> >> >> >>>> company
> >> >> >>>>> > registered in England and Wales, company number
> 04008957)
> >> and
> >> >> IG
> >> >> >>>> Index
> >> >> >>>>> > Limited (a company registered in England and Wales,
> company
> >> >> >> number
> >> >> >>>>> > 01190902). Registered address at Cannon Bridge
> House, 25
> >> >> Dowgate
> >> >> >>>> Hill,
> >> >> >>>>> > London EC4R 2YA. Both IG Markets Limited (register
> number
> >> >> 195355)
> >> >> >>>> and IG
> >> >> >>>>> > Index Limited (register number 114059) are
> authorised and
> >> >> >>>> regulated by the
> >> >> >>>>> > Financial Conduct Authority.
> >> >> >>>>> >
> >> >> >>>>>
> >> >> >>>>>
> >> >> >>>>> The information contained in this email is strictly
> confidential
> >> and
> >> >> >> for
> >> >> >>>> the use of the addressee only, unless otherwise indicated.
> If you
> >> are
> >> >> >> not
> >> >> >>>> the intended recipient, please do not read, copy, use or
> disclose
> >> to
> >> >> >> others
> >> >> >>>> this message or any attachment. Please also notify the
> sender by
> >> >> >> replying
> >> >> >>>> to this email or by telephone (+44(020 7896 0011) and then
> delete
> >> the
> >> >> >> email
> >> >> >>>> and any copies of it. Opinions, conclusion (etc) that do
> not relate
> >> >> to
> >> >> >> the
> >> >> >>>> official business of this company shall be understood as
> neither
> >> >> given
> >> >> >> nor
> >> >> >>>> endorsed by it. IG is a trading name of IG Markets Limited
> (a
> >> company
> >> >> >>>> registered in England and Wales, company number 04008957)
> and IG
> >> >> Index
> >> >> >>>> Limited (a company registered in England and Wales, company
> number
> >> >> >>>> 01190902). Registered address at Cannon Bridge House, 25
> Dowgate
> >> >> Hill,
> >> >> >>>> London EC4R 2YA. Both IG Markets Limited (register number
> 195355)
> >> >> and IG
> >> >> >>>> Index Limited (register number 114059) are authorised and
> regulated
> >> >> by
> >> >> >> the
> >> >> >>>> Financial Conduct Authority.
> >> >> >>>>>
> >> >> >>>>
> >> >> >>>>
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >> >
> >> The information contained in this email is strictly confidential
> and for
> >> the use of the addressee only, unless otherwise indicated. If you
> are not
> >> the intended recipient, please do not read, copy, use or disclose
> to others
> >> this message or any attachment. Please also notify the sender by
> replying
> >> to this email or by telephone (+44(020 7896 0011) and then delete
> the email
> >> and any copies of it. Opinions, conclusion (etc) that do not relate
> to the
> >> official business of this company shall be understood as neither
> given nor
> >> endorsed by it. IG is a trading name of IG Markets Limited (a
> company
> >> registered in England and Wales, company number 04008957) and IG
> Index
> >> Limited (a company registered in England and Wales, company number
> >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
> >> London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
> >> Index Limited (register number 114059) are authorised and regulated
> by the
> >> Financial Conduct Authority.
> >>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>
>
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Michael Pearce <Mi...@ig.com>.
Hi Jay et al.
On the replication using cluster id.
Two options depending on how a company wants or needs to implement their replication as there are options of doing this as a mesh, star, or ring.
Either as some companies enforce a provider/custom producer/consumer wrapper, it can be enforced the interceptors needed to be added, as you can enforce to get support of the platform you must use company x’s internal wrappers.
An alternative approach is to do this at the mirror maker, as Gwen noted we can actually make the mirror maker append this, like wise for a more complex implementation, where you can have multiple hops/routes this can also be the message routing.
Re the size of the clusterId, yes the auto generated one is “cough” hideous in size.
Most organisations though will only need a space to uniquely identify 127 (byte) clusters, or at worst 2 bytes (short) they need to mirror and replicate between, and as such can either as platform teams generally will roll out the mirror makers / replicators with the cluster can either manually adjust the clusterId on ZK, or use a mapping file all their mirror makers have to map the larger id to a smaller one.
Also as Gwen states this is something you should be able to opt into or not, this is the benefit of headers, if you don’t use it, it doesn’t use up the space, its entirely optional.
Regards
Mike
On 09/01/2017, 19:40, "Gwen Shapira" <gw...@confluent.io> wrote:
Mirroring + ClusterIDs:
Kafka clusters already have an automatically generated ID that is
exposed to clients:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
It sounds trivial to get MirrorMaker's consumer to grab this ID (which
always exists) from the source cluster and add it to the message
headers.
Obviously adding stuff to the message makes it larger so this should
be a feature you should be able to turn on/off depending on the
replication requirements. If you need the feature - you pay the price.
I don't think there is an immediate need to put an effort into
shrinking the Cluster ID, but others can comment on that.
On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> Hey Michael,
>
> Sorry for lagging, I've been out for the holiday break. What would help me
> would be to get to a consolidated list of use cases we've worked out and
> think add real value. I think getting down to the details really matters
> for these.
>
> The two lists I'm aware of are the one in the KIP and the one you gave in
> this email thread. Both seem to be more at the sketch level. Let me review
> the use cases and see if I have it right. If so let's get it in the design
> proposal.
>
> Your use cases:
>
> 1. Tracing IDs -- I get this use case and think I see how it would work.
> I think because the tracing is from a third party system there is some
> value to having a header versus a custom format to make this work fully
> transparently.
> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
> thinking below
> 3. End-to-end encryption -- I think what you are saying is that if you
> want "routing' fields that are accessible in transit without decrypting you
> need a scheme to leave this decrypted.I think the idea is to avoid
> decrypting to get a header, but I'm not sure if I get how this is better
> than a custom message format that doesn't encrypt the things you don't want
> encrypted since you have to be aware of the scheme anyway.
> 4. EOS -- I'm not convinced this is a good approach for EOS.
>
> Linked from KIP:
>
> 1. Schema-Registry-aware avro serialization -- not clear how this is
> better than keeping the schema id in the value
> 2. Polyglot kafka topics -- marginally better than a container format,
> but of course you still have to all agree on the the header and
> header=>serialization mapping so not sure if it's really better than a
> container format. Also not wild about polyglot topics :-)
> 3. End-to-End routing trail including mirroring --- I'd put this under
> tracing ids
> 4. Cross-service process tracing -- ditto
> 5. Signing -- I like this use case but it seems to depend on a feature
> we don't have and aren't proposing adding, right?
> 6. Per-record custom compression -- why do you want this? compressing
> per message won't work well compared to batch compression, right?
> 7. Large Message Support -- I think we can agree that although this
> might work it's clearly not the right way to support large messages in Kafka
>
> Matthias
>
> 1. Control messages --- Matthias, not sure if I get the motivation in
> streams for this. If we did want to do it, wouldn't it be easier to
> implement and by allowing users to send control messages (in the EOS sense)
> and adding a consumer config to allow either passing these on or
> suppressing them?
>
> So I if I were to summarize I'd list the use cases as follows:
>
> - Much better with headers: tracing IDs
> - Somewhat better: end-to-end encryption, polyglot topics
> - Don't buy: Mirroring (though perhaps could be made to work with
> changes to the proposal), EOS, large messages, compression
> - Not sure: Control messages
>
> Does this match where other people are at? From my point of view for
> something like this where we are changing the core data model we should
> ideally get a handful of use cases to the point where they are really well
> thought out and we know they can use the feature. I think we have one solid
> one with tracing IDs, and I think maybe mirroring could be another with
> some work. I'm less sure about the others.
>
> Bi-directional Mirroring
> I think bidirectional mirroring could potentially become a compelling use
> case if we an find a way to make it work. If I understand it, the proposal
> is to add a header like "origin-cluster" with the cluster id of the Kafka
> cluster. Then modify the mirroring tool to disallow mirroring a message
> back to it's own origin cluster. This basic idea is definitely workable and
> useful, but can we really get to a good implementation using headers?
> I imagine the approach would be to add this header in the producer if some
> config is set. There are two problems:
>
> - You add a pretty big honking cluster ID to each message. For small
> messages this maybe larger than the contents itself.
> - What happens if someone somewhere forgets to set that config on some
> client somewhere?
>
> If someone forgets to set the config in some client somewhere I guess you
> have various options in mirror maker:
>
> 1. drop the data (pretty bad, now your data is out of sync)
> 2. crash (pretty bad--how do you recover)
> 3. insert some default (pretty bad since would also lead to data loss if
> that's not right)
>
> Basically although I think a cluster tag could be a good way to do
> bi-directional mirroring, I'm not sure this naive approach I described
> would get us to a usable implementation. This could be fixable. I think if
> there was some scheme to check required headers on the server that could
> alleviate the missing header problem. Not really sure what to do about the
> message bloat problem, but maybe there could be some way to shrink the
> cluster id?
>
> Cheers,
>
> -Jay
>
>
> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> @Jay, Ismael and a few others.
>>
>> Based on the discussion on kip-87, and the additional use case by Matthias.
>>
>> Just for clarity do we now have consensus on the earlier points (1)
>> headers are useful (2) headers should be native in Kafka?
>>
>> Cheers
>> Mike
>> ________________________________________
>> From: radai <ra...@gmail.com>
>> Sent: Monday, December 19, 2016 5:01 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>> having said that im perfectly fine with something like a "no payload" flag
>> (as opposed to null payload), which would mean such a msg is not returned
>> from poll() unless the user code specifically "opts in" and would serve
>> only to carry headers (making it a control message)
>>
>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com> wrote:
>>
>> > if all you want is to tag "end of stream" wouldnt a header on the last
>> msg
>> > in the stream be enough?
>> >
>> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <ma...@confluent.io>
>> > wrote:
>> >
>> >> The details about headers for control messages are still to define. But
>> >> yes, the idea is to have some common default behavior that clients would
>> >> need to implement.
>> >>
>> >> The point is, that "regular headers" add meta data to regular messages.
>> >> Thus, those messages will be returned to the user via .poll(). And after
>> >> the message is received the user can check if meta data is present and
>> >> read it.
>> >>
>> >> For control messages, we do not want those to pop up via .poll() as
>> >> those are no regular messages. A client would need to opt-in to see
>> >> those messages (either via poll() or maybe a callback). Thus, we need
>> >> some special (standardized) header IDs that indicate control messages
>> >> that should not be returned to the user via poll() by default.
>> >>
>> >>
>> >> -Matthias
>> >>
>> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
>> >> > Matthias,
>> >> >
>> >> > Thanks for your input. I'm +1 on control messages as they seem to be
>> >> the
>> >> > simplest way to implement watermarks (
>> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102),
>> a
>> >> > feature that would add a lot of value to Kafka Streams IMHO.
>> >> >
>> >> > Your argument that the control-message indicator belongs in the
>> >> client-only
>> >> > section of the record format make sense. Just to make sure I
>> >> understand,
>> >> > are you suggesting that control messages would be indicated by a
>> >> standard
>> >> > reserved header? By standard, I mean that ALL Kafka consumers would
>> >> know
>> >> > to handle these messages differently (possibly just ignoring them).
>> >> This
>> >> > would need to be added to the specification of the consumer protocol
>> so
>> >> > that all Kafka clients implement it, right? I think it's a good idea
>> >> but
>> >> > just checking.
>> >> >
>> >> > Cheers,
>> >> >
>> >> > Roger
>> >> >
>> >> >
>> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>> matthias@confluent.io
>> >> >
>> >> > wrote:
>> >> >
>> >> >> Yes and no. I did overload the term "control message".
>> >> >>
>> >> >> EOS control messages are for client-broker communication and thus
>> never
>> >> >> exposed to any application. And I think this is a good design because
>> >> >> broker needs to understand those control messages. Thus, this should
>> be
>> >> >> a protocol change.
>> >> >>
>> >> >> The type of control messages I have in mind are for client-client
>> >> >> (application-application) communication and the broker is agnostic to
>> >> >> them. Thus, it should not be a protocol change.
>> >> >>
>> >> >>
>> >> >> -Matthias
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 12/14/16 9:42 AM, radai wrote:
>> >> >>> arent control messages getting pushed as their own top level
>> protocol
>> >> >>> change (and a fairly massive one) for the transactions KIP ?
>> >> >>>
>> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>> >> matthias@confluent.io>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> I want to add a completely new angle to this discussion. For this,
>> I
>> >> >>>> want to propose an extension for the headers feature that enables
>> new
>> >> >>>> uses cases -- and those new use cases might convince people to
>> >> support
>> >> >>>> headers (of course including the larger scoped proposal).
>> >> >>>>
>> >> >>>> Extended Proposal:
>> >> >>>>
>> >> >>>> Allow messages with a certain header key to be special "control
>> >> >>>> messages" (w/ o w/o payload) that are not exposed to an application
>> >> via
>> >> >>>> .poll().
>> >> >>>>
>> >> >>>> Thus, a consumer client would automatically skip over those
>> >> messages. If
>> >> >>>> an application knows about embedded control messages, it can "sing
>> >> up"
>> >> >>>> to those messages by the consumer client and either get a callback
>> or
>> >> >>>> the consumer auto-drop for this messages gets disabled (allowing to
>> >> >>>> consumer those messages via poll()).
>> >> >>>>
>> >> >>>> (The details need further considerations/discussion. I just want to
>> >> >>>> sketch the main idea.)
>> >> >>>>
>> >> >>>> Usage:
>> >> >>>>
>> >> >>>> There is a shared topic (ie, used by multiple applications) and a
>> >> >>>> producer application wants to embed a special message in the topic
>> >> for a
>> >> >>>> dedicated consumer application. Because only one application will
>> >> >>>> understand this message, it cannot be a regular message as this
>> would
>> >> >>>> break all applications that do not understand this message. The
>> >> producer
>> >> >>>> application would set a special metadata key and no consumer
>> >> application
>> >> >>>> would see this control message by default because they did not
>> enable
>> >> >>>> their consumer client to return this message in poll() (and the
>> >> client
>> >> >>>> would just drop this message with special metadata key). Only the
>> >> single
>> >> >>>> application that should receive this message, will subscribe to
>> this
>> >> >>>> message on its consumer client and process it.
>> >> >>>>
>> >> >>>>
>> >> >>>> Concrete Use Case: Kafka Streams
>> >> >>>>
>> >> >>>> In Kafka Streams, we would like to propagate "control messages"
>> from
>> >> >>>> subtopology to subtopology. There are multiple scenarios for which
>> >> this
>> >> >>>> would be useful. For example, currently we do not guarantee a
>> >> >>>> "consistent shutdown" of an application. By this, I mean that input
>> >> >>>> records might not be completely processed by the whole topology
>> >> because
>> >> >>>> the application shutdown happens "in between" and an intermediate
>> >> result
>> >> >>>> topic gets "stock" in an intermediate topic. Thus, a user would see
>> >> an
>> >> >>>> committed offset of the source topic of the application, but no
>> >> >>>> corresponding result record in the output topic.
>> >> >>>>
>> >> >>>> Having "shutdown markers" would allow us, to first stop the
>> upstream
>> >> >>>> subtopology and write this marker into the intermediate topic and
>> the
>> >> >>>> downstream subtopology would only shut down itself after is sees
>> the
>> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
>> >> >>>> "in-flight" messages got stuck in intermediate topics.
>> >> >>>>
>> >> >>>>
>> >> >>>> A similar usage would be for KIP-95 (Incremental Batch Processing).
>> >> >>>> There was a discussion about the proposed metadata topic, and we
>> >> could
>> >> >>>> avoid this metadata topic if we would have "control messages".
>> >> >>>>
>> >> >>>>
>> >> >>>> Right now, we cannot insert an "application control message"
>> because
>> >> >>>> Kafka Streams does not own all topics it read/writes and thus might
>> >> >>>> break other consumer application (as described above) if we inject
>> >> >>>> random messages that are not understood by other apps.
>> >> >>>>
>> >> >>>>
>> >> >>>> Of course, one can work around "embedded control messaged" by using
>> >> an
>> >> >>>> additional topic to propagate control messaged between application
>> >> (as
>> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
>> >> there
>> >> >>>> are major concerns about adding this metadata topic in the KIP and
>> >> this
>> >> >>>> shows that other application that need a similar pattern might
>> profit
>> >> >>>> from topic embedded "control messages", too.
>> >> >>>>
>> >> >>>>
>> >> >>>> One last important consideration: those "control messages" are used
>> >> for
>> >> >>>> client to client communication and are not understood by the
>> broker.
>> >> >>>> Thus, those messages should not be enabled within the message
>> format
>> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
>> >> headers
>> >> >>>> would be a nice way to implement them. Because KIP-82 did consider
>> >> key
>> >> >>>> namespaces for metatdata keys, this extension should not be an own
>> >> KIP
>> >> >>>> but should be included in KIP-82 to reserve a namespace for
>> "control
>> >> >>>> message" in the first place.
>> >> >>>>
>> >> >>>>
>> >> >>>> Sorry for the long email... Looking forward to your feedback.
>> >> >>>>
>> >> >>>>
>> >> >>>> -Matthias
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>> >> >>>>> Hi Jun
>> >> >>>>>
>> >> >>>>> 100) each time a transaction exits a jvm for a remote system
>> >> (HTTP/JMS/
>> >> >>>> Hopefully one day kafka) the APM tools stich in a unique id
>> (though I
>> >> >>>> believe it contains the end2end uuid embedded in this id), on
>> >> receiving
>> >> >> the
>> >> >>>> message at the receiving JVM the apm code takes this out, and
>> >> continues
>> >> >> its
>> >> >>>> tracing on the that new thread. Both JVM’s (and other languages the
>> >> APM
>> >> >>>> tool supports) send this data async back to the central controllers
>> >> >> where
>> >> >>>> the stiching togeather occurs. For this they need some header space
>> >> for
>> >> >>>> them to put this id.
>> >> >>>>>
>> >> >>>>> 101) Yes indeed we have a business transaction Id in the payload.
>> >> >> Though
>> >> >>>> this is a system level tracing, that we need to have marry up. Also
>> >> as
>> >> >> per
>> >> >>>> note on end2end encryption we’d be unable to prove the flow if the
>> >> >> payload
>> >> >>>> is encrypted as we’d not have access to this at certain points of
>> the
>> >> >> flow
>> >> >>>> through the infrastructure/platform.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> 103) As said we use this mechanism in IG very successfully, as
>> >> stated
>> >> >>>> per key we guarantee the transaction producing app to handle the
>> >> >>>> transaction of a key at one DC unless at point of critical failure
>> >> >> where we
>> >> >>>> have to flip processing to another. We care about key ordering.
>> >> >>>>> I disagree on the offset comment for the partition solution unless
>> >> you
>> >> >>>> do full ISR, or expensive full XA transactions even with partitions
>> >> you
>> >> >>>> cannot fully guarantee offsets would match.
>> >> >>>>>
>> >> >>>>> 105) Very much so, I need to have access at the platform level to
>> >> the
>> >> >>>> other meta data all mentioned, without having to need to have
>> access
>> >> to
>> >> >> the
>> >> >>>> encryption keys of the payload.
>> >> >>>>>
>> >> >>>>> 106)
>> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d need to have
>> a
>> >> >>>> global producerId register which would be very hard to
>> >> enforce/ensure is
>> >> >>>> current and correct, just to understand the message origins of its
>> >> >>>> region/az/cluster for routing.
>> >> >>>>> The client wrapper version, producerId can be the same, as
>> obviously
>> >> >> the
>> >> >>>> producer could upgrade its wrapper, as such we need to know what
>> >> wrapper
>> >> >>>> version the message is created with.
>> >> >>>>> Likewise the IP address, as stated we can have our producer move,
>> >> where
>> >> >>>> its IP would change.
>> >> >>>>>
>> >> >>>>> 107)
>> >> >>>>> UUID is set on the message by interceptors before actual producer
>> >> >>>> transport send. This is for platform level message dedupe
>> guarantee,
>> >> the
>> >> >>>> business payload should be agnostic to this. Please see
>> >> >>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
>> >> ection.html
>> >> >>>> note this is not touching business payloads.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
>> >> >>>>>
>> >> >>>>> Hi, Michael,
>> >> >>>>>
>> >> >>>>> Thanks for the reply. I find it very helpful.
>> >> >>>>>
>> >> >>>>> Data lineage:
>> >> >>>>> 100. I'd like to understand the APM use case a bit more. It
>> >> sounds
>> >> >>>> like
>> >> >>>>> that those APM plugins can generate a transaction id that we
>> >> could
>> >> >>>>> potentially put in the header of every message. How would you
>> >> >>>> typically
>> >> >>>>> make use of such transaction ids? Are there other metadata
>> >> >>>> associated with
>> >> >>>>> the transaction id and if so, how are they propagated
>> >> downstream?
>> >> >>>>>
>> >> >>>>> 101. For the finance use case, if the concept of transaction
>> is
>> >> >>>> important,
>> >> >>>>> wouldn't it be typically included in the message payload
>> >> instead of
>> >> >>>> as an
>> >> >>>>> optional header field?
>> >> >>>>>
>> >> >>>>> 102. The data lineage that Altas and Navigator support seems
>> to
>> >> be
>> >> >>>> at the
>> >> >>>>> dataset level, not per record level? So, not sure if per
>> message
>> >> >>>> headers
>> >> >>>>> are relevant there.
>> >> >>>>>
>> >> >>>>> Mirroring:
>> >> >>>>> 103. The benefit of using separate partitions is that it
>> >> >> potentially
>> >> >>>> makes
>> >> >>>>> it easy to preserve offsets during mirroring. This will make
>> it
>> >> >>>> easier for
>> >> >>>>> consumer to switch clusters. Currently, the consumers can
>> switch
>> >> >>>> clusters
>> >> >>>>> by using the timestampToOffset() api, but it has to deal with
>> >> >>>> duplicates.
>> >> >>>>> Good point on the issue with log compact and I am not sure how
>> >> to
>> >> >>>> address
>> >> >>>>> this. However, even if we mirror into the existing partitions,
>> >> the
>> >> >>>> ordering
>> >> >>>>> for messages generated from different clusters seems
>> >> >>>> non-deterministic
>> >> >>>>> anyway. So, it seems that the consumers already have to deal
>> >> with
>> >> >>>> that? If
>> >> >>>>> a topic is compacted, does that mean which messages are
>> >> preserved
>> >> >> is
>> >> >>>> also
>> >> >>>>> non-deterministic across clusters?
>> >> >>>>>
>> >> >>>>> 104. Good point on partition key.
>> >> >>>>>
>> >> >>>>> End-to-end encryption:
>> >> >>>>> 105. So, it seems end-to-end encryption is useful. Are headers
>> >> >>>> useful there?
>> >> >>>>>
>> >> >>>>> Auditing:
>> >> >>>>> 106. It seems other than the UUID, all other metadata are per
>> >> >>>> producer?
>> >> >>>>>
>> >> >>>>> EOS:
>> >> >>>>> 107. How are those UUIDs generated? I am not sure if they can
>> be
>> >> >>>> generated
>> >> >>>>> in the producer library. An application may send messages
>> >> through a
>> >> >>>> load
>> >> >>>>> balancer and on retry, the same message could be routed to a
>> >> >>>> different
>> >> >>>>> producer instance. So, it seems that the application has to
>> >> >> generate
>> >> >>>> the
>> >> >>>>> UUIDs. In that case, shouldn't the application just put the
>> >> UUID in
>> >> >>>> the
>> >> >>>>> payload?
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>>
>> >> >>>>> Jun
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>> >> >>>> Michael.Pearce@ig.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> > Hi Jun.
>> >> >>>>> >
>> >> >>>>> > Per Transaction Tracing / Data Lineage.
>> >> >>>>> >
>> >> >>>>> > As Stated in the KIP this has the first use case of how many
>> >> APM
>> >> >>>> tools now
>> >> >>>>> > work.
>> >> >>>>> > I would find it impossible for any one to argue this is not
>> >> >>>> important or a
>> >> >>>>> > niche market as it has its own gartner report for this
>> space.
>> >> >> Such
>> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular are
>> >> but a
>> >> >>>> few.
>> >> >>>>> >
>> >> >>>>> > Likewise these APM tools can help very rapidly track down
>> >> issues
>> >> >>>> and
>> >> >>>>> > automatically capture metrics, perform actions based on
>> >> >> unexpected
>> >> >>>> behavior
>> >> >>>>> > to auto recover services.
>> >> >>>>> >
>> >> >>>>> > Before mentioning looking at aggregated stats, in these
>> cases
>> >> >> where
>> >> >>>>> > actually on critical flows we cannot afford to have
>> aggregated
>> >> >>>> rolled up
>> >> >>>>> > stats only.
>> >> >>>>> >
>> >> >>>>> > With the APM tool we use its actually able to detect a
>> single
>> >> >>>> transaction
>> >> >>>>> > failure and capture the thread traces in the JVM where it
>> >> failed
>> >> >>>> and
>> >> >>>>> > everything for us, to the point it sends us alerts where we
>> >> have
>> >> >>>> this
>> >> >>>>> > giving the line number of the code that caused it, the
>> >> >> transaction
>> >> >>>> trace
>> >> >>>>> > through all the services and endpoints (supported) upto the
>> >> point
>> >> >>>> of
>> >> >>>>> > failure, it can also capture the data in and out (so we can
>> >> >>>> replay).
>> >> >>>>> > Because atm Kafka doesn’t support us being able to stich in
>> >> these
>> >> >>>> tracing
>> >> >>>>> > transaction ids natively, we cannot get these benefits as
>> >> such is
>> >> >>>> limiting
>> >> >>>>> > our ability support apps and monitor them to the same
>> >> standards
>> >> >> we
>> >> >>>> come to
>> >> >>>>> > expect when on a kafka flow.
>> >> >>>>> >
>> >> >>>>> > This actually ties in with Data Lineage, as the same tracing
>> >> can
>> >> >>>> be used
>> >> >>>>> > to back stich this. Essentially many times due to the sums
>> of
>> >> >> money
>> >> >>>>> > involved there are disputes, and typically as a financial
>> >> >>>> institute the
>> >> >>>>> > easiest and cleanest way to prove when disputes arise is to
>> >> >>>> present the
>> >> >>>>> > actual flow and processes involved in a transaction.
>> >> >>>>> >
>> >> >>>>> > Likewise as Hadoop matures its evident this case is
>> >> important, as
>> >> >>>> tools
>> >> >>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera led)
>> >> are
>> >> >>>> evident
>> >> >>>>> > also I believe the importance here is very much NOT just a
>> >> >>>> financial issue.
>> >> >>>>> >
>> >> >>>>> > From a MDM point of view any company wanting to care about
>> >> Data
>> >> >>>> Quality
>> >> >>>>> > and Data Governance - Data Lineage is a key piece in this
>> >> puzzle.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE Mirroring,
>> >> >>>>> >
>> >> >>>>> > As per the KIP in-fact this is exactly what we do re cluster
>> >> id,
>> >> >>>> to mirror
>> >> >>>>> > a network of clusters between AZ’s / Regions. We know a
>> >> >>>> transaction for a
>> >> >>>>> > key will be done within a AZ/Region, as such we know the
>> >> write
>> >> >> to
>> >> >>>> kafka
>> >> >>>>> > would be ordered per key. But we need eventual view of that
>> >> >> across
>> >> >>>> in our
>> >> >>>>> > other regions/az’s. When we have complete AZ or Region
>> >> failure we
>> >> >>>> know
>> >> >>>>> > there will be a brief interruption whilst those transactions
>> >> are
>> >> >>>> moved to
>> >> >>>>> > another region but we expect after it to continue.
>> >> >>>>> >
>> >> >>>>> > As mentioned having separate Partions to do this starts to
>> get
>> >> >>>>> > ugly/complicated for us:
>> >> >>>>> > how would I do compaction where a key is in two partitions?
>> >> >>>>> > How do we balance consumers so where multiple partitions
>> with
>> >> the
>> >> >>>> same key
>> >> >>>>> > goto the same consumer
>> >> >>>>> > What do you do if cluster 1 has 5 partitions but cluster 20
>> >> has
>> >> >> 10
>> >> >>>> because
>> >> >>>>> > its larger kit in our more core DC’s, as such key to
>> partition
>> >> >>>> mappings for
>> >> >>>>> > consumers get even more complicated.
>> >> >>>>> > What do you do if we add or remove a complete region
>> >> >>>>> >
>> >> >>>>> > Where as simple mirror will work we just need to ensure we
>> >> don’t
>> >> >>>> have a
>> >> >>>>> > cycle which we can do with clusterId.
>> >> >>>>> >
>> >> >>>>> > We even have started to look at shortest path mirror routing
>> >> >> based
>> >> >>>> on
>> >> >>>>> > clusterId, if we also had the region and az info on the
>> >> >> originating
>> >> >>>>> > message, this we have not implemented but some ideas come
>> from
>> >> >>>> network
>> >> >>>>> > routing, and also the dispatcher router in apache qpid.
>> >> >>>>> >
>> >> >>>>> > Also we need to have data perimeters e.g. certain data
>> cannot
>> >> >> leave
>> >> >>>>> > certain countries borders. We want this all automated so
>> that
>> >> at
>> >> >>>> the
>> >> >>>>> > platform level without having to touch or look at the
>> business
>> >> >>>> data inside
>> >> >>>>> > we can have headers we can put tags into so that we can
>> ensure
>> >> >>>> this doesn’t
>> >> >>>>> > occur when we mirror. (actually links in to data lineage /
>> >> >> tracing
>> >> >>>> as again
>> >> >>>>> > we need to tag messages at a platform level) Examples are we
>> >> are
>> >> >>>> not
>> >> >>>>> > allowed Private customer details to leave Switzerland, yet
>> we
>> >> >> need
>> >> >>>> those
>> >> >>>>> > systems integrated.
>> >> >>>>> >
>> >> >>>>> > Lastly around mirroring we have a partionKey field, as the
>> key
>> >> >>>> used for
>> >> >>>>> > portioning logic != compaction key all the time but we want
>> to
>> >> >>>> preserve it
>> >> >>>>> > for when we mirror so that if source cluster partition count
>> >> !=
>> >> >>>> destination
>> >> >>>>> > cluster partition count we can honour the same partitioning
>> >> >> logic.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE End 2 End encryption
>> >> >>>>> >
>> >> >>>>> > As I believe mentioned just before, the solution you mention
>> >> just
>> >> >>>> doesn’t
>> >> >>>>> > cut the mustard these days with many regulators. An
>> operations
>> >> >>>> person with
>> >> >>>>> > access to the box should not be able to have access to the
>> >> data.
>> >> >>>> Many now
>> >> >>>>> > actually impose quite literally the implementation expected
>> >> being
>> >> >>>> end2end
>> >> >>>>> > encryption for certain data (Singapore for us is one that I
>> am
>> >> >>>> most aware
>> >> >>>>> > of). In fact we’re even now needing encrypt the data and
>> store
>> >> >> the
>> >> >>>> keys in
>> >> >>>>> > HSM modules.
>> >> >>>>> >
>> >> >>>>> > Likewise the performance penalty on encrypting decrypting as
>> >> you
>> >> >>>> produce
>> >> >>>>> > over wire, then again encrypt decrypt as the data is stored
>> on
>> >> >> the
>> >> >>>> brokers
>> >> >>>>> > disks and back again, then again encrypted and decrypted
>> back
>> >> >> over
>> >> >>>> the wire
>> >> >>>>> > each time for each consumer all adds up, ignoring this
>> >> doubling
>> >> >>>> with mirror
>> >> >>>>> > makers etc. simply encrypting the value once on write by the
>> >> >>>> client and
>> >> >>>>> > again decrypting on consume by the consumer is far more
>> >> >>>> performant, but
>> >> >>>>> > then the routing and platform meta data needs to be separate
>> >> >> (thus
>> >> >>>> headers)
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE Auditing:
>> >> >>>>> >
>> >> >>>>> > Our Auditing needs are:
>> >> >>>>> > Producer Id,
>> >> >>>>> > Origin Cluster Id that message first produced into
>> >> >>>>> > Origin AZ – agreed we can derive this if we have cluster id,
>> >> but
>> >> >>>> it makes
>> >> >>>>> > resolving this for audit reporting a lot easier.
>> >> >>>>> > Origin Region – agreed we can derive this if we have cluster
>> >> id,
>> >> >>>> but it
>> >> >>>>> > makes resolving this for audit reporting a lot easier.
>> >> >>>>> > Unique Message Identification (this is not the same as
>> >> >> transaction
>> >> >>>>> > tracing) – note offset and partition are not the same, as
>> >> when we
>> >> >>>> mirror or
>> >> >>>>> > have for what ever system failure duplicate send,
>> >> >>>>> > Custom Client wrapper version (where organizations have to
>> >> wrap
>> >> >>>> the kafka
>> >> >>>>> > client for added features) so we know what version of the
>> >> wrapper
>> >> >>>> is used
>> >> >>>>> > Producer IP address (in case of clients being in our vm/open
>> >> >> stack
>> >> >>>> infra
>> >> >>>>> > where they can move around, producer id will stay the same
>> but
>> >> >>>> this would
>> >> >>>>> > change)
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE Once and only once delivery case
>> >> >>>>> >
>> >> >>>>> > Using the same Message UUID for auditing we can achieve this
>> >> >> quite
>> >> >>>> simply.
>> >> >>>>> >
>> >> >>>>> > As per how some other brokers do this (cough qpid, artemis)
>> >> >>>> message uuid
>> >> >>>>> > are used to dedupe where message is sent and produced but
>> the
>> >> >>>> client didn’t
>> >> >>>>> > receive the ack, and there for replays the send, by having a
>> >> >>>> unique message
>> >> >>>>> > id per message, this can be filtered out, on consumers where
>> >> >>>> message
>> >> >>>>> > delivery may occur twice for what ever reasons a message
>> uuid
>> >> can
>> >> >>>> be used
>> >> >>>>> > to remove duplicates being deliverd , like wise we can do
>> >> this in
>> >> >>>> the
>> >> >>>>> > mirrormakers so if we detect a dupe message we can avoid
>> >> >>>> replicating it.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > Cheers
>> >> >>>>> > Mike
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io> wrote:
>> >> >>>>> >
>> >> >>>>> > Since this KIP affects message format, wire protocol,
>> >> apis, I
>> >> >>>> think
>> >> >>>>> > it's
>> >> >>>>> > worth spending a bit more time to nail down the concrete
>> >> use
>> >> >>>> cases. It
>> >> >>>>> > would be bad if we add this feature, but when start
>> >> >>>> implementing it
>> >> >>>>> > for say
>> >> >>>>> > mirroring, we then realize that header is not the best
>> >> >>>> approach.
>> >> >>>>> > Initially,
>> >> >>>>> > I thought I was convinced of the use cases of headers
>> and
>> >> was
>> >> >>>> trying to
>> >> >>>>> > write down a few use cases to convince others. That's
>> >> when I
>> >> >>>> became
>> >> >>>>> > less
>> >> >>>>> > certain. For me to be convinced, I just want to see two
>> >> >> strong
>> >> >>>> use
>> >> >>>>> > cases
>> >> >>>>> > (instead of 10 maybe use cases) in the third-party
>> space.
>> >> The
>> >> >>>> reason is
>> >> >>>>> > that when we discussed the use cases within a company,
>> >> often
>> >> >>>> it ends
>> >> >>>>> > with
>> >> >>>>> > "we can't force everyone to use this standard since we
>> may
>> >> >>>> have to
>> >> >>>>> > integrate with third-party tools".
>> >> >>>>> >
>> >> >>>>> > At present, I am not sure why headers are useful for
>> >> things
>> >> >>>> like
>> >> >>>>> > schemaId
>> >> >>>>> > or encryption. In order to do anything useful to the
>> >> value,
>> >> >>>> one needs
>> >> >>>>> > to
>> >> >>>>> > know the schemaId or how data is encrypted, but header
>> is
>> >> >>>> optional.
>> >> >>>>> > But, I
>> >> >>>>> > can be convinced if someone (Radai, Sean, Todd?)
>> provides
>> >> >> more
>> >> >>>> details
>> >> >>>>> > on
>> >> >>>>> > the argument.
>> >> >>>>> >
>> >> >>>>> > I am not very sure header is the best approach for
>> >> mirroring
>> >> >>>> either. If
>> >> >>>>> > someone has thought about this more, I'd be happy to
>> hear.
>> >> >>>>> >
>> >> >>>>> > I can see the data lineage use case. I am just not sure
>> >> how
>> >> >>>> widely
>> >> >>>>> > applicable this is. If someone familiar with this space
>> >> can
>> >> >>>> justify
>> >> >>>>> > this is
>> >> >>>>> > a significant use case, say in the finance industry,
>> this
>> >> >>>> would be a
>> >> >>>>> > strong
>> >> >>>>> > use case.
>> >> >>>>> >
>> >> >>>>> > I can see the auditing use case. I am just not sure if a
>> >> >> native
>> >> >>>>> > producer id
>> >> >>>>> > solves that problem. If there are additional metadata
>> >> that's
>> >> >>>> worth
>> >> >>>>> > collecting but not covered by the producer id, that
>> would
>> >> >> make
>> >> >>>> this a
>> >> >>>>> > strong use case.
>> >> >>>>> >
>> >> >>>>> > Thanks,
>> >> >>>>> >
>> >> >>>>> > Jun
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>> >> >>>> radai.rosenblatt@gmail.com>
>> >> >>>>> > wrote:
>> >> >>>>> >
>> >> >>>>> > > this KIP is about enabling headers, nothing more
>> nothing
>> >> >>>> less - so
>> >> >>>>> > no,
>> >> >>>>> > > broker-side use of headers is not in the KIP scope.
>> >> >>>>> > >
>> >> >>>>> > > obviously though, once you have headers potential use
>> >> cases
>> >> >>>> could
>> >> >>>>> > include
>> >> >>>>> > > broker-side header-aware interceptors (which would be
>> >> the
>> >> >>>> topic of
>> >> >>>>> > other
>> >> >>>>> > > future KIPs).
>> >> >>>>> > >
>> >> >>>>> > > a trivially clear use case (to me) would be using such
>> >> >>>> broker-side
>> >> >>>>> > > interceptors to enforce compliance with organizational
>> >> >>>> policies - it
>> >> >>>>> > would
>> >> >>>>> > > make our SREs lives much easier if instead of
>> >> retroactively
>> >> >>>>> > discovering
>> >> >>>>> > > "rogue" topics/users those messages would have been
>> >> >> rejected
>> >> >>>>> > up-front.
>> >> >>>>> > >
>> >> >>>>> > > the kafka broker code is lacking any such
>> extensibility
>> >> >>>> support
>> >> >>>>> > (beyond
>> >> >>>>> > > maybe authorizer) which is why these use cases were
>> left
>> >> >> out
>> >> >>>> of the
>> >> >>>>> > "case
>> >> >>>>> > > for headers" doc - broker extensibility is a separate
>> >> >>>> discussion.
>> >> >>>>> > >
>> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
>> >> >>>> gwen@confluent.io>
>> >> >>>>> > wrote:
>> >> >>>>> > >
>> >> >>>>> > > > Woah, I wasn't aware this is something we'll do. It
>> >> >> wasn't
>> >> >>>> in the
>> >> >>>>> > KIP,
>> >> >>>>> > > > right?
>> >> >>>>> > > >
>> >> >>>>> > > > I guess we could do it the same way ACLs currently
>> >> work.
>> >> >>>>> > > > I had in mind something that will allow admins to
>> >> apply
>> >> >>>> rules to
>> >> >>>>> > the
>> >> >>>>> > > > new create/delete/config topic APIs. So Todd can
>> >> decide
>> >> >> to
>> >> >>>> reject
>> >> >>>>> > > > "create topic" requests that ask for more than 40
>> >> >>>> partitions, or
>> >> >>>>> > > > require exactly 3 replicas, or no more than 50GB
>> >> >> partition
>> >> >>>> size,
>> >> >>>>> > etc.
>> >> >>>>> > > >
>> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are planning to
>> >> apply
>> >> >>>> more
>> >> >>>>> > rules
>> >> >>>>> > > > to requests (and I think we should), we may want a
>> bit
>> >> >>>> more generic
>> >> >>>>> > > > design around that.
>> >> >>>>> > > >
>> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>> >> >>>> radai.rosenblatt@gmail.com>
>> >> >>>>> > > wrote:
>> >> >>>>> > > > > "wouldn't you be in the business of making sure
>> >> >> everyone
>> >> >>>> uses
>> >> >>>>> > them
>> >> >>>>> > > > > properly?"
>> >> >>>>> > > > >
>> >> >>>>> > > > > thats where a broker-side plugin would come handy
>> -
>> >> any
>> >> >>>> incoming
>> >> >>>>> > > message
>> >> >>>>> > > > > that does not conform to org policy (read - does
>> not
>> >> >>>> have the
>> >> >>>>> > proper
>> >> >>>>> > > > > headers) gets thrown out (with an error returned
>> to
>> >> >> user)
>> >> >>>>> > > > >
>> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
>> >> >>>> tpalino@gmail.com>
>> >> >>>>> > wrote:
>> >> >>>>> > > > >
>> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on this one
>> :)
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> Producing counts to a topic is part of it, but
>> >> that’s
>> >> >>>> only
>> >> >>>>> > part. So
>> >> >>>>> > > you
>> >> >>>>> > > > >> count you have 100 messages in topic A. When you
>> >> >> mirror
>> >> >>>> topic A
>> >> >>>>> > to
>> >> >>>>> > > > another
>> >> >>>>> > > > >> cluster, you have 99 messages. Where was your
>> >> problem?
>> >> >>>> Or
>> >> >>>>> > worse, you
>> >> >>>>> > > > have
>> >> >>>>> > > > >> 100 messages, but one producer duplicated
>> messages
>> >> and
>> >> >>>> another
>> >> >>>>> > one
>> >> >>>>> > > lost
>> >> >>>>> > > > >> messages. You need details about where the
>> message
>> >> >> came
>> >> >>>> from in
>> >> >>>>> > order
>> >> >>>>> > > to
>> >> >>>>> > > > >> pinpoint problems when they happen. Source
>> producer
>> >> >>>> info, where
>> >> >>>>> > it was
>> >> >>>>> > > > >> produced into your infrastructure, and when it
>> was
>> >> >>>> produced.
>> >> >>>>> > This
>> >> >>>>> > > > requires
>> >> >>>>> > > > >> you to add the information to the message.
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> And yes, you still need to maintain your clients.
>> >> So
>> >> >>>> maybe my
>> >> >>>>> > original
>> >> >>>>> > > > >> example was not the best. My thoughts on not
>> >> wanting
>> >> >> to
>> >> >>>> be
>> >> >>>>> > responsible
>> >> >>>>> > > > for
>> >> >>>>> > > > >> message formats stands, because that’s very much
>> >> >>>> separate from
>> >> >>>>> > the
>> >> >>>>> > > > client.
>> >> >>>>> > > > >> As you know, we have our own internal client
>> >> library
>> >> >>>> that can
>> >> >>>>> > insert
>> >> >>>>> > > the
>> >> >>>>> > > > >> right headers, and right now inserts the right
>> >> audit
>> >> >>>>> > information into
>> >> >>>>> > > > the
>> >> >>>>> > > > >> message fields. If they exist, and assuming the
>> >> >> message
>> >> >>>> is Avro
>> >> >>>>> > > encoded.
>> >> >>>>> > > > >> What if someone wants to use JSON instead for a
>> >> good
>> >> >>>> reason?
>> >> >>>>> > What if
>> >> >>>>> > > > user X
>> >> >>>>> > > > >> wants to encrypt messages, but user Y does not?
>> >> >>>> Maintaining the
>> >> >>>>> > client
>> >> >>>>> > > > >> library is still much easier than maintaining the
>> >> >>>> message
>> >> >>>>> > formats.
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> -Todd
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
>> >> >>>> gwen@confluent.io
>> >> >>>>> > >
>> >> >>>>> > > wrote:
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> > Based on your last sentence, consider me
>> >> convinced
>> >> >> :)
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > I get why headers are critical for Mirroring
>> (you
>> >> >>>> need tags to
>> >> >>>>> > > prevent
>> >> >>>>> > > > >> > loops and sometimes to route messages to the
>> >> correct
>> >> >>>>> > destination).
>> >> >>>>> > > > >> > But why do you need headers to audit? We are
>> >> >> auditing
>> >> >>>> by
>> >> >>>>> > producing
>> >> >>>>> > > > >> > counts to a side topic (and I was under the
>> >> >>>> impression you do
>> >> >>>>> > the
>> >> >>>>> > > > >> > same), so we never need to modify the message.
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > Another thing - after we added headers,
>> wouldn't
>> >> you
>> >> >>>> be in the
>> >> >>>>> > > > >> > business of making sure everyone uses them
>> >> properly?
>> >> >>>> Making
>> >> >>>>> > sure
>> >> >>>>> > > > >> > everyone includes the right headers you need,
>> not
>> >> >>>> using the
>> >> >>>>> > header
>> >> >>>>> > > > >> > names you intend to use, etc. I don't think the
>> >> >>>> "policing"
>> >> >>>>> > business
>> >> >>>>> > > > >> > will ever go away.
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <
>> >> >>>>> > tpalino@gmail.com>
>> >> >>>>> > > > wrote:
>> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
>> with
>> >> the
>> >> >>>>> > workaround.
>> >> >>>>> > > Avro
>> >> >>>>> > > > >> means
>> >> >>>>> > > > >> > > that I have to be concerned with the format
>> of
>> >> the
>> >> >>>> messages
>> >> >>>>> > in
>> >> >>>>> > > > order to
>> >> >>>>> > > > >> > run
>> >> >>>>> > > > >> > > the infrastructure (audit, mirroring, etc.).
>> >> That
>> >> >>>> means
>> >> >>>>> > that I
>> >> >>>>> > > have
>> >> >>>>> > > > to
>> >> >>>>> > > > >> > > handle the schemas, and I have to enforce
>> rules
>> >> >>>> about good
>> >> >>>>> > > formats.
>> >> >>>>> > > > >> This
>> >> >>>>> > > > >> > is
>> >> >>>>> > > > >> > > not something I want to be in the business
>> of,
>> >> >>>> because I
>> >> >>>>> > should be
>> >> >>>>> > > > able
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > > run a service infrastructure without needing
>> >> to be
>> >> >>>> in the
>> >> >>>>> > weeds of
>> >> >>>>> > > > >> > dealing
>> >> >>>>> > > > >> > > with customer data formats.
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > > Trust me, a sizable portion of my support
>> time
>> >> is
>> >> >>>> spent
>> >> >>>>> > dealing
>> >> >>>>> > > with
>> >> >>>>> > > > >> > schema
>> >> >>>>> > > > >> > > issues. I really would like to get away from
>> >> that.
>> >> >>>> Maybe
>> >> >>>>> > I'd have
>> >> >>>>> > > > more
>> >> >>>>> > > > >> > time
>> >> >>>>> > > > >> > > for other hobbies. Like writing. ;)
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > > -Todd
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <
>> >> >>>>> > gwen@confluent.io>
>> >> >>>>> > > > wrote:
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > >> I'm pretty satisfied with the current
>> >> workarounds
>> >> >>>> (Avro
>> >> >>>>> > container
>> >> >>>>> > > > >> > >> format), so I'm not too excited about the
>> >> extra
>> >> >>>> work
>> >> >>>>> > required to
>> >> >>>>> > > do
>> >> >>>>> > > > >> > >> headers in Kafka. I absolutely don't mind it
>> >> if
>> >> >>>> you do
>> >> >>>>> > it...
>> >> >>>>> > > > >> > >> I think the Apache convention for "good
>> idea,
>> >> but
>> >> >>>> not
>> >> >>>>> > willing to
>> >> >>>>> > > > put
>> >> >>>>> > > > >> > >> any work toward it" is +0.5? anyway, that's
>> >> what
>> >> >> I
>> >> >>>> was
>> >> >>>>> > trying to
>> >> >>>>> > > > >> > >> convey :)
>> >> >>>>> > > > >> > >>
>> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino
>> <
>> >> >>>>> > tpalino@gmail.com>
>> >> >>>>> > > > >> wrote:
>> >> >>>>> > > > >> > >> > Well I guess my question for you, then, is
>> >> what
>> >> >>>> is
>> >> >>>>> > holding you
>> >> >>>>> > > > back
>> >> >>>>> > > > >> > from
>> >> >>>>> > > > >> > >> > full support for headers? What’s the bit
>> >> that
>> >> >>>> you’re
>> >> >>>>> > missing
>> >> >>>>> > > that
>> >> >>>>> > > > >> has
>> >> >>>>> > > > >> > you
>> >> >>>>> > > > >> > >> > under a full +1?
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> > -Todd
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
>> >> Shapira <
>> >> >>>>> > > gwen@confluent.io>
>> >> >>>>> > > > >> > wrote:
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> >> I know why people who support headers
>> >> support
>> >> >>>> them, and
>> >> >>>>> > I've
>> >> >>>>> > > > seen
>> >> >>>>> > > > >> > what
>> >> >>>>> > > > >> > >> >> the discussion is like.
>> >> >>>>> > > > >> > >> >>
>> >> >>>>> > > > >> > >> >> This is why I'm asking people who are
>> >> against
>> >> >>>> headers
>> >> >>>>> > > > (especially
>> >> >>>>> > > > >> > >> >> committers) what will make them change
>> >> their
>> >> >>>> mind - so
>> >> >>>>> > we can
>> >> >>>>> > > > get
>> >> >>>>> > > > >> > this
>> >> >>>>> > > > >> > >> >> part over one way or another.
>> >> >>>>> > > > >> > >> >>
>> >> >>>>> > > > >> > >> >> If I sound frustrated it is not at Radai,
>> >> Jun
>> >> >>>> or you
>> >> >>>>> > (Todd)...
>> >> >>>>> > > > I am
>> >> >>>>> > > > >> > >> >> just looking for something concrete we
>> can
>> >> do
>> >> >>>> to move
>> >> >>>>> > the
>> >> >>>>> > > > >> discussion
>> >> >>>>> > > > >> > >> >> along to the yummy design details (which
>> is
>> >> >> the
>> >> >>>>> > argument I
>> >> >>>>> > > > really
>> >> >>>>> > > > >> am
>> >> >>>>> > > > >> > >> >> looking forward to).
>> >> >>>>> > > > >> > >> >>
>> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
>> >> Palino <
>> >> >>>>> > > tpalino@gmail.com>
>> >> >>>>> > > > >> > wrote:
>> >> >>>>> > > > >> > >> >> > So, Gwen, to your question (even though
>> >> I’m
>> >> >>>> not a
>> >> >>>>> > > > committer)...
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > I have always been a strong supporter
>> of
>> >> >>>> introducing
>> >> >>>>> > the
>> >> >>>>> > > > concept
>> >> >>>>> > > > >> > of an
>> >> >>>>> > > > >> > >> >> > envelope to messages, which headers
>> >> >>>> accomplishes. The
>> >> >>>>> > > message
>> >> >>>>> > > > key
>> >> >>>>> > > > >> > is
>> >> >>>>> > > > >> > >> >> > already an example of a piece of
>> envelope
>> >> >>>>> > information. By
>> >> >>>>> > > > >> > providing a
>> >> >>>>> > > > >> > >> >> means
>> >> >>>>> > > > >> > >> >> > to do this within Kafka itself, and not
>> >> >>>> relying on
>> >> >>>>> > use-case
>> >> >>>>> > > > >> > specific
>> >> >>>>> > > > >> > >> >> > implementations, you make it much
>> easier
>> >> for
>> >> >>>>> > components to
>> >> >>>>> > > > >> > >> interoperate.
>> >> >>>>> > > > >> > >> >> It
>> >> >>>>> > > > >> > >> >> > simplifies development of all these
>> >> things
>> >> >>>> (message
>> >> >>>>> > routing,
>> >> >>>>> > > > >> > auditing,
>> >> >>>>> > > > >> > >> >> > encryption, etc.) because each one does
>> >> not
>> >> >>>> have to
>> >> >>>>> > reinvent
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> wheel.
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > It also makes it much easier from a
>> >> client
>> >> >>>> point of
>> >> >>>>> > view if
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> headers
>> >> >>>>> > > > >> > >> >> are
>> >> >>>>> > > > >> > >> >> > defined as part of the protocol and/or
>> >> >>>> message format
>> >> >>>>> > in
>> >> >>>>> > > > general
>> >> >>>>> > > > >> > >> because
>> >> >>>>> > > > >> > >> >> > you can easily produce and consume
>> >> messages
>> >> >>>> without
>> >> >>>>> > having
>> >> >>>>> > > to
>> >> >>>>> > > > >> take
>> >> >>>>> > > > >> > >> into
>> >> >>>>> > > > >> > >> >> > account specific cases. For example, I
>> >> want
>> >> >>>> to route
>> >> >>>>> > > messages,
>> >> >>>>> > > > >> but
>> >> >>>>> > > > >> > >> >> client A
>> >> >>>>> > > > >> > >> >> > doesn’t support the way audit
>> implemented
>> >> >>>> headers, and
>> >> >>>>> > > client
>> >> >>>>> > > > B
>> >> >>>>> > > > >> > >> doesn’t
>> >> >>>>> > > > >> > >> >> > support the way encryption or routing
>> >> >>>> implemented
>> >> >>>>> > headers,
>> >> >>>>> > > so
>> >> >>>>> > > > now
>> >> >>>>> > > > >> > my
>> >> >>>>> > > > >> > >> >> > application has to create some really
>> >> >> fragile
>> >> >>>> (my
>> >> >>>>> > > autocorrect
>> >> >>>>> > > > >> just
>> >> >>>>> > > > >> > >> tried
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> > make that “tragic”, which is probably
>> >> >>>> appropriate
>> >> >>>>> > too) code
>> >> >>>>> > > to
>> >> >>>>> > > > >> > strip
>> >> >>>>> > > > >> > >> >> > everything off, rather than just
>> >> consuming
>> >> >> the
>> >> >>>>> > messages,
>> >> >>>>> > > > picking
>> >> >>>>> > > > >> > out
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> 1
>> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested in, and
>> >> >>>> performing its
>> >> >>>>> > > function.
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > Honestly, this discussion has been
>> going
>> >> on
>> >> >>>> for a
>> >> >>>>> > long time,
>> >> >>>>> > > > and
>> >> >>>>> > > > >> > it’s
>> >> >>>>> > > > >> > >> >> > always “Oh, you came up with 2 use
>> cases,
>> >> >> and
>> >> >>>> yeah,
>> >> >>>>> > those
>> >> >>>>> > > use
>> >> >>>>> > > > >> cases
>> >> >>>>> > > > >> > >> are
>> >> >>>>> > > > >> > >> >> > real things that someone would want to
>> >> do.
>> >> >>>> Here’s an
>> >> >>>>> > > alternate
>> >> >>>>> > > > >> way
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > >> >> > implement them so let’s not do
>> headers.”
>> >> If
>> >> >>>> we have a
>> >> >>>>> > few
>> >> >>>>> > > use
>> >> >>>>> > > > >> cases
>> >> >>>>> > > > >> > >> that
>> >> >>>>> > > > >> > >> >> we
>> >> >>>>> > > > >> > >> >> > actually came up with, you can be sure
>> >> that
>> >> >>>> over the
>> >> >>>>> > next
>> >> >>>>> > > year
>> >> >>>>> > > > >> > >> there’s a
>> >> >>>>> > > > >> > >> >> > dozen others that we didn’t think of
>> that
>> >> >>>> someone
>> >> >>>>> > would like
>> >> >>>>> > > > to
>> >> >>>>> > > > >> > do. I
>> >> >>>>> > > > >> > >> >> > really think it’s time to stop
>> rehashing
>> >> >> this
>> >> >>>>> > discussion and
>> >> >>>>> > > > >> > instead
>> >> >>>>> > > > >> > >> >> focus
>> >> >>>>> > > > >> > >> >> > on a workable standard that we can
>> adopt.
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > -Todd
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
>> >> Palino
>> >> >> <
>> >> >>>>> > > > tpalino@gmail.com>
>> >> >>>>> > > > >> > >> wrote:
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> >> C. per message encryption
>> >> >>>>> > > > >> > >> >> >>> One drawback of this approach is that
>> >> this
>> >> >>>>> > significantly
>> >> >>>>> > > > reduce
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> effectiveness of compression, which
>> >> >> happens
>> >> >>>> on a
>> >> >>>>> > set of
>> >> >>>>> > > > >> > serialized
>> >> >>>>> > > > >> > >> >> >>> messages. An alternative is to enable
>> >> SSL
>> >> >>>> for wire
>> >> >>>>> > > > encryption
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> rely
>> >> >>>>> > > > >> > >> >> on
>> >> >>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS) for at
>> >> rest
>> >> >>>>> > encryption.
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
>> this
>> >> >> does
>> >> >>>> cover
>> >> >>>>> > the case
>> >> >>>>> > > > of
>> >> >>>>> > > > >> > >> removing
>> >> >>>>> > > > >> > >> >> a
>> >> >>>>> > > > >> > >> >> >> drive from the system, it will not
>> >> satisfy
>> >> >>>> most
>> >> >>>>> > compliance
>> >> >>>>> > > > >> > >> requirements
>> >> >>>>> > > > >> > >> >> for
>> >> >>>>> > > > >> > >> >> >> encryption of data as whoever has
>> >> access to
>> >> >>>> the
>> >> >>>>> > broker
>> >> >>>>> > > itself
>> >> >>>>> > > > >> > still
>> >> >>>>> > > > >> > >> has
>> >> >>>>> > > > >> > >> >> >> access to the unencrypted data. For
>> >> >>>> end-to-end
>> >> >>>>> > encryption
>> >> >>>>> > > you
>> >> >>>>> > > > >> > need to
>> >> >>>>> > > > >> > >> >> >> encrypt at the producer, before it
>> >> enters
>> >> >> the
>> >> >>>>> > system, and
>> >> >>>>> > > > >> decrypt
>> >> >>>>> > > > >> > at
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> >> consumer, after it exits the system.
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >> -Todd
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai
>> <
>> >> >>>>> > > > >> radai.rosenblatt@gmail.com
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > >> >> wrote:
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >>> another big plus of headers in the
>> >> >> protocol
>> >> >>>> is that
>> >> >>>>> > it
>> >> >>>>> > > would
>> >> >>>>> > > > >> > enable
>> >> >>>>> > > > >> > >> >> rapid
>> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside of core
>> >> kafka
>> >> >>>> and would
>> >> >>>>> > reduce
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> number of
>> >> >>>>> > > > >> > >> >> >>> future wire format changes required.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
>> >> >> represents
>> >> >>>> use
>> >> >>>>> > cases that
>> >> >>>>> > > > are
>> >> >>>>> > > > >> > not
>> >> >>>>> > > > >> > >> 100%
>> >> >>>>> > > > >> > >> >> >>> relevant to all users, and some of
>> them
>> >> >>>> require
>> >> >>>>> > rather
>> >> >>>>> > > > invasive
>> >> >>>>> > > > >> > wire
>> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing a good
>> recent
>> >> >>>> example of
>> >> >>>>> > this is
>> >> >>>>> > > > >> > kip-98.
>> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected to
>> be
>> >> a
>> >> >>>> very small
>> >> >>>>> > > > fraction of
>> >> >>>>> > > > >> > >> total
>> >> >>>>> > > > >> > >> >> >>> traffic and yet the changes are
>> >> invasive.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> every such wire format change
>> >> translates
>> >> >>>> into
>> >> >>>>> > painful and
>> >> >>>>> > > > slow
>> >> >>>>> > > > >> > >> >> adoption of
>> >> >>>>> > > > >> > >> >> >>> new versions.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> i think a lot of functionality
>> >> currently
>> >> >> in
>> >> >>>> KIPs
>> >> >>>>> > could be
>> >> >>>>> > > > "spun
>> >> >>>>> > > > >> > out"
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> implemented as opt-in plugins
>> >> transmitting
>> >> >>>> data over
>> >> >>>>> > > > headers.
>> >> >>>>> > > > >> > this
>> >> >>>>> > > > >> > >> >> would
>> >> >>>>> > > > >> > >> >> >>> keep the core wire format stable(r),
>> >> core
>> >> >>>> codebase
>> >> >>>>> > > smaller,
>> >> >>>>> > > > and
>> >> >>>>> > > > >> > >> avoid
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
>> >> required
>> >> >>>> to prove
>> >> >>>>> > a
>> >> >>>>> > > > certain
>> >> >>>>> > > > >> > >> feature
>> >> >>>>> > > > >> > >> >> is
>> >> >>>>> > > > >> > >> >> >>> useful enough for a wide-enough
>> >> audience
>> >> >> to
>> >> >>>> warrant
>> >> >>>>> > a wire
>> >> >>>>> > > > >> format
>> >> >>>>> > > > >> > >> >> change
>> >> >>>>> > > > >> > >> >> >>> and code complexity additions.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
>> >> "mere"
>> >> >>>> wire format
>> >> >>>>> > > changes
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > im
>> >> >>>>> > > > >> > >> not
>> >> >>>>> > > > >> > >> >> >>> saying it could have been completely
>> >> done
>> >> >>>> with
>> >> >>>>> > headers,
>> >> >>>>> > > but
>> >> >>>>> > > > >> > >> >> exactly-once
>> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen
>> >> >>>> Shapira <
>> >> >>>>> > > > >> gwen@confluent.io
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > >> >> wrote:
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
>> >> radai <
>> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> wrote:
>> >> >>>>> > > > >> > >> >> >>> > > "For use cases within an
>> >> organization,
>> >> >>>> one could
>> >> >>>>> > > always
>> >> >>>>> > > > use
>> >> >>>>> > > > >> > >> other
>> >> >>>>> > > > >> > >> >> >>> > > approaches such as company-wise
>> >> >>>> containers"
>> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin has
>> >> >> traditionally
>> >> >>>> done
>> >> >>>>> > but there
>> >> >>>>> > > > are
>> >> >>>>> > > > >> > now
>> >> >>>>> > > > >> > >> >> cases
>> >> >>>>> > > > >> > >> >> >>> > (read
>> >> >>>>> > > > >> > >> >> >>> > > - topics) where this is not
>> >> >> acceptable.
>> >> >>>> this
>> >> >>>>> > makes
>> >> >>>>> > > > headers
>> >> >>>>> > > > >> > >> useful
>> >> >>>>> > > > >> > >> >> even
>> >> >>>>> > > > >> > >> >> >>> > > within single orgs for cases
>> where
>> >> >>>>> > > > one-container-fits-all
>> >> >>>>> > > > >> > cannot
>> >> >>>>> > > > >> > >> >> >>> apply.
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > > as for the particular use cases
>> >> >> listed,
>> >> >>>> i dont
>> >> >>>>> > want
>> >> >>>>> > > > this to
>> >> >>>>> > > > >> > >> devolve
>> >> >>>>> > > > >> > >> >> >>> to a
>> >> >>>>> > > > >> > >> >> >>> > > discussion of particular use
>> cases
>> >> - i
>> >> >>>> think its
>> >> >>>>> > > enough
>> >> >>>>> > > > >> that
>> >> >>>>> > > > >> > >> some
>> >> >>>>> > > > >> > >> >> of
>> >> >>>>> > > > >> > >> >> >>> them
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I think a main point of contention
>> is
>> >> >>>> that: We
>> >> >>>>> > > identified
>> >> >>>>> > > > few
>> >> >>>>> > > > >> > >> >> >>> > use-cases where headers are useful,
>> >> do
>> >> >> we
>> >> >>>> want
>> >> >>>>> > Kafka to
>> >> >>>>> > > > be a
>> >> >>>>> > > > >> > >> system
>> >> >>>>> > > > >> > >> >> >>> > that supports those use-cases?
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
>> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely useful
>> >> record-level
>> >> >>>> lineage
>> >> >>>>> > is
>> >> >>>>> > > though
>> >> >>>>> > > > >> > since
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> >>> > overhead could
>> >> >>>>> > > > >> > >> >> >>> > be significant."
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports record level
>> >> >>>> lineage. I
>> >> >>>>> > don't
>> >> >>>>> > > think
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > was
>> >> >>>>> > > > >> > >> >> >>> > developed for lols, I think it is
>> >> safe
>> >> >> to
>> >> >>>> assume
>> >> >>>>> > that
>> >> >>>>> > > the
>> >> >>>>> > > > NSA
>> >> >>>>> > > > >> > >> needed
>> >> >>>>> > > > >> > >> >> >>> > that functionality. We also know
>> that
>> >> >>>> certain
>> >> >>>>> > financial
>> >> >>>>> > > > >> > institutes
>> >> >>>>> > > > >> > >> >> >>> > need to track tampering with
>> records
>> >> at
>> >> >> a
>> >> >>>> record
>> >> >>>>> > level
>> >> >>>>> > > and
>> >> >>>>> > > > >> > there
>> >> >>>>> > > > >> > >> are
>> >> >>>>> > > > >> > >> >> >>> > federal regulations that absolutely
>> >> >>>> require
>> >> >>>>> > this. They
>> >> >>>>> > > > also
>> >> >>>>> > > > >> > need
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> >>> > prove that routing apps that
>> >> "touches"
>> >> >> the
>> >> >>>>> > messages and
>> >> >>>>> > > > >> either
>> >> >>>>> > > > >> > >> reads
>> >> >>>>> > > > >> > >> >> >>> > or updates headers couldn't have
>> >> >> possibly
>> >> >>>>> > modified the
>> >> >>>>> > > > >> payload
>> >> >>>>> > > > >> > >> >> itself.
>> >> >>>>> > > > >> > >> >> >>> > They use record level encryption to
>> >> do
>> >> >>>> that -
>> >> >>>>> > apps can
>> >> >>>>> > > > read
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
>> can't
>> >> >>>> touch the
>> >> >>>>> > payload.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > We can totally say "those are
>> corner
>> >> >>>> cases and
>> >> >>>>> > not worth
>> >> >>>>> > > > >> adding
>> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for", they should
>> >> use a
>> >> >>>> different
>> >> >>>>> > > pubsub
>> >> >>>>> > > > >> > message
>> >> >>>>> > > > >> > >> for
>> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the other 1000
>> >> that
>> >> >>>> cater
>> >> >>>>> > > > specifically
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> > financial industry).
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > But this gets us into a catch 22:
>> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific use-case,
>> >> >>>> someone can
>> >> >>>>> > always
>> >> >>>>> > > say
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > isn't
>> >> >>>>> > > > >> > >> >> >>> > interesting enough for Kafka. If we
>> >> >>>> discuss more
>> >> >>>>> > general
>> >> >>>>> > > > >> > trends,
>> >> >>>>> > > > >> > >> >> >>> > others can say "well, we are not
>> sure
>> >> >> any
>> >> >>>> of them
>> >> >>>>> > really
>> >> >>>>> > > > >> needs
>> >> >>>>> > > > >> > >> >> headers
>> >> >>>>> > > > >> > >> >> >>> > specifically. This is just hand
>> >> waving
>> >> >>>> and not
>> >> >>>>> > > > interesting.".
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I think discussing use-cases in
>> >> >> specifics
>> >> >>>> is super
>> >> >>>>> > > > important
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> >> decide
>> >> >>>>> > > > >> > >> >> >>> > implementation details for headers
>> >> (my
>> >> >>>> use-cases
>> >> >>>>> > lean
>> >> >>>>> > > > toward
>> >> >>>>> > > > >> > >> >> numerical
>> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and object
>> >> values,
>> >> >>>> others
>> >> >>>>> > differ),
>> >> >>>>> > > > but I
>> >> >>>>> > > > >> > >> think
>> >> >>>>> > > > >> > >> >> we
>> >> >>>>> > > > >> > >> >> >>> > need to answer the general "Are we
>> >> going
>> >> >>>> to have
>> >> >>>>> > > headers"
>> >> >>>>> > > > >> > question
>> >> >>>>> > > > >> > >> >> >>> > first.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from the other
>> >> >>>> committers in the
>> >> >>>>> > > > discussion:
>> >> >>>>> > > > >> > >> >> >>> > What would it take to convince you
>> >> that
>> >> >>>> headers
>> >> >>>>> > in Kafka
>> >> >>>>> > > > are
>> >> >>>>> > > > >> a
>> >> >>>>> > > > >> > >> good
>> >> >>>>> > > > >> > >> >> >>> > idea in general, so we can move
>> ahead
>> >> >> and
>> >> >>>> try to
>> >> >>>>> > agree
>> >> >>>>> > > on
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> >> details?
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I feel like we keep moving the goal
>> >> >> posts
>> >> >>>> and
>> >> >>>>> > this is
>> >> >>>>> > > > truly
>> >> >>>>> > > > >> > >> >> exhausting.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly support
>> >> adding
>> >> >>>> headers
>> >> >>>>> > to Kafka
>> >> >>>>> > > > >> > (+0.5?).
>> >> >>>>> > > > >> > >> >> >>> > The community can continue to find
>> >> >>>> workarounds to
>> >> >>>>> > the
>> >> >>>>> > > > issue
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> there
>> >> >>>>> > > > >> > >> >> >>> > are some benefits to keeping the
>> >> message
>> >> >>>> format
>> >> >>>>> > and
>> >> >>>>> > > > clients
>> >> >>>>> > > > >> > >> simpler.
>> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness of headers
>> >> to
>> >> >>>> many
>> >> >>>>> > use-cases
>> >> >>>>> > > and
>> >> >>>>> > > > if
>> >> >>>>> > > > >> we
>> >> >>>>> > > > >> > >> can
>> >> >>>>> > > > >> > >> >> >>> > find a good and generally useful
>> way
>> >> to
>> >> >>>> add it to
>> >> >>>>> > Kafka,
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > will
>> >> >>>>> > > > >> > >> make
>> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
>> worthy
>> >> >>>> goal in my
>> >> >>>>> > eyes.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
>> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are use cases
>> >> for
>> >> >>>> polyglot
>> >> >>>>> > topics.
>> >> >>>>> > > > >> > >> especially if
>> >> >>>>> > > > >> > >> >> >>> kafka
>> >> >>>>> > > > >> > >> >> >>> > > is being used to "trunk"
>> something
>> >> >> else.
>> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics would make it
>> >> >> harder
>> >> >>>> to write
>> >> >>>>> > > > portable
>> >> >>>>> > > > >> > >> consumer
>> >> >>>>> > > > >> > >> >> >>> code.
>> >> >>>>> > > > >> > >> >> >>> > > partition remapping would mess
>> with
>> >> >>>> locality of
>> >> >>>>> > > > consumption
>> >> >>>>> > > > >> > >> >> >>> guarantees.
>> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
>> >> >>>> lineage/metadata is
>> >> >>>>> > > > >> > >> billing/chargeback.
>> >> >>>>> > > > >> > >> >> for
>> >> >>>>> > > > >> > >> >> >>> > that
>> >> >>>>> > > > >> > >> >> >>> > > use case it is not enough to
>> simply
>> >> >>>> record the
>> >> >>>>> > point
>> >> >>>>> > > of
>> >> >>>>> > > > >> > origin,
>> >> >>>>> > > > >> > >> but
>> >> >>>>> > > > >> > >> >> >>> every
>> >> >>>>> > > > >> > >> >> >>> > > replication stop (think mirror
>> >> maker)
>> >> >>>> must also
>> >> >>>>> > add a
>> >> >>>>> > > > >> record
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > >> >> form a
>> >> >>>>> > > > >> > >> >> >>> > > "transit log".
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > > as for stream processing on top
>> of
>> >> >>>> kafka - i
>> >> >>>>> > know
>> >> >>>>> > > samza
>> >> >>>>> > > > >> has a
>> >> >>>>> > > > >> > >> >> metadata
>> >> >>>>> > > > >> > >> >> >>> > map
>> >> >>>>> > > > >> > >> >> >>> > > which they carry around in
>> >> addition to
>> >> >>>> user
>> >> >>>>> > values.
>> >> >>>>> > > > headers
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> >>> > perfect
>> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM,
>> >> Jun
>> >> >>>> Rao <
>> >> >>>>> > > > jun@confluent.io
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> In order to answer the first two
>> >> >>>> questions, it
>> >> >>>>> > would
>> >> >>>>> > > be
>> >> >>>>> > > > >> > helpful
>> >> >>>>> > > > >> > >> >> if we
>> >> >>>>> > > > >> > >> >> >>> > could
>> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use cases
>> >> for
>> >> >>>> headers
>> >> >>>>> > in the
>> >> >>>>> > > > space
>> >> >>>>> > > > >> > for
>> >> >>>>> > > > >> > >> >> >>> > third-party
>> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use cases within an
>> >> >>>> organization,
>> >> >>>>> > one
>> >> >>>>> > > > could
>> >> >>>>> > > > >> > always
>> >> >>>>> > > > >> > >> >> use
>> >> >>>>> > > > >> > >> >> >>> > other
>> >> >>>>> > > > >> > >> >> >>> > >> approaches such as company-wise
>> >> >>>> containers to
>> >> >>>>> > get
>> >> >>>>> > > > around
>> >> >>>>> > > > >> w/o
>> >> >>>>> > > > >> > >> >> >>> headers. I
>> >> >>>>> > > > >> > >> >> >>> > >> went through the use cases in
>> the
>> >> KIP
>> >> >>>> and in
>> >> >>>>> > Radai's
>> >> >>>>> > > > wiki
>> >> >>>>> > > > >> (
>> >> >>>>> > > > >> > >> >> >>> > >> https://cwiki.apache.org/confl
>> >> >>>>> > uence/display/KAFKA/A+
>> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>> >> >>>>> > > > >> > >> >> >>> > >> ).
>> >> >>>>> > > > >> > >> >> >>> > >> The following are the ones that
>> >> that
>> >> >> I
>> >> >>>>> > understand and
>> >> >>>>> > > > >> could
>> >> >>>>> > > > >> > be
>> >> >>>>> > > > >> > >> in
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> third-party use case category.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
>> >> >>>>> > > > >> > >> >> >>> > >> It seems that in general,
>> >> >> content-type
>> >> >>>> should
>> >> >>>>> > be set
>> >> >>>>> > > at
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> topic
>> >> >>>>> > > > >> > >> >> >>> level.
>> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages with
>> >> >>>> different
>> >> >>>>> > content
>> >> >>>>> > > > types
>> >> >>>>> > > > >> > >> should be
>> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
>> >> >>>>> > > > >> > >> >> >>> > >> Since the value is mostly
>> useless
>> >> >>>> without
>> >> >>>>> > schema id,
>> >> >>>>> > > it
>> >> >>>>> > > > >> > seems
>> >> >>>>> > > > >> > >> that
>> >> >>>>> > > > >> > >> >> >>> > storing
>> >> >>>>> > > > >> > >> >> >>> > >> the schema id together with
>> >> >> serialized
>> >> >>>> bytes
>> >> >>>>> > in the
>> >> >>>>> > > > value
>> >> >>>>> > > > >> is
>> >> >>>>> > > > >> > >> >> better?
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> C. per message encryption
>> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this approach is
>> >> that
>> >> >>>> this
>> >> >>>>> > > > significantly
>> >> >>>>> > > > >> > reduce
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
>> >> which
>> >> >>>> happens on
>> >> >>>>> > a set
>> >> >>>>> > > of
>> >> >>>>> > > > >> > >> serialized
>> >> >>>>> > > > >> > >> >> >>> > >> messages. An alternative is to
>> >> enable
>> >> >>>> SSL for
>> >> >>>>> > wire
>> >> >>>>> > > > >> > encryption
>> >> >>>>> > > > >> > >> and
>> >> >>>>> > > > >> > >> >> >>> rely
>> >> >>>>> > > > >> > >> >> >>> > on
>> >> >>>>> > > > >> > >> >> >>> > >> the storage system (e.g. LUKS)
>> >> for at
>> >> >>>> rest
>> >> >>>>> > > encryption.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
>> across
>> >> >>>> Kafka
>> >> >>>>> > clusters
>> >> >>>>> > > > >> > >> >> >>> > >> This is actually interesting.
>> >> Today,
>> >> >>>> to avoid
>> >> >>>>> > > > introducing
>> >> >>>>> > > > >> > >> cycles
>> >> >>>>> > > > >> > >> >> when
>> >> >>>>> > > > >> > >> >> >>> > doing
>> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data centers,
>> one
>> >> >>>> would
>> >> >>>>> > either have
>> >> >>>>> > > to
>> >> >>>>> > > > >> set
>> >> >>>>> > > > >> > up
>> >> >>>>> > > > >> > >> two
>> >> >>>>> > > > >> > >> >> >>> Kafka
>> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and an
>> >> aggregate)
>> >> >>>> per data
>> >> >>>>> > center
>> >> >>>>> > > or
>> >> >>>>> > > > >> > rename
>> >> >>>>> > > > >> > >> >> topics.
>> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal. With headers,
>> >> the
>> >> >>>> producer
>> >> >>>>> > could
>> >> >>>>> > > tag
>> >> >>>>> > > > >> each
>> >> >>>>> > > > >> > >> >> message
>> >> >>>>> > > > >> > >> >> >>> with
>> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster ID in the
>> >> >> header.
>> >> >>>>> > MirrorMaker
>> >> >>>>> > > > could
>> >> >>>>> > > > >> > then
>> >> >>>>> > > > >> > >> >> avoid
>> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages to a cluster
>> if
>> >> >>>> they are
>> >> >>>>> > tagged
>> >> >>>>> > > with
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> same
>> >> >>>>> > > > >> > >> >> >>> cluster
>> >> >>>>> > > > >> > >> >> >>> > >> id.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> However, an alternative approach
>> >> is
>> >> >> to
>> >> >>>>> > introduce sth
>> >> >>>>> > > > like
>> >> >>>>> > > > >> > >> >> >>> hierarchical
>> >> >>>>> > > > >> > >> >> >>> > >> topic and store messages from
>> >> >> different
>> >> >>>>> > clusters in
>> >> >>>>> > > > >> > different
>> >> >>>>> > > > >> > >> >> >>> partitions
>> >> >>>>> > > > >> > >> >> >>> > >> under the same topic. This
>> >> approach
>> >> >>>> avoids
>> >> >>>>> > filtering
>> >> >>>>> > > > out
>> >> >>>>> > > > >> > >> unneeded
>> >> >>>>> > > > >> > >> >> >>> data
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> makes offset preserving easier
>> to
>> >> >>>> support. It
>> >> >>>>> > may
>> >> >>>>> > > make
>> >> >>>>> > > > >> > >> compaction
>> >> >>>>> > > > >> > >> >> >>> > trickier
>> >> >>>>> > > > >> > >> >> >>> > >> though since the same key may
>> >> show up
>> >> >>>> in
>> >> >>>>> > different
>> >> >>>>> > > > >> > partitions.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> E. record-level lineage
>> >> >>>>> > > > >> > >> >> >>> > >> For example, a source connector
>> >> could
>> >> >>>> store in
>> >> >>>>> > the
>> >> >>>>> > > > message
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> metadata
>> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
>> record.
>> >> >>>> Similarly,
>> >> >>>>> > if a
>> >> >>>>> > > > stream
>> >> >>>>> > > > >> job
>> >> >>>>> > > > >> > >> >> >>> transforms
>> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A to topic
>> B,
>> >> the
>> >> >>>> library
>> >> >>>>> > could
>> >> >>>>> > > > >> include
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> source
>> >> >>>>> > > > >> > >> >> >>> > >> message offset in each of the
>> >> >>>> transformed
>> >> >>>>> > message in
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> header.
>> >> >>>>> > > > >> > >> >> Not
>> >> >>>>> > > > >> > >> >> >>> > sure
>> >> >>>
>> >> >>>>> > > > >> > >> >> >>> > >> how widely useful record-level
>> >> >> lineage
>> >> >>>> is
>> >> >>>>> > though
>> >> >>>>> > > since
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> >> overhead
>> >> >>>>> > > > >> > >> >> >>> > could
>> >> >>>>> > > > >> > >> >> >>> > >> be significant.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>> >> >>>>> > > > >> > >> >> >>> > >> We could put things like
>> >> >>>> clientId/host/user in
>> >> >>>>> > the
>> >> >>>>> > > > header
>> >> >>>>> > > > >> in
>> >> >>>>> > > > >> > >> each
>> >> >>>>> > > > >> > >> >> >>> > message
>> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These metadata are
>> >> >>>> really at the
>> >> >>>>> > > producer
>> >> >>>>> > > > >> > level
>> >> >>>>> > > > >> > >> >> though.
>> >> >>>>> > > > >> > >> >> >>> > So, a
>> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is to only
>> >> >> include a
>> >> >>>>> > "producerId"
>> >> >>>>> > > > per
>> >> >>>>> > > > >> > >> message
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> > send
>> >> >>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
>> mapping
>> >> >>>>> > independently.
>> >> >>>>> > > > KIP-98
>> >> >>>>> > > > >> is
>> >> >>>>> > > > >> > >> >> actually
>> >> >>>>> > > > >> > >> >> >>> > >> proposing including such a
>> >> producerId
>> >> >>>> natively
>> >> >>>>> > in the
>> >> >>>>> > > > >> > message.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that I
>> am
>> >> >> fully
>> >> >>>>> > convinced of
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > strong
>> >> >>>>> > > > >> > >> >> >>> > third-party
>> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
>> Perhaps
>> >> we
>> >> >>>> could
>> >> >>>>> > discuss a
>> >> >>>>> > > > bit
>> >> >>>>> > > > >> > more
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> make
>> >> >>>>> > > > >> > >> >> >>> > one
>> >> >>>>> > > > >> > >> >> >>> > >> or two really convincing use
>> >> cases.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal question is
>> >> >>>> whether header
>> >> >>>>> > should
>> >> >>>>> > > > be
>> >> >>>>> > > > >> > >> exposed
>> >> >>>>> > > > >> > >> >> in
>> >> >>>>> > > > >> > >> >> >>> > stream
>> >> >>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
>> >> stream,
>> >> >>>> Samza,
>> >> >>>>> > and
>> >> >>>>> > > Spark
>> >> >>>>> > > > >> > >> streaming.
>> >> >>>>> > > > >> > >> >> >>> > >> Currently, those systems just
>> deal
>> >> >> with
>> >> >>>>> > key/value
>> >> >>>>> > > > pairs.
>> >> >>>>> > > > >> > >> Should we
>> >> >>>>> > > > >> > >> >> >>> > expose a
>> >> >>>>> > > > >> > >> >> >>> > >> third thing header there too or
>> >> >>>> somehow map
>> >> >>>>> > header to
>> >> >>>>> > > > key
>> >> >>>>> > > > >> or
>> >> >>>>> > > > >> > >> >> value?
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> Jun
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM,
>> >> >>>> Michael
>> >> >>>>> > Pearce <
>> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>> >> >>>>> > > > >> > >> >> >>> > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that after a period
>> >> of a
>> >> >>>> week,
>> >> >>>>> > that there
>> >> >>>>> > > > is
>> >> >>>>> > > > >> no
>> >> >>>>> > > > >> > >> >> concerns
>> >> >>>>> > > > >> > >> >> >>> now
>> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and now
>> we
>> >> >> have
>> >> >>>>> > agreement that
>> >> >>>>> > > > >> > headers
>> >> >>>>> > > > >> > >> are
>> >> >>>>> > > > >> > >> >> >>> useful
>> >> >>>>> > > > >> > >> >> >>> > >> and
>> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
>> put
>> >> to
>> >> >> a
>> >> >>>> KIP
>> >> >>>>> > vote, this
>> >> >>>>> > > > >> > wouldn’t
>> >> >>>>> > > > >> > >> be
>> >> >>>>> > > > >> > >> >> a
>> >> >>>>> > > > >> > >> >> >>> > reason
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > reject.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > @
>> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
>> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose of getting
>> >> this
>> >> >>>> KIP
>> >> >>>>> > moving past
>> >> >>>>> > > > >> this,
>> >> >>>>> > > > >> > we
>> >> >>>>> > > > >> > >> can
>> >> >>>>> > > > >> > >> >> >>> state
>> >> >>>>> > > > >> > >> >> >>> > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
>> that
>> >> >> can
>> >> >>>> will be
>> >> >>>>> > > > naturally
>> >> >>>>> > > > >> > >> >> interpreted
>> >> >>>>> > > > >> > >> >> >>> as
>> >> >>>>> > > > >> > >> >> >>> > an
>> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is later
>> >> >>>> wanted you can
>> >> >>>>> > > easily
>> >> >>>>> > > > >> split
>> >> >>>>> > > > >> > >> this
>> >> >>>>> > > > >> > >> >> >>> into
>> >> >>>>> > > > >> > >> >> >>> > two
>> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the wire
>> >> >> protocol
>> >> >>>>> > > implementation
>> >> >>>>> > > > >> this
>> >> >>>>> > > > >> > >> makes
>> >> >>>>> > > > >> > >> >> no
>> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t believe. Is
>> >> this
>> >> >>>>> > reasonable to
>> >> >>>>> > > > all?
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4 therefor
>> >> happy
>> >> >>>> we keep
>> >> >>>>> > with 32
>> >> >>>>> > > > >> bits.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
>> >> >>>>> > ignacio.solis@gmail.com on
>> >> >>>>> > > > >> behalf
>> >> >>>>> > > > >> > of
>> >> >>>>> > > > >> > >> >> >>> Ignacio
>> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
>> ignacio.solis@gmail.com
>> >> on
>> >> >>>> behalf of
>> >> >>>>> > > > >> > isolis@igso.net
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> >> >>> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value as
>> >> byte[]
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
>> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
>> maybe
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
>> >> system
>> >> >>>> should
>> >> >>>>> > take a
>> >> >>>>> > > > single
>> >> >>>>> > > > >> > >> int. I
>> >> >>>>> > > > >> > >> >> >>> think
>> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
>> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if you want
>> to
>> >> >>>> interpret
>> >> >>>>> > this as
>> >> >>>>> > > to
>> >> >>>>> > > > >> 16bit
>> >> >>>>> > > > >> > >> >> numbers
>> >> >>>>> > > > >> > >> >> >>> in
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > layer
>> >> >>>>> > > > >> > >> >> >>> > >> > above go right ahead. If
>> >> >>>> somebody wants
>> >> >>>>> > to
>> >> >>>>> > > argue
>> >> >>>>> > > > >> for
>> >> >>>>> > > > >> > 16
>> >> >>>>> > > > >> > >> >> bits
>> >> >>>>> > > > >> > >> >> >>> or
>> >> >>>>> > > > >> > >> >> >>> > 64
>> >> >>>>> > > > >> > >> >> >>> > >> > bits of
>> >> >>>>> > > > >> > >> >> >>> > >> > header key space I would
>> >> >> listen.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
>> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the key space
>> into
>> >> >>>> sub_key_1 and
>> >> >>>>> > > > sub_key_2
>> >> >>>>> > > > >> > >> makes no
>> >> >>>>> > > > >> > >> >> >>> > sense to
>> >> >>>>> > > > >> > >> >> >>> > >> > me at
>> >> >>>>> > > > >> > >> >> >>> > >> > this layer. Are we going
>> to
>> >> >>>> start
>> >> >>>>> > providing
>> >> >>>>> > > > APIs to
>> >> >>>>> > > > >> > get
>> >> >>>>> > > > >> > >> all
>> >> >>>>> > > > >> > >> >> >>> the
>> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
>> >> >>>> sub_key_2s? If
>> >> >>>>> > there is
>> >> >>>>> > > > no
>> >> >>>>> > > > >> > >> >> >>> distinguishing
>> >> >>>>> > > > >> > >> >> >>> > >> > functions
>> >> >>>>> > > > >> > >> >> >>> > >> > that are applied to each
>> one
>> >> >>>> then they
>> >> >>>>> > should
>> >> >>>>> > > be
>> >> >>>>> > > > a
>> >> >>>>> > > > >> > single
>> >> >>>>> > > > >> > >> >> >>> value.
>> >> >>>>> > > > >> > >> >> >>> > At
>> >> >>>>> > > > >> > >> >> >>> > >> > this
>> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're doing is
>> >> >>>> equality.
>> >> >>>>> > > > >> > >> >> >>> > >> > If the above layer wants
>> to
>> >> >>>> interpret
>> >> >>>>> > this as
>> >> >>>>> > > 2,
>> >> >>>>> > > > 3
>> >> >>>>> > > > >> or
>> >> >>>>> > > > >> > >> more
>> >> >>>>> > > > >> > >> >> >>> values
>> >> >>>>> > > > >> > >> >> >>> > >> > that's a
>> >> >>>>> > > > >> > >> >> >>> > >> > different question. I
>> >> >>>> personally think
>> >> >>>>> > it's
>> >> >>>>> > > all
>> >> >>>>> > > > one
>> >> >>>>> > > > >> > >> >> keyspace
>> >> >>>>> > > > >> > >> >> >>> > that is
>> >> >>>>> > > > >> > >> >> >>> > >> > getting assigned using
>> some
>> >> >>>> structure,
>> >> >>>>> > but if
>> >> >>>>> > > you
>> >> >>>>> > > > >> > want to
>> >> >>>>> > > > >> > >> >> >>> > sub-assign
>> >> >>>>> > > > >> > >> >> >>> > >> > parts
>> >> >>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > The same discussion
>> applies
>> >> to
>> >> >>>> strings.
>> >> >>>>> > If
>> >> >>>>> > > > somebody
>> >> >>>>> > > > >> > >> argued
>> >> >>>>> > > > >> > >> >> for
>> >> >>>>> > > > >> > >> >> >>> > >> > strings,
>> >> >>>>> > > > >> > >> >> >>> > >> > would we be arguing to
>> >> divide
>> >> >> the
>> >> >>>>> > strings with
>> >> >>>>> > > > dots
>> >> >>>>> > > > >> > ('.')
>> >> >>>>> > > > >> > >> >> as a
>> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
>> >> >>>>> > > > >> > >> >> >>> > >> > Would we want them to give
>> >> us
>> >> >> the
>> >> >>>>> > different
>> >> >>>>> > > name
>> >> >>>>> > > > >> > segments
>> >> >>>>> > > > >> > >> >> >>> > separately?
>> >> >>>>> > > > >> > >> >> >>> > >> > Would we be performing any
>> >> >>>> actions on
>> >> >>>>> > this key
>> >> >>>>> > > > other
>> >> >>>>> > > > >> > than
>> >> >>>>> > > > >> > >> >> >>> > matching?
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
>> 9:30
>> >> >> AM,
>> >> >>>> Michael
>> >> >>>>> > > Pearce <
>> >> >>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any concerns
>> on
>> >> 1
>> >> >>>> and 2
>> >> >>>>> > still?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > @all
>> >> >>>>> > > > >> > >> >> >>> > >> > > To get this moving
>> along a
>> >> >> bit
>> >> >>>> more
>> >> >>>>> > I'd also
>> >> >>>>> > > > like
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> ask
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> >>> get
>> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
>> >> >>>>> > > > >> > >> >> >>> > >> > > the below last points:
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're all
>> >> >> roughly
>> >> >>>> happy
>> >> >>>>> > with the
>> >> >>>>> > > > >> header
>> >> >>>>> > > > >> > >> value
>> >> >>>>> > > > >> > >> >> >>> > being a
>> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe consensus
>> has
>> >> >>>> been for an
>> >> >>>>> > > > namespace
>> >> >>>>> > > > >> > based
>> >> >>>>> > > > >> > >> int
>> >> >>>>> > > > >> > >> >> >>> > approach
>> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for the key.
>> Any
>> >> >>>> objections
>> >> >>>>> > if this
>> >> >>>>> > > > is
>> >> >>>>> > > > >> > what
>> >> >>>>> > > > >> > >> we
>> >> >>>>> > > > >> > >> >> go
>> >> >>>>> > > > >> > >> >> >>> > with?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
>> >> assumption
>> >> >> in
>> >> >>>> (4) is
>> >> >>>>> > > correct,
>> >> >>>>> > > > >> > >> {int,int}
>> >> >>>>> > > > >> > >> >> >>> keys.
>> >> >>>>> > > > >> > >> >> >>> > >> > > Should both int's be
>> >> int16 or
>> >> >>>> int32?
>> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them being
>> int16(2
>> >> >>>> bytes) as
>> >> >>>>> > combined
>> >> >>>>> > > > is
>> >> >>>>> > > > >> > space
>> >> >>>>> > > > >> > >> of
>> >> >>>>> > > > >> > >> >> >>> > 4bytes as
>> >> >>>>> > > > >> > >> >> >>> > >> > per
>> >> >>>>> > > > >> > >> >> >>> > >> > > original and gives
>> plenty
>> >> of
>> >> >>>>> > combinations for
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> >> >>> foreseeable,
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > keeps
>> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead small.
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any benefit in
>> >> >>>> another kip
>> >> >>>>> > call to
>> >> >>>>> > > > >> discuss
>> >> >>>>> > > > >> > >> >> these at
>> >> >>>>> > > > >> > >> >> >>> > all?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
>> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >> ______________________________
>> >> >>>>> > __________
>> >> >>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
>> >> >>>> k.burstev@yandex.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday, November
>> 18,
>> >> >> 2016
>> >> >>>>> > 7:07:07 AM
>> >> >>>>> > > > >> > >> >> >>> > >> > > To:
>> dev@kafka.apache.org
>> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re: [DISCUSS]
>> >> KIP-82
>> >> >>>> - Add
>> >> >>>>> > Record
>> >> >>>>> > > > Headers
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > For what it is worth
>> also
>> >> i
>> >> >>>> agree. As
>> >> >>>>> > a user:
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
>> >> >>>> worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers should
>> >> be a
>> >> >>>> top level
>> >> >>>>> > > option
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
>> >> "Ignacio
>> >> >>>> Solis" <
>> >> >>>>> > > > >> isolis@igso.net
>> >> >>>>> > > > >> > >:
>> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers are
>> >> >>>> worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
>> should
>> >> be
>> >> >> a
>> >> >>>> top
>> >> >>>>> > level
>> >> >>>>> > > option
>> >> >>>>> > > > >> > >> >> >>> > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14, 2016
>> at
>> >> >> 9:16
>> >> >>>> AM,
>> >> >>>>> > Michael
>> >> >>>>> > > > Pearce
>> >> >>>>> > > > >> <
>> >> >>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
>> >> details/examples
>> >> >>>> the
>> >> >>>>> > original
>> >> >>>>> > > > proposal
>> >> >>>>> > > > >> > for
>> >> >>>>> > > > >> > >> key
>> >> >>>>> > > > >> > >> >> >>> > spacing
>> >> >>>>> > > > >> > >> >> >>> > >> ,
>> >> >>>>> > > > >> > >> >> >>> > >> > not
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as per
>> >> >>>> discussion
>> >> >>>>> > namespace
>> >> >>>>> > > > >> idea.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will need to
>> update
>> >> >> the
>> >> >>>> kip,
>> >> >>>>> > when we
>> >> >>>>> > > get
>> >> >>>>> > > > >> > >> agreement
>> >> >>>>> > > > >> > >> >> >>> this
>> >> >>>>> > > > >> > >> >> >>> > is a
>> >> >>>>> > > > >> > >> >> >>> > >> > better
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach (which
>> seems
>> >> to
>> >> >>>> be the
>> >> >>>>> > case if I
>> >> >>>>> > > > have
>> >> >>>>> > > > >> > >> >> understood
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > general
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
>> >> >>>> conversation)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
>> ints,
>> >> at
>> >> >>>> very
>> >> >>>>> > early stage
>> >> >>>>> > > > we
>> >> >>>>> > > > >> did
>> >> >>>>> > > > >> > >> think
>> >> >>>>> > > > >> > >> >> >>> about
>> >> >>>>> > > > >> > >> >> >>> > >> > this. I
>> >> >>>>> > > > >> > >> >> >>> > >> > > think
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added complexity
>> >> for
>> >> >>>> the
>> >> >>>>> > saving isn't
>> >> >>>>> > > > >> worth
>> >> >>>>> > > > >> > it.
>> >> >>>>> > > > >> > >> >> I'd
>> >> >>>>> > > > >> > >> >> >>> > rather
>> >> >>>>> > > > >> > >> >> >>> > >> go
>> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
>> >> >>>> overheads and
>> >> >>>>> > size
>> >> >>>>> > > int16
>> >> >>>>> > > > >> > (2bytes)
>> >> >>>>> > > > >> > >> >> keys
>> >> >>>>> > > > >> > >> >> >>> as
>> >> >>>>> > > > >> > >> >> >>> > it
>> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
>> >> >> headers,
>> >> >>>> there
>> >> >>>>> > is as
>> >> >>>>> > > per
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > kip
>> >> >>>>> > > > >> > >> as
>> >> >>>>> > > > >> > >> >> we
>> >> >>>>> > > > >> > >> >> >>> > use an
>> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
>> >> headers
>> >> >>>> are
>> >> >>>>> > present or
>> >> >>>>> > > > not as
>> >> >>>>> > > > >> > such
>> >> >>>>> > > > >> > >> >> >>> > provides a
>> >> >>>>> > > > >> > >> >> >>> > >> > zero
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead currently
>> if
>> >> >>>> headers are
>> >> >>>>> > not
>> >> >>>>> > > used.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
>> >> mentions
>> >> >>>> would be
>> >> >>>>> > good
>> >> >>>>> > > > first
>> >> >>>>> > > > >> > if we
>> >> >>>>> > > > >> > >> >> can
>> >> >>>>> > > > >> > >> >> >>> get
>> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
>> >> >>>>> > > > >> > >> >> >>> > >> > > do
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now have general
>> >> >>>> consensus that
>> >> >>>>> > (1)
>> >> >>>>> > > > headers
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> >> >>> > worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> and
>> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want it
>> as
>> >> a
>> >> >>>> top level
>> >> >>>>> > entity.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
>> >> >> obvious i
>> >> >>>>> > believe (1)
>> >> >>>>> > > > >> headers
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> >> >>> > worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as a top level
>> >> >>>> entity.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>> ______________________________
>> >> >>>>> > __________
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> From: Roger Hoover <
>> >> >>>>> > > roger.hoover@gmail.com
>> >> >>>>> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
>> >> November
>> >> >>>> 9, 2016
>> >> >>>>> > 9:10:47
>> >> >>>>> > > > PM
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
>> >> dev@kafka.apache.org
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
>> [DISCUSS]
>> >> >>>> KIP-82 - Add
>> >> >>>>> > > Record
>> >> >>>>> > > > >> > Headers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
>> >> little
>> >> >>>> in the
>> >> >>>>> > weeds but
>> >> >>>>> > > > >> thanks
>> >> >>>>> > > > >> > >> for
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> replies
>> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a prefix
>> >> and
>> >> >>>> {int,
>> >> >>>>> > int} can
>> >> >>>>> > > be
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> same.
>> >> >>>>> > > > >> > >> >> It
>> >> >>>>> > > > >> > >> >> >>> > doesn't
>> >> >>>>> > > > >> > >> >> >>> > >> > look
>> >> >>>>> > > > >> > >> >> >>> > >> > > like
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's what the KIP
>> is
>> >> >>>> saying the
>> >> >>>>> > "Open"
>> >> >>>>> > > > >> > section.
>> >> >>>>> > > > >> > >> The
>> >> >>>>> > > > >> > >> >> >>> > example
>> >> >>>>> > > > >> > >> >> >>> > >> > shows
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
>> >> 210002
>> >> >>>> for App
>> >> >>>>> > Dynamics
>> >> >>>>> > > > >> > implying
>> >> >>>>> > > > >> > >> >> that
>> >> >>>>> > > > >> > >> >> >>> the
>> >> >>>>> > > > >> > >> >> >>> > New
>> >> >>>>> > > > >> > >> >> >>> > >> > Relic
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> organization will
>> have
>> >> >>>> only a
>> >> >>>>> > single
>> >> >>>>> > > > header id
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > >> work
>> >> >>>>> > > > >> > >> >> >>> > with. Or
>> >> >>>>> > > > >> > >> >> >>> > >> > is
>> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The main
>> >> point
>> >> >>>> of a
>> >> >>>>> > namespace
>> >> >>>>> > > or
>> >> >>>>> > > > >> > prefix
>> >> >>>>> > > > >> > >> is
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> >>> > reduce
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead of config
>> >> >> mapping
>> >> >>>> or
>> >> >>>>> > > registration
>> >> >>>>> > > > >> > >> depending
>> >> >>>>> > > > >> > >> >> on
>> >> >>>>> > > > >> > >> >> >>> how
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> namespaces/prefixes
>> >> are
>> >> >>>> managed.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would love to hear
>> >> more
>> >> >>>> feedback
>> >> >>>>> > on the
>> >> >>>>> > > > >> > >> higher-level
>> >> >>>>> > > > >> > >> >> >>> > questions
>> >> >>>>> > > > >> > >> >> >>> > >> > > though...
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9, 2016
>> at
>> >> >>>> 11:38 AM,
>> >> >>>>> > radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think this
>> >> discussion
>> >> >>>> is
>> >> >>>>> > getting a
>> >> >>>>> > > bit
>> >> >>>>> > > > >> into
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> weeds on
>> >> >>>>> > > > >> > >> >> >>> > >> > technical
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > implementation
>> >> details.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to step
>> >> back a
>> >> >>>> minute
>> >> >>>>> > and try
>> >> >>>>> > > > and
>> >> >>>>> > > > >> > >> establish
>> >> >>>>> > > > >> > >> >> >>> > where we
>> >> >>>>> > > > >> > >> >> >>> > >> > are in
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
>> nacho's
>> >> >> last
>> >> >>>>> > paragraph)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all in
>> >> >>>> agreement that
>> >> >>>>> > headers
>> >> >>>>> > > > are
>> >> >>>>> > > > >> a
>> >> >>>>> > > > >> > >> >> >>> worthwhile
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > useful
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > addition to have?
>> >> this
>> >> >>>> was
>> >> >>>>> > contested
>> >> >>>>> > > > early
>> >> >>>>> > > > >> on
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all in
>> >> >>>> agreement on
>> >> >>>>> > headers
>> >> >>>>> > > as
>> >> >>>>> > > > top
>> >> >>>>> > > > >> > >> level
>> >> >>>>> > > > >> > >> >> >>> entity
>> >> >>>>> > > > >> > >> >> >>> > vs
>> >> >>>>> > > > >> > >> >> >>> > >> > headers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > squirreled-away in
>> >> V?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if there are still
>> >> >>>> concerns
>> >> >>>>> > around
>> >> >>>>> > > these
>> >> >>>>> > > > #2
>> >> >>>>> > > > >> > >> points
>> >> >>>>> > > > >> > >> >> >>> (#jay?
>> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and now back to
>> our
>> >> >>>> normal
>> >> >>>>> > programming
>> >> >>>>> > > > ...)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints are nice.
>> >> >> having
>> >> >>>> said
>> >> >>>>> > that, its
>> >> >>>>> > > > >> adding
>> >> >>>>> > > > >> > >> >> >>> complexity
>> >> >>>>> > > > >> > >> >> >>> > >> (see
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>> https://github.com/addthis/
>> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>> main/java/com/clearspring/
>> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
>> >> result)
>> >> >>>> and would
>> >> >>>>> > require
>> >> >>>>> > > > >> anyone
>> >> >>>>> > > > >> > >> >> writing
>> >> >>>>> > > > >> > >> >> >>> > other
>> >> >>>>> > > > >> > >> >> >>> > >> > clients
>> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go? Bash?
>> >> ;-) )
>> >> >>>> to
>> >> >>>>> > > get/implement
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> same,
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> for
>> >> >>>>> > > > >> > >> >> >>> > >> > relatively
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little gain (int
>> vs
>> >> >>>> string is
>> >> >>>>> > order of
>> >> >>>>> > > > >> > magnitude,
>> >> >>>>> > > > >> > >> >> this
>> >> >>>>> > > > >> > >> >> >>> > isnt).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int namespacing vs
>> >> >> {int,
>> >> >>>> int}
>> >> >>>>> > > namespacing
>> >> >>>>> > > > >> are
>> >> >>>>> > > > >> > >> >> basically
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > same
>> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre just
>> >> namespacing
>> >> >>>> an int64
>> >> >>>>> > and
>> >> >>>>> > > > giving
>> >> >>>>> > > > >> > people
>> >> >>>>> > > > >> > >> >> while
>> >> >>>>> > > > >> > >> >> >>> > 2^32
>> >> >>>>> > > > >> > >> >> >>> > >> > ranges
>> >> >>>>> > > > >> > >> >> >>> > >> > > at a
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time. the part i
>> >> like
>> >> >>>> about this
>> >> >>>>> > is
>> >> >>>>> > > > letting
>> >> >>>>> > > > >> > >> people
>> >> >>>>> > > > >> > >> >> >>> have a
>> >> >>>>> > > > >> > >> >> >>> > >> large
>> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers with one
>> >> >>>> registration so
>> >> >>>>> > they
>> >> >>>>> > > > dont
>> >> >>>>> > > > >> > have
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> come
>> >> >>>>> > > > >> > >> >> >>> > back
>> >> >>>>> > > > >> > >> >> >>> > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > every
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
>> plugin/header
>> >> >>>> they want to
>> >> >>>>> > > > "reserve".
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
>> 2016
>> >> at
>> >> >>>> 11:01 AM,
>> >> >>>>> > Roger
>> >> >>>>> > > > >> Hoover
>> >> >>>>> > > > >> > <
>> >> >>>>> > > > >> > >> >> >>> > >> > > roger.hoover@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
>> the
>> >> >>>> debate has
>> >> >>>>> > been
>> >> >>>>> > > about
>> >> >>>>> > > > >> > >> overhead +
>> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if we
>> >> have
>> >> >>>>> > considered a
>> >> >>>>> > > > varint
>> >> >>>>> > > > >> > >> encoding
>> >> >>>>> > > > >> > >> >> (
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>> https://developers.google.com/
>> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
>> >> >>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the header
>> length
>> >> >>>> field (int32
>> >> >>>>> > in the
>> >> >>>>> > > > >> > proposal)
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> for
>> >> >>>>> > > > >> > >> >> >>> > >> > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
>> headers,
>> >> >> the
>> >> >>>>> > overhead would
>> >> >>>>> > > > be a
>> >> >>>>> > > > >> > >> single
>> >> >>>>> > > > >> > >> >> >>> byte
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > for each
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128 would
>> >> also
>> >> >>>> need only a
>> >> >>>>> > > single
>> >> >>>>> > > > >> byte?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov 9,
>> >> 2016
>> >> >> at
>> >> >>>> 6:43 AM,
>> >> >>>>> > > radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus - and
>> >> very
>> >> >>>> dangerous
>> >> >>>>> > (youre
>> >> >>>>> > > > >> > >> essentially
>> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > executing
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary code
>> >> off
>> >> >>>> the
>> >> >>>>> > internet on
>> >> >>>>> > > > your
>> >> >>>>> > > > >> > >> servers
>> >> >>>>> > > > >> > >> >> ...
>> >> >>>>> > > > >> > >> >> >>> > bad
>> >> >>>>> > > > >> > >> >> >>> > >> > idea
>> >> >>>>> > > > >> > >> >> >>> > >> > > without
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox, even
>> >> with)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
>> being
>> >> a
>> >> >>>> purely
>> >> >>>>> > > > administrative
>> >> >>>>> > > > >> > task
>> >> >>>>> > > > >> > >> - i
>> >> >>>>> > > > >> > >> >> >>> > >> disagree.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
>> would,
>> >> >>>> really,
>> >> >>>>> > because
>> >> >>>>> > > > then my
>> >> >>>>> > > > >> > >> earlier
>> >> >>>>> > > > >> > >> >> >>> > point on
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > complexity
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the remapping
>> >> >>>> process would
>> >> >>>>> > be
>> >> >>>>> > > > invalid,
>> >> >>>>> > > > >> > but
>> >> >>>>> > > > >> > >> at
>> >> >>>>> > > > >> > >> >> >>> > linkedin,
>> >> >>>>> > > > >> > >> >> >>> > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > example,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team im
>> in)
>> >> >> run
>> >> >>>> kafka
>> >> >>>>> > as a
>> >> >>>>> > > > service.
>> >> >>>>> > > > >> > we
>> >> >>>>> > > > >> > >> dont
>> >> >>>>> > > > >> > >> >> >>> > really
>> >> >>>>> > > > >> > >> >> >>> > >> > know
>> >> >>>>> > > > >> > >> >> >>> > >> > > what our
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
>> >> >>>> applications
>> >> >>>>> > that use
>> >> >>>>> > > > kafka)
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> up
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> >>> at
>> >> >>>>> > > > >> > >> >> >>> > any
>> >> >>>>> > > > >> > >> >> >>> > >> > given
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
>> possible
>> >> >>>> (given the
>> >> >>>>> > > > existance of
>> >> >>>>> > > > >> > >> headers
>> >> >>>>> > > > >> > >> >> >>> and a
>> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem) for
>> >> some
>> >> >>>>> > application to
>> >> >>>>> > > > >> "equip"
>> >> >>>>> > > > >> > >> their
>> >> >>>>> > > > >> > >> >> >>> > >> producers
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > consumers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
>> >> required
>> >> >>>> plugin
>> >> >>>>> > without us
>> >> >>>>> > > > >> > knowing.
>> >> >>>>> > > > >> > >> i
>> >> >>>>> > > > >> > >> >> dont
>> >> >>>>> > > > >> > >> >> >>> > mean
>> >> >>>>> > > > >> > >> >> >>> > >> > to imply
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
>> >> want to
>> >> >>>> make the
>> >> >>>>> > point
>> >> >>>>> > > > that
>> >> >>>>> > > > >> > its
>> >> >>>>> > > > >> > >> not
>> >> >>>>> > > > >> > >> >> as
>> >> >>>>> > > > >> > >> >> >>> > simple
>> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sync across a
>> >> >>>> large-enough
>> >> >>>>> > > > organization.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed, Nov 9,
>> >> 2016
>> >> >>>> at 6:17
>> >> >>>>> > AM,
>> >> >>>>> > > > Magnus
>> >> >>>>> > > > >> > >> Edenhill
>> >> >>>>> > > > >> > >> >> <
>> >> >>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
>> there
>> >> is
>> >> >> a
>> >> >>>> piece
>> >> >>>>> > missing
>> >> >>>>> > > in
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> >> Strings
>> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
>> >> >>>>> > > > >> > >> >> >>> > >> > > where
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> pro-Stringers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason that
>> by
>> >> >>>> providing
>> >> >>>>> > unique
>> >> >>>>> > > > string
>> >> >>>>> > > > >> > >> >> >>> identifiers
>> >> >>>>> > > > >> > >> >> >>> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > each
>> >> >>>>> > > > >> > >> >> >>> > >> > > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > everything
>> >> will
>> >> >>>> just
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
>> work
>> >> >> for
>> >> >>>> all
>> >> >>>>> > parts of
>> >> >>>>> > > the
>> >> >>>>> > > > >> > stream
>> >> >>>>> > > > >> > >> >> >>> pipeline.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
>> >> strings
>> >> >>>> dont mean
>> >> >>>>> > > anything
>> >> >>>>> > > > by
>> >> >>>>> > > > >> > >> >> themselves,
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > while we
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
>> >> envision
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
>> >> plugin
>> >> >>>> loader
>> >> >>>>> > that
>> >> >>>>> > > > >> downloads,
>> >> >>>>> > > > >> > >> >> compiles,
>> >> >>>>> > > > >> > >> >> >>> > links
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > runs
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon as
>> >> >> they're
>> >> >>>> seen by
>> >> >>>>> > a
>> >> >>>>> > > > >> consumer, I
>> >> >>>>> > > > >> > >> dont
>> >> >>>>> > > > >> > >> >> >>> really
>> >> >>>>> > > > >> > >> >> >>> > >> see
>> >> >>>>> > > > >> > >> >> >>> > >> > a
>> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > so dynamic
>> >> (and
>> >> >>>> fragile) in
>> >> >>>>> > > > practice.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the real
>> >> world
>> >> >>>> an
>> >> >>>>> > application
>> >> >>>>> > > > will
>> >> >>>>> > > > >> be
>> >> >>>>> > > > >> > >> >> >>> configured
>> >> >>>>> > > > >> > >> >> >>> > >> with
>> >> >>>>> > > > >> > >> >> >>> > >> > a set
>> >> >>>>> > > > >> > >> >> >>> > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
>> add
>> >> >>>> (producer)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
>> >> >> (consumer)
>> >> >>>> headers.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is an
>> >> >>>> administrative
>> >> >>>>> > task
>> >> >>>>> > > > based
>> >> >>>>> > > > >> on
>> >> >>>>> > > > >> > >> what
>> >> >>>>> > > > >> > >> >> >>> > features a
>> >> >>>>> > > > >> > >> >> >>> > >> > client
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> needs/provides
>> >> >> and
>> >> >>>> results
>> >> >>>>> > in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some sort of
>> >> >>>> configuration
>> >> >>>>> > to
>> >> >>>>> > > > enable
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> >> >>> configure
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > desired
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since this
>> >> needs
>> >> >>>> to be kept
>> >> >>>>> > > > somewhat
>> >> >>>>> > > > >> in
>> >> >>>>> > > > >> > >> sync
>> >> >>>>> > > > >> > >> >> >>> across
>> >> >>>>> > > > >> > >> >> >>> > an
>> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no point
>> in
>> >> >>>> having
>> >> >>>>> > producers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > add headers
>> no
>> >> >>>> consumers
>> >> >>>>> > will
>> >> >>>>> > > read,
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> vice
>> >> >>>>> > > > >> > >> >> >>> versa),
>> >> >>>>> > > > >> > >> >> >>> > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > added
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > of assigning
>> >> an
>> >> >> id
>> >> >>>>> > namespace
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
>> >> plugin
>> >> >> as
>> >> >>>> it is
>> >> >>>>> > being
>> >> >>>>> > > > >> > configured
>> >> >>>>> > > > >> > >> >> should
>> >> >>>>> > > > >> > >> >> >>> be
>> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > 2016-11-09
>> >> 13:06
>> >> >>>> GMT+01:00
>> >> >>>>> > > Michael
>> >> >>>>> > > > >> > Pearce <
>> >> >>>>> > > > >> > >> >> >>> > >> > > Michael.Pearce@ig.com>:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
>> >> >>>> following/catching
>> >> >>>>> > up on
>> >> >>>>> > > > what
>> >> >>>>> > > > >> > seems
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> be
>> >> >>>>> > > > >> > >> >> >>> an
>> >> >>>>> > > > >> > >> >> >>> > >> > active
>> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
>> >> sorry if
>> >> >>>> it may
>> >> >>>>> > seem
>> >> >>>>> > > > obvious
>> >> >>>>> > > > >> > but
>> >> >>>>> > > > >> > >> what
>> >> >>>>> > > > >> > >> >> >>> does
>> >> >>>>> > > > >> > >> >> >>> > MD
>> >> >>>>> > > > >> > >> >> >>> > >> > stand
>> >> >>>>> > > > >> > >> >> >>> > >> > > for?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My take on
>> >> >>>> String vs Int:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
>> state
>> >> >>>> first I am
>> >> >>>>> > pro Int
>> >> >>>>> > > > (16
>> >> >>>>> > > > >> or
>> >> >>>>> > > > >> > >> 32).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
>> though
>> >> >>>> playing
>> >> >>>>> > devils
>> >> >>>>> > > > advocate
>> >> >>>>> > > > >> > see a
>> >> >>>>> > > > >> > >> >> big
>> >> >>>>> > > > >> > >> >> >>> plus
>> >> >>>>> > > > >> > >> >> >>> > >> > with the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > argument
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
>> keys,
>> >> >>>> this is
>> >> >>>>> > around
>> >> >>>>> > > > >> > integrating
>> >> >>>>> > > > >> > >> >> into an
>> >> >>>>> > > > >> > >> >> >>> > >> > existing
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
>> >> other
>> >> >>>> systems use
>> >> >>>>> > > String
>> >> >>>>> > > > >> based
>> >> >>>>> > > > >> > >> >> headers
>> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
>> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
>> >> >>>>> > > > >> > >> >> >>> > >> > > it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
>> easier
>> >> for
>> >> >>>> these to
>> >> >>>>> > be
>> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
>> >> >>>>> > > > >> > >> >> >>> > >> > into.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How with
>> Int
>> >> >>>> based
>> >> >>>>> > headers
>> >> >>>>> > > could
>> >> >>>>> > > > we
>> >> >>>>> > > > >> > >> provide
>> >> >>>>> > > > >> > >> >> a
>> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> integration
>> >> >>>> simple /
>> >> >>>>> > easy with
>> >> >>>>> > > > >> > transition
>> >> >>>>> > > > >> > >> >> flows
>> >> >>>>> > > > >> > >> >> >>> > over
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
>> luck
>> >> >>>> buddy
>> >> >>>>> > you're on
>> >> >>>>> > > your
>> >> >>>>> > > > >> own
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * simply
>> >> hash
>> >> >>>> the string
>> >> >>>>> > into
>> >> >>>>> > > int
>> >> >>>>> > > > >> code
>> >> >>>>> > > > >> > >> and
>> >> >>>>> > > > >> > >> >> hope
>> >> >>>>> > > > >> > >> >> >>> > for
>> >> >>>>> > > > >> > >> >> >>> > >> no
>> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
>> back
>> >> >>>> though?)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
>> >> style
>> >> >> as
>> >> >>>>> > mentioned by
>> >> >>>>> > > > nacho.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > ______________________________
>> >> >>>>> > > > >> > __________
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
>> radai
>> >> <
>> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
>> >> >> Wednesday,
>> >> >>>>> > November 9,
>> >> >>>>> > > 2016
>> >> >>>>> > > > >> > 8:12 AM
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
>> >> >>>> dev@kafka.apache.org
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Subject:
>> Re:
>> >> >>>> [DISCUSS]
>> >> >>>>> > KIP-82 -
>> >> >>>>> > > > Add
>> >> >>>>> > > > >> > >> Record
>> >> >>>>> > > > >> > >> >> >>> Headers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > thinking
>> >> about
>> >> >>>> it some
>> >> >>>>> > more,
>> >> >>>>> > > the
>> >> >>>>> > > > >> best
>> >> >>>>> > > > >> > >> way to
>> >> >>>>> > > > >> > >> >> >>> > transmit
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
>> >> >>>> consumers would
>> >> >>>>> > be to
>> >> >>>>> > > > put it
>> >> >>>>> > > > >> > in
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> MD
>> >> >>>>> > > > >> > >> >> >>> > >> response
>> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it should
>> be
>> >> >>>> discussed
>> >> >>>>> > now.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
>> Nov
>> >> 9,
>> >> >>>> 2016 at
>> >> >>>>> > 12:09
>> >> >>>>> > > AM,
>> >> >>>>> > > > >> > radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >> radai.rosenblatt@gmail.com
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im not
>> >> >> opposed
>> >> >>>> to the
>> >> >>>>> > idea of
>> >> >>>>> > > > >> > namespace
>> >> >>>>> > > > >> > >> >> >>> mapping.
>> >> >>>>> > > > >> > >> >> >>> > >> all
>> >> >>>>> > > > >> > >> >> >>> > >> > im
>> >> >>>>> > > > >> > >> >> >>> > >> > > saying
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not part
>> >> of
>> >> >>>> the "mvp"
>> >> >>>>> > and,
>> >> >>>>> > > > since
>> >> >>>>> > > > >> it
>> >> >>>>> > > > >> > >> >> requires
>> >> >>>>> > > > >> > >> >> >>> no
>> >> >>>>> > > > >> > >> >> >>> > >> wire
>> >> >>>>> > > > >> > >> >> >>> > >> > format
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > always
>> be
>> >> >>>> added later.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
>> its
>> >> not
>> >> >>>> as
>> >> >>>>> > simple as
>> >> >>>>> > > just
>> >> >>>>> > > > >> > >> >> configuring
>> >> >>>>> > > > >> > >> >> >>> MM
>> >> >>>>> > > > >> > >> >> >>> > to
>> >> >>>>> > > > >> > >> >> >>> > >> do
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > transform:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say i've
>> >> >>>> implemented
>> >> >>>>> > large
>> >> >>>>> > > > message
>> >> >>>>> > > > >> > >> >> support as
>> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
>> >> >>>>> > > > >> > >> >> >>> > >> > > on
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > target
>> >> >> cluster
>> >> >>>> its been
>> >> >>>>> > > > remapped
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> >> {999,1}.
>> >> >>>>> > > > >> > >> >> >>> the
>> >> >>>>> > > > >> > >> >> >>> > >> > consumer
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
>> need
>> >> to
>> >> >>>> be told
>> >> >>>>> > to look
>> >> >>>>> > > > for
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> large
>> >> >>>>> > > > >> > >> >> >>> > message
>> >> >>>>> > > > >> > >> >> >>> > >> > "part X
>> >> >>>>> > > > >> > >> >> >>> > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
>> >> >> {999,1}.
>> >> >>>> doable,
>> >> >>>>> > but
>> >> >>>>> > > > tricky.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On Tue,
>> >> Nov
>> >> >> 8,
>> >> >>>> 2016 at
>> >> >>>>> > 10:29
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > Gwen
>> >> >>>>> > > > >> > >> >> >>> Shapira <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> gwen@confluent.io
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> While
>> you
>> >> >> can
>> >> >>>> do
>> >> >>>>> > whatever
>> >> >>>>> > > you
>> >> >>>>> > > > >> want
>> >> >>>>> > > > >> > >> with a
>> >> >>>>> > > > >> > >> >> >>> > >> namespace
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > your
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
>> I'd
>> >> >>>> expect is
>> >> >>>>> > for each
>> >> >>>>> > > > app
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> >> >>> namespaces
>> >> >>>>> > > > >> > >> >> >>> > >> > > configurable...
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So if I
>> >> >>>> accidentally
>> >> >>>>> > used
>> >> >>>>> > > 666
>> >> >>>>> > > > for
>> >> >>>>> > > > >> > my
>> >> >>>>> > > > >> > >> HR
>> >> >>>>> > > > >> > >> >> >>> > >> department,
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > still
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
>> >> >> RadaiApp,
>> >> >>>> I can
>> >> >>>>> > config
>> >> >>>>> > > > >> > >> "namespace=42"
>> >> >>>>> > > > >> > >> >> >>> for
>> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
>> look
>> >> >>>> normal.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
>> >> means
>> >> >>>> you only
>> >> >>>>> > need to
>> >> >>>>> > > > sync
>> >> >>>>> > > > >> > usage
>> >> >>>>> > > > >> > >> >> >>> inside
>> >> >>>>> > > > >> > >> >> >>> > your
>> >> >>>>> > > > >> > >> >> >>> > >> > own
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> Still
>> >> hard,
>> >> >>>> but
>> >> >>>>> > somewhat
>> >> >>>>> > > > easier
>> >> >>>>> > > > >> > than
>> >> >>>>> > > > >> > >> >> syncing
>> >> >>>>> > > > >> > >> >> >>> > with
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > entire
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > world.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On Tue,
>> >> Nov
>> >> >>>> 8, 2016
>> >> >>>>> > at 10:07
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > >> radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>> radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > and
>> we
>> >> can
>> >> >>>> start
>> >> >>>>> > with
>> >> >>>>> > > > >> {namespace,
>> >> >>>>> > > > >> > >> id}
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> no
>> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> always
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > add
>> it
>> >> >>>> later on
>> >> >>>>> > if/when
>> >> >>>>> > > > >> > collisions
>> >> >>>>> > > > >> > >> >> >>> actually
>> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
>> >> >>>>> > > > >> > >> >> >>> > >> > > dont
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
>> >> >> problem).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > every
>> >> >>>> interested
>> >> >>>>> > party (so
>> >> >>>>> > > > orgs
>> >> >>>>> > > > >> > or
>> >> >>>>> > > > >> > >> >> >>> > individuals)
>> >> >>>>> > > > >> > >> >> >>> > >> > could
>> >> >>>>> > > > >> > >> >> >>> > >> > > then
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > register
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> prefix
>> >> (0
>> >> >> =
>> >> >>>>> > reserved, 1 =
>> >> >>>>> > > > >> > confluent
>> >> >>>>> > > > >> > >> ...
>> >> >>>>> > > > >> > >> >> >>> 666
>> >> >>>>> > > > >> > >> >> >>> > = me
>> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
>> >> >>>>> > > > >> > >> >> >>> > >> > > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > the
>> >> 2nd ID
>> >> >>>> - so once
>> >> >>>>> > > > linkedin
>> >> >>>>> > > > >> > >> >> registers,
>> >> >>>>> > > > >> > >> >> >>> say
>> >> >>>>> > > > >> > >> >> >>> > 3,
>> >> >>>>> > > > >> > >> >> >>> > >> > then
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
>> use
>> >> {3,
>> >> >>>> *} with a
>> >> >>>>> > > > reasonable
>> >> >>>>> > > > >> > >> >> >>> expectation
>> >> >>>>> > > > >> > >> >> >>> > to
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > collide
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > anything
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > else.
>> >> >>>> further
>> >> >>>>> > partitioning
>> >> >>>>> > > > of
>> >> >>>>> > > > >> > that *
>> >> >>>>> > > > >> > >> >> >>> becomes
>> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > problem,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> "upstream
>> >> >>>>> > registration"
>> >> >>>>> > > of a
>> >> >>>>> > > > >> > >> namespace
>> >> >>>>> > > > >> > >> >> >>> only
>> >> >>>>> > > > >> > >> >> >>> > has
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > happen
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
>> Tue,
>> >> >> Nov
>> >> >>>> 8, 2016
>> >> >>>>> > at
>> >> >>>>> > > 9:03
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > >> James
>> >> >>>>> > > > >> > >> >> >>> Cheng <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> wushujames@gmail.com
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>> >> Nov
>> >> >>>> 8, 2016,
>> >> >>>>> > at 5:54
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > Gwen
>> >> >>>>> > > > >> > >> >> >>> Shapira <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > gwen@confluent.io
>> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> Thank
>> >> >>>> you so
>> >> >>>>> > much for
>> >> >>>>> > > > this
>> >> >>>>> > > > >> > clear
>> >> >>>>> > > > >> > >> and
>> >> >>>>> > > > >> > >> >> >>> fair
>> >> >>>>> > > > >> > >> >> >>> > >> > summary of
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > arguments.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> I'm
>> >> in
>> >> >>>> favor of
>> >> >>>>> > ints.
>> >> >>>>> > > > Not a
>> >> >>>>> > > > >> > >> >> >>> deal-breaker,
>> >> >>>>> > > > >> > >> >> >>> > but
>> >> >>>>> > > > >> > >> >> >>> > >> > in
>> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> Even
>> >> >>>> more in
>> >> >>>>> > favor of
>> >> >>>>> > > > >> Magnus's
>> >> >>>>> > > > >> > >> >> >>> > decentralized
>> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> tweak:
>> >> >>>> add a
>> >> >>>>> > namespace
>> >> >>>>> > > > for
>> >> >>>>> > > > >> > >> headers.
>> >> >>>>> > > > >> > >> >> >>> This
>> >> >>>>> > > > >> > >> >> >>> > will
>> >> >>>>> > > > >> > >> >> >>> > >> > allow
>> >> >>>>> > > > >> > >> >> >>> > >> > > each
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> use
>> >> >>>> whatever IDs
>> >> >>>>> > it
>> >> >>>>> > > wants
>> >> >>>>> > > > >> > >> >> internally,
>> >> >>>>> > > > >> > >> >> >>> and
>> >> >>>>> > > > >> > >> >> >>> > >> then
>> >> >>>>> > > > >> > >> >> >>> > >> > let
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> deploying
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> the
>> >> app
>> >> >>>> figure
>> >> >>>>> > out an
>> >> >>>>> > > > >> > available
>> >> >>>>> > > > >> > >> >> >>> namespace
>> >> >>>>> > > > >> > >> >> >>> > ID
>> >> >>>>> > > > >> > >> >> >>> > >> > for the
>> >> >>>>> > > > >> > >> >> >>> > >> > > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > live
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > So
>> >> >>>>> > > > >> > io.confluent.schema-registry
>> >> >>>>> > > > >> > >> can
>> >> >>>>> > > > >> > >> >> be
>> >> >>>>> > > > >> > >> >> >>> > >> > namespace
>> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> deployment
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> and
>> >> >> 0x57
>> >> >>>> on
>> >> >>>>> > yours, and
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > poor
>> >> >>>>> > > > >> > >> guys
>> >> >>>>> > > > >> > >> >> >>> > >> > developing the
>> >> >>>>> > > > >> > >> >> >>> > >> > > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> worry
>> >> >>>> about that.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> Gwen,
>> >> if
>> >> >> I
>> >> >>>>> > understand
>> >> >>>>> > > your
>> >> >>>>> > > > >> > example
>> >> >>>>> > > > >> > >> >> >>> right, an
>> >> >>>>> > > > >> > >> >> >>> > >> > > application
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> decide to
>> >> >>>> use 0x01
>> >> >>>>> > in one
>> >> >>>>> > > > >> > >> deployment,
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> > that
>> >> >>>>> > > > >> > >> >> >>> > >> > means
>> >> >>>>> > > > >> > >> >> >>> > >> > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> message
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> is
>> >> >> written
>> >> >>>> into the
>> >> >>>>> > > > broker, it
>> >> >>>>> > > > >> > >> will be
>> >> >>>>> > > > >> > >> >> >>> > saved on
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > broker
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> specific
>> >> >>>> namespace
>> >> >>>>> > > (0x01).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> If
>> you
>> >> >>>> were to
>> >> >>>>> > mirror
>> >> >>>>> > > that
>> >> >>>>> > > > >> > message
>> >> >>>>> > > > >> > >> >> into
>> >> >>>>> > > > >> > >> >> >>> > another
>> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> accompany
>> >> >>>> the
>> >> >>>>> > message,
>> >> >>>>> > > > right?
>> >> >>>>> > > > >> > What
>> >> >>>>> > > > >> > >> if
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> other
>> >> >>>> cluster uses
>> >> >>>>> > 0x57?
>> >> >>>>> > > > They
>> >> >>>>> > > > >> > won't
>> >> >>>>> > > > >> > >> >> >>> > understand
>> >> >>>>> > > > >> > >> >> >>> > >> > each
>> >> >>>>> > > > >> > >> >> >>> > >> > > other?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> I'm
>> >> not
>> >> >>>> sure
>> >> >>>>> > that's an
>> >> >>>>> > > > >> avoidable
>> >> >>>>> > > > >> > >> >> >>> problem. I
>> >> >>>>> > > > >> > >> >> >>> > >> > think it
>> >> >>>>> > > > >> > >> >> >>> > >> > > simply
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> order
>> >> to
>> >> >>>> share
>> >> >>>>> > data, you
>> >> >>>>> > > > have
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> also
>> >> >>>>> > > > >> > >> >> >>> have a
>> >> >>>>> > > > >> > >> >> >>> > >> > shared
>> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>> understanding of
>> >> >>>>> > what the
>> >> >>>>> > > > >> > >> namespaces
>> >> >>>>> > > > >> > >> >> >>> mean.
>> >> >>>>> > > > >> > >> >> >>> > >> Which
>> >> >>>>> > > > >> > >> >> >>> > >> > I
>> >> >>>>> > > > >> > >> >> >>> > >> > > think
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> because
>> >> >> the
>> >> >>>>> > alternate
>> >> >>>>> > > > (sharing
>> >> >>>>> > > > >> > >> >> *nothing*
>> >> >>>>> > > > >> > >> >> >>> at
>> >> >>>>> > > > >> > >> >> >>> > >> all)
>> >> >>>>> > > > >> > >> >> >>> > >> > would
>> >> >>>>> > > > >> > >> >> >>> > >> > > mean
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> would
>> >> be
>> >> >>>> no way to
>> >> >>>>> > > > understand
>> >> >>>>> > > > >> > each
>> >> >>>>> > > > >> > >> >> other.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> -James
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> Gwen
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>> >> Tue,
>> >> >>>> Nov 8,
>> >> >>>>> > 2016 at
>> >> >>>>> > > > 4:23
>> >> >>>>> > > > >> > PM,
>> >> >>>>> > > > >> > >> >> radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>> radai.rosenblatt@gmail.com
>> >> >>>>> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>> +1
>> >> for
>> >> >>>> sean's
>> >> >>>>> > > document.
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > >> covers
>> >> >>>>> > > > >> > >> >> >>> pretty
>> >> >>>>> > > > >> > >> >> >>> > >> much
>> >> >>>>> > > > >> > >> >> >>> > >> > all
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>> >> >> provides
>> >> >>>>> > concrete
>> >> >>>>> > > > figures
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> argue
>> >> >>>>> > > > >> > >> >> >>> about
>> >> >>>>> > > > >> > >> >> >>> > :-)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>> >> >>>> (nit-picking -
>> >> >>>>> > used
>> >> >>>>> > > the
>> >> >>>>> > > > >> same
>> >> >>>>> > > > >> > >> xkcd
>> >> >>>>> > > > >> > >> >> >>> twice,
>> >> >>>>> > > > >> > >> >> >>> > >> also
>> >> >>>>> > > > >> > >> >> >>> > >> > trove
>> >> >>>>> > > > >> > >> >> >>> > >> > > has
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> superceded
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > --
>> >> >>>>> > > > >> > Gwen Shapira
>> >> >>>>> > > > >> > Product Manager | Confluent
>> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
>> >> >>>>> > > > >> > Follow us: Twitter | blog
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> --
>> >> >>>>> > > > >> *Todd Palino*
>> >> >>>>> > > > >> Staff Site Reliability Engineer
>> >> >>>>> > > > >> Data Infrastructure Streaming
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> linkedin.com/in/toddpalino
>> >> >>>>> > > > >>
>> >> >>>>> > > >
>> >> >>>>> > > >
>> >> >>>>> > > >
>> >> >>>>> > > > --
>> >> >>>>> > > > Gwen Shapira
>> >> >>>>> > > > Product Manager | Confluent
>> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
>> >> >>>>> > > > Follow us: Twitter | blog
>> >> >>>>> > > >
>> >> >>>>> > >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > The information contained in this email is strictly
>> >> confidential
>> >> >>>> and for
>> >> >>>>> > the use of the addressee only, unless otherwise indicated.
>> If
>> >> you
>> >> >>>> are not
>> >> >>>>> > the intended recipient, please do not read, copy, use or
>> >> disclose
>> >> >>>> to others
>> >> >>>>> > this message or any attachment. Please also notify the
>> sender
>> >> by
>> >> >>>> replying
>> >> >>>>> > to this email or by telephone (+44(020 7896 0011) and then
>> >> delete
>> >> >>>> the email
>> >> >>>>> > and any copies of it. Opinions, conclusion (etc) that do not
>> >> >>>> relate to the
>> >> >>>>> > official business of this company shall be understood as
>> >> neither
>> >> >>>> given nor
>> >> >>>>> > endorsed by it. IG is a trading name of IG Markets Limited
>> (a
>> >> >>>> company
>> >> >>>>> > registered in England and Wales, company number 04008957)
>> and
>> >> IG
>> >> >>>> Index
>> >> >>>>> > Limited (a company registered in England and Wales, company
>> >> >> number
>> >> >>>>> > 01190902). Registered address at Cannon Bridge House, 25
>> >> Dowgate
>> >> >>>> Hill,
>> >> >>>>> > London EC4R 2YA. Both IG Markets Limited (register number
>> >> 195355)
>> >> >>>> and IG
>> >> >>>>> > Index Limited (register number 114059) are authorised and
>> >> >>>> regulated by the
>> >> >>>>> > Financial Conduct Authority.
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> The information contained in this email is strictly confidential
>> and
>> >> >> for
>> >> >>>> the use of the addressee only, unless otherwise indicated. If you
>> are
>> >> >> not
>> >> >>>> the intended recipient, please do not read, copy, use or disclose
>> to
>> >> >> others
>> >> >>>> this message or any attachment. Please also notify the sender by
>> >> >> replying
>> >> >>>> to this email or by telephone (+44(020 7896 0011) and then delete
>> the
>> >> >> email
>> >> >>>> and any copies of it. Opinions, conclusion (etc) that do not relate
>> >> to
>> >> >> the
>> >> >>>> official business of this company shall be understood as neither
>> >> given
>> >> >> nor
>> >> >>>> endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> >> >>>> registered in England and Wales, company number 04008957) and IG
>> >> Index
>> >> >>>> Limited (a company registered in England and Wales, company number
>> >> >>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> >> Hill,
>> >> >>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> >> and IG
>> >> >>>> Index Limited (register number 114059) are authorised and regulated
>> >> by
>> >> >> the
>> >> >>>> Financial Conduct Authority.
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog
The information contained in this email is strictly confidential and for the use of the addressee only, unless otherwise indicated. If you are not the intended recipient, please do not read, copy, use or disclose to others this message or any attachment. Please also notify the sender by replying to this email or by telephone (+44(020 7896 0011) and then delete the email and any copies of it. Opinions, conclusion (etc) that do not relate to the official business of this company shall be understood as neither given nor endorsed by it. IG is a trading name of IG Markets Limited (a company registered in England and Wales, company number 04008957) and IG Index Limited (a company registered in England and Wales, company number 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG Index Limited (register number 114059) are authorised and regulated by the Financial Conduct Authority.
Re: [DISCUSS] KIP-82 - Add Record Headers
Posted by Gwen Shapira <gw...@confluent.io>.
Mirroring + ClusterIDs:
Kafka clusters already have an automatically generated ID that is
exposed to clients:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
It sounds trivial to get MirrorMaker's consumer to grab this ID (which
always exists) from the source cluster and add it to the message
headers.
Obviously adding stuff to the message makes it larger so this should
be a feature you should be able to turn on/off depending on the
replication requirements. If you need the feature - you pay the price.
I don't think there is an immediate need to put an effort into
shrinking the Cluster ID, but others can comment on that.
On Mon, Jan 9, 2017 at 11:19 AM, Jay Kreps <ja...@confluent.io> wrote:
> Hey Michael,
>
> Sorry for lagging, I've been out for the holiday break. What would help me
> would be to get to a consolidated list of use cases we've worked out and
> think add real value. I think getting down to the details really matters
> for these.
>
> The two lists I'm aware of are the one in the KIP and the one you gave in
> this email thread. Both seem to be more at the sketch level. Let me review
> the use cases and see if I have it right. If so let's get it in the design
> proposal.
>
> Your use cases:
>
> 1. Tracing IDs -- I get this use case and think I see how it would work.
> I think because the tracing is from a third party system there is some
> value to having a header versus a custom format to make this work fully
> transparently.
> 2. Mirroring -- I'm a bit unsure how this will work, I'll explain my
> thinking below
> 3. End-to-end encryption -- I think what you are saying is that if you
> want "routing' fields that are accessible in transit without decrypting you
> need a scheme to leave this decrypted.I think the idea is to avoid
> decrypting to get a header, but I'm not sure if I get how this is better
> than a custom message format that doesn't encrypt the things you don't want
> encrypted since you have to be aware of the scheme anyway.
> 4. EOS -- I'm not convinced this is a good approach for EOS.
>
> Linked from KIP:
>
> 1. Schema-Registry-aware avro serialization -- not clear how this is
> better than keeping the schema id in the value
> 2. Polyglot kafka topics -- marginally better than a container format,
> but of course you still have to all agree on the the header and
> header=>serialization mapping so not sure if it's really better than a
> container format. Also not wild about polyglot topics :-)
> 3. End-to-End routing trail including mirroring --- I'd put this under
> tracing ids
> 4. Cross-service process tracing -- ditto
> 5. Signing -- I like this use case but it seems to depend on a feature
> we don't have and aren't proposing adding, right?
> 6. Per-record custom compression -- why do you want this? compressing
> per message won't work well compared to batch compression, right?
> 7. Large Message Support -- I think we can agree that although this
> might work it's clearly not the right way to support large messages in Kafka
>
> Matthias
>
> 1. Control messages --- Matthias, not sure if I get the motivation in
> streams for this. If we did want to do it, wouldn't it be easier to
> implement and by allowing users to send control messages (in the EOS sense)
> and adding a consumer config to allow either passing these on or
> suppressing them?
>
> So I if I were to summarize I'd list the use cases as follows:
>
> - Much better with headers: tracing IDs
> - Somewhat better: end-to-end encryption, polyglot topics
> - Don't buy: Mirroring (though perhaps could be made to work with
> changes to the proposal), EOS, large messages, compression
> - Not sure: Control messages
>
> Does this match where other people are at? From my point of view for
> something like this where we are changing the core data model we should
> ideally get a handful of use cases to the point where they are really well
> thought out and we know they can use the feature. I think we have one solid
> one with tracing IDs, and I think maybe mirroring could be another with
> some work. I'm less sure about the others.
>
> Bi-directional Mirroring
> I think bidirectional mirroring could potentially become a compelling use
> case if we an find a way to make it work. If I understand it, the proposal
> is to add a header like "origin-cluster" with the cluster id of the Kafka
> cluster. Then modify the mirroring tool to disallow mirroring a message
> back to it's own origin cluster. This basic idea is definitely workable and
> useful, but can we really get to a good implementation using headers?
> I imagine the approach would be to add this header in the producer if some
> config is set. There are two problems:
>
> - You add a pretty big honking cluster ID to each message. For small
> messages this maybe larger than the contents itself.
> - What happens if someone somewhere forgets to set that config on some
> client somewhere?
>
> If someone forgets to set the config in some client somewhere I guess you
> have various options in mirror maker:
>
> 1. drop the data (pretty bad, now your data is out of sync)
> 2. crash (pretty bad--how do you recover)
> 3. insert some default (pretty bad since would also lead to data loss if
> that's not right)
>
> Basically although I think a cluster tag could be a good way to do
> bi-directional mirroring, I'm not sure this naive approach I described
> would get us to a usable implementation. This could be fixable. I think if
> there was some scheme to check required headers on the server that could
> alleviate the missing header problem. Not really sure what to do about the
> message bloat problem, but maybe there could be some way to shrink the
> cluster id?
>
> Cheers,
>
> -Jay
>
>
> On Mon, Dec 19, 2016 at 10:08 PM, Michael Pearce <Mi...@ig.com>
> wrote:
>
>> @Jay, Ismael and a few others.
>>
>> Based on the discussion on kip-87, and the additional use case by Matthias.
>>
>> Just for clarity do we now have consensus on the earlier points (1)
>> headers are useful (2) headers should be native in Kafka?
>>
>> Cheers
>> Mike
>> ________________________________________
>> From: radai <ra...@gmail.com>
>> Sent: Monday, December 19, 2016 5:01 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>> having said that im perfectly fine with something like a "no payload" flag
>> (as opposed to null payload), which would mean such a msg is not returned
>> from poll() unless the user code specifically "opts in" and would serve
>> only to carry headers (making it a control message)
>>
>> On Mon, Dec 19, 2016 at 8:58 AM, radai <ra...@gmail.com> wrote:
>>
>> > if all you want is to tag "end of stream" wouldnt a header on the last
>> msg
>> > in the stream be enough?
>> >
>> > On Sun, Dec 18, 2016 at 2:03 PM, Matthias J. Sax <ma...@confluent.io>
>> > wrote:
>> >
>> >> The details about headers for control messages are still to define. But
>> >> yes, the idea is to have some common default behavior that clients would
>> >> need to implement.
>> >>
>> >> The point is, that "regular headers" add meta data to regular messages.
>> >> Thus, those messages will be returned to the user via .poll(). And after
>> >> the message is received the user can check if meta data is present and
>> >> read it.
>> >>
>> >> For control messages, we do not want those to pop up via .poll() as
>> >> those are no regular messages. A client would need to opt-in to see
>> >> those messages (either via poll() or maybe a callback). Thus, we need
>> >> some special (standardized) header IDs that indicate control messages
>> >> that should not be returned to the user via poll() by default.
>> >>
>> >>
>> >> -Matthias
>> >>
>> >> On 12/17/16 9:37 PM, Roger Hoover wrote:
>> >> > Matthias,
>> >> >
>> >> > Thanks for your input. I'm +1 on control messages as they seem to be
>> >> the
>> >> > simplest way to implement watermarks (
>> >> > https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102),
>> a
>> >> > feature that would add a lot of value to Kafka Streams IMHO.
>> >> >
>> >> > Your argument that the control-message indicator belongs in the
>> >> client-only
>> >> > section of the record format make sense. Just to make sure I
>> >> understand,
>> >> > are you suggesting that control messages would be indicated by a
>> >> standard
>> >> > reserved header? By standard, I mean that ALL Kafka consumers would
>> >> know
>> >> > to handle these messages differently (possibly just ignoring them).
>> >> This
>> >> > would need to be added to the specification of the consumer protocol
>> so
>> >> > that all Kafka clients implement it, right? I think it's a good idea
>> >> but
>> >> > just checking.
>> >> >
>> >> > Cheers,
>> >> >
>> >> > Roger
>> >> >
>> >> >
>> >> > On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax <
>> matthias@confluent.io
>> >> >
>> >> > wrote:
>> >> >
>> >> >> Yes and no. I did overload the term "control message".
>> >> >>
>> >> >> EOS control messages are for client-broker communication and thus
>> never
>> >> >> exposed to any application. And I think this is a good design because
>> >> >> broker needs to understand those control messages. Thus, this should
>> be
>> >> >> a protocol change.
>> >> >>
>> >> >> The type of control messages I have in mind are for client-client
>> >> >> (application-application) communication and the broker is agnostic to
>> >> >> them. Thus, it should not be a protocol change.
>> >> >>
>> >> >>
>> >> >> -Matthias
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 12/14/16 9:42 AM, radai wrote:
>> >> >>> arent control messages getting pushed as their own top level
>> protocol
>> >> >>> change (and a fairly massive one) for the transactions KIP ?
>> >> >>>
>> >> >>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax <
>> >> matthias@confluent.io>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> I want to add a completely new angle to this discussion. For this,
>> I
>> >> >>>> want to propose an extension for the headers feature that enables
>> new
>> >> >>>> uses cases -- and those new use cases might convince people to
>> >> support
>> >> >>>> headers (of course including the larger scoped proposal).
>> >> >>>>
>> >> >>>> Extended Proposal:
>> >> >>>>
>> >> >>>> Allow messages with a certain header key to be special "control
>> >> >>>> messages" (w/ o w/o payload) that are not exposed to an application
>> >> via
>> >> >>>> .poll().
>> >> >>>>
>> >> >>>> Thus, a consumer client would automatically skip over those
>> >> messages. If
>> >> >>>> an application knows about embedded control messages, it can "sing
>> >> up"
>> >> >>>> to those messages by the consumer client and either get a callback
>> or
>> >> >>>> the consumer auto-drop for this messages gets disabled (allowing to
>> >> >>>> consumer those messages via poll()).
>> >> >>>>
>> >> >>>> (The details need further considerations/discussion. I just want to
>> >> >>>> sketch the main idea.)
>> >> >>>>
>> >> >>>> Usage:
>> >> >>>>
>> >> >>>> There is a shared topic (ie, used by multiple applications) and a
>> >> >>>> producer application wants to embed a special message in the topic
>> >> for a
>> >> >>>> dedicated consumer application. Because only one application will
>> >> >>>> understand this message, it cannot be a regular message as this
>> would
>> >> >>>> break all applications that do not understand this message. The
>> >> producer
>> >> >>>> application would set a special metadata key and no consumer
>> >> application
>> >> >>>> would see this control message by default because they did not
>> enable
>> >> >>>> their consumer client to return this message in poll() (and the
>> >> client
>> >> >>>> would just drop this message with special metadata key). Only the
>> >> single
>> >> >>>> application that should receive this message, will subscribe to
>> this
>> >> >>>> message on its consumer client and process it.
>> >> >>>>
>> >> >>>>
>> >> >>>> Concrete Use Case: Kafka Streams
>> >> >>>>
>> >> >>>> In Kafka Streams, we would like to propagate "control messages"
>> from
>> >> >>>> subtopology to subtopology. There are multiple scenarios for which
>> >> this
>> >> >>>> would be useful. For example, currently we do not guarantee a
>> >> >>>> "consistent shutdown" of an application. By this, I mean that input
>> >> >>>> records might not be completely processed by the whole topology
>> >> because
>> >> >>>> the application shutdown happens "in between" and an intermediate
>> >> result
>> >> >>>> topic gets "stock" in an intermediate topic. Thus, a user would see
>> >> an
>> >> >>>> committed offset of the source topic of the application, but no
>> >> >>>> corresponding result record in the output topic.
>> >> >>>>
>> >> >>>> Having "shutdown markers" would allow us, to first stop the
>> upstream
>> >> >>>> subtopology and write this marker into the intermediate topic and
>> the
>> >> >>>> downstream subtopology would only shut down itself after is sees
>> the
>> >> >>>> "shutdown marker". Thus, we can guarantee on shutdown, that no
>> >> >>>> "in-flight" messages got stuck in intermediate topics.
>> >> >>>>
>> >> >>>>
>> >> >>>> A similar usage would be for KIP-95 (Incremental Batch Processing).
>> >> >>>> There was a discussion about the proposed metadata topic, and we
>> >> could
>> >> >>>> avoid this metadata topic if we would have "control messages".
>> >> >>>>
>> >> >>>>
>> >> >>>> Right now, we cannot insert an "application control message"
>> because
>> >> >>>> Kafka Streams does not own all topics it read/writes and thus might
>> >> >>>> break other consumer application (as described above) if we inject
>> >> >>>> random messages that are not understood by other apps.
>> >> >>>>
>> >> >>>>
>> >> >>>> Of course, one can work around "embedded control messaged" by using
>> >> an
>> >> >>>> additional topic to propagate control messaged between application
>> >> (as
>> >> >>>> suggestion in KIP-95 via a metadata topic for Kafka Streams). But
>> >> there
>> >> >>>> are major concerns about adding this metadata topic in the KIP and
>> >> this
>> >> >>>> shows that other application that need a similar pattern might
>> profit
>> >> >>>> from topic embedded "control messages", too.
>> >> >>>>
>> >> >>>>
>> >> >>>> One last important consideration: those "control messages" are used
>> >> for
>> >> >>>> client to client communication and are not understood by the
>> broker.
>> >> >>>> Thus, those messages should not be enabled within the message
>> format
>> >> >>>> (c.f. tombstone flag -- KIP-87). However, "client land" record
>> >> headers
>> >> >>>> would be a nice way to implement them. Because KIP-82 did consider
>> >> key
>> >> >>>> namespaces for metatdata keys, this extension should not be an own
>> >> KIP
>> >> >>>> but should be included in KIP-82 to reserve a namespace for
>> "control
>> >> >>>> message" in the first place.
>> >> >>>>
>> >> >>>>
>> >> >>>> Sorry for the long email... Looking forward to your feedback.
>> >> >>>>
>> >> >>>>
>> >> >>>> -Matthias
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >>>> On 12/8/16 12:12 AM, Michael Pearce wrote:
>> >> >>>>> Hi Jun
>> >> >>>>>
>> >> >>>>> 100) each time a transaction exits a jvm for a remote system
>> >> (HTTP/JMS/
>> >> >>>> Hopefully one day kafka) the APM tools stich in a unique id
>> (though I
>> >> >>>> believe it contains the end2end uuid embedded in this id), on
>> >> receiving
>> >> >> the
>> >> >>>> message at the receiving JVM the apm code takes this out, and
>> >> continues
>> >> >> its
>> >> >>>> tracing on the that new thread. Both JVM’s (and other languages the
>> >> APM
>> >> >>>> tool supports) send this data async back to the central controllers
>> >> >> where
>> >> >>>> the stiching togeather occurs. For this they need some header space
>> >> for
>> >> >>>> them to put this id.
>> >> >>>>>
>> >> >>>>> 101) Yes indeed we have a business transaction Id in the payload.
>> >> >> Though
>> >> >>>> this is a system level tracing, that we need to have marry up. Also
>> >> as
>> >> >> per
>> >> >>>> note on end2end encryption we’d be unable to prove the flow if the
>> >> >> payload
>> >> >>>> is encrypted as we’d not have access to this at certain points of
>> the
>> >> >> flow
>> >> >>>> through the infrastructure/platform.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> 103) As said we use this mechanism in IG very successfully, as
>> >> stated
>> >> >>>> per key we guarantee the transaction producing app to handle the
>> >> >>>> transaction of a key at one DC unless at point of critical failure
>> >> >> where we
>> >> >>>> have to flip processing to another. We care about key ordering.
>> >> >>>>> I disagree on the offset comment for the partition solution unless
>> >> you
>> >> >>>> do full ISR, or expensive full XA transactions even with partitions
>> >> you
>> >> >>>> cannot fully guarantee offsets would match.
>> >> >>>>>
>> >> >>>>> 105) Very much so, I need to have access at the platform level to
>> >> the
>> >> >>>> other meta data all mentioned, without having to need to have
>> access
>> >> to
>> >> >> the
>> >> >>>> encryption keys of the payload.
>> >> >>>>>
>> >> >>>>> 106)
>> >> >>>>> Techincally yes for AZ/Region/Cluster, but then we’d need to have
>> a
>> >> >>>> global producerId register which would be very hard to
>> >> enforce/ensure is
>> >> >>>> current and correct, just to understand the message origins of its
>> >> >>>> region/az/cluster for routing.
>> >> >>>>> The client wrapper version, producerId can be the same, as
>> obviously
>> >> >> the
>> >> >>>> producer could upgrade its wrapper, as such we need to know what
>> >> wrapper
>> >> >>>> version the message is created with.
>> >> >>>>> Likewise the IP address, as stated we can have our producer move,
>> >> where
>> >> >>>> its IP would change.
>> >> >>>>>
>> >> >>>>> 107)
>> >> >>>>> UUID is set on the message by interceptors before actual producer
>> >> >>>> transport send. This is for platform level message dedupe
>> guarantee,
>> >> the
>> >> >>>> business payload should be agnostic to this. Please see
>> >> >>>> https://activemq.apache.org/artemis/docs/1.5.0/duplicate-det
>> >> ection.html
>> >> >>>> note this is not touching business payloads.
>> >> >>>>>
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On 06/12/2016, 18:22, "Jun Rao" <ju...@confluent.io> wrote:
>> >> >>>>>
>> >> >>>>> Hi, Michael,
>> >> >>>>>
>> >> >>>>> Thanks for the reply. I find it very helpful.
>> >> >>>>>
>> >> >>>>> Data lineage:
>> >> >>>>> 100. I'd like to understand the APM use case a bit more. It
>> >> sounds
>> >> >>>> like
>> >> >>>>> that those APM plugins can generate a transaction id that we
>> >> could
>> >> >>>>> potentially put in the header of every message. How would you
>> >> >>>> typically
>> >> >>>>> make use of such transaction ids? Are there other metadata
>> >> >>>> associated with
>> >> >>>>> the transaction id and if so, how are they propagated
>> >> downstream?
>> >> >>>>>
>> >> >>>>> 101. For the finance use case, if the concept of transaction
>> is
>> >> >>>> important,
>> >> >>>>> wouldn't it be typically included in the message payload
>> >> instead of
>> >> >>>> as an
>> >> >>>>> optional header field?
>> >> >>>>>
>> >> >>>>> 102. The data lineage that Altas and Navigator support seems
>> to
>> >> be
>> >> >>>> at the
>> >> >>>>> dataset level, not per record level? So, not sure if per
>> message
>> >> >>>> headers
>> >> >>>>> are relevant there.
>> >> >>>>>
>> >> >>>>> Mirroring:
>> >> >>>>> 103. The benefit of using separate partitions is that it
>> >> >> potentially
>> >> >>>> makes
>> >> >>>>> it easy to preserve offsets during mirroring. This will make
>> it
>> >> >>>> easier for
>> >> >>>>> consumer to switch clusters. Currently, the consumers can
>> switch
>> >> >>>> clusters
>> >> >>>>> by using the timestampToOffset() api, but it has to deal with
>> >> >>>> duplicates.
>> >> >>>>> Good point on the issue with log compact and I am not sure how
>> >> to
>> >> >>>> address
>> >> >>>>> this. However, even if we mirror into the existing partitions,
>> >> the
>> >> >>>> ordering
>> >> >>>>> for messages generated from different clusters seems
>> >> >>>> non-deterministic
>> >> >>>>> anyway. So, it seems that the consumers already have to deal
>> >> with
>> >> >>>> that? If
>> >> >>>>> a topic is compacted, does that mean which messages are
>> >> preserved
>> >> >> is
>> >> >>>> also
>> >> >>>>> non-deterministic across clusters?
>> >> >>>>>
>> >> >>>>> 104. Good point on partition key.
>> >> >>>>>
>> >> >>>>> End-to-end encryption:
>> >> >>>>> 105. So, it seems end-to-end encryption is useful. Are headers
>> >> >>>> useful there?
>> >> >>>>>
>> >> >>>>> Auditing:
>> >> >>>>> 106. It seems other than the UUID, all other metadata are per
>> >> >>>> producer?
>> >> >>>>>
>> >> >>>>> EOS:
>> >> >>>>> 107. How are those UUIDs generated? I am not sure if they can
>> be
>> >> >>>> generated
>> >> >>>>> in the producer library. An application may send messages
>> >> through a
>> >> >>>> load
>> >> >>>>> balancer and on retry, the same message could be routed to a
>> >> >>>> different
>> >> >>>>> producer instance. So, it seems that the application has to
>> >> >> generate
>> >> >>>> the
>> >> >>>>> UUIDs. In that case, shouldn't the application just put the
>> >> UUID in
>> >> >>>> the
>> >> >>>>> payload?
>> >> >>>>>
>> >> >>>>> Thanks,
>> >> >>>>>
>> >> >>>>> Jun
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> On Fri, Dec 2, 2016 at 4:57 PM, Michael Pearce <
>> >> >>>> Michael.Pearce@ig.com>
>> >> >>>>> wrote:
>> >> >>>>>
>> >> >>>>> > Hi Jun.
>> >> >>>>> >
>> >> >>>>> > Per Transaction Tracing / Data Lineage.
>> >> >>>>> >
>> >> >>>>> > As Stated in the KIP this has the first use case of how many
>> >> APM
>> >> >>>> tools now
>> >> >>>>> > work.
>> >> >>>>> > I would find it impossible for any one to argue this is not
>> >> >>>> important or a
>> >> >>>>> > niche market as it has its own gartner report for this
>> space.
>> >> >> Such
>> >> >>>>> > companies as Appdynamics, NewRelic, Dynatrace, Hawqular are
>> >> but a
>> >> >>>> few.
>> >> >>>>> >
>> >> >>>>> > Likewise these APM tools can help very rapidly track down
>> >> issues
>> >> >>>> and
>> >> >>>>> > automatically capture metrics, perform actions based on
>> >> >> unexpected
>> >> >>>> behavior
>> >> >>>>> > to auto recover services.
>> >> >>>>> >
>> >> >>>>> > Before mentioning looking at aggregated stats, in these
>> cases
>> >> >> where
>> >> >>>>> > actually on critical flows we cannot afford to have
>> aggregated
>> >> >>>> rolled up
>> >> >>>>> > stats only.
>> >> >>>>> >
>> >> >>>>> > With the APM tool we use its actually able to detect a
>> single
>> >> >>>> transaction
>> >> >>>>> > failure and capture the thread traces in the JVM where it
>> >> failed
>> >> >>>> and
>> >> >>>>> > everything for us, to the point it sends us alerts where we
>> >> have
>> >> >>>> this
>> >> >>>>> > giving the line number of the code that caused it, the
>> >> >> transaction
>> >> >>>> trace
>> >> >>>>> > through all the services and endpoints (supported) upto the
>> >> point
>> >> >>>> of
>> >> >>>>> > failure, it can also capture the data in and out (so we can
>> >> >>>> replay).
>> >> >>>>> > Because atm Kafka doesn’t support us being able to stich in
>> >> these
>> >> >>>> tracing
>> >> >>>>> > transaction ids natively, we cannot get these benefits as
>> >> such is
>> >> >>>> limiting
>> >> >>>>> > our ability support apps and monitor them to the same
>> >> standards
>> >> >> we
>> >> >>>> come to
>> >> >>>>> > expect when on a kafka flow.
>> >> >>>>> >
>> >> >>>>> > This actually ties in with Data Lineage, as the same tracing
>> >> can
>> >> >>>> be used
>> >> >>>>> > to back stich this. Essentially many times due to the sums
>> of
>> >> >> money
>> >> >>>>> > involved there are disputes, and typically as a financial
>> >> >>>> institute the
>> >> >>>>> > easiest and cleanest way to prove when disputes arise is to
>> >> >>>> present the
>> >> >>>>> > actual flow and processes involved in a transaction.
>> >> >>>>> >
>> >> >>>>> > Likewise as Hadoop matures its evident this case is
>> >> important, as
>> >> >>>> tools
>> >> >>>>> > such as Atlas (Hortonworks led) and Navigator (cloudera led)
>> >> are
>> >> >>>> evident
>> >> >>>>> > also I believe the importance here is very much NOT just a
>> >> >>>> financial issue.
>> >> >>>>> >
>> >> >>>>> > From a MDM point of view any company wanting to care about
>> >> Data
>> >> >>>> Quality
>> >> >>>>> > and Data Governance - Data Lineage is a key piece in this
>> >> puzzle.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE Mirroring,
>> >> >>>>> >
>> >> >>>>> > As per the KIP in-fact this is exactly what we do re cluster
>> >> id,
>> >> >>>> to mirror
>> >> >>>>> > a network of clusters between AZ’s / Regions. We know a
>> >> >>>> transaction for a
>> >> >>>>> > key will be done within a AZ/Region, as such we know the
>> >> write
>> >> >> to
>> >> >>>> kafka
>> >> >>>>> > would be ordered per key. But we need eventual view of that
>> >> >> across
>> >> >>>> in our
>> >> >>>>> > other regions/az’s. When we have complete AZ or Region
>> >> failure we
>> >> >>>> know
>> >> >>>>> > there will be a brief interruption whilst those transactions
>> >> are
>> >> >>>> moved to
>> >> >>>>> > another region but we expect after it to continue.
>> >> >>>>> >
>> >> >>>>> > As mentioned having separate Partions to do this starts to
>> get
>> >> >>>>> > ugly/complicated for us:
>> >> >>>>> > how would I do compaction where a key is in two partitions?
>> >> >>>>> > How do we balance consumers so where multiple partitions
>> with
>> >> the
>> >> >>>> same key
>> >> >>>>> > goto the same consumer
>> >> >>>>> > What do you do if cluster 1 has 5 partitions but cluster 20
>> >> has
>> >> >> 10
>> >> >>>> because
>> >> >>>>> > its larger kit in our more core DC’s, as such key to
>> partition
>> >> >>>> mappings for
>> >> >>>>> > consumers get even more complicated.
>> >> >>>>> > What do you do if we add or remove a complete region
>> >> >>>>> >
>> >> >>>>> > Where as simple mirror will work we just need to ensure we
>> >> don’t
>> >> >>>> have a
>> >> >>>>> > cycle which we can do with clusterId.
>> >> >>>>> >
>> >> >>>>> > We even have started to look at shortest path mirror routing
>> >> >> based
>> >> >>>> on
>> >> >>>>> > clusterId, if we also had the region and az info on the
>> >> >> originating
>> >> >>>>> > message, this we have not implemented but some ideas come
>> from
>> >> >>>> network
>> >> >>>>> > routing, and also the dispatcher router in apache qpid.
>> >> >>>>> >
>> >> >>>>> > Also we need to have data perimeters e.g. certain data
>> cannot
>> >> >> leave
>> >> >>>>> > certain countries borders. We want this all automated so
>> that
>> >> at
>> >> >>>> the
>> >> >>>>> > platform level without having to touch or look at the
>> business
>> >> >>>> data inside
>> >> >>>>> > we can have headers we can put tags into so that we can
>> ensure
>> >> >>>> this doesn’t
>> >> >>>>> > occur when we mirror. (actually links in to data lineage /
>> >> >> tracing
>> >> >>>> as again
>> >> >>>>> > we need to tag messages at a platform level) Examples are we
>> >> are
>> >> >>>> not
>> >> >>>>> > allowed Private customer details to leave Switzerland, yet
>> we
>> >> >> need
>> >> >>>> those
>> >> >>>>> > systems integrated.
>> >> >>>>> >
>> >> >>>>> > Lastly around mirroring we have a partionKey field, as the
>> key
>> >> >>>> used for
>> >> >>>>> > portioning logic != compaction key all the time but we want
>> to
>> >> >>>> preserve it
>> >> >>>>> > for when we mirror so that if source cluster partition count
>> >> !=
>> >> >>>> destination
>> >> >>>>> > cluster partition count we can honour the same partitioning
>> >> >> logic.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE End 2 End encryption
>> >> >>>>> >
>> >> >>>>> > As I believe mentioned just before, the solution you mention
>> >> just
>> >> >>>> doesn’t
>> >> >>>>> > cut the mustard these days with many regulators. An
>> operations
>> >> >>>> person with
>> >> >>>>> > access to the box should not be able to have access to the
>> >> data.
>> >> >>>> Many now
>> >> >>>>> > actually impose quite literally the implementation expected
>> >> being
>> >> >>>> end2end
>> >> >>>>> > encryption for certain data (Singapore for us is one that I
>> am
>> >> >>>> most aware
>> >> >>>>> > of). In fact we’re even now needing encrypt the data and
>> store
>> >> >> the
>> >> >>>> keys in
>> >> >>>>> > HSM modules.
>> >> >>>>> >
>> >> >>>>> > Likewise the performance penalty on encrypting decrypting as
>> >> you
>> >> >>>> produce
>> >> >>>>> > over wire, then again encrypt decrypt as the data is stored
>> on
>> >> >> the
>> >> >>>> brokers
>> >> >>>>> > disks and back again, then again encrypted and decrypted
>> back
>> >> >> over
>> >> >>>> the wire
>> >> >>>>> > each time for each consumer all adds up, ignoring this
>> >> doubling
>> >> >>>> with mirror
>> >> >>>>> > makers etc. simply encrypting the value once on write by the
>> >> >>>> client and
>> >> >>>>> > again decrypting on consume by the consumer is far more
>> >> >>>> performant, but
>> >> >>>>> > then the routing and platform meta data needs to be separate
>> >> >> (thus
>> >> >>>> headers)
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE Auditing:
>> >> >>>>> >
>> >> >>>>> > Our Auditing needs are:
>> >> >>>>> > Producer Id,
>> >> >>>>> > Origin Cluster Id that message first produced into
>> >> >>>>> > Origin AZ – agreed we can derive this if we have cluster id,
>> >> but
>> >> >>>> it makes
>> >> >>>>> > resolving this for audit reporting a lot easier.
>> >> >>>>> > Origin Region – agreed we can derive this if we have cluster
>> >> id,
>> >> >>>> but it
>> >> >>>>> > makes resolving this for audit reporting a lot easier.
>> >> >>>>> > Unique Message Identification (this is not the same as
>> >> >> transaction
>> >> >>>>> > tracing) – note offset and partition are not the same, as
>> >> when we
>> >> >>>> mirror or
>> >> >>>>> > have for what ever system failure duplicate send,
>> >> >>>>> > Custom Client wrapper version (where organizations have to
>> >> wrap
>> >> >>>> the kafka
>> >> >>>>> > client for added features) so we know what version of the
>> >> wrapper
>> >> >>>> is used
>> >> >>>>> > Producer IP address (in case of clients being in our vm/open
>> >> >> stack
>> >> >>>> infra
>> >> >>>>> > where they can move around, producer id will stay the same
>> but
>> >> >>>> this would
>> >> >>>>> > change)
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > RE Once and only once delivery case
>> >> >>>>> >
>> >> >>>>> > Using the same Message UUID for auditing we can achieve this
>> >> >> quite
>> >> >>>> simply.
>> >> >>>>> >
>> >> >>>>> > As per how some other brokers do this (cough qpid, artemis)
>> >> >>>> message uuid
>> >> >>>>> > are used to dedupe where message is sent and produced but
>> the
>> >> >>>> client didn’t
>> >> >>>>> > receive the ack, and there for replays the send, by having a
>> >> >>>> unique message
>> >> >>>>> > id per message, this can be filtered out, on consumers where
>> >> >>>> message
>> >> >>>>> > delivery may occur twice for what ever reasons a message
>> uuid
>> >> can
>> >> >>>> be used
>> >> >>>>> > to remove duplicates being deliverd , like wise we can do
>> >> this in
>> >> >>>> the
>> >> >>>>> > mirrormakers so if we detect a dupe message we can avoid
>> >> >>>> replicating it.
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > Cheers
>> >> >>>>> > Mike
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > On 02/12/2016, 22:09, "Jun Rao" <ju...@confluent.io> wrote:
>> >> >>>>> >
>> >> >>>>> > Since this KIP affects message format, wire protocol,
>> >> apis, I
>> >> >>>> think
>> >> >>>>> > it's
>> >> >>>>> > worth spending a bit more time to nail down the concrete
>> >> use
>> >> >>>> cases. It
>> >> >>>>> > would be bad if we add this feature, but when start
>> >> >>>> implementing it
>> >> >>>>> > for say
>> >> >>>>> > mirroring, we then realize that header is not the best
>> >> >>>> approach.
>> >> >>>>> > Initially,
>> >> >>>>> > I thought I was convinced of the use cases of headers
>> and
>> >> was
>> >> >>>> trying to
>> >> >>>>> > write down a few use cases to convince others. That's
>> >> when I
>> >> >>>> became
>> >> >>>>> > less
>> >> >>>>> > certain. For me to be convinced, I just want to see two
>> >> >> strong
>> >> >>>> use
>> >> >>>>> > cases
>> >> >>>>> > (instead of 10 maybe use cases) in the third-party
>> space.
>> >> The
>> >> >>>> reason is
>> >> >>>>> > that when we discussed the use cases within a company,
>> >> often
>> >> >>>> it ends
>> >> >>>>> > with
>> >> >>>>> > "we can't force everyone to use this standard since we
>> may
>> >> >>>> have to
>> >> >>>>> > integrate with third-party tools".
>> >> >>>>> >
>> >> >>>>> > At present, I am not sure why headers are useful for
>> >> things
>> >> >>>> like
>> >> >>>>> > schemaId
>> >> >>>>> > or encryption. In order to do anything useful to the
>> >> value,
>> >> >>>> one needs
>> >> >>>>> > to
>> >> >>>>> > know the schemaId or how data is encrypted, but header
>> is
>> >> >>>> optional.
>> >> >>>>> > But, I
>> >> >>>>> > can be convinced if someone (Radai, Sean, Todd?)
>> provides
>> >> >> more
>> >> >>>> details
>> >> >>>>> > on
>> >> >>>>> > the argument.
>> >> >>>>> >
>> >> >>>>> > I am not very sure header is the best approach for
>> >> mirroring
>> >> >>>> either. If
>> >> >>>>> > someone has thought about this more, I'd be happy to
>> hear.
>> >> >>>>> >
>> >> >>>>> > I can see the data lineage use case. I am just not sure
>> >> how
>> >> >>>> widely
>> >> >>>>> > applicable this is. If someone familiar with this space
>> >> can
>> >> >>>> justify
>> >> >>>>> > this is
>> >> >>>>> > a significant use case, say in the finance industry,
>> this
>> >> >>>> would be a
>> >> >>>>> > strong
>> >> >>>>> > use case.
>> >> >>>>> >
>> >> >>>>> > I can see the auditing use case. I am just not sure if a
>> >> >> native
>> >> >>>>> > producer id
>> >> >>>>> > solves that problem. If there are additional metadata
>> >> that's
>> >> >>>> worth
>> >> >>>>> > collecting but not covered by the producer id, that
>> would
>> >> >> make
>> >> >>>> this a
>> >> >>>>> > strong use case.
>> >> >>>>> >
>> >> >>>>> > Thanks,
>> >> >>>>> >
>> >> >>>>> > Jun
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > On Fri, Dec 2, 2016 at 1:41 PM, radai <
>> >> >>>> radai.rosenblatt@gmail.com>
>> >> >>>>> > wrote:
>> >> >>>>> >
>> >> >>>>> > > this KIP is about enabling headers, nothing more
>> nothing
>> >> >>>> less - so
>> >> >>>>> > no,
>> >> >>>>> > > broker-side use of headers is not in the KIP scope.
>> >> >>>>> > >
>> >> >>>>> > > obviously though, once you have headers potential use
>> >> cases
>> >> >>>> could
>> >> >>>>> > include
>> >> >>>>> > > broker-side header-aware interceptors (which would be
>> >> the
>> >> >>>> topic of
>> >> >>>>> > other
>> >> >>>>> > > future KIPs).
>> >> >>>>> > >
>> >> >>>>> > > a trivially clear use case (to me) would be using such
>> >> >>>> broker-side
>> >> >>>>> > > interceptors to enforce compliance with organizational
>> >> >>>> policies - it
>> >> >>>>> > would
>> >> >>>>> > > make our SREs lives much easier if instead of
>> >> retroactively
>> >> >>>>> > discovering
>> >> >>>>> > > "rogue" topics/users those messages would have been
>> >> >> rejected
>> >> >>>>> > up-front.
>> >> >>>>> > >
>> >> >>>>> > > the kafka broker code is lacking any such
>> extensibility
>> >> >>>> support
>> >> >>>>> > (beyond
>> >> >>>>> > > maybe authorizer) which is why these use cases were
>> left
>> >> >> out
>> >> >>>> of the
>> >> >>>>> > "case
>> >> >>>>> > > for headers" doc - broker extensibility is a separate
>> >> >>>> discussion.
>> >> >>>>> > >
>> >> >>>>> > > On Fri, Dec 2, 2016 at 12:59 PM, Gwen Shapira <
>> >> >>>> gwen@confluent.io>
>> >> >>>>> > wrote:
>> >> >>>>> > >
>> >> >>>>> > > > Woah, I wasn't aware this is something we'll do. It
>> >> >> wasn't
>> >> >>>> in the
>> >> >>>>> > KIP,
>> >> >>>>> > > > right?
>> >> >>>>> > > >
>> >> >>>>> > > > I guess we could do it the same way ACLs currently
>> >> work.
>> >> >>>>> > > > I had in mind something that will allow admins to
>> >> apply
>> >> >>>> rules to
>> >> >>>>> > the
>> >> >>>>> > > > new create/delete/config topic APIs. So Todd can
>> >> decide
>> >> >> to
>> >> >>>> reject
>> >> >>>>> > > > "create topic" requests that ask for more than 40
>> >> >>>> partitions, or
>> >> >>>>> > > > require exactly 3 replicas, or no more than 50GB
>> >> >> partition
>> >> >>>> size,
>> >> >>>>> > etc.
>> >> >>>>> > > >
>> >> >>>>> > > > ACLs were added a bit ad-hoc, if we are planning to
>> >> apply
>> >> >>>> more
>> >> >>>>> > rules
>> >> >>>>> > > > to requests (and I think we should), we may want a
>> bit
>> >> >>>> more generic
>> >> >>>>> > > > design around that.
>> >> >>>>> > > >
>> >> >>>>> > > > On Fri, Dec 2, 2016 at 7:16 AM, radai <
>> >> >>>> radai.rosenblatt@gmail.com>
>> >> >>>>> > > wrote:
>> >> >>>>> > > > > "wouldn't you be in the business of making sure
>> >> >> everyone
>> >> >>>> uses
>> >> >>>>> > them
>> >> >>>>> > > > > properly?"
>> >> >>>>> > > > >
>> >> >>>>> > > > > thats where a broker-side plugin would come handy
>> -
>> >> any
>> >> >>>> incoming
>> >> >>>>> > > message
>> >> >>>>> > > > > that does not conform to org policy (read - does
>> not
>> >> >>>> have the
>> >> >>>>> > proper
>> >> >>>>> > > > > headers) gets thrown out (with an error returned
>> to
>> >> >> user)
>> >> >>>>> > > > >
>> >> >>>>> > > > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <
>> >> >>>> tpalino@gmail.com>
>> >> >>>>> > wrote:
>> >> >>>>> > > > >
>> >> >>>>> > > > >> Come on, I’ve done at least 2 talks on this one
>> :)
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> Producing counts to a topic is part of it, but
>> >> that’s
>> >> >>>> only
>> >> >>>>> > part. So
>> >> >>>>> > > you
>> >> >>>>> > > > >> count you have 100 messages in topic A. When you
>> >> >> mirror
>> >> >>>> topic A
>> >> >>>>> > to
>> >> >>>>> > > > another
>> >> >>>>> > > > >> cluster, you have 99 messages. Where was your
>> >> problem?
>> >> >>>> Or
>> >> >>>>> > worse, you
>> >> >>>>> > > > have
>> >> >>>>> > > > >> 100 messages, but one producer duplicated
>> messages
>> >> and
>> >> >>>> another
>> >> >>>>> > one
>> >> >>>>> > > lost
>> >> >>>>> > > > >> messages. You need details about where the
>> message
>> >> >> came
>> >> >>>> from in
>> >> >>>>> > order
>> >> >>>>> > > to
>> >> >>>>> > > > >> pinpoint problems when they happen. Source
>> producer
>> >> >>>> info, where
>> >> >>>>> > it was
>> >> >>>>> > > > >> produced into your infrastructure, and when it
>> was
>> >> >>>> produced.
>> >> >>>>> > This
>> >> >>>>> > > > requires
>> >> >>>>> > > > >> you to add the information to the message.
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> And yes, you still need to maintain your clients.
>> >> So
>> >> >>>> maybe my
>> >> >>>>> > original
>> >> >>>>> > > > >> example was not the best. My thoughts on not
>> >> wanting
>> >> >> to
>> >> >>>> be
>> >> >>>>> > responsible
>> >> >>>>> > > > for
>> >> >>>>> > > > >> message formats stands, because that’s very much
>> >> >>>> separate from
>> >> >>>>> > the
>> >> >>>>> > > > client.
>> >> >>>>> > > > >> As you know, we have our own internal client
>> >> library
>> >> >>>> that can
>> >> >>>>> > insert
>> >> >>>>> > > the
>> >> >>>>> > > > >> right headers, and right now inserts the right
>> >> audit
>> >> >>>>> > information into
>> >> >>>>> > > > the
>> >> >>>>> > > > >> message fields. If they exist, and assuming the
>> >> >> message
>> >> >>>> is Avro
>> >> >>>>> > > encoded.
>> >> >>>>> > > > >> What if someone wants to use JSON instead for a
>> >> good
>> >> >>>> reason?
>> >> >>>>> > What if
>> >> >>>>> > > > user X
>> >> >>>>> > > > >> wants to encrypt messages, but user Y does not?
>> >> >>>> Maintaining the
>> >> >>>>> > client
>> >> >>>>> > > > >> library is still much easier than maintaining the
>> >> >>>> message
>> >> >>>>> > formats.
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> -Todd
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <
>> >> >>>> gwen@confluent.io
>> >> >>>>> > >
>> >> >>>>> > > wrote:
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> > Based on your last sentence, consider me
>> >> convinced
>> >> >> :)
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > I get why headers are critical for Mirroring
>> (you
>> >> >>>> need tags to
>> >> >>>>> > > prevent
>> >> >>>>> > > > >> > loops and sometimes to route messages to the
>> >> correct
>> >> >>>>> > destination).
>> >> >>>>> > > > >> > But why do you need headers to audit? We are
>> >> >> auditing
>> >> >>>> by
>> >> >>>>> > producing
>> >> >>>>> > > > >> > counts to a side topic (and I was under the
>> >> >>>> impression you do
>> >> >>>>> > the
>> >> >>>>> > > > >> > same), so we never need to modify the message.
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > Another thing - after we added headers,
>> wouldn't
>> >> you
>> >> >>>> be in the
>> >> >>>>> > > > >> > business of making sure everyone uses them
>> >> properly?
>> >> >>>> Making
>> >> >>>>> > sure
>> >> >>>>> > > > >> > everyone includes the right headers you need,
>> not
>> >> >>>> using the
>> >> >>>>> > header
>> >> >>>>> > > > >> > names you intend to use, etc. I don't think the
>> >> >>>> "policing"
>> >> >>>>> > business
>> >> >>>>> > > > >> > will ever go away.
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <
>> >> >>>>> > tpalino@gmail.com>
>> >> >>>>> > > > wrote:
>> >> >>>>> > > > >> > > Got it. As an ops guy, I'm not very happy
>> with
>> >> the
>> >> >>>>> > workaround.
>> >> >>>>> > > Avro
>> >> >>>>> > > > >> means
>> >> >>>>> > > > >> > > that I have to be concerned with the format
>> of
>> >> the
>> >> >>>> messages
>> >> >>>>> > in
>> >> >>>>> > > > order to
>> >> >>>>> > > > >> > run
>> >> >>>>> > > > >> > > the infrastructure (audit, mirroring, etc.).
>> >> That
>> >> >>>> means
>> >> >>>>> > that I
>> >> >>>>> > > have
>> >> >>>>> > > > to
>> >> >>>>> > > > >> > > handle the schemas, and I have to enforce
>> rules
>> >> >>>> about good
>> >> >>>>> > > formats.
>> >> >>>>> > > > >> This
>> >> >>>>> > > > >> > is
>> >> >>>>> > > > >> > > not something I want to be in the business
>> of,
>> >> >>>> because I
>> >> >>>>> > should be
>> >> >>>>> > > > able
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > > run a service infrastructure without needing
>> >> to be
>> >> >>>> in the
>> >> >>>>> > weeds of
>> >> >>>>> > > > >> > dealing
>> >> >>>>> > > > >> > > with customer data formats.
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > > Trust me, a sizable portion of my support
>> time
>> >> is
>> >> >>>> spent
>> >> >>>>> > dealing
>> >> >>>>> > > with
>> >> >>>>> > > > >> > schema
>> >> >>>>> > > > >> > > issues. I really would like to get away from
>> >> that.
>> >> >>>> Maybe
>> >> >>>>> > I'd have
>> >> >>>>> > > > more
>> >> >>>>> > > > >> > time
>> >> >>>>> > > > >> > > for other hobbies. Like writing. ;)
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > > -Todd
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <
>> >> >>>>> > gwen@confluent.io>
>> >> >>>>> > > > wrote:
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > >> I'm pretty satisfied with the current
>> >> workarounds
>> >> >>>> (Avro
>> >> >>>>> > container
>> >> >>>>> > > > >> > >> format), so I'm not too excited about the
>> >> extra
>> >> >>>> work
>> >> >>>>> > required to
>> >> >>>>> > > do
>> >> >>>>> > > > >> > >> headers in Kafka. I absolutely don't mind it
>> >> if
>> >> >>>> you do
>> >> >>>>> > it...
>> >> >>>>> > > > >> > >> I think the Apache convention for "good
>> idea,
>> >> but
>> >> >>>> not
>> >> >>>>> > willing to
>> >> >>>>> > > > put
>> >> >>>>> > > > >> > >> any work toward it" is +0.5? anyway, that's
>> >> what
>> >> >> I
>> >> >>>> was
>> >> >>>>> > trying to
>> >> >>>>> > > > >> > >> convey :)
>> >> >>>>> > > > >> > >>
>> >> >>>>> > > > >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino
>> <
>> >> >>>>> > tpalino@gmail.com>
>> >> >>>>> > > > >> wrote:
>> >> >>>>> > > > >> > >> > Well I guess my question for you, then, is
>> >> what
>> >> >>>> is
>> >> >>>>> > holding you
>> >> >>>>> > > > back
>> >> >>>>> > > > >> > from
>> >> >>>>> > > > >> > >> > full support for headers? What’s the bit
>> >> that
>> >> >>>> you’re
>> >> >>>>> > missing
>> >> >>>>> > > that
>> >> >>>>> > > > >> has
>> >> >>>>> > > > >> > you
>> >> >>>>> > > > >> > >> > under a full +1?
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> > -Todd
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen
>> >> Shapira <
>> >> >>>>> > > gwen@confluent.io>
>> >> >>>>> > > > >> > wrote:
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> >> I know why people who support headers
>> >> support
>> >> >>>> them, and
>> >> >>>>> > I've
>> >> >>>>> > > > seen
>> >> >>>>> > > > >> > what
>> >> >>>>> > > > >> > >> >> the discussion is like.
>> >> >>>>> > > > >> > >> >>
>> >> >>>>> > > > >> > >> >> This is why I'm asking people who are
>> >> against
>> >> >>>> headers
>> >> >>>>> > > > (especially
>> >> >>>>> > > > >> > >> >> committers) what will make them change
>> >> their
>> >> >>>> mind - so
>> >> >>>>> > we can
>> >> >>>>> > > > get
>> >> >>>>> > > > >> > this
>> >> >>>>> > > > >> > >> >> part over one way or another.
>> >> >>>>> > > > >> > >> >>
>> >> >>>>> > > > >> > >> >> If I sound frustrated it is not at Radai,
>> >> Jun
>> >> >>>> or you
>> >> >>>>> > (Todd)...
>> >> >>>>> > > > I am
>> >> >>>>> > > > >> > >> >> just looking for something concrete we
>> can
>> >> do
>> >> >>>> to move
>> >> >>>>> > the
>> >> >>>>> > > > >> discussion
>> >> >>>>> > > > >> > >> >> along to the yummy design details (which
>> is
>> >> >> the
>> >> >>>>> > argument I
>> >> >>>>> > > > really
>> >> >>>>> > > > >> am
>> >> >>>>> > > > >> > >> >> looking forward to).
>> >> >>>>> > > > >> > >> >>
>> >> >>>>> > > > >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd
>> >> Palino <
>> >> >>>>> > > tpalino@gmail.com>
>> >> >>>>> > > > >> > wrote:
>> >> >>>>> > > > >> > >> >> > So, Gwen, to your question (even though
>> >> I’m
>> >> >>>> not a
>> >> >>>>> > > > committer)...
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > I have always been a strong supporter
>> of
>> >> >>>> introducing
>> >> >>>>> > the
>> >> >>>>> > > > concept
>> >> >>>>> > > > >> > of an
>> >> >>>>> > > > >> > >> >> > envelope to messages, which headers
>> >> >>>> accomplishes. The
>> >> >>>>> > > message
>> >> >>>>> > > > key
>> >> >>>>> > > > >> > is
>> >> >>>>> > > > >> > >> >> > already an example of a piece of
>> envelope
>> >> >>>>> > information. By
>> >> >>>>> > > > >> > providing a
>> >> >>>>> > > > >> > >> >> means
>> >> >>>>> > > > >> > >> >> > to do this within Kafka itself, and not
>> >> >>>> relying on
>> >> >>>>> > use-case
>> >> >>>>> > > > >> > specific
>> >> >>>>> > > > >> > >> >> > implementations, you make it much
>> easier
>> >> for
>> >> >>>>> > components to
>> >> >>>>> > > > >> > >> interoperate.
>> >> >>>>> > > > >> > >> >> It
>> >> >>>>> > > > >> > >> >> > simplifies development of all these
>> >> things
>> >> >>>> (message
>> >> >>>>> > routing,
>> >> >>>>> > > > >> > auditing,
>> >> >>>>> > > > >> > >> >> > encryption, etc.) because each one does
>> >> not
>> >> >>>> have to
>> >> >>>>> > reinvent
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> wheel.
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > It also makes it much easier from a
>> >> client
>> >> >>>> point of
>> >> >>>>> > view if
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> headers
>> >> >>>>> > > > >> > >> >> are
>> >> >>>>> > > > >> > >> >> > defined as part of the protocol and/or
>> >> >>>> message format
>> >> >>>>> > in
>> >> >>>>> > > > general
>> >> >>>>> > > > >> > >> because
>> >> >>>>> > > > >> > >> >> > you can easily produce and consume
>> >> messages
>> >> >>>> without
>> >> >>>>> > having
>> >> >>>>> > > to
>> >> >>>>> > > > >> take
>> >> >>>>> > > > >> > >> into
>> >> >>>>> > > > >> > >> >> > account specific cases. For example, I
>> >> want
>> >> >>>> to route
>> >> >>>>> > > messages,
>> >> >>>>> > > > >> but
>> >> >>>>> > > > >> > >> >> client A
>> >> >>>>> > > > >> > >> >> > doesn’t support the way audit
>> implemented
>> >> >>>> headers, and
>> >> >>>>> > > client
>> >> >>>>> > > > B
>> >> >>>>> > > > >> > >> doesn’t
>> >> >>>>> > > > >> > >> >> > support the way encryption or routing
>> >> >>>> implemented
>> >> >>>>> > headers,
>> >> >>>>> > > so
>> >> >>>>> > > > now
>> >> >>>>> > > > >> > my
>> >> >>>>> > > > >> > >> >> > application has to create some really
>> >> >> fragile
>> >> >>>> (my
>> >> >>>>> > > autocorrect
>> >> >>>>> > > > >> just
>> >> >>>>> > > > >> > >> tried
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> > make that “tragic”, which is probably
>> >> >>>> appropriate
>> >> >>>>> > too) code
>> >> >>>>> > > to
>> >> >>>>> > > > >> > strip
>> >> >>>>> > > > >> > >> >> > everything off, rather than just
>> >> consuming
>> >> >> the
>> >> >>>>> > messages,
>> >> >>>>> > > > picking
>> >> >>>>> > > > >> > out
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> 1
>> >> >>>>> > > > >> > >> >> > or 2 headers it’s interested in, and
>> >> >>>> performing its
>> >> >>>>> > > function.
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > Honestly, this discussion has been
>> going
>> >> on
>> >> >>>> for a
>> >> >>>>> > long time,
>> >> >>>>> > > > and
>> >> >>>>> > > > >> > it’s
>> >> >>>>> > > > >> > >> >> > always “Oh, you came up with 2 use
>> cases,
>> >> >> and
>> >> >>>> yeah,
>> >> >>>>> > those
>> >> >>>>> > > use
>> >> >>>>> > > > >> cases
>> >> >>>>> > > > >> > >> are
>> >> >>>>> > > > >> > >> >> > real things that someone would want to
>> >> do.
>> >> >>>> Here’s an
>> >> >>>>> > > alternate
>> >> >>>>> > > > >> way
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > >> >> > implement them so let’s not do
>> headers.”
>> >> If
>> >> >>>> we have a
>> >> >>>>> > few
>> >> >>>>> > > use
>> >> >>>>> > > > >> cases
>> >> >>>>> > > > >> > >> that
>> >> >>>>> > > > >> > >> >> we
>> >> >>>>> > > > >> > >> >> > actually came up with, you can be sure
>> >> that
>> >> >>>> over the
>> >> >>>>> > next
>> >> >>>>> > > year
>> >> >>>>> > > > >> > >> there’s a
>> >> >>>>> > > > >> > >> >> > dozen others that we didn’t think of
>> that
>> >> >>>> someone
>> >> >>>>> > would like
>> >> >>>>> > > > to
>> >> >>>>> > > > >> > do. I
>> >> >>>>> > > > >> > >> >> > really think it’s time to stop
>> rehashing
>> >> >> this
>> >> >>>>> > discussion and
>> >> >>>>> > > > >> > instead
>> >> >>>>> > > > >> > >> >> focus
>> >> >>>>> > > > >> > >> >> > on a workable standard that we can
>> adopt.
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > -Todd
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd
>> >> Palino
>> >> >> <
>> >> >>>>> > > > tpalino@gmail.com>
>> >> >>>>> > > > >> > >> wrote:
>> >> >>>>> > > > >> > >> >> >
>> >> >>>>> > > > >> > >> >> >> C. per message encryption
>> >> >>>>> > > > >> > >> >> >>> One drawback of this approach is that
>> >> this
>> >> >>>>> > significantly
>> >> >>>>> > > > reduce
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> effectiveness of compression, which
>> >> >> happens
>> >> >>>> on a
>> >> >>>>> > set of
>> >> >>>>> > > > >> > serialized
>> >> >>>>> > > > >> > >> >> >>> messages. An alternative is to enable
>> >> SSL
>> >> >>>> for wire
>> >> >>>>> > > > encryption
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> rely
>> >> >>>>> > > > >> > >> >> on
>> >> >>>>> > > > >> > >> >> >>> the storage system (e.g. LUKS) for at
>> >> rest
>> >> >>>>> > encryption.
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >> Jun, this is not sufficient. While
>> this
>> >> >> does
>> >> >>>> cover
>> >> >>>>> > the case
>> >> >>>>> > > > of
>> >> >>>>> > > > >> > >> removing
>> >> >>>>> > > > >> > >> >> a
>> >> >>>>> > > > >> > >> >> >> drive from the system, it will not
>> >> satisfy
>> >> >>>> most
>> >> >>>>> > compliance
>> >> >>>>> > > > >> > >> requirements
>> >> >>>>> > > > >> > >> >> for
>> >> >>>>> > > > >> > >> >> >> encryption of data as whoever has
>> >> access to
>> >> >>>> the
>> >> >>>>> > broker
>> >> >>>>> > > itself
>> >> >>>>> > > > >> > still
>> >> >>>>> > > > >> > >> has
>> >> >>>>> > > > >> > >> >> >> access to the unencrypted data. For
>> >> >>>> end-to-end
>> >> >>>>> > encryption
>> >> >>>>> > > you
>> >> >>>>> > > > >> > need to
>> >> >>>>> > > > >> > >> >> >> encrypt at the producer, before it
>> >> enters
>> >> >> the
>> >> >>>>> > system, and
>> >> >>>>> > > > >> decrypt
>> >> >>>>> > > > >> > at
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> >> consumer, after it exits the system.
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >> -Todd
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai
>> <
>> >> >>>>> > > > >> radai.rosenblatt@gmail.com
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > >> >> wrote:
>> >> >>>>> > > > >> > >> >> >>
>> >> >>>>> > > > >> > >> >> >>> another big plus of headers in the
>> >> >> protocol
>> >> >>>> is that
>> >> >>>>> > it
>> >> >>>>> > > would
>> >> >>>>> > > > >> > enable
>> >> >>>>> > > > >> > >> >> rapid
>> >> >>>>> > > > >> > >> >> >>> iteration on ideas outside of core
>> >> kafka
>> >> >>>> and would
>> >> >>>>> > reduce
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> number of
>> >> >>>>> > > > >> > >> >> >>> future wire format changes required.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> a lot of what is currently a KIP
>> >> >> represents
>> >> >>>> use
>> >> >>>>> > cases that
>> >> >>>>> > > > are
>> >> >>>>> > > > >> > not
>> >> >>>>> > > > >> > >> 100%
>> >> >>>>> > > > >> > >> >> >>> relevant to all users, and some of
>> them
>> >> >>>> require
>> >> >>>>> > rather
>> >> >>>>> > > > invasive
>> >> >>>>> > > > >> > wire
>> >> >>>>> > > > >> > >> >> >>> protocol changes. a thing a good
>> recent
>> >> >>>> example of
>> >> >>>>> > this is
>> >> >>>>> > > > >> > kip-98.
>> >> >>>>> > > > >> > >> >> >>> tx-utilizing traffic is expected to
>> be
>> >> a
>> >> >>>> very small
>> >> >>>>> > > > fraction of
>> >> >>>>> > > > >> > >> total
>> >> >>>>> > > > >> > >> >> >>> traffic and yet the changes are
>> >> invasive.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> every such wire format change
>> >> translates
>> >> >>>> into
>> >> >>>>> > painful and
>> >> >>>>> > > > slow
>> >> >>>>> > > > >> > >> >> adoption of
>> >> >>>>> > > > >> > >> >> >>> new versions.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> i think a lot of functionality
>> >> currently
>> >> >> in
>> >> >>>> KIPs
>> >> >>>>> > could be
>> >> >>>>> > > > "spun
>> >> >>>>> > > > >> > out"
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> implemented as opt-in plugins
>> >> transmitting
>> >> >>>> data over
>> >> >>>>> > > > headers.
>> >> >>>>> > > > >> > this
>> >> >>>>> > > > >> > >> >> would
>> >> >>>>> > > > >> > >> >> >>> keep the core wire format stable(r),
>> >> core
>> >> >>>> codebase
>> >> >>>>> > > smaller,
>> >> >>>>> > > > and
>> >> >>>>> > > > >> > >> avoid
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> "burden of proof" thats sometimes
>> >> required
>> >> >>>> to prove
>> >> >>>>> > a
>> >> >>>>> > > > certain
>> >> >>>>> > > > >> > >> feature
>> >> >>>>> > > > >> > >> >> is
>> >> >>>>> > > > >> > >> >> >>> useful enough for a wide-enough
>> >> audience
>> >> >> to
>> >> >>>> warrant
>> >> >>>>> > a wire
>> >> >>>>> > > > >> format
>> >> >>>>> > > > >> > >> >> change
>> >> >>>>> > > > >> > >> >> >>> and code complexity additions.
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> (to be clear - kip-98 goes beyond
>> >> "mere"
>> >> >>>> wire format
>> >> >>>>> > > changes
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > im
>> >> >>>>> > > > >> > >> not
>> >> >>>>> > > > >> > >> >> >>> saying it could have been completely
>> >> done
>> >> >>>> with
>> >> >>>>> > headers,
>> >> >>>>> > > but
>> >> >>>>> > > > >> > >> >> exactly-once
>> >> >>>>> > > > >> > >> >> >>> delivery certainly could)
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen
>> >> >>>> Shapira <
>> >> >>>>> > > > >> gwen@confluent.io
>> >> >>>>> > > > >> > >
>> >> >>>>> > > > >> > >> >> wrote:
>> >> >>>>> > > > >> > >> >> >>>
>> >> >>>>> > > > >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM,
>> >> radai <
>> >> >>>>> > > > >> > >> radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> wrote:
>> >> >>>>> > > > >> > >> >> >>> > > "For use cases within an
>> >> organization,
>> >> >>>> one could
>> >> >>>>> > > always
>> >> >>>>> > > > use
>> >> >>>>> > > > >> > >> other
>> >> >>>>> > > > >> > >> >> >>> > > approaches such as company-wise
>> >> >>>> containers"
>> >> >>>>> > > > >> > >> >> >>> > > this is what linkedin has
>> >> >> traditionally
>> >> >>>> done
>> >> >>>>> > but there
>> >> >>>>> > > > are
>> >> >>>>> > > > >> > now
>> >> >>>>> > > > >> > >> >> cases
>> >> >>>>> > > > >> > >> >> >>> > (read
>> >> >>>>> > > > >> > >> >> >>> > > - topics) where this is not
>> >> >> acceptable.
>> >> >>>> this
>> >> >>>>> > makes
>> >> >>>>> > > > headers
>> >> >>>>> > > > >> > >> useful
>> >> >>>>> > > > >> > >> >> even
>> >> >>>>> > > > >> > >> >> >>> > > within single orgs for cases
>> where
>> >> >>>>> > > > one-container-fits-all
>> >> >>>>> > > > >> > cannot
>> >> >>>>> > > > >> > >> >> >>> apply.
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > > as for the particular use cases
>> >> >> listed,
>> >> >>>> i dont
>> >> >>>>> > want
>> >> >>>>> > > > this to
>> >> >>>>> > > > >> > >> devolve
>> >> >>>>> > > > >> > >> >> >>> to a
>> >> >>>>> > > > >> > >> >> >>> > > discussion of particular use
>> cases
>> >> - i
>> >> >>>> think its
>> >> >>>>> > > enough
>> >> >>>>> > > > >> that
>> >> >>>>> > > > >> > >> some
>> >> >>>>> > > > >> > >> >> of
>> >> >>>>> > > > >> > >> >> >>> them
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I think a main point of contention
>> is
>> >> >>>> that: We
>> >> >>>>> > > identified
>> >> >>>>> > > > few
>> >> >>>>> > > > >> > >> >> >>> > use-cases where headers are useful,
>> >> do
>> >> >> we
>> >> >>>> want
>> >> >>>>> > Kafka to
>> >> >>>>> > > > be a
>> >> >>>>> > > > >> > >> system
>> >> >>>>> > > > >> > >> >> >>> > that supports those use-cases?
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > For example, Jun said:
>> >> >>>>> > > > >> > >> >> >>> > "Not sure how widely useful
>> >> record-level
>> >> >>>> lineage
>> >> >>>>> > is
>> >> >>>>> > > though
>> >> >>>>> > > > >> > since
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> >>> > overhead could
>> >> >>>>> > > > >> > >> >> >>> > be significant."
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > We know NiFi supports record level
>> >> >>>> lineage. I
>> >> >>>>> > don't
>> >> >>>>> > > think
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > was
>> >> >>>>> > > > >> > >> >> >>> > developed for lols, I think it is
>> >> safe
>> >> >> to
>> >> >>>> assume
>> >> >>>>> > that
>> >> >>>>> > > the
>> >> >>>>> > > > NSA
>> >> >>>>> > > > >> > >> needed
>> >> >>>>> > > > >> > >> >> >>> > that functionality. We also know
>> that
>> >> >>>> certain
>> >> >>>>> > financial
>> >> >>>>> > > > >> > institutes
>> >> >>>>> > > > >> > >> >> >>> > need to track tampering with
>> records
>> >> at
>> >> >> a
>> >> >>>> record
>> >> >>>>> > level
>> >> >>>>> > > and
>> >> >>>>> > > > >> > there
>> >> >>>>> > > > >> > >> are
>> >> >>>>> > > > >> > >> >> >>> > federal regulations that absolutely
>> >> >>>> require
>> >> >>>>> > this. They
>> >> >>>>> > > > also
>> >> >>>>> > > > >> > need
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> >>> > prove that routing apps that
>> >> "touches"
>> >> >> the
>> >> >>>>> > messages and
>> >> >>>>> > > > >> either
>> >> >>>>> > > > >> > >> reads
>> >> >>>>> > > > >> > >> >> >>> > or updates headers couldn't have
>> >> >> possibly
>> >> >>>>> > modified the
>> >> >>>>> > > > >> payload
>> >> >>>>> > > > >> > >> >> itself.
>> >> >>>>> > > > >> > >> >> >>> > They use record level encryption to
>> >> do
>> >> >>>> that -
>> >> >>>>> > apps can
>> >> >>>>> > > > read
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> >> >>> > (sometimes) modify headers but
>> can't
>> >> >>>> touch the
>> >> >>>>> > payload.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > We can totally say "those are
>> corner
>> >> >>>> cases and
>> >> >>>>> > not worth
>> >> >>>>> > > > >> adding
>> >> >>>>> > > > >> > >> >> >>> > headers to Kafka for", they should
>> >> use a
>> >> >>>> different
>> >> >>>>> > > pubsub
>> >> >>>>> > > > >> > message
>> >> >>>>> > > > >> > >> for
>> >> >>>>> > > > >> > >> >> >>> > that (Nifi or one of the other 1000
>> >> that
>> >> >>>> cater
>> >> >>>>> > > > specifically
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> > financial industry).
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > But this gets us into a catch 22:
>> >> >>>>> > > > >> > >> >> >>> > If we discuss a specific use-case,
>> >> >>>> someone can
>> >> >>>>> > always
>> >> >>>>> > > say
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > isn't
>> >> >>>>> > > > >> > >> >> >>> > interesting enough for Kafka. If we
>> >> >>>> discuss more
>> >> >>>>> > general
>> >> >>>>> > > > >> > trends,
>> >> >>>>> > > > >> > >> >> >>> > others can say "well, we are not
>> sure
>> >> >> any
>> >> >>>> of them
>> >> >>>>> > really
>> >> >>>>> > > > >> needs
>> >> >>>>> > > > >> > >> >> headers
>> >> >>>>> > > > >> > >> >> >>> > specifically. This is just hand
>> >> waving
>> >> >>>> and not
>> >> >>>>> > > > interesting.".
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I think discussing use-cases in
>> >> >> specifics
>> >> >>>> is super
>> >> >>>>> > > > important
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> >> decide
>> >> >>>>> > > > >> > >> >> >>> > implementation details for headers
>> >> (my
>> >> >>>> use-cases
>> >> >>>>> > lean
>> >> >>>>> > > > toward
>> >> >>>>> > > > >> > >> >> numerical
>> >> >>>>> > > > >> > >> >> >>> > keys with namespaces and object
>> >> values,
>> >> >>>> others
>> >> >>>>> > differ),
>> >> >>>>> > > > but I
>> >> >>>>> > > > >> > >> think
>> >> >>>>> > > > >> > >> >> we
>> >> >>>>> > > > >> > >> >> >>> > need to answer the general "Are we
>> >> going
>> >> >>>> to have
>> >> >>>>> > > headers"
>> >> >>>>> > > > >> > question
>> >> >>>>> > > > >> > >> >> >>> > first.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I'd love to hear from the other
>> >> >>>> committers in the
>> >> >>>>> > > > discussion:
>> >> >>>>> > > > >> > >> >> >>> > What would it take to convince you
>> >> that
>> >> >>>> headers
>> >> >>>>> > in Kafka
>> >> >>>>> > > > are
>> >> >>>>> > > > >> a
>> >> >>>>> > > > >> > >> good
>> >> >>>>> > > > >> > >> >> >>> > idea in general, so we can move
>> ahead
>> >> >> and
>> >> >>>> try to
>> >> >>>>> > agree
>> >> >>>>> > > on
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> >> details?
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > I feel like we keep moving the goal
>> >> >> posts
>> >> >>>> and
>> >> >>>>> > this is
>> >> >>>>> > > > truly
>> >> >>>>> > > > >> > >> >> exhausting.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > For the record, I mildly support
>> >> adding
>> >> >>>> headers
>> >> >>>>> > to Kafka
>> >> >>>>> > > > >> > (+0.5?).
>> >> >>>>> > > > >> > >> >> >>> > The community can continue to find
>> >> >>>> workarounds to
>> >> >>>>> > the
>> >> >>>>> > > > issue
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> there
>> >> >>>>> > > > >> > >> >> >>> > are some benefits to keeping the
>> >> message
>> >> >>>> format
>> >> >>>>> > and
>> >> >>>>> > > > clients
>> >> >>>>> > > > >> > >> simpler.
>> >> >>>>> > > > >> > >> >> >>> > But I see the usefulness of headers
>> >> to
>> >> >>>> many
>> >> >>>>> > use-cases
>> >> >>>>> > > and
>> >> >>>>> > > > if
>> >> >>>>> > > > >> we
>> >> >>>>> > > > >> > >> can
>> >> >>>>> > > > >> > >> >> >>> > find a good and generally useful
>> way
>> >> to
>> >> >>>> add it to
>> >> >>>>> > Kafka,
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > will
>> >> >>>>> > > > >> > >> make
>> >> >>>>> > > > >> > >> >> >>> > Kafka easier to use for many -
>> worthy
>> >> >>>> goal in my
>> >> >>>>> > eyes.
>> >> >>>>> > > > >> > >> >> >>> >
>> >> >>>>> > > > >> > >> >> >>> > > are interesting/feasible, but:
>> >> >>>>> > > > >> > >> >> >>> > > A+B. i think there are use cases
>> >> for
>> >> >>>> polyglot
>> >> >>>>> > topics.
>> >> >>>>> > > > >> > >> especially if
>> >> >>>>> > > > >> > >> >> >>> kafka
>> >> >>>>> > > > >> > >> >> >>> > > is being used to "trunk"
>> something
>> >> >> else.
>> >> >>>>> > > > >> > >> >> >>> > > D. multiple topics would make it
>> >> >> harder
>> >> >>>> to write
>> >> >>>>> > > > portable
>> >> >>>>> > > > >> > >> consumer
>> >> >>>>> > > > >> > >> >> >>> code.
>> >> >>>>> > > > >> > >> >> >>> > > partition remapping would mess
>> with
>> >> >>>> locality of
>> >> >>>>> > > > consumption
>> >> >>>>> > > > >> > >> >> >>> guarantees.
>> >> >>>>> > > > >> > >> >> >>> > > E+F. a use case I see for
>> >> >>>> lineage/metadata is
>> >> >>>>> > > > >> > >> billing/chargeback.
>> >> >>>>> > > > >> > >> >> for
>> >> >>>>> > > > >> > >> >> >>> > that
>> >> >>>>> > > > >> > >> >> >>> > > use case it is not enough to
>> simply
>> >> >>>> record the
>> >> >>>>> > point
>> >> >>>>> > > of
>> >> >>>>> > > > >> > origin,
>> >> >>>>> > > > >> > >> but
>> >> >>>>> > > > >> > >> >> >>> every
>> >> >>>>> > > > >> > >> >> >>> > > replication stop (think mirror
>> >> maker)
>> >> >>>> must also
>> >> >>>>> > add a
>> >> >>>>> > > > >> record
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > >> >> form a
>> >> >>>>> > > > >> > >> >> >>> > > "transit log".
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > > as for stream processing on top
>> of
>> >> >>>> kafka - i
>> >> >>>>> > know
>> >> >>>>> > > samza
>> >> >>>>> > > > >> has a
>> >> >>>>> > > > >> > >> >> metadata
>> >> >>>>> > > > >> > >> >> >>> > map
>> >> >>>>> > > > >> > >> >> >>> > > which they carry around in
>> >> addition to
>> >> >>>> user
>> >> >>>>> > values.
>> >> >>>>> > > > headers
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> >>> > perfect
>> >> >>>>> > > > >> > >> >> >>> > > fit for these things.
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM,
>> >> Jun
>> >> >>>> Rao <
>> >> >>>>> > > > jun@confluent.io
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >
>> >> >>>>> > > > >> > >> >> >>> > >> Hi, Michael,
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> In order to answer the first two
>> >> >>>> questions, it
>> >> >>>>> > would
>> >> >>>>> > > be
>> >> >>>>> > > > >> > helpful
>> >> >>>>> > > > >> > >> >> if we
>> >> >>>>> > > > >> > >> >> >>> > could
>> >> >>>>> > > > >> > >> >> >>> > >> identify 1 or 2 strong use cases
>> >> for
>> >> >>>> headers
>> >> >>>>> > in the
>> >> >>>>> > > > space
>> >> >>>>> > > > >> > for
>> >> >>>>> > > > >> > >> >> >>> > third-party
>> >> >>>>> > > > >> > >> >> >>> > >> vendors. For use cases within an
>> >> >>>> organization,
>> >> >>>>> > one
>> >> >>>>> > > > could
>> >> >>>>> > > > >> > always
>> >> >>>>> > > > >> > >> >> use
>> >> >>>>> > > > >> > >> >> >>> > other
>> >> >>>>> > > > >> > >> >> >>> > >> approaches such as company-wise
>> >> >>>> containers to
>> >> >>>>> > get
>> >> >>>>> > > > around
>> >> >>>>> > > > >> w/o
>> >> >>>>> > > > >> > >> >> >>> headers. I
>> >> >>>>> > > > >> > >> >> >>> > >> went through the use cases in
>> the
>> >> KIP
>> >> >>>> and in
>> >> >>>>> > Radai's
>> >> >>>>> > > > wiki
>> >> >>>>> > > > >> (
>> >> >>>>> > > > >> > >> >> >>> > >> https://cwiki.apache.org/confl
>> >> >>>>> > uence/display/KAFKA/A+
>> >> >>>>> > > > >> > >> >> >>> > Case+for+Kafka+Headers
>> >> >>>>> > > > >> > >> >> >>> > >> ).
>> >> >>>>> > > > >> > >> >> >>> > >> The following are the ones that
>> >> that
>> >> >> I
>> >> >>>>> > understand and
>> >> >>>>> > > > >> could
>> >> >>>>> > > > >> > be
>> >> >>>>> > > > >> > >> in
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> third-party use case category.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> A. content-type
>> >> >>>>> > > > >> > >> >> >>> > >> It seems that in general,
>> >> >> content-type
>> >> >>>> should
>> >> >>>>> > be set
>> >> >>>>> > > at
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> topic
>> >> >>>>> > > > >> > >> >> >>> level.
>> >> >>>>> > > > >> > >> >> >>> > >> Not sure if mixing messages with
>> >> >>>> different
>> >> >>>>> > content
>> >> >>>>> > > > types
>> >> >>>>> > > > >> > >> should be
>> >> >>>>> > > > >> > >> >> >>> > >> encouraged.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> B. schema id
>> >> >>>>> > > > >> > >> >> >>> > >> Since the value is mostly
>> useless
>> >> >>>> without
>> >> >>>>> > schema id,
>> >> >>>>> > > it
>> >> >>>>> > > > >> > seems
>> >> >>>>> > > > >> > >> that
>> >> >>>>> > > > >> > >> >> >>> > storing
>> >> >>>>> > > > >> > >> >> >>> > >> the schema id together with
>> >> >> serialized
>> >> >>>> bytes
>> >> >>>>> > in the
>> >> >>>>> > > > value
>> >> >>>>> > > > >> is
>> >> >>>>> > > > >> > >> >> better?
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> C. per message encryption
>> >> >>>>> > > > >> > >> >> >>> > >> One drawback of this approach is
>> >> that
>> >> >>>> this
>> >> >>>>> > > > significantly
>> >> >>>>> > > > >> > reduce
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> effectiveness of compression,
>> >> which
>> >> >>>> happens on
>> >> >>>>> > a set
>> >> >>>>> > > of
>> >> >>>>> > > > >> > >> serialized
>> >> >>>>> > > > >> > >> >> >>> > >> messages. An alternative is to
>> >> enable
>> >> >>>> SSL for
>> >> >>>>> > wire
>> >> >>>>> > > > >> > encryption
>> >> >>>>> > > > >> > >> and
>> >> >>>>> > > > >> > >> >> >>> rely
>> >> >>>>> > > > >> > >> >> >>> > on
>> >> >>>>> > > > >> > >> >> >>> > >> the storage system (e.g. LUKS)
>> >> for at
>> >> >>>> rest
>> >> >>>>> > > encryption.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> D. cluster ID for mirroring
>> across
>> >> >>>> Kafka
>> >> >>>>> > clusters
>> >> >>>>> > > > >> > >> >> >>> > >> This is actually interesting.
>> >> Today,
>> >> >>>> to avoid
>> >> >>>>> > > > introducing
>> >> >>>>> > > > >> > >> cycles
>> >> >>>>> > > > >> > >> >> when
>> >> >>>>> > > > >> > >> >> >>> > doing
>> >> >>>>> > > > >> > >> >> >>> > >> mirroring across data centers,
>> one
>> >> >>>> would
>> >> >>>>> > either have
>> >> >>>>> > > to
>> >> >>>>> > > > >> set
>> >> >>>>> > > > >> > up
>> >> >>>>> > > > >> > >> two
>> >> >>>>> > > > >> > >> >> >>> Kafka
>> >> >>>>> > > > >> > >> >> >>> > >> clusters (a local and an
>> >> aggregate)
>> >> >>>> per data
>> >> >>>>> > center
>> >> >>>>> > > or
>> >> >>>>> > > > >> > rename
>> >> >>>>> > > > >> > >> >> topics.
>> >> >>>>> > > > >> > >> >> >>> > >> Neither is ideal. With headers,
>> >> the
>> >> >>>> producer
>> >> >>>>> > could
>> >> >>>>> > > tag
>> >> >>>>> > > > >> each
>> >> >>>>> > > > >> > >> >> message
>> >> >>>>> > > > >> > >> >> >>> with
>> >> >>>>> > > > >> > >> >> >>> > >> the producing cluster ID in the
>> >> >> header.
>> >> >>>>> > MirrorMaker
>> >> >>>>> > > > could
>> >> >>>>> > > > >> > then
>> >> >>>>> > > > >> > >> >> avoid
>> >> >>>>> > > > >> > >> >> >>> > >> mirroring messages to a cluster
>> if
>> >> >>>> they are
>> >> >>>>> > tagged
>> >> >>>>> > > with
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> same
>> >> >>>>> > > > >> > >> >> >>> cluster
>> >> >>>>> > > > >> > >> >> >>> > >> id.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> However, an alternative approach
>> >> is
>> >> >> to
>> >> >>>>> > introduce sth
>> >> >>>>> > > > like
>> >> >>>>> > > > >> > >> >> >>> hierarchical
>> >> >>>>> > > > >> > >> >> >>> > >> topic and store messages from
>> >> >> different
>> >> >>>>> > clusters in
>> >> >>>>> > > > >> > different
>> >> >>>>> > > > >> > >> >> >>> partitions
>> >> >>>>> > > > >> > >> >> >>> > >> under the same topic. This
>> >> approach
>> >> >>>> avoids
>> >> >>>>> > filtering
>> >> >>>>> > > > out
>> >> >>>>> > > > >> > >> unneeded
>> >> >>>>> > > > >> > >> >> >>> data
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> makes offset preserving easier
>> to
>> >> >>>> support. It
>> >> >>>>> > may
>> >> >>>>> > > make
>> >> >>>>> > > > >> > >> compaction
>> >> >>>>> > > > >> > >> >> >>> > trickier
>> >> >>>>> > > > >> > >> >> >>> > >> though since the same key may
>> >> show up
>> >> >>>> in
>> >> >>>>> > different
>> >> >>>>> > > > >> > partitions.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> E. record-level lineage
>> >> >>>>> > > > >> > >> >> >>> > >> For example, a source connector
>> >> could
>> >> >>>> store in
>> >> >>>>> > the
>> >> >>>>> > > > message
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> metadata
>> >> >>>>> > > > >> > >> >> >>> > >> (e.g. UUID) of the source
>> record.
>> >> >>>> Similarly,
>> >> >>>>> > if a
>> >> >>>>> > > > stream
>> >> >>>>> > > > >> job
>> >> >>>>> > > > >> > >> >> >>> transforms
>> >> >>>>> > > > >> > >> >> >>> > >> messages from topic A to topic
>> B,
>> >> the
>> >> >>>> library
>> >> >>>>> > could
>> >> >>>>> > > > >> include
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> source
>> >> >>>>> > > > >> > >> >> >>> > >> message offset in each of the
>> >> >>>> transformed
>> >> >>>>> > message in
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> header.
>> >> >>>>> > > > >> > >> >> Not
>> >> >>>>> > > > >> > >> >> >>> > sure
>> >> >>>
>> >> >>>>> > > > >> > >> >> >>> > >> how widely useful record-level
>> >> >> lineage
>> >> >>>> is
>> >> >>>>> > though
>> >> >>>>> > > since
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> >> overhead
>> >> >>>>> > > > >> > >> >> >>> > could
>> >> >>>>> > > > >> > >> >> >>> > >> be significant.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> F. auditing metadata
>> >> >>>>> > > > >> > >> >> >>> > >> We could put things like
>> >> >>>> clientId/host/user in
>> >> >>>>> > the
>> >> >>>>> > > > header
>> >> >>>>> > > > >> in
>> >> >>>>> > > > >> > >> each
>> >> >>>>> > > > >> > >> >> >>> > message
>> >> >>>>> > > > >> > >> >> >>> > >> for auditing. These metadata are
>> >> >>>> really at the
>> >> >>>>> > > producer
>> >> >>>>> > > > >> > level
>> >> >>>>> > > > >> > >> >> though.
>> >> >>>>> > > > >> > >> >> >>> > So, a
>> >> >>>>> > > > >> > >> >> >>> > >> more efficient way is to only
>> >> >> include a
>> >> >>>>> > "producerId"
>> >> >>>>> > > > per
>> >> >>>>> > > > >> > >> message
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> > send
>> >> >>>>> > > > >> > >> >> >>> > >> the producerId -> metadata
>> mapping
>> >> >>>>> > independently.
>> >> >>>>> > > > KIP-98
>> >> >>>>> > > > >> is
>> >> >>>>> > > > >> > >> >> actually
>> >> >>>>> > > > >> > >> >> >>> > >> proposing including such a
>> >> producerId
>> >> >>>> natively
>> >> >>>>> > in the
>> >> >>>>> > > > >> > message.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> So, overall, I not sure that I
>> am
>> >> >> fully
>> >> >>>>> > convinced of
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > strong
>> >> >>>>> > > > >> > >> >> >>> > third-party
>> >> >>>>> > > > >> > >> >> >>> > >> use cases of headers yet.
>> Perhaps
>> >> we
>> >> >>>> could
>> >> >>>>> > discuss a
>> >> >>>>> > > > bit
>> >> >>>>> > > > >> > more
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> make
>> >> >>>>> > > > >> > >> >> >>> > one
>> >> >>>>> > > > >> > >> >> >>> > >> or two really convincing use
>> >> cases.
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> Another orthogonal question is
>> >> >>>> whether header
>> >> >>>>> > should
>> >> >>>>> > > > be
>> >> >>>>> > > > >> > >> exposed
>> >> >>>>> > > > >> > >> >> in
>> >> >>>>> > > > >> > >> >> >>> > stream
>> >> >>>>> > > > >> > >> >> >>> > >> processing systems such Kafka
>> >> stream,
>> >> >>>> Samza,
>> >> >>>>> > and
>> >> >>>>> > > Spark
>> >> >>>>> > > > >> > >> streaming.
>> >> >>>>> > > > >> > >> >> >>> > >> Currently, those systems just
>> deal
>> >> >> with
>> >> >>>>> > key/value
>> >> >>>>> > > > pairs.
>> >> >>>>> > > > >> > >> Should we
>> >> >>>>> > > > >> > >> >> >>> > expose a
>> >> >>>>> > > > >> > >> >> >>> > >> third thing header there too or
>> >> >>>> somehow map
>> >> >>>>> > header to
>> >> >>>>> > > > key
>> >> >>>>> > > > >> or
>> >> >>>>> > > > >> > >> >> value?
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> Thanks,
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> Jun
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM,
>> >> >>>> Michael
>> >> >>>>> > Pearce <
>> >> >>>>> > > > >> > >> >> >>> Michael.Pearce@ig.com>
>> >> >>>>> > > > >> > >> >> >>> > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > I assume, that after a period
>> >> of a
>> >> >>>> week,
>> >> >>>>> > that there
>> >> >>>>> > > > is
>> >> >>>>> > > > >> no
>> >> >>>>> > > > >> > >> >> concerns
>> >> >>>>> > > > >> > >> >> >>> now
>> >> >>>>> > > > >> > >> >> >>> > >> > with points 1, and 2 and now
>> we
>> >> >> have
>> >> >>>>> > agreement that
>> >> >>>>> > > > >> > headers
>> >> >>>>> > > > >> > >> are
>> >> >>>>> > > > >> > >> >> >>> useful
>> >> >>>>> > > > >> > >> >> >>> > >> and
>> >> >>>>> > > > >> > >> >> >>> > >> > needed in Kafka. As such if
>> put
>> >> to
>> >> >> a
>> >> >>>> KIP
>> >> >>>>> > vote, this
>> >> >>>>> > > > >> > wouldn’t
>> >> >>>>> > > > >> > >> be
>> >> >>>>> > > > >> > >> >> a
>> >> >>>>> > > > >> > >> >> >>> > reason
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > reject.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > @
>> >> >>>>> > > > >> > >> >> >>> > >> > Ignacio on point 4).
>> >> >>>>> > > > >> > >> >> >>> > >> > I think for purpose of getting
>> >> this
>> >> >>>> KIP
>> >> >>>>> > moving past
>> >> >>>>> > > > >> this,
>> >> >>>>> > > > >> > we
>> >> >>>>> > > > >> > >> can
>> >> >>>>> > > > >> > >> >> >>> state
>> >> >>>>> > > > >> > >> >> >>> > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > key will be a 4 bytes space
>> that
>> >> >> can
>> >> >>>> will be
>> >> >>>>> > > > naturally
>> >> >>>>> > > > >> > >> >> interpreted
>> >> >>>>> > > > >> > >> >> >>> as
>> >> >>>>> > > > >> > >> >> >>> > an
>> >> >>>>> > > > >> > >> >> >>> > >> > Int32 (if namespacing is later
>> >> >>>> wanted you can
>> >> >>>>> > > easily
>> >> >>>>> > > > >> split
>> >> >>>>> > > > >> > >> this
>> >> >>>>> > > > >> > >> >> >>> into
>> >> >>>>> > > > >> > >> >> >>> > two
>> >> >>>>> > > > >> > >> >> >>> > >> > int16 spaces), from the wire
>> >> >> protocol
>> >> >>>>> > > implementation
>> >> >>>>> > > > >> this
>> >> >>>>> > > > >> > >> makes
>> >> >>>>> > > > >> > >> >> no
>> >> >>>>> > > > >> > >> >> >>> > >> > difference I don’t believe. Is
>> >> this
>> >> >>>>> > reasonable to
>> >> >>>>> > > > all?
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > On 5) as per point 4 therefor
>> >> happy
>> >> >>>> we keep
>> >> >>>>> > with 32
>> >> >>>>> > > > >> bits.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "
>> >> >>>>> > ignacio.solis@gmail.com on
>> >> >>>>> > > > >> behalf
>> >> >>>>> > > > >> > of
>> >> >>>>> > > > >> > >> >> >>> Ignacio
>> >> >>>>> > > > >> > >> >> >>> > >> > Solis" <
>> ignacio.solis@gmail.com
>> >> on
>> >> >>>> behalf of
>> >> >>>>> > > > >> > isolis@igso.net
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> > >> >> >>> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > Summary:
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > 3) Yes - Header value as
>> >> byte[]
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > 4a) Int,Int - No
>> >> >>>>> > > > >> > >> >> >>> > >> > 4b) Int - Yes
>> >> >>>>> > > > >> > >> >> >>> > >> > 4c) String - Reluctant
>> maybe
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > 5) I believe the header
>> >> system
>> >> >>>> should
>> >> >>>>> > take a
>> >> >>>>> > > > single
>> >> >>>>> > > > >> > >> int. I
>> >> >>>>> > > > >> > >> >> >>> think
>> >> >>>>> > > > >> > >> >> >>> > >> > 32bits is
>> >> >>>>> > > > >> > >> >> >>> > >> > a good size, if you want
>> to
>> >> >>>> interpret
>> >> >>>>> > this as
>> >> >>>>> > > to
>> >> >>>>> > > > >> 16bit
>> >> >>>>> > > > >> > >> >> numbers
>> >> >>>>> > > > >> > >> >> >>> in
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > layer
>> >> >>>>> > > > >> > >> >> >>> > >> > above go right ahead. If
>> >> >>>> somebody wants
>> >> >>>>> > to
>> >> >>>>> > > argue
>> >> >>>>> > > > >> for
>> >> >>>>> > > > >> > 16
>> >> >>>>> > > > >> > >> >> bits
>> >> >>>>> > > > >> > >> >> >>> or
>> >> >>>>> > > > >> > >> >> >>> > 64
>> >> >>>>> > > > >> > >> >> >>> > >> > bits of
>> >> >>>>> > > > >> > >> >> >>> > >> > header key space I would
>> >> >> listen.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > Discussion:
>> >> >>>>> > > > >> > >> >> >>> > >> > Dividing the key space
>> into
>> >> >>>> sub_key_1 and
>> >> >>>>> > > > sub_key_2
>> >> >>>>> > > > >> > >> makes no
>> >> >>>>> > > > >> > >> >> >>> > sense to
>> >> >>>>> > > > >> > >> >> >>> > >> > me at
>> >> >>>>> > > > >> > >> >> >>> > >> > this layer. Are we going
>> to
>> >> >>>> start
>> >> >>>>> > providing
>> >> >>>>> > > > APIs to
>> >> >>>>> > > > >> > get
>> >> >>>>> > > > >> > >> all
>> >> >>>>> > > > >> > >> >> >>> the
>> >> >>>>> > > > >> > >> >> >>> > >> > sub_key_1s? or all the
>> >> >>>> sub_key_2s? If
>> >> >>>>> > there is
>> >> >>>>> > > > no
>> >> >>>>> > > > >> > >> >> >>> distinguishing
>> >> >>>>> > > > >> > >> >> >>> > >> > functions
>> >> >>>>> > > > >> > >> >> >>> > >> > that are applied to each
>> one
>> >> >>>> then they
>> >> >>>>> > should
>> >> >>>>> > > be
>> >> >>>>> > > > a
>> >> >>>>> > > > >> > single
>> >> >>>>> > > > >> > >> >> >>> value.
>> >> >>>>> > > > >> > >> >> >>> > At
>> >> >>>>> > > > >> > >> >> >>> > >> > this
>> >> >>>>> > > > >> > >> >> >>> > >> > layer all we're doing is
>> >> >>>> equality.
>> >> >>>>> > > > >> > >> >> >>> > >> > If the above layer wants
>> to
>> >> >>>> interpret
>> >> >>>>> > this as
>> >> >>>>> > > 2,
>> >> >>>>> > > > 3
>> >> >>>>> > > > >> or
>> >> >>>>> > > > >> > >> more
>> >> >>>>> > > > >> > >> >> >>> values
>> >> >>>>> > > > >> > >> >> >>> > >> > that's a
>> >> >>>>> > > > >> > >> >> >>> > >> > different question. I
>> >> >>>> personally think
>> >> >>>>> > it's
>> >> >>>>> > > all
>> >> >>>>> > > > one
>> >> >>>>> > > > >> > >> >> keyspace
>> >> >>>>> > > > >> > >> >> >>> > that is
>> >> >>>>> > > > >> > >> >> >>> > >> > getting assigned using
>> some
>> >> >>>> structure,
>> >> >>>>> > but if
>> >> >>>>> > > you
>> >> >>>>> > > > >> > want to
>> >> >>>>> > > > >> > >> >> >>> > sub-assign
>> >> >>>>> > > > >> > >> >> >>> > >> > parts
>> >> >>>>> > > > >> > >> >> >>> > >> > of it then that's fine.
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > The same discussion
>> applies
>> >> to
>> >> >>>> strings.
>> >> >>>>> > If
>> >> >>>>> > > > somebody
>> >> >>>>> > > > >> > >> argued
>> >> >>>>> > > > >> > >> >> for
>> >> >>>>> > > > >> > >> >> >>> > >> > strings,
>> >> >>>>> > > > >> > >> >> >>> > >> > would we be arguing to
>> >> divide
>> >> >> the
>> >> >>>>> > strings with
>> >> >>>>> > > > dots
>> >> >>>>> > > > >> > ('.')
>> >> >>>>> > > > >> > >> >> as a
>> >> >>>>> > > > >> > >> >> >>> > >> > requirement?
>> >> >>>>> > > > >> > >> >> >>> > >> > Would we want them to give
>> >> us
>> >> >> the
>> >> >>>>> > different
>> >> >>>>> > > name
>> >> >>>>> > > > >> > segments
>> >> >>>>> > > > >> > >> >> >>> > separately?
>> >> >>>>> > > > >> > >> >> >>> > >> > Would we be performing any
>> >> >>>> actions on
>> >> >>>>> > this key
>> >> >>>>> > > > other
>> >> >>>>> > > > >> > than
>> >> >>>>> > > > >> > >> >> >>> > matching?
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > Nacho
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at
>> 9:30
>> >> >> AM,
>> >> >>>> Michael
>> >> >>>>> > > Pearce <
>> >> >>>>> > > > >> > >> >> >>> > >> Michael.Pearce@ig.com
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > #jay #jun any concerns
>> on
>> >> 1
>> >> >>>> and 2
>> >> >>>>> > still?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > @all
>> >> >>>>> > > > >> > >> >> >>> > >> > > To get this moving
>> along a
>> >> >> bit
>> >> >>>> more
>> >> >>>>> > I'd also
>> >> >>>>> > > > like
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> ask
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> >>> get
>> >> >>>>> > > > >> > >> >> >>> > >> > clarity on
>> >> >>>>> > > > >> > >> >> >>> > >> > > the below last points:
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 3) I believe we're all
>> >> >> roughly
>> >> >>>> happy
>> >> >>>>> > with the
>> >> >>>>> > > > >> header
>> >> >>>>> > > > >> > >> value
>> >> >>>>> > > > >> > >> >> >>> > being a
>> >> >>>>> > > > >> > >> >> >>> > >> > byte[]?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 4) I believe consensus
>> has
>> >> >>>> been for an
>> >> >>>>> > > > namespace
>> >> >>>>> > > > >> > based
>> >> >>>>> > > > >> > >> int
>> >> >>>>> > > > >> > >> >> >>> > approach
>> >> >>>>> > > > >> > >> >> >>> > >> > > {int,int} for the key.
>> Any
>> >> >>>> objections
>> >> >>>>> > if this
>> >> >>>>> > > > is
>> >> >>>>> > > > >> > what
>> >> >>>>> > > > >> > >> we
>> >> >>>>> > > > >> > >> >> go
>> >> >>>>> > > > >> > >> >> >>> > with?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 5) as we have if
>> >> assumption
>> >> >> in
>> >> >>>> (4) is
>> >> >>>>> > > correct,
>> >> >>>>> > > > >> > >> {int,int}
>> >> >>>>> > > > >> > >> >> >>> keys.
>> >> >>>>> > > > >> > >> >> >>> > >> > > Should both int's be
>> >> int16 or
>> >> >>>> int32?
>> >> >>>>> > > > >> > >> >> >>> > >> > > I'm for them being
>> int16(2
>> >> >>>> bytes) as
>> >> >>>>> > combined
>> >> >>>>> > > > is
>> >> >>>>> > > > >> > space
>> >> >>>>> > > > >> > >> of
>> >> >>>>> > > > >> > >> >> >>> > 4bytes as
>> >> >>>>> > > > >> > >> >> >>> > >> > per
>> >> >>>>> > > > >> > >> >> >>> > >> > > original and gives
>> plenty
>> >> of
>> >> >>>>> > combinations for
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> >> >>> foreseeable,
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > keeps
>> >> >>>>> > > > >> > >> >> >>> > >> > > the overhead small.
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > Do we see any benefit in
>> >> >>>> another kip
>> >> >>>>> > call to
>> >> >>>>> > > > >> discuss
>> >> >>>>> > > > >> > >> >> these at
>> >> >>>>> > > > >> > >> >> >>> > all?
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > Cheers
>> >> >>>>> > > > >> > >> >> >>> > >> > > Mike
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >> ______________________________
>> >> >>>>> > __________
>> >> >>>>> > > > >> > >> >> >>> > >> > > From: K Burstev <
>> >> >>>> k.burstev@yandex.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > Sent: Friday, November
>> 18,
>> >> >> 2016
>> >> >>>>> > 7:07:07 AM
>> >> >>>>> > > > >> > >> >> >>> > >> > > To:
>> dev@kafka.apache.org
>> >> >>>>> > > > >> > >> >> >>> > >> > > Subject: Re: [DISCUSS]
>> >> KIP-82
>> >> >>>> - Add
>> >> >>>>> > Record
>> >> >>>>> > > > Headers
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > For what it is worth
>> also
>> >> i
>> >> >>>> agree. As
>> >> >>>>> > a user:
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 1) Yes - Headers are
>> >> >>>> worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> > > 2) Yes - Headers should
>> >> be a
>> >> >>>> top level
>> >> >>>>> > > option
>> >> >>>>> > > > >> > >> >> >>> > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > 14.11.2016, 21:15,
>> >> "Ignacio
>> >> >>>> Solis" <
>> >> >>>>> > > > >> isolis@igso.net
>> >> >>>>> > > > >> > >:
>> >> >>>>> > > > >> > >> >> >>> > >> > > > 1) Yes - Headers are
>> >> >>>> worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> > > > 2) Yes - Headers
>> should
>> >> be
>> >> >> a
>> >> >>>> top
>> >> >>>>> > level
>> >> >>>>> > > option
>> >> >>>>> > > > >> > >> >> >>> > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14, 2016
>> at
>> >> >> 9:16
>> >> >>>> AM,
>> >> >>>>> > Michael
>> >> >>>>> > > > Pearce
>> >> >>>>> > > > >> <
>> >> >>>>> > > > >> > >> >> >>> > >> > Michael.Pearce@ig.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Hi Roger,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> The kip
>> >> details/examples
>> >> >>>> the
>> >> >>>>> > original
>> >> >>>>> > > > proposal
>> >> >>>>> > > > >> > for
>> >> >>>>> > > > >> > >> key
>> >> >>>>> > > > >> > >> >> >>> > spacing
>> >> >>>>> > > > >> > >> >> >>> > >> ,
>> >> >>>>> > > > >> > >> >> >>> > >> > not
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> new mentioned as per
>> >> >>>> discussion
>> >> >>>>> > namespace
>> >> >>>>> > > > >> idea.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> We will need to
>> update
>> >> >> the
>> >> >>>> kip,
>> >> >>>>> > when we
>> >> >>>>> > > get
>> >> >>>>> > > > >> > >> agreement
>> >> >>>>> > > > >> > >> >> >>> this
>> >> >>>>> > > > >> > >> >> >>> > is a
>> >> >>>>> > > > >> > >> >> >>> > >> > better
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> approach (which
>> seems
>> >> to
>> >> >>>> be the
>> >> >>>>> > case if I
>> >> >>>>> > > > have
>> >> >>>>> > > > >> > >> >> understood
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > general
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> feeling in the
>> >> >>>> conversation)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Re the variable
>> ints,
>> >> at
>> >> >>>> very
>> >> >>>>> > early stage
>> >> >>>>> > > > we
>> >> >>>>> > > > >> did
>> >> >>>>> > > > >> > >> think
>> >> >>>>> > > > >> > >> >> >>> about
>> >> >>>>> > > > >> > >> >> >>> > >> > this. I
>> >> >>>>> > > > >> > >> >> >>> > >> > > think
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> the added complexity
>> >> for
>> >> >>>> the
>> >> >>>>> > saving isn't
>> >> >>>>> > > > >> worth
>> >> >>>>> > > > >> > it.
>> >> >>>>> > > > >> > >> >> I'd
>> >> >>>>> > > > >> > >> >> >>> > rather
>> >> >>>>> > > > >> > >> >> >>> > >> go
>> >> >>>>> > > > >> > >> >> >>> > >> > > with, if
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> we want to reduce
>> >> >>>> overheads and
>> >> >>>>> > size
>> >> >>>>> > > int16
>> >> >>>>> > > > >> > (2bytes)
>> >> >>>>> > > > >> > >> >> keys
>> >> >>>>> > > > >> > >> >> >>> as
>> >> >>>>> > > > >> > >> >> >>> > it
>> >> >>>>> > > > >> > >> >> >>> > >> > keeps it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> simple.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> On the note of no
>> >> >> headers,
>> >> >>>> there
>> >> >>>>> > is as
>> >> >>>>> > > per
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > kip
>> >> >>>>> > > > >> > >> as
>> >> >>>>> > > > >> > >> >> we
>> >> >>>>> > > > >> > >> >> >>> > use an
>> >> >>>>> > > > >> > >> >> >>> > >> > > attribute
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> bit to denote if
>> >> headers
>> >> >>>> are
>> >> >>>>> > present or
>> >> >>>>> > > > not as
>> >> >>>>> > > > >> > such
>> >> >>>>> > > > >> > >> >> >>> > provides a
>> >> >>>>> > > > >> > >> >> >>> > >> > zero
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead currently
>> if
>> >> >>>> headers are
>> >> >>>>> > not
>> >> >>>>> > > used.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> I think as radai
>> >> mentions
>> >> >>>> would be
>> >> >>>>> > good
>> >> >>>>> > > > first
>> >> >>>>> > > > >> > if we
>> >> >>>>> > > > >> > >> >> can
>> >> >>>>> > > > >> > >> >> >>> get
>> >> >>>>> > > > >> > >> >> >>> > >> > clarity if
>> >> >>>>> > > > >> > >> >> >>> > >> > > do
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> we now have general
>> >> >>>> consensus that
>> >> >>>>> > (1)
>> >> >>>>> > > > headers
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> >> >>> > worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> and
>> >> >>>>> > > > >> > >> >> >>> > >> > > useful,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> and (2) we want it
>> as
>> >> a
>> >> >>>> top level
>> >> >>>>> > entity.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Just to state the
>> >> >> obvious i
>> >> >>>>> > believe (1)
>> >> >>>>> > > > >> headers
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> >> >>> > worthwhile
>> >> >>>>> > > > >> > >> >> >>> > >> > and (2)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> agree as a top level
>> >> >>>> entity.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Mike
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>> ______________________________
>> >> >>>>> > __________
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> From: Roger Hoover <
>> >> >>>>> > > roger.hoover@gmail.com
>> >> >>>>> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sent: Wednesday,
>> >> November
>> >> >>>> 9, 2016
>> >> >>>>> > 9:10:47
>> >> >>>>> > > > PM
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> To:
>> >> dev@kafka.apache.org
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Subject: Re:
>> [DISCUSS]
>> >> >>>> KIP-82 - Add
>> >> >>>>> > > Record
>> >> >>>>> > > > >> > Headers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Sorry for going a
>> >> little
>> >> >>>> in the
>> >> >>>>> > weeds but
>> >> >>>>> > > > >> thanks
>> >> >>>>> > > > >> > >> for
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> replies
>> >> >>>>> > > > >> > >> >> >>> > >> > > regarding
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> varint.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Agreed that a prefix
>> >> and
>> >> >>>> {int,
>> >> >>>>> > int} can
>> >> >>>>> > > be
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> same.
>> >> >>>>> > > > >> > >> >> It
>> >> >>>>> > > > >> > >> >> >>> > doesn't
>> >> >>>>> > > > >> > >> >> >>> > >> > look
>> >> >>>>> > > > >> > >> >> >>> > >> > > like
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> that's what the KIP
>> is
>> >> >>>> saying the
>> >> >>>>> > "Open"
>> >> >>>>> > > > >> > section.
>> >> >>>>> > > > >> > >> The
>> >> >>>>> > > > >> > >> >> >>> > example
>> >> >>>>> > > > >> > >> >> >>> > >> > shows
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> 2100001
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> for New Relic and
>> >> 210002
>> >> >>>> for App
>> >> >>>>> > Dynamics
>> >> >>>>> > > > >> > implying
>> >> >>>>> > > > >> > >> >> that
>> >> >>>>> > > > >> > >> >> >>> the
>> >> >>>>> > > > >> > >> >> >>> > New
>> >> >>>>> > > > >> > >> >> >>> > >> > Relic
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> organization will
>> have
>> >> >>>> only a
>> >> >>>>> > single
>> >> >>>>> > > > header id
>> >> >>>>> > > > >> > to
>> >> >>>>> > > > >> > >> work
>> >> >>>>> > > > >> > >> >> >>> > with. Or
>> >> >>>>> > > > >> > >> >> >>> > >> > is
>> >> >>>>> > > > >> > >> >> >>> > >> > > 2100001
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> a prefix? The main
>> >> point
>> >> >>>> of a
>> >> >>>>> > namespace
>> >> >>>>> > > or
>> >> >>>>> > > > >> > prefix
>> >> >>>>> > > > >> > >> is
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> >>> > reduce
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> overhead of config
>> >> >> mapping
>> >> >>>> or
>> >> >>>>> > > registration
>> >> >>>>> > > > >> > >> depending
>> >> >>>>> > > > >> > >> >> on
>> >> >>>>> > > > >> > >> >> >>> how
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> namespaces/prefixes
>> >> are
>> >> >>>> managed.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Would love to hear
>> >> more
>> >> >>>> feedback
>> >> >>>>> > on the
>> >> >>>>> > > > >> > >> higher-level
>> >> >>>>> > > > >> > >> >> >>> > questions
>> >> >>>>> > > > >> > >> >> >>> > >> > > though...
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Cheers,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Roger
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> On Wed, Nov 9, 2016
>> at
>> >> >>>> 11:38 AM,
>> >> >>>>> > radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I think this
>> >> discussion
>> >> >>>> is
>> >> >>>>> > getting a
>> >> >>>>> > > bit
>> >> >>>>> > > > >> into
>> >> >>>>> > > > >> > the
>> >> >>>>> > > > >> > >> >> >>> weeds on
>> >> >>>>> > > > >> > >> >> >>> > >> > technical
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > implementation
>> >> details.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > I'd liek to step
>> >> back a
>> >> >>>> minute
>> >> >>>>> > and try
>> >> >>>>> > > > and
>> >> >>>>> > > > >> > >> establish
>> >> >>>>> > > > >> > >> >> >>> > where we
>> >> >>>>> > > > >> > >> >> >>> > >> > are in
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > larger picture:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (re-wording
>> nacho's
>> >> >> last
>> >> >>>>> > paragraph)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 1. are we all in
>> >> >>>> agreement that
>> >> >>>>> > headers
>> >> >>>>> > > > are
>> >> >>>>> > > > >> a
>> >> >>>>> > > > >> > >> >> >>> worthwhile
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > useful
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > addition to have?
>> >> this
>> >> >>>> was
>> >> >>>>> > contested
>> >> >>>>> > > > early
>> >> >>>>> > > > >> on
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > 2. are we all in
>> >> >>>> agreement on
>> >> >>>>> > headers
>> >> >>>>> > > as
>> >> >>>>> > > > top
>> >> >>>>> > > > >> > >> level
>> >> >>>>> > > > >> > >> >> >>> entity
>> >> >>>>> > > > >> > >> >> >>> > vs
>> >> >>>>> > > > >> > >> >> >>> > >> > headers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > squirreled-away in
>> >> V?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > if there are still
>> >> >>>> concerns
>> >> >>>>> > around
>> >> >>>>> > > these
>> >> >>>>> > > > #2
>> >> >>>>> > > > >> > >> points
>> >> >>>>> > > > >> > >> >> >>> (#jay?
>> >> >>>>> > > > >> > >> >> >>> > >> > #jun?)?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > (and now back to
>> our
>> >> >>>> normal
>> >> >>>>> > programming
>> >> >>>>> > > > ...)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > varints are nice.
>> >> >> having
>> >> >>>> said
>> >> >>>>> > that, its
>> >> >>>>> > > > >> adding
>> >> >>>>> > > > >> > >> >> >>> complexity
>> >> >>>>> > > > >> > >> >> >>> > >> (see
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>> https://github.com/addthis/
>> >> >>>>> > > > >> > >> >> stream-lib/blob/master/src/
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>> main/java/com/clearspring/
>> >> >>>>> > > > >> > >> >> analytics/util/Varint.java
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > as 1st google
>> >> result)
>> >> >>>> and would
>> >> >>>>> > require
>> >> >>>>> > > > >> anyone
>> >> >>>>> > > > >> > >> >> writing
>> >> >>>>> > > > >> > >> >> >>> > other
>> >> >>>>> > > > >> > >> >> >>> > >> > clients
>> >> >>>>> > > > >> > >> >> >>> > >> > > (C?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > Python? Go? Bash?
>> >> ;-) )
>> >> >>>> to
>> >> >>>>> > > get/implement
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > >> same,
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> for
>> >> >>>>> > > > >> > >> >> >>> > >> > relatively
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > little gain (int
>> vs
>> >> >>>> string is
>> >> >>>>> > order of
>> >> >>>>> > > > >> > magnitude,
>> >> >>>>> > > > >> > >> >> this
>> >> >>>>> > > > >> > >> >> >>> > isnt).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > int namespacing vs
>> >> >> {int,
>> >> >>>> int}
>> >> >>>>> > > namespacing
>> >> >>>>> > > > >> are
>> >> >>>>> > > > >> > >> >> basically
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > same
>> >> >>>>> > > > >> > >> >> >>> > >> > > thing -
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > youre just
>> >> namespacing
>> >> >>>> an int64
>> >> >>>>> > and
>> >> >>>>> > > > giving
>> >> >>>>> > > > >> > people
>> >> >>>>> > > > >> > >> >> while
>> >> >>>>> > > > >> > >> >> >>> > 2^32
>> >> >>>>> > > > >> > >> >> >>> > >> > ranges
>> >> >>>>> > > > >> > >> >> >>> > >> > > at a
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > time. the part i
>> >> like
>> >> >>>> about this
>> >> >>>>> > is
>> >> >>>>> > > > letting
>> >> >>>>> > > > >> > >> people
>> >> >>>>> > > > >> > >> >> >>> have a
>> >> >>>>> > > > >> > >> >> >>> > >> large
>> >> >>>>> > > > >> > >> >> >>> > >> > > swath of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > numbers with one
>> >> >>>> registration so
>> >> >>>>> > they
>> >> >>>>> > > > dont
>> >> >>>>> > > > >> > have
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> come
>> >> >>>>> > > > >> > >> >> >>> > back
>> >> >>>>> > > > >> > >> >> >>> > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > every
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > single
>> plugin/header
>> >> >>>> they want to
>> >> >>>>> > > > "reserve".
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > On Wed, Nov 9,
>> 2016
>> >> at
>> >> >>>> 11:01 AM,
>> >> >>>>> > Roger
>> >> >>>>> > > > >> Hoover
>> >> >>>>> > > > >> > <
>> >> >>>>> > > > >> > >> >> >>> > >> > > roger.hoover@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > Since some of
>> the
>> >> >>>> debate has
>> >> >>>>> > been
>> >> >>>>> > > about
>> >> >>>>> > > > >> > >> overhead +
>> >> >>>>> > > > >> > >> >> >>> > >> > performance, I'm
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > wondering if we
>> >> have
>> >> >>>>> > considered a
>> >> >>>>> > > > varint
>> >> >>>>> > > > >> > >> encoding
>> >> >>>>> > > > >> > >> >> (
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>> https://developers.google.com/
>> >> >>>>> > > > >> > >> >> protocol-buffers/docs/
>> >> >>>>> > > > >> > >> >> >>> > >> > > encoding#varints)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > the header
>> length
>> >> >>>> field (int32
>> >> >>>>> > in the
>> >> >>>>> > > > >> > proposal)
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> for
>> >> >>>>> > > > >> > >> >> >>> > >> > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > ids? If
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > you
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > don't use
>> headers,
>> >> >> the
>> >> >>>>> > overhead would
>> >> >>>>> > > > be a
>> >> >>>>> > > > >> > >> single
>> >> >>>>> > > > >> > >> >> >>> byte
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > for each
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > id < 128 would
>> >> also
>> >> >>>> need only a
>> >> >>>>> > > single
>> >> >>>>> > > > >> byte?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > On Wed, Nov 9,
>> >> 2016
>> >> >> at
>> >> >>>> 6:43 AM,
>> >> >>>>> > > radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > @magnus - and
>> >> very
>> >> >>>> dangerous
>> >> >>>>> > (youre
>> >> >>>>> > > > >> > >> essentially
>> >> >>>>> > > > >> > >> >> >>> > >> > downloading and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > executing
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > arbitrary code
>> >> off
>> >> >>>> the
>> >> >>>>> > internet on
>> >> >>>>> > > > your
>> >> >>>>> > > > >> > >> servers
>> >> >>>>> > > > >> > >> >> ...
>> >> >>>>> > > > >> > >> >> >>> > bad
>> >> >>>>> > > > >> > >> >> >>> > >> > idea
>> >> >>>>> > > > >> > >> >> >>> > >> > > without
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> a
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sandbox, even
>> >> with)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > as for it
>> being
>> >> a
>> >> >>>> purely
>> >> >>>>> > > > administrative
>> >> >>>>> > > > >> > task
>> >> >>>>> > > > >> > >> - i
>> >> >>>>> > > > >> > >> >> >>> > >> disagree.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > i wish it
>> would,
>> >> >>>> really,
>> >> >>>>> > because
>> >> >>>>> > > > then my
>> >> >>>>> > > > >> > >> earlier
>> >> >>>>> > > > >> > >> >> >>> > point on
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > complexity
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the remapping
>> >> >>>> process would
>> >> >>>>> > be
>> >> >>>>> > > > invalid,
>> >> >>>>> > > > >> > but
>> >> >>>>> > > > >> > >> at
>> >> >>>>> > > > >> > >> >> >>> > linkedin,
>> >> >>>>> > > > >> > >> >> >>> > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > example,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > we
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (the team im
>> in)
>> >> >> run
>> >> >>>> kafka
>> >> >>>>> > as a
>> >> >>>>> > > > service.
>> >> >>>>> > > > >> > we
>> >> >>>>> > > > >> > >> dont
>> >> >>>>> > > > >> > >> >> >>> > really
>> >> >>>>> > > > >> > >> >> >>> > >> > know
>> >> >>>>> > > > >> > >> >> >>> > >> > > what our
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > users
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (developing
>> >> >>>> applications
>> >> >>>>> > that use
>> >> >>>>> > > > kafka)
>> >> >>>>> > > > >> > are
>> >> >>>>> > > > >> > >> up
>> >> >>>>> > > > >> > >> >> to
>> >> >>>>> > > > >> > >> >> >>> at
>> >> >>>>> > > > >> > >> >> >>> > any
>> >> >>>>> > > > >> > >> >> >>> > >> > given
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> moment.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > is very
>> possible
>> >> >>>> (given the
>> >> >>>>> > > > existance of
>> >> >>>>> > > > >> > >> headers
>> >> >>>>> > > > >> > >> >> >>> and a
>> >> >>>>> > > > >> > >> >> >>> > >> > > corresponding
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > ecosystem) for
>> >> some
>> >> >>>>> > application to
>> >> >>>>> > > > >> "equip"
>> >> >>>>> > > > >> > >> their
>> >> >>>>> > > > >> > >> >> >>> > >> producers
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > consumers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > with the
>> >> required
>> >> >>>> plugin
>> >> >>>>> > without us
>> >> >>>>> > > > >> > knowing.
>> >> >>>>> > > > >> > >> i
>> >> >>>>> > > > >> > >> >> dont
>> >> >>>>> > > > >> > >> >> >>> > mean
>> >> >>>>> > > > >> > >> >> >>> > >> > to imply
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> thats
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > bad, i just
>> >> want to
>> >> >>>> make the
>> >> >>>>> > point
>> >> >>>>> > > > that
>> >> >>>>> > > > >> > its
>> >> >>>>> > > > >> > >> not
>> >> >>>>> > > > >> > >> >> as
>> >> >>>>> > > > >> > >> >> >>> > simple
>> >> >>>>> > > > >> > >> >> >>> > >> > > keeping it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > sync across a
>> >> >>>> large-enough
>> >> >>>>> > > > organization.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > On Wed, Nov 9,
>> >> 2016
>> >> >>>> at 6:17
>> >> >>>>> > AM,
>> >> >>>>> > > > Magnus
>> >> >>>>> > > > >> > >> Edenhill
>> >> >>>>> > > > >> > >> >> <
>> >> >>>>> > > > >> > >> >> >>> > >> > > magnus@edenhill.se>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > I think
>> there
>> >> is
>> >> >> a
>> >> >>>> piece
>> >> >>>>> > missing
>> >> >>>>> > > in
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> >> Strings
>> >> >>>>> > > > >> > >> >> >>> > >> > discussion,
>> >> >>>>> > > > >> > >> >> >>> > >> > > where
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> pro-Stringers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > reason that
>> by
>> >> >>>> providing
>> >> >>>>> > unique
>> >> >>>>> > > > string
>> >> >>>>> > > > >> > >> >> >>> identifiers
>> >> >>>>> > > > >> > >> >> >>> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > each
>> >> >>>>> > > > >> > >> >> >>> > >> > > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > everything
>> >> will
>> >> >>>> just
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > magically
>> work
>> >> >> for
>> >> >>>> all
>> >> >>>>> > parts of
>> >> >>>>> > > the
>> >> >>>>> > > > >> > stream
>> >> >>>>> > > > >> > >> >> >>> pipeline.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > But the
>> >> strings
>> >> >>>> dont mean
>> >> >>>>> > > anything
>> >> >>>>> > > > by
>> >> >>>>> > > > >> > >> >> themselves,
>> >> >>>>> > > > >> > >> >> >>> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > while we
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> could
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > probably
>> >> envision
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some auto
>> >> plugin
>> >> >>>> loader
>> >> >>>>> > that
>> >> >>>>> > > > >> downloads,
>> >> >>>>> > > > >> > >> >> compiles,
>> >> >>>>> > > > >> > >> >> >>> > links
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > runs
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > on-demand
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > as soon as
>> >> >> they're
>> >> >>>> seen by
>> >> >>>>> > a
>> >> >>>>> > > > >> consumer, I
>> >> >>>>> > > > >> > >> dont
>> >> >>>>> > > > >> > >> >> >>> really
>> >> >>>>> > > > >> > >> >> >>> > >> see
>> >> >>>>> > > > >> > >> >> >>> > >> > a
>> >> >>>>> > > > >> > >> >> >>> > >> > > use-case
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > for
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > something
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > so dynamic
>> >> (and
>> >> >>>> fragile) in
>> >> >>>>> > > > practice.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > In the real
>> >> world
>> >> >>>> an
>> >> >>>>> > application
>> >> >>>>> > > > will
>> >> >>>>> > > > >> be
>> >> >>>>> > > > >> > >> >> >>> configured
>> >> >>>>> > > > >> > >> >> >>> > >> with
>> >> >>>>> > > > >> > >> >> >>> > >> > a set
>> >> >>>>> > > > >> > >> >> >>> > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > plugins
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > to either
>> add
>> >> >>>> (producer)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > or read
>> >> >> (consumer)
>> >> >>>> headers.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > This is an
>> >> >>>> administrative
>> >> >>>>> > task
>> >> >>>>> > > > based
>> >> >>>>> > > > >> on
>> >> >>>>> > > > >> > >> what
>> >> >>>>> > > > >> > >> >> >>> > features a
>> >> >>>>> > > > >> > >> >> >>> > >> > client
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> needs/provides
>> >> >> and
>> >> >>>> results
>> >> >>>>> > in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > some sort of
>> >> >>>> configuration
>> >> >>>>> > to
>> >> >>>>> > > > enable
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> >> >>> configure
>> >> >>>>> > > > >> > >> >> >>> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > desired
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > plugins.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > Since this
>> >> needs
>> >> >>>> to be kept
>> >> >>>>> > > > somewhat
>> >> >>>>> > > > >> in
>> >> >>>>> > > > >> > >> sync
>> >> >>>>> > > > >> > >> >> >>> across
>> >> >>>>> > > > >> > >> >> >>> > an
>> >> >>>>> > > > >> > >> >> >>> > >> > > organisation
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > (there
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > is no point
>> in
>> >> >>>> having
>> >> >>>>> > producers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > add headers
>> no
>> >> >>>> consumers
>> >> >>>>> > will
>> >> >>>>> > > read,
>> >> >>>>> > > > >> and
>> >> >>>>> > > > >> > >> vice
>> >> >>>>> > > > >> > >> >> >>> versa),
>> >> >>>>> > > > >> > >> >> >>> > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > added
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > complexity
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > of assigning
>> >> an
>> >> >> id
>> >> >>>>> > namespace
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > for each
>> >> plugin
>> >> >> as
>> >> >>>> it is
>> >> >>>>> > being
>> >> >>>>> > > > >> > configured
>> >> >>>>> > > > >> > >> >> should
>> >> >>>>> > > > >> > >> >> >>> be
>> >> >>>>> > > > >> > >> >> >>> > >> > tolerable.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > /Magnus
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > 2016-11-09
>> >> 13:06
>> >> >>>> GMT+01:00
>> >> >>>>> > > Michael
>> >> >>>>> > > > >> > Pearce <
>> >> >>>>> > > > >> > >> >> >>> > >> > > Michael.Pearce@ig.com>:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Just
>> >> >>>> following/catching
>> >> >>>>> > up on
>> >> >>>>> > > > what
>> >> >>>>> > > > >> > seems
>> >> >>>>> > > > >> > >> to
>> >> >>>>> > > > >> > >> >> be
>> >> >>>>> > > > >> > >> >> >>> an
>> >> >>>>> > > > >> > >> >> >>> > >> > active
>> >> >>>>> > > > >> > >> >> >>> > >> > > night :)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > @Radai
>> >> sorry if
>> >> >>>> it may
>> >> >>>>> > seem
>> >> >>>>> > > > obvious
>> >> >>>>> > > > >> > but
>> >> >>>>> > > > >> > >> what
>> >> >>>>> > > > >> > >> >> >>> does
>> >> >>>>> > > > >> > >> >> >>> > MD
>> >> >>>>> > > > >> > >> >> >>> > >> > stand
>> >> >>>>> > > > >> > >> >> >>> > >> > > for?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > My take on
>> >> >>>> String vs Int:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I will
>> state
>> >> >>>> first I am
>> >> >>>>> > pro Int
>> >> >>>>> > > > (16
>> >> >>>>> > > > >> or
>> >> >>>>> > > > >> > >> 32).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > I do
>> though
>> >> >>>> playing
>> >> >>>>> > devils
>> >> >>>>> > > > advocate
>> >> >>>>> > > > >> > see a
>> >> >>>>> > > > >> > >> >> big
>> >> >>>>> > > > >> > >> >> >>> plus
>> >> >>>>> > > > >> > >> >> >>> > >> > with the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > argument
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > String
>> keys,
>> >> >>>> this is
>> >> >>>>> > around
>> >> >>>>> > > > >> > integrating
>> >> >>>>> > > > >> > >> >> into an
>> >> >>>>> > > > >> > >> >> >>> > >> > existing
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > eco-system.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > As many
>> >> other
>> >> >>>> systems use
>> >> >>>>> > > String
>> >> >>>>> > > > >> based
>> >> >>>>> > > > >> > >> >> headers
>> >> >>>>> > > > >> > >> >> >>> > >> (Flume,
>> >> >>>>> > > > >> > >> >> >>> > >> > JMS)
>> >> >>>>> > > > >> > >> >> >>> > >> > > it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > it
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > much
>> easier
>> >> for
>> >> >>>> these to
>> >> >>>>> > be
>> >> >>>>> > > > >> > >> >> >>> > incorporated/integrated
>> >> >>>>> > > > >> > >> >> >>> > >> > into.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > How with
>> Int
>> >> >>>> based
>> >> >>>>> > headers
>> >> >>>>> > > could
>> >> >>>>> > > > we
>> >> >>>>> > > > >> > >> provide
>> >> >>>>> > > > >> > >> >> a
>> >> >>>>> > > > >> > >> >> >>> > >> > way/guidence to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> make
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > this
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> integration
>> >> >>>> simple /
>> >> >>>>> > easy with
>> >> >>>>> > > > >> > transition
>> >> >>>>> > > > >> > >> >> flows
>> >> >>>>> > > > >> > >> >> >>> > over
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > kafka?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * tough
>> luck
>> >> >>>> buddy
>> >> >>>>> > you're on
>> >> >>>>> > > your
>> >> >>>>> > > > >> own
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * simply
>> >> hash
>> >> >>>> the string
>> >> >>>>> > into
>> >> >>>>> > > int
>> >> >>>>> > > > >> code
>> >> >>>>> > > > >> > >> and
>> >> >>>>> > > > >> > >> >> hope
>> >> >>>>> > > > >> > >> >> >>> > for
>> >> >>>>> > > > >> > >> >> >>> > >> no
>> >> >>>>> > > > >> > >> >> >>> > >> > > collisions
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > (how
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > convert
>> back
>> >> >>>> though?)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > * http2
>> >> style
>> >> >> as
>> >> >>>>> > mentioned by
>> >> >>>>> > > > nacho.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > cheers,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Mike
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > ______________________________
>> >> >>>>> > > > >> > __________
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > From:
>> radai
>> >> <
>> >> >>>>> > > > >> > radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Sent:
>> >> >> Wednesday,
>> >> >>>>> > November 9,
>> >> >>>>> > > 2016
>> >> >>>>> > > > >> > 8:12 AM
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > To:
>> >> >>>> dev@kafka.apache.org
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Subject:
>> Re:
>> >> >>>> [DISCUSS]
>> >> >>>>> > KIP-82 -
>> >> >>>>> > > > Add
>> >> >>>>> > > > >> > >> Record
>> >> >>>>> > > > >> > >> >> >>> Headers
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > thinking
>> >> about
>> >> >>>> it some
>> >> >>>>> > more,
>> >> >>>>> > > the
>> >> >>>>> > > > >> best
>> >> >>>>> > > > >> > >> way to
>> >> >>>>> > > > >> > >> >> >>> > transmit
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > remapping
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > data to
>> >> >>>> consumers would
>> >> >>>>> > be to
>> >> >>>>> > > > put it
>> >> >>>>> > > > >> > in
>> >> >>>>> > > > >> > >> the
>> >> >>>>> > > > >> > >> >> MD
>> >> >>>>> > > > >> > >> >> >>> > >> response
>> >> >>>>> > > > >> > >> >> >>> > >> > > payload,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> so
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > maybe
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > it should
>> be
>> >> >>>> discussed
>> >> >>>>> > now.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > On Wed,
>> Nov
>> >> 9,
>> >> >>>> 2016 at
>> >> >>>>> > 12:09
>> >> >>>>> > > AM,
>> >> >>>>> > > > >> > radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >>
>> >> >> radai.rosenblatt@gmail.com
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > im not
>> >> >> opposed
>> >> >>>> to the
>> >> >>>>> > idea of
>> >> >>>>> > > > >> > namespace
>> >> >>>>> > > > >> > >> >> >>> mapping.
>> >> >>>>> > > > >> > >> >> >>> > >> all
>> >> >>>>> > > > >> > >> >> >>> > >> > im
>> >> >>>>> > > > >> > >> >> >>> > >> > > saying
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> is
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > its
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > not part
>> >> of
>> >> >>>> the "mvp"
>> >> >>>>> > and,
>> >> >>>>> > > > since
>> >> >>>>> > > > >> it
>> >> >>>>> > > > >> > >> >> requires
>> >> >>>>> > > > >> > >> >> >>> no
>> >> >>>>> > > > >> > >> >> >>> > >> wire
>> >> >>>>> > > > >> > >> >> >>> > >> > format
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > change,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > can
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > always
>> be
>> >> >>>> added later.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also,
>> its
>> >> not
>> >> >>>> as
>> >> >>>>> > simple as
>> >> >>>>> > > just
>> >> >>>>> > > > >> > >> >> configuring
>> >> >>>>> > > > >> > >> >> >>> MM
>> >> >>>>> > > > >> > >> >> >>> > to
>> >> >>>>> > > > >> > >> >> >>> > >> do
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > transform:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > lets
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > say i've
>> >> >>>> implemented
>> >> >>>>> > large
>> >> >>>>> > > > message
>> >> >>>>> > > > >> > >> >> support as
>> >> >>>>> > > > >> > >> >> >>> > >> > {666,1} and
>> >> >>>>> > > > >> > >> >> >>> > >> > > on
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> some
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > mirror
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > target
>> >> >> cluster
>> >> >>>> its been
>> >> >>>>> > > > remapped
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> >> {999,1}.
>> >> >>>>> > > > >> > >> >> >>> the
>> >> >>>>> > > > >> > >> >> >>> > >> > consumer
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> plugin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > code
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > also
>> need
>> >> to
>> >> >>>> be told
>> >> >>>>> > to look
>> >> >>>>> > > > for
>> >> >>>>> > > > >> the
>> >> >>>>> > > > >> > >> large
>> >> >>>>> > > > >> > >> >> >>> > message
>> >> >>>>> > > > >> > >> >> >>> > >> > "part X
>> >> >>>>> > > > >> > >> >> >>> > >> > > of
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> Y"
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > header
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > under
>> >> >> {999,1}.
>> >> >>>> doable,
>> >> >>>>> > but
>> >> >>>>> > > > tricky.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > > On Tue,
>> >> Nov
>> >> >> 8,
>> >> >>>> 2016 at
>> >> >>>>> > 10:29
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > Gwen
>> >> >>>>> > > > >> > >> >> >>> Shapira <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> gwen@confluent.io
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> While
>> you
>> >> >> can
>> >> >>>> do
>> >> >>>>> > whatever
>> >> >>>>> > > you
>> >> >>>>> > > > >> want
>> >> >>>>> > > > >> > >> with a
>> >> >>>>> > > > >> > >> >> >>> > >> namespace
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > your
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > code,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> what
>> I'd
>> >> >>>> expect is
>> >> >>>>> > for each
>> >> >>>>> > > > app
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> >> >>> namespaces
>> >> >>>>> > > > >> > >> >> >>> > >> > > configurable...
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> So if I
>> >> >>>> accidentally
>> >> >>>>> > used
>> >> >>>>> > > 666
>> >> >>>>> > > > for
>> >> >>>>> > > > >> > my
>> >> >>>>> > > > >> > >> HR
>> >> >>>>> > > > >> > >> >> >>> > >> department,
>> >> >>>>> > > > >> > >> >> >>> > >> > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > still
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > want
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> run
>> >> >> RadaiApp,
>> >> >>>> I can
>> >> >>>>> > config
>> >> >>>>> > > > >> > >> "namespace=42"
>> >> >>>>> > > > >> > >> >> >>> for
>> >> >>>>> > > > >> > >> >> >>> > >> > RadaiApp and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > everything
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> will
>> look
>> >> >>>> normal.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> This
>> >> means
>> >> >>>> you only
>> >> >>>>> > need to
>> >> >>>>> > > > sync
>> >> >>>>> > > > >> > usage
>> >> >>>>> > > > >> > >> >> >>> inside
>> >> >>>>> > > > >> > >> >> >>> > your
>> >> >>>>> > > > >> > >> >> >>> > >> > own
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > organization.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> Still
>> >> hard,
>> >> >>>> but
>> >> >>>>> > somewhat
>> >> >>>>> > > > easier
>> >> >>>>> > > > >> > than
>> >> >>>>> > > > >> > >> >> syncing
>> >> >>>>> > > > >> > >> >> >>> > with
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > entire
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > world.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> On Tue,
>> >> Nov
>> >> >>>> 8, 2016
>> >> >>>>> > at 10:07
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > >> radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> >>>> radai.rosenblatt@gmail.com>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > and
>> we
>> >> can
>> >> >>>> start
>> >> >>>>> > with
>> >> >>>>> > > > >> {namespace,
>> >> >>>>> > > > >> > >> id}
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> no
>> >> >>>>> > > > >> > >> >> >>> > >> > re-mapping
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > support
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> always
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > add
>> it
>> >> >>>> later on
>> >> >>>>> > if/when
>> >> >>>>> > > > >> > collisions
>> >> >>>>> > > > >> > >> >> >>> actually
>> >> >>>>> > > > >> > >> >> >>> > >> > happen (i
>> >> >>>>> > > > >> > >> >> >>> > >> > > dont
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > think
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > they'd
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> be
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > a
>> >> >> problem).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > every
>> >> >>>> interested
>> >> >>>>> > party (so
>> >> >>>>> > > > orgs
>> >> >>>>> > > > >> > or
>> >> >>>>> > > > >> > >> >> >>> > individuals)
>> >> >>>>> > > > >> > >> >> >>> > >> > could
>> >> >>>>> > > > >> > >> >> >>> > >> > > then
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > register
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > a
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> prefix
>> >> (0
>> >> >> =
>> >> >>>>> > reserved, 1 =
>> >> >>>>> > > > >> > confluent
>> >> >>>>> > > > >> > >> ...
>> >> >>>>> > > > >> > >> >> >>> 666
>> >> >>>>> > > > >> > >> >> >>> > = me
>> >> >>>>> > > > >> > >> >> >>> > >> > :-) )
>> >> >>>>> > > > >> > >> >> >>> > >> > > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> do
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > whatever
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > the
>> >> 2nd ID
>> >> >>>> - so once
>> >> >>>>> > > > linkedin
>> >> >>>>> > > > >> > >> >> registers,
>> >> >>>>> > > > >> > >> >> >>> say
>> >> >>>>> > > > >> > >> >> >>> > 3,
>> >> >>>>> > > > >> > >> >> >>> > >> > then
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> linkedin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > devs
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > are
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> free
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > to
>> use
>> >> {3,
>> >> >>>> *} with a
>> >> >>>>> > > > reasonable
>> >> >>>>> > > > >> > >> >> >>> expectation
>> >> >>>>> > > > >> > >> >> >>> > to
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > collide
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > anything
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > else.
>> >> >>>> further
>> >> >>>>> > partitioning
>> >> >>>>> > > > of
>> >> >>>>> > > > >> > that *
>> >> >>>>> > > > >> > >> >> >>> becomes
>> >> >>>>> > > > >> > >> >> >>> > >> > linkedin's
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > problem,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > but
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> "upstream
>> >> >>>>> > registration"
>> >> >>>>> > > of a
>> >> >>>>> > > > >> > >> namespace
>> >> >>>>> > > > >> > >> >> >>> only
>> >> >>>>> > > > >> > >> >> >>> > has
>> >> >>>>> > > > >> > >> >> >>> > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > happen
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> > On
>> Tue,
>> >> >> Nov
>> >> >>>> 8, 2016
>> >> >>>>> > at
>> >> >>>>> > > 9:03
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > >> James
>> >> >>>>> > > > >> > >> >> >>> Cheng <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > >
>> >> wushujames@gmail.com
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>> >> Nov
>> >> >>>> 8, 2016,
>> >> >>>>> > at 5:54
>> >> >>>>> > > > PM,
>> >> >>>>> > > > >> > Gwen
>> >> >>>>> > > > >> > >> >> >>> Shapira <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > gwen@confluent.io
>> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> Thank
>> >> >>>> you so
>> >> >>>>> > much for
>> >> >>>>> > > > this
>> >> >>>>> > > > >> > clear
>> >> >>>>> > > > >> > >> and
>> >> >>>>> > > > >> > >> >> >>> fair
>> >> >>>>> > > > >> > >> >> >>> > >> > summary of
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > arguments.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> I'm
>> >> in
>> >> >>>> favor of
>> >> >>>>> > ints.
>> >> >>>>> > > > Not a
>> >> >>>>> > > > >> > >> >> >>> deal-breaker,
>> >> >>>>> > > > >> > >> >> >>> > but
>> >> >>>>> > > > >> > >> >> >>> > >> > in
>> >> >>>>> > > > >> > >> >> >>> > >> > > favor.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> Even
>> >> >>>> more in
>> >> >>>>> > favor of
>> >> >>>>> > > > >> Magnus's
>> >> >>>>> > > > >> > >> >> >>> > decentralized
>> >> >>>>> > > > >> > >> >> >>> > >> > > suggestion
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > Roger's
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> tweak:
>> >> >>>> add a
>> >> >>>>> > namespace
>> >> >>>>> > > > for
>> >> >>>>> > > > >> > >> headers.
>> >> >>>>> > > > >> > >> >> >>> This
>> >> >>>>> > > > >> > >> >> >>> > will
>> >> >>>>> > > > >> > >> >> >>> > >> > allow
>> >> >>>>> > > > >> > >> >> >>> > >> > > each
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > just
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> use
>> >> >>>> whatever IDs
>> >> >>>>> > it
>> >> >>>>> > > wants
>> >> >>>>> > > > >> > >> >> internally,
>> >> >>>>> > > > >> > >> >> >>> and
>> >> >>>>> > > > >> > >> >> >>> > >> then
>> >> >>>>> > > > >> > >> >> >>> > >> > let
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > admin
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> deploying
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> the
>> >> app
>> >> >>>> figure
>> >> >>>>> > out an
>> >> >>>>> > > > >> > available
>> >> >>>>> > > > >> > >> >> >>> namespace
>> >> >>>>> > > > >> > >> >> >>> > ID
>> >> >>>>> > > > >> > >> >> >>> > >> > for the
>> >> >>>>> > > > >> > >> >> >>> > >> > > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > live
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > So
>> >> >>>>> > > > >> > io.confluent.schema-registry
>> >> >>>>> > > > >> > >> can
>> >> >>>>> > > > >> > >> >> be
>> >> >>>>> > > > >> > >> >> >>> > >> > namespace
>> >> >>>>> > > > >> > >> >> >>> > >> > > 0x01 on
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> my
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> deployment
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> and
>> >> >> 0x57
>> >> >>>> on
>> >> >>>>> > yours, and
>> >> >>>>> > > > the
>> >> >>>>> > > > >> > poor
>> >> >>>>> > > > >> > >> guys
>> >> >>>>> > > > >> > >> >> >>> > >> > developing the
>> >> >>>>> > > > >> > >> >> >>> > >> > > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > don't
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > need
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > to
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> worry
>> >> >>>> about that.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> Gwen,
>> >> if
>> >> >> I
>> >> >>>>> > understand
>> >> >>>>> > > your
>> >> >>>>> > > > >> > example
>> >> >>>>> > > > >> > >> >> >>> right, an
>> >> >>>>> > > > >> > >> >> >>> > >> > > application
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > deployer
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > might
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> decide to
>> >> >>>> use 0x01
>> >> >>>>> > in one
>> >> >>>>> > > > >> > >> deployment,
>> >> >>>>> > > > >> > >> >> and
>> >> >>>>> > > > >> > >> >> >>> > that
>> >> >>>>> > > > >> > >> >> >>> > >> > means
>> >> >>>>> > > > >> > >> >> >>> > >> > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > once
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> message
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> is
>> >> >> written
>> >> >>>> into the
>> >> >>>>> > > > broker, it
>> >> >>>>> > > > >> > >> will be
>> >> >>>>> > > > >> > >> >> >>> > saved on
>> >> >>>>> > > > >> > >> >> >>> > >> > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > broker
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > with
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> specific
>> >> >>>> namespace
>> >> >>>>> > > (0x01).
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> If
>> you
>> >> >>>> were to
>> >> >>>>> > mirror
>> >> >>>>> > > that
>> >> >>>>> > > > >> > message
>> >> >>>>> > > > >> > >> >> into
>> >> >>>>> > > > >> > >> >> >>> > another
>> >> >>>>> > > > >> > >> >> >>> > >> > > cluster,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > 0x01
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > would
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> accompany
>> >> >>>> the
>> >> >>>>> > message,
>> >> >>>>> > > > right?
>> >> >>>>> > > > >> > What
>> >> >>>>> > > > >> > >> if
>> >> >>>>> > > > >> > >> >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > deployers of
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > same
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > app
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> other
>> >> >>>> cluster uses
>> >> >>>>> > 0x57?
>> >> >>>>> > > > They
>> >> >>>>> > > > >> > won't
>> >> >>>>> > > > >> > >> >> >>> > understand
>> >> >>>>> > > > >> > >> >> >>> > >> > each
>> >> >>>>> > > > >> > >> >> >>> > >> > > other?
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> I'm
>> >> not
>> >> >>>> sure
>> >> >>>>> > that's an
>> >> >>>>> > > > >> avoidable
>> >> >>>>> > > > >> > >> >> >>> problem. I
>> >> >>>>> > > > >> > >> >> >>> > >> > think it
>> >> >>>>> > > > >> > >> >> >>> > >> > > simply
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > means
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> in
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> order
>> >> to
>> >> >>>> share
>> >> >>>>> > data, you
>> >> >>>>> > > > have
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> also
>> >> >>>>> > > > >> > >> >> >>> have a
>> >> >>>>> > > > >> > >> >> >>> > >> > shared
>> >> >>>>> > > > >> > >> >> >>> > >> > > (agreed
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > upon)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>> understanding of
>> >> >>>>> > what the
>> >> >>>>> > > > >> > >> namespaces
>> >> >>>>> > > > >> > >> >> >>> mean.
>> >> >>>>> > > > >> > >> >> >>> > >> Which
>> >> >>>>> > > > >> > >> >> >>> > >> > I
>> >> >>>>> > > > >> > >> >> >>> > >> > > think
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > makes
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > sense,
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> because
>> >> >> the
>> >> >>>>> > alternate
>> >> >>>>> > > > (sharing
>> >> >>>>> > > > >> > >> >> *nothing*
>> >> >>>>> > > > >> > >> >> >>> at
>> >> >>>>> > > > >> > >> >> >>> > >> all)
>> >> >>>>> > > > >> > >> >> >>> > >> > would
>> >> >>>>> > > > >> > >> >> >>> > >> > > mean
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > that
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > there
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> would
>> >> be
>> >> >>>> no way to
>> >> >>>>> > > > understand
>> >> >>>>> > > > >> > each
>> >> >>>>> > > > >> > >> >> other.
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> -James
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> Gwen
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> > On
>> >> Tue,
>> >> >>>> Nov 8,
>> >> >>>>> > 2016 at
>> >> >>>>> > > > 4:23
>> >> >>>>> > > > >> > PM,
>> >> >>>>> > > > >> > >> >> radai <
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > >
>> >> >>>> radai.rosenblatt@gmail.com
>> >> >>>>> > >
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >>
>> wrote:
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>> +1
>> >> for
>> >> >>>> sean's
>> >> >>>>> > > document.
>> >> >>>>> > > > it
>> >> >>>>> > > > >> > >> covers
>> >> >>>>> > > > >> > >> >> >>> pretty
>> >> >>>>> > > > >> > >> >> >>> > >> much
>> >> >>>>> > > > >> > >> >> >>> > >> > all
>> >> >>>>> > > > >> > >> >> >>> > >> > > the
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > trade-offs
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > and
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>> >> >> provides
>> >> >>>>> > concrete
>> >> >>>>> > > > figures
>> >> >>>>> > > > >> to
>> >> >>>>> > > > >> > >> argue
>> >> >>>>> > > > >> > >> >> >>> about
>> >> >>>>> > > > >> > >> >> >>> > :-)
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >> >> >>
>> >> >>>> (nit-picking -
>> >> >>>>> > used
>> >> >>>>> > > the
>> >> >>>>> > > > >> same
>> >> >>>>> > > > >> > >> xkcd
>> >> >>>>> > > > >> > >> >> >>> twice,
>> >> >>>>> > > > >> > >> >> >>> > >> also
>> >> >>>>> > > > >> > >> >> >>> > >> > trove
>> >> >>>>> > > > >> > >> >> >>> > >> > > has
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > been
>> >> >>>>> > > > >> > >> >> >>> > >> > > >> > > > > > >>
>> >> superceded
>> >> >>>>> > > > >> > >> >
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >> > --
>> >> >>>>> > > > >> > Gwen Shapira
>> >> >>>>> > > > >> > Product Manager | Confluent
>> >> >>>>> > > > >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
>> >> >>>>> > > > >> > Follow us: Twitter | blog
>> >> >>>>> > > > >> >
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> --
>> >> >>>>> > > > >> *Todd Palino*
>> >> >>>>> > > > >> Staff Site Reliability Engineer
>> >> >>>>> > > > >> Data Infrastructure Streaming
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >>
>> >> >>>>> > > > >> linkedin.com/in/toddpalino
>> >> >>>>> > > > >>
>> >> >>>>> > > >
>> >> >>>>> > > >
>> >> >>>>> > > >
>> >> >>>>> > > > --
>> >> >>>>> > > > Gwen Shapira
>> >> >>>>> > > > Product Manager | Confluent
>> >> >>>>> > > > 650.450.2760 <(650)%20450-2760> | @gwenshap
>> >> >>>>> > > > Follow us: Twitter | blog
>> >> >>>>> > > >
>> >> >>>>> > >
>> >> >>>>> >
>> >> >>>>> >
>> >> >>>>> > The information contained in this email is strictly
>> >> confidential
>> >> >>>> and for
>> >> >>>>> > the use of the addressee only, unless otherwise indicated.
>> If
>> >> you
>> >> >>>> are not
>> >> >>>>> > the intended recipient, please do not read, copy, use or
>> >> disclose
>> >> >>>> to others
>> >> >>>>> > this message or any attachment. Please also notify the
>> sender
>> >> by
>> >> >>>> replying
>> >> >>>>> > to this email or by telephone (+44(020 7896 0011) and then
>> >> delete
>> >> >>>> the email
>> >> >>>>> > and any copies of it. Opinions, conclusion (etc) that do not
>> >> >>>> relate to the
>> >> >>>>> > official business of this company shall be understood as
>> >> neither
>> >> >>>> given nor
>> >> >>>>> > endorsed by it. IG is a trading name of IG Markets Limited
>> (a
>> >> >>>> company
>> >> >>>>> > registered in England and Wales, company number 04008957)
>> and
>> >> IG
>> >> >>>> Index
>> >> >>>>> > Limited (a company registered in England and Wales, company
>> >> >> number
>> >> >>>>> > 01190902). Registered address at Cannon Bridge House, 25
>> >> Dowgate
>> >> >>>> Hill,
>> >> >>>>> > London EC4R 2YA. Both IG Markets Limited (register number
>> >> 195355)
>> >> >>>> and IG
>> >> >>>>> > Index Limited (register number 114059) are authorised and
>> >> >>>> regulated by the
>> >> >>>>> > Financial Conduct Authority.
>> >> >>>>> >
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> The information contained in this email is strictly confidential
>> and
>> >> >> for
>> >> >>>> the use of the addressee only, unless otherwise indicated. If you
>> are
>> >> >> not
>> >> >>>> the intended recipient, please do not read, copy, use or disclose
>> to
>> >> >> others
>> >> >>>> this message or any attachment. Please also notify the sender by
>> >> >> replying
>> >> >>>> to this email or by telephone (+44(020 7896 0011) and then delete
>> the
>> >> >> email
>> >> >>>> and any copies of it. Opinions, conclusion (etc) that do not relate
>> >> to
>> >> >> the
>> >> >>>> official business of this company shall be understood as neither
>> >> given
>> >> >> nor
>> >> >>>> endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> >> >>>> registered in England and Wales, company number 04008957) and IG
>> >> Index
>> >> >>>> Limited (a company registered in England and Wales, company number
>> >> >>>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> >> Hill,
>> >> >>>> London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> >> and IG
>> >> >>>> Index Limited (register number 114059) are authorised and regulated
>> >> by
>> >> >> the
>> >> >>>> Financial Conduct Authority.
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>>
--
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog