You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Guozhang Wang <wa...@gmail.com> on 2017/06/27 18:08:11 UTC
Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging
Hello all,
I want to highlight some changes in the implementation design of KIP-98
since the proposal is voted in February.
As many of you know, exactly-once support is one of the biggest features we
have added in the 0.11.0.0 release. So we have been carefully think through
all the corner cases during the implementation phase, and indeed we have
dug some devils in the details and potential optimizations that have not
been covered in the proposal. Corresponding changes have been updated in
the design doc
<https://docs.google.com/document/d/11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/edit#heading=h.xq0ee1vnpz4o>,
and here is a short summary:
1.
We have collapsed the two types of transactional log messages
TransactionalId
mapping and Transaction metadata into a single type with the format as:
TransactionalId -> {pid, epoch, transaction_timeout, transaction_state,
[topic partition
<https://confluentinc.atlassian.net/wiki/display/KAFKA/partition> ],
transaction_start_timestamp, transaction_last_update_timestamp }
If there is no transaction ever produced for a certain transactionalId,
its transaction_state field will be Empty. Also inside transactional
coordinator a single memory map for the above transaction metadata will be
kept in in cache. This change simplifies the concurrency control between
the transactional id management and the transaction state management and
the log format on the transaction topic. For details, please read this PR
<https://github.com/apache/kafka/pull/2846>.
2.
We added a new exception type in o.a.k.common.errors for invalid
transaction timeout values namely InvalidTxnTimeoutException. This
error is used to define the case of invalid user-specified transaction
timeout value, which should be treated as a fatal error and be
distinguished with other retriable error cases. For details, please read
this PR <https://github.com/apache/kafka/pull/2825>; we have also added
another re-triable exception named ConcurrentTransactionsException to
guard old clients with the same transactionalId to override current ongoing
transactions.
3.
We stopped wrapping the internal retriable exception encountered during
offset commits in RetriableOffsetCommitException. There isn't a really
good reason to do this, since the user will not handle each individual
exception differently: they will just retry anyway. By only returning the
wrapped RetriableOffsetCommitException we can make the client-broker
protocol simpler, and also enable broker side to change things underneath
with more flexibility. For details, please read this JIRA
<https://issues.apache.org/jira/browse/KAFKA-5052>.
4.
We have extended WriteTxnMarkerRequest to be able to contain multiple
markers for multiple PIDs with a single request. By doing this we can have
less round trips between the transaction coordinator and brokers for
sending markers for multiple PIDs. For details, please read this PR
<https://github.com/apache/kafka/pull/2797> and this PR
<https://github.com/apache/kafka/pull/2964>.
5.
We enforced acks=all and max.in.flight.requests.per.connection=1 on the
producer client when idempotence is enabled. This is not necessary though:
with some additional work on the broker side we can guarantee idempotency
on the producer even with max.in.flight.requests.per.connection>1. For
details please read this JIRA
<https://issues.apache.org/jira/browse/KAFKA-5494>.
Guozhang
On Fri, Feb 24, 2017 at 7:44 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Correction: Non-Binding +0 count is 2.
>
> On Fri, Feb 24, 2017 at 7:44 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
>> Hello all,
>>
>> Thanks a billion for your votes and comments. We have collected the
>> following votes so far (+1 from myself as well):
>>
>> Binding +1: 9
>>
>> Gwen Shapira
>> Jay Kreps
>> Becket Qin
>> Jun Rao
>> Ismael Juma
>> Jason Gustafson
>> Sriram Subramanian
>> Joel Koshy
>> Guozhang Wang
>>
>> Binding -1: 0
>>
>> Non-Binding +1: 4
>>
>> Eno Thereska
>> Matthias J. Sax
>> Bill Bejeck
>> Rajini Sivaram
>>
>> Non Binding -1: 0
>>
>> Non binding +0: 0
>>
>> Tom Crayford
>> Michael Pearce
>>
>>
>> Based on the above result, I am now closing this voting thread as
>> accepted.
>>
>> Thanks again for your comments.
>>
>> Guozhang
>>
>>
>> On Fri, Feb 24, 2017 at 11:50 AM, Sriram Subramanian <ra...@confluent.io>
>> wrote:
>>
>>> +1. Great work in driving this to a consensus. Lots of good constructive
>>> conversations.
>>>
>>> On Fri, Feb 24, 2017 at 11:48 AM, Jason Gustafson <ja...@confluent.io>
>>> wrote:
>>>
>>> > +1 from me (duh). Thanks to all the reviewers. The KIP has been much
>>> > improved because of it!
>>> >
>>> > -Jason
>>> >
>>> > On Wed, Feb 22, 2017 at 8:48 AM, Ismael Juma <is...@juma.me.uk>
>>> wrote:
>>> >
>>> > > Great work on the proposal and iterating on it based on community
>>> > feedback.
>>> > > As Jun (and others) said, it's likely that minor changes will happen
>>> as
>>> > the
>>> > > PR is reviewed and additional testing takes place since this is a
>>> > > significant change.
>>> > >
>>> > > I am +1 (binding) on the proposal without optional keys and values to
>>> > keep
>>> > > things consistent. If we show during performance testing that this is
>>> > > worthwhile, we can update the proposal.
>>> > >
>>> > > Ismael
>>> > >
>>> > > On Tue, Feb 21, 2017 at 6:23 PM, Jun Rao <ju...@confluent.io> wrote:
>>> > >
>>> > > > It seems that it's simpler and more consistent to avoid optional
>>> keys
>>> > and
>>> > > > values. Not sure if it's worth squeezing every byte at the expense
>>> of
>>> > > > additional complexity. Other than that, +1 from me.
>>> > > >
>>> > > > Also, since this is a large KIP, minor changes may arise as we
>>> start
>>> > the
>>> > > > implementation. It would be good if we can keep the community
>>> posted of
>>> > > > those changes, if any.
>>> > > >
>>> > > > Thanks,
>>> > > >
>>> > > > Jun
>>> > > >
>>> > > > On Tue, Feb 21, 2017 at 4:33 PM, Michael Pearce <
>>> Michael.Pearce@ig.com
>>> > >
>>> > > > wrote:
>>> > > >
>>> > > > > If the argument and objective within this KIP is to keep the
>>> overhead
>>> > > of
>>> > > > > the protocol as small as possible and remove redundancy, and
>>> every
>>> > byte
>>> > > > is
>>> > > > > being counted and the introduction of varInts, then it would make
>>> > sense
>>> > > > to
>>> > > > > use attributes to me.
>>> > > > >
>>> > > > >
>>> > > > > On 22/02/2017, 00:14, "Jason Gustafson" <ja...@confluent.io>
>>> wrote:
>>> > > > >
>>> > > > > Done. I've left the key and value as optional since we may
>>> not
>>> > have
>>> > > > > reached
>>> > > > > consensus on whether to use attributes or not. Perhaps we
>>> should
>>> > > just
>>> > > > > keep
>>> > > > > it simple and not do it? The benefit seems small.
>>> > > > >
>>> > > > > -Jason
>>> > > > >
>>> > > > > On Tue, Feb 21, 2017 at 4:05 PM, Michael Pearce <
>>> > > > Michael.Pearce@ig.com
>>> > > > > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Ok, no worries, can you add it back ValueLen on this KIP,
>>> and
>>> > > > update
>>> > > > > the
>>> > > > > > doc, then we can work from that ☺
>>> > > > > >
>>> > > > > > Cheers
>>> > > > > > Mike
>>> > > > > >
>>> > > > > > On 22/02/2017, 00:02, "Jason Gustafson" <
>>> jason@confluent.io>
>>> > > > wrote:
>>> > > > > >
>>> > > > > > I feel it was a little odd to leave out the value
>>> length
>>> > > > anyway,
>>> > > > > so I
>>> > > > > > would
>>> > > > > > rather add it back and put headers at the end. This is
>>> more
>>> > > > > consistent
>>> > > > > > with
>>> > > > > > the rest of the Kafka protocol.
>>> > > > > >
>>> > > > > > -Jason
>>> > > > > >
>>> > > > > > On Tue, Feb 21, 2017 at 3:58 PM, Michael Pearce <
>>> > > > > Michael.Pearce@ig.com
>>> > > > > > >
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Or we keep as is (valuelen removed), and headers are
>>> > added
>>> > > > with
>>> > > > > > headers
>>> > > > > > > length..
>>> > > > > > >
>>> > > > > > > On 21/02/2017, 23:38, "Apurva Mehta" <
>>> > apurva@confluent.io>
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > > Right now, we don't need the value length: since
>>> it
>>> > is
>>> > > > the
>>> > > > > last
>>> > > > > > item
>>> > > > > > > in the
>>> > > > > > > message, and we have the message length, we can
>>> > deduce
>>> > > > the
>>> > > > > value
>>> > > > > > > length.
>>> > > > > > > However, if we are adding record headers to the
>>> end,
>>> > we
>>> > > > > would
>>> > > > > > need to
>>> > > > > > > introduce the value length along with that
>>> change.
>>> > > > > > >
>>> > > > > > > On Tue, Feb 21, 2017 at 3:34 PM, Michael Pearce <
>>> > > > > > Michael.Pearce@ig.com
>>> > > > > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > It seems I cannot add comment on the doc.
>>> > > > > > > >
>>> > > > > > > > In the section around the message protocol.
>>> > > > > > > >
>>> > > > > > > > It has stated:
>>> > > > > > > >
>>> > > > > > > > Message =>
>>> > > > > > > > Length => uintVar
>>> > > > > > > > Attributes => int8
>>> > > > > > > > TimestampDelta => intVar
>>> > > > > > > > OffsetDelta => uintVar
>>> > > > > > > > KeyLen => uintVar [OPTIONAL]
>>> > > > > > > > Key => data [OPTIONAL]
>>> > > > > > > > Value => data [OPTIONAL]
>>> > > > > > > >
>>> > > > > > > > Should it not be: (added missing value len)
>>> > > > > > > >
>>> > > > > > > > Message =>
>>> > > > > > > > Length => uintVar
>>> > > > > > > > Attributes => int8
>>> > > > > > > > TimestampDelta => intVar
>>> > > > > > > > OffsetDelta => uintVar
>>> > > > > > > > KeyLen => uintVar [OPTIONAL]
>>> > > > > > > > Key => data [OPTIONAL]
>>> > > > > > > > ValueLen => uintVar [OPTIONAL]
>>> > > > > > > > Value => data [OPTIONAL]
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On 21/02/2017, 23:07, "Joel Koshy" <
>>> > > > jjkoshy.w@gmail.com>
>>> > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > I left a couple of comments/questions
>>> directly
>>> > on
>>> > > > the
>>> > > > > > google-doc
>>> > > > > > > > <https://docs.google.com/document/d/11Jqy_
>>> > > > > > > > GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8>
>>> > > > > > > > - I found it much more tractable for a
>>> proposal
>>> > > of
>>> > > > > this
>>> > > > > > size to
>>> > > > > > > > discuss in
>>> > > > > > > > context within the doc. The permissions on
>>> the
>>> > > doc
>>> > > > > don't
>>> > > > > > let
>>> > > > > > > everyone
>>> > > > > > > > view
>>> > > > > > > > comments, so if there are any material
>>> changes
>>> > > that
>>> > > > > come
>>> > > > > > out of
>>> > > > > > > the
>>> > > > > > > > discussions in those comment threads we can
>>> > > > > summarize here.
>>> > > > > > > >
>>> > > > > > > > Thanks,
>>> > > > > > > >
>>> > > > > > > > Joel
>>> > > > > > > >
>>> > > > > > > > On Mon, Feb 20, 2017 at 4:08 PM, Becket
>>> Qin <
>>> > > > > > > becket.qin@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > Thanks for the explanation, Guozhang.
>>> That
>>> > > makes
>>> > > > > sense.
>>> > > > > > > > >
>>> > > > > > > > > On Sun, Feb 19, 2017 at 7:28 PM, Guozhang
>>> > Wang
>>> > > <
>>> > > > > > > wangguoz@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > Thanks Becket.
>>> > > > > > > > > >
>>> > > > > > > > > > Actually sequence is associated with a
>>> > > message,
>>> > > > > not a
>>> > > > > > > message set.
>>> > > > > > > > For
>>> > > > > > > > > > example if a message set sent by
>>> producer
>>> > > > > contains 100
>>> > > > > > > messages,
>>> > > > > > > > and the
>>> > > > > > > > > > first message's sequence is 5, then the
>>> > last
>>> > > > > message's
>>> > > > > > > sequence
>>> > > > > > > > number
>>> > > > > > > > > > would be 104, and the next message
>>> set's
>>> > > first
>>> > > > > > sequence is
>>> > > > > > > > expected to be
>>> > > > > > > > > > 105.
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > Guozhang
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > On Sun, Feb 19, 2017 at 4:48 PM, Becket
>>> > Qin <
>>> > > > > > > becket.qin@gmail.com>
>>> > > > > > > > > wrote:
>>> > > > > > > > > >
>>> > > > > > > > > > > +1. Thanks for the great work on the
>>> KIP!
>>> > > > > > > > > > >
>>> > > > > > > > > > > I have only one minor question, in
>>> the
>>> > wiki
>>> > > > > (and the
>>> > > > > > doc)
>>> > > > > > > the new
>>> > > > > > > > > message
>>> > > > > > > > > > > set format has a "FirstSequence"
>>> field,
>>> > > > should
>>> > > > > it
>>> > > > > > just be
>>> > > > > > > > "Sequence" if
>>> > > > > > > > > > the
>>> > > > > > > > > > > sequence is always associated with a
>>> > > message
>>> > > > > set?
>>> > > > > > > > > > >
>>> > > > > > > > > > > On Fri, Feb 17, 2017 at 3:28 AM,
>>> Michael
>>> > > > > Pearce <
>>> > > > > > > > Michael.Pearce@ig.com
>>> > > > > > > > > >
>>> > > > > > > > > > > wrote:
>>> > > > > > > > > > >
>>> > > > > > > > > > > > +0
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > I think need some unified
>>> agreement on
>>> > > the
>>> > > > > VarInts.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Would this also change in all other
>>> > > area’s
>>> > > > > of the
>>> > > > > > > protocol,
>>> > > > > > > > e.g.
>>> > > > > > > > > value
>>> > > > > > > > > > > and
>>> > > > > > > > > > > > key length in message protocol, to
>>> keep
>>> > > > this
>>> > > > > > uniform
>>> > > > > > > across all
>>> > > > > > > > > > protocols
>>> > > > > > > > > > > > going forwards?
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > On 17/02/2017, 00:23, "Apurva
>>> Mehta" <
>>> > > > > > > apurva@confluent.io>
>>> > > > > > > > wrote:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Hi Jun,
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Thanks for the reply. Comments
>>> > > inline.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > On Thu, Feb 16, 2017 at 2:29
>>> PM,
>>> > Jun
>>> > > > Rao
>>> > > > > <
>>> > > > > > > jun@confluent.io
>>> > > > > > > > >
>>> > > > > > > > > wrote:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > Hi, Apurva,
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Thanks for the reply. A
>>> couple of
>>> > > > > comment
>>> > > > > > below.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > On Wed, Feb 15, 2017 at 9:45
>>> PM,
>>> > > > Apurva
>>> > > > > > Mehta <
>>> > > > > > > > > > apurva@confluent.io
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > wrote:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > > Hi Jun,
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > Answers inline:
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > 210. Pid snapshots: Is the
>>> > number
>>> > > > of
>>> > > > > pid
>>> > > > > > snapshot
>>> > > > > > > > > configurable
>>> > > > > > > > > > or
>>> > > > > > > > > > > > > hardcoded
>>> > > > > > > > > > > > > > > with 2? When do we
>>> decide to
>>> > > roll
>>> > > > > a new
>>> > > > > > > snapshot?
>>> > > > > > > > Based on
>>> > > > > > > > > > > time,
>>> > > > > > > > > > > > byte,
>>> > > > > > > > > > > > > or
>>> > > > > > > > > > > > > > > offset? Is that
>>> configurable
>>> > > too?
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > When a replica becomes a
>>> > follower,
>>> > > we
>>> > > > > do a
>>> > > > > > bit log
>>> > > > > > > > truncation.
>>> > > > > > > > > > > > Having an
>>> > > > > > > > > > > > > older snapshot allows us to
>>> > recover
>>> > > > the
>>> > > > > > > PID->sequence
>>> > > > > > > > mapping
>>> > > > > > > > > > much
>>> > > > > > > > > > > > quicker
>>> > > > > > > > > > > > > than rescanning the whole
>>> log.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > This is a good point. I have
>>> > updated
>>> > > > the
>>> > > > > doc
>>> > > > > > with a
>>> > > > > > > more
>>> > > > > > > > detailed
>>> > > > > > > > > > > > proposal.
>>> > > > > > > > > > > > Essentially, snapshots will be
>>> > > created
>>> > > > > on a
>>> > > > > > periodic
>>> > > > > > > > basis. A
>>> > > > > > > > > > > > reasonable
>>> > > > > > > > > > > > period would be every 30 or 60
>>> > > seconds.
>>> > > > > We
>>> > > > > > will keep
>>> > > > > > > at
>>> > > > > > > > most 2
>>> > > > > > > > > > copies
>>> > > > > > > > > > > > of
>>> > > > > > > > > > > > the snapshot file. With this
>>> setup,
>>> > > we
>>> > > > > would
>>> > > > > > have to
>>> > > > > > > > replay at
>>> > > > > > > > > most
>>> > > > > > > > > > > 60
>>> > > > > > > > > > > > or
>>> > > > > > > > > > > > 120 seconds of the log in the
>>> event
>>> > > of
>>> > > > > log
>>> > > > > > truncation
>>> > > > > > > > during
>>> > > > > > > > > leader
>>> > > > > > > > > > > > failover.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > If we need to make any of this
>>> > > > > configurable,
>>> > > > > > we can
>>> > > > > > > expose
>>> > > > > > > > a
>>> > > > > > > > > config
>>> > > > > > > > > > > in
>>> > > > > > > > > > > > the
>>> > > > > > > > > > > > future. It would be easier to
>>> add a
>>> > > > > config we
>>> > > > > > need
>>> > > > > > > than
>>> > > > > > > > remove
>>> > > > > > > > > one
>>> > > > > > > > > > > with
>>> > > > > > > > > > > > marginal utility.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > 211. I am wondering if we
>>> > > should
>>> > > > > store
>>> > > > > > > > ExpirationTime in
>>> > > > > > > > > the
>>> > > > > > > > > > > > producer
>>> > > > > > > > > > > > > > > transactionalId mapping
>>> > message
>>> > > > as
>>> > > > > we do
>>> > > > > > in the
>>> > > > > > > > producer
>>> > > > > > > > > > > > transaction
>>> > > > > > > > > > > > > > status
>>> > > > > > > > > > > > > > > message. If a producer
>>> only
>>> > > calls
>>> > > > > > > > initTransactions(), but
>>> > > > > > > > > > never
>>> > > > > > > > > > > > > publishes
>>> > > > > > > > > > > > > > > any data, we still want
>>> to be
>>> > > > able
>>> > > > > to
>>> > > > > > expire
>>> > > > > > > and
>>> > > > > > > > remove the
>>> > > > > > > > > > > > producer
>>> > > > > > > > > > > > > > > transactionalId mapping
>>> > > message.
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > Actually, the document was
>>> > > > > inaccurate. The
>>> > > > > > > > transactionalId
>>> > > > > > > > > will
>>> > > > > > > > > > > be
>>> > > > > > > > > > > > > expired
>>> > > > > > > > > > > > > > only if there is no active
>>> > > > > transaction,
>>> > > > > > and the
>>> > > > > > > age of
>>> > > > > > > > the
>>> > > > > > > > > last
>>> > > > > > > > > > > > > transaction
>>> > > > > > > > > > > > > > with that transactionalId
>>> is
>>> > > older
>>> > > > > than the
>>> > > > > > > > transactioanlId
>>> > > > > > > > > > > > expiration
>>> > > > > > > > > > > > > > time. With these semantics,
>>> > > storing
>>> > > > > the
>>> > > > > > > expiration
>>> > > > > > > > time in
>>> > > > > > > > > the
>>> > > > > > > > > > > > > > transactionalId mapping
>>> message
>>> > > > > won't be
>>> > > > > > useful,
>>> > > > > > > since
>>> > > > > > > > the
>>> > > > > > > > > > > > expiration
>>> > > > > > > > > > > > > time
>>> > > > > > > > > > > > > > is a moving target based on
>>> > > > > transaction
>>> > > > > > activity.
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > I have updated the doc
>>> with a
>>> > > > > > clarification.
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > Currently, the producer
>>> > > > transactionalId
>>> > > > > > mapping
>>> > > > > > > message
>>> > > > > > > > doesn't
>>> > > > > > > > > > > carry
>>> > > > > > > > > > > > > ExpirationTime, but the
>>> producer
>>> > > > > transaction
>>> > > > > > status
>>> > > > > > > > message
>>> > > > > > > > > does.
>>> > > > > > > > > > > > It would
>>> > > > > > > > > > > > > be useful if they are
>>> consistent.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > You are right. The document has
>>> > been
>>> > > > > updated to
>>> > > > > > > remove the
>>> > > > > > > > > > > > ExpirationTime
>>> > > > > > > > > > > > from the transaction status
>>> > messages
>>> > > as
>>> > > > > well.
>>> > > > > > Any
>>> > > > > > > utility
>>> > > > > > > > for
>>> > > > > > > > > this
>>> > > > > > > > > > > > field
>>> > > > > > > > > > > > can be achieved by using the
>>> > > timestamp
>>> > > > > of the
>>> > > > > > message
>>> > > > > > > > itself
>>> > > > > > > > > along
>>> > > > > > > > > > > with
>>> > > > > > > > > > > > another expiration time (like
>>> > > > > transactionalId
>>> > > > > > > expiration
>>> > > > > > > > time,
>>> > > > > > > > > > > > transaction
>>> > > > > > > > > > > > expiration time, etc.).
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Thanks,
>>> > > > > > > > > > > > Apurva
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > The information contained in this
>>> email
>>> > > is
>>> > > > > strictly
>>> > > > > > > > confidential and
>>> > > > > > > > > > for
>>> > > > > > > > > > > > the use of the addressee only,
>>> unless
>>> > > > > otherwise
>>> > > > > > > indicated. If
>>> > > > > > > > you are
>>> > > > > > > > > > not
>>> > > > > > > > > > > > the intended recipient, please do
>>> not
>>> > > read,
>>> > > > > copy,
>>> > > > > > use or
>>> > > > > > > > disclose to
>>> > > > > > > > > > > others
>>> > > > > > > > > > > > this message or any attachment.
>>> Please
>>> > > also
>>> > > > > notify
>>> > > > > > the
>>> > > > > > > sender
>>> > > > > > > > by
>>> > > > > > > > > > replying
>>> > > > > > > > > > > > to this email or by telephone
>>> (+44(020
>>> > > 7896
>>> > > > > 0011)
>>> > > > > > and
>>> > > > > > > then
>>> > > > > > > > delete the
>>> > > > > > > > > > > email
>>> > > > > > > > > > > > and any copies of it. Opinions,
>>> > > conclusion
>>> > > > > (etc)
>>> > > > > > that do
>>> > > > > > > not
>>> > > > > > > > relate
>>> > > > > > > > > to
>>> > > > > > > > > > > the
>>> > > > > > > > > > > > official business of this company
>>> shall
>>> > > be
>>> > > > > > understood as
>>> > > > > > > > neither
>>> > > > > > > > > given
>>> > > > > > > > > > > nor
>>> > > > > > > > > > > > endorsed by it. IG is a trading
>>> name of
>>> > > IG
>>> > > > > Markets
>>> > > > > > > Limited (a
>>> > > > > > > > company
>>> > > > > > > > > > > > registered in England and Wales,
>>> > company
>>> > > > > number
>>> > > > > > > 04008957) and
>>> > > > > > > > IG
>>> > > > > > > > > Index
>>> > > > > > > > > > > > Limited (a company registered in
>>> > England
>>> > > > and
>>> > > > > Wales,
>>> > > > > > > company
>>> > > > > > > > number
>>> > > > > > > > > > > > 01190902). Registered address at
>>> Cannon
>>> > > > > Bridge
>>> > > > > > House, 25
>>> > > > > > > > Dowgate
>>> > > > > > > > > Hill,
>>> > > > > > > > > > > > London EC4R 2YA. Both IG Markets
>>> > Limited
>>> > > > > (register
>>> > > > > > number
>>> > > > > > > > 195355) and
>>> > > > > > > > > > IG
>>> > > > > > > > > > > > Index Limited (register number
>>> 114059)
>>> > > are
>>> > > > > > authorised and
>>> > > > > > > > regulated
>>> > > > > > > > > by
>>> > > > > > > > > > > the
>>> > > > > > > > > > > > Financial Conduct Authority.
>>> > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > --
>>> > > > > > > > > > -- Guozhang
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > The information contained in this email is
>>> strictly
>>> > > > > > confidential and
>>> > > > > > > for
>>> > > > > > > > the use of the addressee only, unless otherwise
>>> > > > > indicated. If
>>> > > > > > you
>>> > > > > > > are not
>>> > > > > > > > the intended recipient, please do not read,
>>> copy,
>>> > use
>>> > > > or
>>> > > > > > disclose to
>>> > > > > > > others
>>> > > > > > > > this message or any attachment. Please also
>>> notify
>>> > > the
>>> > > > > sender
>>> > > > > > by
>>> > > > > > > replying
>>> > > > > > > > to this email or by telephone (+44(020 7896
>>> 0011)
>>> > and
>>> > > > > then
>>> > > > > > delete
>>> > > > > > > the email
>>> > > > > > > > and any copies of it. Opinions, conclusion
>>> (etc)
>>> > that
>>> > > > do
>>> > > > > not
>>> > > > > > relate
>>> > > > > > > to the
>>> > > > > > > > official business of this company shall be
>>> > understood
>>> > > > as
>>> > > > > > neither
>>> > > > > > > given nor
>>> > > > > > > > endorsed by it. IG is a trading name of IG
>>> Markets
>>> > > > > Limited (a
>>> > > > > > company
>>> > > > > > > > registered in England and Wales, company number
>>> > > > > 04008957) and
>>> > > > > > IG
>>> > > > > > > Index
>>> > > > > > > > Limited (a company registered in England and
>>> Wales,
>>> > > > > company
>>> > > > > > number
>>> > > > > > > > 01190902). Registered address at Cannon Bridge
>>> > House,
>>> > > > 25
>>> > > > > > Dowgate
>>> > > > > > > Hill,
>>> > > > > > > > London EC4R 2YA. Both IG Markets Limited
>>> (register
>>> > > > number
>>> > > > > > 195355)
>>> > > > > > > and IG
>>> > > > > > > > Index Limited (register number 114059) are
>>> > authorised
>>> > > > and
>>> > > > > > regulated
>>> > > > > > > by the
>>> > > > > > > > Financial Conduct Authority.
>>> > > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > The information contained in this email is strictly
>>> > > > > confidential and
>>> > > > > > for
>>> > > > > > > the use of the addressee only, unless otherwise
>>> > indicated.
>>> > > If
>>> > > > > you
>>> > > > > > are not
>>> > > > > > > the intended recipient, please do not read, copy,
>>> use or
>>> > > > > disclose to
>>> > > > > > others
>>> > > > > > > this message or any attachment. Please also notify
>>> the
>>> > > sender
>>> > > > > by
>>> > > > > > replying
>>> > > > > > > to this email or by telephone (+44(020 7896 0011) and
>>> > then
>>> > > > > delete
>>> > > > > > the email
>>> > > > > > > and any copies of it. Opinions, conclusion (etc)
>>> that do
>>> > > not
>>> > > > > relate
>>> > > > > > to the
>>> > > > > > > official business of this company shall be
>>> understood as
>>> > > > > neither
>>> > > > > > given nor
>>> > > > > > > endorsed by it. IG is a trading name of IG Markets
>>> > Limited
>>> > > (a
>>> > > > > company
>>> > > > > > > registered in England and Wales, company number
>>> 04008957)
>>> > > and
>>> > > > > IG
>>> > > > > > Index
>>> > > > > > > Limited (a company registered in England and Wales,
>>> > company
>>> > > > > number
>>> > > > > > > 01190902). Registered address at Cannon Bridge
>>> House, 25
>>> > > > > Dowgate
>>> > > > > > Hill,
>>> > > > > > > London EC4R 2YA. Both IG Markets Limited (register
>>> number
>>> > > > > 195355)
>>> > > > > > and IG
>>> > > > > > > Index Limited (register number 114059) are
>>> authorised and
>>> > > > > regulated
>>> > > > > > by the
>>> > > > > > > Financial Conduct Authority.
>>> > > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > The information contained in this email is strictly
>>> > confidential
>>> > > > and
>>> > > > > for
>>> > > > > > the use of the addressee only, unless otherwise indicated.
>>> If
>>> > you
>>> > > > > are not
>>> > > > > > the intended recipient, please do not read, copy, use or
>>> > disclose
>>> > > > to
>>> > > > > others
>>> > > > > > this message or any attachment. Please also notify the
>>> sender
>>> > by
>>> > > > > replying
>>> > > > > > to this email or by telephone (+44(020 7896 0011) and then
>>> > delete
>>> > > > > the email
>>> > > > > > and any copies of it. Opinions, conclusion (etc) that do
>>> not
>>> > > relate
>>> > > > > to the
>>> > > > > > official business of this company shall be understood as
>>> > neither
>>> > > > > given nor
>>> > > > > > endorsed by it. IG is a trading name of IG Markets Limited
>>> (a
>>> > > > company
>>> > > > > > registered in England and Wales, company number 04008957)
>>> and
>>> > IG
>>> > > > > Index
>>> > > > > > Limited (a company registered in England and Wales, company
>>> > > number
>>> > > > > > 01190902). Registered address at Cannon Bridge House, 25
>>> > Dowgate
>>> > > > > Hill,
>>> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
>>> > 195355)
>>> > > > > and IG
>>> > > > > > Index Limited (register number 114059) are authorised and
>>> > > regulated
>>> > > > > by the
>>> > > > > > Financial Conduct Authority.
>>> > > > > >
>>> > > > >
>>> > > > >
>>> > > > > The information contained in this email is strictly confidential
>>> and
>>> > > for
>>> > > > > the use of the addressee only, unless otherwise indicated. If
>>> you are
>>> > > not
>>> > > > > the intended recipient, please do not read, copy, use or
>>> disclose to
>>> > > > others
>>> > > > > this message or any attachment. Please also notify the sender by
>>> > > replying
>>> > > > > to this email or by telephone (+44(020 7896 0011) and then
>>> delete the
>>> > > > email
>>> > > > > and any copies of it. Opinions, conclusion (etc) that do not
>>> relate
>>> > to
>>> > > > the
>>> > > > > official business of this company shall be understood as neither
>>> > given
>>> > > > nor
>>> > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>>> company
>>> > > > > registered in England and Wales, company number 04008957) and IG
>>> > Index
>>> > > > > Limited (a company registered in England and Wales, company
>>> number
>>> > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>>> > Hill,
>>> > > > > London EC4R 2YA. Both IG Markets Limited (register number
>>> 195355) and
>>> > > IG
>>> > > > > Index Limited (register number 114059) are authorised and
>>> regulated
>>> > by
>>> > > > the
>>> > > > > Financial Conduct Authority.
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang