TLS Session Ticket Key Format

[Bill Zeng <bi...@gmail.com>]

Hi all,

I am new to ATS and my understanding of ATS is limited. I am working on a
project to enable session resumption using session tickets. Session tickets
are encrypted with session ticket keys which need to be rotated for
security. Currently, one session ticket key file (only one key is used) is
specified by the ticket_key_name option and used for the entire time.

I would like to propose to rotate the session ticket key periodically, say,
once every 12 hours. A new session ticket key is generated by a key server,
and distributed to ATS's. Note that locally generated keys cannot be shared
among multiple ATS's on different boxes easily. An ATS needs to load new
keys into memory every 12 hours. In order to smooth the key transition
process, multiple keys can be in use at the same time. That is, when a new
key is generated, the older can be used for a while before it gets retired.
These keys can be stored in multiple files, but it is neat to store them in
just one file.

A session ticket key contains three 16-byte fields: key name, AES key, and
HMAC key. Right now,  they are stored as an opaque 48-byte blob. A 48-byte
blob works for the simple case with just one session ticket key. But it can
be inconvenient for multiple keys with different versions. I would like to
propose to use JSON as the format for session ticket keys. We can store
rich metadata such as the version, lifetime etc. in JSON. Each key can also
have a lifetime associated with it. JSON is also human-readable and
inter-operable with other tools. A simple JSON parser is needed to parse
the session ticket keys.

Here is Twitter's approach to this problem:
https://blog.twitter.com/2013/forward-secrecy-at-twitter

Comments are welcome.
Thanks!
Bin

Re: TLS Session Ticket Key Format

[Bill Zeng <bi...@gmail.com>]

ping


On Mon, Aug 25, 2014 at 1:21 PM, Bill Zeng <bi...@gmail.com> wrote:

> Thanks for the comments. We can stick to the current byte-blob ticket key
> format. I would like to extend it a little to store multiple ticket keys
> for rotation. A new key is generated and appended to the ticket key file
> while the oldest one (at the beginning of the file) gets removed (assume we
> have more than one keys in rotation). Multiple keys are kept in the ticket
> key file for the convenience of deployment. Sessions encrypted with older
> keys can still be resumed after an ATS restart. It seems that
> SSL_CTX_set_tlsext_ticket_key_cb callback is helpful for this situation.
> Every time a ticket key is presented, we try to decrypt it with one of the
> ticket keys in rotation, and if it succeeds, we always encrypt it with the
> most recent ticket key. The callback checks whether the ticket key file has
> changed once every rotation cycle, say 12 hours.
>
>
>
> On Fri, Aug 22, 2014 at 1:44 PM, Bill Zeng <bi...@gmail.com> wrote:
>
>>
>>
>>
>> On Fri, Aug 22, 2014 at 12:14 PM, James Peach <jp...@apache.org> wrote:
>>
>>> On Aug 22, 2014, at 10:35 AM, Manjesh Nilange <ma...@gmail.com>
>>> wrote:
>>>
>>> > On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org>
>>> wrote:
>>> >
>>> >> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com>
>>> wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> I am new to ATS and my understanding of ATS is limited. I am working
>>> on a
>>> >>> project to enable session resumption using session tickets. Session
>>> >> tickets
>>> >>> are encrypted with session ticket keys which need to be rotated for
>>> >>> security. Currently, one session ticket key file (only one key is
>>> used)
>>> >> is
>>> >>> specified by the ticket_key_name option and used for the entire time.
>>> >>>
>>> >>> I would like to propose to rotate the session ticket key
>>> periodically,
>>> >> say,
>>> >>> once every 12 hours. A new session ticket key is generated by a key
>>> >> server,
>>> >>> and distributed to ATS's. Note that locally generated keys cannot be
>>> >> shared
>>> >>> among multiple ATS's on different boxes easily. An ATS needs to load
>>> new
>>> >>> keys into memory every 12 hours. In order to smooth the key
>>> transition
>>> >>> process, multiple keys can be in use at the same time. That is, when
>>> a
>>> >> new
>>> >>> key is generated, the older can be used for a while before it gets
>>> >> retired.
>>> >>> These keys can be stored in multiple files, but it is neat to store
>>> them
>>> >> in
>>> >>> just one file.
>>> >>>
>>> >>> A session ticket key contains three 16-byte fields: key name, AES
>>> key,
>>> >> and
>>> >>> HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
>>> >> 48-byte
>>> >>> blob works for the simple case with just one session ticket key. But
>>> it
>>> >> can
>>> >>> be inconvenient for multiple keys with different versions.
>>> >>
>>> >> We use the same ticket key format as httpd <
>>> >>
>>> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
>>> >>> .
>>> >>
>>> >>> I would like to
>>> >>> propose to use JSON as the format for session ticket keys. We can
>>> store
>>> >>> rich metadata such as the version, lifetime etc. in JSON. Each key
>>> can
>>> >> also
>>> >>> have a lifetime associated with it. JSON is also human-readable and
>>> >>> inter-operable with other tools. A simple JSON parser is needed to
>>> parse
>>> >>> the session ticket keys.
>>> >>>
>>> >>> Here is Twitter's approach to this problem:
>>> >>> https://blog.twitter.com/2013/forward-secrecy-at-twitter
>>> >>
>>> >> You can implement a scheme similar to that described in the Twitter
>>> post
>>> >> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
>>> >> filesystem helper), and reload the SSL config (touch
>>> ssl_multicert.config
>>> >> && traffic_line -x) once the new version is in place everywhere. I
>>> guess
>>> >> that you could use something like etcd to co-ordinate the process.
>>> >>
>>> >
>>> > Won't traffic_line -x cause other config changes to be picked up as
>>> well? I
>>> > think that is risky.
>>>
>>> Well it's intrinsically risky to leave un-applied changes that you are
>>> not willing to accept at any time. I would expect any config management
>>> (puppet, salt, etc) will run traffic_line -x when it makes any config
>>> changes.
>>>
>>
>> It is not really a change to the configuration files. The contents in the
>> ticket key file are changed, of which the file name stays the same. When
>> you run traffic_line -x, the configuration files get parsed but the ticket
>> keys do not get loaded again.
>>
>>
>>> > And operationally, I'm not sure how maintainable it is
>>> > to co-ordinate all servers like this.
>>>
>>> I dunno, it seems straightforward to me. Rotate the keys, load new ones.
>>> The failure mode is that there is a window where SSL sessions can't get
>>> resumed, so you'll take a CPU and latency hit for a short time.
>>>
>>
>> The failure mode is not something I am worried about. We have multiple
>> keys in rotation. Session tickets decrypted with old keys will get
>> encrypted with the most recent one.
>>
>>
>>>
>>> J
>>>
>>>
>>
>

Re: TLS Session Ticket Key Format

[Bill Zeng <bi...@gmail.com>]

Thanks for the feedback. The implementation is not as complex as it sounds.
It's only a few lines of code and requires adding one configuration option
to SSL config, which is the number of keys in rotation. Even that can be
obviated.


On Tue, Aug 26, 2014 at 3:53 PM, James Peach <jp...@apache.org> wrote:

> On Aug 25, 2014, at 1:21 PM, Bill Zeng <bi...@gmail.com> wrote:
>
> > Thanks for the comments. We can stick to the current byte-blob ticket key
> > format. I would like to extend it a little to store multiple ticket keys
> > for rotation. A new key is generated and appended to the ticket key file
> > while the oldest one (at the beginning of the file) gets removed (assume
> we
> > have more than one keys in rotation). Multiple keys are kept in the
> ticket
> > key file for the convenience of deployment. Sessions encrypted with older
> > keys can still be resumed after an ATS restart. It seems that
> > SSL_CTX_set_tlsext_ticket_key_cb callback is helpful for this situation.
> > Every time a ticket key is presented, we try to decrypt it with one of
> the
> > ticket keys in rotation, and if it succeeds, we always encrypt it with
> the
> > most recent ticket key. The callback checks whether the ticket key file
> has
> > changed once every rotation cycle, say 12 hours.
>
> Sorry, this still seems like more complexity than it's worth. I'd just
> push the new keys and let connections renegotiate. SSL config is already
> complicated enough.
>
> > On Fri, Aug 22, 2014 at 1:44 PM, Bill Zeng <bi...@gmail.com>
> wrote:
> >
> >>
> >>
> >>
> >> On Fri, Aug 22, 2014 at 12:14 PM, James Peach <jp...@apache.org>
> wrote:
> >>
> >>> On Aug 22, 2014, at 10:35 AM, Manjesh Nilange <
> manjeshnilange@gmail.com>
> >>> wrote:
> >>>
> >>>> On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org>
> wrote:
> >>>>
> >>>>> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com>
> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I am new to ATS and my understanding of ATS is limited. I am working
> >>> on a
> >>>>>> project to enable session resumption using session tickets. Session
> >>>>> tickets
> >>>>>> are encrypted with session ticket keys which need to be rotated for
> >>>>>> security. Currently, one session ticket key file (only one key is
> >>> used)
> >>>>> is
> >>>>>> specified by the ticket_key_name option and used for the entire
> time.
> >>>>>>
> >>>>>> I would like to propose to rotate the session ticket key
> periodically,
> >>>>> say,
> >>>>>> once every 12 hours. A new session ticket key is generated by a key
> >>>>> server,
> >>>>>> and distributed to ATS's. Note that locally generated keys cannot be
> >>>>> shared
> >>>>>> among multiple ATS's on different boxes easily. An ATS needs to load
> >>> new
> >>>>>> keys into memory every 12 hours. In order to smooth the key
> transition
> >>>>>> process, multiple keys can be in use at the same time. That is,
> when a
> >>>>> new
> >>>>>> key is generated, the older can be used for a while before it gets
> >>>>> retired.
> >>>>>> These keys can be stored in multiple files, but it is neat to store
> >>> them
> >>>>> in
> >>>>>> just one file.
> >>>>>>
> >>>>>> A session ticket key contains three 16-byte fields: key name, AES
> key,
> >>>>> and
> >>>>>> HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
> >>>>> 48-byte
> >>>>>> blob works for the simple case with just one session ticket key. But
> >>> it
> >>>>> can
> >>>>>> be inconvenient for multiple keys with different versions.
> >>>>>
> >>>>> We use the same ticket key format as httpd <
> >>>>>
> >>>
> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
> >>>>>> .
> >>>>>
> >>>>>> I would like to
> >>>>>> propose to use JSON as the format for session ticket keys. We can
> >>> store
> >>>>>> rich metadata such as the version, lifetime etc. in JSON. Each key
> can
> >>>>> also
> >>>>>> have a lifetime associated with it. JSON is also human-readable and
> >>>>>> inter-operable with other tools. A simple JSON parser is needed to
> >>> parse
> >>>>>> the session ticket keys.
> >>>>>>
> >>>>>> Here is Twitter's approach to this problem:
> >>>>>> https://blog.twitter.com/2013/forward-secrecy-at-twitter
> >>>>>
> >>>>> You can implement a scheme similar to that described in the Twitter
> >>> post
> >>>>> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
> >>>>> filesystem helper), and reload the SSL config (touch
> >>> ssl_multicert.config
> >>>>> && traffic_line -x) once the new version is in place everywhere. I
> >>> guess
> >>>>> that you could use something like etcd to co-ordinate the process.
> >>>>>
> >>>>
> >>>> Won't traffic_line -x cause other config changes to be picked up as
> >>> well? I
> >>>> think that is risky.
> >>>
> >>> Well it's intrinsically risky to leave un-applied changes that you are
> >>> not willing to accept at any time. I would expect any config management
> >>> (puppet, salt, etc) will run traffic_line -x when it makes any config
> >>> changes.
> >>>
> >>
> >> It is not really a change to the configuration files. The contents in
> the
> >> ticket key file are changed, of which the file name stays the same. When
> >> you run traffic_line -x, the configuration files get parsed but the
> ticket
> >> keys do not get loaded again.
> >>
> >>
> >>>> And operationally, I'm not sure how maintainable it is
> >>>> to co-ordinate all servers like this.
> >>>
> >>> I dunno, it seems straightforward to me. Rotate the keys, load new
> ones.
> >>> The failure mode is that there is a window where SSL sessions can't get
> >>> resumed, so you'll take a CPU and latency hit for a short time.
> >>>
> >>
> >> The failure mode is not something I am worried about. We have multiple
> >> keys in rotation. Session tickets decrypted with old keys will get
> >> encrypted with the most recent one.
> >>
> >>
> >>>
> >>> J
> >>>
> >>>
> >>
>
>

Re: TLS Session Ticket Key Format

[James Peach <jp...@apache.org>]

On Aug 25, 2014, at 1:21 PM, Bill Zeng <bi...@gmail.com> wrote:

> Thanks for the comments. We can stick to the current byte-blob ticket key
> format. I would like to extend it a little to store multiple ticket keys
> for rotation. A new key is generated and appended to the ticket key file
> while the oldest one (at the beginning of the file) gets removed (assume we
> have more than one keys in rotation). Multiple keys are kept in the ticket
> key file for the convenience of deployment. Sessions encrypted with older
> keys can still be resumed after an ATS restart. It seems that
> SSL_CTX_set_tlsext_ticket_key_cb callback is helpful for this situation.
> Every time a ticket key is presented, we try to decrypt it with one of the
> ticket keys in rotation, and if it succeeds, we always encrypt it with the
> most recent ticket key. The callback checks whether the ticket key file has
> changed once every rotation cycle, say 12 hours.

Sorry, this still seems like more complexity than it's worth. I'd just push the new keys and let connections renegotiate. SSL config is already complicated enough.

> On Fri, Aug 22, 2014 at 1:44 PM, Bill Zeng <bi...@gmail.com> wrote:
> 
>> 
>> 
>> 
>> On Fri, Aug 22, 2014 at 12:14 PM, James Peach <jp...@apache.org> wrote:
>> 
>>> On Aug 22, 2014, at 10:35 AM, Manjesh Nilange <ma...@gmail.com>
>>> wrote:
>>> 
>>>> On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:
>>>> 
>>>>> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
>>>>> 
>>>>>> Hi all,
>>>>>> 
>>>>>> I am new to ATS and my understanding of ATS is limited. I am working
>>> on a
>>>>>> project to enable session resumption using session tickets. Session
>>>>> tickets
>>>>>> are encrypted with session ticket keys which need to be rotated for
>>>>>> security. Currently, one session ticket key file (only one key is
>>> used)
>>>>> is
>>>>>> specified by the ticket_key_name option and used for the entire time.
>>>>>> 
>>>>>> I would like to propose to rotate the session ticket key periodically,
>>>>> say,
>>>>>> once every 12 hours. A new session ticket key is generated by a key
>>>>> server,
>>>>>> and distributed to ATS's. Note that locally generated keys cannot be
>>>>> shared
>>>>>> among multiple ATS's on different boxes easily. An ATS needs to load
>>> new
>>>>>> keys into memory every 12 hours. In order to smooth the key transition
>>>>>> process, multiple keys can be in use at the same time. That is, when a
>>>>> new
>>>>>> key is generated, the older can be used for a while before it gets
>>>>> retired.
>>>>>> These keys can be stored in multiple files, but it is neat to store
>>> them
>>>>> in
>>>>>> just one file.
>>>>>> 
>>>>>> A session ticket key contains three 16-byte fields: key name, AES key,
>>>>> and
>>>>>> HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
>>>>> 48-byte
>>>>>> blob works for the simple case with just one session ticket key. But
>>> it
>>>>> can
>>>>>> be inconvenient for multiple keys with different versions.
>>>>> 
>>>>> We use the same ticket key format as httpd <
>>>>> 
>>> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
>>>>>> .
>>>>> 
>>>>>> I would like to
>>>>>> propose to use JSON as the format for session ticket keys. We can
>>> store
>>>>>> rich metadata such as the version, lifetime etc. in JSON. Each key can
>>>>> also
>>>>>> have a lifetime associated with it. JSON is also human-readable and
>>>>>> inter-operable with other tools. A simple JSON parser is needed to
>>> parse
>>>>>> the session ticket keys.
>>>>>> 
>>>>>> Here is Twitter's approach to this problem:
>>>>>> https://blog.twitter.com/2013/forward-secrecy-at-twitter
>>>>> 
>>>>> You can implement a scheme similar to that described in the Twitter
>>> post
>>>>> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
>>>>> filesystem helper), and reload the SSL config (touch
>>> ssl_multicert.config
>>>>> && traffic_line -x) once the new version is in place everywhere. I
>>> guess
>>>>> that you could use something like etcd to co-ordinate the process.
>>>>> 
>>>> 
>>>> Won't traffic_line -x cause other config changes to be picked up as
>>> well? I
>>>> think that is risky.
>>> 
>>> Well it's intrinsically risky to leave un-applied changes that you are
>>> not willing to accept at any time. I would expect any config management
>>> (puppet, salt, etc) will run traffic_line -x when it makes any config
>>> changes.
>>> 
>> 
>> It is not really a change to the configuration files. The contents in the
>> ticket key file are changed, of which the file name stays the same. When
>> you run traffic_line -x, the configuration files get parsed but the ticket
>> keys do not get loaded again.
>> 
>> 
>>>> And operationally, I'm not sure how maintainable it is
>>>> to co-ordinate all servers like this.
>>> 
>>> I dunno, it seems straightforward to me. Rotate the keys, load new ones.
>>> The failure mode is that there is a window where SSL sessions can't get
>>> resumed, so you'll take a CPU and latency hit for a short time.
>>> 
>> 
>> The failure mode is not something I am worried about. We have multiple
>> keys in rotation. Session tickets decrypted with old keys will get
>> encrypted with the most recent one.
>> 
>> 
>>> 
>>> J
>>> 
>>> 
>> 

Re: TLS Session Ticket Key Format

[Bill Zeng <bi...@gmail.com>]

Thanks for the comments. We can stick to the current byte-blob ticket key
format. I would like to extend it a little to store multiple ticket keys
for rotation. A new key is generated and appended to the ticket key file
while the oldest one (at the beginning of the file) gets removed (assume we
have more than one keys in rotation). Multiple keys are kept in the ticket
key file for the convenience of deployment. Sessions encrypted with older
keys can still be resumed after an ATS restart. It seems that
SSL_CTX_set_tlsext_ticket_key_cb callback is helpful for this situation.
Every time a ticket key is presented, we try to decrypt it with one of the
ticket keys in rotation, and if it succeeds, we always encrypt it with the
most recent ticket key. The callback checks whether the ticket key file has
changed once every rotation cycle, say 12 hours.



On Fri, Aug 22, 2014 at 1:44 PM, Bill Zeng <bi...@gmail.com> wrote:

>
>
>
> On Fri, Aug 22, 2014 at 12:14 PM, James Peach <jp...@apache.org> wrote:
>
>> On Aug 22, 2014, at 10:35 AM, Manjesh Nilange <ma...@gmail.com>
>> wrote:
>>
>> > On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:
>> >
>> >> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> I am new to ATS and my understanding of ATS is limited. I am working
>> on a
>> >>> project to enable session resumption using session tickets. Session
>> >> tickets
>> >>> are encrypted with session ticket keys which need to be rotated for
>> >>> security. Currently, one session ticket key file (only one key is
>> used)
>> >> is
>> >>> specified by the ticket_key_name option and used for the entire time.
>> >>>
>> >>> I would like to propose to rotate the session ticket key periodically,
>> >> say,
>> >>> once every 12 hours. A new session ticket key is generated by a key
>> >> server,
>> >>> and distributed to ATS's. Note that locally generated keys cannot be
>> >> shared
>> >>> among multiple ATS's on different boxes easily. An ATS needs to load
>> new
>> >>> keys into memory every 12 hours. In order to smooth the key transition
>> >>> process, multiple keys can be in use at the same time. That is, when a
>> >> new
>> >>> key is generated, the older can be used for a while before it gets
>> >> retired.
>> >>> These keys can be stored in multiple files, but it is neat to store
>> them
>> >> in
>> >>> just one file.
>> >>>
>> >>> A session ticket key contains three 16-byte fields: key name, AES key,
>> >> and
>> >>> HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
>> >> 48-byte
>> >>> blob works for the simple case with just one session ticket key. But
>> it
>> >> can
>> >>> be inconvenient for multiple keys with different versions.
>> >>
>> >> We use the same ticket key format as httpd <
>> >>
>> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
>> >>> .
>> >>
>> >>> I would like to
>> >>> propose to use JSON as the format for session ticket keys. We can
>> store
>> >>> rich metadata such as the version, lifetime etc. in JSON. Each key can
>> >> also
>> >>> have a lifetime associated with it. JSON is also human-readable and
>> >>> inter-operable with other tools. A simple JSON parser is needed to
>> parse
>> >>> the session ticket keys.
>> >>>
>> >>> Here is Twitter's approach to this problem:
>> >>> https://blog.twitter.com/2013/forward-secrecy-at-twitter
>> >>
>> >> You can implement a scheme similar to that described in the Twitter
>> post
>> >> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
>> >> filesystem helper), and reload the SSL config (touch
>> ssl_multicert.config
>> >> && traffic_line -x) once the new version is in place everywhere. I
>> guess
>> >> that you could use something like etcd to co-ordinate the process.
>> >>
>> >
>> > Won't traffic_line -x cause other config changes to be picked up as
>> well? I
>> > think that is risky.
>>
>> Well it's intrinsically risky to leave un-applied changes that you are
>> not willing to accept at any time. I would expect any config management
>> (puppet, salt, etc) will run traffic_line -x when it makes any config
>> changes.
>>
>
> It is not really a change to the configuration files. The contents in the
> ticket key file are changed, of which the file name stays the same. When
> you run traffic_line -x, the configuration files get parsed but the ticket
> keys do not get loaded again.
>
>
>> > And operationally, I'm not sure how maintainable it is
>> > to co-ordinate all servers like this.
>>
>> I dunno, it seems straightforward to me. Rotate the keys, load new ones.
>> The failure mode is that there is a window where SSL sessions can't get
>> resumed, so you'll take a CPU and latency hit for a short time.
>>
>
> The failure mode is not something I am worried about. We have multiple
> keys in rotation. Session tickets decrypted with old keys will get
> encrypted with the most recent one.
>
>
>>
>> J
>>
>>
>

Re: TLS Session Ticket Key Format

[Bill Zeng <bi...@gmail.com>]

On Fri, Aug 22, 2014 at 12:14 PM, James Peach <jp...@apache.org> wrote:

> On Aug 22, 2014, at 10:35 AM, Manjesh Nilange <ma...@gmail.com>
> wrote:
>
> > On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:
> >
> >> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I am new to ATS and my understanding of ATS is limited. I am working
> on a
> >>> project to enable session resumption using session tickets. Session
> >> tickets
> >>> are encrypted with session ticket keys which need to be rotated for
> >>> security. Currently, one session ticket key file (only one key is used)
> >> is
> >>> specified by the ticket_key_name option and used for the entire time.
> >>>
> >>> I would like to propose to rotate the session ticket key periodically,
> >> say,
> >>> once every 12 hours. A new session ticket key is generated by a key
> >> server,
> >>> and distributed to ATS's. Note that locally generated keys cannot be
> >> shared
> >>> among multiple ATS's on different boxes easily. An ATS needs to load
> new
> >>> keys into memory every 12 hours. In order to smooth the key transition
> >>> process, multiple keys can be in use at the same time. That is, when a
> >> new
> >>> key is generated, the older can be used for a while before it gets
> >> retired.
> >>> These keys can be stored in multiple files, but it is neat to store
> them
> >> in
> >>> just one file.
> >>>
> >>> A session ticket key contains three 16-byte fields: key name, AES key,
> >> and
> >>> HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
> >> 48-byte
> >>> blob works for the simple case with just one session ticket key. But it
> >> can
> >>> be inconvenient for multiple keys with different versions.
> >>
> >> We use the same ticket key format as httpd <
> >>
> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
> >>> .
> >>
> >>> I would like to
> >>> propose to use JSON as the format for session ticket keys. We can store
> >>> rich metadata such as the version, lifetime etc. in JSON. Each key can
> >> also
> >>> have a lifetime associated with it. JSON is also human-readable and
> >>> inter-operable with other tools. A simple JSON parser is needed to
> parse
> >>> the session ticket keys.
> >>>
> >>> Here is Twitter's approach to this problem:
> >>> https://blog.twitter.com/2013/forward-secrecy-at-twitter
> >>
> >> You can implement a scheme similar to that described in the Twitter post
> >> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
> >> filesystem helper), and reload the SSL config (touch
> ssl_multicert.config
> >> && traffic_line -x) once the new version is in place everywhere. I guess
> >> that you could use something like etcd to co-ordinate the process.
> >>
> >
> > Won't traffic_line -x cause other config changes to be picked up as
> well? I
> > think that is risky.
>
> Well it's intrinsically risky to leave un-applied changes that you are not
> willing to accept at any time. I would expect any config management
> (puppet, salt, etc) will run traffic_line -x when it makes any config
> changes.
>

It is not really a change to the configuration files. The contents in the
ticket key file are changed, of which the file name stays the same. When
you run traffic_line -x, the configuration files get parsed but the ticket
keys do not get loaded again.


> > And operationally, I'm not sure how maintainable it is
> > to co-ordinate all servers like this.
>
> I dunno, it seems straightforward to me. Rotate the keys, load new ones.
> The failure mode is that there is a window where SSL sessions can't get
> resumed, so you'll take a CPU and latency hit for a short time.
>

The failure mode is not something I am worried about. We have multiple keys
in rotation. Session tickets decrypted with old keys will get encrypted
with the most recent one.


>
> J
>
>

Re: TLS Session Ticket Key Format

[James Peach <jp...@apache.org>]

On Aug 22, 2014, at 10:35 AM, Manjesh Nilange <ma...@gmail.com> wrote:

> On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:
> 
>> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
>> 
>>> Hi all,
>>> 
>>> I am new to ATS and my understanding of ATS is limited. I am working on a
>>> project to enable session resumption using session tickets. Session
>> tickets
>>> are encrypted with session ticket keys which need to be rotated for
>>> security. Currently, one session ticket key file (only one key is used)
>> is
>>> specified by the ticket_key_name option and used for the entire time.
>>> 
>>> I would like to propose to rotate the session ticket key periodically,
>> say,
>>> once every 12 hours. A new session ticket key is generated by a key
>> server,
>>> and distributed to ATS's. Note that locally generated keys cannot be
>> shared
>>> among multiple ATS's on different boxes easily. An ATS needs to load new
>>> keys into memory every 12 hours. In order to smooth the key transition
>>> process, multiple keys can be in use at the same time. That is, when a
>> new
>>> key is generated, the older can be used for a while before it gets
>> retired.
>>> These keys can be stored in multiple files, but it is neat to store them
>> in
>>> just one file.
>>> 
>>> A session ticket key contains three 16-byte fields: key name, AES key,
>> and
>>> HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
>> 48-byte
>>> blob works for the simple case with just one session ticket key. But it
>> can
>>> be inconvenient for multiple keys with different versions.
>> 
>> We use the same ticket key format as httpd <
>> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
>>> .
>> 
>>> I would like to
>>> propose to use JSON as the format for session ticket keys. We can store
>>> rich metadata such as the version, lifetime etc. in JSON. Each key can
>> also
>>> have a lifetime associated with it. JSON is also human-readable and
>>> inter-operable with other tools. A simple JSON parser is needed to parse
>>> the session ticket keys.
>>> 
>>> Here is Twitter's approach to this problem:
>>> https://blog.twitter.com/2013/forward-secrecy-at-twitter
>> 
>> You can implement a scheme similar to that described in the Twitter post
>> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
>> filesystem helper), and reload the SSL config (touch ssl_multicert.config
>> && traffic_line -x) once the new version is in place everywhere. I guess
>> that you could use something like etcd to co-ordinate the process.
>> 
> 
> Won't traffic_line -x cause other config changes to be picked up as well? I
> think that is risky.

Well it's intrinsically risky to leave un-applied changes that you are not willing to accept at any time. I would expect any config management (puppet, salt, etc) will run traffic_line -x when it makes any config changes.

> And operationally, I'm not sure how maintainable it is
> to co-ordinate all servers like this.

I dunno, it seems straightforward to me. Rotate the keys, load new ones. The failure mode is that there is a window where SSL sessions can't get resumed, so you'll take a CPU and latency hit for a short time.

J

Re: TLS Session Ticket Key Format

[Manjesh Nilange <ma...@gmail.com>]

On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:

> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
>
> > Hi all,
> >
> > I am new to ATS and my understanding of ATS is limited. I am working on a
> > project to enable session resumption using session tickets. Session
> tickets
> > are encrypted with session ticket keys which need to be rotated for
> > security. Currently, one session ticket key file (only one key is used)
> is
> > specified by the ticket_key_name option and used for the entire time.
> >
> > I would like to propose to rotate the session ticket key periodically,
> say,
> > once every 12 hours. A new session ticket key is generated by a key
> server,
> > and distributed to ATS's. Note that locally generated keys cannot be
> shared
> > among multiple ATS's on different boxes easily. An ATS needs to load new
> > keys into memory every 12 hours. In order to smooth the key transition
> > process, multiple keys can be in use at the same time. That is, when a
> new
> > key is generated, the older can be used for a while before it gets
> retired.
> > These keys can be stored in multiple files, but it is neat to store them
> in
> > just one file.
> >
> > A session ticket key contains three 16-byte fields: key name, AES key,
> and
> > HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
> 48-byte
> > blob works for the simple case with just one session ticket key. But it
> can
> > be inconvenient for multiple keys with different versions.
>
> We use the same ticket key format as httpd <
> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
> >.
>
> > I would like to
> > propose to use JSON as the format for session ticket keys. We can store
> > rich metadata such as the version, lifetime etc. in JSON. Each key can
> also
> > have a lifetime associated with it. JSON is also human-readable and
> > inter-operable with other tools. A simple JSON parser is needed to parse
> > the session ticket keys.
> >
> > Here is Twitter's approach to this problem:
> > https://blog.twitter.com/2013/forward-secrecy-at-twitter
>
> You can implement a scheme similar to that described in the Twitter post
> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
> filesystem helper), and reload the SSL config (touch ssl_multicert.config
> && traffic_line -x) once the new version is in place everywhere. I guess
> that you could use something like etcd to co-ordinate the process.
>

Won't traffic_line -x cause other config changes to be picked up as well? I
think that is risky. And operationally, I'm not sure how maintainable it is
to co-ordinate all servers like this.


>
> J
>

Re: TLS Session Ticket Key Format

[Bill Zeng <bi...@gmail.com>]

Thanks for the reply!

On Thu, Aug 21, 2014 at 7:50 PM, Wei Sun <su...@yahoo-inc.com.invalid>
wrote:

>
>
> On 8/22/14, 10:06, "Bill Zeng" <bi...@gmail.com> wrote:
>
> >Hi James,
> >
> >Thanks for the reply!
> >
> >On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:
> >
> >> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I am new to ATS and my understanding of ATS is limited. I am working
> >>on a
> >> > project to enable session resumption using session tickets. Session
> >> tickets
> >> > are encrypted with session ticket keys which need to be rotated for
> >> > security. Currently, one session ticket key file (only one key is
> >>used)
> >> is
> >> > specified by the ticket_key_name option and used for the entire time.
> >> >
> >> > I would like to propose to rotate the session ticket key periodically,
> >> say,
> >> > once every 12 hours. A new session ticket key is generated by a key
> >> server,
> >> > and distributed to ATS's. Note that locally generated keys cannot be
> >> shared
> >> > among multiple ATS's on different boxes easily. An ATS needs to load
> >>new
> >> > keys into memory every 12 hours. In order to smooth the key transition
> >> > process, multiple keys can be in use at the same time. That is, when a
> >> new
> >> > key is generated, the older can be used for a while before it gets
> >> retired.
> >> > These keys can be stored in multiple files, but it is neat to store
> >>them
> >> in
> >> > just one file.
> >> >
> >> > A session ticket key contains three 16-byte fields: key name, AES key,
> >> and
> >> > HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
> >> 48-byte
> >> > blob works for the simple case with just one session ticket key. But
> >>it
> >> can
> >> > be inconvenient for multiple keys with different versions.
> >>
> >> We use the same ticket key format as httpd <
> >>
> >>
> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
> >> >.
> >>
> >
> >Compatibility is important.
> >
> >
> >> > I would like to
> >> > propose to use JSON as the format for session ticket keys. We can
> >>store
> >> > rich metadata such as the version, lifetime etc. in JSON. Each key can
> >> also
> >> > have a lifetime associated with it. JSON is also human-readable and
> >> > inter-operable with other tools. A simple JSON parser is needed to
> >>parse
> >> > the session ticket keys.
> >> >
> >> > Here is Twitter's approach to this problem:
> >> > https://blog.twitter.com/2013/forward-secrecy-at-twitter
> >>
> >> You can implement a scheme similar to that described in the Twitter post
> >> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
> >> filesystem helper), and reload the SSL config (touch
> >>ssl_multicert.config
> >> && traffic_line -x) once the new version is in place everywhere. I guess
> >> that you could use something like etcd to co-ordinate the process.
> >>
> >
> >traffic_line -x might work. It does not seem like an efficient approach.
> >You need to do that on every box. I would like to push a new key to ATS's
> >and they suck in the new key periodically without intervention. We can add
> >a new configuration option to records.config and checks for key update
> >only
> >once every 12 hours.
>
> Another approach is to use the lifecycle API
> (TS_LIFECYCLE_SERVER_SSL_CTX_INITIALIZED_HOOK) and handle any SSL_CTX
> related callbacks (e.g. ssl_callback_session_ticket) in a plug-in where
> you may spawn a dedicate thread to rotate the keys, this helps to
> integrate with specific CKMS that keys are all in memory.
>
>
A separate thread seems like an overkill for this problem. A session ticket
key is renewed once for every 12 hours. Most of the time, the thread would
be idle.


> >
> >
> >> J
> >>
>
>

Re: TLS Session Ticket Key Format

[Wei Sun <su...@yahoo-inc.com.INVALID>]

On 8/22/14, 10:06, "Bill Zeng" <bi...@gmail.com> wrote:

>Hi James,
>
>Thanks for the reply!
>
>On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:
>
>> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I am new to ATS and my understanding of ATS is limited. I am working
>>on a
>> > project to enable session resumption using session tickets. Session
>> tickets
>> > are encrypted with session ticket keys which need to be rotated for
>> > security. Currently, one session ticket key file (only one key is
>>used)
>> is
>> > specified by the ticket_key_name option and used for the entire time.
>> >
>> > I would like to propose to rotate the session ticket key periodically,
>> say,
>> > once every 12 hours. A new session ticket key is generated by a key
>> server,
>> > and distributed to ATS's. Note that locally generated keys cannot be
>> shared
>> > among multiple ATS's on different boxes easily. An ATS needs to load
>>new
>> > keys into memory every 12 hours. In order to smooth the key transition
>> > process, multiple keys can be in use at the same time. That is, when a
>> new
>> > key is generated, the older can be used for a while before it gets
>> retired.
>> > These keys can be stored in multiple files, but it is neat to store
>>them
>> in
>> > just one file.
>> >
>> > A session ticket key contains three 16-byte fields: key name, AES key,
>> and
>> > HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
>> 48-byte
>> > blob works for the simple case with just one session ticket key. But
>>it
>> can
>> > be inconvenient for multiple keys with different versions.
>>
>> We use the same ticket key format as httpd <
>> 
>>http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
>> >.
>>
>
>Compatibility is important.
>
>
>> > I would like to
>> > propose to use JSON as the format for session ticket keys. We can
>>store
>> > rich metadata such as the version, lifetime etc. in JSON. Each key can
>> also
>> > have a lifetime associated with it. JSON is also human-readable and
>> > inter-operable with other tools. A simple JSON parser is needed to
>>parse
>> > the session ticket keys.
>> >
>> > Here is Twitter's approach to this problem:
>> > https://blog.twitter.com/2013/forward-secrecy-at-twitter
>>
>> You can implement a scheme similar to that described in the Twitter post
>> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
>> filesystem helper), and reload the SSL config (touch
>>ssl_multicert.config
>> && traffic_line -x) once the new version is in place everywhere. I guess
>> that you could use something like etcd to co-ordinate the process.
>>
>
>traffic_line -x might work. It does not seem like an efficient approach.
>You need to do that on every box. I would like to push a new key to ATS's
>and they suck in the new key periodically without intervention. We can add
>a new configuration option to records.config and checks for key update
>only
>once every 12 hours.

Another approach is to use the lifecycle API
(TS_LIFECYCLE_SERVER_SSL_CTX_INITIALIZED_HOOK) and handle any SSL_CTX
related callbacks (e.g. ssl_callback_session_ticket) in a plug-in where
you may spawn a dedicate thread to rotate the keys, this helps to
integrate with specific CKMS that keys are all in memory.

>
>
>> J
>>

Re: TLS Session Ticket Key Format

[Bill Zeng <bi...@gmail.com>]

Hi James,

Thanks for the reply!

On Thu, Aug 21, 2014 at 4:37 PM, James Peach <jp...@apache.org> wrote:

> On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:
>
> > Hi all,
> >
> > I am new to ATS and my understanding of ATS is limited. I am working on a
> > project to enable session resumption using session tickets. Session
> tickets
> > are encrypted with session ticket keys which need to be rotated for
> > security. Currently, one session ticket key file (only one key is used)
> is
> > specified by the ticket_key_name option and used for the entire time.
> >
> > I would like to propose to rotate the session ticket key periodically,
> say,
> > once every 12 hours. A new session ticket key is generated by a key
> server,
> > and distributed to ATS's. Note that locally generated keys cannot be
> shared
> > among multiple ATS's on different boxes easily. An ATS needs to load new
> > keys into memory every 12 hours. In order to smooth the key transition
> > process, multiple keys can be in use at the same time. That is, when a
> new
> > key is generated, the older can be used for a while before it gets
> retired.
> > These keys can be stored in multiple files, but it is neat to store them
> in
> > just one file.
> >
> > A session ticket key contains three 16-byte fields: key name, AES key,
> and
> > HMAC key. Right now,  they are stored as an opaque 48-byte blob. A
> 48-byte
> > blob works for the simple case with just one session ticket key. But it
> can
> > be inconvenient for multiple keys with different versions.
>
> We use the same ticket key format as httpd <
> http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile
> >.
>

Compatibility is important.


> > I would like to
> > propose to use JSON as the format for session ticket keys. We can store
> > rich metadata such as the version, lifetime etc. in JSON. Each key can
> also
> > have a lifetime associated with it. JSON is also human-readable and
> > inter-operable with other tools. A simple JSON parser is needed to parse
> > the session ticket keys.
> >
> > Here is Twitter's approach to this problem:
> > https://blog.twitter.com/2013/forward-secrecy-at-twitter
>
> You can implement a scheme similar to that described in the Twitter post
> with some external tooling. Keep the tickets on a RAM disk (or a FUSE
> filesystem helper), and reload the SSL config (touch ssl_multicert.config
> && traffic_line -x) once the new version is in place everywhere. I guess
> that you could use something like etcd to co-ordinate the process.
>

traffic_line -x might work. It does not seem like an efficient approach.
You need to do that on every box. I would like to push a new key to ATS's
and they suck in the new key periodically without intervention. We can add
a new configuration option to records.config and checks for key update only
once every 12 hours.


> J
>

Re: TLS Session Ticket Key Format

[James Peach <jp...@apache.org>]

On Aug 21, 2014, at 3:56 PM, Bill Zeng <bi...@gmail.com> wrote:

> Hi all,
> 
> I am new to ATS and my understanding of ATS is limited. I am working on a
> project to enable session resumption using session tickets. Session tickets
> are encrypted with session ticket keys which need to be rotated for
> security. Currently, one session ticket key file (only one key is used) is
> specified by the ticket_key_name option and used for the entire time.
> 
> I would like to propose to rotate the session ticket key periodically, say,
> once every 12 hours. A new session ticket key is generated by a key server,
> and distributed to ATS's. Note that locally generated keys cannot be shared
> among multiple ATS's on different boxes easily. An ATS needs to load new
> keys into memory every 12 hours. In order to smooth the key transition
> process, multiple keys can be in use at the same time. That is, when a new
> key is generated, the older can be used for a while before it gets retired.
> These keys can be stored in multiple files, but it is neat to store them in
> just one file.
> 
> A session ticket key contains three 16-byte fields: key name, AES key, and
> HMAC key. Right now,  they are stored as an opaque 48-byte blob. A 48-byte
> blob works for the simple case with just one session ticket key. But it can
> be inconvenient for multiple keys with different versions.

We use the same ticket key format as httpd <http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslsessionticketkeyfile>.

> I would like to
> propose to use JSON as the format for session ticket keys. We can store
> rich metadata such as the version, lifetime etc. in JSON. Each key can also
> have a lifetime associated with it. JSON is also human-readable and
> inter-operable with other tools. A simple JSON parser is needed to parse
> the session ticket keys.
> 
> Here is Twitter's approach to this problem:
> https://blog.twitter.com/2013/forward-secrecy-at-twitter

You can implement a scheme similar to that described in the Twitter post with some external tooling. Keep the tickets on a RAM disk (or a FUSE filesystem helper), and reload the SSL config (touch ssl_multicert.config && traffic_line -x) once the new version is in place everywhere. I guess that you could use something like etcd to co-ordinate the process.

J