You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by Rawlin Peters <ra...@apache.org> on 2020/12/07 20:47:44 UTC
Replace Riak w/ PostgreSQL
Hey folks,
I hope by now everyone can agree that we need to replace Riak (it's
been unmaintained for quite some time now). However, we might not all
agree yet on what it should be replaced with (at least not
officially). We've discussed it in threads here and there, but I'd
like to get some official consensus before we really hit the ground
running.
I would like to propose that we replace Riak with PostgreSQL.
Here are some of the reasons that I can think of (and have been
mentioned by others in the past) for us to use PostgreSQL:
- we all have much experience running it in production (because we
already run it for the Traffic Ops database)
- it would simplify ATC deployments by removing one more component
from the system
- it would simplify development as ATC devs are already familiar with
traditional SQL databases, and we could reuse a lot of the existing
code
- it has a healthy community of support and doesn't seem to be losing
steam anytime soon (it still remains the 2nd most popular OSS
relational database behind MySQL [1])
I would like this thread to focus on the merits (or lack thereof) of
using PostgreSQL as a replacement for Riak. We can discuss the
low-level implementation details separately in the blueprint I will
propose as a follow-up to this discussion. Unless someone is
vehemently -1 on using PostgreSQL to replace Riak, I will take silence
as assent and move forward with the blueprint process.
- Rawlin
[1] https://db-engines.com/en/ranking_osvsc
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by Rawlin Peters <ra...@apache.org>.
I've never used the experimental Emulated Vault, but if I understand
it correctly, it is supposed to mimic the Riak interface. The plan is
to keep Riak support while also adding postgres support, and you would
choose which backend to use via a new config option (which will
default to Riak for backwards-compatibility). I'm not positive the
Emulated Vault works out of the box with the current version of
Traffic Ops, since we use the Riak gRPC library instead of the Riak
HTTP API (which is what the old Perl version of Traffic Ops used). So
someone would probably need to add support for the Emulated Vault as
another Traffic Vault backend once the plugin interface is
implemented.
To answer your 2nd question, yes, everything that is currently stored
in Riak would be stored in Postgres instead if you were to choose the
Postgres option. If interested, keep an eye out for the blueprint I'll
be proposing within the next month or so.
- Rawlin
On Wed, Mar 3, 2021 at 4:43 AM Amarnath B S <am...@gmail.com> wrote:
>
> Is the plugin interface planned around the "emulated vault" experimental feature already in ATC? https://github.com/apache/trafficcontrol/tree/master/experimental/emulated_vault. The emulated vault shim layer looks quite clean and extensible.
> Also, just want to confirm that both the use cases of storing DS certs/keys and DNSSEC keys are planned to be covered.
> Thanks,
> -Amar
>
> On 2020/12/16 18:03:53, Rawlin Peters <ra...@apache.org> wrote:
> > Thank you all for the feedback. It sounds like Postgres can do the
> > job, so I'll move forward with the blueprint for implementing it as a
> > type of plugin interface with Postgres and Riak as the two
> > implementations to begin with. If we wanted to add HashiCorp Vault (or
> > something else) as another plugin implementation later on, the plugin
> > interface should make that easier.
> >
> > - Rawlin
> >
> > On Mon, Dec 7, 2020 at 9:48 PM Chatterjee, Srijeet
> > <SR...@comcast.com.invalid> wrote:
> > >
> > > +1 on moving away from Riak and using postgres instead. Riak can be a pain(atleast for new users) to set up and debug.
> > >
> > > --Srijeet
> > >
> > > From: John Rushford <jj...@gmail.com>
> > > Date: Monday, December 7, 2020 at 8:07 PM
> > > To: dev@trafficcontrol.apache.org <de...@trafficcontrol.apache.org>
> > > Subject: Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
> > > +1 on using Postgres. I’ve played around with Postgres and pgcrypto. Keeping certs and sig keys encrypted is easily done in postgres. I used asymmetric public private key encryption with pgcrypto where I stored the public key to encrypt in a database table. The private key was stored apart from the database and used by an api that had secure access to the private key. This worked quite well and is mostly done with Postgres queries
> > >
> > > Sent from my iPhone
> > >
> > > > On Dec 7, 2020, at 4:00 PM, Gray, Jonathan <Jo...@comcast.com.invalid> wrote:
> > > >
> > > > HashiCorp Vault and/or Consul is the only other primary contender I think we've had proposed, but I'm +1/+1 as well.
> > > >
> > > > Jonathan G
> > > >
> > > > On 12/7/20, 3:58 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > > >
> > > > Yes, I agree with the plugin interface as well, but that is what I was
> > > > hoping to defer to a follow-up thread, preferably with a rough draft
> > > > of a blueprint in hand. First, I just want to get an official
> > > > consensus on PostgreSQL (in this case as the _main_ plugin
> > > > implementation).
> > > >
> > > > - Rawlin
> > > >
> > > >> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> > > >>
> > > >> +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> > > >> TO secret store to a Plugin interface, and ATC will provide a Postgres
> > > >> Plugin."
> > > >>
> > > >> I'd also like to note, I believe our company has a legal requirement to
> > > >> have a separate "secret" database, so the Postgres secret store needs to at
> > > >> least have the ability to be a separate DB URL+auth than the primary TO
> > > >> Postgres DB.
> > > >>
> > > >>
> > > >>> On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> > > >>>
> > > >>> I am +1 for using Postgres, however we should consider implementing the
> > > >>> "secret store" functionality in such a way that people can choose to
> > > >>> implement whatever backend they want. I think it can be accomplished using
> > > >>> the TO plugin functionality but I am sure people more familiar with the
> > > >>> code these days would know better. This would also provide a built in way
> > > >>> to migrate from one to the other without forcing everyone to change.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> > > >>>
> > > >>>> Hey folks,
> > > >>>>
> > > >>>> I hope by now everyone can agree that we need to replace Riak (it's
> > > >>>> been unmaintained for quite some time now). However, we might not all
> > > >>>> agree yet on what it should be replaced with (at least not
> > > >>>> officially). We've discussed it in threads here and there, but I'd
> > > >>>> like to get some official consensus before we really hit the ground
> > > >>>> running.
> > > >>>>
> > > >>>> I would like to propose that we replace Riak with PostgreSQL.
> > > >>>>
> > > >>>> Here are some of the reasons that I can think of (and have been
> > > >>>> mentioned by others in the past) for us to use PostgreSQL:
> > > >>>> - we all have much experience running it in production (because we
> > > >>>> already run it for the Traffic Ops database)
> > > >>>> - it would simplify ATC deployments by removing one more component
> > > >>>> from the system
> > > >>>> - it would simplify development as ATC devs are already familiar with
> > > >>>> traditional SQL databases, and we could reuse a lot of the existing
> > > >>>> code
> > > >>>> - it has a healthy community of support and doesn't seem to be losing
> > > >>>> steam anytime soon (it still remains the 2nd most popular OSS
> > > >>>> relational database behind MySQL [1])
> > > >>>>
> > > >>>> I would like this thread to focus on the merits (or lack thereof) of
> > > >>>> using PostgreSQL as a replacement for Riak. We can discuss the
> > > >>>> low-level implementation details separately in the blueprint I will
> > > >>>> propose as a follow-up to this discussion. Unless someone is
> > > >>>> vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > > >>>> as assent and move forward with the blueprint process.
> > > >>>>
> > > >>>> - Rawlin
> > > >>>>
> > > >>>> [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$<https://urldefense.com/v3/__https:/db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$>
> > > >>>>
> > > >>>
> > > >
> >
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by Amarnath B S <am...@gmail.com>.
Is the plugin interface planned around the "emulated vault" experimental feature already in ATC? https://github.com/apache/trafficcontrol/tree/master/experimental/emulated_vault. The emulated vault shim layer looks quite clean and extensible.
Also, just want to confirm that both the use cases of storing DS certs/keys and DNSSEC keys are planned to be covered.
Thanks,
-Amar
On 2020/12/16 18:03:53, Rawlin Peters <ra...@apache.org> wrote:
> Thank you all for the feedback. It sounds like Postgres can do the
> job, so I'll move forward with the blueprint for implementing it as a
> type of plugin interface with Postgres and Riak as the two
> implementations to begin with. If we wanted to add HashiCorp Vault (or
> something else) as another plugin implementation later on, the plugin
> interface should make that easier.
>
> - Rawlin
>
> On Mon, Dec 7, 2020 at 9:48 PM Chatterjee, Srijeet
> <SR...@comcast.com.invalid> wrote:
> >
> > +1 on moving away from Riak and using postgres instead. Riak can be a pain(atleast for new users) to set up and debug.
> >
> > --Srijeet
> >
> > From: John Rushford <jj...@gmail.com>
> > Date: Monday, December 7, 2020 at 8:07 PM
> > To: dev@trafficcontrol.apache.org <de...@trafficcontrol.apache.org>
> > Subject: Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
> > +1 on using Postgres. I’ve played around with Postgres and pgcrypto. Keeping certs and sig keys encrypted is easily done in postgres. I used asymmetric public private key encryption with pgcrypto where I stored the public key to encrypt in a database table. The private key was stored apart from the database and used by an api that had secure access to the private key. This worked quite well and is mostly done with Postgres queries
> >
> > Sent from my iPhone
> >
> > > On Dec 7, 2020, at 4:00 PM, Gray, Jonathan <Jo...@comcast.com.invalid> wrote:
> > >
> > > HashiCorp Vault and/or Consul is the only other primary contender I think we've had proposed, but I'm +1/+1 as well.
> > >
> > > Jonathan G
> > >
> > > On 12/7/20, 3:58 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > >
> > > Yes, I agree with the plugin interface as well, but that is what I was
> > > hoping to defer to a follow-up thread, preferably with a rough draft
> > > of a blueprint in hand. First, I just want to get an official
> > > consensus on PostgreSQL (in this case as the _main_ plugin
> > > implementation).
> > >
> > > - Rawlin
> > >
> > >> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> > >>
> > >> +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> > >> TO secret store to a Plugin interface, and ATC will provide a Postgres
> > >> Plugin."
> > >>
> > >> I'd also like to note, I believe our company has a legal requirement to
> > >> have a separate "secret" database, so the Postgres secret store needs to at
> > >> least have the ability to be a separate DB URL+auth than the primary TO
> > >> Postgres DB.
> > >>
> > >>
> > >>> On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> > >>>
> > >>> I am +1 for using Postgres, however we should consider implementing the
> > >>> "secret store" functionality in such a way that people can choose to
> > >>> implement whatever backend they want. I think it can be accomplished using
> > >>> the TO plugin functionality but I am sure people more familiar with the
> > >>> code these days would know better. This would also provide a built in way
> > >>> to migrate from one to the other without forcing everyone to change.
> > >>>
> > >>>
> > >>>
> > >>>> On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> > >>>
> > >>>> Hey folks,
> > >>>>
> > >>>> I hope by now everyone can agree that we need to replace Riak (it's
> > >>>> been unmaintained for quite some time now). However, we might not all
> > >>>> agree yet on what it should be replaced with (at least not
> > >>>> officially). We've discussed it in threads here and there, but I'd
> > >>>> like to get some official consensus before we really hit the ground
> > >>>> running.
> > >>>>
> > >>>> I would like to propose that we replace Riak with PostgreSQL.
> > >>>>
> > >>>> Here are some of the reasons that I can think of (and have been
> > >>>> mentioned by others in the past) for us to use PostgreSQL:
> > >>>> - we all have much experience running it in production (because we
> > >>>> already run it for the Traffic Ops database)
> > >>>> - it would simplify ATC deployments by removing one more component
> > >>>> from the system
> > >>>> - it would simplify development as ATC devs are already familiar with
> > >>>> traditional SQL databases, and we could reuse a lot of the existing
> > >>>> code
> > >>>> - it has a healthy community of support and doesn't seem to be losing
> > >>>> steam anytime soon (it still remains the 2nd most popular OSS
> > >>>> relational database behind MySQL [1])
> > >>>>
> > >>>> I would like this thread to focus on the merits (or lack thereof) of
> > >>>> using PostgreSQL as a replacement for Riak. We can discuss the
> > >>>> low-level implementation details separately in the blueprint I will
> > >>>> propose as a follow-up to this discussion. Unless someone is
> > >>>> vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > >>>> as assent and move forward with the blueprint process.
> > >>>>
> > >>>> - Rawlin
> > >>>>
> > >>>> [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$<https://urldefense.com/v3/__https:/db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$>
> > >>>>
> > >>>
> > >
>
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by Rawlin Peters <ra...@apache.org>.
Thank you all for the feedback. It sounds like Postgres can do the
job, so I'll move forward with the blueprint for implementing it as a
type of plugin interface with Postgres and Riak as the two
implementations to begin with. If we wanted to add HashiCorp Vault (or
something else) as another plugin implementation later on, the plugin
interface should make that easier.
- Rawlin
On Mon, Dec 7, 2020 at 9:48 PM Chatterjee, Srijeet
<SR...@comcast.com.invalid> wrote:
>
> +1 on moving away from Riak and using postgres instead. Riak can be a pain(atleast for new users) to set up and debug.
>
> --Srijeet
>
> From: John Rushford <jj...@gmail.com>
> Date: Monday, December 7, 2020 at 8:07 PM
> To: dev@trafficcontrol.apache.org <de...@trafficcontrol.apache.org>
> Subject: Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
> +1 on using Postgres. I’ve played around with Postgres and pgcrypto. Keeping certs and sig keys encrypted is easily done in postgres. I used asymmetric public private key encryption with pgcrypto where I stored the public key to encrypt in a database table. The private key was stored apart from the database and used by an api that had secure access to the private key. This worked quite well and is mostly done with Postgres queries
>
> Sent from my iPhone
>
> > On Dec 7, 2020, at 4:00 PM, Gray, Jonathan <Jo...@comcast.com.invalid> wrote:
> >
> > HashiCorp Vault and/or Consul is the only other primary contender I think we've had proposed, but I'm +1/+1 as well.
> >
> > Jonathan G
> >
> > On 12/7/20, 3:58 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> >
> > Yes, I agree with the plugin interface as well, but that is what I was
> > hoping to defer to a follow-up thread, preferably with a rough draft
> > of a blueprint in hand. First, I just want to get an official
> > consensus on PostgreSQL (in this case as the _main_ plugin
> > implementation).
> >
> > - Rawlin
> >
> >> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> >>
> >> +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> >> TO secret store to a Plugin interface, and ATC will provide a Postgres
> >> Plugin."
> >>
> >> I'd also like to note, I believe our company has a legal requirement to
> >> have a separate "secret" database, so the Postgres secret store needs to at
> >> least have the ability to be a separate DB URL+auth than the primary TO
> >> Postgres DB.
> >>
> >>
> >>> On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> >>>
> >>> I am +1 for using Postgres, however we should consider implementing the
> >>> "secret store" functionality in such a way that people can choose to
> >>> implement whatever backend they want. I think it can be accomplished using
> >>> the TO plugin functionality but I am sure people more familiar with the
> >>> code these days would know better. This would also provide a built in way
> >>> to migrate from one to the other without forcing everyone to change.
> >>>
> >>>
> >>>
> >>>> On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> >>>
> >>>> Hey folks,
> >>>>
> >>>> I hope by now everyone can agree that we need to replace Riak (it's
> >>>> been unmaintained for quite some time now). However, we might not all
> >>>> agree yet on what it should be replaced with (at least not
> >>>> officially). We've discussed it in threads here and there, but I'd
> >>>> like to get some official consensus before we really hit the ground
> >>>> running.
> >>>>
> >>>> I would like to propose that we replace Riak with PostgreSQL.
> >>>>
> >>>> Here are some of the reasons that I can think of (and have been
> >>>> mentioned by others in the past) for us to use PostgreSQL:
> >>>> - we all have much experience running it in production (because we
> >>>> already run it for the Traffic Ops database)
> >>>> - it would simplify ATC deployments by removing one more component
> >>>> from the system
> >>>> - it would simplify development as ATC devs are already familiar with
> >>>> traditional SQL databases, and we could reuse a lot of the existing
> >>>> code
> >>>> - it has a healthy community of support and doesn't seem to be losing
> >>>> steam anytime soon (it still remains the 2nd most popular OSS
> >>>> relational database behind MySQL [1])
> >>>>
> >>>> I would like this thread to focus on the merits (or lack thereof) of
> >>>> using PostgreSQL as a replacement for Riak. We can discuss the
> >>>> low-level implementation details separately in the blueprint I will
> >>>> propose as a follow-up to this discussion. Unless someone is
> >>>> vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> >>>> as assent and move forward with the blueprint process.
> >>>>
> >>>> - Rawlin
> >>>>
> >>>> [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$<https://urldefense.com/v3/__https:/db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$>
> >>>>
> >>>
> >
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by "Chatterjee, Srijeet" <SR...@comcast.com.INVALID>.
+1 on moving away from Riak and using postgres instead. Riak can be a pain(atleast for new users) to set up and debug.
--Srijeet
From: John Rushford <jj...@gmail.com>
Date: Monday, December 7, 2020 at 8:07 PM
To: dev@trafficcontrol.apache.org <de...@trafficcontrol.apache.org>
Subject: Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
+1 on using Postgres. I’ve played around with Postgres and pgcrypto. Keeping certs and sig keys encrypted is easily done in postgres. I used asymmetric public private key encryption with pgcrypto where I stored the public key to encrypt in a database table. The private key was stored apart from the database and used by an api that had secure access to the private key. This worked quite well and is mostly done with Postgres queries
Sent from my iPhone
> On Dec 7, 2020, at 4:00 PM, Gray, Jonathan <Jo...@comcast.com.invalid> wrote:
>
> HashiCorp Vault and/or Consul is the only other primary contender I think we've had proposed, but I'm +1/+1 as well.
>
> Jonathan G
>
> On 12/7/20, 3:58 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> Yes, I agree with the plugin interface as well, but that is what I was
> hoping to defer to a follow-up thread, preferably with a rough draft
> of a blueprint in hand. First, I just want to get an official
> consensus on PostgreSQL (in this case as the _main_ plugin
> implementation).
>
> - Rawlin
>
>> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
>>
>> +1 and +1 to what @neuman said. I'd vote this be framed more like "change
>> TO secret store to a Plugin interface, and ATC will provide a Postgres
>> Plugin."
>>
>> I'd also like to note, I believe our company has a legal requirement to
>> have a separate "secret" database, so the Postgres secret store needs to at
>> least have the ability to be a separate DB URL+auth than the primary TO
>> Postgres DB.
>>
>>
>>> On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
>>>
>>> I am +1 for using Postgres, however we should consider implementing the
>>> "secret store" functionality in such a way that people can choose to
>>> implement whatever backend they want. I think it can be accomplished using
>>> the TO plugin functionality but I am sure people more familiar with the
>>> code these days would know better. This would also provide a built in way
>>> to migrate from one to the other without forcing everyone to change.
>>>
>>>
>>>
>>>> On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
>>>
>>>> Hey folks,
>>>>
>>>> I hope by now everyone can agree that we need to replace Riak (it's
>>>> been unmaintained for quite some time now). However, we might not all
>>>> agree yet on what it should be replaced with (at least not
>>>> officially). We've discussed it in threads here and there, but I'd
>>>> like to get some official consensus before we really hit the ground
>>>> running.
>>>>
>>>> I would like to propose that we replace Riak with PostgreSQL.
>>>>
>>>> Here are some of the reasons that I can think of (and have been
>>>> mentioned by others in the past) for us to use PostgreSQL:
>>>> - we all have much experience running it in production (because we
>>>> already run it for the Traffic Ops database)
>>>> - it would simplify ATC deployments by removing one more component
>>>> from the system
>>>> - it would simplify development as ATC devs are already familiar with
>>>> traditional SQL databases, and we could reuse a lot of the existing
>>>> code
>>>> - it has a healthy community of support and doesn't seem to be losing
>>>> steam anytime soon (it still remains the 2nd most popular OSS
>>>> relational database behind MySQL [1])
>>>>
>>>> I would like this thread to focus on the merits (or lack thereof) of
>>>> using PostgreSQL as a replacement for Riak. We can discuss the
>>>> low-level implementation details separately in the blueprint I will
>>>> propose as a follow-up to this discussion. Unless someone is
>>>> vehemently -1 on using PostgreSQL to replace Riak, I will take silence
>>>> as assent and move forward with the blueprint process.
>>>>
>>>> - Rawlin
>>>>
>>>> [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$<https://urldefense.com/v3/__https:/db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$>
>>>>
>>>
>
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by John Rushford <jj...@gmail.com>.
+1 on using Postgres. I’ve played around with Postgres and pgcrypto. Keeping certs and sig keys encrypted is easily done in postgres. I used asymmetric public private key encryption with pgcrypto where I stored the public key to encrypt in a database table. The private key was stored apart from the database and used by an api that had secure access to the private key. This worked quite well and is mostly done with Postgres queries
Sent from my iPhone
> On Dec 7, 2020, at 4:00 PM, Gray, Jonathan <Jo...@comcast.com.invalid> wrote:
>
> HashiCorp Vault and/or Consul is the only other primary contender I think we've had proposed, but I'm +1/+1 as well.
>
> Jonathan G
>
> On 12/7/20, 3:58 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> Yes, I agree with the plugin interface as well, but that is what I was
> hoping to defer to a follow-up thread, preferably with a rough draft
> of a blueprint in hand. First, I just want to get an official
> consensus on PostgreSQL (in this case as the _main_ plugin
> implementation).
>
> - Rawlin
>
>> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
>>
>> +1 and +1 to what @neuman said. I'd vote this be framed more like "change
>> TO secret store to a Plugin interface, and ATC will provide a Postgres
>> Plugin."
>>
>> I'd also like to note, I believe our company has a legal requirement to
>> have a separate "secret" database, so the Postgres secret store needs to at
>> least have the ability to be a separate DB URL+auth than the primary TO
>> Postgres DB.
>>
>>
>>> On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
>>>
>>> I am +1 for using Postgres, however we should consider implementing the
>>> "secret store" functionality in such a way that people can choose to
>>> implement whatever backend they want. I think it can be accomplished using
>>> the TO plugin functionality but I am sure people more familiar with the
>>> code these days would know better. This would also provide a built in way
>>> to migrate from one to the other without forcing everyone to change.
>>>
>>>
>>>
>>>> On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
>>>
>>>> Hey folks,
>>>>
>>>> I hope by now everyone can agree that we need to replace Riak (it's
>>>> been unmaintained for quite some time now). However, we might not all
>>>> agree yet on what it should be replaced with (at least not
>>>> officially). We've discussed it in threads here and there, but I'd
>>>> like to get some official consensus before we really hit the ground
>>>> running.
>>>>
>>>> I would like to propose that we replace Riak with PostgreSQL.
>>>>
>>>> Here are some of the reasons that I can think of (and have been
>>>> mentioned by others in the past) for us to use PostgreSQL:
>>>> - we all have much experience running it in production (because we
>>>> already run it for the Traffic Ops database)
>>>> - it would simplify ATC deployments by removing one more component
>>>> from the system
>>>> - it would simplify development as ATC devs are already familiar with
>>>> traditional SQL databases, and we could reuse a lot of the existing
>>>> code
>>>> - it has a healthy community of support and doesn't seem to be losing
>>>> steam anytime soon (it still remains the 2nd most popular OSS
>>>> relational database behind MySQL [1])
>>>>
>>>> I would like this thread to focus on the merits (or lack thereof) of
>>>> using PostgreSQL as a replacement for Riak. We can discuss the
>>>> low-level implementation details separately in the blueprint I will
>>>> propose as a follow-up to this discussion. Unless someone is
>>>> vehemently -1 on using PostgreSQL to replace Riak, I will take silence
>>>> as assent and move forward with the blueprint process.
>>>>
>>>> - Rawlin
>>>>
>>>> [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$
>>>>
>>>
>
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by Zach Hoffman <ho...@gmail.com>.
+1 on PostgreSQL so we can stop coding around Riak bugs. We can work out
the logistics later, but there are definitely secure ways to do it.
-Zach
On Mon, Dec 7, 2020 at 4:00 PM Gray, Jonathan
<Jo...@comcast.com.invalid> wrote:
> HashiCorp Vault and/or Consul is the only other primary contender I think
> we've had proposed, but I'm +1/+1 as well.
>
> Jonathan G
>
> On 12/7/20, 3:58 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> Yes, I agree with the plugin interface as well, but that is what I was
> hoping to defer to a follow-up thread, preferably with a rough draft
> of a blueprint in hand. First, I just want to get an official
> consensus on PostgreSQL (in this case as the _main_ plugin
> implementation).
>
> - Rawlin
>
> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> >
> > +1 and +1 to what @neuman said. I'd vote this be framed more like
> "change
> > TO secret store to a Plugin interface, and ATC will provide a
> Postgres
> > Plugin."
> >
> > I'd also like to note, I believe our company has a legal requirement
> to
> > have a separate "secret" database, so the Postgres secret store
> needs to at
> > least have the ability to be a separate DB URL+auth than the primary
> TO
> > Postgres DB.
> >
> >
> > On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org>
> wrote:
> >
> > > I am +1 for using Postgres, however we should consider
> implementing the
> > > "secret store" functionality in such a way that people can choose
> to
> > > implement whatever backend they want. I think it can be
> accomplished using
> > > the TO plugin functionality but I am sure people more familiar
> with the
> > > code these days would know better. This would also provide a
> built in way
> > > to migrate from one to the other without forcing everyone to
> change.
> > >
> > >
> > >
> > > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org>
> wrote:
> > >
> > > > Hey folks,
> > > >
> > > > I hope by now everyone can agree that we need to replace Riak
> (it's
> > > > been unmaintained for quite some time now). However, we might
> not all
> > > > agree yet on what it should be replaced with (at least not
> > > > officially). We've discussed it in threads here and there, but
> I'd
> > > > like to get some official consensus before we really hit the
> ground
> > > > running.
> > > >
> > > > I would like to propose that we replace Riak with PostgreSQL.
> > > >
> > > > Here are some of the reasons that I can think of (and have been
> > > > mentioned by others in the past) for us to use PostgreSQL:
> > > > - we all have much experience running it in production (because
> we
> > > > already run it for the Traffic Ops database)
> > > > - it would simplify ATC deployments by removing one more
> component
> > > > from the system
> > > > - it would simplify development as ATC devs are already familiar
> with
> > > > traditional SQL databases, and we could reuse a lot of the
> existing
> > > > code
> > > > - it has a healthy community of support and doesn't seem to be
> losing
> > > > steam anytime soon (it still remains the 2nd most popular OSS
> > > > relational database behind MySQL [1])
> > > >
> > > > I would like this thread to focus on the merits (or lack
> thereof) of
> > > > using PostgreSQL as a replacement for Riak. We can discuss the
> > > > low-level implementation details separately in the blueprint I
> will
> > > > propose as a follow-up to this discussion. Unless someone is
> > > > vehemently -1 on using PostgreSQL to replace Riak, I will take
> silence
> > > > as assent and move forward with the blueprint process.
> > > >
> > > > - Rawlin
> > > >
> > > > [1]
> https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$
> > > >
> > >
>
>
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by "Gray, Jonathan" <Jo...@comcast.com.INVALID>.
HashiCorp Vault and/or Consul is the only other primary contender I think we've had proposed, but I'm +1/+1 as well.
Jonathan G
On 12/7/20, 3:58 PM, "Rawlin Peters" <ra...@apache.org> wrote:
Yes, I agree with the plugin interface as well, but that is what I was
hoping to defer to a follow-up thread, preferably with a rough draft
of a blueprint in hand. First, I just want to get an official
consensus on PostgreSQL (in this case as the _main_ plugin
implementation).
- Rawlin
On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
>
> +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> TO secret store to a Plugin interface, and ATC will provide a Postgres
> Plugin."
>
> I'd also like to note, I believe our company has a legal requirement to
> have a separate "secret" database, so the Postgres secret store needs to at
> least have the ability to be a separate DB URL+auth than the primary TO
> Postgres DB.
>
>
> On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
>
> > I am +1 for using Postgres, however we should consider implementing the
> > "secret store" functionality in such a way that people can choose to
> > implement whatever backend they want. I think it can be accomplished using
> > the TO plugin functionality but I am sure people more familiar with the
> > code these days would know better. This would also provide a built in way
> > to migrate from one to the other without forcing everyone to change.
> >
> >
> >
> > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> >
> > > Hey folks,
> > >
> > > I hope by now everyone can agree that we need to replace Riak (it's
> > > been unmaintained for quite some time now). However, we might not all
> > > agree yet on what it should be replaced with (at least not
> > > officially). We've discussed it in threads here and there, but I'd
> > > like to get some official consensus before we really hit the ground
> > > running.
> > >
> > > I would like to propose that we replace Riak with PostgreSQL.
> > >
> > > Here are some of the reasons that I can think of (and have been
> > > mentioned by others in the past) for us to use PostgreSQL:
> > > - we all have much experience running it in production (because we
> > > already run it for the Traffic Ops database)
> > > - it would simplify ATC deployments by removing one more component
> > > from the system
> > > - it would simplify development as ATC devs are already familiar with
> > > traditional SQL databases, and we could reuse a lot of the existing
> > > code
> > > - it has a healthy community of support and doesn't seem to be losing
> > > steam anytime soon (it still remains the 2nd most popular OSS
> > > relational database behind MySQL [1])
> > >
> > > I would like this thread to focus on the merits (or lack thereof) of
> > > using PostgreSQL as a replacement for Riak. We can discuss the
> > > low-level implementation details separately in the blueprint I will
> > > propose as a follow-up to this discussion. Unless someone is
> > > vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > > as assent and move forward with the blueprint process.
> > >
> > > - Rawlin
> > >
> > > [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!Xy7f_5hSaPy1VOI6G3s7dnOKEWWmrpYJK1noQ67A3gTlldrSM596Zx4YjjbMHFUITPk4$
> > >
> >
Re: Replace Riak w/ PostgreSQL
Posted by Matthew Jackson <mj...@alumni.nd.edu>.
+1 as well
On Mon, Dec 7, 2020 at 4:09 PM Derek Gelinas <mr...@gmail.com> wrote:
> +1
> On Dec 7, 2020, 5:58 PM -0500, Rawlin Peters <ra...@apache.org>, wrote:
> > Yes, I agree with the plugin interface as well, but that is what I was
> > hoping to defer to a follow-up thread, preferably with a rough draft
> > of a blueprint in hand. First, I just want to get an official
> > consensus on PostgreSQL (in this case as the _main_ plugin
> > implementation).
> >
> > - Rawlin
> >
> > On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> > >
> > > +1 and +1 to what @neuman said. I'd vote this be framed more like
> "change
> > > TO secret store to a Plugin interface, and ATC will provide a Postgres
> > > Plugin."
> > >
> > > I'd also like to note, I believe our company has a legal requirement to
> > > have a separate "secret" database, so the Postgres secret store needs
> to at
> > > least have the ability to be a separate DB URL+auth than the primary TO
> > > Postgres DB.
> > >
> > >
> > > On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> > >
> > > > I am +1 for using Postgres, however we should consider implementing
> the
> > > > "secret store" functionality in such a way that people can choose to
> > > > implement whatever backend they want. I think it can be accomplished
> using
> > > > the TO plugin functionality but I am sure people more familiar with
> the
> > > > code these days would know better. This would also provide a built
> in way
> > > > to migrate from one to the other without forcing everyone to change.
> > > >
> > > >
> > > >
> > > > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org>
> wrote:
> > > >
> > > > > Hey folks,
> > > > >
> > > > > I hope by now everyone can agree that we need to replace Riak (it's
> > > > > been unmaintained for quite some time now). However, we might not
> all
> > > > > agree yet on what it should be replaced with (at least not
> > > > > officially). We've discussed it in threads here and there, but I'd
> > > > > like to get some official consensus before we really hit the ground
> > > > > running.
> > > > >
> > > > > I would like to propose that we replace Riak with PostgreSQL.
> > > > >
> > > > > Here are some of the reasons that I can think of (and have been
> > > > > mentioned by others in the past) for us to use PostgreSQL:
> > > > > - we all have much experience running it in production (because we
> > > > > already run it for the Traffic Ops database)
> > > > > - it would simplify ATC deployments by removing one more component
> > > > > from the system
> > > > > - it would simplify development as ATC devs are already familiar
> with
> > > > > traditional SQL databases, and we could reuse a lot of the existing
> > > > > code
> > > > > - it has a healthy community of support and doesn't seem to be
> losing
> > > > > steam anytime soon (it still remains the 2nd most popular OSS
> > > > > relational database behind MySQL [1])
> > > > >
> > > > > I would like this thread to focus on the merits (or lack thereof)
> of
> > > > > using PostgreSQL as a replacement for Riak. We can discuss the
> > > > > low-level implementation details separately in the blueprint I will
> > > > > propose as a follow-up to this discussion. Unless someone is
> > > > > vehemently -1 on using PostgreSQL to replace Riak, I will take
> silence
> > > > > as assent and move forward with the blueprint process.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > [1] https://db-engines.com/en/ranking_osvsc
> > > > >
> > > >
>
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by "Gray, Jonathan" <Jo...@comcast.com.INVALID>.
Agreed. I raised it as an option because we've contemplated them before in past iterations of this conversation and didn't want it to be completely one-sided. If riak or its replacement went down, it wouldn't be any worse than losing TODB. Both pieces have to work for the application stack to work properly. ATS, TM, & TR cache the data they need locally in case this were to ever occur, and failsafe should TO/TODB/TV-plugin(riak) become unavailable.
Postgres is effectively free from a technical requirement and complexity perspective (or massive reduction depending on how you look at it). Vault vs Postgres as far as security is concerned is basically the same. We've adopted the design method of application-style permissions as opposed to datastore-style permissions. An implementation of Vault wouldn't end up entailing suddenly having more than one application service account touching it. It's also a significant amount of complexity and overhead comparatively. While mostly out of scope of the project does impact implementors. When it comes to HA on postgres, there are ways to do it they're just not free out of the box (a debt to pay anyway for TODB really). If an implementor already has an infrastructure team using Vault such that it's free to them, great; the plugin api would make adding one nowhere near as painful as it is today. If a wall does need to exist with postgres, it can exist with separate credentials + separate schema or databases (even instances in extreme). If you can compromise TODB, you can compromise RIAK or any secure store for that matter via the API by simply forging an admin user account. Having it be a separate datastore is mostly a mitigation for some kind of side-channel attack where the instance is shared among multiple applications and/or a superuser account is compromised. Being that the number of connection sources and credentials are so small with today's implementation, it's much easier to lockdown via instance isolation and firewalls. At the end of the day, all the data in riak is basically just strings and integers so basically any keyed datastore can work (should is a different matter).
Jonathan G
On 12/7/20, 6:29 PM, "Dave Neuman" <ne...@apache.org> wrote:
We should offer a simple solution like Postgres as the default option and
allow people to get more elaborate if they want.
If we are being honest with ourselves, Riak is not really a secure keystore
either.
On Mon, Dec 7, 2020 at 5:34 PM Resino, Robert
<Ro...@comcast.com.invalid> wrote:
>
> As Jonathan said, Hashicorp Vault on top of Consul replication handles
> this well.
>
> Not sure PostgreSQL is a secure replacement for an actual system designed
> to store/serve secrets.
>
> Bob
>
> On 12/7/20, 4:25 PM, "Villa, Joseph" <Jo...@comcast.com.INVALID>
> wrote:
>
> Question.. If you’re replacing Riak with Postgresql and logical
> replication doesn’t have a method touchless failover, don’t you leave
> Traffic Control open to have a single point of failure?
>
> On 12/7/20, 4:10 PM, "Derek Gelinas" <mr...@gmail.com> wrote:
>
> +1
> On Dec 7, 2020, 5:58 PM -0500, Rawlin Peters <ra...@apache.org>,
> wrote:
> > Yes, I agree with the plugin interface as well, but that is what
> I was
> > hoping to defer to a follow-up thread, preferably with a rough
> draft
> > of a blueprint in hand. First, I just want to get an official
> > consensus on PostgreSQL (in this case as the _main_ plugin
> > implementation).
> >
> > - Rawlin
> >
> > On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org>
> wrote:
> > >
> > > +1 and +1 to what @neuman said. I'd vote this be framed more
> like "change
> > > TO secret store to a Plugin interface, and ATC will provide a
> Postgres
> > > Plugin."
> > >
> > > I'd also like to note, I believe our company has a legal
> requirement to
> > > have a separate "secret" database, so the Postgres secret
> store needs to at
> > > least have the ability to be a separate DB URL+auth than the
> primary TO
> > > Postgres DB.
> > >
> > >
> > > On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org>
> wrote:
> > >
> > > > I am +1 for using Postgres, however we should consider
> implementing the
> > > > "secret store" functionality in such a way that people can
> choose to
> > > > implement whatever backend they want. I think it can be
> accomplished using
> > > > the TO plugin functionality but I am sure people more
> familiar with the
> > > > code these days would know better. This would also provide a
> built in way
> > > > to migrate from one to the other without forcing everyone to
> change.
> > > >
> > > >
> > > >
> > > > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <
> rawlin@apache.org> wrote:
> > > >
> > > > > Hey folks,
> > > > >
> > > > > I hope by now everyone can agree that we need to replace
> Riak (it's
> > > > > been unmaintained for quite some time now). However, we
> might not all
> > > > > agree yet on what it should be replaced with (at least not
> > > > > officially). We've discussed it in threads here and there,
> but I'd
> > > > > like to get some official consensus before we really hit
> the ground
> > > > > running.
> > > > >
> > > > > I would like to propose that we replace Riak with
> PostgreSQL.
> > > > >
> > > > > Here are some of the reasons that I can think of (and have
> been
> > > > > mentioned by others in the past) for us to use PostgreSQL:
> > > > > - we all have much experience running it in production
> (because we
> > > > > already run it for the Traffic Ops database)
> > > > > - it would simplify ATC deployments by removing one more
> component
> > > > > from the system
> > > > > - it would simplify development as ATC devs are already
> familiar with
> > > > > traditional SQL databases, and we could reuse a lot of the
> existing
> > > > > code
> > > > > - it has a healthy community of support and doesn't seem
> to be losing
> > > > > steam anytime soon (it still remains the 2nd most popular
> OSS
> > > > > relational database behind MySQL [1])
> > > > >
> > > > > I would like this thread to focus on the merits (or lack
> thereof) of
> > > > > using PostgreSQL as a replacement for Riak. We can discuss
> the
> > > > > low-level implementation details separately in the
> blueprint I will
> > > > > propose as a follow-up to this discussion. Unless someone
> is
> > > > > vehemently -1 on using PostgreSQL to replace Riak, I will
> take silence
> > > > > as assent and move forward with the blueprint process.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > [1]
> https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!V5-XK-1VGFXUitQ0q1--84NBl9T9bbsX8TdT__z0s7RLcZEhpi5-xX0fBWbTd2Nhx6Xz$
> > > > >
> > > >
>
>
>
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by Dave Neuman <ne...@apache.org>.
We should offer a simple solution like Postgres as the default option and
allow people to get more elaborate if they want.
If we are being honest with ourselves, Riak is not really a secure keystore
either.
On Mon, Dec 7, 2020 at 5:34 PM Resino, Robert
<Ro...@comcast.com.invalid> wrote:
>
> As Jonathan said, Hashicorp Vault on top of Consul replication handles
> this well.
>
> Not sure PostgreSQL is a secure replacement for an actual system designed
> to store/serve secrets.
>
> Bob
>
> On 12/7/20, 4:25 PM, "Villa, Joseph" <Jo...@comcast.com.INVALID>
> wrote:
>
> Question.. If you’re replacing Riak with Postgresql and logical
> replication doesn’t have a method touchless failover, don’t you leave
> Traffic Control open to have a single point of failure?
>
> On 12/7/20, 4:10 PM, "Derek Gelinas" <mr...@gmail.com> wrote:
>
> +1
> On Dec 7, 2020, 5:58 PM -0500, Rawlin Peters <ra...@apache.org>,
> wrote:
> > Yes, I agree with the plugin interface as well, but that is what
> I was
> > hoping to defer to a follow-up thread, preferably with a rough
> draft
> > of a blueprint in hand. First, I just want to get an official
> > consensus on PostgreSQL (in this case as the _main_ plugin
> > implementation).
> >
> > - Rawlin
> >
> > On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org>
> wrote:
> > >
> > > +1 and +1 to what @neuman said. I'd vote this be framed more
> like "change
> > > TO secret store to a Plugin interface, and ATC will provide a
> Postgres
> > > Plugin."
> > >
> > > I'd also like to note, I believe our company has a legal
> requirement to
> > > have a separate "secret" database, so the Postgres secret
> store needs to at
> > > least have the ability to be a separate DB URL+auth than the
> primary TO
> > > Postgres DB.
> > >
> > >
> > > On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org>
> wrote:
> > >
> > > > I am +1 for using Postgres, however we should consider
> implementing the
> > > > "secret store" functionality in such a way that people can
> choose to
> > > > implement whatever backend they want. I think it can be
> accomplished using
> > > > the TO plugin functionality but I am sure people more
> familiar with the
> > > > code these days would know better. This would also provide a
> built in way
> > > > to migrate from one to the other without forcing everyone to
> change.
> > > >
> > > >
> > > >
> > > > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <
> rawlin@apache.org> wrote:
> > > >
> > > > > Hey folks,
> > > > >
> > > > > I hope by now everyone can agree that we need to replace
> Riak (it's
> > > > > been unmaintained for quite some time now). However, we
> might not all
> > > > > agree yet on what it should be replaced with (at least not
> > > > > officially). We've discussed it in threads here and there,
> but I'd
> > > > > like to get some official consensus before we really hit
> the ground
> > > > > running.
> > > > >
> > > > > I would like to propose that we replace Riak with
> PostgreSQL.
> > > > >
> > > > > Here are some of the reasons that I can think of (and have
> been
> > > > > mentioned by others in the past) for us to use PostgreSQL:
> > > > > - we all have much experience running it in production
> (because we
> > > > > already run it for the Traffic Ops database)
> > > > > - it would simplify ATC deployments by removing one more
> component
> > > > > from the system
> > > > > - it would simplify development as ATC devs are already
> familiar with
> > > > > traditional SQL databases, and we could reuse a lot of the
> existing
> > > > > code
> > > > > - it has a healthy community of support and doesn't seem
> to be losing
> > > > > steam anytime soon (it still remains the 2nd most popular
> OSS
> > > > > relational database behind MySQL [1])
> > > > >
> > > > > I would like this thread to focus on the merits (or lack
> thereof) of
> > > > > using PostgreSQL as a replacement for Riak. We can discuss
> the
> > > > > low-level implementation details separately in the
> blueprint I will
> > > > > propose as a follow-up to this discussion. Unless someone
> is
> > > > > vehemently -1 on using PostgreSQL to replace Riak, I will
> take silence
> > > > > as assent and move forward with the blueprint process.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > [1]
> https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!V5-XK-1VGFXUitQ0q1--84NBl9T9bbsX8TdT__z0s7RLcZEhpi5-xX0fBWbTd2Nhx6Xz$
> > > > >
> > > >
>
>
>
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by "Resino, Robert" <Ro...@comcast.com.INVALID>.
As Jonathan said, Hashicorp Vault on top of Consul replication handles this well.
Not sure PostgreSQL is a secure replacement for an actual system designed to store/serve secrets.
Bob
On 12/7/20, 4:25 PM, "Villa, Joseph" <Jo...@comcast.com.INVALID> wrote:
Question.. If you’re replacing Riak with Postgresql and logical replication doesn’t have a method touchless failover, don’t you leave Traffic Control open to have a single point of failure?
On 12/7/20, 4:10 PM, "Derek Gelinas" <mr...@gmail.com> wrote:
+1
On Dec 7, 2020, 5:58 PM -0500, Rawlin Peters <ra...@apache.org>, wrote:
> Yes, I agree with the plugin interface as well, but that is what I was
> hoping to defer to a follow-up thread, preferably with a rough draft
> of a blueprint in hand. First, I just want to get an official
> consensus on PostgreSQL (in this case as the _main_ plugin
> implementation).
>
> - Rawlin
>
> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> >
> > +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> > TO secret store to a Plugin interface, and ATC will provide a Postgres
> > Plugin."
> >
> > I'd also like to note, I believe our company has a legal requirement to
> > have a separate "secret" database, so the Postgres secret store needs to at
> > least have the ability to be a separate DB URL+auth than the primary TO
> > Postgres DB.
> >
> >
> > On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> >
> > > I am +1 for using Postgres, however we should consider implementing the
> > > "secret store" functionality in such a way that people can choose to
> > > implement whatever backend they want. I think it can be accomplished using
> > > the TO plugin functionality but I am sure people more familiar with the
> > > code these days would know better. This would also provide a built in way
> > > to migrate from one to the other without forcing everyone to change.
> > >
> > >
> > >
> > > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> > >
> > > > Hey folks,
> > > >
> > > > I hope by now everyone can agree that we need to replace Riak (it's
> > > > been unmaintained for quite some time now). However, we might not all
> > > > agree yet on what it should be replaced with (at least not
> > > > officially). We've discussed it in threads here and there, but I'd
> > > > like to get some official consensus before we really hit the ground
> > > > running.
> > > >
> > > > I would like to propose that we replace Riak with PostgreSQL.
> > > >
> > > > Here are some of the reasons that I can think of (and have been
> > > > mentioned by others in the past) for us to use PostgreSQL:
> > > > - we all have much experience running it in production (because we
> > > > already run it for the Traffic Ops database)
> > > > - it would simplify ATC deployments by removing one more component
> > > > from the system
> > > > - it would simplify development as ATC devs are already familiar with
> > > > traditional SQL databases, and we could reuse a lot of the existing
> > > > code
> > > > - it has a healthy community of support and doesn't seem to be losing
> > > > steam anytime soon (it still remains the 2nd most popular OSS
> > > > relational database behind MySQL [1])
> > > >
> > > > I would like this thread to focus on the merits (or lack thereof) of
> > > > using PostgreSQL as a replacement for Riak. We can discuss the
> > > > low-level implementation details separately in the blueprint I will
> > > > propose as a follow-up to this discussion. Unless someone is
> > > > vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > > > as assent and move forward with the blueprint process.
> > > >
> > > > - Rawlin
> > > >
> > > > [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!V5-XK-1VGFXUitQ0q1--84NBl9T9bbsX8TdT__z0s7RLcZEhpi5-xX0fBWbTd2Nhx6Xz$
> > > >
> > >
Re: [EXTERNAL] Re: Replace Riak w/ PostgreSQL
Posted by "Villa, Joseph" <Jo...@comcast.com.INVALID>.
Question.. If you’re replacing Riak with Postgresql and logical replication doesn’t have a method touchless failover, don’t you leave Traffic Control open to have a single point of failure?
On 12/7/20, 4:10 PM, "Derek Gelinas" <mr...@gmail.com> wrote:
+1
On Dec 7, 2020, 5:58 PM -0500, Rawlin Peters <ra...@apache.org>, wrote:
> Yes, I agree with the plugin interface as well, but that is what I was
> hoping to defer to a follow-up thread, preferably with a rough draft
> of a blueprint in hand. First, I just want to get an official
> consensus on PostgreSQL (in this case as the _main_ plugin
> implementation).
>
> - Rawlin
>
> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> >
> > +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> > TO secret store to a Plugin interface, and ATC will provide a Postgres
> > Plugin."
> >
> > I'd also like to note, I believe our company has a legal requirement to
> > have a separate "secret" database, so the Postgres secret store needs to at
> > least have the ability to be a separate DB URL+auth than the primary TO
> > Postgres DB.
> >
> >
> > On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> >
> > > I am +1 for using Postgres, however we should consider implementing the
> > > "secret store" functionality in such a way that people can choose to
> > > implement whatever backend they want. I think it can be accomplished using
> > > the TO plugin functionality but I am sure people more familiar with the
> > > code these days would know better. This would also provide a built in way
> > > to migrate from one to the other without forcing everyone to change.
> > >
> > >
> > >
> > > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> > >
> > > > Hey folks,
> > > >
> > > > I hope by now everyone can agree that we need to replace Riak (it's
> > > > been unmaintained for quite some time now). However, we might not all
> > > > agree yet on what it should be replaced with (at least not
> > > > officially). We've discussed it in threads here and there, but I'd
> > > > like to get some official consensus before we really hit the ground
> > > > running.
> > > >
> > > > I would like to propose that we replace Riak with PostgreSQL.
> > > >
> > > > Here are some of the reasons that I can think of (and have been
> > > > mentioned by others in the past) for us to use PostgreSQL:
> > > > - we all have much experience running it in production (because we
> > > > already run it for the Traffic Ops database)
> > > > - it would simplify ATC deployments by removing one more component
> > > > from the system
> > > > - it would simplify development as ATC devs are already familiar with
> > > > traditional SQL databases, and we could reuse a lot of the existing
> > > > code
> > > > - it has a healthy community of support and doesn't seem to be losing
> > > > steam anytime soon (it still remains the 2nd most popular OSS
> > > > relational database behind MySQL [1])
> > > >
> > > > I would like this thread to focus on the merits (or lack thereof) of
> > > > using PostgreSQL as a replacement for Riak. We can discuss the
> > > > low-level implementation details separately in the blueprint I will
> > > > propose as a follow-up to this discussion. Unless someone is
> > > > vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > > > as assent and move forward with the blueprint process.
> > > >
> > > > - Rawlin
> > > >
> > > > [1] https://urldefense.com/v3/__https://db-engines.com/en/ranking_osvsc__;!!CQl3mcHX2A!V5-XK-1VGFXUitQ0q1--84NBl9T9bbsX8TdT__z0s7RLcZEhpi5-xX0fBWbTd2Nhx6Xz$
> > > >
> > >
Re: Replace Riak w/ PostgreSQL
Posted by Derek Gelinas <mr...@gmail.com>.
+1
On Dec 7, 2020, 5:58 PM -0500, Rawlin Peters <ra...@apache.org>, wrote:
> Yes, I agree with the plugin interface as well, but that is what I was
> hoping to defer to a follow-up thread, preferably with a rough draft
> of a blueprint in hand. First, I just want to get an official
> consensus on PostgreSQL (in this case as the _main_ plugin
> implementation).
>
> - Rawlin
>
> On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
> >
> > +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> > TO secret store to a Plugin interface, and ATC will provide a Postgres
> > Plugin."
> >
> > I'd also like to note, I believe our company has a legal requirement to
> > have a separate "secret" database, so the Postgres secret store needs to at
> > least have the ability to be a separate DB URL+auth than the primary TO
> > Postgres DB.
> >
> >
> > On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> >
> > > I am +1 for using Postgres, however we should consider implementing the
> > > "secret store" functionality in such a way that people can choose to
> > > implement whatever backend they want. I think it can be accomplished using
> > > the TO plugin functionality but I am sure people more familiar with the
> > > code these days would know better. This would also provide a built in way
> > > to migrate from one to the other without forcing everyone to change.
> > >
> > >
> > >
> > > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> > >
> > > > Hey folks,
> > > >
> > > > I hope by now everyone can agree that we need to replace Riak (it's
> > > > been unmaintained for quite some time now). However, we might not all
> > > > agree yet on what it should be replaced with (at least not
> > > > officially). We've discussed it in threads here and there, but I'd
> > > > like to get some official consensus before we really hit the ground
> > > > running.
> > > >
> > > > I would like to propose that we replace Riak with PostgreSQL.
> > > >
> > > > Here are some of the reasons that I can think of (and have been
> > > > mentioned by others in the past) for us to use PostgreSQL:
> > > > - we all have much experience running it in production (because we
> > > > already run it for the Traffic Ops database)
> > > > - it would simplify ATC deployments by removing one more component
> > > > from the system
> > > > - it would simplify development as ATC devs are already familiar with
> > > > traditional SQL databases, and we could reuse a lot of the existing
> > > > code
> > > > - it has a healthy community of support and doesn't seem to be losing
> > > > steam anytime soon (it still remains the 2nd most popular OSS
> > > > relational database behind MySQL [1])
> > > >
> > > > I would like this thread to focus on the merits (or lack thereof) of
> > > > using PostgreSQL as a replacement for Riak. We can discuss the
> > > > low-level implementation details separately in the blueprint I will
> > > > propose as a follow-up to this discussion. Unless someone is
> > > > vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > > > as assent and move forward with the blueprint process.
> > > >
> > > > - Rawlin
> > > >
> > > > [1] https://db-engines.com/en/ranking_osvsc
> > > >
> > >
Re: Replace Riak w/ PostgreSQL
Posted by Rawlin Peters <ra...@apache.org>.
Yes, I agree with the plugin interface as well, but that is what I was
hoping to defer to a follow-up thread, preferably with a rough draft
of a blueprint in hand. First, I just want to get an official
consensus on PostgreSQL (in this case as the _main_ plugin
implementation).
- Rawlin
On Mon, Dec 7, 2020 at 2:24 PM Robert O Butts <ro...@apache.org> wrote:
>
> +1 and +1 to what @neuman said. I'd vote this be framed more like "change
> TO secret store to a Plugin interface, and ATC will provide a Postgres
> Plugin."
>
> I'd also like to note, I believe our company has a legal requirement to
> have a separate "secret" database, so the Postgres secret store needs to at
> least have the ability to be a separate DB URL+auth than the primary TO
> Postgres DB.
>
>
> On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
>
> > I am +1 for using Postgres, however we should consider implementing the
> > "secret store" functionality in such a way that people can choose to
> > implement whatever backend they want. I think it can be accomplished using
> > the TO plugin functionality but I am sure people more familiar with the
> > code these days would know better. This would also provide a built in way
> > to migrate from one to the other without forcing everyone to change.
> >
> >
> >
> > On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> >
> > > Hey folks,
> > >
> > > I hope by now everyone can agree that we need to replace Riak (it's
> > > been unmaintained for quite some time now). However, we might not all
> > > agree yet on what it should be replaced with (at least not
> > > officially). We've discussed it in threads here and there, but I'd
> > > like to get some official consensus before we really hit the ground
> > > running.
> > >
> > > I would like to propose that we replace Riak with PostgreSQL.
> > >
> > > Here are some of the reasons that I can think of (and have been
> > > mentioned by others in the past) for us to use PostgreSQL:
> > > - we all have much experience running it in production (because we
> > > already run it for the Traffic Ops database)
> > > - it would simplify ATC deployments by removing one more component
> > > from the system
> > > - it would simplify development as ATC devs are already familiar with
> > > traditional SQL databases, and we could reuse a lot of the existing
> > > code
> > > - it has a healthy community of support and doesn't seem to be losing
> > > steam anytime soon (it still remains the 2nd most popular OSS
> > > relational database behind MySQL [1])
> > >
> > > I would like this thread to focus on the merits (or lack thereof) of
> > > using PostgreSQL as a replacement for Riak. We can discuss the
> > > low-level implementation details separately in the blueprint I will
> > > propose as a follow-up to this discussion. Unless someone is
> > > vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > > as assent and move forward with the blueprint process.
> > >
> > > - Rawlin
> > >
> > > [1] https://db-engines.com/en/ranking_osvsc
> > >
> >
Re: Replace Riak w/ PostgreSQL
Posted by Robert O Butts <ro...@apache.org>.
+1 and +1 to what @neuman said. I'd vote this be framed more like "change
TO secret store to a Plugin interface, and ATC will provide a Postgres
Plugin."
I'd also like to note, I believe our company has a legal requirement to
have a separate "secret" database, so the Postgres secret store needs to at
least have the ability to be a separate DB URL+auth than the primary TO
Postgres DB.
On Mon, Dec 7, 2020 at 2:13 PM Dave Neuman <ne...@apache.org> wrote:
> I am +1 for using Postgres, however we should consider implementing the
> "secret store" functionality in such a way that people can choose to
> implement whatever backend they want. I think it can be accomplished using
> the TO plugin functionality but I am sure people more familiar with the
> code these days would know better. This would also provide a built in way
> to migrate from one to the other without forcing everyone to change.
>
>
>
> On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
>
> > Hey folks,
> >
> > I hope by now everyone can agree that we need to replace Riak (it's
> > been unmaintained for quite some time now). However, we might not all
> > agree yet on what it should be replaced with (at least not
> > officially). We've discussed it in threads here and there, but I'd
> > like to get some official consensus before we really hit the ground
> > running.
> >
> > I would like to propose that we replace Riak with PostgreSQL.
> >
> > Here are some of the reasons that I can think of (and have been
> > mentioned by others in the past) for us to use PostgreSQL:
> > - we all have much experience running it in production (because we
> > already run it for the Traffic Ops database)
> > - it would simplify ATC deployments by removing one more component
> > from the system
> > - it would simplify development as ATC devs are already familiar with
> > traditional SQL databases, and we could reuse a lot of the existing
> > code
> > - it has a healthy community of support and doesn't seem to be losing
> > steam anytime soon (it still remains the 2nd most popular OSS
> > relational database behind MySQL [1])
> >
> > I would like this thread to focus on the merits (or lack thereof) of
> > using PostgreSQL as a replacement for Riak. We can discuss the
> > low-level implementation details separately in the blueprint I will
> > propose as a follow-up to this discussion. Unless someone is
> > vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> > as assent and move forward with the blueprint process.
> >
> > - Rawlin
> >
> > [1] https://db-engines.com/en/ranking_osvsc
> >
>
Re: Replace Riak w/ PostgreSQL
Posted by Dave Neuman <ne...@apache.org>.
I am +1 for using Postgres, however we should consider implementing the
"secret store" functionality in such a way that people can choose to
implement whatever backend they want. I think it can be accomplished using
the TO plugin functionality but I am sure people more familiar with the
code these days would know better. This would also provide a built in way
to migrate from one to the other without forcing everyone to change.
On Mon, Dec 7, 2020 at 1:48 PM Rawlin Peters <ra...@apache.org> wrote:
> Hey folks,
>
> I hope by now everyone can agree that we need to replace Riak (it's
> been unmaintained for quite some time now). However, we might not all
> agree yet on what it should be replaced with (at least not
> officially). We've discussed it in threads here and there, but I'd
> like to get some official consensus before we really hit the ground
> running.
>
> I would like to propose that we replace Riak with PostgreSQL.
>
> Here are some of the reasons that I can think of (and have been
> mentioned by others in the past) for us to use PostgreSQL:
> - we all have much experience running it in production (because we
> already run it for the Traffic Ops database)
> - it would simplify ATC deployments by removing one more component
> from the system
> - it would simplify development as ATC devs are already familiar with
> traditional SQL databases, and we could reuse a lot of the existing
> code
> - it has a healthy community of support and doesn't seem to be losing
> steam anytime soon (it still remains the 2nd most popular OSS
> relational database behind MySQL [1])
>
> I would like this thread to focus on the merits (or lack thereof) of
> using PostgreSQL as a replacement for Riak. We can discuss the
> low-level implementation details separately in the blueprint I will
> propose as a follow-up to this discussion. Unless someone is
> vehemently -1 on using PostgreSQL to replace Riak, I will take silence
> as assent and move forward with the blueprint process.
>
> - Rawlin
>
> [1] https://db-engines.com/en/ranking_osvsc
>