You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by "Gray, Jonathan" <Jo...@comcast.com> on 2018/11/10 01:02:37 UTC
Re: [EXTERNAL] The future of the db/admin.pl script
+1 There is already precedence in the repo for python for other purposes. The only caveat I would include is to be sure you include backward compatibility for python 2.x for the next year or so until it goes EOL.
Jonathan G
On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
Hey TC devs,
As we eliminate the need for Traffic Ops Perl, it will no longer
really make sense for the db/admin.pl script to be Perl as it is
today. This is because it depends on the Perl modules that are
installed via Carton for running Traffic Ops Perl. However, it's
mostly just a Perl wrapper around shell commands except for a YAML
Perl module and the `reverse_schema` command which uses the DBIx Perl
module to generate the ORM schema for Traffic Ops Perl (i.e. you've
added a new DB table/column and need to get the ORM files updated to
use it).
Without TO-Perl, there's no need for the `reverse_schema` command in
db/admin.pl and its dependency on the Perl DBIx module. At that point
db/admin.pl is just a Perl script that parses some YAML and shells out
commands.
Part of the problem with TO-Perl is that there are a bunch of random
non-API Perl scripts that basically assume all the modules in the
cpanfile are installed in the environment, even though a lot of those
dependencies are probably only required for the Perl TO API or UI. So
unless we go through all those Perl dependencies to eliminate
everything we don't really need once the Perl TO API and UI are
completely removed, we'll continue to have a handful of Perl scripts
like db/admin.pl that still require downloading and installing the
full set of TO Perl dependencies. On fresh installs, running Carton to
install these dependencies can take nearly half an hour.
I'm working on adding tests for the DB upgrade/downgrade process which
I'd like to run automatically on PR submissions, but it really only
needs the db/admin.pl script out of TO which assumes the entire set of
Perl TO dependencies even though it mostly just shells out to `goose
up` and `goose down`. I'd like the test to emulate an actual TO
environment as closely as possible to match what would actually happen
in a production DB upgrade/downgrade. Right now I'm reusing the
Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
to port db/admin.pl into Python to give it its own set of dependencies
(just a YAML package) and not require the full set of TO Perl
dependencies. Then I can spin up a much lighter-weight container with
just the TO Python packages rather than setting up a separate cpanfile
with just the Perl packages needed for db/admin.pl.
+1/-1 for adding Python as a dependency of Traffic Ops for porting
scripts like db/admin.pl?
-Rawlin
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Rawlin Peters <ra...@gmail.com>.
I'm kind of on the fence in terms of python 2.x compatibility for
something like db/admin.pl. It's a really small script, but I do think
a lot of popular python libraries are compatible with either 2.7 or
3.something. For instance, the only package I'm interested in using
for this is PyYAML, which requires "Python 2.7 or Python 3.4+". One
potential benefit of keeping 2.x compatibility is that I believe
Python 2.x is still included as part of the standard CentOS7
distribution whereas Python 3.x might require using different package
sources (EPEL?) to install from. But on the other hand I do prefer
only supporting 1 major version and allowing the use Python 3
features.
- Rawlin
On Sat, Nov 10, 2018 at 11:44 AM Dan Kirkwood <da...@gmail.com> wrote:
>
> +1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
> seem to be gaining a lot of Python expertise recently.
>
> -1 on 2.x compatibility -- writing something new with compatibility for 2
> major versions makes no sense to me. It limits the features and libraries
> that can be used and potentially doubles the amount of testing required.
>
>
>
> On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
>
> > +1, seems reasonable. I don’t really have an opinion on python 2.x
> > compatibility, but whatever makes the most sense for the amount of work is
> > what I would prefer.
> >
> > On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <Jo...@comcast.com>
> > wrote:
> >
> > > +1 There is already precedence in the repo for python for other purposes.
> > > The only caveat I would include is to be sure you include backward
> > > compatibility for python 2.x for the next year or so until it goes EOL.
> > >
> > > Jonathan G
> > >
> > > On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
> > >
> > > Hey TC devs,
> > >
> > > As we eliminate the need for Traffic Ops Perl, it will no longer
> > > really make sense for the db/admin.pl script to be Perl as it is
> > > today. This is because it depends on the Perl modules that are
> > > installed via Carton for running Traffic Ops Perl. However, it's
> > > mostly just a Perl wrapper around shell commands except for a YAML
> > > Perl module and the `reverse_schema` command which uses the DBIx Perl
> > > module to generate the ORM schema for Traffic Ops Perl (i.e. you've
> > > added a new DB table/column and need to get the ORM files updated to
> > > use it).
> > >
> > > Without TO-Perl, there's no need for the `reverse_schema` command in
> > > db/admin.pl and its dependency on the Perl DBIx module. At that
> > point
> > > db/admin.pl is just a Perl script that parses some YAML and shells
> > out
> > > commands.
> > >
> > > Part of the problem with TO-Perl is that there are a bunch of random
> > > non-API Perl scripts that basically assume all the modules in the
> > > cpanfile are installed in the environment, even though a lot of those
> > > dependencies are probably only required for the Perl TO API or UI. So
> > > unless we go through all those Perl dependencies to eliminate
> > > everything we don't really need once the Perl TO API and UI are
> > > completely removed, we'll continue to have a handful of Perl scripts
> > > like db/admin.pl that still require downloading and installing the
> > > full set of TO Perl dependencies. On fresh installs, running Carton
> > to
> > > install these dependencies can take nearly half an hour.
> > >
> > > I'm working on adding tests for the DB upgrade/downgrade process
> > which
> > > I'd like to run automatically on PR submissions, but it really only
> > > needs the db/admin.pl script out of TO which assumes the entire set
> > of
> > > Perl TO dependencies even though it mostly just shells out to `goose
> > > up` and `goose down`. I'd like the test to emulate an actual TO
> > > environment as closely as possible to match what would actually
> > happen
> > > in a production DB upgrade/downgrade. Right now I'm reusing the
> > > Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
> > > to port db/admin.pl into Python to give it its own set of
> > dependencies
> > > (just a YAML package) and not require the full set of TO Perl
> > > dependencies. Then I can spin up a much lighter-weight container with
> > > just the TO Python packages rather than setting up a separate
> > cpanfile
> > > with just the Perl packages needed for db/admin.pl.
> > >
> > > +1/-1 for adding Python as a dependency of Traffic Ops for porting
> > > scripts like db/admin.pl?
> > >
> > > -Rawlin
> > >
> > >
> > >
> >
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Dave Neuman <ne...@apache.org>.
Well said Rawlin. This thread is now 24 replies long, so I think we have
enough consensus to just build it in golang and move on.
On Thu, Nov 15, 2018 at 8:58 AM Rawlin Peters <ra...@gmail.com>
wrote:
> I think Go really is the best path forward for this. A simple Bash
> script would be nice, but the yaml parsing issue really takes that out
> of the running IMO. If we added a dbconf.env file with the same data
> as dbconf.yml except in a format that can be sourced from a Bash
> script, it would be easy enough to just generate the dbconf.yml file
> on the fly from the sourced variables in order to use it for the
> `goose` commands. But then that adds another manual step to the
> upgrade path where an admin has to either use a tool we provide to
> convert the dbconf.yml to the Bash source-able file.
>
> But, I think the Bash path is really just a pseudo-low-dependency
> solution that still probably isn't ideal because it relies on external
> commands that need to be installed onto the system from somewhere else
> (probably requiring internet access at install time). I think the
> postgres commands come from the postgresql96 package. What if we
> upgrade postgres to the latest version and those external commands are
> renamed or go away entirely?
>
> Ideally I'd think we'd be using the lib/pq and database/sql Go
> packages to run all the sql files. Then it would be pretty
> self-contained and probably wouldn't need any changes if we upgraded
> postgres to a newer version (maybe just need to update the vendored
> packages and retest). Or if we decide that FooBarDB is actually better
> than postgres, swapping it out would probably just require changing a
> package import in the Go tool rather than replacing all the external
> commands.
>
> So to summarize some of what we've discussed so far:
> - Bash would be nice because it doesn't add a lot of extra
> dependencies, except it would still rely on installing external
> commands which might require internet access at install time
> - Python could also work but requires a bunch of extra build
> dependencies/steps to essentially "compile" it into a self-contained
> executable
> - Go requires a bit more boilerplate for something that would
> typically just be a Bash script, but Go is already a dependency of
> Traffic Ops, is used for at least one other TO tool (convert_profile),
> and would just require vendoring the go-yaml package. It would also
> give us the opportunity to use native Go DB packages.
>
> At this point I think we are really just bikeshedding for something
> that takes less than a day's work to write, test, and compare
> functionality with the existing script. I've already done this twice
> now for both Python and Go, so given what we've discussed so far I'm
> inclined to just move forward with the Go version unless someone can
> come up with a compelling reason not to.
>
> - Rawlin
>
> On Wed, Nov 14, 2018 at 3:31 PM Gray, Jonathan
> <Jo...@comcast.com> wrote:
> >
> > That project is just a thin wrapper atop github.com/spf13/viper which
> has a lot more nuance and power to trip up on if you're not careful.
> >
> > Jonathan G
> >
> >
> > On 11/14/18, 2:36 PM, "Dan Kirkwood" <da...@gmail.com> wrote:
> >
> > if yaml parsing is the only barrier for bash, try this:
> > https://github.com/030/go-yq
> >
> >
> >
> > On Wed, Nov 14, 2018 at 2:18 PM Fieck, Brennan <
> Brennan_Fieck@comcast.com>
> > wrote:
> >
> > > Specifically 2.7 and 3.6? We'd want it to run on 3.4, ideally, but
> 3.5 can
> > > work. Python 2 is going away, and there's no RHEL repo for
> anything over
> > > 3.5 afaik.
> > > ________________________________________
> > > From: Chris Lemmons <al...@gmail.com>
> > > Sent: Wednesday, November 14, 2018 1:41 PM
> > > To: dev@trafficcontrol.apache.org
> > > Subject: Re: [EXTERNAL] The future of the db/admin.pl script
> > >
> > > Yeah, I looked at yq. For one... it's written in Python. :) But it
> > > does appear to be compatible with 2.7 and 3.6, so it may be easier
> to
> > > run.
> > > On Wed, Nov 14, 2018 at 1:35 PM Dewayne Richardson <
> dewrich@apache.org>
> > > wrote:
> > > >
> > > > No strong opinion here, the admin.pl was built for a different
> time in
> > > the
> > > > history of TC. The reason we used Perl was for config parsing
> back in
> > > the
> > > > day, but there is a yaml version of jq, called yq. I honestly
> agree with
> > > > bash if we can get away with it.
> > > >
> > > > FYI:
> > > > https://github.com/kislyuk/yq
> > > >
> > > > -Dew
> > > >
> > > >
> > > > On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <
> rawlin.peters@gmail.com>
> > > > wrote:
> > > >
> > > > > Eric,
> > > > >
> > > > > I share your sentiment about being reluctant to introduce
> another
> > > > > language as a dependency for Traffic Ops, but I wasn't able to
> find a
> > > > > really good, easily-available utility for parsing yaml (a la
> `jq` for
> > > > > json parsing) in a Bash script. Since the goose config is in
> yaml,
> > > > > `db/admin.pl` uses a yaml package to parse the goose config
> into
> > > > > variables which are then passed to the external `psql` et al.
> > > > > commands. It is possible to parse yaml using sed, but the
> example I
> > > > > found for doing that seemed really sketchy and fragile. So I
> figured
> > > > > using a solid YAML-parsing library like PyYAML in Python would
> be a
> > > > > safer bet while still allowing the use of a fully-featured
> programming
> > > > > language rather than "Bash + <insert yaml-parsing CLI tool
> here>". It
> > > > > would also allow us to potentially use a DB library to
> interface with
> > > > > the DB directly in Python rather than requiring `psql` et al.
> and just
> > > > > shelling out to those external commands (although I plan to
> continue
> > > > > doing it that way for now).
> > > > >
> > > > > As a side note, it also paves the way for moving other scripts
> to
> > > > > Python like WebDep.pm, which uses a Perl package that is
> virtually
> > > > > impossible to install/get running on Mac because of Perl's
> broken SSL
> > > > > on Mac, which would make it much easier to start as a new
> developer on
> > > > > the project. I remember when I started working on Traffic
> Control, I
> > > > > had to copy someone else's Perl `traffic_ops/app/local`
> directory who
> > > > > had been on the project a long time and had actually gotten it
> to
> > > > > build on Mac before it became unusable. Eliminating issues
> like that
> > > > > by using a more popular and supportable language is a win in
> my book,
> > > > > but right now I'm just focusing on `db/admin.pl` to allow for
> better
> > > > > testability of the DB migration operations.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri -
> TRITON
> > > > > UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> > > > > <ef...@cisco.com.invalid> wrote:
> > > > > >
> > > > > > I’m only slightly familiar with all the different options
> for db/
> > > > > admin.pl.
> > > > > >
> > > > > > I’m a big fan of Python, but reluctant to introduce another
> language
> > > > > into TC without a strong reason.
> > > > > >
> > > > > > Once the reverse_schema option is removed, what would be the
> main
> > > > > purposes of the script?
> > > > > >
> > > > > > Looks like this is something that could be easily converted
> to bash
> > > > > without need for another dependency.
> > > > > >
> > > > > > —Eric
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <
> dangogh@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > > +1 on rewriting admin.pl -- Python seems a reasonable
> choice, esp
> > > > > since we
> > > > > > > seem to be gaining a lot of Python expertise recently.
> > > > > > >
> > > > > > > -1 on 2.x compatibility -- writing something new with
> compatibility
> > > > > for 2
> > > > > > > major versions makes no sense to me. It limits the
> features and
> > > > > libraries
> > > > > > > that can be used and potentially doubles the amount of
> testing
> > > > > required.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <
> neuman@apache.org>
> > > wrote:
> > > > > > >
> > > > > > >> +1, seems reasonable. I don’t really have an opinion on
> python
> > > 2.x
> > > > > > >> compatibility, but whatever makes the most sense for the
> amount of
> > > > > work is
> > > > > > >> what I would prefer.
> > > > > > >>
> > > > > > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> > > > > Jonathan_Gray@comcast.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> +1 There is already precedence in the repo for python
> for other
> > > > > purposes.
> > > > > > >>> The only caveat I would include is to be sure you include
> > > backward
> > > > > > >>> compatibility for python 2.x for the next year or so
> until it
> > > goes
> > > > > EOL.
> > > > > > >>>
> > > > > > >>> Jonathan G
> > > > > > >>>
> > > > > > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <
> rawlin.peters@gmail.com>
> > > > > wrote:
> > > > > > >>>
> > > > > > >>> Hey TC devs,
> > > > > > >>>
> > > > > > >>> As we eliminate the need for Traffic Ops Perl, it
> will no
> > > longer
> > > > > > >>> really make sense for the db/admin.pl script to be
> Perl as
> > > it is
> > > > > > >>> today. This is because it depends on the Perl modules
> that are
> > > > > > >>> installed via Carton for running Traffic Ops Perl.
> However,
> > > it's
> > > > > > >>> mostly just a Perl wrapper around shell commands
> except for a
> > > YAML
> > > > > > >>> Perl module and the `reverse_schema` command which
> uses the
> > > DBIx
> > > > > Perl
> > > > > > >>> module to generate the ORM schema for Traffic Ops
> Perl (i.e.
> > > > > you've
> > > > > > >>> added a new DB table/column and need to get the ORM
> files
> > > updated
> > > > > to
> > > > > > >>> use it).
> > > > > > >>>
> > > > > > >>> Without TO-Perl, there's no need for the
> `reverse_schema`
> > > command
> > > > > in
> > > > > > >>> db/admin.pl and its dependency on the Perl DBIx
> module. At
> > > that
> > > > > > >> point
> > > > > > >>> db/admin.pl is just a Perl script that parses some
> YAML and
> > > > > shells
> > > > > > >> out
> > > > > > >>> commands.
> > > > > > >>>
> > > > > > >>> Part of the problem with TO-Perl is that there are a
> bunch of
> > > > > random
> > > > > > >>> non-API Perl scripts that basically assume all the
> modules in
> > > the
> > > > > > >>> cpanfile are installed in the environment, even
> though a lot
> > > of
> > > > > those
> > > > > > >>> dependencies are probably only required for the Perl
> TO API or
> > > > > UI. So
> > > > > > >>> unless we go through all those Perl dependencies to
> eliminate
> > > > > > >>> everything we don't really need once the Perl TO API
> and UI
> > > are
> > > > > > >>> completely removed, we'll continue to have a handful
> of Perl
> > > > > scripts
> > > > > > >>> like db/admin.pl that still require downloading and
> > > installing
> > > > > the
> > > > > > >>> full set of TO Perl dependencies. On fresh installs,
> running
> > > > > Carton
> > > > > > >> to
> > > > > > >>> install these dependencies can take nearly half an
> hour.
> > > > > > >>>
> > > > > > >>> I'm working on adding tests for the DB
> upgrade/downgrade
> > > process
> > > > > > >> which
> > > > > > >>> I'd like to run automatically on PR submissions, but
> it really
> > > > > only
> > > > > > >>> needs the db/admin.pl script out of TO which assumes
> the
> > > entire
> > > > > set
> > > > > > >> of
> > > > > > >>> Perl TO dependencies even though it mostly just
> shells out to
> > > > > `goose
> > > > > > >>> up` and `goose down`. I'd like the test to emulate an
> actual
> > > TO
> > > > > > >>> environment as closely as possible to match what would
> > > actually
> > > > > > >> happen
> > > > > > >>> in a production DB upgrade/downgrade. Right now I'm
> reusing
> > > the
> > > > > > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but
> ideally
> > > I'd
> > > > > like
> > > > > > >>> to port db/admin.pl into Python to give it its own
> set of
> > > > > > >> dependencies
> > > > > > >>> (just a YAML package) and not require the full set of
> TO Perl
> > > > > > >>> dependencies. Then I can spin up a much lighter-weight
> > > container
> > > > > with
> > > > > > >>> just the TO Python packages rather than setting up a
> separate
> > > > > > >> cpanfile
> > > > > > >>> with just the Perl packages needed for db/admin.pl.
> > > > > > >>>
> > > > > > >>> +1/-1 for adding Python as a dependency of Traffic
> Ops for
> > > porting
> > > > > > >>> scripts like db/admin.pl?
> > > > > > >>>
> > > > > > >>> -Rawlin
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > >
> >
> >
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Rawlin Peters <ra...@gmail.com>.
I think Go really is the best path forward for this. A simple Bash
script would be nice, but the yaml parsing issue really takes that out
of the running IMO. If we added a dbconf.env file with the same data
as dbconf.yml except in a format that can be sourced from a Bash
script, it would be easy enough to just generate the dbconf.yml file
on the fly from the sourced variables in order to use it for the
`goose` commands. But then that adds another manual step to the
upgrade path where an admin has to either use a tool we provide to
convert the dbconf.yml to the Bash source-able file.
But, I think the Bash path is really just a pseudo-low-dependency
solution that still probably isn't ideal because it relies on external
commands that need to be installed onto the system from somewhere else
(probably requiring internet access at install time). I think the
postgres commands come from the postgresql96 package. What if we
upgrade postgres to the latest version and those external commands are
renamed or go away entirely?
Ideally I'd think we'd be using the lib/pq and database/sql Go
packages to run all the sql files. Then it would be pretty
self-contained and probably wouldn't need any changes if we upgraded
postgres to a newer version (maybe just need to update the vendored
packages and retest). Or if we decide that FooBarDB is actually better
than postgres, swapping it out would probably just require changing a
package import in the Go tool rather than replacing all the external
commands.
So to summarize some of what we've discussed so far:
- Bash would be nice because it doesn't add a lot of extra
dependencies, except it would still rely on installing external
commands which might require internet access at install time
- Python could also work but requires a bunch of extra build
dependencies/steps to essentially "compile" it into a self-contained
executable
- Go requires a bit more boilerplate for something that would
typically just be a Bash script, but Go is already a dependency of
Traffic Ops, is used for at least one other TO tool (convert_profile),
and would just require vendoring the go-yaml package. It would also
give us the opportunity to use native Go DB packages.
At this point I think we are really just bikeshedding for something
that takes less than a day's work to write, test, and compare
functionality with the existing script. I've already done this twice
now for both Python and Go, so given what we've discussed so far I'm
inclined to just move forward with the Go version unless someone can
come up with a compelling reason not to.
- Rawlin
On Wed, Nov 14, 2018 at 3:31 PM Gray, Jonathan
<Jo...@comcast.com> wrote:
>
> That project is just a thin wrapper atop github.com/spf13/viper which has a lot more nuance and power to trip up on if you're not careful.
>
> Jonathan G
>
>
> On 11/14/18, 2:36 PM, "Dan Kirkwood" <da...@gmail.com> wrote:
>
> if yaml parsing is the only barrier for bash, try this:
> https://github.com/030/go-yq
>
>
>
> On Wed, Nov 14, 2018 at 2:18 PM Fieck, Brennan <Br...@comcast.com>
> wrote:
>
> > Specifically 2.7 and 3.6? We'd want it to run on 3.4, ideally, but 3.5 can
> > work. Python 2 is going away, and there's no RHEL repo for anything over
> > 3.5 afaik.
> > ________________________________________
> > From: Chris Lemmons <al...@gmail.com>
> > Sent: Wednesday, November 14, 2018 1:41 PM
> > To: dev@trafficcontrol.apache.org
> > Subject: Re: [EXTERNAL] The future of the db/admin.pl script
> >
> > Yeah, I looked at yq. For one... it's written in Python. :) But it
> > does appear to be compatible with 2.7 and 3.6, so it may be easier to
> > run.
> > On Wed, Nov 14, 2018 at 1:35 PM Dewayne Richardson <de...@apache.org>
> > wrote:
> > >
> > > No strong opinion here, the admin.pl was built for a different time in
> > the
> > > history of TC. The reason we used Perl was for config parsing back in
> > the
> > > day, but there is a yaml version of jq, called yq. I honestly agree with
> > > bash if we can get away with it.
> > >
> > > FYI:
> > > https://github.com/kislyuk/yq
> > >
> > > -Dew
> > >
> > >
> > > On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <ra...@gmail.com>
> > > wrote:
> > >
> > > > Eric,
> > > >
> > > > I share your sentiment about being reluctant to introduce another
> > > > language as a dependency for Traffic Ops, but I wasn't able to find a
> > > > really good, easily-available utility for parsing yaml (a la `jq` for
> > > > json parsing) in a Bash script. Since the goose config is in yaml,
> > > > `db/admin.pl` uses a yaml package to parse the goose config into
> > > > variables which are then passed to the external `psql` et al.
> > > > commands. It is possible to parse yaml using sed, but the example I
> > > > found for doing that seemed really sketchy and fragile. So I figured
> > > > using a solid YAML-parsing library like PyYAML in Python would be a
> > > > safer bet while still allowing the use of a fully-featured programming
> > > > language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> > > > would also allow us to potentially use a DB library to interface with
> > > > the DB directly in Python rather than requiring `psql` et al. and just
> > > > shelling out to those external commands (although I plan to continue
> > > > doing it that way for now).
> > > >
> > > > As a side note, it also paves the way for moving other scripts to
> > > > Python like WebDep.pm, which uses a Perl package that is virtually
> > > > impossible to install/get running on Mac because of Perl's broken SSL
> > > > on Mac, which would make it much easier to start as a new developer on
> > > > the project. I remember when I started working on Traffic Control, I
> > > > had to copy someone else's Perl `traffic_ops/app/local` directory who
> > > > had been on the project a long time and had actually gotten it to
> > > > build on Mac before it became unusable. Eliminating issues like that
> > > > by using a more popular and supportable language is a win in my book,
> > > > but right now I'm just focusing on `db/admin.pl` to allow for better
> > > > testability of the DB migration operations.
> > > >
> > > > - Rawlin
> > > >
> > > > On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> > > > UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> > > > <ef...@cisco.com.invalid> wrote:
> > > > >
> > > > > I’m only slightly familiar with all the different options for db/
> > > > admin.pl.
> > > > >
> > > > > I’m a big fan of Python, but reluctant to introduce another language
> > > > into TC without a strong reason.
> > > > >
> > > > > Once the reverse_schema option is removed, what would be the main
> > > > purposes of the script?
> > > > >
> > > > > Looks like this is something that could be easily converted to bash
> > > > without need for another dependency.
> > > > >
> > > > > —Eric
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com>
> > wrote:
> > > > > >
> > > > > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> > > > since we
> > > > > > seem to be gaining a lot of Python expertise recently.
> > > > > >
> > > > > > -1 on 2.x compatibility -- writing something new with compatibility
> > > > for 2
> > > > > > major versions makes no sense to me. It limits the features and
> > > > libraries
> > > > > > that can be used and potentially doubles the amount of testing
> > > > required.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org>
> > wrote:
> > > > > >
> > > > > >> +1, seems reasonable. I don’t really have an opinion on python
> > 2.x
> > > > > >> compatibility, but whatever makes the most sense for the amount of
> > > > work is
> > > > > >> what I would prefer.
> > > > > >>
> > > > > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> > > > Jonathan_Gray@comcast.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> +1 There is already precedence in the repo for python for other
> > > > purposes.
> > > > > >>> The only caveat I would include is to be sure you include
> > backward
> > > > > >>> compatibility for python 2.x for the next year or so until it
> > goes
> > > > EOL.
> > > > > >>>
> > > > > >>> Jonathan G
> > > > > >>>
> > > > > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> > > > wrote:
> > > > > >>>
> > > > > >>> Hey TC devs,
> > > > > >>>
> > > > > >>> As we eliminate the need for Traffic Ops Perl, it will no
> > longer
> > > > > >>> really make sense for the db/admin.pl script to be Perl as
> > it is
> > > > > >>> today. This is because it depends on the Perl modules that are
> > > > > >>> installed via Carton for running Traffic Ops Perl. However,
> > it's
> > > > > >>> mostly just a Perl wrapper around shell commands except for a
> > YAML
> > > > > >>> Perl module and the `reverse_schema` command which uses the
> > DBIx
> > > > Perl
> > > > > >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> > > > you've
> > > > > >>> added a new DB table/column and need to get the ORM files
> > updated
> > > > to
> > > > > >>> use it).
> > > > > >>>
> > > > > >>> Without TO-Perl, there's no need for the `reverse_schema`
> > command
> > > > in
> > > > > >>> db/admin.pl and its dependency on the Perl DBIx module. At
> > that
> > > > > >> point
> > > > > >>> db/admin.pl is just a Perl script that parses some YAML and
> > > > shells
> > > > > >> out
> > > > > >>> commands.
> > > > > >>>
> > > > > >>> Part of the problem with TO-Perl is that there are a bunch of
> > > > random
> > > > > >>> non-API Perl scripts that basically assume all the modules in
> > the
> > > > > >>> cpanfile are installed in the environment, even though a lot
> > of
> > > > those
> > > > > >>> dependencies are probably only required for the Perl TO API or
> > > > UI. So
> > > > > >>> unless we go through all those Perl dependencies to eliminate
> > > > > >>> everything we don't really need once the Perl TO API and UI
> > are
> > > > > >>> completely removed, we'll continue to have a handful of Perl
> > > > scripts
> > > > > >>> like db/admin.pl that still require downloading and
> > installing
> > > > the
> > > > > >>> full set of TO Perl dependencies. On fresh installs, running
> > > > Carton
> > > > > >> to
> > > > > >>> install these dependencies can take nearly half an hour.
> > > > > >>>
> > > > > >>> I'm working on adding tests for the DB upgrade/downgrade
> > process
> > > > > >> which
> > > > > >>> I'd like to run automatically on PR submissions, but it really
> > > > only
> > > > > >>> needs the db/admin.pl script out of TO which assumes the
> > entire
> > > > set
> > > > > >> of
> > > > > >>> Perl TO dependencies even though it mostly just shells out to
> > > > `goose
> > > > > >>> up` and `goose down`. I'd like the test to emulate an actual
> > TO
> > > > > >>> environment as closely as possible to match what would
> > actually
> > > > > >> happen
> > > > > >>> in a production DB upgrade/downgrade. Right now I'm reusing
> > the
> > > > > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally
> > I'd
> > > > like
> > > > > >>> to port db/admin.pl into Python to give it its own set of
> > > > > >> dependencies
> > > > > >>> (just a YAML package) and not require the full set of TO Perl
> > > > > >>> dependencies. Then I can spin up a much lighter-weight
> > container
> > > > with
> > > > > >>> just the TO Python packages rather than setting up a separate
> > > > > >> cpanfile
> > > > > >>> with just the Perl packages needed for db/admin.pl.
> > > > > >>>
> > > > > >>> +1/-1 for adding Python as a dependency of Traffic Ops for
> > porting
> > > > > >>> scripts like db/admin.pl?
> > > > > >>>
> > > > > >>> -Rawlin
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> >
>
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
That project is just a thin wrapper atop github.com/spf13/viper which has a lot more nuance and power to trip up on if you're not careful.
Jonathan G
On 11/14/18, 2:36 PM, "Dan Kirkwood" <da...@gmail.com> wrote:
if yaml parsing is the only barrier for bash, try this:
https://github.com/030/go-yq
On Wed, Nov 14, 2018 at 2:18 PM Fieck, Brennan <Br...@comcast.com>
wrote:
> Specifically 2.7 and 3.6? We'd want it to run on 3.4, ideally, but 3.5 can
> work. Python 2 is going away, and there's no RHEL repo for anything over
> 3.5 afaik.
> ________________________________________
> From: Chris Lemmons <al...@gmail.com>
> Sent: Wednesday, November 14, 2018 1:41 PM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] The future of the db/admin.pl script
>
> Yeah, I looked at yq. For one... it's written in Python. :) But it
> does appear to be compatible with 2.7 and 3.6, so it may be easier to
> run.
> On Wed, Nov 14, 2018 at 1:35 PM Dewayne Richardson <de...@apache.org>
> wrote:
> >
> > No strong opinion here, the admin.pl was built for a different time in
> the
> > history of TC. The reason we used Perl was for config parsing back in
> the
> > day, but there is a yaml version of jq, called yq. I honestly agree with
> > bash if we can get away with it.
> >
> > FYI:
> > https://github.com/kislyuk/yq
> >
> > -Dew
> >
> >
> > On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <ra...@gmail.com>
> > wrote:
> >
> > > Eric,
> > >
> > > I share your sentiment about being reluctant to introduce another
> > > language as a dependency for Traffic Ops, but I wasn't able to find a
> > > really good, easily-available utility for parsing yaml (a la `jq` for
> > > json parsing) in a Bash script. Since the goose config is in yaml,
> > > `db/admin.pl` uses a yaml package to parse the goose config into
> > > variables which are then passed to the external `psql` et al.
> > > commands. It is possible to parse yaml using sed, but the example I
> > > found for doing that seemed really sketchy and fragile. So I figured
> > > using a solid YAML-parsing library like PyYAML in Python would be a
> > > safer bet while still allowing the use of a fully-featured programming
> > > language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> > > would also allow us to potentially use a DB library to interface with
> > > the DB directly in Python rather than requiring `psql` et al. and just
> > > shelling out to those external commands (although I plan to continue
> > > doing it that way for now).
> > >
> > > As a side note, it also paves the way for moving other scripts to
> > > Python like WebDep.pm, which uses a Perl package that is virtually
> > > impossible to install/get running on Mac because of Perl's broken SSL
> > > on Mac, which would make it much easier to start as a new developer on
> > > the project. I remember when I started working on Traffic Control, I
> > > had to copy someone else's Perl `traffic_ops/app/local` directory who
> > > had been on the project a long time and had actually gotten it to
> > > build on Mac before it became unusable. Eliminating issues like that
> > > by using a more popular and supportable language is a win in my book,
> > > but right now I'm just focusing on `db/admin.pl` to allow for better
> > > testability of the DB migration operations.
> > >
> > > - Rawlin
> > >
> > > On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> > > UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> > > <ef...@cisco.com.invalid> wrote:
> > > >
> > > > I’m only slightly familiar with all the different options for db/
> > > admin.pl.
> > > >
> > > > I’m a big fan of Python, but reluctant to introduce another language
> > > into TC without a strong reason.
> > > >
> > > > Once the reverse_schema option is removed, what would be the main
> > > purposes of the script?
> > > >
> > > > Looks like this is something that could be easily converted to bash
> > > without need for another dependency.
> > > >
> > > > —Eric
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com>
> wrote:
> > > > >
> > > > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> > > since we
> > > > > seem to be gaining a lot of Python expertise recently.
> > > > >
> > > > > -1 on 2.x compatibility -- writing something new with compatibility
> > > for 2
> > > > > major versions makes no sense to me. It limits the features and
> > > libraries
> > > > > that can be used and potentially doubles the amount of testing
> > > required.
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org>
> wrote:
> > > > >
> > > > >> +1, seems reasonable. I don’t really have an opinion on python
> 2.x
> > > > >> compatibility, but whatever makes the most sense for the amount of
> > > work is
> > > > >> what I would prefer.
> > > > >>
> > > > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> > > Jonathan_Gray@comcast.com>
> > > > >> wrote:
> > > > >>
> > > > >>> +1 There is already precedence in the repo for python for other
> > > purposes.
> > > > >>> The only caveat I would include is to be sure you include
> backward
> > > > >>> compatibility for python 2.x for the next year or so until it
> goes
> > > EOL.
> > > > >>>
> > > > >>> Jonathan G
> > > > >>>
> > > > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> > > wrote:
> > > > >>>
> > > > >>> Hey TC devs,
> > > > >>>
> > > > >>> As we eliminate the need for Traffic Ops Perl, it will no
> longer
> > > > >>> really make sense for the db/admin.pl script to be Perl as
> it is
> > > > >>> today. This is because it depends on the Perl modules that are
> > > > >>> installed via Carton for running Traffic Ops Perl. However,
> it's
> > > > >>> mostly just a Perl wrapper around shell commands except for a
> YAML
> > > > >>> Perl module and the `reverse_schema` command which uses the
> DBIx
> > > Perl
> > > > >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> > > you've
> > > > >>> added a new DB table/column and need to get the ORM files
> updated
> > > to
> > > > >>> use it).
> > > > >>>
> > > > >>> Without TO-Perl, there's no need for the `reverse_schema`
> command
> > > in
> > > > >>> db/admin.pl and its dependency on the Perl DBIx module. At
> that
> > > > >> point
> > > > >>> db/admin.pl is just a Perl script that parses some YAML and
> > > shells
> > > > >> out
> > > > >>> commands.
> > > > >>>
> > > > >>> Part of the problem with TO-Perl is that there are a bunch of
> > > random
> > > > >>> non-API Perl scripts that basically assume all the modules in
> the
> > > > >>> cpanfile are installed in the environment, even though a lot
> of
> > > those
> > > > >>> dependencies are probably only required for the Perl TO API or
> > > UI. So
> > > > >>> unless we go through all those Perl dependencies to eliminate
> > > > >>> everything we don't really need once the Perl TO API and UI
> are
> > > > >>> completely removed, we'll continue to have a handful of Perl
> > > scripts
> > > > >>> like db/admin.pl that still require downloading and
> installing
> > > the
> > > > >>> full set of TO Perl dependencies. On fresh installs, running
> > > Carton
> > > > >> to
> > > > >>> install these dependencies can take nearly half an hour.
> > > > >>>
> > > > >>> I'm working on adding tests for the DB upgrade/downgrade
> process
> > > > >> which
> > > > >>> I'd like to run automatically on PR submissions, but it really
> > > only
> > > > >>> needs the db/admin.pl script out of TO which assumes the
> entire
> > > set
> > > > >> of
> > > > >>> Perl TO dependencies even though it mostly just shells out to
> > > `goose
> > > > >>> up` and `goose down`. I'd like the test to emulate an actual
> TO
> > > > >>> environment as closely as possible to match what would
> actually
> > > > >> happen
> > > > >>> in a production DB upgrade/downgrade. Right now I'm reusing
> the
> > > > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally
> I'd
> > > like
> > > > >>> to port db/admin.pl into Python to give it its own set of
> > > > >> dependencies
> > > > >>> (just a YAML package) and not require the full set of TO Perl
> > > > >>> dependencies. Then I can spin up a much lighter-weight
> container
> > > with
> > > > >>> just the TO Python packages rather than setting up a separate
> > > > >> cpanfile
> > > > >>> with just the Perl packages needed for db/admin.pl.
> > > > >>>
> > > > >>> +1/-1 for adding Python as a dependency of Traffic Ops for
> porting
> > > > >>> scripts like db/admin.pl?
> > > > >>>
> > > > >>> -Rawlin
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Dan Kirkwood <da...@gmail.com>.
if yaml parsing is the only barrier for bash, try this:
https://github.com/030/go-yq
On Wed, Nov 14, 2018 at 2:18 PM Fieck, Brennan <Br...@comcast.com>
wrote:
> Specifically 2.7 and 3.6? We'd want it to run on 3.4, ideally, but 3.5 can
> work. Python 2 is going away, and there's no RHEL repo for anything over
> 3.5 afaik.
> ________________________________________
> From: Chris Lemmons <al...@gmail.com>
> Sent: Wednesday, November 14, 2018 1:41 PM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] The future of the db/admin.pl script
>
> Yeah, I looked at yq. For one... it's written in Python. :) But it
> does appear to be compatible with 2.7 and 3.6, so it may be easier to
> run.
> On Wed, Nov 14, 2018 at 1:35 PM Dewayne Richardson <de...@apache.org>
> wrote:
> >
> > No strong opinion here, the admin.pl was built for a different time in
> the
> > history of TC. The reason we used Perl was for config parsing back in
> the
> > day, but there is a yaml version of jq, called yq. I honestly agree with
> > bash if we can get away with it.
> >
> > FYI:
> > https://github.com/kislyuk/yq
> >
> > -Dew
> >
> >
> > On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <ra...@gmail.com>
> > wrote:
> >
> > > Eric,
> > >
> > > I share your sentiment about being reluctant to introduce another
> > > language as a dependency for Traffic Ops, but I wasn't able to find a
> > > really good, easily-available utility for parsing yaml (a la `jq` for
> > > json parsing) in a Bash script. Since the goose config is in yaml,
> > > `db/admin.pl` uses a yaml package to parse the goose config into
> > > variables which are then passed to the external `psql` et al.
> > > commands. It is possible to parse yaml using sed, but the example I
> > > found for doing that seemed really sketchy and fragile. So I figured
> > > using a solid YAML-parsing library like PyYAML in Python would be a
> > > safer bet while still allowing the use of a fully-featured programming
> > > language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> > > would also allow us to potentially use a DB library to interface with
> > > the DB directly in Python rather than requiring `psql` et al. and just
> > > shelling out to those external commands (although I plan to continue
> > > doing it that way for now).
> > >
> > > As a side note, it also paves the way for moving other scripts to
> > > Python like WebDep.pm, which uses a Perl package that is virtually
> > > impossible to install/get running on Mac because of Perl's broken SSL
> > > on Mac, which would make it much easier to start as a new developer on
> > > the project. I remember when I started working on Traffic Control, I
> > > had to copy someone else's Perl `traffic_ops/app/local` directory who
> > > had been on the project a long time and had actually gotten it to
> > > build on Mac before it became unusable. Eliminating issues like that
> > > by using a more popular and supportable language is a win in my book,
> > > but right now I'm just focusing on `db/admin.pl` to allow for better
> > > testability of the DB migration operations.
> > >
> > > - Rawlin
> > >
> > > On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> > > UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> > > <ef...@cisco.com.invalid> wrote:
> > > >
> > > > I’m only slightly familiar with all the different options for db/
> > > admin.pl.
> > > >
> > > > I’m a big fan of Python, but reluctant to introduce another language
> > > into TC without a strong reason.
> > > >
> > > > Once the reverse_schema option is removed, what would be the main
> > > purposes of the script?
> > > >
> > > > Looks like this is something that could be easily converted to bash
> > > without need for another dependency.
> > > >
> > > > —Eric
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com>
> wrote:
> > > > >
> > > > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> > > since we
> > > > > seem to be gaining a lot of Python expertise recently.
> > > > >
> > > > > -1 on 2.x compatibility -- writing something new with compatibility
> > > for 2
> > > > > major versions makes no sense to me. It limits the features and
> > > libraries
> > > > > that can be used and potentially doubles the amount of testing
> > > required.
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org>
> wrote:
> > > > >
> > > > >> +1, seems reasonable. I don’t really have an opinion on python
> 2.x
> > > > >> compatibility, but whatever makes the most sense for the amount of
> > > work is
> > > > >> what I would prefer.
> > > > >>
> > > > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> > > Jonathan_Gray@comcast.com>
> > > > >> wrote:
> > > > >>
> > > > >>> +1 There is already precedence in the repo for python for other
> > > purposes.
> > > > >>> The only caveat I would include is to be sure you include
> backward
> > > > >>> compatibility for python 2.x for the next year or so until it
> goes
> > > EOL.
> > > > >>>
> > > > >>> Jonathan G
> > > > >>>
> > > > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> > > wrote:
> > > > >>>
> > > > >>> Hey TC devs,
> > > > >>>
> > > > >>> As we eliminate the need for Traffic Ops Perl, it will no
> longer
> > > > >>> really make sense for the db/admin.pl script to be Perl as
> it is
> > > > >>> today. This is because it depends on the Perl modules that are
> > > > >>> installed via Carton for running Traffic Ops Perl. However,
> it's
> > > > >>> mostly just a Perl wrapper around shell commands except for a
> YAML
> > > > >>> Perl module and the `reverse_schema` command which uses the
> DBIx
> > > Perl
> > > > >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> > > you've
> > > > >>> added a new DB table/column and need to get the ORM files
> updated
> > > to
> > > > >>> use it).
> > > > >>>
> > > > >>> Without TO-Perl, there's no need for the `reverse_schema`
> command
> > > in
> > > > >>> db/admin.pl and its dependency on the Perl DBIx module. At
> that
> > > > >> point
> > > > >>> db/admin.pl is just a Perl script that parses some YAML and
> > > shells
> > > > >> out
> > > > >>> commands.
> > > > >>>
> > > > >>> Part of the problem with TO-Perl is that there are a bunch of
> > > random
> > > > >>> non-API Perl scripts that basically assume all the modules in
> the
> > > > >>> cpanfile are installed in the environment, even though a lot
> of
> > > those
> > > > >>> dependencies are probably only required for the Perl TO API or
> > > UI. So
> > > > >>> unless we go through all those Perl dependencies to eliminate
> > > > >>> everything we don't really need once the Perl TO API and UI
> are
> > > > >>> completely removed, we'll continue to have a handful of Perl
> > > scripts
> > > > >>> like db/admin.pl that still require downloading and
> installing
> > > the
> > > > >>> full set of TO Perl dependencies. On fresh installs, running
> > > Carton
> > > > >> to
> > > > >>> install these dependencies can take nearly half an hour.
> > > > >>>
> > > > >>> I'm working on adding tests for the DB upgrade/downgrade
> process
> > > > >> which
> > > > >>> I'd like to run automatically on PR submissions, but it really
> > > only
> > > > >>> needs the db/admin.pl script out of TO which assumes the
> entire
> > > set
> > > > >> of
> > > > >>> Perl TO dependencies even though it mostly just shells out to
> > > `goose
> > > > >>> up` and `goose down`. I'd like the test to emulate an actual
> TO
> > > > >>> environment as closely as possible to match what would
> actually
> > > > >> happen
> > > > >>> in a production DB upgrade/downgrade. Right now I'm reusing
> the
> > > > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally
> I'd
> > > like
> > > > >>> to port db/admin.pl into Python to give it its own set of
> > > > >> dependencies
> > > > >>> (just a YAML package) and not require the full set of TO Perl
> > > > >>> dependencies. Then I can spin up a much lighter-weight
> container
> > > with
> > > > >>> just the TO Python packages rather than setting up a separate
> > > > >> cpanfile
> > > > >>> with just the Perl packages needed for db/admin.pl.
> > > > >>>
> > > > >>> +1/-1 for adding Python as a dependency of Traffic Ops for
> porting
> > > > >>> scripts like db/admin.pl?
> > > > >>>
> > > > >>> -Rawlin
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > >
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Fieck, Brennan" <Br...@comcast.com>.
Specifically 2.7 and 3.6? We'd want it to run on 3.4, ideally, but 3.5 can work. Python 2 is going away, and there's no RHEL repo for anything over 3.5 afaik.
________________________________________
From: Chris Lemmons <al...@gmail.com>
Sent: Wednesday, November 14, 2018 1:41 PM
To: dev@trafficcontrol.apache.org
Subject: Re: [EXTERNAL] The future of the db/admin.pl script
Yeah, I looked at yq. For one... it's written in Python. :) But it
does appear to be compatible with 2.7 and 3.6, so it may be easier to
run.
On Wed, Nov 14, 2018 at 1:35 PM Dewayne Richardson <de...@apache.org> wrote:
>
> No strong opinion here, the admin.pl was built for a different time in the
> history of TC. The reason we used Perl was for config parsing back in the
> day, but there is a yaml version of jq, called yq. I honestly agree with
> bash if we can get away with it.
>
> FYI:
> https://github.com/kislyuk/yq
>
> -Dew
>
>
> On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <ra...@gmail.com>
> wrote:
>
> > Eric,
> >
> > I share your sentiment about being reluctant to introduce another
> > language as a dependency for Traffic Ops, but I wasn't able to find a
> > really good, easily-available utility for parsing yaml (a la `jq` for
> > json parsing) in a Bash script. Since the goose config is in yaml,
> > `db/admin.pl` uses a yaml package to parse the goose config into
> > variables which are then passed to the external `psql` et al.
> > commands. It is possible to parse yaml using sed, but the example I
> > found for doing that seemed really sketchy and fragile. So I figured
> > using a solid YAML-parsing library like PyYAML in Python would be a
> > safer bet while still allowing the use of a fully-featured programming
> > language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> > would also allow us to potentially use a DB library to interface with
> > the DB directly in Python rather than requiring `psql` et al. and just
> > shelling out to those external commands (although I plan to continue
> > doing it that way for now).
> >
> > As a side note, it also paves the way for moving other scripts to
> > Python like WebDep.pm, which uses a Perl package that is virtually
> > impossible to install/get running on Mac because of Perl's broken SSL
> > on Mac, which would make it much easier to start as a new developer on
> > the project. I remember when I started working on Traffic Control, I
> > had to copy someone else's Perl `traffic_ops/app/local` directory who
> > had been on the project a long time and had actually gotten it to
> > build on Mac before it became unusable. Eliminating issues like that
> > by using a more popular and supportable language is a win in my book,
> > but right now I'm just focusing on `db/admin.pl` to allow for better
> > testability of the DB migration operations.
> >
> > - Rawlin
> >
> > On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> > UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> > <ef...@cisco.com.invalid> wrote:
> > >
> > > I’m only slightly familiar with all the different options for db/
> > admin.pl.
> > >
> > > I’m a big fan of Python, but reluctant to introduce another language
> > into TC without a strong reason.
> > >
> > > Once the reverse_schema option is removed, what would be the main
> > purposes of the script?
> > >
> > > Looks like this is something that could be easily converted to bash
> > without need for another dependency.
> > >
> > > —Eric
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
> > > >
> > > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> > since we
> > > > seem to be gaining a lot of Python expertise recently.
> > > >
> > > > -1 on 2.x compatibility -- writing something new with compatibility
> > for 2
> > > > major versions makes no sense to me. It limits the features and
> > libraries
> > > > that can be used and potentially doubles the amount of testing
> > required.
> > > >
> > > >
> > > >
> > > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
> > > >
> > > >> +1, seems reasonable. I don’t really have an opinion on python 2.x
> > > >> compatibility, but whatever makes the most sense for the amount of
> > work is
> > > >> what I would prefer.
> > > >>
> > > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> > Jonathan_Gray@comcast.com>
> > > >> wrote:
> > > >>
> > > >>> +1 There is already precedence in the repo for python for other
> > purposes.
> > > >>> The only caveat I would include is to be sure you include backward
> > > >>> compatibility for python 2.x for the next year or so until it goes
> > EOL.
> > > >>>
> > > >>> Jonathan G
> > > >>>
> > > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> > wrote:
> > > >>>
> > > >>> Hey TC devs,
> > > >>>
> > > >>> As we eliminate the need for Traffic Ops Perl, it will no longer
> > > >>> really make sense for the db/admin.pl script to be Perl as it is
> > > >>> today. This is because it depends on the Perl modules that are
> > > >>> installed via Carton for running Traffic Ops Perl. However, it's
> > > >>> mostly just a Perl wrapper around shell commands except for a YAML
> > > >>> Perl module and the `reverse_schema` command which uses the DBIx
> > Perl
> > > >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> > you've
> > > >>> added a new DB table/column and need to get the ORM files updated
> > to
> > > >>> use it).
> > > >>>
> > > >>> Without TO-Perl, there's no need for the `reverse_schema` command
> > in
> > > >>> db/admin.pl and its dependency on the Perl DBIx module. At that
> > > >> point
> > > >>> db/admin.pl is just a Perl script that parses some YAML and
> > shells
> > > >> out
> > > >>> commands.
> > > >>>
> > > >>> Part of the problem with TO-Perl is that there are a bunch of
> > random
> > > >>> non-API Perl scripts that basically assume all the modules in the
> > > >>> cpanfile are installed in the environment, even though a lot of
> > those
> > > >>> dependencies are probably only required for the Perl TO API or
> > UI. So
> > > >>> unless we go through all those Perl dependencies to eliminate
> > > >>> everything we don't really need once the Perl TO API and UI are
> > > >>> completely removed, we'll continue to have a handful of Perl
> > scripts
> > > >>> like db/admin.pl that still require downloading and installing
> > the
> > > >>> full set of TO Perl dependencies. On fresh installs, running
> > Carton
> > > >> to
> > > >>> install these dependencies can take nearly half an hour.
> > > >>>
> > > >>> I'm working on adding tests for the DB upgrade/downgrade process
> > > >> which
> > > >>> I'd like to run automatically on PR submissions, but it really
> > only
> > > >>> needs the db/admin.pl script out of TO which assumes the entire
> > set
> > > >> of
> > > >>> Perl TO dependencies even though it mostly just shells out to
> > `goose
> > > >>> up` and `goose down`. I'd like the test to emulate an actual TO
> > > >>> environment as closely as possible to match what would actually
> > > >> happen
> > > >>> in a production DB upgrade/downgrade. Right now I'm reusing the
> > > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd
> > like
> > > >>> to port db/admin.pl into Python to give it its own set of
> > > >> dependencies
> > > >>> (just a YAML package) and not require the full set of TO Perl
> > > >>> dependencies. Then I can spin up a much lighter-weight container
> > with
> > > >>> just the TO Python packages rather than setting up a separate
> > > >> cpanfile
> > > >>> with just the Perl packages needed for db/admin.pl.
> > > >>>
> > > >>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
> > > >>> scripts like db/admin.pl?
> > > >>>
> > > >>> -Rawlin
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> >
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Chris Lemmons <al...@gmail.com>.
Yeah, I looked at yq. For one... it's written in Python. :) But it
does appear to be compatible with 2.7 and 3.6, so it may be easier to
run.
On Wed, Nov 14, 2018 at 1:35 PM Dewayne Richardson <de...@apache.org> wrote:
>
> No strong opinion here, the admin.pl was built for a different time in the
> history of TC. The reason we used Perl was for config parsing back in the
> day, but there is a yaml version of jq, called yq. I honestly agree with
> bash if we can get away with it.
>
> FYI:
> https://github.com/kislyuk/yq
>
> -Dew
>
>
> On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <ra...@gmail.com>
> wrote:
>
> > Eric,
> >
> > I share your sentiment about being reluctant to introduce another
> > language as a dependency for Traffic Ops, but I wasn't able to find a
> > really good, easily-available utility for parsing yaml (a la `jq` for
> > json parsing) in a Bash script. Since the goose config is in yaml,
> > `db/admin.pl` uses a yaml package to parse the goose config into
> > variables which are then passed to the external `psql` et al.
> > commands. It is possible to parse yaml using sed, but the example I
> > found for doing that seemed really sketchy and fragile. So I figured
> > using a solid YAML-parsing library like PyYAML in Python would be a
> > safer bet while still allowing the use of a fully-featured programming
> > language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> > would also allow us to potentially use a DB library to interface with
> > the DB directly in Python rather than requiring `psql` et al. and just
> > shelling out to those external commands (although I plan to continue
> > doing it that way for now).
> >
> > As a side note, it also paves the way for moving other scripts to
> > Python like WebDep.pm, which uses a Perl package that is virtually
> > impossible to install/get running on Mac because of Perl's broken SSL
> > on Mac, which would make it much easier to start as a new developer on
> > the project. I remember when I started working on Traffic Control, I
> > had to copy someone else's Perl `traffic_ops/app/local` directory who
> > had been on the project a long time and had actually gotten it to
> > build on Mac before it became unusable. Eliminating issues like that
> > by using a more popular and supportable language is a win in my book,
> > but right now I'm just focusing on `db/admin.pl` to allow for better
> > testability of the DB migration operations.
> >
> > - Rawlin
> >
> > On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> > UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> > <ef...@cisco.com.invalid> wrote:
> > >
> > > I’m only slightly familiar with all the different options for db/
> > admin.pl.
> > >
> > > I’m a big fan of Python, but reluctant to introduce another language
> > into TC without a strong reason.
> > >
> > > Once the reverse_schema option is removed, what would be the main
> > purposes of the script?
> > >
> > > Looks like this is something that could be easily converted to bash
> > without need for another dependency.
> > >
> > > —Eric
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
> > > >
> > > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> > since we
> > > > seem to be gaining a lot of Python expertise recently.
> > > >
> > > > -1 on 2.x compatibility -- writing something new with compatibility
> > for 2
> > > > major versions makes no sense to me. It limits the features and
> > libraries
> > > > that can be used and potentially doubles the amount of testing
> > required.
> > > >
> > > >
> > > >
> > > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
> > > >
> > > >> +1, seems reasonable. I don’t really have an opinion on python 2.x
> > > >> compatibility, but whatever makes the most sense for the amount of
> > work is
> > > >> what I would prefer.
> > > >>
> > > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> > Jonathan_Gray@comcast.com>
> > > >> wrote:
> > > >>
> > > >>> +1 There is already precedence in the repo for python for other
> > purposes.
> > > >>> The only caveat I would include is to be sure you include backward
> > > >>> compatibility for python 2.x for the next year or so until it goes
> > EOL.
> > > >>>
> > > >>> Jonathan G
> > > >>>
> > > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> > wrote:
> > > >>>
> > > >>> Hey TC devs,
> > > >>>
> > > >>> As we eliminate the need for Traffic Ops Perl, it will no longer
> > > >>> really make sense for the db/admin.pl script to be Perl as it is
> > > >>> today. This is because it depends on the Perl modules that are
> > > >>> installed via Carton for running Traffic Ops Perl. However, it's
> > > >>> mostly just a Perl wrapper around shell commands except for a YAML
> > > >>> Perl module and the `reverse_schema` command which uses the DBIx
> > Perl
> > > >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> > you've
> > > >>> added a new DB table/column and need to get the ORM files updated
> > to
> > > >>> use it).
> > > >>>
> > > >>> Without TO-Perl, there's no need for the `reverse_schema` command
> > in
> > > >>> db/admin.pl and its dependency on the Perl DBIx module. At that
> > > >> point
> > > >>> db/admin.pl is just a Perl script that parses some YAML and
> > shells
> > > >> out
> > > >>> commands.
> > > >>>
> > > >>> Part of the problem with TO-Perl is that there are a bunch of
> > random
> > > >>> non-API Perl scripts that basically assume all the modules in the
> > > >>> cpanfile are installed in the environment, even though a lot of
> > those
> > > >>> dependencies are probably only required for the Perl TO API or
> > UI. So
> > > >>> unless we go through all those Perl dependencies to eliminate
> > > >>> everything we don't really need once the Perl TO API and UI are
> > > >>> completely removed, we'll continue to have a handful of Perl
> > scripts
> > > >>> like db/admin.pl that still require downloading and installing
> > the
> > > >>> full set of TO Perl dependencies. On fresh installs, running
> > Carton
> > > >> to
> > > >>> install these dependencies can take nearly half an hour.
> > > >>>
> > > >>> I'm working on adding tests for the DB upgrade/downgrade process
> > > >> which
> > > >>> I'd like to run automatically on PR submissions, but it really
> > only
> > > >>> needs the db/admin.pl script out of TO which assumes the entire
> > set
> > > >> of
> > > >>> Perl TO dependencies even though it mostly just shells out to
> > `goose
> > > >>> up` and `goose down`. I'd like the test to emulate an actual TO
> > > >>> environment as closely as possible to match what would actually
> > > >> happen
> > > >>> in a production DB upgrade/downgrade. Right now I'm reusing the
> > > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd
> > like
> > > >>> to port db/admin.pl into Python to give it its own set of
> > > >> dependencies
> > > >>> (just a YAML package) and not require the full set of TO Perl
> > > >>> dependencies. Then I can spin up a much lighter-weight container
> > with
> > > >>> just the TO Python packages rather than setting up a separate
> > > >> cpanfile
> > > >>> with just the Perl packages needed for db/admin.pl.
> > > >>>
> > > >>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
> > > >>> scripts like db/admin.pl?
> > > >>>
> > > >>> -Rawlin
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > >
> >
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Dewayne Richardson <de...@apache.org>.
No strong opinion here, the admin.pl was built for a different time in the
history of TC. The reason we used Perl was for config parsing back in the
day, but there is a yaml version of jq, called yq. I honestly agree with
bash if we can get away with it.
FYI:
https://github.com/kislyuk/yq
-Dew
On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <ra...@gmail.com>
wrote:
> Eric,
>
> I share your sentiment about being reluctant to introduce another
> language as a dependency for Traffic Ops, but I wasn't able to find a
> really good, easily-available utility for parsing yaml (a la `jq` for
> json parsing) in a Bash script. Since the goose config is in yaml,
> `db/admin.pl` uses a yaml package to parse the goose config into
> variables which are then passed to the external `psql` et al.
> commands. It is possible to parse yaml using sed, but the example I
> found for doing that seemed really sketchy and fragile. So I figured
> using a solid YAML-parsing library like PyYAML in Python would be a
> safer bet while still allowing the use of a fully-featured programming
> language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> would also allow us to potentially use a DB library to interface with
> the DB directly in Python rather than requiring `psql` et al. and just
> shelling out to those external commands (although I plan to continue
> doing it that way for now).
>
> As a side note, it also paves the way for moving other scripts to
> Python like WebDep.pm, which uses a Perl package that is virtually
> impossible to install/get running on Mac because of Perl's broken SSL
> on Mac, which would make it much easier to start as a new developer on
> the project. I remember when I started working on Traffic Control, I
> had to copy someone else's Perl `traffic_ops/app/local` directory who
> had been on the project a long time and had actually gotten it to
> build on Mac before it became unusable. Eliminating issues like that
> by using a more popular and supportable language is a win in my book,
> but right now I'm just focusing on `db/admin.pl` to allow for better
> testability of the DB migration operations.
>
> - Rawlin
>
> On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> <ef...@cisco.com.invalid> wrote:
> >
> > I’m only slightly familiar with all the different options for db/
> admin.pl.
> >
> > I’m a big fan of Python, but reluctant to introduce another language
> into TC without a strong reason.
> >
> > Once the reverse_schema option is removed, what would be the main
> purposes of the script?
> >
> > Looks like this is something that could be easily converted to bash
> without need for another dependency.
> >
> > —Eric
> >
> >
> >
> >
> >
> >
> >
> > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
> > >
> > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> since we
> > > seem to be gaining a lot of Python expertise recently.
> > >
> > > -1 on 2.x compatibility -- writing something new with compatibility
> for 2
> > > major versions makes no sense to me. It limits the features and
> libraries
> > > that can be used and potentially doubles the amount of testing
> required.
> > >
> > >
> > >
> > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
> > >
> > >> +1, seems reasonable. I don’t really have an opinion on python 2.x
> > >> compatibility, but whatever makes the most sense for the amount of
> work is
> > >> what I would prefer.
> > >>
> > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> Jonathan_Gray@comcast.com>
> > >> wrote:
> > >>
> > >>> +1 There is already precedence in the repo for python for other
> purposes.
> > >>> The only caveat I would include is to be sure you include backward
> > >>> compatibility for python 2.x for the next year or so until it goes
> EOL.
> > >>>
> > >>> Jonathan G
> > >>>
> > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> wrote:
> > >>>
> > >>> Hey TC devs,
> > >>>
> > >>> As we eliminate the need for Traffic Ops Perl, it will no longer
> > >>> really make sense for the db/admin.pl script to be Perl as it is
> > >>> today. This is because it depends on the Perl modules that are
> > >>> installed via Carton for running Traffic Ops Perl. However, it's
> > >>> mostly just a Perl wrapper around shell commands except for a YAML
> > >>> Perl module and the `reverse_schema` command which uses the DBIx
> Perl
> > >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> you've
> > >>> added a new DB table/column and need to get the ORM files updated
> to
> > >>> use it).
> > >>>
> > >>> Without TO-Perl, there's no need for the `reverse_schema` command
> in
> > >>> db/admin.pl and its dependency on the Perl DBIx module. At that
> > >> point
> > >>> db/admin.pl is just a Perl script that parses some YAML and
> shells
> > >> out
> > >>> commands.
> > >>>
> > >>> Part of the problem with TO-Perl is that there are a bunch of
> random
> > >>> non-API Perl scripts that basically assume all the modules in the
> > >>> cpanfile are installed in the environment, even though a lot of
> those
> > >>> dependencies are probably only required for the Perl TO API or
> UI. So
> > >>> unless we go through all those Perl dependencies to eliminate
> > >>> everything we don't really need once the Perl TO API and UI are
> > >>> completely removed, we'll continue to have a handful of Perl
> scripts
> > >>> like db/admin.pl that still require downloading and installing
> the
> > >>> full set of TO Perl dependencies. On fresh installs, running
> Carton
> > >> to
> > >>> install these dependencies can take nearly half an hour.
> > >>>
> > >>> I'm working on adding tests for the DB upgrade/downgrade process
> > >> which
> > >>> I'd like to run automatically on PR submissions, but it really
> only
> > >>> needs the db/admin.pl script out of TO which assumes the entire
> set
> > >> of
> > >>> Perl TO dependencies even though it mostly just shells out to
> `goose
> > >>> up` and `goose down`. I'd like the test to emulate an actual TO
> > >>> environment as closely as possible to match what would actually
> > >> happen
> > >>> in a production DB upgrade/downgrade. Right now I'm reusing the
> > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd
> like
> > >>> to port db/admin.pl into Python to give it its own set of
> > >> dependencies
> > >>> (just a YAML package) and not require the full set of TO Perl
> > >>> dependencies. Then I can spin up a much lighter-weight container
> with
> > >>> just the TO Python packages rather than setting up a separate
> > >> cpanfile
> > >>> with just the Perl packages needed for db/admin.pl.
> > >>>
> > >>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
> > >>> scripts like db/admin.pl?
> > >>>
> > >>> -Rawlin
> > >>>
> > >>>
> > >>>
> > >>
> >
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Dan Kirkwood <da...@gmail.com>.
a side note.. webdep.pm is required to install the UI dependencies of
traffic_ops, but the UI is going away in favor of traffic_portal. So,
that one should be left alone.
-dan
On Mon, Nov 12, 2018 at 11:29 AM Rawlin Peters <ra...@gmail.com>
wrote:
> Eric,
>
> I share your sentiment about being reluctant to introduce another
> language as a dependency for Traffic Ops, but I wasn't able to find a
> really good, easily-available utility for parsing yaml (a la `jq` for
> json parsing) in a Bash script. Since the goose config is in yaml,
> `db/admin.pl` uses a yaml package to parse the goose config into
> variables which are then passed to the external `psql` et al.
> commands. It is possible to parse yaml using sed, but the example I
> found for doing that seemed really sketchy and fragile. So I figured
> using a solid YAML-parsing library like PyYAML in Python would be a
> safer bet while still allowing the use of a fully-featured programming
> language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> would also allow us to potentially use a DB library to interface with
> the DB directly in Python rather than requiring `psql` et al. and just
> shelling out to those external commands (although I plan to continue
> doing it that way for now).
>
> As a side note, it also paves the way for moving other scripts to
> Python like WebDep.pm, which uses a Perl package that is virtually
> impossible to install/get running on Mac because of Perl's broken SSL
> on Mac, which would make it much easier to start as a new developer on
> the project. I remember when I started working on Traffic Control, I
> had to copy someone else's Perl `traffic_ops/app/local` directory who
> had been on the project a long time and had actually gotten it to
> build on Mac before it became unusable. Eliminating issues like that
> by using a more popular and supportable language is a win in my book,
> but right now I'm just focusing on `db/admin.pl` to allow for better
> testability of the DB migration operations.
>
> - Rawlin
>
> On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> <ef...@cisco.com.invalid> wrote:
> >
> > I’m only slightly familiar with all the different options for db/
> admin.pl.
> >
> > I’m a big fan of Python, but reluctant to introduce another language
> into TC without a strong reason.
> >
> > Once the reverse_schema option is removed, what would be the main
> purposes of the script?
> >
> > Looks like this is something that could be easily converted to bash
> without need for another dependency.
> >
> > —Eric
> >
> >
> >
> >
> >
> >
> >
> > > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
> > >
> > > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> since we
> > > seem to be gaining a lot of Python expertise recently.
> > >
> > > -1 on 2.x compatibility -- writing something new with compatibility
> for 2
> > > major versions makes no sense to me. It limits the features and
> libraries
> > > that can be used and potentially doubles the amount of testing
> required.
> > >
> > >
> > >
> > > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
> > >
> > >> +1, seems reasonable. I don’t really have an opinion on python 2.x
> > >> compatibility, but whatever makes the most sense for the amount of
> work is
> > >> what I would prefer.
> > >>
> > >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> Jonathan_Gray@comcast.com>
> > >> wrote:
> > >>
> > >>> +1 There is already precedence in the repo for python for other
> purposes.
> > >>> The only caveat I would include is to be sure you include backward
> > >>> compatibility for python 2.x for the next year or so until it goes
> EOL.
> > >>>
> > >>> Jonathan G
> > >>>
> > >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> wrote:
> > >>>
> > >>> Hey TC devs,
> > >>>
> > >>> As we eliminate the need for Traffic Ops Perl, it will no longer
> > >>> really make sense for the db/admin.pl script to be Perl as it is
> > >>> today. This is because it depends on the Perl modules that are
> > >>> installed via Carton for running Traffic Ops Perl. However, it's
> > >>> mostly just a Perl wrapper around shell commands except for a YAML
> > >>> Perl module and the `reverse_schema` command which uses the DBIx
> Perl
> > >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> you've
> > >>> added a new DB table/column and need to get the ORM files updated
> to
> > >>> use it).
> > >>>
> > >>> Without TO-Perl, there's no need for the `reverse_schema` command
> in
> > >>> db/admin.pl and its dependency on the Perl DBIx module. At that
> > >> point
> > >>> db/admin.pl is just a Perl script that parses some YAML and
> shells
> > >> out
> > >>> commands.
> > >>>
> > >>> Part of the problem with TO-Perl is that there are a bunch of
> random
> > >>> non-API Perl scripts that basically assume all the modules in the
> > >>> cpanfile are installed in the environment, even though a lot of
> those
> > >>> dependencies are probably only required for the Perl TO API or
> UI. So
> > >>> unless we go through all those Perl dependencies to eliminate
> > >>> everything we don't really need once the Perl TO API and UI are
> > >>> completely removed, we'll continue to have a handful of Perl
> scripts
> > >>> like db/admin.pl that still require downloading and installing
> the
> > >>> full set of TO Perl dependencies. On fresh installs, running
> Carton
> > >> to
> > >>> install these dependencies can take nearly half an hour.
> > >>>
> > >>> I'm working on adding tests for the DB upgrade/downgrade process
> > >> which
> > >>> I'd like to run automatically on PR submissions, but it really
> only
> > >>> needs the db/admin.pl script out of TO which assumes the entire
> set
> > >> of
> > >>> Perl TO dependencies even though it mostly just shells out to
> `goose
> > >>> up` and `goose down`. I'd like the test to emulate an actual TO
> > >>> environment as closely as possible to match what would actually
> > >> happen
> > >>> in a production DB upgrade/downgrade. Right now I'm reusing the
> > >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd
> like
> > >>> to port db/admin.pl into Python to give it its own set of
> > >> dependencies
> > >>> (just a YAML package) and not require the full set of TO Perl
> > >>> dependencies. Then I can spin up a much lighter-weight container
> with
> > >>> just the TO Python packages rather than setting up a separate
> > >> cpanfile
> > >>> with just the Perl packages needed for db/admin.pl.
> > >>>
> > >>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
> > >>> scripts like db/admin.pl?
> > >>>
> > >>> -Rawlin
> > >>>
> > >>>
> > >>>
> > >>
> >
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus
(UK) Limited -OBO at Cisco)" <ef...@cisco.com.INVALID>.
Hey Rawlin-
Thanks for doing more digging on the licensing aspect. I think we could find a way to make this work. The Apache FAQs are fuzzy for stuff like the boot loader exception, but would probably be ok.
I dont think there is any issue including the python interpreter as PSF is an Apache friendly license.
I think Go sounds like a great idea for the db/admin.pl replacement script
—Eric
> On Nov 13, 2018, at 12:55 PM, Rawlin Peters <ra...@gmail.com> wrote:
>
> replies inline
>
> On Mon, Nov 12, 2018 at 4:59 PM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> <ef...@cisco.com.invalid> wrote:
>> Is pyyaml part of the batteries-included packages?
>> If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops RPM. (For those of us without permitted Internet access in our deployments, installing via pip means standing up our own private PyPi repo) at each of our customers- something I would really like to avoid.
>>
>> I don't believe it is part of the standard library, but I do believe
>> we can figure out a way to keep it all self-contained to the traffic
>> ops RPM without requiring internet access at install time. I think
>> https://github.com/pyinstaller/pyinstaller is more than capable of
>> doing that on Linux/Mac/Windows. I just tried it out on my Python
>> version of db/admin.pl on my mac, and it created a self-contained
>> binary of 4.7MB size that seems to work just fine. It does still
>> depend on the external commands like `psql` et al. from the postgres
>> package, but that's no different from the current Perl version. Do you
>> think that would be sufficient?
>>
>> I think pyinstaller also solves the python2 vs python3 availability
>> problem, since the interpreter is packaged into the resulting binary
>> itself.
>> EF> I think that would be OK. The licensing on that one is GPLv2 which is category X to include in source (we can distribute the boot loader though). Do you know of any alternatives with more compatible licensing?
>
> It wouldn't need to be included in the source; it would just be used
> at build time. But there is also this information on its website
> (http://www.pyinstaller.org/license.html):
>
>> PyInstaller is distributed under the GPL license (see the file COPYING.txt in the source code), with a special exception which allows to use PyInstaller to build and distribute non-free programs (including commercial ones). In other words, you have no restrictions in using PyInstaller as-is, but any kind of modifications to it will have to comply with the GPL license. See also our FAQ.
>
> According to https://www.apache.org/legal/resolved.html#prohibited we
> are ok to use a GPL'ed tool during the build but cannot include the
> GPL'ed source code in our distribution.
>
> There is also the question of actually including/distributing the
> python interpreter+standard library in our rpm (as part of the
> standalone binary produced by pyinstaller), but I believe that should
> be fine according to
> https://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq#Can_I_bundle_Python_with_my_non-open-source_application.3F.
> Here is is the relevant quote:
>
>> Can I bundle Python with my non-open-source application?
>> Yes. Unlike some open source licenses, the PSF License allows Python to be included in non-open applications, either in unmodified or modified form.
>
> That said, I'm starting to lean more towards Golang for replacing
> db/admin.pl, because we already use it heavily and it checks off a lot
> of the boxes. It might seem a bit like overkill for scripting stuff,
> but there is definitely growing usage of Go as a script-like
> executable (see
> and
> https://github.com/erning/gorun) for things you'd normally just write
> in Bash or some other scripting language. There's just a lot more
> involved in using Python for this (pyinstaller, pipenv, 2 vs 3, etc)
> than Go.
>
> - Rawlin
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Rawlin Peters <ra...@gmail.com>.
replies inline
On Mon, Nov 12, 2018 at 4:59 PM Eric Friedrich -X (efriedri - TRITON
UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
<ef...@cisco.com.invalid> wrote:
> Is pyyaml part of the batteries-included packages?
> If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops RPM. (For those of us without permitted Internet access in our deployments, installing via pip means standing up our own private PyPi repo) at each of our customers- something I would really like to avoid.
>
> I don't believe it is part of the standard library, but I do believe
> we can figure out a way to keep it all self-contained to the traffic
> ops RPM without requiring internet access at install time. I think
> https://github.com/pyinstaller/pyinstaller is more than capable of
> doing that on Linux/Mac/Windows. I just tried it out on my Python
> version of db/admin.pl on my mac, and it created a self-contained
> binary of 4.7MB size that seems to work just fine. It does still
> depend on the external commands like `psql` et al. from the postgres
> package, but that's no different from the current Perl version. Do you
> think that would be sufficient?
>
> I think pyinstaller also solves the python2 vs python3 availability
> problem, since the interpreter is packaged into the resulting binary
> itself.
> EF> I think that would be OK. The licensing on that one is GPLv2 which is category X to include in source (we can distribute the boot loader though). Do you know of any alternatives with more compatible licensing?
It wouldn't need to be included in the source; it would just be used
at build time. But there is also this information on its website
(http://www.pyinstaller.org/license.html):
> PyInstaller is distributed under the GPL license (see the file COPYING.txt in the source code), with a special exception which allows to use PyInstaller to build and distribute non-free programs (including commercial ones). In other words, you have no restrictions in using PyInstaller as-is, but any kind of modifications to it will have to comply with the GPL license. See also our FAQ.
According to https://www.apache.org/legal/resolved.html#prohibited we
are ok to use a GPL'ed tool during the build but cannot include the
GPL'ed source code in our distribution.
There is also the question of actually including/distributing the
python interpreter+standard library in our rpm (as part of the
standalone binary produced by pyinstaller), but I believe that should
be fine according to
https://wiki.python.org/moin/PythonSoftwareFoundationLicenseFaq#Can_I_bundle_Python_with_my_non-open-source_application.3F.
Here is is the relevant quote:
> Can I bundle Python with my non-open-source application?
> Yes. Unlike some open source licenses, the PSF License allows Python to be included in non-open applications, either in unmodified or modified form.
That said, I'm starting to lean more towards Golang for replacing
db/admin.pl, because we already use it heavily and it checks off a lot
of the boxes. It might seem a bit like overkill for scripting stuff,
but there is definitely growing usage of Go as a script-like
executable (see https://github.com/codeskyblue/go-sh and
https://github.com/erning/gorun) for things you'd normally just write
in Bash or some other scripting language. There's just a lot more
involved in using Python for this (pyinstaller, pipenv, 2 vs 3, etc)
than Go.
- Rawlin
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Fieck, Brennan" <Br...@comcast.com>.
> You either need Internet access to download python, to have python embedded in the build at build time, or to have it pre-installed on the system
Linux Standard Base (LSB) compliant Linux distributions _will_ have a Python interpreter. Unfortunately, the spec only specifies:
"The default installed Python version shall be 2.4.2 or greater." (Section 6.3 of the Runtime Languages specification - https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Languages/LSB-Languages.pdf)
I always like to point that out, but that being said, I agree that Python isn't really the best fit here. For as long as we keep using 'goose', I think Go is the way to go (pun intended).
> As a side note, it also paves the way for moving other scripts to
> Python like WebDep.pm,
That may be a bad example because, as was mentioned, it should be going away :P
I think that scripts that need to interface with Traffic Ops and have no really tight performance constraints are currently our best use-case for Python, since we already have a working, supported TO client in Python.
But for as long as CentOS (as our only officially supported distro) stubbornly clings to the rotting carcass of Python2, using it for much else - especially in the build process of major TC components - seems like a bad idea.
________________________________________
From: Chris Lemmons <al...@gmail.com>
Sent: Tuesday, November 13, 2018 2:16 PM
To: dev@trafficcontrol.apache.org
Subject: Re: [EXTERNAL] The future of the db/admin.pl script
Stepping back for a moment, I see a few goals:
- Install without requiring Internet access. Since local security
policies may (rightfully!) restrict upline resources, it would be best
if we can install without needing to download things.
- Not distribute CatX dependencies. This would be a bit of a
non-starter, I think.
- Use technologies well-understood by the team.
- Install quickly.
- Maintain the script easily.
- Minimize system dependencies.
When I analyze some of the suggestions, here's what I see.
Python: You either need Internet access to download python, to have
python embedded in the build at build time, or to have it
pre-installed on the system. The same goes for every dependent
library. The first option doesn't meet the goal of not requiring
Internet access. We can embed python if we wish without violating
licenses, since Python is licensed under the PSF, which is Category A.
However, while PyInstaller itself is GPL, which is CatX, it has a
bootstrapper exemption that applies to the files that it stores
directly as part of the installer-creator. Unfortunately, the
exemption does not allow modification of the files and contains what I
think is a Field of Use limitation which requires that they be used
only when linked into another executable. I suspect that this causes a
PyInstaller binary to be incompatible with the distribution mechanisms
of the Apache project. Perhaps there is another python single-binary
system we could consider here, though. Ultimately, though Python isn't
really designed as a compiled language and if we want to compile it,
we might want to choose a language designed for that.
Having python pre-installed would increase deployment challenges,
particularly since many system repositories don't have the modern
versions of Python that we'd prefer to use.
Bash: It's pre-installed everywhere, but doesn't come with a sane YAML
implementation. We could build or add such a YAML implementation, but
it sounds gnarly. I've seen some attempts at doing YAML parsing with
sed... that way madness lies. There are some binary YAML-parsing
tools, which we'd need to vendor and build to include as binaries. But
if we're compiling binaries anyway, we've already side-stepped the
advantage of bash.
Go: Since all the choices amount to needing to compile something down
to a binary, it makes the most sense to use a language that compiles
nicely down to a binary. Go is the obvious choice, since it has nice
multithreading options, is a technology the team is already familiar
with, is already a dependency for existing systems (so it's exactly as
portable as most of ATC), and compiles quickly and easily using
existing tools. It's also got a solid YAML library suitably licensed
for us.
C or something like that: Basically the same advantages of Go, but
without the developer familiarity or multithreading options.
All things considered, I think Go is the best choice for this
particular problem. It avoids all the bad things we want to avoid and
gives us some neat advantages in return.
On Mon, Nov 12, 2018 at 4:59 PM Eric Friedrich -X (efriedri - TRITON
UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
<ef...@cisco.com.invalid> wrote:
>
>
>
> On Nov 12, 2018, at 4:03 PM, Rawlin Peters <ra...@gmail.com>> wrote:
>
> replies inline
>
> On Mon, Nov 12, 2018 at 11:51 AM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> <ef...@cisco.com.invalid>> wrote:
>
> Hey Rawlin-
> Both good points worth some serious thought.
>
> On Nov 12, 2018, at 1:29 PM, Rawlin Peters <ra...@gmail.com>> wrote:
>
> Eric,
>
> I share your sentiment about being reluctant to introduce another
> language as a dependency for Traffic Ops, but I wasn't able to find a
> really good, easily-available utility for parsing yaml (a la `jq` for
> json parsing) in a Bash script. Since the goose config is in yaml,
> `db/admin.pl` uses a yaml package to parse the goose config into
> variables which are then passed to the external `psql` et al.
> commands. It is possible to parse yaml using sed, but the example I
> found for doing that seemed really sketchy and fragile. So I figured
> using a solid YAML-parsing library like PyYAML in Python would be a
> safer bet while still allowing the use of a fully-featured programming
> language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> would also allow us to potentially use a DB library to interface with
> the DB directly in Python rather than requiring `psql` et al. and just
> shelling out to those external commands (although I plan to continue
> doing it that way for now).
> EF> Yeah, the YAML parsing was the only thing I saw that was not a perfect fit for bash.
> Given how concise that db.yaml file that is, I wouldn’t think twice about getting the open line via:
> $ DB=“development” grep -A2 “$DB_NAME:” dbconf.yml | grep open | awk -F ":" '{print $2}'
>
> No special tool needed.
>
> I wish we could depend on the dbconf.yml file being a standard format
> like that in order to just be able to grep it with context, but there
> are no restrictions on empty lines or arbitrary comments in the yaml
> syntax that would easily break the grep command. We shouldn't have to
> enforce arbitrary formatting of a yaml config file just to remove the
> need for a yaml parser.
>
> EF> Yeah good point. I rarely touch the file so was not too worried about the syntax. But yeah, if people regularly change that file we should be more permissive with our parsing
>
> Is pyyaml part of the batteries-included packages?
> If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops RPM. (For those of us without permitted Internet access in our deployments, installing via pip means standing up our own private PyPi repo) at each of our customers- something I would really like to avoid.
>
> I don't believe it is part of the standard library, but I do believe
> we can figure out a way to keep it all self-contained to the traffic
> ops RPM without requiring internet access at install time. I think
> https://github.com/pyinstaller/pyinstaller is more than capable of
> doing that on Linux/Mac/Windows. I just tried it out on my Python
> version of db/admin.pl on my mac, and it created a self-contained
> binary of 4.7MB size that seems to work just fine. It does still
> depend on the external commands like `psql` et al. from the postgres
> package, but that's no different from the current Perl version. Do you
> think that would be sufficient?
>
> I think pyinstaller also solves the python2 vs python3 availability
> problem, since the interpreter is packaged into the resulting binary
> itself.
> EF> I think that would be OK. The licensing on that one is GPLv2 which is category X to include in source (we can distribute the boot loader though). Do you know of any alternatives with more compatible licensing?
>
>
>
>
> As a side note, it also paves the way for moving other scripts to
> Python like WebDep.pm, which uses a Perl package that is virtually
> impossible to install/get running on Mac because of Perl's broken SSL
> on Mac, which would make it much easier to start as a new developer on
> the project. I remember when I started working on Traffic Control, I
> had to copy someone else's Perl `traffic_ops/app/local` directory who
> had been on the project a long time and had actually gotten it to
> build on Mac before it became unusable. Eliminating issues like that
> by using a more popular and supportable language is a win in my book,
> but right now I'm just focusing on `db/admin.pl` to allow for better
> testability of the DB migration operations.
> EF>All the web_deps stuff should go away with the rest of perl, right?
>
> We really should be targeting RPMs that contain all dependencies (or can be resolved via yum/rpm), rather than asking our users to install stuff from the Internet at install time. Its fragile (packages disappear) and a bunch of TC users do not run in environments with access to the net.
>
> I’m not opposed to Python, its probably my favorite language, certainly the one I'm most comfortable in. I just don't see a compelling need for it with these changes.
> At some point soon we will need to rewrite perl scripts into something else (postinstall, ORT, etc…). We should closely consider our use of language for those as well- Go, Python, bash, etc…
>
> I think Go could be a reasonable language for stuff like db/admin.pl,
> but it seemed more natural to keep scripts as scripts for small stuff
> like that.
>
> - Rawlin
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Chris Lemmons <al...@gmail.com>.
Stepping back for a moment, I see a few goals:
- Install without requiring Internet access. Since local security
policies may (rightfully!) restrict upline resources, it would be best
if we can install without needing to download things.
- Not distribute CatX dependencies. This would be a bit of a
non-starter, I think.
- Use technologies well-understood by the team.
- Install quickly.
- Maintain the script easily.
- Minimize system dependencies.
When I analyze some of the suggestions, here's what I see.
Python: You either need Internet access to download python, to have
python embedded in the build at build time, or to have it
pre-installed on the system. The same goes for every dependent
library. The first option doesn't meet the goal of not requiring
Internet access. We can embed python if we wish without violating
licenses, since Python is licensed under the PSF, which is Category A.
However, while PyInstaller itself is GPL, which is CatX, it has a
bootstrapper exemption that applies to the files that it stores
directly as part of the installer-creator. Unfortunately, the
exemption does not allow modification of the files and contains what I
think is a Field of Use limitation which requires that they be used
only when linked into another executable. I suspect that this causes a
PyInstaller binary to be incompatible with the distribution mechanisms
of the Apache project. Perhaps there is another python single-binary
system we could consider here, though. Ultimately, though Python isn't
really designed as a compiled language and if we want to compile it,
we might want to choose a language designed for that.
Having python pre-installed would increase deployment challenges,
particularly since many system repositories don't have the modern
versions of Python that we'd prefer to use.
Bash: It's pre-installed everywhere, but doesn't come with a sane YAML
implementation. We could build or add such a YAML implementation, but
it sounds gnarly. I've seen some attempts at doing YAML parsing with
sed... that way madness lies. There are some binary YAML-parsing
tools, which we'd need to vendor and build to include as binaries. But
if we're compiling binaries anyway, we've already side-stepped the
advantage of bash.
Go: Since all the choices amount to needing to compile something down
to a binary, it makes the most sense to use a language that compiles
nicely down to a binary. Go is the obvious choice, since it has nice
multithreading options, is a technology the team is already familiar
with, is already a dependency for existing systems (so it's exactly as
portable as most of ATC), and compiles quickly and easily using
existing tools. It's also got a solid YAML library suitably licensed
for us.
C or something like that: Basically the same advantages of Go, but
without the developer familiarity or multithreading options.
All things considered, I think Go is the best choice for this
particular problem. It avoids all the bad things we want to avoid and
gives us some neat advantages in return.
On Mon, Nov 12, 2018 at 4:59 PM Eric Friedrich -X (efriedri - TRITON
UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
<ef...@cisco.com.invalid> wrote:
>
>
>
> On Nov 12, 2018, at 4:03 PM, Rawlin Peters <ra...@gmail.com>> wrote:
>
> replies inline
>
> On Mon, Nov 12, 2018 at 11:51 AM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> <ef...@cisco.com.invalid>> wrote:
>
> Hey Rawlin-
> Both good points worth some serious thought.
>
> On Nov 12, 2018, at 1:29 PM, Rawlin Peters <ra...@gmail.com>> wrote:
>
> Eric,
>
> I share your sentiment about being reluctant to introduce another
> language as a dependency for Traffic Ops, but I wasn't able to find a
> really good, easily-available utility for parsing yaml (a la `jq` for
> json parsing) in a Bash script. Since the goose config is in yaml,
> `db/admin.pl` uses a yaml package to parse the goose config into
> variables which are then passed to the external `psql` et al.
> commands. It is possible to parse yaml using sed, but the example I
> found for doing that seemed really sketchy and fragile. So I figured
> using a solid YAML-parsing library like PyYAML in Python would be a
> safer bet while still allowing the use of a fully-featured programming
> language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> would also allow us to potentially use a DB library to interface with
> the DB directly in Python rather than requiring `psql` et al. and just
> shelling out to those external commands (although I plan to continue
> doing it that way for now).
> EF> Yeah, the YAML parsing was the only thing I saw that was not a perfect fit for bash.
> Given how concise that db.yaml file that is, I wouldn’t think twice about getting the open line via:
> $ DB=“development” grep -A2 “$DB_NAME:” dbconf.yml | grep open | awk -F ":" '{print $2}'
>
> No special tool needed.
>
> I wish we could depend on the dbconf.yml file being a standard format
> like that in order to just be able to grep it with context, but there
> are no restrictions on empty lines or arbitrary comments in the yaml
> syntax that would easily break the grep command. We shouldn't have to
> enforce arbitrary formatting of a yaml config file just to remove the
> need for a yaml parser.
>
> EF> Yeah good point. I rarely touch the file so was not too worried about the syntax. But yeah, if people regularly change that file we should be more permissive with our parsing
>
> Is pyyaml part of the batteries-included packages?
> If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops RPM. (For those of us without permitted Internet access in our deployments, installing via pip means standing up our own private PyPi repo) at each of our customers- something I would really like to avoid.
>
> I don't believe it is part of the standard library, but I do believe
> we can figure out a way to keep it all self-contained to the traffic
> ops RPM without requiring internet access at install time. I think
> https://github.com/pyinstaller/pyinstaller is more than capable of
> doing that on Linux/Mac/Windows. I just tried it out on my Python
> version of db/admin.pl on my mac, and it created a self-contained
> binary of 4.7MB size that seems to work just fine. It does still
> depend on the external commands like `psql` et al. from the postgres
> package, but that's no different from the current Perl version. Do you
> think that would be sufficient?
>
> I think pyinstaller also solves the python2 vs python3 availability
> problem, since the interpreter is packaged into the resulting binary
> itself.
> EF> I think that would be OK. The licensing on that one is GPLv2 which is category X to include in source (we can distribute the boot loader though). Do you know of any alternatives with more compatible licensing?
>
>
>
>
> As a side note, it also paves the way for moving other scripts to
> Python like WebDep.pm, which uses a Perl package that is virtually
> impossible to install/get running on Mac because of Perl's broken SSL
> on Mac, which would make it much easier to start as a new developer on
> the project. I remember when I started working on Traffic Control, I
> had to copy someone else's Perl `traffic_ops/app/local` directory who
> had been on the project a long time and had actually gotten it to
> build on Mac before it became unusable. Eliminating issues like that
> by using a more popular and supportable language is a win in my book,
> but right now I'm just focusing on `db/admin.pl` to allow for better
> testability of the DB migration operations.
> EF>All the web_deps stuff should go away with the rest of perl, right?
>
> We really should be targeting RPMs that contain all dependencies (or can be resolved via yum/rpm), rather than asking our users to install stuff from the Internet at install time. Its fragile (packages disappear) and a bunch of TC users do not run in environments with access to the net.
>
> I’m not opposed to Python, its probably my favorite language, certainly the one I'm most comfortable in. I just don't see a compelling need for it with these changes.
> At some point soon we will need to rewrite perl scripts into something else (postinstall, ORT, etc…). We should closely consider our use of language for those as well- Go, Python, bash, etc…
>
> I think Go could be a reasonable language for stuff like db/admin.pl,
> but it seemed more natural to keep scripts as scripts for small stuff
> like that.
>
> - Rawlin
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus
(UK) Limited -OBO at Cisco)" <ef...@cisco.com.INVALID>.
On Nov 12, 2018, at 4:03 PM, Rawlin Peters <ra...@gmail.com>> wrote:
replies inline
On Mon, Nov 12, 2018 at 11:51 AM Eric Friedrich -X (efriedri - TRITON
UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
<ef...@cisco.com.invalid>> wrote:
Hey Rawlin-
Both good points worth some serious thought.
On Nov 12, 2018, at 1:29 PM, Rawlin Peters <ra...@gmail.com>> wrote:
Eric,
I share your sentiment about being reluctant to introduce another
language as a dependency for Traffic Ops, but I wasn't able to find a
really good, easily-available utility for parsing yaml (a la `jq` for
json parsing) in a Bash script. Since the goose config is in yaml,
`db/admin.pl` uses a yaml package to parse the goose config into
variables which are then passed to the external `psql` et al.
commands. It is possible to parse yaml using sed, but the example I
found for doing that seemed really sketchy and fragile. So I figured
using a solid YAML-parsing library like PyYAML in Python would be a
safer bet while still allowing the use of a fully-featured programming
language rather than "Bash + <insert yaml-parsing CLI tool here>". It
would also allow us to potentially use a DB library to interface with
the DB directly in Python rather than requiring `psql` et al. and just
shelling out to those external commands (although I plan to continue
doing it that way for now).
EF> Yeah, the YAML parsing was the only thing I saw that was not a perfect fit for bash.
Given how concise that db.yaml file that is, I wouldn’t think twice about getting the open line via:
$ DB=“development” grep -A2 “$DB_NAME:” dbconf.yml | grep open | awk -F ":" '{print $2}'
No special tool needed.
I wish we could depend on the dbconf.yml file being a standard format
like that in order to just be able to grep it with context, but there
are no restrictions on empty lines or arbitrary comments in the yaml
syntax that would easily break the grep command. We shouldn't have to
enforce arbitrary formatting of a yaml config file just to remove the
need for a yaml parser.
EF> Yeah good point. I rarely touch the file so was not too worried about the syntax. But yeah, if people regularly change that file we should be more permissive with our parsing
Is pyyaml part of the batteries-included packages?
If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops RPM. (For those of us without permitted Internet access in our deployments, installing via pip means standing up our own private PyPi repo) at each of our customers- something I would really like to avoid.
I don't believe it is part of the standard library, but I do believe
we can figure out a way to keep it all self-contained to the traffic
ops RPM without requiring internet access at install time. I think
https://github.com/pyinstaller/pyinstaller is more than capable of
doing that on Linux/Mac/Windows. I just tried it out on my Python
version of db/admin.pl on my mac, and it created a self-contained
binary of 4.7MB size that seems to work just fine. It does still
depend on the external commands like `psql` et al. from the postgres
package, but that's no different from the current Perl version. Do you
think that would be sufficient?
I think pyinstaller also solves the python2 vs python3 availability
problem, since the interpreter is packaged into the resulting binary
itself.
EF> I think that would be OK. The licensing on that one is GPLv2 which is category X to include in source (we can distribute the boot loader though). Do you know of any alternatives with more compatible licensing?
As a side note, it also paves the way for moving other scripts to
Python like WebDep.pm, which uses a Perl package that is virtually
impossible to install/get running on Mac because of Perl's broken SSL
on Mac, which would make it much easier to start as a new developer on
the project. I remember when I started working on Traffic Control, I
had to copy someone else's Perl `traffic_ops/app/local` directory who
had been on the project a long time and had actually gotten it to
build on Mac before it became unusable. Eliminating issues like that
by using a more popular and supportable language is a win in my book,
but right now I'm just focusing on `db/admin.pl` to allow for better
testability of the DB migration operations.
EF>All the web_deps stuff should go away with the rest of perl, right?
We really should be targeting RPMs that contain all dependencies (or can be resolved via yum/rpm), rather than asking our users to install stuff from the Internet at install time. Its fragile (packages disappear) and a bunch of TC users do not run in environments with access to the net.
I’m not opposed to Python, its probably my favorite language, certainly the one I'm most comfortable in. I just don't see a compelling need for it with these changes.
At some point soon we will need to rewrite perl scripts into something else (postinstall, ORT, etc…). We should closely consider our use of language for those as well- Go, Python, bash, etc…
I think Go could be a reasonable language for stuff like db/admin.pl,
but it seemed more natural to keep scripts as scripts for small stuff
like that.
- Rawlin
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Rawlin Peters <ra...@gmail.com>.
replies inline
On Mon, Nov 12, 2018 at 11:51 AM Eric Friedrich -X (efriedri - TRITON
UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
<ef...@cisco.com.invalid> wrote:
>
> Hey Rawlin-
> Both good points worth some serious thought.
>
> > On Nov 12, 2018, at 1:29 PM, Rawlin Peters <ra...@gmail.com> wrote:
> >
> > Eric,
> >
> > I share your sentiment about being reluctant to introduce another
> > language as a dependency for Traffic Ops, but I wasn't able to find a
> > really good, easily-available utility for parsing yaml (a la `jq` for
> > json parsing) in a Bash script. Since the goose config is in yaml,
> > `db/admin.pl` uses a yaml package to parse the goose config into
> > variables which are then passed to the external `psql` et al.
> > commands. It is possible to parse yaml using sed, but the example I
> > found for doing that seemed really sketchy and fragile. So I figured
> > using a solid YAML-parsing library like PyYAML in Python would be a
> > safer bet while still allowing the use of a fully-featured programming
> > language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> > would also allow us to potentially use a DB library to interface with
> > the DB directly in Python rather than requiring `psql` et al. and just
> > shelling out to those external commands (although I plan to continue
> > doing it that way for now).
> EF> Yeah, the YAML parsing was the only thing I saw that was not a perfect fit for bash.
> Given how concise that db.yaml file that is, I wouldn’t think twice about getting the open line via:
> $ DB=“development” grep -A2 “$DB_NAME:” dbconf.yml | grep open | awk -F ":" '{print $2}'
>
> No special tool needed.
I wish we could depend on the dbconf.yml file being a standard format
like that in order to just be able to grep it with context, but there
are no restrictions on empty lines or arbitrary comments in the yaml
syntax that would easily break the grep command. We shouldn't have to
enforce arbitrary formatting of a yaml config file just to remove the
need for a yaml parser.
> Is pyyaml part of the batteries-included packages?
> If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops RPM. (For those of us without permitted Internet access in our deployments, installing via pip means standing up our own private PyPi repo) at each of our customers- something I would really like to avoid.
I don't believe it is part of the standard library, but I do believe
we can figure out a way to keep it all self-contained to the traffic
ops RPM without requiring internet access at install time. I think
https://github.com/pyinstaller/pyinstaller is more than capable of
doing that on Linux/Mac/Windows. I just tried it out on my Python
version of db/admin.pl on my mac, and it created a self-contained
binary of 4.7MB size that seems to work just fine. It does still
depend on the external commands like `psql` et al. from the postgres
package, but that's no different from the current Perl version. Do you
think that would be sufficient?
I think pyinstaller also solves the python2 vs python3 availability
problem, since the interpreter is packaged into the resulting binary
itself.
> > As a side note, it also paves the way for moving other scripts to
> > Python like WebDep.pm, which uses a Perl package that is virtually
> > impossible to install/get running on Mac because of Perl's broken SSL
> > on Mac, which would make it much easier to start as a new developer on
> > the project. I remember when I started working on Traffic Control, I
> > had to copy someone else's Perl `traffic_ops/app/local` directory who
> > had been on the project a long time and had actually gotten it to
> > build on Mac before it became unusable. Eliminating issues like that
> > by using a more popular and supportable language is a win in my book,
> > but right now I'm just focusing on `db/admin.pl` to allow for better
> > testability of the DB migration operations.
> EF>All the web_deps stuff should go away with the rest of perl, right?
>
> We really should be targeting RPMs that contain all dependencies (or can be resolved via yum/rpm), rather than asking our users to install stuff from the Internet at install time. Its fragile (packages disappear) and a bunch of TC users do not run in environments with access to the net.
>
> I’m not opposed to Python, its probably my favorite language, certainly the one I'm most comfortable in. I just don't see a compelling need for it with these changes.
> At some point soon we will need to rewrite perl scripts into something else (postinstall, ORT, etc…). We should closely consider our use of language for those as well- Go, Python, bash, etc…
I think Go could be a reasonable language for stuff like db/admin.pl,
but it seemed more natural to keep scripts as scripts for small stuff
like that.
- Rawlin
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus
(UK) Limited -OBO at Cisco)" <ef...@cisco.com.INVALID>.
Hey Rawlin-
Both good points worth some serious thought.
> On Nov 12, 2018, at 1:29 PM, Rawlin Peters <ra...@gmail.com> wrote:
>
> Eric,
>
> I share your sentiment about being reluctant to introduce another
> language as a dependency for Traffic Ops, but I wasn't able to find a
> really good, easily-available utility for parsing yaml (a la `jq` for
> json parsing) in a Bash script. Since the goose config is in yaml,
> `db/admin.pl` uses a yaml package to parse the goose config into
> variables which are then passed to the external `psql` et al.
> commands. It is possible to parse yaml using sed, but the example I
> found for doing that seemed really sketchy and fragile. So I figured
> using a solid YAML-parsing library like PyYAML in Python would be a
> safer bet while still allowing the use of a fully-featured programming
> language rather than "Bash + <insert yaml-parsing CLI tool here>". It
> would also allow us to potentially use a DB library to interface with
> the DB directly in Python rather than requiring `psql` et al. and just
> shelling out to those external commands (although I plan to continue
> doing it that way for now).
EF> Yeah, the YAML parsing was the only thing I saw that was not a perfect fit for bash.
Given how concise that db.yaml file that is, I wouldn’t think twice about getting the open line via:
$ DB=“development” grep -A2 “$DB_NAME:” dbconf.yml | grep open | awk -F ":" '{print $2}'
No special tool needed.
Is pyyaml part of the batteries-included packages?
If not, we’ll need a way to distribute pyyaml.whl as part of the traffic ops RPM. (For those of us without permitted Internet access in our deployments, installing via pip means standing up our own private PyPi repo) at each of our customers- something I would really like to avoid.
>
> As a side note, it also paves the way for moving other scripts to
> Python like WebDep.pm, which uses a Perl package that is virtually
> impossible to install/get running on Mac because of Perl's broken SSL
> on Mac, which would make it much easier to start as a new developer on
> the project. I remember when I started working on Traffic Control, I
> had to copy someone else's Perl `traffic_ops/app/local` directory who
> had been on the project a long time and had actually gotten it to
> build on Mac before it became unusable. Eliminating issues like that
> by using a more popular and supportable language is a win in my book,
> but right now I'm just focusing on `db/admin.pl` to allow for better
> testability of the DB migration operations.
EF>All the web_deps stuff should go away with the rest of perl, right?
We really should be targeting RPMs that contain all dependencies (or can be resolved via yum/rpm), rather than asking our users to install stuff from the Internet at install time. Its fragile (packages disappear) and a bunch of TC users do not run in environments with access to the net.
I’m not opposed to Python, its probably my favorite language, certainly the one I'm most comfortable in. I just don't see a compelling need for it with these changes.
At some point soon we will need to rewrite perl scripts into something else (postinstall, ORT, etc…). We should closely consider our use of language for those as well- Go, Python, bash, etc…
—Eric
>
> - Rawlin
>
> On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
> UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
> <ef...@cisco.com.invalid> wrote:
>>
>> I’m only slightly familiar with all the different options for db/admin.pl.
>>
>> I’m a big fan of Python, but reluctant to introduce another language into TC without a strong reason.
>>
>> Once the reverse_schema option is removed, what would be the main purposes of the script?
>>
>> Looks like this is something that could be easily converted to bash without need for another dependency.
>>
>> —Eric
>>
>>
>>
>>
>>
>>
>>
>>> On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
>>>
>>> +1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
>>> seem to be gaining a lot of Python expertise recently.
>>>
>>> -1 on 2.x compatibility -- writing something new with compatibility for 2
>>> major versions makes no sense to me. It limits the features and libraries
>>> that can be used and potentially doubles the amount of testing required.
>>>
>>>
>>>
>>> On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
>>>
>>>> +1, seems reasonable. I don’t really have an opinion on python 2.x
>>>> compatibility, but whatever makes the most sense for the amount of work is
>>>> what I would prefer.
>>>>
>>>> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <Jo...@comcast.com>
>>>> wrote:
>>>>
>>>>> +1 There is already precedence in the repo for python for other purposes.
>>>>> The only caveat I would include is to be sure you include backward
>>>>> compatibility for python 2.x for the next year or so until it goes EOL.
>>>>>
>>>>> Jonathan G
>>>>>
>>>>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
>>>>>
>>>>> Hey TC devs,
>>>>>
>>>>> As we eliminate the need for Traffic Ops Perl, it will no longer
>>>>> really make sense for the db/admin.pl script to be Perl as it is
>>>>> today. This is because it depends on the Perl modules that are
>>>>> installed via Carton for running Traffic Ops Perl. However, it's
>>>>> mostly just a Perl wrapper around shell commands except for a YAML
>>>>> Perl module and the `reverse_schema` command which uses the DBIx Perl
>>>>> module to generate the ORM schema for Traffic Ops Perl (i.e. you've
>>>>> added a new DB table/column and need to get the ORM files updated to
>>>>> use it).
>>>>>
>>>>> Without TO-Perl, there's no need for the `reverse_schema` command in
>>>>> db/admin.pl and its dependency on the Perl DBIx module. At that
>>>> point
>>>>> db/admin.pl is just a Perl script that parses some YAML and shells
>>>> out
>>>>> commands.
>>>>>
>>>>> Part of the problem with TO-Perl is that there are a bunch of random
>>>>> non-API Perl scripts that basically assume all the modules in the
>>>>> cpanfile are installed in the environment, even though a lot of those
>>>>> dependencies are probably only required for the Perl TO API or UI. So
>>>>> unless we go through all those Perl dependencies to eliminate
>>>>> everything we don't really need once the Perl TO API and UI are
>>>>> completely removed, we'll continue to have a handful of Perl scripts
>>>>> like db/admin.pl that still require downloading and installing the
>>>>> full set of TO Perl dependencies. On fresh installs, running Carton
>>>> to
>>>>> install these dependencies can take nearly half an hour.
>>>>>
>>>>> I'm working on adding tests for the DB upgrade/downgrade process
>>>> which
>>>>> I'd like to run automatically on PR submissions, but it really only
>>>>> needs the db/admin.pl script out of TO which assumes the entire set
>>>> of
>>>>> Perl TO dependencies even though it mostly just shells out to `goose
>>>>> up` and `goose down`. I'd like the test to emulate an actual TO
>>>>> environment as closely as possible to match what would actually
>>>> happen
>>>>> in a production DB upgrade/downgrade. Right now I'm reusing the
>>>>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
>>>>> to port db/admin.pl into Python to give it its own set of
>>>> dependencies
>>>>> (just a YAML package) and not require the full set of TO Perl
>>>>> dependencies. Then I can spin up a much lighter-weight container with
>>>>> just the TO Python packages rather than setting up a separate
>>>> cpanfile
>>>>> with just the Perl packages needed for db/admin.pl.
>>>>>
>>>>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
>>>>> scripts like db/admin.pl?
>>>>>
>>>>> -Rawlin
>>>>>
>>>>>
>>>>>
>>>>
>>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Rawlin Peters <ra...@gmail.com>.
Eric,
I share your sentiment about being reluctant to introduce another
language as a dependency for Traffic Ops, but I wasn't able to find a
really good, easily-available utility for parsing yaml (a la `jq` for
json parsing) in a Bash script. Since the goose config is in yaml,
`db/admin.pl` uses a yaml package to parse the goose config into
variables which are then passed to the external `psql` et al.
commands. It is possible to parse yaml using sed, but the example I
found for doing that seemed really sketchy and fragile. So I figured
using a solid YAML-parsing library like PyYAML in Python would be a
safer bet while still allowing the use of a fully-featured programming
language rather than "Bash + <insert yaml-parsing CLI tool here>". It
would also allow us to potentially use a DB library to interface with
the DB directly in Python rather than requiring `psql` et al. and just
shelling out to those external commands (although I plan to continue
doing it that way for now).
As a side note, it also paves the way for moving other scripts to
Python like WebDep.pm, which uses a Perl package that is virtually
impossible to install/get running on Mac because of Perl's broken SSL
on Mac, which would make it much easier to start as a new developer on
the project. I remember when I started working on Traffic Control, I
had to copy someone else's Perl `traffic_ops/app/local` directory who
had been on the project a long time and had actually gotten it to
build on Mac before it became unusable. Eliminating issues like that
by using a more popular and supportable language is a win in my book,
but right now I'm just focusing on `db/admin.pl` to allow for better
testability of the DB migration operations.
- Rawlin
On Mon, Nov 12, 2018 at 6:46 AM Eric Friedrich -X (efriedri - TRITON
UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)
<ef...@cisco.com.invalid> wrote:
>
> I’m only slightly familiar with all the different options for db/admin.pl.
>
> I’m a big fan of Python, but reluctant to introduce another language into TC without a strong reason.
>
> Once the reverse_schema option is removed, what would be the main purposes of the script?
>
> Looks like this is something that could be easily converted to bash without need for another dependency.
>
> —Eric
>
>
>
>
>
>
>
> > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
> >
> > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
> > seem to be gaining a lot of Python expertise recently.
> >
> > -1 on 2.x compatibility -- writing something new with compatibility for 2
> > major versions makes no sense to me. It limits the features and libraries
> > that can be used and potentially doubles the amount of testing required.
> >
> >
> >
> > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
> >
> >> +1, seems reasonable. I don’t really have an opinion on python 2.x
> >> compatibility, but whatever makes the most sense for the amount of work is
> >> what I would prefer.
> >>
> >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <Jo...@comcast.com>
> >> wrote:
> >>
> >>> +1 There is already precedence in the repo for python for other purposes.
> >>> The only caveat I would include is to be sure you include backward
> >>> compatibility for python 2.x for the next year or so until it goes EOL.
> >>>
> >>> Jonathan G
> >>>
> >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
> >>>
> >>> Hey TC devs,
> >>>
> >>> As we eliminate the need for Traffic Ops Perl, it will no longer
> >>> really make sense for the db/admin.pl script to be Perl as it is
> >>> today. This is because it depends on the Perl modules that are
> >>> installed via Carton for running Traffic Ops Perl. However, it's
> >>> mostly just a Perl wrapper around shell commands except for a YAML
> >>> Perl module and the `reverse_schema` command which uses the DBIx Perl
> >>> module to generate the ORM schema for Traffic Ops Perl (i.e. you've
> >>> added a new DB table/column and need to get the ORM files updated to
> >>> use it).
> >>>
> >>> Without TO-Perl, there's no need for the `reverse_schema` command in
> >>> db/admin.pl and its dependency on the Perl DBIx module. At that
> >> point
> >>> db/admin.pl is just a Perl script that parses some YAML and shells
> >> out
> >>> commands.
> >>>
> >>> Part of the problem with TO-Perl is that there are a bunch of random
> >>> non-API Perl scripts that basically assume all the modules in the
> >>> cpanfile are installed in the environment, even though a lot of those
> >>> dependencies are probably only required for the Perl TO API or UI. So
> >>> unless we go through all those Perl dependencies to eliminate
> >>> everything we don't really need once the Perl TO API and UI are
> >>> completely removed, we'll continue to have a handful of Perl scripts
> >>> like db/admin.pl that still require downloading and installing the
> >>> full set of TO Perl dependencies. On fresh installs, running Carton
> >> to
> >>> install these dependencies can take nearly half an hour.
> >>>
> >>> I'm working on adding tests for the DB upgrade/downgrade process
> >> which
> >>> I'd like to run automatically on PR submissions, but it really only
> >>> needs the db/admin.pl script out of TO which assumes the entire set
> >> of
> >>> Perl TO dependencies even though it mostly just shells out to `goose
> >>> up` and `goose down`. I'd like the test to emulate an actual TO
> >>> environment as closely as possible to match what would actually
> >> happen
> >>> in a production DB upgrade/downgrade. Right now I'm reusing the
> >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
> >>> to port db/admin.pl into Python to give it its own set of
> >> dependencies
> >>> (just a YAML package) and not require the full set of TO Perl
> >>> dependencies. Then I can spin up a much lighter-weight container with
> >>> just the TO Python packages rather than setting up a separate
> >> cpanfile
> >>> with just the Perl packages needed for db/admin.pl.
> >>>
> >>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
> >>> scripts like db/admin.pl?
> >>>
> >>> -Rawlin
> >>>
> >>>
> >>>
> >>
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Jeremy Mitchell <mi...@gmail.com>.
@jonathon - i think goose up/down runs migrations and migrations should be
limited to structural database changes. If data changes (inserts, updates,
deletes, etc) are introduced into migrations then the ability to migrate
down can get real tricky. i think that's why patches.sql was introduced for
data changes.
Jeremy
On Mon, Nov 12, 2018 at 9:27 AM Gray, Jonathan <Jo...@comcast.com>
wrote:
> Correct me if I'm wrong, but one of the important functions of the
> admin.pl script is the db record seeds. It seems like there's kindof a
> separation when comes to upgrading the database that structural changes run
> through goose while record initialization runs through admin.pl. I'm not
> aware of a reason why goose couldn't insert rows into the database, but I
> havn't dug too deep on that.
>
> Jonathan G
>
> On 11/12/18, 6:46 AM, "Eric Friedrich -X (efriedri - TRITON UK BIDCO
> LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)"
> <ef...@cisco.com.INVALID> wrote:
>
> I’m only slightly familiar with all the different options for db/
> admin.pl.
>
> I’m a big fan of Python, but reluctant to introduce another language
> into TC without a strong reason.
>
> Once the reverse_schema option is removed, what would be the main
> purposes of the script?
>
> Looks like this is something that could be easily converted to bash
> without need for another dependency.
>
> —Eric
>
>
>
>
>
>
>
> > On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
> >
> > +1 on rewriting admin.pl -- Python seems a reasonable choice, esp
> since we
> > seem to be gaining a lot of Python expertise recently.
> >
> > -1 on 2.x compatibility -- writing something new with compatibility
> for 2
> > major versions makes no sense to me. It limits the features and
> libraries
> > that can be used and potentially doubles the amount of testing
> required.
> >
> >
> >
> > On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org>
> wrote:
> >
> >> +1, seems reasonable. I don’t really have an opinion on python 2.x
> >> compatibility, but whatever makes the most sense for the amount of
> work is
> >> what I would prefer.
> >>
> >> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <
> Jonathan_Gray@comcast.com>
> >> wrote:
> >>
> >>> +1 There is already precedence in the repo for python for other
> purposes.
> >>> The only caveat I would include is to be sure you include backward
> >>> compatibility for python 2.x for the next year or so until it goes
> EOL.
> >>>
> >>> Jonathan G
> >>>
> >>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com>
> wrote:
> >>>
> >>> Hey TC devs,
> >>>
> >>> As we eliminate the need for Traffic Ops Perl, it will no longer
> >>> really make sense for the db/admin.pl script to be Perl as it
> is
> >>> today. This is because it depends on the Perl modules that are
> >>> installed via Carton for running Traffic Ops Perl. However, it's
> >>> mostly just a Perl wrapper around shell commands except for a
> YAML
> >>> Perl module and the `reverse_schema` command which uses the
> DBIx Perl
> >>> module to generate the ORM schema for Traffic Ops Perl (i.e.
> you've
> >>> added a new DB table/column and need to get the ORM files
> updated to
> >>> use it).
> >>>
> >>> Without TO-Perl, there's no need for the `reverse_schema`
> command in
> >>> db/admin.pl and its dependency on the Perl DBIx module. At that
> >> point
> >>> db/admin.pl is just a Perl script that parses some YAML and
> shells
> >> out
> >>> commands.
> >>>
> >>> Part of the problem with TO-Perl is that there are a bunch of
> random
> >>> non-API Perl scripts that basically assume all the modules in
> the
> >>> cpanfile are installed in the environment, even though a lot of
> those
> >>> dependencies are probably only required for the Perl TO API or
> UI. So
> >>> unless we go through all those Perl dependencies to eliminate
> >>> everything we don't really need once the Perl TO API and UI are
> >>> completely removed, we'll continue to have a handful of Perl
> scripts
> >>> like db/admin.pl that still require downloading and installing
> the
> >>> full set of TO Perl dependencies. On fresh installs, running
> Carton
> >> to
> >>> install these dependencies can take nearly half an hour.
> >>>
> >>> I'm working on adding tests for the DB upgrade/downgrade process
> >> which
> >>> I'd like to run automatically on PR submissions, but it really
> only
> >>> needs the db/admin.pl script out of TO which assumes the
> entire set
> >> of
> >>> Perl TO dependencies even though it mostly just shells out to
> `goose
> >>> up` and `goose down`. I'd like the test to emulate an actual TO
> >>> environment as closely as possible to match what would actually
> >> happen
> >>> in a production DB upgrade/downgrade. Right now I'm reusing the
> >>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally
> I'd like
> >>> to port db/admin.pl into Python to give it its own set of
> >> dependencies
> >>> (just a YAML package) and not require the full set of TO Perl
> >>> dependencies. Then I can spin up a much lighter-weight
> container with
> >>> just the TO Python packages rather than setting up a separate
> >> cpanfile
> >>> with just the Perl packages needed for db/admin.pl.
> >>>
> >>> +1/-1 for adding Python as a dependency of Traffic Ops for
> porting
> >>> scripts like db/admin.pl?
> >>>
> >>> -Rawlin
> >>>
> >>>
> >>>
> >>
>
>
>
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
Correct me if I'm wrong, but one of the important functions of the admin.pl script is the db record seeds. It seems like there's kindof a separation when comes to upgrading the database that structural changes run through goose while record initialization runs through admin.pl. I'm not aware of a reason why goose couldn't insert rows into the database, but I havn't dug too deep on that.
Jonathan G
On 11/12/18, 6:46 AM, "Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus (UK) Limited -OBO at Cisco)" <ef...@cisco.com.INVALID> wrote:
I’m only slightly familiar with all the different options for db/admin.pl.
I’m a big fan of Python, but reluctant to introduce another language into TC without a strong reason.
Once the reverse_schema option is removed, what would be the main purposes of the script?
Looks like this is something that could be easily converted to bash without need for another dependency.
—Eric
> On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
>
> +1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
> seem to be gaining a lot of Python expertise recently.
>
> -1 on 2.x compatibility -- writing something new with compatibility for 2
> major versions makes no sense to me. It limits the features and libraries
> that can be used and potentially doubles the amount of testing required.
>
>
>
> On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
>
>> +1, seems reasonable. I don’t really have an opinion on python 2.x
>> compatibility, but whatever makes the most sense for the amount of work is
>> what I would prefer.
>>
>> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <Jo...@comcast.com>
>> wrote:
>>
>>> +1 There is already precedence in the repo for python for other purposes.
>>> The only caveat I would include is to be sure you include backward
>>> compatibility for python 2.x for the next year or so until it goes EOL.
>>>
>>> Jonathan G
>>>
>>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
>>>
>>> Hey TC devs,
>>>
>>> As we eliminate the need for Traffic Ops Perl, it will no longer
>>> really make sense for the db/admin.pl script to be Perl as it is
>>> today. This is because it depends on the Perl modules that are
>>> installed via Carton for running Traffic Ops Perl. However, it's
>>> mostly just a Perl wrapper around shell commands except for a YAML
>>> Perl module and the `reverse_schema` command which uses the DBIx Perl
>>> module to generate the ORM schema for Traffic Ops Perl (i.e. you've
>>> added a new DB table/column and need to get the ORM files updated to
>>> use it).
>>>
>>> Without TO-Perl, there's no need for the `reverse_schema` command in
>>> db/admin.pl and its dependency on the Perl DBIx module. At that
>> point
>>> db/admin.pl is just a Perl script that parses some YAML and shells
>> out
>>> commands.
>>>
>>> Part of the problem with TO-Perl is that there are a bunch of random
>>> non-API Perl scripts that basically assume all the modules in the
>>> cpanfile are installed in the environment, even though a lot of those
>>> dependencies are probably only required for the Perl TO API or UI. So
>>> unless we go through all those Perl dependencies to eliminate
>>> everything we don't really need once the Perl TO API and UI are
>>> completely removed, we'll continue to have a handful of Perl scripts
>>> like db/admin.pl that still require downloading and installing the
>>> full set of TO Perl dependencies. On fresh installs, running Carton
>> to
>>> install these dependencies can take nearly half an hour.
>>>
>>> I'm working on adding tests for the DB upgrade/downgrade process
>> which
>>> I'd like to run automatically on PR submissions, but it really only
>>> needs the db/admin.pl script out of TO which assumes the entire set
>> of
>>> Perl TO dependencies even though it mostly just shells out to `goose
>>> up` and `goose down`. I'd like the test to emulate an actual TO
>>> environment as closely as possible to match what would actually
>> happen
>>> in a production DB upgrade/downgrade. Right now I'm reusing the
>>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
>>> to port db/admin.pl into Python to give it its own set of
>> dependencies
>>> (just a YAML package) and not require the full set of TO Perl
>>> dependencies. Then I can spin up a much lighter-weight container with
>>> just the TO Python packages rather than setting up a separate
>> cpanfile
>>> with just the Perl packages needed for db/admin.pl.
>>>
>>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
>>> scripts like db/admin.pl?
>>>
>>> -Rawlin
>>>
>>>
>>>
>>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by "Eric Friedrich -X (efriedri - TRITON UK BIDCO LIMITED c/o Alter Domus
(UK) Limited -OBO at Cisco)" <ef...@cisco.com.INVALID>.
I’m only slightly familiar with all the different options for db/admin.pl.
I’m a big fan of Python, but reluctant to introduce another language into TC without a strong reason.
Once the reverse_schema option is removed, what would be the main purposes of the script?
Looks like this is something that could be easily converted to bash without need for another dependency.
—Eric
> On Nov 10, 2018, at 1:44 PM, Dan Kirkwood <da...@gmail.com> wrote:
>
> +1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
> seem to be gaining a lot of Python expertise recently.
>
> -1 on 2.x compatibility -- writing something new with compatibility for 2
> major versions makes no sense to me. It limits the features and libraries
> that can be used and potentially doubles the amount of testing required.
>
>
>
> On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
>
>> +1, seems reasonable. I don’t really have an opinion on python 2.x
>> compatibility, but whatever makes the most sense for the amount of work is
>> what I would prefer.
>>
>> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <Jo...@comcast.com>
>> wrote:
>>
>>> +1 There is already precedence in the repo for python for other purposes.
>>> The only caveat I would include is to be sure you include backward
>>> compatibility for python 2.x for the next year or so until it goes EOL.
>>>
>>> Jonathan G
>>>
>>> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
>>>
>>> Hey TC devs,
>>>
>>> As we eliminate the need for Traffic Ops Perl, it will no longer
>>> really make sense for the db/admin.pl script to be Perl as it is
>>> today. This is because it depends on the Perl modules that are
>>> installed via Carton for running Traffic Ops Perl. However, it's
>>> mostly just a Perl wrapper around shell commands except for a YAML
>>> Perl module and the `reverse_schema` command which uses the DBIx Perl
>>> module to generate the ORM schema for Traffic Ops Perl (i.e. you've
>>> added a new DB table/column and need to get the ORM files updated to
>>> use it).
>>>
>>> Without TO-Perl, there's no need for the `reverse_schema` command in
>>> db/admin.pl and its dependency on the Perl DBIx module. At that
>> point
>>> db/admin.pl is just a Perl script that parses some YAML and shells
>> out
>>> commands.
>>>
>>> Part of the problem with TO-Perl is that there are a bunch of random
>>> non-API Perl scripts that basically assume all the modules in the
>>> cpanfile are installed in the environment, even though a lot of those
>>> dependencies are probably only required for the Perl TO API or UI. So
>>> unless we go through all those Perl dependencies to eliminate
>>> everything we don't really need once the Perl TO API and UI are
>>> completely removed, we'll continue to have a handful of Perl scripts
>>> like db/admin.pl that still require downloading and installing the
>>> full set of TO Perl dependencies. On fresh installs, running Carton
>> to
>>> install these dependencies can take nearly half an hour.
>>>
>>> I'm working on adding tests for the DB upgrade/downgrade process
>> which
>>> I'd like to run automatically on PR submissions, but it really only
>>> needs the db/admin.pl script out of TO which assumes the entire set
>> of
>>> Perl TO dependencies even though it mostly just shells out to `goose
>>> up` and `goose down`. I'd like the test to emulate an actual TO
>>> environment as closely as possible to match what would actually
>> happen
>>> in a production DB upgrade/downgrade. Right now I'm reusing the
>>> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
>>> to port db/admin.pl into Python to give it its own set of
>> dependencies
>>> (just a YAML package) and not require the full set of TO Perl
>>> dependencies. Then I can spin up a much lighter-weight container with
>>> just the TO Python packages rather than setting up a separate
>> cpanfile
>>> with just the Perl packages needed for db/admin.pl.
>>>
>>> +1/-1 for adding Python as a dependency of Traffic Ops for porting
>>> scripts like db/admin.pl?
>>>
>>> -Rawlin
>>>
>>>
>>>
>>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Dan Kirkwood <da...@gmail.com>.
+1 on rewriting admin.pl -- Python seems a reasonable choice, esp since we
seem to be gaining a lot of Python expertise recently.
-1 on 2.x compatibility -- writing something new with compatibility for 2
major versions makes no sense to me. It limits the features and libraries
that can be used and potentially doubles the amount of testing required.
On Sat, Nov 10, 2018 at 8:47 AM Dave Neuman <ne...@apache.org> wrote:
> +1, seems reasonable. I don’t really have an opinion on python 2.x
> compatibility, but whatever makes the most sense for the amount of work is
> what I would prefer.
>
> On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <Jo...@comcast.com>
> wrote:
>
> > +1 There is already precedence in the repo for python for other purposes.
> > The only caveat I would include is to be sure you include backward
> > compatibility for python 2.x for the next year or so until it goes EOL.
> >
> > Jonathan G
> >
> > On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
> >
> > Hey TC devs,
> >
> > As we eliminate the need for Traffic Ops Perl, it will no longer
> > really make sense for the db/admin.pl script to be Perl as it is
> > today. This is because it depends on the Perl modules that are
> > installed via Carton for running Traffic Ops Perl. However, it's
> > mostly just a Perl wrapper around shell commands except for a YAML
> > Perl module and the `reverse_schema` command which uses the DBIx Perl
> > module to generate the ORM schema for Traffic Ops Perl (i.e. you've
> > added a new DB table/column and need to get the ORM files updated to
> > use it).
> >
> > Without TO-Perl, there's no need for the `reverse_schema` command in
> > db/admin.pl and its dependency on the Perl DBIx module. At that
> point
> > db/admin.pl is just a Perl script that parses some YAML and shells
> out
> > commands.
> >
> > Part of the problem with TO-Perl is that there are a bunch of random
> > non-API Perl scripts that basically assume all the modules in the
> > cpanfile are installed in the environment, even though a lot of those
> > dependencies are probably only required for the Perl TO API or UI. So
> > unless we go through all those Perl dependencies to eliminate
> > everything we don't really need once the Perl TO API and UI are
> > completely removed, we'll continue to have a handful of Perl scripts
> > like db/admin.pl that still require downloading and installing the
> > full set of TO Perl dependencies. On fresh installs, running Carton
> to
> > install these dependencies can take nearly half an hour.
> >
> > I'm working on adding tests for the DB upgrade/downgrade process
> which
> > I'd like to run automatically on PR submissions, but it really only
> > needs the db/admin.pl script out of TO which assumes the entire set
> of
> > Perl TO dependencies even though it mostly just shells out to `goose
> > up` and `goose down`. I'd like the test to emulate an actual TO
> > environment as closely as possible to match what would actually
> happen
> > in a production DB upgrade/downgrade. Right now I'm reusing the
> > Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
> > to port db/admin.pl into Python to give it its own set of
> dependencies
> > (just a YAML package) and not require the full set of TO Perl
> > dependencies. Then I can spin up a much lighter-weight container with
> > just the TO Python packages rather than setting up a separate
> cpanfile
> > with just the Perl packages needed for db/admin.pl.
> >
> > +1/-1 for adding Python as a dependency of Traffic Ops for porting
> > scripts like db/admin.pl?
> >
> > -Rawlin
> >
> >
> >
>
Re: [EXTERNAL] The future of the db/admin.pl script
Posted by Dave Neuman <ne...@apache.org>.
+1, seems reasonable. I don’t really have an opinion on python 2.x
compatibility, but whatever makes the most sense for the amount of work is
what I would prefer.
On Fri, Nov 9, 2018 at 18:06 Gray, Jonathan <Jo...@comcast.com>
wrote:
> +1 There is already precedence in the repo for python for other purposes.
> The only caveat I would include is to be sure you include backward
> compatibility for python 2.x for the next year or so until it goes EOL.
>
> Jonathan G
>
> On 11/9/18, 5:23 PM, "Rawlin Peters" <ra...@gmail.com> wrote:
>
> Hey TC devs,
>
> As we eliminate the need for Traffic Ops Perl, it will no longer
> really make sense for the db/admin.pl script to be Perl as it is
> today. This is because it depends on the Perl modules that are
> installed via Carton for running Traffic Ops Perl. However, it's
> mostly just a Perl wrapper around shell commands except for a YAML
> Perl module and the `reverse_schema` command which uses the DBIx Perl
> module to generate the ORM schema for Traffic Ops Perl (i.e. you've
> added a new DB table/column and need to get the ORM files updated to
> use it).
>
> Without TO-Perl, there's no need for the `reverse_schema` command in
> db/admin.pl and its dependency on the Perl DBIx module. At that point
> db/admin.pl is just a Perl script that parses some YAML and shells out
> commands.
>
> Part of the problem with TO-Perl is that there are a bunch of random
> non-API Perl scripts that basically assume all the modules in the
> cpanfile are installed in the environment, even though a lot of those
> dependencies are probably only required for the Perl TO API or UI. So
> unless we go through all those Perl dependencies to eliminate
> everything we don't really need once the Perl TO API and UI are
> completely removed, we'll continue to have a handful of Perl scripts
> like db/admin.pl that still require downloading and installing the
> full set of TO Perl dependencies. On fresh installs, running Carton to
> install these dependencies can take nearly half an hour.
>
> I'm working on adding tests for the DB upgrade/downgrade process which
> I'd like to run automatically on PR submissions, but it really only
> needs the db/admin.pl script out of TO which assumes the entire set of
> Perl TO dependencies even though it mostly just shells out to `goose
> up` and `goose down`. I'd like the test to emulate an actual TO
> environment as closely as possible to match what would actually happen
> in a production DB upgrade/downgrade. Right now I'm reusing the
> Traffic-Ops-Perl Docker image from cdn-in-a-box, but ideally I'd like
> to port db/admin.pl into Python to give it its own set of dependencies
> (just a YAML package) and not require the full set of TO Perl
> dependencies. Then I can spin up a much lighter-weight container with
> just the TO Python packages rather than setting up a separate cpanfile
> with just the Perl packages needed for db/admin.pl.
>
> +1/-1 for adding Python as a dependency of Traffic Ops for porting
> scripts like db/admin.pl?
>
> -Rawlin
>
>
>