You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by ocket 8888 <oc...@gmail.com> on 2019/09/12 17:25:48 UTC
Changing dbdump output format
I'm rewriting the dbdump endpoint (PR #3912) and the output format used by
Perl is a binary, compressed SQL script. It makes editing and even testing
just a bit harder than it could be. The data is still compressed for
transfers, but a plain SQL script is much easier for a human to analyze.
pg_restore can work with any format that pg_dump outputs - and it
automatically detects which one to use, so the user doesn't even need to
know, really.
But is this a problem for anyone, for any reason?
Re: Changing dbdump output format
Posted by ocket 8888 <oc...@gmail.com>.
Yes, it seems I misunderstood the manpage. pg_restore will *create* SQL
scripts (if dbname isn't given), it can't *use* SQL scripts.
So although I still think this represents an improvement,
testing/deployment/restore pipelines in use for the database may need to be
altered to accommodate it.
On Fri, Sep 13, 2019 at 10:59 AM ocket 8888 <oc...@gmail.com> wrote:
> AFAIK the current output format is incomprehensible to anything but
> pg_restore
>
> On Fri, Sep 13, 2019 at 10:21 AM Jeremy Mitchell <mi...@gmail.com>
> wrote:
>
>> If pg_restore can handle both (the current binary and your suggested plain
>> sql) seamlessly then doesn't seem like a problem to me to change the
>> format
>> to plain SQL but i can't speak for everyone using this endpoint. is
>> everyone using the endpoint's response in tandem with pg_restore?
>>
>> if it helps testability, i think your suggestion makes a lot of sense.
>>
>> jeremy
>>
>> On Thu, Sep 12, 2019 at 11:26 AM ocket 8888 <oc...@gmail.com> wrote:
>>
>> > I'm rewriting the dbdump endpoint (PR #3912) and the output format used
>> by
>> > Perl is a binary, compressed SQL script. It makes editing and even
>> testing
>> > just a bit harder than it could be. The data is still compressed for
>> > transfers, but a plain SQL script is much easier for a human to analyze.
>> > pg_restore can work with any format that pg_dump outputs - and it
>> > automatically detects which one to use, so the user doesn't even need to
>> > know, really.
>> >
>> > But is this a problem for anyone, for any reason?
>> >
>>
>
Re: Changing dbdump output format
Posted by ocket 8888 <oc...@gmail.com>.
I could add content negotiation, but that'd be redoing some stuff I did for
the DS stats rewrite. Plus, I agree that this endpoint shouldn't exist.
I'll make it use the custom format, but it'll be harder to tell why when
tests fail.
On Fri, Sep 13, 2019 at 12:58 PM Rawlin Peters <ra...@gmail.com>
wrote:
> We have tools that use the /dbdump API for things like DB backups and
> DB testing. If the API output changed out beneath those things, they
> would 100% break. IMO the default output should remain as it is today,
> and if we think a new format is needed then we should just add an
> optional query parameter to the /dbdump endpoint. For example:
> /api/1.1/dbdump (uses default compressed format)
> /api/1.1/dbdump?format=compressed (same as default)
> /api/1.1/dbdump?format=sql (uses new "SQL script" output format for
> whatever reason)
>
> That is, if we think we even have a valid use case for a new format.
> We should be careful not to add things to the API that we don't really
> need. Already it seems like this endpoint is very dangerous and
> _probably_ shouldn't exist (although it does and things depend on it
> now). If a test can just read the current compressed format and
> convert it to a sql script for analysis or whatever, that's probably
> better than adding support for various output formats.
>
> - Rawlin
>
> On Fri, Sep 13, 2019 at 10:59 AM ocket 8888 <oc...@gmail.com> wrote:
> >
> > AFAIK the current output format is incomprehensible to anything but
> > pg_restore
> >
> > On Fri, Sep 13, 2019 at 10:21 AM Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> >
> > > If pg_restore can handle both (the current binary and your suggested
> plain
> > > sql) seamlessly then doesn't seem like a problem to me to change the
> format
> > > to plain SQL but i can't speak for everyone using this endpoint. is
> > > everyone using the endpoint's response in tandem with pg_restore?
> > >
> > > if it helps testability, i think your suggestion makes a lot of sense.
> > >
> > > jeremy
> > >
> > > On Thu, Sep 12, 2019 at 11:26 AM ocket 8888 <oc...@gmail.com>
> wrote:
> > >
> > > > I'm rewriting the dbdump endpoint (PR #3912) and the output format
> used
> > > by
> > > > Perl is a binary, compressed SQL script. It makes editing and even
> > > testing
> > > > just a bit harder than it could be. The data is still compressed for
> > > > transfers, but a plain SQL script is much easier for a human to
> analyze.
> > > > pg_restore can work with any format that pg_dump outputs - and it
> > > > automatically detects which one to use, so the user doesn't even
> need to
> > > > know, really.
> > > >
> > > > But is this a problem for anyone, for any reason?
> > > >
> > >
>
Re: Changing dbdump output format
Posted by Rawlin Peters <ra...@gmail.com>.
We have tools that use the /dbdump API for things like DB backups and
DB testing. If the API output changed out beneath those things, they
would 100% break. IMO the default output should remain as it is today,
and if we think a new format is needed then we should just add an
optional query parameter to the /dbdump endpoint. For example:
/api/1.1/dbdump (uses default compressed format)
/api/1.1/dbdump?format=compressed (same as default)
/api/1.1/dbdump?format=sql (uses new "SQL script" output format for
whatever reason)
That is, if we think we even have a valid use case for a new format.
We should be careful not to add things to the API that we don't really
need. Already it seems like this endpoint is very dangerous and
_probably_ shouldn't exist (although it does and things depend on it
now). If a test can just read the current compressed format and
convert it to a sql script for analysis or whatever, that's probably
better than adding support for various output formats.
- Rawlin
On Fri, Sep 13, 2019 at 10:59 AM ocket 8888 <oc...@gmail.com> wrote:
>
> AFAIK the current output format is incomprehensible to anything but
> pg_restore
>
> On Fri, Sep 13, 2019 at 10:21 AM Jeremy Mitchell <mi...@gmail.com>
> wrote:
>
> > If pg_restore can handle both (the current binary and your suggested plain
> > sql) seamlessly then doesn't seem like a problem to me to change the format
> > to plain SQL but i can't speak for everyone using this endpoint. is
> > everyone using the endpoint's response in tandem with pg_restore?
> >
> > if it helps testability, i think your suggestion makes a lot of sense.
> >
> > jeremy
> >
> > On Thu, Sep 12, 2019 at 11:26 AM ocket 8888 <oc...@gmail.com> wrote:
> >
> > > I'm rewriting the dbdump endpoint (PR #3912) and the output format used
> > by
> > > Perl is a binary, compressed SQL script. It makes editing and even
> > testing
> > > just a bit harder than it could be. The data is still compressed for
> > > transfers, but a plain SQL script is much easier for a human to analyze.
> > > pg_restore can work with any format that pg_dump outputs - and it
> > > automatically detects which one to use, so the user doesn't even need to
> > > know, really.
> > >
> > > But is this a problem for anyone, for any reason?
> > >
> >
Re: Changing dbdump output format
Posted by ocket 8888 <oc...@gmail.com>.
AFAIK the current output format is incomprehensible to anything but
pg_restore
On Fri, Sep 13, 2019 at 10:21 AM Jeremy Mitchell <mi...@gmail.com>
wrote:
> If pg_restore can handle both (the current binary and your suggested plain
> sql) seamlessly then doesn't seem like a problem to me to change the format
> to plain SQL but i can't speak for everyone using this endpoint. is
> everyone using the endpoint's response in tandem with pg_restore?
>
> if it helps testability, i think your suggestion makes a lot of sense.
>
> jeremy
>
> On Thu, Sep 12, 2019 at 11:26 AM ocket 8888 <oc...@gmail.com> wrote:
>
> > I'm rewriting the dbdump endpoint (PR #3912) and the output format used
> by
> > Perl is a binary, compressed SQL script. It makes editing and even
> testing
> > just a bit harder than it could be. The data is still compressed for
> > transfers, but a plain SQL script is much easier for a human to analyze.
> > pg_restore can work with any format that pg_dump outputs - and it
> > automatically detects which one to use, so the user doesn't even need to
> > know, really.
> >
> > But is this a problem for anyone, for any reason?
> >
>
Re: [EXTERNAL] Re: Changing dbdump output format
Posted by "Rivas, Jesse" <Je...@comcast.com>.
pg_restore cannot handle plain-text formats -- psql has to be used in order
to restore from .sql format.
Jesse
On 9/13/19, 10:21 AM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
If pg_restore can handle both (the current binary and your suggested plain
sql) seamlessly then doesn't seem like a problem to me to change the format
to plain SQL but i can't speak for everyone using this endpoint. is
everyone using the endpoint's response in tandem with pg_restore?
if it helps testability, i think your suggestion makes a lot of sense.
jeremy
On Thu, Sep 12, 2019 at 11:26 AM ocket 8888 <oc...@gmail.com> wrote:
> I'm rewriting the dbdump endpoint (PR #3912) and the output format used by
> Perl is a binary, compressed SQL script. It makes editing and even testing
> just a bit harder than it could be. The data is still compressed for
> transfers, but a plain SQL script is much easier for a human to analyze.
> pg_restore can work with any format that pg_dump outputs - and it
> automatically detects which one to use, so the user doesn't even need to
> know, really.
>
> But is this a problem for anyone, for any reason?
>
Re: Changing dbdump output format
Posted by Jeremy Mitchell <mi...@gmail.com>.
If pg_restore can handle both (the current binary and your suggested plain
sql) seamlessly then doesn't seem like a problem to me to change the format
to plain SQL but i can't speak for everyone using this endpoint. is
everyone using the endpoint's response in tandem with pg_restore?
if it helps testability, i think your suggestion makes a lot of sense.
jeremy
On Thu, Sep 12, 2019 at 11:26 AM ocket 8888 <oc...@gmail.com> wrote:
> I'm rewriting the dbdump endpoint (PR #3912) and the output format used by
> Perl is a binary, compressed SQL script. It makes editing and even testing
> just a bit harder than it could be. The data is still compressed for
> transfers, but a plain SQL script is much easier for a human to analyze.
> pg_restore can work with any format that pg_dump outputs - and it
> automatically detects which one to use, so the user doesn't even need to
> know, really.
>
> But is this a problem for anyone, for any reason?
>