You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Alex Dubov <oa...@yahoo.com> on 2006/06/01 04:03:52 UTC
Re: LONGs and LOBs [Was: improved error reporting for apr_dbd_mysql.c]
>
> Second, instead of paying attention to various
> letters like %B, %L, %s
> and so on, which come from when the statement was
> prepared, we ignore
> them in p[v]bquery/bselect, but we pass
> apr_dbd_entry_t ** instead of
> const char ** into the function. Then we have both
> the type/length
> figured out explicitly and can pass binary data in
> safely, without the
> need to do dubious strlen()'s.
>
I'm afraid that this will unnecessary complicate an
already ugly code. That is, data types are most often
known in advance and any above typical database code
is not a pretty picture even without the need to
assemble additional data structures. I would rather
prefer something like:
void* apr_dbd_get_entry_as(apr_dbd_type_t type)
or may be even
int apr_dbd_get_entries(char *format, ...)
(akin to scanf) which returns data cast as desired
type if possible, or null.
NULLs are moderately rare type to fetch - I think it's
always better to filter them out on SQL level.
Therefore:
int apr_dbd_entry_is_null(const apr_dbd_row_t *row,
int n)
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: LONGs and LOBs [Was: improved error reporting for
apr_dbd_mysql.c]
Posted by Bojan Smojver <bo...@rexursive.com>.
On Fri, 2006-06-02 at 20:12 +1000, Bojan Smojver wrote:
> length:column:table:payload
> ------ ------ ----- -------
> ^ ^ ^ ^
> | | | +-----------> binary data, length in size
> +------+------+-----------------> ascii
>
> Obviously, for most databases column and table could be omitted,
> reducing the thing to:
>
> length:::payload
If we wanted to be even more pedantic, we could call the above %B and
the following %b:
> length:payload
Just a thought...
--
Bojan
Re: LONGs and LOBs [Was: improved error reporting for
apr_dbd_mysql.c]
Posted by Bojan Smojver <bo...@rexursive.com>.
On Fri, 2006-06-02 at 13:18 -0400, Garrett Rooney wrote:
> If it has never been present on a release branch we have zero need to
> worry about backwards compatibility. If someone is depending on the
> code in trunk clearly they have enough clue to deal with the breakages
> that can result.
Henceforth, longs shall be known as %l and %L ;-)
Recap of the proposal after the change:
- %s - varchar
- %S - text
- %c - char
- %C - unsigned char
- %h - short
- %H - unsigned short
- %d - integer
- %D - unsigned integer
- %l - long
- %L - unsigned long
- %q - quad (i.e. long long)
- %Q - unsigned quad
- %f - float
- %F - double
- %i - time
- %a - date
- %A - datetime
- %t - timestamp
- %T - timestamp with zone
- %b - blob (encoded on C side as per previous e-mail)
--
Bojan
Re: LONGs and LOBs [Was: improved error reporting for apr_dbd_mysql.c]
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 6/2/06, Chris Darroch <ch...@pearsoncmg.com> wrote:
> Bojan Smojver wrote:
>
> > length:column:table:payload
> > ------ ------ ----- -------
> > ^ ^ ^ ^
> > | | | +-----------> binary data, length in size
> > +------+------+-----------------> ascii
>
> That could work for me, but if others feel it's too much of a hack,
> fair enough.
>
> > - %O - unsigned long (big L has been taken by Oracle driver already!)
>
> A question re APR policy, really -- do we need to care about this,
> since the Oracle driver only exists in trunk, IIRC? I'm all for
> backwards compatibility, but thought I should clarify if it's required
> in this case.
If it has never been present on a release branch we have zero need to
worry about backwards compatibility. If someone is depending on the
code in trunk clearly they have enough clue to deal with the breakages
that can result.
-garrett
Re: LONGs and LOBs [Was: improved error reporting for apr_dbd_mysql.c]
Posted by Chris Darroch <ch...@pearsoncmg.com>.
Bojan Smojver wrote:
> length:column:table:payload
> ------ ------ ----- -------
> ^ ^ ^ ^
> | | | +-----------> binary data, length in size
> +------+------+-----------------> ascii
That could work for me, but if others feel it's too much of a hack,
fair enough.
> - %O - unsigned long (big L has been taken by Oracle driver already!)
A question re APR policy, really -- do we need to care about this,
since the Oracle driver only exists in trunk, IIRC? I'm all for
backwards compatibility, but thought I should clarify if it's required
in this case.
Chris.
--
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263 E4DE C8E3 FA36 366A 375B
Re: LONGs and LOBs [Was: improved error reporting for
apr_dbd_mysql.c]
Posted by Bojan Smojver <bo...@rexursive.com>.
On Fri, 2006-06-02 at 07:58 +1000, Bojan Smojver wrote:
> One more note here: if we go with this approach, this is almost
> certainly something for APU 2.0 due to existing implementation
> incompatibility. For instance, PostgreSQL driver uses %d to cast to
> integer in SQL, but fetches a char* as an argument nonetheless. If we
> were to implement a change in which an int* is expected with %d,
> current code would break.
Now this got me thinking (I know, I shouldn't be doing this :-), because
waiting for APR 2.0 probably means waiting for Apache 2.4 as well...
Instead of introducing various types of pointers on C side and then
using that in p[v]query/select, maybe we can stick to const char *
instead, but encode other information into the string. For instance,
something referred to as %b (blob, binary data) could be passed into
p[v]query/select and fetched fromo apr_dbd_binary_get() as:
length:column:table:payload
------ ------ ----- -------
^ ^ ^ ^
| | | +-----------> binary data, length in size
+------+------+-----------------> ascii
Obviously, for most databases column and table could be omitted,
reducing the thing to:
length:::payload
or even:
length:payload
With this approach we can:
- keep binary compatibility
- cater for Oracle "sing and dance" routine
- keep passing strings around for everything, but most importantly the
simple stuff (strings, integers, floats etc.), as we are doing now
- get stuff back in this format from apr_dbd_binary_get()
I know it's going to a bit of hassle to "pack" and "unpack" data, but we
can have handy little functions (or even macros) for that, making the
job much easier.
In such a case, the "format identifiers" would have to be adjusted
slightly, in order to preserve current meanings. The types would mostly
have meaning on the SQL side and less on C side (i.e. everything would
be picked up as const char *). Something like this:
- %s - varchar
- %S - text
- %c - char
- %C - unsigned char
- %h - short
- %H - unsigned short
- %d - integer
- %D - unsigned integer
- %o - long
- %O - unsigned long (big L has been taken by Oracle driver already!)
- %q - quad (i.e. long long)
- %Q - unsigned quad
- %f - float
- %F - double
- %i - time
- %a - date
- %A - datetime
- %t - timestamp
- %T - timestamp with zone
- %b - blob (only this one is encoded on C side as per above!)
The above would be the only things we'd advertise in APU 1.3.x, so that
people know they can rely on this. All other letters that are currently
implemented differently (e.g. %L in Oracle driver), would be kept
silently to avoid breaking things.
Then in APR 2.0, we can do the fancy stuff with buckets, brigades and
other goodies.
<fire_proof_suit_on/> ;-)
--
Bojan
Re: LONGs and LOBs [Was: improved error reporting for
apr_dbd_mysql.c]
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Bojan Smojver <bo...@rexursive.com>:
> I'm sure there are many more combos that we can consider, but these
> should be a decent start.
One more note here: if we go with this approach, this is almost
certainly something for APU 2.0 due to existing implementation
incompatibility. For instance, PostgreSQL driver uses %d to cast to
integer in SQL, but fetches a char* as an argument nonetheless. If we
were to implement a change in which an int* is expected with %d,
current code would break.
--
Bojan
Re: LONGs and LOBs [Was: improved error reporting for apr_dbd_mysql.c]
Posted by Chris Darroch <ch...@pearsoncmg.com>.
Alex Dubov wrote:
> int apr_dbd_entry_is_null(const apr_dbd_row_t *row, int n)
That strikes me as a good solution for the current API; thanks!
I suppose it might have to be:
int apr_dbd_entry_is_null(const apr_dbd_driver_t *driver,
const apr_dbd_row_t *row, int n);
but I haven't thought about it enough yet to be sure either way.
Bojan Smojver wrote:
> However, we can have a new function instead, so that we remain
> backward compatible. Someting like apr_dbd_entry_get(). Or, if that's
> no good because people may expect intuitively that this is the same as
> apr_dbd_get_entry(), then maybe apr_dbd_binary_get() or something like
> that.
Yeah, I'd suggest that any renaming of functions that we do
to match APR/APU (driver_get, dbname_set, row_get, entry_get, name_get?)
should not also add or change the functionality from the old versions.
Otherwise it'll just add confusion for everyone.
Bojan Smojver wrote:
> Quoting Alex Dubov <oa...@yahoo.com>:
>> from here everything works just like with
>> printf - if you push pointer incompatible to
>> specifier, it's most likely a crash.
[snip]
> I think we should stick with a single letter convention, for
> simplicity of parsing. So, how about:
>
> - %S - string (i.e. char *, text in SQL)
> - %D - unsigned integer
> - %c - char
> - %C - unsigned char
> - %h - short
> - %H - unsigned short
> - %l - long
> - %L - unsigned long
> - %q - quad (i.e. long long)
> - %Q - unsigned quad
> - %F - double
> - %b - blob (brigade, backend specific blob type 1 in SQL*)
> - %B - blob (brigade, backend specific blob type 2 in SQL*)
> - %o - blob (brigade, backend specific blob type 3 in SQL*)
> - %O - blob (brigade, backend specific blob type 4 in SQL*)
> - %i - time (string, time in SQL)
> - %a - date (string, date in SQL)
> - %A - datetime (string, datetime in SQL)
> - %t - timestamp (string, timestamp in SQL)
> - %T - timestamp with zone (string, timestamp with zone in SQL)
I confess I'm alarmed and frightened by all this, although I am
still willing to be convinced of its necessity. At the very least,
I'd hope that most of the processing can occur in set of routines
common to all drivers, otherwise it'll just be terribly difficult to
maintain going forward if each driver implements the parsing for
all this on its own.
Dealing just with LOB types, I'm also concerned about whether
using just a brigade as input data is possible. It may also
prove to be difficult to provide a portable interface; from what
little I know about MySQL, it handles LOBs like any other data
type, so a simple brigade might be OK. Oracle, as I noted, requires
a whole song and dance if you want to insert or update LOB data
into a row as an atomic transaction. If you want to make Oracle
do that, you really need to pass in the following:
- LOB data
- LOB data length
- table name
- LOB column name
(Although it's sometimes possible to determine the last two from
the SQL query, it's not always so.)
Anyway, I'll think on the various suggestions for a bit before
making other comments. And maybe someday I'll actually get back
to writing some code. :-)
Chris.
--
GPG Key ID: 366A375B
GPG Key Fingerprint: 485E 5041 17E1 E2BB C263 E4DE C8E3 FA36 366A 375B
Re: LONGs and LOBs [Was: improved error reporting for
apr_dbd_mysql.c]
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Alex Dubov <oa...@yahoo.com>:
> I'm actually ok with this. But why do you need 4 blob
> types? I say - give backend a bucket and it should
> figure by itself what to do with it (use smallest blob
> type available that fits). I would also propose a
> plain bucket type, to simplify things:
>
> %b - bucket
> %B - bucket brigade
The above was actually my first thought, but then I had a look at the
Oracle driver and noticed that column types are being explicitly set.
Like this:
------------------------------------------
sql->status = OCIAttrSet(statement->args[i].value.lobval,
OCI_DTYPE_LOB, &null, 0,
OCI_ATTR_LOBEMPTY, sql->err);
------------------------------------------
I don't know Oracle API at all (nor did I actually check any
documentation), but I left the possiblity for more than one of those
types, just in case.
But, if we can simplify that to %b (bucket) and %B (brigade) as per
above, I would be more than happy to go with it. I'm sure Oracle folks
will let us know if that's good or not.
> By the way, what are the plans for apu-2 (when it
> should be ready)?
Dunno. I think some long time APR/APU folks will have to answer this.
--
Bojan
Re: LONGs and LOBs [Was: improved error reporting for apr_dbd_mysql.c]
Posted by Alex Dubov <oa...@yahoo.com>.
> We already have:
>
> - %s - string (i.e. char *, varchar in SQL)
> - %d - integer
> - %f - float
>
> I think we should stick with a single letter
> convention, for
> simplicity of parsing. So, how about:
>
> - %S - string (i.e. char *, text in SQL)
> - %D - unsigned integer
> - %c - char
> - %C - unsigned char
> - %h - short
> - %H - unsigned short
> - %l - long
> - %L - unsigned long
> - %q - quad (i.e. long long)
> - %Q - unsigned quad
> - %F - double
> - %b - blob (brigade, backend specific blob type 1
> in SQL*)
> - %B - blob (brigade, backend specific blob type 2
> in SQL*)
> - %o - blob (brigade, backend specific blob type 3
> in SQL*)
> - %O - blob (brigade, backend specific blob type 4
> in SQL*)
> - %i - time (string, time in SQL)
> - %a - date (string, date in SQL)
> - %A - datetime (string, datetime in SQL)
> - %t - timestamp (string, timestamp in SQL)
> - %T - timestamp with zone (string, timestamp with
> zone in SQL)
>
> * Blob types vary between backends a lot (e.g. MySQL
> has four types,
> Oracle at least 3 and so on). We should pick a few
> we'd like to
> implement in each.
I'm actually ok with this. But why do you need 4 blob
types? I say - give backend a bucket and it should
figure by itself what to do with it (use smallest blob
type available that fits). I would also propose a
plain bucket type, to simplify things:
%b - bucket
%B - bucket brigade
By the way, what are the plans for apu-2 (when it
should be ready)?
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: LONGs and LOBs [Was: improved error reporting for
apr_dbd_mysql.c]
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Alex Dubov <oa...@yahoo.com>:
> from here everything works just like with
> printf - if you push pointer incompatible to
> specifier, it's most likely a crash.
Ah, right. I think I'm starting to understand this now - I thought we
were going to end up with a bunch of backend specific stuff (which I
don't like much :-), but it doesn't have to be that way.
So, it seems that we need to define, on DBD level, what identifiers
mean. They have to convey both the meaning in C and SQL, because
drivers needs hints on how to map C types to SQL types in _prepare.
Getting the stuff once the select is done is easier, but we should
probably stick with the same letters, to make it easier for callers.
We already have:
- %s - string (i.e. char *, varchar in SQL)
- %d - integer
- %f - float
I think we should stick with a single letter convention, for
simplicity of parsing. So, how about:
- %S - string (i.e. char *, text in SQL)
- %D - unsigned integer
- %c - char
- %C - unsigned char
- %h - short
- %H - unsigned short
- %l - long
- %L - unsigned long
- %q - quad (i.e. long long)
- %Q - unsigned quad
- %F - double
- %b - blob (brigade, backend specific blob type 1 in SQL*)
- %B - blob (brigade, backend specific blob type 2 in SQL*)
- %o - blob (brigade, backend specific blob type 3 in SQL*)
- %O - blob (brigade, backend specific blob type 4 in SQL*)
- %i - time (string, time in SQL)
- %a - date (string, date in SQL)
- %A - datetime (string, datetime in SQL)
- %t - timestamp (string, timestamp in SQL)
- %T - timestamp with zone (string, timestamp with zone in SQL)
I'm sure there are many more combos that we can consider, but these
should be a decent start.
Let the flame wars begin... ;-)
* Blob types vary between backends a lot (e.g. MySQL has four types,
Oracle at least 3 and so on). We should pick a few we'd like to
implement in each.
--
Bojan
Re: LONGs and LOBs [Was: improved error reporting for apr_dbd_mysql.c]
Posted by Alex Dubov <oa...@yahoo.com>.
> The other problem is of input arguments for prepared
> statements, where
> again we don't have support for anything but zero
> terminated strings
> at the moment. With a generic structure that Nick
> suggested, we could
> solve that problem too.
I think we need to straighten this out. Nick basically
proposes to use structure similar to MYSQL_BIND one -
you grab the structure, fill it with data (type,
length) and pass it to the dbd (I saw that Nick
currently suggests to append only type tag). I'm
thinking about different approach - you have your data
(including bucket or bb for binaries); you have format
specifier; from here everything works just like with
printf - if you push pointer incompatible to
specifier, it's most likely a crash.
There's nothing in p(v)select that prevents it from
using generic pointers. People are doing this for
years with printf and it never was a problem. Another
way to view this: specifier is a type tag, but there's
no need to carry it all over as there's no easy way
for user to act upon it. It's not like type of any
specific field is going to change suddenly and apr_dbd
is not implemented in Scheme (may be it should be).
I'm using this approach for a while now and it works
rather well.
Of course, back-ends should not be obligated to
support anything beside strings, but it will be nice
for them to actively reject specifiers they don't
support.
The symmetric case, with get_entry implemented in a
scanf style is not difficult to implement, at least
with mysql. libmysqlclient does all the needed
conversions itself (I don't know about other
databases, but they can honour only strings and reject
everything else, if they want to). Additional
advantage here is movement of numbers - you can skip
allocation and simply fetch into user supplied
pointers. Even better: this approach makes get_row
unneeded, simplifying typical db access.
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Re: LONGs and LOBs [Was: improved error reporting for
apr_dbd_mysql.c]
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Alex Dubov <oa...@yahoo.com>:
>> Second, instead of paying attention to various
>> letters like %B, %L, %s
>> and so on, which come from when the statement was
>> prepared, we ignore
>> them in p[v]bquery/bselect, but we pass
>> apr_dbd_entry_t ** instead of
>> const char ** into the function. Then we have both
>> the type/length
>> figured out explicitly and can pass binary data in
>> safely, without the
>> need to do dubious strlen()'s.
I was actually referring there to the input arguments of prepared
statements, not values pulled out of select.
> void* apr_dbd_get_entry_as(apr_dbd_type_t type)
>
> or may be even
>
> int apr_dbd_get_entries(char *format, ...)
Right. But we're still no better off - we still need to be able to
store all kinds of types in something we understand with just APU DBD.
The underlying problem is that we don't have a way of returning binary
data (i.e. buffer + length or something even smarter, like a brigade)
from get_entry(). Nick's little structure solves that problem for
getting the entries out of select, because it tells us all we need to
know. Your solution goes a step further and attempts to cast things as
well for the caller, but will still need something (or many of those)
known to the caller on APU DBD level to store all kinds of stuff.
The other problem is of input arguments for prepared statements, where
again we don't have support for anything but zero terminated strings
at the moment. With a generic structure that Nick suggested, we could
solve that problem too.
--
Bojan