You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "D. Richard Hipp" <dr...@hwaci.com> on 2009/05/04 13:56:45 UTC
Suggestion from the SQLite developers
We received the following post on the SQLite mailing list today:
http://article.gmane.org/gmane.comp.db.sqlite.general/46505
I have not looked at the SVN code, but it would appear that in order
to compile SVN one must first install SQLite libraries on the build
system. In other words, it appears that SVN has a dependency SQLite.
May I suggest you do away with this dependency. Grab a copy of the
SQLite amalgamation source file and its header (sqlite3.c and
sqlite3.h) for whatever version of SQLite works for you, and check in
those files as part of your own source tree. Adjust your makefiles so
that they compile the built-in sqlite3.c instead of trying to link
against a library (which may or may not be available on the system.)
This will make it that much easier for people to compile SVN.
Upgrading is easy. When you decide you want to use the latest version
of SQLite (because it is faster or has some new feature you want)
simple download the latest sqlite3.c and sqlite3.h from the SQLite
website, drop them into your own source tree, and recompile.
This will also make SVN more resistant to breakage if somebody updates
the system SQLite library - since SVN will not depend on the system
SQLite library but will have its own. We work hard to keep SQLite
fully compatible from one release to the next. But sometimes a bug
fix in SQLite will break systems that depend on the old buggy
behavior. So it never hurts to stick with a specific version of
SQLite unless you are 100% sure that your use of SQLite never depends
on any undefined or ill-defined behavior.
PHP, Mozilla, Python, Perl, Fossil, Tcl, and many other projects
manage SQLite this way. All commercial projects that we are aware of
use SQLite this way. The SQLite library is not that big. It will not
bloat your source tree. But it will tend to make your users happier.
D. Richard Hipp
drh@sqlite.org
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2056719
Re: Suggestion from the SQLite developers
Posted by Stefan Sperling <st...@elego.de>.
On Mon, May 04, 2009 at 07:27:25PM -0400, D. Richard Hipp wrote:
> So, even though we can honestly claim that each version of SQLite is
> backwards compatible, because of issues such as the above, it is
> normally a good idea to retest everything when changing SQLite
> versions. If SQLite is a shared library that dozens of applications
> use, upgrading it might break one or more of those apps. Is that
> really something you want to do?
Yes. Applications breaking because they have bugs is a good thing,
because it means that the bugs will have to be fixed. Leaving such
bugs unfixed can cause more harm than the breakage endured while
the bug has been fixed in the library but not yet in the application.
> I hear your point about wanting to do security updates quickly and
> efficiency. But consider that zlib and xpdf (and libpng, and libjpeg,
> etc) are libraries that are usually handed untrusted and unchecked
> inputs for processing. So a bug in them can easily lead to an
> exploit. It is much, much less common to feed untrusted and unchecked
> inputs into your SQL database engine on the other hand.
I would say that this is a very dangerous assumption to make.
> There is usually (hopefully) several layers of application in between the
> database engine and the black-hat hacker, and those intervening layers
> go a long way toward mitigating any potential for exploits.
And they go a long way towards adding more security problems, too.
Because more code generally means more bugs.
And what if there is no input checking done by the application?
In security, people tend to assume the worst-case scenario, and
there are good reasons for doing so. Programmers using a library
will often tend to assume that the library implicitly handles certain
tasks for them, because it saves them from doing extra work.
That's human nature.
The recent advisories in OpenSSL and Debian's apt, both of which were
due to failures in checking values returned from library code properly,
are good examples of this pattern:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-5077
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1358
In both cases, it was assumed that safety checks on certificates by the
library were working correctly, but they were not. The libraries even
indicated errors, but not in ways that the applications were programed
to handle. What guarantees that users of sqlite won't make similar errors,
possibly on the input data path (the examples I gave were on the output
data path)?
Here is a recent example of a different SQL database (mysql) not
doing proper input validation, leading to problems:
http://debian.org/security/2009/dsa-1783
So it happens all the time. Assuming it won't happen because it
will be caught by "higher layers" is dangerous.
> So I argue that a vulnerability in SQLite is unlikely to effect most
> application and most applications need not be recompiled for
> security. Hence it is not really necessary to factor out SQLite into
> a single shared library.
I would hope most people will go the route that makes it easier
to fix security problems quickly. Nevertheless, we provide the
option of embedding sqlite in Subversion to our users, so they
can decide for themselves. But I would not recommend doing so
in builds that are used in production, and especially not in
environments were some amount of security is required.
I embed sqlite in builds I use for Subversion development,
but just because it is a very convenient way of compiling.
sqlite is a very important piece of software, thank you for making
it and ensuring that it is of good quality. It helps us a lot, since
it is a core part of our new working copy library. Thanks :)
Stefan
Re: Suggestion from the SQLite developers
Posted by "D. Richard Hipp" <dr...@hwaci.com>.
On May 4, 2009, at 4:33 PM, Peter Samuelson wrote:
>
> [Greg Stein]
>> We chose to allow options other than amalgamation because we'd heard
>> feedback, "let the sysadmins upgrade sqlite independently on their
>> system", rather than pinning svn to whatever-we-shipped. IOW, if we
>> ship an amalgamation, then it becomes *our problem* when bugs appear
>> in sqlite, and we need to spin up a new release.
>
> Yes, and this choice is valuable. In Debian, we learned our lesson
> about redundant copies of library code in our binaries. It used to be
> quite common for projects to copy zlib code into their own projects
> ... until a potential security bug was found in it. For weeks
> afterwards, the Debian Security Team was patching random packages and
> releasing advisories for them. Not much fun for them, I think.
>
> Some years later, the same thing happened with xpdf, another popular
> codebase to copy and paste. So these days, we're pretty strict about
> _not_ embedding library code into our binaries, unless the code really
> is specific to the one project.
There are two sides to this coin.
We work very hard at SQLite to make it backwards compatible in every
way: Query language, C/C++ interface, and file format. But sometimes
applications will come to depend on undefined behavior, such at the
output order of SELECT statement that lack an ORDER BY clause. Then
when the library is upgraded and the arbitrary output order changes,
the application breaks.
Sometimes applications depend on buggy behavior. This has recently
hit SQLite hard. There was a bug in the SUBSTR. The function
SUBSTR(x, 1, 5) should return the first 5 characters of string x, and
it does and always has. However SUBSTR(x, 0, 5) was also returning
the first 5 characters of x. That was a bug. It should have only
returned the first 4. Yet, many programmers, accustomed to zero-based
C or javascript code were writing SUBSTR(x, 0, 5) when SUBSTR(x, 1, 5)
was correct. And because of the bug in SQLite they were getting the
right answer. Now that we have fixed the bug, they are getting the
wrong answer. And that is causing a lot of grief for people who are
upgrading their SQLite libraries.
So, even though we can honestly claim that each version of SQLite is
backwards compatible, because of issues such as the above, it is
normally a good idea to retest everything when changing SQLite
versions. If SQLite is a shared library that dozens of applications
use, upgrading it might break one or more of those apps. Is that
really something you want to do?
I hear your point about wanting to do security updates quickly and
efficiency. But consider that zlib and xpdf (and libpng, and libjpeg,
etc) are libraries that are usually handed untrusted and unchecked
inputs for processing. So a bug in them can easily lead to an
exploit. It is much, much less common to feed untrusted and unchecked
inputs into your SQL database engine on the other hand. There is
usually (hopefully) several layers of application in between the
database engine and the black-hat hacker, and those intervening layers
go a long way toward mitigating any potential for exploits. So I
argue that a vulnerability in SQLite is unlikely to effect most
application and most applications need not be recompiled for
security. Hence it is not really necessary to factor out SQLite into
a single shared library.
Do whatever you want at Debian and with my blessings. (www.sqlite.org
runs on Debian, fwiw, so I do trust and admire your product.) But I
will argue that factoring out all uses of SQLite into a single shared
library will likely result in more application breakage, not less, and
so I encourage you to reconsider your policy.
A number of groups are doing as SVN does and building in SQLite but
providing a compile-time option to use a shared library instead.
That's OK. But if it were me, I would include some disclaimer with
the system-shared-library option to the effect that if it breaks, the
user gets to keep both pieces and that the SVN developers will only
debug problems when using the built-in SQLite. But that's just me....
D. Richard Hipp
drh@hwaci.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2061553
Re: Suggestion from the SQLite developers
Posted by Peter Samuelson <pe...@p12n.org>.
[Greg Stein]
> We chose to allow options other than amalgamation because we'd heard
> feedback, "let the sysadmins upgrade sqlite independently on their
> system", rather than pinning svn to whatever-we-shipped. IOW, if we
> ship an amalgamation, then it becomes *our problem* when bugs appear
> in sqlite, and we need to spin up a new release.
Yes, and this choice is valuable. In Debian, we learned our lesson
about redundant copies of library code in our binaries. It used to be
quite common for projects to copy zlib code into their own projects
... until a potential security bug was found in it. For weeks
afterwards, the Debian Security Team was patching random packages and
releasing advisories for them. Not much fun for them, I think.
Some years later, the same thing happened with xpdf, another popular
codebase to copy and paste. So these days, we're pretty strict about
_not_ embedding library code into our binaries, unless the code really
is specific to the one project.
--
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/
Re: Suggestion from the SQLite developers
Posted by "D. Richard Hipp" <dr...@hwaci.com>.
On May 4, 2009, at 10:06 AM, Greg Stein wrote:
>
> We chose to allow options other than amalgamation because we'd heard
> feedback, "let the sysadmins upgrade sqlite independently on their
> system", rather than pinning svn to whatever-we-shipped. IOW, if we
> ship an amalgamation, then it becomes *our problem* when bugs appear
> in sqlite, and we need to spin up a new release.
Excellent. This is a good and common choice.
>
> For your user with the problem, please redirect them to
> users@subversion.tigris.org, and we'll help them out.
>
It has been done as you suggest.
We are honored that you are using SQLite. We hope that you continue
to find it useful. Please let us know if you encounter any
difficulties with it.
D. Richard Hipp
drh@hwaci.com
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2056935
Re: Suggestion from the SQLite developers
Posted by Greg Stein <gs...@gmail.com>.
Hey Richard!
(long time, no see... I still remember our fun Foo session a couple
years ago...)
Yup. Subversion has grown a SQLite dependency. It is "light" in 1.6,
but we are movin to a serious dependency upon it for 1.7.
Our build system allows for using SQLite that has been installed, a
specific SQLite directory, or an amalgamation file. For the latter, we
compile it privately into our libsvn_subr library (where your user
noted the problem).
We chose to allow options other than amalgamation because we'd heard
feedback, "let the sysadmins upgrade sqlite independently on their
system", rather than pinning svn to whatever-we-shipped. IOW, if we
ship an amalgamation, then it becomes *our problem* when bugs appear
in sqlite, and we need to spin up a new release.
Your thoughts are most welcome! SQLite is (now) a key technology
within Subversion. We'd certainly like to be a Good User.
For your user with the problem, please redirect them to
users@subversion.tigris.org, and we'll help them out.
Cheers,
-g
On Mon, May 4, 2009 at 15:56, D. Richard Hipp <dr...@hwaci.com> wrote:
> We received the following post on the SQLite mailing list today:
>
> http://article.gmane.org/gmane.comp.db.sqlite.general/46505
>
> I have not looked at the SVN code, but it would appear that in order
> to compile SVN one must first install SQLite libraries on the build
> system. In other words, it appears that SVN has a dependency SQLite.
>
> May I suggest you do away with this dependency. Grab a copy of the
> SQLite amalgamation source file and its header (sqlite3.c and
> sqlite3.h) for whatever version of SQLite works for you, and check in
> those files as part of your own source tree. Adjust your makefiles so
> that they compile the built-in sqlite3.c instead of trying to link
> against a library (which may or may not be available on the system.)
> This will make it that much easier for people to compile SVN.
>
> Upgrading is easy. When you decide you want to use the latest version
> of SQLite (because it is faster or has some new feature you want)
> simple download the latest sqlite3.c and sqlite3.h from the SQLite
> website, drop them into your own source tree, and recompile.
>
> This will also make SVN more resistant to breakage if somebody updates
> the system SQLite library - since SVN will not depend on the system
> SQLite library but will have its own. We work hard to keep SQLite
> fully compatible from one release to the next. But sometimes a bug
> fix in SQLite will break systems that depend on the old buggy
> behavior. So it never hurts to stick with a specific version of
> SQLite unless you are 100% sure that your use of SQLite never depends
> on any undefined or ill-defined behavior.
>
> PHP, Mozilla, Python, Perl, Fossil, Tcl, and many other projects
> manage SQLite this way. All commercial projects that we are aware of
> use SQLite this way. The SQLite library is not that big. It will not
> bloat your source tree. But it will tend to make your users happier.
>
> D. Richard Hipp
> drh@sqlite.org
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2056719
>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2056828