You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@corinthia.apache.org by jan i <ja...@apache.org> on 2015/03/23 09:29:14 UTC
libxml2 and iconv question, do we really need iconv ?
Hi.
I am nearly finished with the x64 port for windows, I have compiled all
libraries we need.
I wanted to see if we could keep the library sources in our Repo, and
integrate them in our build
system, as it would simplify things. Since I am biased I asked on
general@i.a.o to get the opinion
of outside Mentors. I got a few responses (strangely enough all in
private), and it turns out many projects do as I suggest.
However one of the others mentors found a license problem with iconv. We do
not use iconv, but by default libxml2 does. I can configure libxml2 not to
include iconv, but I wanted to ask in here, if we need the functionality ?
If not my plan has changed a bit, and I will include the sources in
platform, make a single commit with each original source, noting the URL
and version in the commit text (and add it to LICENSE). Then I will add my
changes, which will be solely CMAKE files NO source changes.
Having the source in our build system, will also allow us to decide how to
handle other platforms.
Rgds
jan I.
Re: libxml2 and iconv question, do we really need iconv ?
Posted by jan i <ja...@apache.org>.
On 23 March 2015 at 15:03, Peter Kelly <ke...@gmail.com> wrote:
> I have about 70% of an XML parser done that I haven’t gotten around to
> finishing yet. Once that’s completed, we don’t need libxml. I can’t
> remember if we need iconv; it’s used for character set conversions but
> DocFormats is UTF-8 native (as is libxml), and support for UTF-16 would not
> be hard (if it’s even needed).
>
> I would strongly argue against keeping sources of libraries in our
> repositories except in the case where we have made changes to those
> libraries. If we do that we have to include all of SDL_image and thus all
> of SDL, and that leads to quite a big repository. The same goes for libz
> etc. Most projects have lots and lots of dependencies and are handled just
> fine by standard package dependency systems e.g. apt-get.
>
I tried to add it to see how much it grows, and yes it is getting bigger.
We do need to modify:
- libz, because there are no 64bit windows version around.
- libxml, because it depends on iconv which has a license apache does not
like (LGL), and because there are no 64bit windows version around
- SDL and SDL_Image we could do without, because the deliver both x32 and
x64.
Btw SDL and SDL_Image why do we need them ?
Can we agree to keep them there until the 64bit version is ready, then we
know how much has really been changed ? If you disagree then I will move it
out again.
>
> Furthermore, we want to use the system libxml where available, both to
> take advantage of shared libraries (libxml only needs to exist in memory
> once, the OS maps it into the address space of each process that uses it),
> and for security updates (system libxml updated due to vulnerability,
> programs using DocFormats are still vulnerable until we go and update our
> own version).
>
I have to ask the license experts again, but I fear when we use the
standard libxml, it includes iconv and I am not sure how that affect us.
(but anyhow if you are nearly finished, that aspect will go away).
You are right about shared libraries, but it makes it complicated on e.g.
windows to test both 32bit and 64bit.
Franz@ since Iconv is LGPL we cannot depend on it.
Is it all about getImageDimension() ? if so it might be easier to simply
write that function.
I am sorry, the new desktop editor, seems to draw bigger circles than I
thought. But the good thing is we get a lot of future problems solved.
rgds
jan I
>
> --
> Dr. Peter M. Kelly
> kellypmk@gmail.com
> http://www.kellypmk.net/
>
> PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
> (fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
>
> > On 23 Mar 2015, at 3:29 pm, jan i <ja...@apache.org> wrote:
> >
> > Hi.
> >
> > I am nearly finished with the x64 port for windows, I have compiled all
> > libraries we need.
> >
> > I wanted to see if we could keep the library sources in our Repo, and
> > integrate them in our build
> > system, as it would simplify things. Since I am biased I asked on
> > general@i.a.o to get the opinion
> > of outside Mentors. I got a few responses (strangely enough all in
> > private), and it turns out many projects do as I suggest.
> >
> > However one of the others mentors found a license problem with iconv. We
> do
> > not use iconv, but by default libxml2 does. I can configure libxml2 not
> to
> > include iconv, but I wanted to ask in here, if we need the functionality
> ?
> >
> > If not my plan has changed a bit, and I will include the sources in
> > platform, make a single commit with each original source, noting the URL
> > and version in the commit text (and add it to LICENSE). Then I will add
> my
> > changes, which will be solely CMAKE files NO source changes.
> >
> > Having the source in our build system, will also allow us to decide how
> to
> > handle other platforms.
> >
> > Rgds
> > jan I.
>
>
Re: libxml2 and iconv question, do we really need iconv ?
Posted by Peter Kelly <ke...@gmail.com>.
I have about 70% of an XML parser done that I haven’t gotten around to finishing yet. Once that’s completed, we don’t need libxml. I can’t remember if we need iconv; it’s used for character set conversions but DocFormats is UTF-8 native (as is libxml), and support for UTF-16 would not be hard (if it’s even needed).
I would strongly argue against keeping sources of libraries in our repositories except in the case where we have made changes to those libraries. If we do that we have to include all of SDL_image and thus all of SDL, and that leads to quite a big repository. The same goes for libz etc. Most projects have lots and lots of dependencies and are handled just fine by standard package dependency systems e.g. apt-get.
Furthermore, we want to use the system libxml where available, both to take advantage of shared libraries (libxml only needs to exist in memory once, the OS maps it into the address space of each process that uses it), and for security updates (system libxml updated due to vulnerability, programs using DocFormats are still vulnerable until we go and update our own version).
--
Dr. Peter M. Kelly
kellypmk@gmail.com
http://www.kellypmk.net/
PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
(fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
> On 23 Mar 2015, at 3:29 pm, jan i <ja...@apache.org> wrote:
>
> Hi.
>
> I am nearly finished with the x64 port for windows, I have compiled all
> libraries we need.
>
> I wanted to see if we could keep the library sources in our Repo, and
> integrate them in our build
> system, as it would simplify things. Since I am biased I asked on
> general@i.a.o to get the opinion
> of outside Mentors. I got a few responses (strangely enough all in
> private), and it turns out many projects do as I suggest.
>
> However one of the others mentors found a license problem with iconv. We do
> not use iconv, but by default libxml2 does. I can configure libxml2 not to
> include iconv, but I wanted to ask in here, if we need the functionality ?
>
> If not my plan has changed a bit, and I will include the sources in
> platform, make a single commit with each original source, noting the URL
> and version in the commit text (and add it to LICENSE). Then I will add my
> changes, which will be solely CMAKE files NO source changes.
>
> Having the source in our build system, will also allow us to decide how to
> handle other platforms.
>
> Rgds
> jan I.
Re: libxml2 and iconv question, do we really need iconv ?
Posted by Peter Kelly <pm...@apache.org>.
> On 23 Mar 2015, at 9:04 pm, Peter Kelly <pm...@apache.org> wrote:
>
> Furthermore, we want to use the system libxml where available, both to take advantage of shared libraries (libxml only needs to exist in memory once, the OS maps it into the address space of each process that uses it), and for security updates (system libxml updated due to vulnerability, programs using DocFormats are still vulnerable until we go and update our own version).
For reference, here’s a list of security vulnerabilities that have been discovered in libxml over the years:
http://www.cvedetails.com/vulnerability-list/vendor_id-1962/product_id-3311/Xmlsoft-Libxml2.html
On a standard Linux setup where libxml is a 3rd-party package, all that’s required when one of these are discovered is an upgrade of that single package.
If we keep 3rd-party sources in the repository, then every vulnerability in every library we use suddenly becomes a vulnerability in Corinthia as well, and we have to track these and issue a new version whenever one of the libraries are patched.
If we were to ever include OpenSSL as a dependency - as *many* projects do (and we might, e.g. to cater for encryption in OOXML documents), this would be an even more serious problem. I’ve lost count of the number of vulnerabilities that have been patched in OpenSSL over just over the past year.
—
Dr Peter M. Kelly
pmkelly@apache.org
PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
(fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
Re: libxml2 and iconv question, do we really need iconv ?
Posted by jan i <ja...@apache.org>.
On 23 March 2015 at 15:31, Peter Kelly <pm...@apache.org> wrote:
> Also I just realised that SDL2, SDL2_image, and even zlib are in 3rd
> party. This brings the total size of our source tree to 165 mb. Before that
> it was 41 Mb. Could we *please* not have these in there?
>
> Dennis raised the point previously about it being a bad idea to provide
> our own binary distributions of these libraries - a point I disagreed with
> (I think it is useful to do so, for convenience purposes). But I definitely
> think the sources to other projects do not belong in our own repository.
> The issues of size (medium) and responsibility for updating after security
> vulnerabilities are found in these (major) are the main reasons for my
> objection, but another minor one is that these are not Apache licensed.
>
As I just wrote in another mail, we need nearly all of them to be able to
do the 64bit for windows.
A lot of projects do like I just did, and often for the same purpose to be
able to compile the library so it fits the need of the project.
The only alternative I can see, is that I compile them and make them
available as zip files, that reduces the size of everybody else repo, but
we still have update problems etc. Looking at the revision history of these
libraries, they do not exactly have many updates.
Can anyone suggest another alternative:
I need 32bit and 64bit libs (prefer no-shared, because shared is gives dll
conflicts when testing).
We need both a release version and a debug version (to avoid linker
warnings).
We also need a way to avoid dependency on iconv.
rgds
jan i.
>
> —
> Dr Peter M. Kelly
> pmkelly@apache.org
>
> PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
> (fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
>
> > On 23 Mar 2015, at 9:04 pm, Peter Kelly <pm...@apache.org> wrote:
> >
> > (apologies if this is duplicated; I sent from the wrong address before)
> >
> > I have about 70% of an XML parser done that I haven’t gotten around to
> finishing yet. Once that’s completed, we don’t need libxml. I can’t
> remember if we need iconv; it’s used for character set conversions but
> DocFormats is UTF-8 native (as is libxml), and support for UTF-16 would not
> be hard (if it’s even needed).
> >
> > I would strongly argue against keeping sources of libraries in our
> repositories except in the case where we have made changes to those
> libraries. If we do that we have to include all of SDL_image and thus all
> of SDL, and that leads to quite a big repository. The same goes for libz
> etc. Most projects have lots and lots of dependencies and are handled just
> fine by standard package dependency systems e.g. apt-get.
> >
> > Furthermore, we want to use the system libxml where available, both to
> take advantage of shared libraries (libxml only needs to exist in memory
> once, the OS maps it into the address space of each process that uses it),
> and for security updates (system libxml updated due to vulnerability,
> programs using DocFormats are still vulnerable until we go and update our
> own version).
> >
> > —
> > Dr Peter M. Kelly
> > pmkelly@apache.org
> >
> > PGP key: http://www.kellypmk.net/pgp-key <
> http://www.kellypmk.net/pgp-key>
> > (fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
> >
> >> On 23 Mar 2015, at 3:29 pm, jan i <ja...@apache.org> wrote:
> >>
> >> Hi.
> >>
> >> I am nearly finished with the x64 port for windows, I have compiled all
> >> libraries we need.
> >>
> >> I wanted to see if we could keep the library sources in our Repo, and
> >> integrate them in our build
> >> system, as it would simplify things. Since I am biased I asked on
> >> general@i.a.o to get the opinion
> >> of outside Mentors. I got a few responses (strangely enough all in
> >> private), and it turns out many projects do as I suggest.
> >>
> >> However one of the others mentors found a license problem with iconv.
> We do
> >> not use iconv, but by default libxml2 does. I can configure libxml2 not
> to
> >> include iconv, but I wanted to ask in here, if we need the
> functionality ?
> >>
> >> If not my plan has changed a bit, and I will include the sources in
> >> platform, make a single commit with each original source, noting the URL
> >> and version in the commit text (and add it to LICENSE). Then I will add
> my
> >> changes, which will be solely CMAKE files NO source changes.
> >>
> >> Having the source in our build system, will also allow us to decide how
> to
> >> handle other platforms.
> >>
> >> Rgds
> >> jan I.
> >
>
>
Re: libxml2 and iconv question, do we really need iconv ?
Posted by Peter Kelly <pm...@apache.org>.
Also I just realised that SDL2, SDL2_image, and even zlib are in 3rd party. This brings the total size of our source tree to 165 mb. Before that it was 41 Mb. Could we *please* not have these in there?
Dennis raised the point previously about it being a bad idea to provide our own binary distributions of these libraries - a point I disagreed with (I think it is useful to do so, for convenience purposes). But I definitely think the sources to other projects do not belong in our own repository. The issues of size (medium) and responsibility for updating after security vulnerabilities are found in these (major) are the main reasons for my objection, but another minor one is that these are not Apache licensed.
—
Dr Peter M. Kelly
pmkelly@apache.org
PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
(fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
> On 23 Mar 2015, at 9:04 pm, Peter Kelly <pm...@apache.org> wrote:
>
> (apologies if this is duplicated; I sent from the wrong address before)
>
> I have about 70% of an XML parser done that I haven’t gotten around to finishing yet. Once that’s completed, we don’t need libxml. I can’t remember if we need iconv; it’s used for character set conversions but DocFormats is UTF-8 native (as is libxml), and support for UTF-16 would not be hard (if it’s even needed).
>
> I would strongly argue against keeping sources of libraries in our repositories except in the case where we have made changes to those libraries. If we do that we have to include all of SDL_image and thus all of SDL, and that leads to quite a big repository. The same goes for libz etc. Most projects have lots and lots of dependencies and are handled just fine by standard package dependency systems e.g. apt-get.
>
> Furthermore, we want to use the system libxml where available, both to take advantage of shared libraries (libxml only needs to exist in memory once, the OS maps it into the address space of each process that uses it), and for security updates (system libxml updated due to vulnerability, programs using DocFormats are still vulnerable until we go and update our own version).
>
> —
> Dr Peter M. Kelly
> pmkelly@apache.org
>
> PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
> (fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
>
>> On 23 Mar 2015, at 3:29 pm, jan i <ja...@apache.org> wrote:
>>
>> Hi.
>>
>> I am nearly finished with the x64 port for windows, I have compiled all
>> libraries we need.
>>
>> I wanted to see if we could keep the library sources in our Repo, and
>> integrate them in our build
>> system, as it would simplify things. Since I am biased I asked on
>> general@i.a.o to get the opinion
>> of outside Mentors. I got a few responses (strangely enough all in
>> private), and it turns out many projects do as I suggest.
>>
>> However one of the others mentors found a license problem with iconv. We do
>> not use iconv, but by default libxml2 does. I can configure libxml2 not to
>> include iconv, but I wanted to ask in here, if we need the functionality ?
>>
>> If not my plan has changed a bit, and I will include the sources in
>> platform, make a single commit with each original source, noting the URL
>> and version in the commit text (and add it to LICENSE). Then I will add my
>> changes, which will be solely CMAKE files NO source changes.
>>
>> Having the source in our build system, will also allow us to decide how to
>> handle other platforms.
>>
>> Rgds
>> jan I.
>
Re: libxml2 and iconv question, do we really need iconv ?
Posted by Peter Kelly <pm...@apache.org>.
(apologies if this is duplicated; I sent from the wrong address before)
I have about 70% of an XML parser done that I haven’t gotten around to finishing yet. Once that’s completed, we don’t need libxml. I can’t remember if we need iconv; it’s used for character set conversions but DocFormats is UTF-8 native (as is libxml), and support for UTF-16 would not be hard (if it’s even needed).
I would strongly argue against keeping sources of libraries in our repositories except in the case where we have made changes to those libraries. If we do that we have to include all of SDL_image and thus all of SDL, and that leads to quite a big repository. The same goes for libz etc. Most projects have lots and lots of dependencies and are handled just fine by standard package dependency systems e.g. apt-get.
Furthermore, we want to use the system libxml where available, both to take advantage of shared libraries (libxml only needs to exist in memory once, the OS maps it into the address space of each process that uses it), and for security updates (system libxml updated due to vulnerability, programs using DocFormats are still vulnerable until we go and update our own version).
—
Dr Peter M. Kelly
pmkelly@apache.org
PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
(fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
> On 23 Mar 2015, at 3:29 pm, jan i <ja...@apache.org> wrote:
>
> Hi.
>
> I am nearly finished with the x64 port for windows, I have compiled all
> libraries we need.
>
> I wanted to see if we could keep the library sources in our Repo, and
> integrate them in our build
> system, as it would simplify things. Since I am biased I asked on
> general@i.a.o to get the opinion
> of outside Mentors. I got a few responses (strangely enough all in
> private), and it turns out many projects do as I suggest.
>
> However one of the others mentors found a license problem with iconv. We do
> not use iconv, but by default libxml2 does. I can configure libxml2 not to
> include iconv, but I wanted to ask in here, if we need the functionality ?
>
> If not my plan has changed a bit, and I will include the sources in
> platform, make a single commit with each original source, noting the URL
> and version in the commit text (and add it to LICENSE). Then I will add my
> changes, which will be solely CMAKE files NO source changes.
>
> Having the source in our build system, will also allow us to decide how to
> handle other platforms.
>
> Rgds
> jan I.
RE: libxml2 and iconv question, do we really need iconv ?
Posted by Franz de Copenhague <fr...@outlook.com>.
> -----Original Message-----
> iconv deals with character set conversion, not images; SDL_image does that.
>
My fault, I was confuse with a previous email talking about iconv and image resolutions.
> Note of course that dfwebserver could have dependencies on SDL or
> another library (e.g. ImageMagick - either linked against it or calling out to
> separate processes). Library functionality used only by dfwebserver doesn’t
> necessarily have to be depended on by DocFormats itself.
So far, DocFormats is responsible to unzip image resources in a GET call. It makes me think that DocFormats can rescale the images as well.
Franz
Re: libxml2 and iconv question, do we really need iconv ?
Posted by jan i <ja...@apache.org>.
On 23 March 2015 at 15:08, Peter Kelly <pm...@apache.org> wrote:
> > On 23 Mar 2015, at 7:13 pm, Franz de Copenhague <
> franzdecopenhague@outlook.com> wrote:
> >
> > dfwebserver will need iconv or similar functionality to reduce the
> image resources resolution in documents to improve download performance in
> the client editor.
>
> iconv deals with character set conversion, not images; SDL_image does that.
>
But do we really need character set conversation ?
DocFormat does not use it directly, it is only linked to libxml2.
If I were to redo getImageDimension() we could remove SDL_image and SDL ?
>
> Currently we have SDL_image treated as a platform-dependent library, since
> we’re only using it for one purpose, which is to get the width and height
> of an image, and that is handled by the system-supplied ImageIO framework
> on OS X/iOS (I suspect there’s something similar on Windows as well). For
> image resizing, if that’s the only extra thing we need to do with images,
> we could probably implement both an SDL_image and ImageIO version of that
> as well. If we’re going to be doing more extensive image manipulation, I
> think it would be better to rely on one library (that is, use SDL_image on
> Apple platforms as well as others).
>
> Note of course that dfwebserver could have dependencies on SDL or another
> library (e.g. ImageMagick - either linked against it or calling out to
> separate processes). Library functionality used only by dfwebserver doesn’t
> necessarily have to be depended on by DocFormats itself.
>
+1
rgds
jan i
>
> —
> Dr Peter M. Kelly
> pmkelly@apache.org
>
> PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
> (fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
>
>
Re: libxml2 and iconv question, do we really need iconv ?
Posted by Peter Kelly <pm...@apache.org>.
> On 23 Mar 2015, at 7:13 pm, Franz de Copenhague <fr...@outlook.com> wrote:
>
> dfwebserver will need iconv or similar functionality to reduce the image resources resolution in documents to improve download performance in the client editor.
iconv deals with character set conversion, not images; SDL_image does that.
Currently we have SDL_image treated as a platform-dependent library, since we’re only using it for one purpose, which is to get the width and height of an image, and that is handled by the system-supplied ImageIO framework on OS X/iOS (I suspect there’s something similar on Windows as well). For image resizing, if that’s the only extra thing we need to do with images, we could probably implement both an SDL_image and ImageIO version of that as well. If we’re going to be doing more extensive image manipulation, I think it would be better to rely on one library (that is, use SDL_image on Apple platforms as well as others).
Note of course that dfwebserver could have dependencies on SDL or another library (e.g. ImageMagick - either linked against it or calling out to separate processes). Library functionality used only by dfwebserver doesn’t necessarily have to be depended on by DocFormats itself.
—
Dr Peter M. Kelly
pmkelly@apache.org
PGP key: http://www.kellypmk.net/pgp-key <http://www.kellypmk.net/pgp-key>
(fingerprint 5435 6718 59F0 DD1F BFA0 5E46 2523 BAA1 44AE 2966)
RE: libxml2 and iconv question, do we really need iconv ?
Posted by Franz de Copenhague <fr...@outlook.com>.
> -----Original Message-----
> From: jan i [mailto:jani@apache.org]
> Sent: Monday, March 23, 2015 4:29 AM
> To: dev@corinthia.incubator.apache.org
> Subject: libxml2 and iconv question, do we really need iconv ?
>
> Hi.
>
> I am nearly finished with the x64 port for windows, I have compiled all
> libraries we need.
>
> I wanted to see if we could keep the library sources in our Repo, and
> integrate them in our build system, as it would simplify things. Since I am
> biased I asked on general@i.a.o to get the opinion of outside Mentors. I got a
> few responses (strangely enough all in private), and it turns out many
> projects do as I suggest.
>
> However one of the others mentors found a license problem with iconv. We
> do not use iconv, but by default libxml2 does. I can configure libxml2 not to
> include iconv, but I wanted to ask in here, if we need the functionality ?
>
> If not my plan has changed a bit, and I will include the sources in platform,
> make a single commit with each original source, noting the URL and version in
> the commit text (and add it to LICENSE). Then I will add my changes, which
> will be solely CMAKE files NO source changes.
>
> Having the source in our build system, will also allow us to decide how to
> handle other platforms.
>
> Rgds
> jan I.
dfwebserver will need iconv or similar functionality to reduce the image resources resolution in documents to improve download performance in the client editor.
franz