PCRE 10 and puzzling edge cases

[William A Rowe Jr <wr...@rowe-clan.net>]

I've written the following patch to trunk to allow us to configure, compile
and link against PCRE2 (10.x). The autoconf in particular is streamlined
for cross-compilation detection, while retaining the ability to override
the path to (and name of) pcre[2]-config.

It isn't in a commit-ready state due to t/TEST t/apache/expr.t failures
(among others), and the defects appear to revolve around the way substring
patterns are recorded.

Attached the test failure cases (many similar test patterns do succeed,
interestingly.) One test looks outright wrong. I'd rather not beat my head
against these if the answer is blatantly obvious.

If anyone has patience for exploring this further, any help is welcomed.
Philip starts with this assertion; "The original, very widely deployed PCRE
library, originally released in 1997, is at version 8.39, and the API and
feature set are stable—future releases will be for bugfixes only. All new
future features will be to PCRE2, not the original PCRE 8.x series." But he
has gone on to state that many fuzzing error cases which are handled
correctly in PCRE2 cannot be realistically fixed in PCRE 8.x. I've placed
this up there with other parsing rewrites in httpd, that starting over is
simply the correct answer, and I'd like to see if we can have httpd 3.0
choosing PCRE2 over PCRE in the near future (and perhaps backport this if
we determine behavior is consistent.)

Cheers,

Bill

Re: PCRE 10 and puzzling edge cases

[Reindl Harald <h....@thelounge.net>]

Am 12.12.2016 um 10:52 schrieb Petr Pisar:
> I made sure I have installed all Perl modules I found relevant, but I was
> unable to run the tests against SVN httpd sources. I played with
> LD_LIBRARY_PATH, apxs etc. but without any good result. At the end
> I reconfigured httpd sources and installed the binaries into a /tmp
> subdirectory and that allowed me to run the tests. It's quite annoying to have
> to install httpd into the system just only to test it

since your email is @redhat.com take a look at the Fedora php.spec and 
how the testsuite is called there from the rpm builddir *before* make 
install - same for mariadb/mysql - i guess httpd wouldn't bee that different

Re: PCRE 10 and puzzling edge cases

[Petr Pisar <pp...@redhat.com>]

On Fri, Dec 09, 2016 at 01:07:54PM -0600, William A Rowe Jr wrote:
> 
> Thanks again, if there is anything that we didn't document well about
> getting the tests to run, we would like to fix that and make it easier for
> developers to cobble together their own test environment. We certainly don't
> want it to take hours for an experienced newcomer to get from point a to
> point b.
> 
First problem was how to generate ./configure from SVN sources. Obvious
autoreconf or autoconf did not work because of some missing macros. Then
I found ./buildconf.

But it still failed because of missing APR sources despite I had installed
APR 1.5.4 including header files and M4 macros from my distribution packages.
Then I found in the documentation that building from SVN requires APR sources.
So I cloned APR sources. Then I was able to build httpd.

I wanted to run test, but there are no tests you complained about a failure.
After searching web, I got an idea that httpd-framework is what I need.

I made sure I have installed all Perl modules I found relevant, but I was
unable to run the tests against SVN httpd sources. I played with
LD_LIBRARY_PATH, apxs etc. but without any good result. At the end
I reconfigured httpd sources and installed the binaries into a /tmp
subdirectory and that allowed me to run the tests. It's quite annoying to have
to install httpd into the system just only to test it.

But still the tests failed because of an undefined symbol in a session test
module. The symbol was defined in the httpd session module but the session
module was not named by the LoadModule keyword in the generated
t/conf/httpd.conf. I did not found a way how to disable the session tests
other than removing c-modules/test_session/mod_test_session.c file.

Finally I obtained a pass.

I think the inability to build httpd against system APR and to run test
against not yet installed httpd is quite surprising.

-- Petr

Re: PCRE 10 and puzzling edge cases

[William A Rowe Jr <wr...@rowe-clan.net>]

On Fri, Dec 9, 2016 at 8:05 AM, Petr Pisar <pp...@redhat.com> wrote:

> On Thu, Dec 08, 2016 at 11:09:42AM -0600, William A Rowe Jr wrote:
> > I've beaten my head against this wall for a bit longer, and came up with
> > several places where pcre2 changed return types for void *what query
> > interogations (especially from int to uint32, badness on x86_64-linux).
> >
> > The attached patch picks up these bad void * type assignments. Still
> > no tremendous improvement, missing something blatantly obvious,
> > I expect.
> >
> After few hours of getting httpd sources and tests and hacking them to
> actually obtain a pass, I applied your patch and looked what's wrong with
> your
> PCRE2 code.
>

Thanks again, if there is anything that we didn't document well about
getting
the tests to run, we would like to fix that and make it easier for
developers
to cobble together their own test environment. We certainly don't want it
to take hours for an experienced newcomer to get from point a to point b.

Happy to improve this based on your observations.

The t/apache/expr.t failed because the pcre2_match() alwayes returned -51
> that means PCRE2_ERROR_NULL. The reason is you used the old PCRE
> optimization
> path and called pcre2_match_data_create(nmatch, NULL) only if nmatch was
> positive. As a result, pcre2_match_data_create() was never called, so you
> passed NULL matchdata to pcre2_match(), hence the failure.
>

Yup, that was a bad assumption on my part, reading pcre2api in parallel
with pcreapi.


> The tests still do not pass, but that's probably another (PCRE2) problem.
> I hope I helped you at lest somehow.
>

I do have it working, now committed. Failing tests are likely some missing
cpan modules in your setup still, or perhaps some tests that are making
bogus assumptions about the supported dependency libraries. Thanks for
the pointer, we seem to have succeeded!

Re: PCRE 10 and puzzling edge cases

[William A Rowe Jr <wr...@rowe-clan.net>]

On Dec 9, 2016 8:06 AM, "Petr Pisar" <pp...@redhat.com> wrote:

On Thu, Dec 08, 2016 at 11:09:42AM -0600, William A Rowe Jr wrote:
> I've beaten my head against this wall for a bit longer, and came up with
> several places where pcre2 changed return types for void *what query
> interogations (especially from int to uint32, badness on x86_64-linux).
>
> The attached patch picks up these bad void * type assignments. Still
> no tremendous improvement, missing something blatantly obvious,
> I expect.
>
After few hours of getting httpd sources and tests and hacking them to
actually obtain a pass, I applied your patch and looked what's wrong with
your
PCRE2 code.

The t/apache/expr.t failed because the pcre2_match() alwayes returned -51
that means PCRE2_ERROR_NULL. The reason is you used the old PCRE
optimization
path and called pcre2_match_data_create(nmatch, NULL) only if nmatch was
positive. As a result, pcre2_match_data_create() was never called, so you
passed NULL matchdata to pcre2_match(), hence the failure.

See the attached fix.

The tests still do not pass, but that's probably another (PCRE2) problem.
I hope I helped you at lest somehow


That's a huge help, thanks Petr.

I'm also curious along that path if adding PCRE2 (or PCRE) study pattern
would also be helpful. Something to experiment with once both code paths
are working.

Cheers,

Bill

Re: PCRE 10 and puzzling edge cases

[Petr Pisar <pp...@redhat.com>]

On Thu, Dec 08, 2016 at 11:09:42AM -0600, William A Rowe Jr wrote:
> I've beaten my head against this wall for a bit longer, and came up with
> several places where pcre2 changed return types for void *what query
> interogations (especially from int to uint32, badness on x86_64-linux).
> 
> The attached patch picks up these bad void * type assignments. Still
> no tremendous improvement, missing something blatantly obvious,
> I expect.
> 
After few hours of getting httpd sources and tests and hacking them to
actually obtain a pass, I applied your patch and looked what's wrong with your
PCRE2 code.

The t/apache/expr.t failed because the pcre2_match() alwayes returned -51
that means PCRE2_ERROR_NULL. The reason is you used the old PCRE optimization
path and called pcre2_match_data_create(nmatch, NULL) only if nmatch was
positive. As a result, pcre2_match_data_create() was never called, so you
passed NULL matchdata to pcre2_match(), hence the failure.

See the attached fix.

The tests still do not pass, but that's probably another (PCRE2) problem.
I hope I helped you at lest somehow.

-- Petr

Re: PCRE 10 and puzzling edge cases

[William A Rowe Jr <wr...@rowe-clan.net>]

I've beaten my head against this wall for a bit longer, and came up with
several places where pcre2 changed return types for void *what query
interogations (especially from int to uint32, badness on x86_64-linux).

The attached patch picks up these bad void * type assignments. Still
no tremendous improvement, missing something blatantly obvious,
I expect.



On Mon, Dec 5, 2016 at 10:59 PM, William A Rowe Jr <wr...@rowe-clan.net>
wrote:

> I've written the following patch to trunk to allow us to configure,
> compile and link against PCRE2 (10.x). The autoconf in particular is
> streamlined for cross-compilation detection, while retaining the ability to
> override the path to (and name of) pcre[2]-config.
>
> It isn't in a commit-ready state due to t/TEST t/apache/expr.t failures
> (among others), and the defects appear to revolve around the way substring
> patterns are recorded.
>
> Attached the test failure cases (many similar test patterns do succeed,
> interestingly.) One test looks outright wrong. I'd rather not beat my head
> against these if the answer is blatantly obvious.
>
> If anyone has patience for exploring this further, any help is welcomed.
> Philip starts with this assertion; "The original, very widely deployed PCRE
> library, originally released in 1997, is at version 8.39, and the API and
> feature set are stable—future releases will be for bugfixes only. All new
> future features will be to PCRE2, not the original PCRE 8.x series." But he
> has gone on to state that many fuzzing error cases which are handled
> correctly in PCRE2 cannot be realistically fixed in PCRE 8.x. I've placed
> this up there with other parsing rewrites in httpd, that starting over is
> simply the correct answer, and I'd like to see if we can have httpd 3.0
> choosing PCRE2 over PCRE in the near future (and perhaps backport this if
> we determine behavior is consistent.)
>
> Cheers,
>
> Bill
>
>