You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomee.apache.org by Doychin Bondzhev <do...@dsoft-bg.com> on 2019/01/06 10:25:16 UTC
Investigating https://issues.apache.org/jira/browse/TOMEE-2403
Hi,
I would like to investigate this issue:
https://issues.apache.org/jira/browse/TOMEE-2403
Where I can get more detailed error log regarding this issue?
I tried to run the test locally but it works.
--
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116
Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly
Posted by Carl Mosca <ca...@gmail.com>.
Pretty cool technique... never have done mvn in a loop like that...but I
think I will. ☺️
On Sun, Jan 6, 2019, 12:47 PM David Blevins <david.blevins@gmail.com wrote:
> > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <do...@dsoft-bg.com>
> wrote:
> >
> > I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
> >
> > Where I can get more detailed error log regarding this issue?
> >
> > I tried to run the test locally but it works.
>
> Here's a technique I use sometimes to check random failures:
>
> $ for n in {1..50}; do mvn clean install
> -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
>
> So far 3 out 17 have failed. Debugging them will still be a pain in the
> butt, but at least there's some hope.
>
>
> My shallow observation is that the approach taken in the test itself looks
> conspicuous. Particularly, I'd say this test is not "correct" until it can
> pass without the System.gc() statements.
>
> -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
>
> More importantly, this method is using thread sleeps and retries to wait
> for log output which *will* cause random failures. We should have a
> discussion on if there is a better way to make this more deterministic.
>
> -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
>
> Testing log output can work, however the test logic is ultimately "if you
> do or don't see this message in n seconds, we're good." It needs to be "if
> you do or don't see this message *when the test is over*, we're good."
>
> Since it's looking in log output, one way to do that might be to simply
> log a "test complete" statement (perhaps with a random number on it), and
> then wait for that in the log output.
>
>
> -David
>
>
>
>
>
Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly
Posted by Jonathan Gallimore <jo...@gmail.com>.
Thanks. I'll set the rig up again and run some volume through with these
changes.
Jon
On Mon, Jan 7, 2019 at 8:46 PM Doychin Bondzhev <do...@dsoft-bg.com>
wrote:
> Hi Jon,
>
> Please check this PR:
> https://github.com/apache/tomee/pull/360
>
> We can discuss the details there.
>
> I also found this bug report which is still open:
> https://issues.apache.org/jira/browse/TOMEE-2257
>
> Is that the problem you describe?
>
> On 7.1.2019 г. 18:51, jgallimore [via TomEE & OpenEJB] wrote:
>
> This test was a pain to write, and I am grateful for the feedback. Please
> do send a PR and I'll take a look.
>
> Some background to the problem here is that the the AutoConnectionTracker
> failed in some circumstances when servers were under heavy load with lots
> of GC happening - there appeared to be a timing issue where a resource
> that
> was actually in use was being forcefully closed by the auto connection
> tracker, but because something else still had a hold of it, it ended up
> going back in the pool and a broken resource being served up to the next
> thing that requested it. The requests for GC were quite key in reproducing
> the actual issue originally, so removing them might not actually be the
> right approach.
>
> It was particularly hard to reproduce and pin down and I ended up going
> through something like 40GB of heap dumps and logs to figure out what was
> happening. Previously there was no test, which made this even harder to
> pin
> down. I'm definitely happy to explore some options to make this more
> solid.
>
> Jon
>
> On Sun, Jan 6, 2019 at 8:28 PM Doychin Bondzhev <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=4687328&i=0>>
> wrote:
>
> > I use IDE to run test multiple times. It failed only once here.
> >
> > I also did some changes in the test code.
> >
> > I removed the line
> >
> >
> >
> https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162
> >
> > This made test many times faster. It is not needed. This tests non
> leaking
> > business method so we expect everything to be cleared.
> >
> > Also I removed the all the code that waits for log messages. I left only
> > the line
> >
> > assertEquals(message, times, logCapture.find(message).size());
> >
> >
> > There is no need to wait for anything. All the waiting is already done
> for
> > the first and second tests in
> >
> > runTest method
> >
> > Regarding
> >
> > System.gc();
> >
> >
> > It is needed only for the tests that leak so I removed it from all other
> > tests.
> >
> > Third and fourth test are run in the main thread and there is no need to
> > wait before processing the log to.
> > You can see the change here :
> >
> >
> https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a
> >
> > In case you like it I can create PR.
> >
> > On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
> >
> > > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote:
> > >
> > > I would like to investigate this issue:
> > https://issues.apache.org/jira/browse/TOMEE-2403
> > >
> > > Where I can get more detailed error log regarding this issue?
> > >
> > > I tried to run the test locally but it works.
> >
> > Here's a technique I use sometimes to check random failures:
> >
> > $ for n in {1..50}; do mvn clean install
> > -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
> >
> > So far 3 out 17 have failed. Debugging them will still be a pain in the
> > butt, but at least there's some hope.
> >
> >
> > My shallow observation is that the approach taken in the test itself
> looks
> > conspicuous. Particularly, I'd say this test is not "correct" until it
> can
> > pass without the System.gc() statements.
> >
> > -
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
> >
> > More importantly, this method is using thread sleeps and retries to wait
> > for log output which *will* cause random failures. We should have a
> > discussion on if there is a better way to make this more deterministic.
> >
> > -
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
> >
> > Testing log output can work, however the test logic is ultimately "if
> you
> > do or don't see this message in n seconds, we're good." It needs to be
> "if
> > you do or don't see this message *when the test is over*, we're good."
> >
> > Since it's looking in log output, one way to do that might be to simply
> > log a "test complete" statement (perhaps with a random number on it),
> and
> > then wait for that in the log output.
> >
> >
> > -David
> >
> >
> >
> >
> >
> > *signature.asc* (499 bytes) Download Attachment
> > <
> http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc>
>
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html
> > To start a new topic under TomEE Dev, email
> > [hidden email] <http:///user/SendEmail.jtp?type=node&node=4687328&i=1>
> > To unsubscribe from TomEE Dev, click here
> > <
> > .
> > NAML
> > <
> http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> >
> >
> > --
> > Doychin Bondzhev
> > dSoft-Bulgaria Ltd.
> > PowerPro - billing & provisioning solution for Service providershttp://
> www.dsoft-bg.com/
> > Mobile: +359888243116
> >
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687328.html
> To start a new topic under TomEE Dev, email
> ml+s979440n982480h80@n4.nabble.com
> To unsubscribe from TomEE Dev, click here
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=982480&code=ZG95Y2hpbkBkc29mdC1iZy5jb218OTgyNDgwfC0xNTI2MjM5MTA3>
> .
> NAML
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providershttp://www.dsoft-bg.com/
> Mobile: +359888243116
>
>
Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly
Posted by Doychin Bondzhev <do...@dsoft-bg.com>.
Hi Jon,
Please check this PR:
https://github.com/apache/tomee/pull/360
We can discuss the details there.
I also found this bug report which is still open:
https://issues.apache.org/jira/browse/TOMEE-2257
Is that the problem you describe?
On 7.1.2019 �. 18:51, jgallimore [via TomEE & OpenEJB] wrote:
> This test was a pain to write, and I am grateful for the feedback. Please
> do send a PR and I'll take a look.
>
> Some background to the problem here is that the the AutoConnectionTracker
> failed in some circumstances when servers were under heavy load with lots
> of GC happening - there appeared to be a timing issue where a resource
> that
> was actually in use was being forcefully closed by the auto connection
> tracker, but because something else still had a hold of it, it ended up
> going back in the pool and a broken resource being served up to the next
> thing that requested it. The requests for GC were quite key in
> reproducing
> the actual issue originally, so removing them might not actually be the
> right approach.
>
> It was particularly hard to reproduce and pin down and I ended up going
> through something like 40GB of heap dumps and logs to figure out what was
> happening. Previously there was no test, which made this even harder
> to pin
> down. I'm definitely happy to explore some options to make this more
> solid.
>
> Jon
>
> On Sun, Jan 6, 2019 at 8:28 PM Doychin Bondzhev <[hidden email]
> </user/SendEmail.jtp?type=node&node=4687328&i=0>>
> wrote:
>
> > I use IDE to run test multiple times. It failed only once here.
> >
> > I also did some changes in the test code.
> >
> > I removed the line
> >
> >
> >
> https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162
> >
> > This made test many times faster. It is not needed. This tests non
> leaking
> > business method so we expect everything to be cleared.
> >
> > Also I removed the all the code that waits for log messages. I left
> only
> > the line
> >
> > assertEquals(message, times, logCapture.find(message).size());
> >
> >
> > There is no need to wait for anything. All the waiting is already
> done for
> > the first and second tests in
> >
> > runTest method
> >
> > Regarding
> >
> > System.gc();
> >
> >
> > It is needed only for the tests that leak so I removed it from all
> other
> > tests.
> >
> > Third and fourth test are run in the main thread and there is no
> need to
> > wait before processing the log to.
> > You can see the change here :
> >
> >
> https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a
> >
> > In case you like it I can create PR.
> >
> > On 6.1.2019 �. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
> >
> > > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]
> > <http:///user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote:
> > >
> > > I would like to investigate this issue:
> > https://issues.apache.org/jira/browse/TOMEE-2403
> > >
> > > Where I can get more detailed error log regarding this issue?
> > >
> > > I tried to run the test locally but it works.
> >
> > Here's a technique I use sometimes to check random failures:
> >
> > � � $ for n in {1..50}; do mvn clean install
> > -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
> >
> > So far 3 out 17 have failed. �Debugging them will still be a pain in
> the
> > butt, but at least there's some hope.
> >
> >
> > My shallow observation is that the approach taken in the test itself
> looks
> > conspicuous. �Particularly, I'd say this test is not "correct" until
> it can
> > pass without the System.gc() statements.
> >
> > �-
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
> >
> > More importantly, this method is using thread sleeps and retries to
> wait
> > for log output which *will* cause random failures. �We should have a
> > discussion on if there is a better way to make this more deterministic.
> >
> > �-
> >
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
> >
> > Testing log output can work, however the test logic is ultimately
> "if you
> > do or don't see this message in n seconds, we're good." �It needs to
> be "if
> > you do or don't see this message *when the test is over*, we're good."
> >
> > Since it's looking in log output, one way to do that might be to simply
> > log a "test complete" statement (perhaps with a random number on
> it), and
> > then wait for that in the log output.
> >
> >
> > -David
> >
> >
> >
> >
> >
> > *signature.asc* (499 bytes) Download Attachment
> >
> <http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc>
>
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the
> discussion
> > below:
> >
> >
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html
> > To start a new topic under TomEE Dev, email
> > [hidden email] </user/SendEmail.jtp?type=node&node=4687328&i=1>
> > To unsubscribe from TomEE Dev, click here
> > <
> > .
> > NAML
> >
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> >
> >
> > --
> > Doychin Bondzhev
> > dSoft-Bulgaria Ltd.
> > PowerPro - billing & provisioning solution for Service
> providershttp://www.dsoft-bg.com/
> > Mobile: +359888243116
> >
> >
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687328.html
>
> To start a new topic under TomEE Dev, email
> ml+s979440n982480h80@n4.nabble.com
> To unsubscribe from TomEE Dev, click here
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=982480&code=ZG95Y2hpbkBkc29mdC1iZy5jb218OTgyNDgwfC0xNTI2MjM5MTA3>.
> NAML
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116
Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly
Posted by Jonathan Gallimore <jo...@gmail.com>.
This test was a pain to write, and I am grateful for the feedback. Please
do send a PR and I'll take a look.
Some background to the problem here is that the the AutoConnectionTracker
failed in some circumstances when servers were under heavy load with lots
of GC happening - there appeared to be a timing issue where a resource that
was actually in use was being forcefully closed by the auto connection
tracker, but because something else still had a hold of it, it ended up
going back in the pool and a broken resource being served up to the next
thing that requested it. The requests for GC were quite key in reproducing
the actual issue originally, so removing them might not actually be the
right approach.
It was particularly hard to reproduce and pin down and I ended up going
through something like 40GB of heap dumps and logs to figure out what was
happening. Previously there was no test, which made this even harder to pin
down. I'm definitely happy to explore some options to make this more solid.
Jon
On Sun, Jan 6, 2019 at 8:28 PM Doychin Bondzhev <do...@dsoft-bg.com>
wrote:
> I use IDE to run test multiple times. It failed only once here.
>
> I also did some changes in the test code.
>
> I removed the line
>
>
> https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162
>
> This made test many times faster. It is not needed. This tests non leaking
> business method so we expect everything to be cleared.
>
> Also I removed the all the code that waits for log messages. I left only
> the line
>
> assertEquals(message, times, logCapture.find(message).size());
>
>
> There is no need to wait for anything. All the waiting is already done for
> the first and second tests in
>
> runTest method
>
> Regarding
>
> System.gc();
>
>
> It is needed only for the tests that leak so I removed it from all other
> tests.
>
> Third and fourth test are run in the main thread and there is no need to
> wait before processing the log to.
> You can see the change here :
>
> https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a
>
> In case you like it I can create PR.
>
> On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
>
> > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote:
> >
> > I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
> >
> > Where I can get more detailed error log regarding this issue?
> >
> > I tried to run the test locally but it works.
>
> Here's a technique I use sometimes to check random failures:
>
> $ for n in {1..50}; do mvn clean install
> -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
>
> So far 3 out 17 have failed. Debugging them will still be a pain in the
> butt, but at least there's some hope.
>
>
> My shallow observation is that the approach taken in the test itself looks
> conspicuous. Particularly, I'd say this test is not "correct" until it can
> pass without the System.gc() statements.
>
> -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
>
> More importantly, this method is using thread sleeps and retries to wait
> for log output which *will* cause random failures. We should have a
> discussion on if there is a better way to make this more deterministic.
>
> -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
>
> Testing log output can work, however the test logic is ultimately "if you
> do or don't see this message in n seconds, we're good." It needs to be "if
> you do or don't see this message *when the test is over*, we're good."
>
> Since it's looking in log output, one way to do that might be to simply
> log a "test complete" statement (perhaps with a random number on it), and
> then wait for that in the log output.
>
>
> -David
>
>
>
>
>
> *signature.asc* (499 bytes) Download Attachment
> <http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html
> To start a new topic under TomEE Dev, email
> ml+s979440n982480h80@n4.nabble.com
> To unsubscribe from TomEE Dev, click here
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=982480&code=ZG95Y2hpbkBkc29mdC1iZy5jb218OTgyNDgwfC0xNTI2MjM5MTA3>
> .
> NAML
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providershttp://www.dsoft-bg.com/
> Mobile: +359888243116
>
>
Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly
Posted by Doychin Bondzhev <do...@dsoft-bg.com>.
I use IDE to run test multiple times. It failed only once here.
I also did some changes in the test code.
I removed the line
https://github.com/apache/tomee/blob/7318a1db025878b3f1f56587c60b47a14d55ef68/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L162
This made test many times faster. It is not needed. This tests non
leaking business method so we expect everything to be cleared.
Also I removed the all the code that waits for log messages. I left only
the line
assertEquals(message, times, logCapture.find(message).size());
There is no need to wait for anything. All the waiting is already done
for the first and second tests in
runTest method
Regarding
System.gc();
It is needed only for the tests that leak so I removed it from all other
tests.
Third and fourth test are run in the main thread and there is no need to
wait before processing the log to.
You can see the change here :
https://github.com/doychin/tomee/commit/76e090b482825682a859d6a1637f572d8128f30a
In case you like it I can create PR.
On 6.1.2019 г. 19:47, David Blevins-2 [via TomEE & OpenEJB] wrote:
> > On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <[hidden email]
> </user/SendEmail.jtp?type=node&node=4687291&i=0>> wrote:
> >
> > I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
> >
> > Where I can get more detailed error log regarding this issue?
> >
> > I tried to run the test locally but it works.
>
> Here's a technique I use sometimes to check random failures:
>
> $ for n in {1..50}; do mvn clean install
> -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
>
> So far 3 out 17 have failed. Debugging them will still be a pain in
> the butt, but at least there's some hope.
>
>
> My shallow observation is that the approach taken in the test itself
> looks conspicuous. Particularly, I'd say this test is not "correct"
> until it can pass without the System.gc() statements.
>
> -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
>
> More importantly, this method is using thread sleeps and retries to
> wait for log output which *will* cause random failures. We should
> have a discussion on if there is a better way to make this more
> deterministic.
>
> -
> https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
>
> Testing log output can work, however the test logic is ultimately "if
> you do or don't see this message in n seconds, we're good." It needs
> to be "if you do or don't see this message *when the test is over*,
> we're good."
>
> Since it's looking in log output, one way to do that might be to
> simply log a "test complete" statement (perhaps with a random number
> on it), and then wait for that in the log output.
>
>
> -David
>
>
>
>
>
> *signature.asc* (499 bytes) Download Attachment
> <http://tomee-openejb.979440.n4.nabble.com/attachment/4687291/0/signature.asc>
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687291.html
>
> To start a new topic under TomEE Dev, email
> ml+s979440n982480h80@n4.nabble.com
> To unsubscribe from TomEE Dev, click here
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=982480&code=ZG95Y2hpbkBkc29mdC1iZy5jb218OTgyNDgwfC0xNTI2MjM5MTA3>.
> NAML
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116
Re: Investigating TOMEE-2403 AutoConnectionTrackerTest fails randomly
Posted by David Blevins <da...@gmail.com>.
> On Jan 6, 2019, at 2:25 AM, Doychin Bondzhev <do...@dsoft-bg.com> wrote:
>
> I would like to investigate this issue: https://issues.apache.org/jira/browse/TOMEE-2403
>
> Where I can get more detailed error log regarding this issue?
>
> I tried to run the test locally but it works.
Here's a technique I use sometimes to check random failures:
$ for n in {1..50}; do mvn clean install -Dtest=AutoConnectionTrackerTest 2>&1 | tee run-$n.log; done
So far 3 out 17 have failed. Debugging them will still be a pain in the butt, but at least there's some hope.
My shallow observation is that the approach taken in the test itself looks conspicuous. Particularly, I'd say this test is not "correct" until it can pass without the System.gc() statements.
- https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L197
More importantly, this method is using thread sleeps and retries to wait for log output which *will* cause random failures. We should have a discussion on if there is a better way to make this more deterministic.
- https://github.com/apache/tomee/blob/master/container/openejb-core/src/test/java/org/apache/openejb/resource/AutoConnectionTrackerTest.java#L216
Testing log output can work, however the test logic is ultimately "if you do or don't see this message in n seconds, we're good." It needs to be "if you do or don't see this message *when the test is over*, we're good."
Since it's looking in log output, one way to do that might be to simply log a "test complete" statement (perhaps with a random number on it), and then wait for that in the log output.
-David
Re: Investigating https://issues.apache.org/jira/browse/TOMEE-2403
Posted by Doychin Bondzhev <do...@dsoft-bg.com>.
Hi César,
I found the CI pages but looks like logs are not kept for long enough.
Any way I managed to reproduce the problem and I already implemented
some changes in the test code and in the code that is tested.
You can check them here in case you are interested and let me know if
you have comments.
http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tt4687290.html#a4687294
On 7.1.2019 г. 16:44, César Hernández Mendoza [via TomEE & OpenEJB] wrote:
> Hi Doychin,
> By looking at TomEE-2403, it seems the issue was occurring only on
> TomEE CI
> [1].
> If you click on a specific build you can have access to the entire
> log, see
> for instance:
> https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8/builds/987/steps/test/logs/stdio
>
> As a side note, I think this CI information can be something we can
> improve
> on the TomEE website.
>
> [1] https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8
>
> El dom., 6 ene. 2019 a las 4:25, Doychin Bondzhev (<[hidden email]
> </user/SendEmail.jtp?type=node&node=4687324&i=0>>)
> escribió:
>
> > Hi,
> >
> > I would like to investigate this issue:
> > https://issues.apache.org/jira/browse/TOMEE-2403
> >
> > Where I can get more detailed error log regarding this issue?
> >
> > I tried to run the test locally but it works.
> >
> > --
> > Doychin Bondzhev
> > dSoft-Bulgaria Ltd.
> > PowerPro - billing & provisioning solution for Service providers
> > http://www.dsoft-bg.com/
> > Mobile: +359888243116
> >
> >
>
> --
> Atentamente:
> César Hernández Mendoza.
>
>
> ------------------------------------------------------------------------
> If you reply to this email, your message will be added to the
> discussion below:
> http://tomee-openejb.979440.n4.nabble.com/Investigating-https-issues-apache-org-jira-browse-TOMEE-2403-tp4687290p4687324.html
>
> To start a new topic under TomEE Dev, email
> ml+s979440n982480h80@n4.nabble.com
> To unsubscribe from TomEE Dev, click here
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=982480&code=ZG95Y2hpbkBkc29mdC1iZy5jb218OTgyNDgwfC0xNTI2MjM5MTA3>.
> NAML
> <http://tomee-openejb.979440.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Doychin Bondzhev
dSoft-Bulgaria Ltd.
PowerPro - billing & provisioning solution for Service providers
http://www.dsoft-bg.com/
Mobile: +359888243116
Re: Investigating https://issues.apache.org/jira/browse/TOMEE-2403
Posted by César Hernández Mendoza <ce...@gmail.com>.
Hi Doychin,
By looking at TomEE-2403, it seems the issue was occurring only on TomEE CI
[1].
If you click on a specific build you can have access to the entire log, see
for instance:
https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8/builds/987/steps/test/logs/stdio
As a side note, I think this CI information can be something we can improve
on the TomEE website.
[1] https://ci.apache.org/builders/tomee-trunk-ubuntu-jvm8
El dom., 6 ene. 2019 a las 4:25, Doychin Bondzhev (<do...@dsoft-bg.com>)
escribió:
> Hi,
>
> I would like to investigate this issue:
> https://issues.apache.org/jira/browse/TOMEE-2403
>
> Where I can get more detailed error log regarding this issue?
>
> I tried to run the test locally but it works.
>
> --
> Doychin Bondzhev
> dSoft-Bulgaria Ltd.
> PowerPro - billing & provisioning solution for Service providers
> http://www.dsoft-bg.com/
> Mobile: +359888243116
>
>
--
Atentamente:
César Hernández Mendoza.