You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2021/03/10 23:24:03 UTC
[GitHub] [couchdb] nickva opened a new pull request #3414: Fix couch_jobs to be less flaky
nickva opened a new pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414
It turns out fabric is dependent on couch_jobs because of db expiration module.
So when couch_jobs was restarted multiple times per test case it could have
brought down fabric. However, since couch_jobs needs fabric for transactional
stuff it ended up brining couch_jobs app down as well.
To fix it:
* Switch to explicitly starting/stopping fabric and couch_jobs together
* Break apart bad_messages* tests to individually test each type of message
as app restarts in the middle of the tests kept killing fabric and
intermittently killing couch_jobs a well.
* Also make the tests look nicer by re-using ?TDEF_FE macros from
`fabric2_test`, this we can avoid the `?_test(begin... end).` pattern.
* Remove `meck:unload` since we don't really `meck` anything in the module
* Don't need to spend time cleaning out database as we don't really create
that many dbs (just one) and that one gets cleaned out in its own test.
To test run:
```
make eunit apps=couch_jobs
```
Or just observer the jenkins CI tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797266385
To see if the db_expiration circular dependency on couch_jobs is messing up the tests would you mind trying running the tests with this diff:
```
diff --git a/src/fabric/src/fabric2_db_expiration.erl b/src/fabric/src/fabric2_db_expiration.erl
index 92f22e749..4a79e5474 100644
--- a/src/fabric/src/fabric2_db_expiration.erl
+++ b/src/fabric/src/fabric2_db_expiration.erl
@@ -56,7 +56,7 @@ start_link() ->
init(_) ->
process_flag(trap_exit, true),
- {ok, #st{job = undefined}, 0}.
+ {ok, #st{job = undefined}, 99999999}.
```
We are basically disabling db expiration from initialization and using couch_jobs, so then when it restart, it doesn't bring fabric application down. Wonder if you'd see a different set of failures with that or no difference at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797120928
@bessbd sorry, I meant `couch_jobs`, thanks for the logs.
No obvious errors based on the failures so far. I can probably update few places to do the `test_util:wait(...)` pattern to wait for the value to change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797120928
@bessbd sorry, I mean couch_jobs, thanks for the logs.
No obvious errors based on the failures so far. I can probably update few places to do the `test_util:wait(...)` pattern to wait for the value to change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
bessbd commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797107773
> @bessbd thank you!
>
> Would you be able to fetch the logs in `src/couch_views/.eunit/couch.log` and share in gist or txt attachment.
```
[donat:~/couchdb] 23a2d3709* 7h45m25s ± ls -la src/couch_views/.eunit
ls: src/couch_views/.eunit: No such file or directory
```
I got a `src/couch_jobs/.eunit/couch.log` though if that's what you meant: https://gist.github.com/bessbd/d34d94f9888993d07b149ce88684428d
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva merged pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva merged pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd edited a comment on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
bessbd edited a comment on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797358363
[`(git diff && make eunit apps=couch_jobs 2>&1) |pbcopy`](https://gist.github.com/bessbd/02c0101db549e74dac59a157832fa1f7)
Do you get 100% pass locally? Is it failing only on my machine?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-796830709
@bessbd thanks for taking a look! Good find.
They should be passing yeah. I saw the passed in Jenkins and locally for me so I submitted the PR. But there is still some flakiness in there it seems.
Could you share more about your environment so I can reproduce the issue (Erlang version? Do you use docker?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
bessbd commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797061637
> Could you share more about your environment so I can reproduce the issue (Erlang version? Do you use docker?)
`[donat:~/couchdb] 23a2d3709* 24m44s ± erl -eval 'erlang:display(erlang:system_info(otp_release)), halt().' -noshell
"21"`
No docker.
Please let me know if you need any further info.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797069943
@bessbd thank you!
Would you be able to fetch the logs in `src/couch_views/.eunit/couch.log` and share in gist or txt attachment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797696388
I'll merge then and make a new issue to keep track of the failure @bessbd found
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
bessbd commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797359839
If we get a consistent green in CI, then I think we should merge this and let's figure out what could be wrong later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
bessbd commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-796825381
Thank you for this PR, @nickva !
I'm getting some fails on my machine: https://gist.github.com/bessbd/906e50015a8ace1432f11538a08151e1 (I got `Failed: 8.` on a previous run.)
Do you have them all passing when you run them locally?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797120928
@bessbd sorry, I meant `couch_jobs`, thanks for the logs.
No obvious errors based on the log so far. I can probably update few places to do the `test_util:wait(...)` pattern to wait for the value to change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797700715
New issue https://github.com/apache/couchdb/issues/3417
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] nickva edited a comment on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
nickva edited a comment on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-796830709
@bessbd thanks for taking a look! Good find.
They should be passing yeah. I saw they passed in Jenkins and locally for me so I submitted the PR. But there is still some flakiness in there it seems.
Could you share more about your environment so I can reproduce the issue (Erlang version? Do you use docker?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] bessbd commented on pull request #3414: Fix couch_jobs to be less flaky
Posted by GitBox <gi...@apache.org>.
bessbd commented on pull request #3414:
URL: https://github.com/apache/couchdb/pull/3414#issuecomment-797358363
[`(git diff && make eunit apps=couch_jobs 2>&1) |pbcopy`](https://gist.github.com/bessbd/02c0101db549e74dac59a157832fa1f7)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org