You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2022/08/11 21:11:36 UTC
[GitHub] [cloudstack] mrog opened a new issue, #6633: Affinity group creation sometimes fails due to race condition
mrog opened a new issue, #6633:
URL: https://github.com/apache/cloudstack/issues/6633
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
API
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.14.1.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
N/A
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
CentOS 7, but this is likely N/A.
##### SUMMARY
<!-- Explain the problem/feature briefly -->
When using the API to create affinity groups, it usually works. On rare instances, a race condition causes a NullPointerException.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
Use the API to create an affinity group. This may take many tries to reproduce the issue.
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
The affinity group should be created.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
There is sometimes a NullPointerException.
See the attached log. It shows that an async job is created and runs. The output from the async job includes the message `Update db status for job-131043`, which indicates that the transaction has started which will set the removed date for the async job. Then when ApiServer.getBaseAsyncCreateResponse() looks for the job on line 662, it gets a null value. This causes the NPE to happen on line 663.
[NullPointerException.log](https://github.com/apache/cloudstack/files/9312143/NullPointerException.log)
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] mrog commented on issue #6633: Affinity group creation sometimes fails due to race condition
Posted by GitBox <gi...@apache.org>.
mrog commented on issue #6633:
URL: https://github.com/apache/cloudstack/issues/6633#issuecomment-1216833119
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #6633: Affinity group creation sometimes fails due to race condition
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #6633:
URL: https://github.com/apache/cloudstack/issues/6633#issuecomment-1216240739
@mrog I'm closing this for now as the issue has been fixed in 4.17.0.
Please re-open if it is reproducible there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #6633: Affinity group creation sometimes fails due to race condition
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #6633:
URL: https://github.com/apache/cloudstack/issues/6633#issuecomment-1213072722
@mrog how to reproduce this? I tested with 4.17 branch env, trying to create 50 groups simultaneously and it seem to work okay with following script using cmk,
```
#!/bin/bash
create_ag_func(){
echo "Create AG $1 times..."
cmk create affinitygroup name="abcd${1}" type='host affinity' &
}
for i in {1..50}
do
create_ag_func $i &
done
wait
echo "All done"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr closed issue #6633: Affinity group creation sometimes fails due to race condition
Posted by GitBox <gi...@apache.org>.
shwstppr closed issue #6633: Affinity group creation sometimes fails due to race condition
URL: https://github.com/apache/cloudstack/issues/6633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] mrog commented on issue #6633: Affinity group creation sometimes fails due to race condition
Posted by GitBox <gi...@apache.org>.
mrog commented on issue #6633:
URL: https://github.com/apache/cloudstack/issues/6633#issuecomment-1213368338
If ApiServer.getBaseAsyncCreateResponse() was able to find jobs that have been removed, that would fix this issue. But I don't know if it would cause other problems. Is there a reason why it shouldn't be allowed to find removed jobs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] mrog commented on issue #6633: Affinity group creation sometimes fails due to race condition
Posted by GitBox <gi...@apache.org>.
mrog commented on issue #6633:
URL: https://github.com/apache/cloudstack/issues/6633#issuecomment-1213332061
@shwstppr It might be hard to reproduce. Creating affinity groups usually works, but one of our developers saw this issue 1 out of 4 tries yesterday.
We have the CloudStack management service and MySQL running on the same machine. It has 2 CPU cores, and CPU utilization was around 60-65% at the time of the error. It makes me wonder if the error will only happen if the machine is fairly busy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #6633: Affinity group creation sometimes fails due to race condition
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #6633:
URL: https://github.com/apache/cloudstack/issues/6633#issuecomment-1213380169
@mrog that change is already there in 4.17.0
https://github.com/apache/cloudstack/commit/293517ad728
You will have to upgrade or create your own packages for 4.14
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org