Scale testing in Openstack has required some creative thinking. While gated commit testing of the same things over and over again is quite valuable in determining the effects of code changes, when we broaden our scope to the infrastructure as a whole and start looking at scale we need to run tests over and over at different sizes. To this end we have written Odin.
Odin is a simple BASH loader for running multiple Openstack Rally tests one after another. Due to the nature of these larger sample sets, testing can run for days or even weeks. It is also nice to have a robot loading the next test run during days or at times when you aren’t actually at work, taking advantage of what would otherwise be downtime to meet deadlines and project goals, or simply to have the ability to do something else until the testing is completed.
I would like to put this on Github somewhere but have not found a home for it yet. The components of Odin are:
1. The Script – Currently BASH (needs to be Python eventually but I’m not very good at it yet…) allows the user to set a range of tests to run using the START and END variables at the start of the script. The script uses a loop that keeps track of how many iterations it has run already and steps through them in sequence until the run matching the END variable count has completed. Refer to the basic README for more details.
2. Service Restart Scripts (Optional) – The nature of scale testing dictates that generating failures is where the most interesting data is provided. As such we expect to crash services, overload queues, and generally bring the house down to identify issues in our architectures. This leaves a mess. Odin provides a place for you to provide your own Openstack service restart scripts (in our case an Ansible playbook) to make sure services are up and running for each test run, and to make sure all services are in as close to the same working state each time to prevent artifacts from previous runs from altering our results for the current run.
3. Rally Cleanup Scripts (KloudBuster) – Openstack Rally is great at creating things, but is not so good at cleaning them up. While it is definitely on the Rally project roadmap to implement various improvements to ensure it cleans its room after each run, we are dropping in a script from our sister KloudBuster project (missing LBaaS cleanup and some other functions still…) and simply changing the search string to look for rally generated components.
4. Test Plans – When I have time I will write something to auto-generate these, but for now I am just taking the base plan, copying it, and doing a find and replace on the appropriate fields. These plans are numbered for the count mechanism to load them in the proper sequence.
5. Alfred The Butler – A simple script that can be called outside the workflow to clean up Rally messes and restart your Openstack services since these are just generally useful operations.
Here is what I have so far.