Troubleshooting

This page documents issues that are commonly encountered and provides advice on how to troubleshoot them.

Increasing verbosity and obtaining debug logging

When there is an exception, an error or something doesn’t work, it is likely that increasing verbosity or looking at the debug logs can point us in the right direction.

The ansible and ansible-playbook commands do not run with enough verbosity by default to display tracebacks from Ansible plugins.

In order to retrieve the tracebacks, run these commands with increased verbosity (-vvv):

> ansible-playbook playbook.yml
# ...
TASK [Gathering Facts] *****************************************************
[WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f6d4abcfe50>):
# ...

> ansible-playbook -vvv playbook.yml
# ...
TASK [Gathering Facts] *****************************************************
[WARNING]: Failure using method (v2_playbook_on_task_start) in callback plugin (<ansible.plugins.callback.ara_default.CallbackModule object at 0x7f1e7fd5ce50>):
Callback Exception:
  File "/usr/lib/python3.9/site-packages/ansible/executor/task_queue_manager.py", line 389, in send_callback
    method(*new_args, **kwargs)
  File "/usr/lib/python3.9/site-packages/ara/plugins/callback/ara_default.py", line 354, in v2_playbook_on_task_start
    raise Exception
# ...

Additional debug logging from ara’s perspective can be added by setting ARA_DEBUG to true and ARA_LOG_LEVEL to DEBUG:

> export ARA_DEBUG=true
> export ARA_LOG_LEVEL=DEBUG
> export ANSIBLE_CALLBACK_PLUGINS=$(python3 -m ara.setup.callback_plugins)
> ansible-playbook tests/integration/hosts.yaml

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
Operations to perform:
Apply all migrations: admin, api, auth, contenttypes, db, sessions
Running migrations:
No migrations to apply.
2021-04-20 09:56:14,678 DEBUG ara.plugins.callback.default: v2_playbook_on_start
2021-04-20 09:56:14,679 DEBUG ara.plugins.callback.default: Ignoring argument: extra_vars
2021-04-20 09:56:14,680 DEBUG urllib3.connectionpool: Starting new HTTP connection (1): localhost:35697
2021-04-20 09:56:14,708 DEBUG urllib3.connectionpool: http://localhost:35697 "POST /api/v1/playbooks HTTP/1.1" 201 1116
2021-04-20 09:56:14,708 DEBUG ara.clients.http: HTTP 201: post on /api/v1/playbooks
2021-04-20 09:56:14,708 DEBUG ara.plugins.callback.default: File not in cache, getting or creating: /home/dmsimard/dev/git/ansible-community/ara/tests/integration/hosts.yaml
2021-04-20 09:56:14,723 DEBUG urllib3.connectionpool: http://localhost:35697 "POST /api/v1/files HTTP/1.1" 201 1784
2021-04-20 09:56:14,724 DEBUG ara.clients.http: HTTP 201: post on /api/v1/files

PLAY [Create fake hosts for host tests] ****************************************************************************************************************************************************************************************************************************************************
2021-04-20 09:56:14,726 DEBUG ara.plugins.callback.default: v2_playbook_on_play_start
2021-04-20 09:56:14,726 DEBUG ara.plugins.callback.default: Updating playbook labels to match: check:False,tags:all
2021-04-20 09:56:14,750 DEBUG urllib3.connectionpool: http://localhost:35697 "PATCH /api/v1/playbooks/13 HTTP/1.1" 200 1172
2021-04-20 09:56:14,751 DEBUG ara.clients.http: HTTP 200: patch on /api/v1/playbooks/13
2021-04-20 09:56:14,806 DEBUG urllib3.connectionpool: http://localhost:35697 "POST /api/v1/plays HTTP/1.1" 201 294
2021-04-20 09:56:14,807 DEBUG ara.clients.http: HTTP 201: post on /api/v1/plays

# [...]

Playbooks are not recorded

If it is expected that playbooks should be recorded but they are not, there are a few things to double check:

  1. Are both ara and ansible installed and running under python >=3.5 ?
  2. Are the ara and ansible packages installed for the same python interpreter ? Mixing packages from distributions (such as rpm or deb) and PyPi tends to be problematic.
  3. Is the ara callback enabled ? This can be done with export ANSIBLE_CALLBACK_PLUGINS=$(python3 -m ara.setup.callback_plugins) prior to running playbooks.
  4. By default, playbooks are saved in a sqlite database at ~/.ara/server/ansible.sqlite. Are playbooks running under a different user or location than the server ?

Database exceptions

By default, ara’s Ansible callback plugin takes care of creating a sqlite database and runs SQL migrations automatically.

Attempting to start the server process or run CLI commands (such as ara playbook list) before running these SQL migrations can result in exceptions such as:

> ara playbook list
2021-04-15 20:01:23,224 ERROR django.request: Internal Server Error: /api/v1/playbooks
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 383, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: playbooks

SQL migrations can be run manually by executing ara-manage migrate.

For MySQL and PostgreSQL, the database must have already been created before attempting to run migrations.

Bad Request (400) and ALLOWED_HOSTS

Attempting to browse the API server or it’s built-in reporting UI can result in a HTTP 400 (Bad Request) error if the hostname is not included in the ARA_ALLOWED_HOSTS of ~/.ara/server/settings.yaml.

This error in also manifested in the server’s logs as such:

2021-04-20 10:32:45,219 ERROR django.security.DisallowedHost: Invalid HTTP_HOST header: 'ara.example.org:8000'. You may need to add 'ara.example.org' to ALLOWED_HOSTS.
2021-04-20 10:32:45,250 WARNING django.request: Bad Request: /

Adding the hostname to your settings.yaml configuration file and restarting the server should resolve the issue.

Degraded playbook execution performance

Recording playbooks with the ara callback plugin introduces a performance overhead because we are now recording playbooks instead of just printing to the console:

_images/recording-workflow.png

Some general tips in order to minimize the overhead and improve performance:

  1. Run the API server with an application and frontend server (ex: gunicorn with nginx in front)
  2. Use MySQL or PostgreSQL in order to allow callback multi-threading without running into sqlite database locking constraints
  3. When using MySQL or PostgreSQL, enable callback multi-threading by setting ARA_API_CLIENT=http and ARA_CALLBACK_THREADS=4
  4. Latency is important because it adds up quickly over the course of a playbook. Keep the latency between your Ansible control node, your API server and database server as small as possible.
  5. While not specific to ara, tuning Ansible’s SSH pipelining, forks and other parameters can yield significant performance improvements.