ManageIQ
Oparin
Versions: Latest / Quinteros / Petrosian / Oparin / Najdorf / Morphy / Lasker / Kasparov / Jansa / Ivanchuk / Hammer / Gaprindashvili / Fine / Euwe

Troubleshooting

Logging

ManageIQ has extensive logging to assist in troubleshooting issues or just seeing what is being worked on.

Appliance Logging

On appliances logging is managed using the built-in systemd-journal system. Journald uses database-like binary files to store log lines along with extensive metadata.

There is a journalctl command-line utility which can be used to query the journal. Documentation for journalctl can be found at https://man7.org/linux/man-pages/man1/journalctl.1.html

If you want to tail the main evm logs then you can use journalctl -f -t evm. Other logs (e.g.: automation or api) can be viewed by using their syslog_identifier e.g. journalctl -f -t automation. You can also view multiple logs by using journalctl -f -t evm -t automation -t api.

journalctl allows you to do much more like view logs for a specific unit (aka worker), for a time range, and for a group of workers (e.g. every generic worker).

Here are some helpful examples:

Viewing:

  • View all logs: journalctl
  • View all logs, most recent first: journalctl -r
  • View all logs since the last boot: journalctl -b
  • View all logs from the previous boot: journalctl -b -1
  • View all logs from a time range: journalctl --since "1 hour ago"
  • View all logs with extra metadata: journalctl --output verbose
  • Tail all logs: journalctl -f

Filtering:

  • Filter warnings: journalctl -f -p warning
  • Filter errors: journalctl -f -p err
  • Filter warnings and errors: journalctl -f -p err -p warning
  • Logs for a specific unit: journalctl -f -u manageiq-generic@51113329-8942-4a4c-98d6-1e532656c569.service
  • Logs for a group of workers: journalctl -f -u manageiq-generic.slice
  • Logs matching a pattern: journalctl -g '(Started|Stopping) EVM server'' (accepts perl compatible regular expressions)
  • Logs for a specific identifier: journalctl -f -t evm

Other:

  • Export to a file: journalctl --output export > export.log
  • Rotate the current journal to an archive: journalctl --rotate
  • Clear out all archived journals: journalctl --vacuum-time=1s
  • Clear out journal archives over a day old: journalctl --vacuum-time=1d

Almost all of these can be combined together where they don’t conflict (e.g. -f can’t be used with -r)

Podified Logging

When running in podified all logs are output to the container’s stdout which is picked up by Kubernetes/OpenShift and made available via kubectl logs/oc logs.

The log output format is in JSON and includes extra metadata like the log-level, timestamp:

{
  "@timestamp":"2020-01-01T12:34:56.789012",
  "pid":12345,
  "tid":"6789abcdef01",
  "level":"info",
  "message":"MIQ(Class#method) Example message"
}

You can also pass a label selector to oc logs with -l so for example to tail all generic pods: oc logs -f -l name=1-generic

If you prefer to view the container JSON logs in the standard MIQ format you can use the unjson_logs utility. This utility is in /var/www/miq/vmdb/tools/unjson_logs

oc logs -l name=1-ui | tools/unjson_logs
[----] I, [2022-06-16T15:40:21.095098 #7:b4e4c]  INFO -- production: Started GET "/ems_infra/56?display=vms" for 127.0.0.1 at 2022-06-16 15:40:21 +0000
[----] I, [2022-06-16T15:40:21.100999 #7:b4e4c]  INFO -- production: Processing by EmsInfraController#show as HTML
[----] I, [2022-06-16T15:40:21.101242 #7:b4e4c]  INFO -- production:   Parameters: {"display"=>"vms", "id"=>"56"}