Experimenting with Icinga 2: a quick way to test an upgrade

Icinga 2 which was initially released around July 2014. Icinga is a fork of the Nagios monitoring system, meaning that it can do periodic checks to servers, services and networks to make sure everything is running smoothly. If not, it can e-mail, text or phone a system administrator.

A bit of a tangent: Nagios was written a long time ago. Initially released on the same year as the first Matrix movie (1999), it was awesome, but it did not age so well. It has scaling issues, performance issues, an ugly user web interface, harsh documentation and shenanigans from Nagios Enterprises, the main company behind the project. As a consequence, developers forked the code into many variants and the plugin community became more neutral. New solutions appeared, but no silver bullet. Icinga seems to have gotten some traction because it uses a Nagios-compatible plugin system. It wasn't a complete rewrite that made your head hurt (Zabbix). The configuration syntax of Icinga 1 was compatible with Nagios 3, and they had a shiny-ish web interface.

I should mention that Nagios is still by far the most used solution. Zabbix is also pretty popular. Here are some numbers loosely based on Debian statistics (popularity contest):

Nagios is still by far the most popular monitoring solution, but Icinga 2 and Zabbix are rising

Staying close to Nagios syntax had its comforts, but it also meant keeping some of its limitations. Icinga 2 is more of a complete rewrite. The documentation keeps talking about node communication with SSL and a 12 step program to get started... with nodes. What if you just want to convert your Icinga 1 configuration and see how it goes?

Testing of a brand new feature
(source: devops reactions)

The quick upgrade

OK, so enough warnings, here's the quick procedure I opted for:

Icinga 2 has good documentation (also available in a more plain format on github). For example, there's a chapter on upgrades and the differences between 1.x and 2.x. However, you might have dozens or hundreds of hosts and services to monitor, and editing that by hand would take a while.

If you are using Puppet, Chef or Ansible, the Icinga team has made available recipes that can be re-used. Whether you can convert your current recipes might be another issue. Since all that inevitably takes more time (although probably a better option), and I just wanted to quick test Icinga 2, the migration scripts were very practical.

The script available in the icinga2-migrate github repository.

To use it, download it to your Icinga test server:

sudo apt-get install icinga2
sudo apt-get install php5-cli zendframework
wget -O icinga2-migration.zip https://github.com/Icinga/icinga2-migration/archive/master.zip
unzip icinga2-migration.zip
cd icinga2-migration-master
./bin/icinga-conftool migrate v1 /etc/icinga/icinga.cfg > /etc/icinga2/conf.d/migrate.conf

The migration script will recursively process all configurations and generate one huge migrate.conf file with all commands, hosts, services and so on. I found it an easy way to learn the basics of new syntax. You will still want to review your configurations later on, in order to use the new syntax fully (lots of cool tricks that help to reduce the number of declarations necessary), but it's a quick way to test Icinga 2.

At this point you can try to start Icinga 2:

sudo systemctl enable icinga2
sudo systemctl start icinga2
sudo systemctl status icinga2 -l
sudo /etc/init.d/icinga2 checkconfig

One annoyance I ran into using the Icinga 2 Debian package, is that config syntax errors didn't generate a warning when starting the service. You have to run "checkconfig" and also keep an eye on the logs to make sure it started correctly. Besides a few service dependencies that had weird errors and some duplicate services with the default configuration, I didn't have many changes to do in order to get the system running.

icli

Update: See icingacli (part of icingaweb2)! Thanks @dnsmichi for the pointer.

I did however have to do a few changes to get "icli" running. According to Google, I'm one of the rare people using icli, but it really rocks. It's a great command line tool to talk to Icinga. I'm not a big fan of web-interfaces. I always ended up having dozens of tabs opened and not really sure what I'm looking at. The icli tool makes it easy to send quick queries to Icinga.

The icli tool requires that the 'command' and 'statusdata' features be enabled:

sudo icinga2 feature enable command
sudo icinga2 feature enable statusdata
sudo systemctl restart icinga2

Since I use the icli Debian Package, but I didn't want to modify the source code of Icli to point to icinga2 instead of icinga (1.x), I did the following:

  • symlinked: /var/lib/icinga2 to /var/lib/icinga
  • symlinked: /var/cache/icinga2 to /var/cache/icinga
  • created: /var/lib/icinga2/rw/
  • symlinked: /var/run/icinga2/cmd/icinga2.cmd to /var/lib/icinga/rw/icinga.cmd

I also had to modify the statusdata feature configuration in /etc/icinga2/features-enabled/statusdata.conf

library "compat"

object StatusDataWriter "status" {
    status_path = "/var/lib/icinga2/status.dat"
    objects_path = "/var/cache/icinga2/objects.cache"
    update_interval = 10s
}

Then using "icli" on the command line worked. I could also send actions, ex:

# icli -h host.example.org
ping6                   OK    PING OK - Packet loss = 0%, RTA = 0.02 ms
ping4                   OK    PING OK - Packet loss = 0%, RTA = 0.03 ms
ssh                      OK    SSH OK - OpenSSH_6.7p1 Debian-5 (protocol 2.0)

# icli -h example.org -a r
Scheduled check of * on 'host.example.org'

nsca-ng

I use nsca-ng for passive notifications, so I had to edit its configuration to point to the Icinga 2 command file:

File: /etc/nsca-ng/nsca-ng.local.cfg

command_file= "/var/run/icinga2/cmd/icinga2.cmd"

Although I guess that isn't really necessary with the above changes for icli. Also make sure you read the above comments on how to enable the 'command' feature if you have skipped over the icli section.

Presumably the server/node architecture of Icinga 2 is similar to how Zabbix has an agent that runs on node, to avoid using passive notifications such as NSCA, but I haven't really gotten into that yet.

Conclusion!

In conclusion, migrating to Icinga 2 is a rather big change. It took a few hours to try it out, which was more than I hoped. The syntax of the configuration files is completely different, but the logic is pretty much the same. It also removes a few annoyances of the Nagios syntax (ex: for command definitions, with ARG1, ARG2, etc). The syntax is more explicit, readable and the migration script really helps to save time and experiment. I look forward to experimenting more with it, especially the server/node architecture, "apply" rules, better IPv6 testing and variable inheritance.. I feel like I could probably simplify my configuration a lot.

Misc notes

  • /usr/share/icinga2/include/command-plugins.conf : has the definitions for the basic plugins. I found it easier to grep for options in this file.
  • In Icinga1/Nagios, I did not have the habit of specifying the IP address for each Host. It seems a requirement for Icinga 2 (or recommended practice), although that may be because of the default "apply" rules.
  • Icinga 2 makes it much easier to manage IPv6 and making sure both IPv4/IPv6 services are monitored. Yay!!

Archives