In March 14, 1999 Ethan Galstad released the first version of Nagios. Then, nearly exactly 10 years later (May 2009), Icinga (a fork of Nagios) was born. What happened there? Why a fork? In this article, I will shed some light about what made the Icinga developers decide to fork (although they still send patches to Nagios). In this article, I will talk to both Ethan Galstad himself, and Michael Lübben (one of the founding Icinga team members and Nagios addon developer). I will quote Michael and Ethan in the article. You get to read their points of view here.
The beginning: Netsaint/Nagios, debugging, features
Nagios, like many other free software projects, started as a hobby. In Ethan's own words, "I first started developing Nagios in the fall of 1998 under its original name of 'NetSaint'. Development started because I was looking for a hobby project to work on in my spare time and because I thought it would be useful for offering commercial monitoring services in the future. When I surveyed the other monitoring solutions that existed at the time I didn't think they were flexible enough to meet my future needs, so I started my own project. I decided to release NetSaint as an Open Source project because I wanted to contribute something back to the community and I knew that community involvement would help improve its capabilities quickly."
Ethan talks about "Nagios Core" as Nagios is divided into two main parts: Nagios Core itself, and its plugins. Nagios real strength is in its ability to be enhanced with add-ons. Ethan talks about how he opened up Nagios to plugin developers from early on: "It wasn't more than a few months after the initial release of NetSaint when the first contributors came onboard to help. I remained the 'gatekeeper' of the core code for many years afterwards, but I made an early decision to spin the plugins off as their own project to bring in more people.". As the gatekeeper, Ethan had to deal with every maintainer's dilemma: accepting features and patches and feature-creep. He says "For many years I wrote the majority of the core code myself and reviewed patches by contributors before either rejecting or incorporating them. Keeping a balance between never-ending feature creep and maintaining the scope of having a focused monitoring engine has always been a challenge, but I think its one of the reasons that Nagios has become the most widely deployed monitoring solution out there." What was it like, accepting code from third party developers? He says "Opening up the development team isn't always easy, nor is it always appropriate. You have to build trust with the contributors who are submitting patches -- both in terms of code quality and their understanding of architectural issues that affect the long-term viability of a project. Some projects are more suited to multiple developers, whereas others are not. I've been quite happy with how the Nagios development team has grown over time and how community members have stepped up and offered to contribute in whatever way they can".
From the very beginning, Ethan was very serious about trademarks. He told me "I took steps to protect the Nagios trademark immediately in early 2002 when I had to change the project's name from 'NetSaint' to 'Nagios'. The name change was the result of 'NetSaint' being too close to another party's trademark, so I thought a name change was the best way to go. It also made me aware of the importance of trademarks, how trademark law worked, the importance of protecting your brand name, and striking a balance with the community's use of a mark. Beginning with the first release of Nagios in 2002 I published guidelines on how to use (and not use) the Nagios trademark, as well as how to provide attribution of the mark's ownership. Community members (including core contributors, plugins developers, and add-on developers) by in large understood the need for the rules and honored the trademark usage and attribution requirements. We later adopted a modified version of the Ubuntu trademark policy for use with Nagios, as we felt it served as a good example of balancing trademark protection requirements with community advocacy and usage."
Six months before the split
According to Michael, Some developers, especially towards the end, grew displeased with Ethan's management of Nagios Core. Michael, who was there at the time, says "Six months before the fork, there was a bit of unrest among Nagios' extension developers: Nagios Core was maintained just by Ethan Galstad (this was in contrast with Nagios plugins and add ons, developed by several programmers). Up until that point, Ethan hadn't action the request of the community to get more developers. Community patches went unapplied for a long time, and so did their requests to support more databases." So, Ethan was keeping what he considered a balance, whereas other developers didn't feel that balance was strong enough.
When I asked Michael what bug was bugging him especially before the fork, he answered "No one bug, to see community patches be more regularly integrated was more important for me. Beyond the bugs, it was also the fact that much needed to be improved in the core that none of our patches or addons could resolve. PostgreSQL and Oracle database support for instance, enabling data to be retrieved for reporting addons, an API to ease addon integration, and the list goes on. These were not simply superfluous feature requests – their absence in the core was hindering the community’s ability to develop extensions for Nagios. At that point there was a 2-year standstill on Nagios Core releases. So when we decided to fork, we set our priorities to offer the database flexibility the community had been asking for, introduce a PHP based interface as was promised by Nagios, and some way of making addon developers lives easier – at first this was the Icinga API. All while maintaining configuration and plugin compatibility to Nagios, as well as improving our version of the CGIs. Not only that, since forking, we've also sent back bug-fixes to the new Nagios Core team for their re-integration into the Nagios Core. There is an overview of this flow of fixes to and fro between the two Nagios and Icinga development groups. You can also find them scattered in the Nagios change logs."
Ethan's stance on the "pre-fork" era is different. He says "It's not clear what specific bugs/features they were unhappy with when they decided to fork. Plus, none of the members on the original Icinga team offered to become a core contributor or help with bug fixes or core feature enhancements before they made a decision to fork. That's not how a true community effort should work.". Ethan then adds that "The Icinga bug and feature comparison tables are misleading, in that they show "feature enhancements" alongside of bugs. This makes the bug list appear skewed, as the Nagios Core development team has chosen to not implement certain things in Nagios Core due to negative side effects with key Nagios addons.".
Ethan's take on the matter will sound familiar to many programmers: "Bugs weren't as much the issue as was a few people's expectations that more be done with the core project. The challenge with Nagios Core is that, for the most part, it already does what it was scoped to do and some features that users want shouldn't be added to the core engine. Keeping the core engine lean and architecturally separate from external addons has been a major factor in Nagios' success. Additional features and functionality that users often want do not belong in the core monitoring engine, but rather in plugins and addons to the core. Examples of such functions/features are web interfaces, additional APIs, reporting, distributed monitoring, etc. A good architectural understanding of what Nagios is and how things are designed to fit together as separate components, rather than allowing Nagios to become a monolithic solution, is essential for anyone who wants to develop for Nagios."
His plan worked: to have an idea of the sheer amount of code we are talking about, just have a look at Nagios exchange: there are hundreds and hundreds of extensions, plugins, and information documents. It's all comunity-driven, so the occasional "File not found" error might pop up. However, they are quickly caught internally -- and if not, the community works towards fixing them.
Just before the fork
Two years ago, more or less when the split happened, Ethan was having problems resolving issues with a company called "Netways". Nagios had tried to solve the issue amicably with them for over two years -- but didn't manage. Ethan writes: "Over four years ago we discovered serious, intentional trademark violations by a commercial company in Germany by the name of Netways. These violations included a Nagios trademark registration in Germany that the company's founder - Julian Hein - had initiated with the purpose of subverting the Nagios brand for his own company's commercial benefit. We attempted to first resolve the violations amicably with Netways and Julian Hein, but we had no choice other than to get our attorneys involved when they refused to work with us."
Which is when things got interesting. The battle was on several fronts: there was also a PR issue. According to Ethan, "after more than two years of ignoring requests from us and our attorneys, and refusing to resolve the matter as they should have, Netways was able to convince a few members of the Nagios community that we were attacking the entire community with our trademark policy. Netways then planned and launched the Icinga fork of Nagios using a variety of FUD tactics that included references to our trademark policy and trademark protection."
While talking to Michael about Nagios' trademark protection attacks, Michael's says that he wasn't personally approached by them, but then he says "I know of others that Nagios Enterprises (Ethan / Mary) spoke to and emailed, requesting domain names to be transferred. It was covered in an IT journal (http://www.zdnet.de/news/41527909/nagios-kluft-zwischen-autor-und-community-vertieft-sich.htm) which is unfortunately in German, but you can use Google Translate!". (Note: the article is considered inaccurate by the Nagios team: according to them, it contains "inaccurate information and opinion delivered as if it is fact").
Ethan's stance on the alleged "attacks" is that Nagios never attacked the community. He says "Talking about 'Nagios trademark protection attacks' is very misleading here. The community was untouched. We simply took steps to protect the Nagios trademark from commercial misuse. The only steps we took was to email trademark violators that appeared to be commercial entities -- not community members. So, the entities Michael is referring to were some of the commercial trademark violators."
Michael wasn't personally asked to surrender any domains. However, he points out that "if you look online, nagios-fr.org now is monitoring-fr.org, the first nagiosexchange.org is now monitoringexchange.org and nagiosplugins.org now belongs to Nagios Enterprises.".
Ethan's take on this is simple: "The nagios-fr.org domain was in direct violation of our policy. At the time we asked the domain owner to stop using the domain and transfer it to us, he was attempting to setup a commercial "Nagios" entity in France. We attempted to work with him on solving the issue amicably (even to the point of helping with the commercial entity), but he refused to work with us. The nagiosexchange.org domain was one of the over 60 "nagios" domains that Netways owned in violation of our trademark policy. It was only after we exposed (on social media) what they were doing that they transferred the domains to us."
The fork
So, Icinga was born. After these events, Nagios was forked into Icinga. The new Icinga team included:
- Some members of the community advisory board (the ones with stronger ties to Netways)
- Six developers of Nagios extensions (which had a pool of hundreds of developers)
- Members of Netways
Five days after the fork (on the 6th of May 2009), Ethan finally got two more programmers on board, in order to stop being "the" developer and sole gatekeeper of Nagios' codebase; Andreas Ericsson and Ton Voon became part of the Nagios development team:
The announcement of a Nagios fork has brought to light several weaknesses in the Nagios project that have hampered its evolution. Two of these being the fact that I have been “the” developer and sole gatekeeper of the codebase.
The time I can commit to Nagios development has decreased in recent times. It will continue to decrease further as time goes by. I need to transition my role away from that of “the” Nagios developer. To do that, I’m empowering others to take over development, and moving into an architectural/mentor role. And eventually, when the time is right, I’ll pass that role on to someone else.
About people joining the core team, Ethan today points out that "Despite the Icinga team claiming otherwise, I've been quite open to others getting involved and helping out with the project. All it would have taken for the Icinga developers to have joined was for them to have offered their help. Andreas and Ton both volunteered to help out and I gladly accepted their offer. That's how community should work."
What about copyrights and trademark issues?
About the fork, Ethan also points out some interesting facts about Netways registering the Icinga trademark. Ethan says "While unnoticed by the people that initially supported Icinga, it did not surprise us to see that Netways was quick to register the Icinga trademark. We thought that was pretty interesting, given the fact that they didn't want to honor the Nagios trademark policy, and on several occasions acted as though they weren't aware of trademark laws. It was only after the details of what Netways and Julian Hein had done was publicly exposed on social media networks that they finally transferred dozens of Nagios domains and the German trademark filing to us. I personally, and we as a company, have had to defend the Nagios trademark against misuse (especially commercial misuse) for over a decade, and will continue to do so to ensure the longevity of the project. We've found that most of the people who complain about trademark issues are usually exploiting the trademark themselves or work for companies that are exploiting the trademark for their own benefit. Our experience serves as a great example of why every Open Source project needs to have a trademark policy and defend its mark from the onset."
When I asked Michael who actually "owns Icinga" as such, he says "Good question! In terms of copyright, it's owned by whoever contributed code to it. A big portion is owned by Ethan Galstad himself, as well as anybody else who contributed to Nagios' code." So, the owners are all those individuals who have contributed code - their snippets belong to them and each new Icinga version is the result of this collective effort over time.
According to Ethan, however, the real issue is not in the copyright, but in the Trademark: the Iginca trademark (which is probably the most valuable asset, since the code is open source) is owned by Netways. Michael disagrees: "In representing the Icinga Team here, I don't see the Icinga trademark to be the most valuable asset - that is its community of developers, contributors and users. Netways contributes much by way of administrative assistance such as hosting the website and offering a physical postal address for formalities. That has included helping in the current legal transition to a foundation, or what we call here a registered association. Their stewardship over the legalities of the Icinga name does not conflict with the development of our code. This trademark stuff does not interest us -- we choose to focus our time on improving Icinga's functionality. That is the addition of PostgreSQL and Oracle database support that was long desired in the community, improvements to the user interface, in-built reporting capabilities and simply development that users can count on. These results speak for themselves. And our users (http://www.icinga.org/users/) speak their minds too".
The aftermaths: Icinga and Nagios
One year after releasing Icinga, they celebrated their 10,000th (ten thousandth) downloads. They proudly said so in their press release, where they pointed out the success the project was enjoying: a reliable roadmap, support for PostgreSQL and Oracle, 16 active contributors, availability of Icinga in several languages, corporate endorsement, 1000 twitter followers, and so on. So, a year on, the project was doing fine although they were releasing separate versions of Core, API and Web.
About 8 months ago, Icinga turned 2. In their announcement, they celebrated: Core, API and Web was unified and included dual IPv6 / IPv4 support, optimized database connectivity and a new look web interface. Plus, integrated addons (PNP, LConf, Heatmap and Business Process Addon), more than 70,000 downloads, and 23 members.
The Icinga team told me that Icinga is not Nagios plus a custom web interface, but rather it's a framework. Actually most of Icinga's component parts are optional, because Icinga is rather like a toolkit. You take: Icinga Core + a database (MySQL/PostgreSLQ/Oracle) and can add:
- UI (Web or Classic, or a 3rd party interface)
- optional Icinga Reporting
- add ons (popular are: NagVis, Business Process Addons, PNP4Nagios, etc)
- plugins as necessary (all Nagios plugins are compatible)
- optional Icinga Mobile ( web interface for iPhone & Android)
So really, there is a lot of flexibility offered in using Icinga.
What about Nagios? Ethan writes "We've brought additional developers on to the core development team in recent years to help with bug fixes and new developments with Nagios Core, NSCA, NRPE, and other new Nagios projects we've released. For example, we've developed a new PHP interface for Nagios (Nagios V-Shell), new business intelligence addon (Nagios BPI), a new mobile interface (Nagios Mobile), and a new SNMP trap interface (NSTI). We are also working on a new web-based configuration interface for Nagios. These are all Open Source projects that have been developed by our company for use by both our commercial customers and the greater Nagios community". Ethan continues: "The development pace with Nagios has grown steadily since the first release in 1999, and has certainly improved further since we began our commercial operations and were able to start hiring additional developers. Last year (in 2011) there were over six hundred (600) new Nagios projects that were developed and released to the community. There are now thousands of free and Open Source addons for Nagios that extend its functionality and help make it fit the needs of a large variety of end users. The flexibility of Nagios and what it can do (both natively and through the use of optional addons) has been a major factor in Nagios becoming the industry standard in monitoring."
What about plugins? Are plugins developed for Icinga compatible to the ones developed for Nagios? Things are pretty bright, luckily; about this, Michael says "All plugins that have been developed for Nagios also work in Icinga and vice-versa. The Nagios Plugins Development Team is an independent team that develops and supports plugins for Nagios. Like configurations for Nagios, these are always compatible with Icinga - so migration is very simple. The majority of Nagios addons work with Icinga Classic UI but for Icinga Web they need to be modified. This can be done as a kind of widget window to the external add-on with a simple xml snippet, though for full functionality more coding is required. That being said, a couple add ons have been comprehensively integrated -- including Business Process Monitoring by Bernd Strössenreuther, Jörg Linge's PNP4Nagios, LConf by Jannis Mosshammer and my own Nagtrap is in the works. These can be considered pretty much "native". We'd like to see more so we're working on making the process easier for add-on developers with Icinga Web development guides that also explain what is needed to work with the database and REST API. On the smartphone front, a few Android and iOS developers have created apps for, or made compatible to Icinga too. As more people use Icinga, we expect more add-on developers will do the same."
Ethan points out however that compatibility is not always easy and explains: "Plugins may work with both Nagios Core and Icinga Core, but not all Nagios addons will with with Icinga. There are several addons and extensions that will only work with Nagios."
Today
Today, Icinga is definitely a reality.
This brings us to today: Icinga developers have a nice bug and feature comparison between Icinga and Nagios, which gives you a good idea about where each project is.
Ethan's comment about the wiki page above is quite critical: "This comparison on the wiki is both inaccurate and skewed, as it contains incorrect bug data and mixes new features that have not yet been implemented. This has apparently been done with the intention of trying to make Nagios look bad in comparison to Icinga."
In terms of code, things are interesting. Michael writes about the source code ownership: "The original code comes from Ethan Galstad. But the code falls under GPL. That means anyone can take the code, modify and redistribute it. In this case, the Icinga Team did so with the fork. My knowledge of law is limited so I cannot judge if there could be any copyright problems, but I doubt it." What about making money with Icinga? Michael says "I personally do not make any money with Icinga, and there are currently no such plans. Some in the Icinga Team provide fee-based service and support for Icinga. This is also important, as many companies use Icinga or in general open source software, and like to have a point of contact for problems. There are no intentions to make Icinga or any of its components fee-based at any time."
About code, Ethan writes "While I own the majority of the code in the core engine, a portion of the ownership belongs to contributors. We don't have any plans of dual-licensing the core code, as we're quite happy with it remaining a free, community-developed Open Source project. Our commercial efforts focus on other offerings. Nagios XI - our commercial version of Nagios - builds on top of the core monitoring engine and provides features that many long-time Nagios users and newcomers are looking for - including integrated performance graphs, dashboards, advanced reports, configuration wizards, web configuration GUIs, visualization tools, fine-grain control of user settings, and more. We also offer support contracts, training, and will shortly be offering official certifications."
When I asked Ethan about how he felt about the split, he was very straightforward: "It was disappointing to see a few community members join with Netways to launch the Icinga project. Despite outwardly saying that they forked because of slow development with Nagios, they never offered to help improve the Nagios project before making the decision to fork. In my opinion that's not how things should work in Open Source, but you sometimes get the bad with the good when it comes to situations like this. Most of the original Icinga developers (aside from Netways employees) have left the project, which, when combined with discussions I've had with various community members, seems to indicate they started to have misgivings about the methodologies the project and its parent company Netways have employed. There are dozens of Nagios forks in the wild. A few have survived, but most were abandoned after a short time. What makes Nagios stand apart from its forks is that people know Nagios, they build for Nagios, and they trust in Nagios. We haven't seen any fork come close to having the same strength in numbers in terms of community members and addons that Nagios has. That's what keeps bringing people to Nagios. Its a name they trust."
Ethan is obviously very proud of what he achieved. He says "Our statistics show that there are now well over one million active Nagios installations worldwide. While there are dozens of Nagios forks like Icinga that exist, people continue to choose Nagios over its competitors due to its long track record of proven stability and development."
Michael and the rest of the Icinga team still get in touch with the Nagios Core development team at times: "We have some contact and exchange ideas. Some more than others, though Ethan is fairly quiet -- it's mostly on the mailing lists between developers. Especially when it comes to bugs, etc. You can see this from the bug and feature comparison list where we've exchanged patches. From my own feelings, the attitude is positive as it is about making progress in the code."
The future
When I asked Ethan about the future of Nagios, he wrote "There are a number of exciting things that are just on the horizon for Nagios. We're currently working on a new web configuration utility which will make managing a Nagios setup much easier. We're also close to launching official certification tests for Nagios. There will be both Nagios Professional and Nagios Administrator certification levels available for community members to choose from. Our office whiteboards are full of new projects (over 40) that are on our development agenda for 2012. The projects cover a wide variety of technical needs that we've seen from our commercial customers and the Open Source community, including SNMP, Netflow, log file monitoring, incident management, visualization, and virtualization. We place a high level of importance on customer-driven innovation and the things we work on directly reflect what people are asking for. Generally speaking, half of what we work on is for our commercial customers, and half is devoted to Open Source projects that benefit the entire community. We think its a great way to operate our company. Based on the continued growth we see happening worldwide, we expect 2012 to be another record year for Nagios development, with several hundred new projects being released before year's end. Users can watch how new projects shape up by visiting our development blog (http://labs.nagios.com) or by checking Nagios Exchange (http://exchange.nagios.org) for new and updated projects that are released each week. Nagios remains the industry standard in monitoring due to its long history, flexible architecture, its active community, widespread support, and the thousands of free addons that are available for it. That's what makes Nagios a great product and a great community."
Regarding Icinga, Michael had this to say. "Coming up to version 1.7, our Icinga team continues to roll out an exciting roadmap for Icinga. We’re working on major new features like IcingaMQ, a messaging queue for high performance distributed monitoring, improved reporting, graphing and business monitoring capabilities. These will reinforce the existing modular architecture of Icinga Core, database, Web / Classic UI, Reporting and Mobile, which combined with community extensions form a tailored, enterprise grade monitoring solution that is 100% open and free. And when we look at the 128,000+ downloads thus far, next to the growing community of contributors and users who share their appreciation for our project – we are certain that we’re headed in the right direction as a leading open source monitoring project."
What about Ethan's thought on Icinga's development? He says "Although Icinga is still putting out new releases, it has had an extremely high developer turnover rate in its relatively short history. I believe this could cause problems for the project in the long term. Both Netways and the Icinga team have also violated international IP laws (including trademark and copyright) on multiple occasions, which has hurt them substantially in terms of adoption. Many companies have stayed away from the fork due to potential liability issues. More information on some of the trademark and IP problems can be found in my blog article at: http://community.nagios.org/2010/09/28/nagios-trademark-truth/
About the high turnover, Michael says "Upon forking, our team had a settling-in phase where we had to find our feet in working together. Many people wanted to join and contribute to the project and till today many still do. There are always people who find they no longer have time to contribute and step down from the official team for a break. That's why we've introduced an "6 month induction program" to give new contributors time to learn the ropes before they are announced as new official team members. We called it the "Icinga Jedi Apprenticeship" to make sure it doesn't get too serious. Our two latest graduates and official team members have since released our new Icinga Virtual Appliances, so we think we're making much progress here.
Final words
Writing this article was a huge challenge. It took weeks of communication with Michael and Ethan. It fails at providing a clear-cut explanation of what happened, but in some cases clear-cut explanations just don't cut it.
Sometimes, as they say, "it's complicated". And it actually is so!