Hortonworks and Pivotal, two rival companies in Hadoop software and service, are joining forces to develop the Apache Ambari project to create a component that will handle the provisioning, monitoring and management of large Hadoop clusters.

The partnership is somewhat unlikely, especially considering the two drastically different approaches of the two companies in terms of utilizing the Hadoop data platform. While Hortonworks brands itself as a non-proprietary and pure open source product, Pivotal boasts of its power and utility as a data system for enterprises.

The two companies have come together primarily due to their shared belief in the importance of the underlying open source projects that make up the foundation of Hadoop.

"We fundamentally believe in the efforts of a vibrant open source community under the governance of the Apache Software Foundation (ASF) as the fastest path to innovation for Hadoop, and we are excited to work with the Pivotal team to accelerate the innovation, features, and momentum of Ambari," writes Hortonworks vice president for corporate strategy Shaun Connolly in the company's official blog post that details the forged partnership between the two companies.

The decision of Pivotal does not mean that the company will abandon its cluster management software Command Center, which has a similar function to the Apache Ambari.

Apache Ambari is not as popular as the other famous components of Hadoop, namely Apache Spark and Apache Hive. However, as the importance of cluster management is being placed at the forefront, Ambari will most likely soon rise in popularity as well.

"Ambari has been building momentum for more than 18 months now, to the point where the folks at Pivotal, who have made their own operations tools, have been taking a good hard look at it and wondering if they can adopt it and invest in it," Connolly said.

The major benefit of Ambari is the ability to extend and expand the management framework that the project provides for Hadoop, enabling an infrastructure that allows developers to define external components. The project also provides developers with a dashboard to continuously monitor that operational integrity of a Hadoop cluster. The dashboard can also be used to launch, halt, or reconfigure the Hadoop installation.

Hortonworks and Pivotal are looking to jointly increase the number of nodes that Ambari can manage at a time, which currently reaches a number up to the thousands, and to simplify how Ambari can be tweaked to more easily operate Hadoop as a hosted service.

According to Markets and Markets, the Hadoop market is expected to grow to $13.9 billion by 2017.