Hadoop is one of the biggest names in big data analytics. It’s no secret that numerous companies have made a name for themselves using the open-sourced software framework.
One of the biggest advantages for Hadoop has been its ability to integrate with other software. It has spawned many subprojects that aim to capitalize on Hadoop’s processing and storage capabilities.
Companies making products that use Hadoop are popping up everywhere and being bought at a rapid rate. That is expected to heat up even more in the coming year.
But while many see Hadoop growing and evolving, there are some who see challenges ahead.
Saggi Neumann, CTO of Xplenty, a startup that wants to make Hadoop processing easier on new adopters, is one of them. He believes 2014 was a trial year for Hadoop and that 2015 will be the year Apache Spark takes off and gives Hadoop’s MapReduce a run for its money.
“Apache Spark is not the only scalable in-memory data processing engine,” Neumann said, “but it sure is a very popular one because its APIs are less cumbersome than what we’ve grown accustomed to with MapReduce.”
He adds that Apache Spark is 100x faster than MapReduce “and it comes with a stack of other data processing tools including machine learning and graph processing libraries. All major Hadoop distributions already come bundled with Spark and Spark will have better integration with Apache YARN.”
But while he believes that Apache Spark is set to break out, he believes that Apache’s relational database, Hive, is set to face stiff competition.
“Hive is steadily growing to be on par with current relational databases in terms of functionality and performance,” he said, but added that “other open-source SQL projects such as Impala and SparkSQL are major contenders.
He also sees competition heating up among Hadoop’s big distributors including IBM, Hewlett-Packard, Cloudera, Hortonworks and MapR. Each have been pushing its own way to commercialize Hadoop.
However, with all these challenges, Neumann said he believes Hadoop may overcome a major obstacle in 2015: the transition to the cloud.
“The cloud is an elastic and cost-effective environment to experiment on and develop Hadoop solutions against,” he said. “It’s also ideal for offloading heavy-duty data processing tasks. Altogether, it’s bound to become the platform of choice for many companies.”
But while Neumann believes Hadoop is primed to overcome the cloud hurdle, he does not believe it will overcome the struggle to find qualified experts to use the software. That has been a major obstacle to Hadoop’s growth since the beginning.
“The demand for Hadoop professionals will increase and is likely to peak in 2015 or the following year,” he said. “Companies will continue to try to train in-house and look for expensive Hadoop experts, as well as tools and services that will allow them to use Hadoop technologies without the steep and continuous learning curve.”
But some disagree.
In a countdown of top ten predictions for Hadoop in Computer Business Review, Mike Gualtieri, an analyst of application development at Forrester Research said: “CIOs won’t have to hire high-priced Hadoop consultants to get projects done. Hadoop projects will get done faster because the enterprise’s very own application developers and operations professionals know the data, the integration points, the applications and the business challenges.”
Whatever the outcome, it doesn’t appear that Hadoop is going anywhere in the coming year, but there does appear to be big changes ahead.
