IBM is making a big push into Spark and plans to invest $300 Million and plans to assign 3,500 engineers to Spark related analytics projects. Spark is a unique technology that can work stand-alone or can be boot-strapped to work on top of HDFS. So, this technology presents developers with the deployment flexibility and simplicity that they will appreciate. Andreessen Horowitz and New Enterprise Associates have invested $47 Million into a Spark-centered start-up called Databricks.
Spark is a potential replacement to MapReduce. It gained a lot of attention when it won the Gray Sort Benchmark in 2014. It sorted 100 TB of data or 1 Trillion records in 23 minutes, handily breaking the previous record of 72 minutes.