Apache Spark gaining momentum, big backers.

IBM is making a big push into Spark and plans to invest $300 Million and plans to assign 3,500 engineers to Spark related analytics projects. Spark is a unique technology that can work stand-alone or can be boot-strapped to work on top of HDFS. So, this technology presents developers with the deployment flexibility and simplicity that they will appreciate. Andreessen Horowitz and New Enterprise Associates have invested $47 Million into a Spark-centered start-up called Databricks.

Spark is a potential replacement to MapReduce. It gained a lot of attention when it won the Gray Sort Benchmark in 2014. It sorted 100 TB of data or 1 Trillion records in 23 minutes, handily breaking the previous record of 72 minutes.

Apache Spark claims to be 10x faster than MapReduce on disk and 100x faster on memory.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s