Microsoft Research in 2013 released this article that nobody got fired for buying a cluster. At that time, optimizations on CPU were already a very interesting point in computation.
Nowadays, it’s even more the case with GPU :
Benchmarks with BIDMach library show that main classification algorithms run on a single instance with a GPU are faster than on a cluster of hundred CPU instances with distributed technologies such as SPARK.
The new approach of deep learning:
Practical examples from NVIDIA :
The traditional approach of feature engineering :
where the main problem was to find the correct definition of features.
And the new deep learning approach :
is inspired by nature :
with the following advantages :
Installs on mobile phones :
Clusters remain very interesting for parsing and manipulating large files such as for example parsing Wikipedia pages with Spark.