Objective
Evolution is an important thing in the technology world. All technologies are born out of purpose. For example, search engines were created to sort through the massive amounts of data online. With each new upgrade, technology compounds existing technologies to create something better than what was previously used. And on and on it goes.
However due to the big data evolution, there are lot of scalable architecture and platforms that are evolved quickly. There is nothing wrong with that, but the maturity/experience of a product that has been evolved over a period of many decades can never be ignored while we embrace new things and amazing innovations. It’s like saying experience cannot be compromised — regardless of whether humans or technology. Those with years of tech experience in financial sectors, please ask yourselves a question, whether the mainframe processing could be replaced by any technology. Similarly in every big data related talks, I advocate not to replace your RDBMS with big data — as this question always comes from audience. It’s not possible. RDBMS is meant for a unique purpose, and its evolved and battle tested across the world across industries based on very strong foundations. This approach creates a void in the world of technology. What we mean is there are some stable products that are existing and proven. At the same time there are many new products which address the scalability factor. These new products might not have the maturity equivalent to those which are evolved over our period of time. Hence if we want to have the flexibility of the products evolved in time with the scalability offered by the new, we have a compatibility issue. For example Python is a well matured language for machine learning. Apache Spark is well known for unified distributed computing. Spark, though supported with MLLIb for machine learning, is not much evolved as that of Python libraries and packages in that field. What we need is the ability to use all the Python features in a Spark application so that we get the best of both worlds. That’s what we are covering in this article.
