[EDIT: 17-Feb-2023]: As of this update Redis can be used as a primary database and is durable. The slides in this original post below are dated as for as Redis is concerned or rather Redis has evolved a LOT.

Recently I had a real-life encounter with a scenario where I had to design a solution (for systems that might not fit into a single primary data model like an RDBMS). It turned out to be a strategy called Polyglot persistence,  which I did not know at that time. After some digging realized that actually 4 years before when I wrote this post Martin Fowler put forth the concept of Polyglot persistence.

So what’s polyglot persistence?

A quick history – we all know RDMBS came into existence beating network and hierarchical database systems. Until this decade the concept of tackling unstructured data with No-SQL stores didn’t reach the mainstream. There are quite an open source and commercial products today but discussions and the origin started way back. We all know the limitations of using RDBMS for unstructured data or data which doesn’t fit into a fixed data model.  When the dominance of RDBMS cracked with very few success stories the delirium of “RDBMS must go and NoSQL is the replacement” went on for some time. But that cannot be right as well isn’t it? Still, we have some data that perfectly fit into a data model and we very well might need RDBMS. Quora CTO argued that he is still going with MySQL for the “questions, answers, comments and replies” data model of Quora (even though it looks like a Graph data model would serve better,  rather than joining multiple tables). Also for storing relationships between entities came RDF-based Triple stores. Later more refined Query-able graph databases came into existence like Neo4J and TitanDB. Here is a good quora discussion about the same. So we may have different forms of data which do or do not fit into a data model, with or without relationships. How do we tackle this?

Looks like the market has reached a state of maturity to have the wisdom to say “let’s use the right data store which makes sense for the type of data we have. it’s okay to mix and match data stores”. Hence born Polyglot persistence and it is just a fancy term to describe mixing and matching primary data models with different datastores.  This apparently is the future of data storage. Here is the presentation from Martin Flower and Pramod Sadalage.

I have borrowed a slide from the above presentation to give you a glimpse of how polyglot persistence might look like and here it is.

Ok, What is the challenges in doing polyglot persistence?

Just think about your Data Access layer, and how will build a unified data access layer to access different data stores. What will be the impact on the business logic layer, and what levels of abstraction we should design for? These are a few challenges to start with.

Here is a great article – this should get you started.

Think about this and share your thoughts. I will write about my opinions in a follow-up post.