Dynamic DataSource Routing : Spring Boot

Dynamic DataSource Routing: Spring Boot

Multi data source routing(DB per satellite)

Sometimes we need to create a web application that can connect to multiple databases. We might need to connect multiple databases based on the client, customers so on.

Spring provides AbstractRoutingDataSource class which we can extend and achieve the data source routing dynamically and we can do it easily on Spring Boot.

Use cases

Sometimes for security reasons, performance, scalability, or isolation, we need data source routing such as MASTER and REPLICA DB where REPLICA DB can only be read-only.

Another use case is the real-time processing data of satellites. These are very huge volume data sets and created very frequent on an interval.

Let’s take the satellite SARAL(https://en.wikipedia.org/wiki/SARAL) and Aryabhata (https://en.wikipedia.org/wiki/Aryabhata_(satellite)) example which pushed the data on a very frequent interval of time. Suppose we have an application that can retrieve the data of the satellite and each satellite has its own database. To achieve this, we can take the help of Data source routing.

Satellite Data (ftp://scatsat1:%22%22@ftp.mosdac.gov.in/2020/JUL/L3BT/). Data copied from https://www.mosdac.gov.in/open-data

Each satellite creates a huge set of files on their FTP server (https://www.mosdac.gov.in/open-data) and based on the type, we want to retrieve the files.

In this article, we will provide the steps to configure multiple data sources dynamically using Spring Boot & JPA.

For the complete example find the code from GitHub.

Maven Dependencies

Here is the list of maven dependencies. We used MySQL Database and Spring MVC.

Find it from GitHub.

DataSource Router :  AbstractRoutingDataSource, Spring 2.0.1 introduced which acts as an intermediary. However, the real data source can be determined dynamically at run time based on a lookup key. In our example, we will use the notion of a DatabaseContext which has the following implementation. Find it from GitHub.Take a DataBaseContextHolder which is a utility class that can set the data source context dynamically which has the following implementation. Find it from GitHub.

Finally, we will define our MultiRoutingDataSource which will extend the spring provide AbstractRoutingDataSource and override the method. Find it from GitHub.

Configuring Datasources

Let’s configure the data source in the application.properties file

Data Source Configuration :

In this example, we created three (3) data source bean such as mainDataSource,arayabhatDataSource & saralDataSource which can be decided based on the type of data source requested dynamically. Find it from GitHub.

Testing :

In this example, when we will click on File Storage-Saral satellite, it will access the satellite SaralData and will navigate the satellite Saral data related file server of ftp://scatsat1:%22%22@ftp.mosdac.gov.in/ and similar to Aryabhata(But for Aryabhata satellite, we have used same FTP server with different data set for illustration purpose only).

Access files example :

Satellite Data (ftp://scatsat1:%22%22@ftp.mosdac.gov.in/2020/JUL/L3BT/). Data copied from https://www.mosdac.gov.in/open-data

Conclusion

Spring boot provides additional configuration support which we can do if we want to configure multi-data source connectivity dynamically.

For the complete example find the code from GitHub.