Data Architecture: Migration, Management and Navigation of Cloud

Words Manish Kumar

NEW DELHI (India CSR): The migration of large on-premises databases to the cloud is not only a technical task, rather it represents a strategic shift for organizations aiming to enhance operational efficiency and harness the full potential of their data. The landscape of data management is continually evolving. The trend of data lakes, that gained momentum from 2017 to 2020, has fundamentally shifted how organizations handle their data.

As a Subject Matter Expert (SME), Ramasankar Molleti, spent a few years immersing himself in this arena, working diligently to secure significant achievements while navigating complex challenges. He was the single point of contact for database migration as well as data lake setups at a few esteemed organisations, where he has played a pivotal role in helping clients transition their data architectures. The primary focus has been on implementing high availability and disaster recovery capabilities, while ensuring data governance, and enhancing data quality and streaming processes. The migration of databases, particularly those hosted on Oracle and SQL Server, has become a defining feature of his career.

Molleti also migrated a substantial Oracle database from on-premises systems to Amazon RDS Oracle. This project involved a straightforward data transfer along with the intricate task of setting up the initial load and ongoing migration using Change Data Capture (CDC). Moving petabytes of data was challenging and required a meticulous plan; so, he effectively divided the schemas into non-blob and blob objects. This strategic approach assisted in seamless migration, as each type could be optimized separately based on their unique replication requirements.

These initiatives have been profoundly consequential. By migrating from Oracle to Amazon Aurora, licensing costs reduced drastically, providing significant savings for the organization. The establishment of an S3 data lake for large datasets facilitated efficient data storage and lifecycle management. These efforts have resulted in enhancement of data availability and durability, and have optimized resource allocation, leading to increased operational efficiency.

Even beyond cost savings, Molleti’s contributions have had a measurable impact on the organization’s bottom line. For instance, the move to cloud-based solutions has streamlined data access and retrieval processes, resulting in improved productivity across departments. The adoption of Amazon Web Services Lake Formation further accelerated this transition, allowing for near-instant data availability and up-to-date reporting, a marked improvement over traditional data warehousing methods that required lengthy batching processes.

However, there were several challenges that hindered the progress. One of them was the migration of Oracle databases containing blob objects. Successful tuning of the Data Migration Service (DMS) replication task parameters was essential for overcoming data transfer hurdles. Establishing data lakes using S3 posed its own set of difficulties. Throttling issues were encountered with AWS Lambda due to event-based actions triggered by data uploads. Implementing an extra layer of queuing through Amazon SQS effectively resolved these scaling challenges and enabled smooth operations even under heavy loads.

In conclusion, as companies transition from traditional data warehousing to more agile cloud-based solutions, the emphasis on real-time data availability and machine learning strategies is becoming increasingly crucial. The shift from traditional data warehousing to cloud-based solutions is reshaping how organizations manage and leverage their data. Strategic initiatives of experts like Ramasankar Molleti not only enhance operational efficiency and reduce costs but also lay the groundwork for real-time data access, positioning companies to thrive in an evolving digital landscape.