Data Engineering on Google Cloud – Automated Data replication approach from Cloud SQL to Big Query using Python

Google Cloud has some rich features and products w.r.t SQL when your apps are built on MySQL or PostgreSQL. But If you have a requirement to replicate the data and schema from Cloud SQL(SQL Server) to Big Query then the options are limited. Reasons: Google Cloud Data Flows doesn’t support Cloud SQL(SQL Server) as Source as per official docs Big

Data Engineering on Google Cloud- python docker cloud SQL for sqlserver

Feeling good to be back blogging again!! I was recently fortunate to have got a chance to work on Google cloud and i found it super interesting!!! Hence the urge to start writing again. Hope this is useful for anyone looking for a serverles data engineering solution on GCP. Google cloud offers 3 flavors of SQL PostgreSQL, MySQL and SQL

Azure Policy As Code – Accelerate governance in cloud

The cloud, in the context of technology, has many definitions and types—ranging fromits simplest form of using shared resources, to a fully automated environment withextreme standardization. Various departments in your organization would be creating, updating and terminating resources in cloud. What does this really mean in practice, and how is it relevant to your organization?The answer is simple: technology exists

How to experience Northern lights in the midst of Pandemic?

This story  is about an incredible experience I had with my family last evening. Before I move further in my story I live in north west part of Singapore in a south west facing flat on 9th floor with completely open front facing view which drives some very cool breeze around the our hall area both in the mornings and

Horizontal pod autoscaling(HPA) in Azure aks

Hey folks!! its been a couple of months that couldn’t do a writeup on a topic. Having been closely watching the Rise and the Rise of Kubernetes based solutions off late I found the Auto scaling concepts pretty exciting. There are 3 different ways of scaling in kuberenetes Horizontal Pod AutoScaler(HPA) – Declarative Pod Resource Request based scaling Cluster Node