Wejo Ltd is a data-centric company specializing in real-time connected car data and data analytics services. In my role as the Head of DevOps, I lead a growing team of 8 DevOps Engineers while maintaining a hands-on approach (30% management, 70% hands-on). My responsibilities span across multi-cloud infrastructures on AWS, Azure, and GCP, which collectively stream ~330 billion data points per month and store 17PB of data.

Challenges

• Managing a complex multi-cloud infrastructure that includes AWS, Azure, and GCP.
• Streamlining enormous data streams and storage.
• Migrating and modernizing legacy platforms.
• Implementing financial optimizations without compromising performance.
• Collaborating with data teams to build data warehouses and lakes across various cloud providers.

Solutions

Extension to GCP

Besides the existing AWS and Azure environments, I spearheaded the extension of our multi-cloud platform to Google Cloud Platform (GCP), primarily focusing on Kubernetes (GKE) and Google Cloud Storage (GCS) services.

FinOps Initiatives

Took charge of cloud platform costs, implementing various cost-saving measures that led to a 50% reduction in costs for Kubernetes workloads and cloud storage. Strategies included usage monitoring, resource optimization, storage tiering, and implementing retention policies.

Kafka Cluster Migration

Successfully migrated over 30 Strimzi deployed Kafka clusters to the Confluent Platform, enhancing performance and reliability.

DataOps and Data Warehousing

Collaborated with data teams to implement a multi-cloud data warehouse and lake architecture. This incorporates services like Apache Spark, Kafka Streams, AWS S3, Redshift, Google Cloud Storage, Azure Blob Storage, Databricks, and Tableau.

Reliability and Monitoring

In my previous role as Principal DevOps Engineer, I took ownership of AWS and Azure cloud platform reliability by implementing comprehensive monitoring solutions and provisioning high availability, low-latency platforms.

Multi-Cloud Transformation

Led a transformational project that extended our AWS infrastructure to Microsoft Azure, thereby creating a more resilient and scalable system.

GitOps and Kubernetes Monitoring

Implemented GitOps strategies (ArgoCD, Kustomize, Helm) and built a centralized monitoring and logging platform for Kubernetes using Prometheus, Grafana, and Loki.

Results

• Successfully extended to a multi-cloud environment including AWS, Azure, and GCP.
• Achieved a 50% reduction in Kubernetes and storage costs.
• Enhanced data operations by building a multi-cloud data warehouse and lake architecture.
• Optimized performance and supported multi-tenancy on Kubernetes.

Conclusion

My tenure at Wejo Ltd has been a journey in scaling DevOps initiatives, optimizing costs, and implementing data-centric solutions in a multi-cloud environment. The success of these projects has not only streamlined operations but has also set the stage for future innovations in data analytics and cloud computing.