THE DEV SPECTRUM

← Back to Spectrum

The EMR Migration Handbook: Moving Massive Data Clusters Without Data Loss

1 min read #Cloud Architecture PK Pramod Krishna

Migrating an EMR cluster isn't just about moving code; it's about moving the gravity of your data. When I recently managed an EMR migration, the challenge wasn't the spark jobs—it was the S3 consistency and the HDFS metadata.

The "Lift and Shift" Fallacy

Many architects try to simply replicate the instance types and configurations. However, a migration is the perfect time to modernize. We moved from m5.xlarge to r6g.xlarge (Graviton2) instances, which provided a 20% cost reduction and better performance for our memory-intensive Spark jobs.

Step 1: S3 DistCp for Data Transfer

For petabyte-scale data, s3-dist-cp is your best friend. It allows you to parallelize the copying of data between buckets across different regions or accounts.

# Example of running S3DistCp to sync data for migration
hadoop jar /usr/lib/hadoop-lzo/lib/s3-dist-cp.jar \
  --src s3://old-data-bucket/logs/ \
  --dest s3://new-data-bucket/logs/ \
  --groupBy '.*(2024-[0-9][0-9]-[0-9][0-9]).*'

Step 2: The Metadata Challenge

Your Hive Metastore is the brain of your cluster. We opted for an External Hive Metastore using AWS Glue. This decoupled the data catalog from the lifecycle of the EMR cluster, making the "cutover" as simple as pointing the new cluster to the existing Glue Catalog.

Continue Reading

Read S3 + CloudFront vs Nginx + Docker: Where Should You Host a React App?
#DevOps

S3 + CloudFront vs Nginx + Docker: Where Should You Host a React App?

If a React application becomes static files after npm run build, why do teams still package it inside Docker and serve it through Nginx? Let's compare S3 + CloudFront with Nginx + Docker and understand when each architecture makes sense.

Read post ->
Read Why Do We Build React Apps for Production Instead of Running npm run dev?
#Frontend

Why Do We Build React Apps for Production Instead of Running npm run dev?

Have you ever wondered why we run npm run build before deploying a React application instead of simply running npm run dev on the production server? Let's understand what actually happens behind the scenes.

Read post ->