Staff Big Data Engineer (Python/AWS)

We are looking for a highly experienced Staff Big Data Engineer to lead a critical Data Migration project for our client, designing the migration strategy and architecting scalable data pipelines. The role requires deep expertise in AWS, Spark, and modern table formats like Apache Iceberg.

• Design and execute a comprehensive data migration strategy from on-premise/legacy systems to a modern AWS ecosystem.
• Design, develop, and optimize robust big data pipelines using Python, Apache Spark, and Apache Airflow.
• Deploy and manage Spark jobs on Kubernetes (K8s/EKS) and utilize AWS services (S3, EMR, Glue, Athena) to build highly scalable data architectures.
• Implement and maintain top-tier security standards across the Hadoop ecosystem and AWS infrastructure.
• Act as the ultimate technical authority on the project, mentoring senior engineers and collaborating with stakeholders.
• Implement and optimize data storage using Apache Iceberg.

• Extensive Experience (8+ years) in data engineering with large-scale distributed systems and data migrations.
• Deep knowledge of AWS data services, including S3, EMR, Glue, and Athena.
• Experience running Spark jobs on K8s or EKS.
• Exceptional hands-on experience with Apache Spark and strong programming skills in Python.
• Solid experience building and managing complex DAGs with Apache Airflow.
• Production experience with Apache Iceberg.
• Strong understanding and practical experience with Hadoop Ecosystem Security.