Job Summary:
We are seeking an experienced Data Pipeline Engineer with 13+ years of experience to design, develop, optimize, and support scalable batch and real-time data pipelines in an Investment Banking environment.

Key Responsibilities:

• Design, develop, test, deploy, and maintain batch and real-time data pipelines.
• Process high-volume and multi-terabyte datasets using distributed and cloud-based technologies.
• Build data ingestion, transformation, validation, enrichment, and delivery processes.
• Optimize pipeline performance, reliability, cost, and recoverability.
• Implement data quality, reconciliation, lineage, monitoring, alerting, and error-handling mechanisms.
• Integrate pipelines with AWS cloud platforms, data lakes, warehouses, databases, APIs, and messaging systems.
• Collaborate with architects, engineers, analysts, data scientists, and business stakeholders.
• Maintain technical documentation, data mappings, pipeline designs, and operational runbooks.

Required Skills:

• 13+ years of experience in Data Engineering, ETL/ELT, or Data Integration.
• Strong SQL and Python skills.
• Hands-on experience with AWS, including Lambda and Kafka.
• Experience with Apache Airflow or Azure Data Factory.
• Strong knowledge of relational databases and large-scale data processing.
• Experience with logging, monitoring, retry logic, idempotency, and failure recovery.

Preferred Skills:

• Data Modeling, Metadata Management, Data Governance, and MDM.
• Docker and Kubernetes.
• Financial/Investment Banking industry experience.