Job Descrption
•Debugging issues related to Spark performance in a big data environment.
•Coding and architecting of end-to-end applications on modern data processing technology stack (e.g. Hadoop, Cloud, Spark ecosystem technologies)
•Build continuous integration/continuous delivery, test-driven development, and production deployment frameworks
•Lead conversations with infrastructure teams (on-prem & cloud) on analytics application requirements (e.g., configuration, access, tools, services, compute capacity, etc.)
•Platforms: Hadoop, Spark, Kafka, Kinesis, Oracle, TD
•Languages: Python, PySpark, Hive, Shell Scripting, SQL, Pig, Java
•Proficient in Map-Reduce, Conda, H2O, Spark, Airflow / Oozie / Jenkins, Hbase, Pig, No-SQL, Chef / Puppet, Git
•Familiarity with building data pipelines, data modeling, architecture & governance concepts
•Experience implementing ML models and building highly scalable and high availability systems
•Experience operating in distributed environments including cloud (Azure, GCP, AWS etc.)
•Experience building, launching and maintaining complex analytics pipelines in production
Key Skills:
Spark Developer