Share this Job

Go Back

Python Developer with Unix cron & AWS

Sunnyvale, CA
2018-02-16 16:50:14
Job Type: Contract

JobDescription:

Minimum 9+ years of softwareprogramming experience

Strongexperience in ETL

SQLscripting experience

StrongPython development experience around data management.

Strongin Unix cron not just exposure

Musthave the ability to work in a dynamic, fast-paced environment

Strongcommunication skills to interact with Agile team members

 

DataSources: Cloudbased as well as on premise example: SAP CRM, ECC, people soft, PLM,Calliduscloud

 

High-levelarchitecture:

Amazon S3 bucket have foldersfor specific source files delivered by Fusion middleware. Data quality check isnot handled by the sources.

AnacondaPython based programs are in place to clean any abnormalities in the sourcefiles and re-write them back in their respective folders this is to avoid loadfailures.

mostof the known errors are handled using this mechanism, moving forward you willrequire to enhance these scripts to include a fix for any new errors.

currentlymechanism is processing one file at a time, target is to move it to parallelprocessing.

Nextis a Python program for data pipeline, based on AWS API and snowflake API. Thisis towrite files on snowflake EDW

Thereare 3 different layers on the snowflake side Pre=processing, staging and basetables

Unixcron is used for scheduling purpose.

 

 


Key Skills: