As an Azure Data and ML Engineer, you will work closely with this company's data scientists and data engineers to design, build and deploy data engineering and machine learning solutions for business usage in a production environment on Azure, .i.e. design self-running software to automate data pipelines and predictive models. The consultant will require deep hands on experience in all phases of a machine learning lifecycle to harden, build, deploy, scale, tune, monitor these machine learning models for operationalization of solutions using them.
Responsibilities will include:
* Design and package deployments of data pipelines and machine learning models for production on Azure environments working very closely with data scientists and data engineers.
* Design and code production ready high-quality data application code on Azure databricks/ Azure functions/ Azure Data factory/Synapse Spark pools as well as harden experimentation code developed by Data Scientists and deploy using Azure ML services and MLOps.
* Select appropriate datasets and data representation methods to enable quick turnaround time for feature engineering using key Azure services like Azure Databricks/Functions/Data Factory/Synapse Spark Pools
* Define and incorporate software engineering practices into data engineering pipelines, model implementation code and deployments
* Choose, design and use the right ML libraries, tools, programming languages, deployment infrastructures and frameworks for scaling data pipelines and experimented models in production on Azure environment
* Exploring and visualizing data to gain an understanding of it, then identifying differences in data distribution that could affect performance when deploying the model in the real world
**Candidates must be willing to travel up to 80%
* Bachelor's degree in Computer Science, Engineering, Statistics, Technical Science or 3+ years of IT/Software Development/Programming experience
* Minimum 1-2 years' experience of building and deploying production applications for data products as well as deploying of ML applications using Azure native services
* Minimum 1-2 years' experience in designing and deploying production grade data products / pipelines and ML models Azure Infrastructure
* Minimum 1-2 years' experience setting and using model parameters and hyperparameters i.e. containerize and externalize to tune and scale the model for large datasets. Must have experience of deploying containerized models and ML pipelines using Docker, Azure Kubernetes Services (AKS) or equivalent technologies
* Minimum 1-2 years' experience of building, containerizing and deploying end to end automated data and ML pipelines using technologies like Spark and Azure Native services in a large-scale production environment
* Minimum 3-4 years of experience of using Azure data factory, Azure databricks, Azure functions, Azure Synapse Analytics, Delta lake for data engineering and building scaled data pipelines as well as minimum 1-2 years' experience of using Jupyterhub, Anaconda , Spyder, Azure Databricks, Azure ML Services, ML Ops for model engineering, deployments and monitoring.
* Minimum 1-2 years' experience with performance engineering of large data pipelines and with very large-scale datasets on a large distributed infrastructure using technologies like Azure Databricks, Azure data factory, Azure functions, Azure logic apps etc.
* Minimum 1-2 years' experience using tools like MLFlow, Azure Machine Learning Service , Azure Kubernetes servicesfor managing end-to-end machine learning lifecycle for tracking experiments, packaging ML code and deploying models from various ML libraries to model serving and inference platforms
* Minimum 2+ years of strong programming skills in at least 2 languages from Python, Scala (and Spark), R on Azure.
* Minimum 1-2 years' experience working in an Agile environment
* Deep understanding of software engineering and software architecture principles for building and deploying business critical applications.
- Relevant certifications in AI, ML or Data Engineering from Microsoft
- Understanding of all phases of a complete data engineering and Data Science Life-cycle
- Strong Experience delivering scaled solutions that generated business outcomes and impacts
- Knowledge and Understanding of Cloud, Hybrid and On-Premise DevOps