Senior AI or ML Engineer – ADB, Pyspark, Scala, SQL, DevOps

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by diversity and inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health equity on a global scale. Join us to start Caring. Connecting. Growing together.

We are looking for a talented and hands-on Azure Engineer to join our team. The ideal candidate will have significant experience working on Azure, as well as a solid background in cloud data engineering, data pipelines, and analytics solutions. You will be responsible for designing, building, and managing scalable data architectures, enabling seamless data integration, and leveraging advanced analytics capabilities to drive business insights.

Primary Responsibilities:

• Azure Platform Implementation:
  • Develop, manage, and optimize data pipelines using AML workspace on Azure
  • Design and implement end-to-end data processing workflows, leveraging Databricks notebooks and jobs for data transformation, modeling, and analysis
  • Build and maintain scalable data models in Databricks using Apache Spark for big data processing
  • Integrate Databricks with other Azure services, including Azure Data Lake, Azure Synapse, and Azure Blob Storage
• Data Engineering & ETL Development:
  • Design and implement robust ETL/ELT pipelines to ingest, transform, and load large volumes of data
  • Optimize data processing jobs for performance, reliability, and scalability
  • Use Apache Spark and other Databricks features to process structured, semi-structured, and unstructured data efficiently
• Azure Cloud Architecture:
  • Work with Azure cloud services to design and deploy cloud-based data solutions
  • Architect and implement data lakes, data warehouses, and analytics solutions within the Azure ecosystem
  • Ensure security, compliance, and governance best practices for cloud-based data solutions
• Collaboration & Analytics:
  • Collaborate with data scientists, analysts, and business stakeholders to deliver actionable insights
  • Build advanced analytics models and solutions using Databricks, leveraging Python, SQL, and Spark-based technologies
  • Provide guidance and technical expertise to other teams on best practices for working with Databricks and Azure
• Performance Optimization & Monitoring:
  • Monitor and optimize the performance of data pipelines and Databricks jobs
  • Troubleshoot and resolve performance and reliability issues within the data engineering pipelines
  • Ensure high availability, fault tolerance, and efficient resource utilization on Databricks
• Continuous Improvement:
  • Stay up-to-date with the latest features of Databricks, Azure, and related technologies
  • Continuously improve data architectures, pipelines, and processes for better performance and scalability
  • Propose and implement innovative solutions to meet evolving business needs
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• 10+ years of hands-on experience with Azure ecosystem
• Solid experience with cloud-based data engineering, particularly with Azure services (Azure Data Lake, Azure Synapse, Azure Blob Storage, etc.)
• Experience with Databricks notebooks and managing Databricks environments
• Hands-on experience with data storage technologies (Data Lake, Data Warehouse, Blob Storage)
• Solid knowledge of SQL and Python for data processing and transformation
• Familiarity with cloud infrastructure management on Azure and using Azure DevOps for CI/CD
• Solid understanding of data modeling, data warehousing, and data lake architectures
• Expertise in building and managing ETL/ELT pipelines using Apache Spark, Databricks, and other related technologies
• Proficiency in Apache Spark (PySpark, Scala, SQL)
• Proven solid problem-solving skills with a proactive approach to identifying and addressing issues
• Proven ability to communicate complex technical concepts to non-technical stakeholders
• Proven excellent collaboration skills to work effectively with cross-functional teams

Preferred Qualifications:

• Certifications in Azure (Azure Data Engineer, Azure Solutions Architect)
• Experience with advanced analytics techniques, including machine learning and AI, using Databricks
• Experience with other big data processing frameworks or platforms
• Experience with data governance and security best practices in cloud environments
• Knowledge of DevOps practices and CI/CD pipelines for cloud environments

At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone–of every race, gender, sexuality, age, location and income–deserves the opportunity to live their healthiest life. Today, however, there are still far too many barriers to good health which are disproportionately experienced by people of color, historically marginalized groups and those with lower incomes. We are committed to mitigating our impact on the environment and enabling and delivering equitable care that addresses health disparities and improves health outcomes — an enterprise priority reflected in our mission.