Overview

We are seeking an experienced Lead Data Engineer to build an Oil and Gas data processing prototype leveraging Azure Databricks, ADLS Gen2, and Delta Lakehouse medallion architecture. This role focuses on data mastering using exact matching and geospatial calculations. The prototype must be delivered rapidly to enable extension into a fully-fledged solution.

Responsibilities

  • Design and build a data processing prototype for the Oil and Gas domain
  • Develop scalable data pipelines using Azure Databricks and PySpark
  • Implement Delta Lakehouse medallion architecture for structured data flow
  • Manage and organize data storage in ADLS Gen2, working with Parquet and JSON formats
  • Perform data mastering using exact matching and geospatial calculations
  • Write and optimize queries using Spark SQL for efficient data transformation
  • Collaborate with stakeholders to ensure the prototype meets business requirements
  • Deliver the prototype rapidly to support extension into a full-scale solution

Requirements

  • 5+ years of experience in data engineering
  • At least 1 year of relevant leadership experience
  • Expertise in Azure Databricks, Azure Data Lake Storage, and ADLS Gen2 (Parquet and JSON)
  • Proficiency in PySpark and Spark SQL
  • Background in building data pipelines using Delta Lakehouse medallion architecture
  • Capability to deliver prototypes rapidly under tight timelines
  • Excellent written and verbal communication skills in English (B2+ level)

Nice to have

  • Skills in data matching and deduplication using geospatial calculations
  • Familiarity with Databricks DLT, Databricks Unity Catalog, and Databricks Asset Bundles
  • Oil and Gas domain experience

[GTS] Benefits (generic, except India)

  • International projects with top brands
  • Work with global teams of highly skilled, diverse peers
  • Healthcare benefits
  • Employee financial programs
  • Paid time off and sick leave
  • Upskilling, reskilling and certification courses
  • Unlimited access to the LinkedIn Learning library and 22,000+ courses
  • Global career opportunities
  • Volunteer and community involvement opportunities
  • EPAM Employee Groups
  • Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn