Overview

We are seeking a visionary and experienced Lead Data Engineer to join our Data Engineering team.

In this role, you will lead a Platform Team, driving the design, development, and scaling of data pipelines in Databricks with PySpark on Microsoft Azure for our AI Factory team. This is a unique opportunity to shape the future of our data platforms, influence technical strategy, and mentor a talented team, all while working at the forefront of big data and cloud engineering.

Responsibilities

  • Lead the design and architecture of cloud-native analytical solutions using Big Data and NoSQL technologies
  • Oversee the development and optimization of scalable data pipelines in Databricks with PySpark on Azure
  • Own the strategy for building and maintaining data lakes and warehouses, ensuring reliability, performance, and scalability
  • Define and implement ETL/ELT workflows and best practices for data collection, cleaning, and structuring
  • Establish and enforce data quality, lineage, and monitoring frameworks across the team
  • Collaborate closely with ML, analytics, and business teams to deliver production-ready datasets and solutions
  • Conduct and lead code reviews, setting technical standards and fostering best practices
  • Mentor and coach data engineers, cultivating a high-performance and collaborative team culture
  • Champion CI/CD methodologies in data engineering workflows using tools like Jenkins or GitLab CI/CD
  • Drive requirements gathering and solution alignment with architects, technical leads, and cross-functional teams
  • Engage with stakeholders at all levels to understand business processes, model input data, and ensure deliverable alignment with strategic goals

Requirements

  • 6+ years of experience in Data Engineering or a related field, with at least 2 years in a technical leadership role
  • Deep proficiency in Python and PySpark
  • Extensive hands-on experience with Databricks and Microsoft Azure cloud services
  • Strong background in software version control tools (e.g., GitHub, Git)
  • Proven track record with CI/CD frameworks such as Jenkins, Concourse, or GitLab CI/CD
  • Demonstrated expertise in architecting and building scalable, robust, and highly available data solutions
  • Excellent problem-solving, analytical, and stakeholder management skills
  • Experience in mentoring and leading engineering teams

Nice to have

  • Experience with additional programming languages such as Java, SQL, or Scala
  • Knowledge of SAP BTP or similar enterprise data platforms
  • Familiarity with agile development methodologies
  • Experience in strategic planning and technical roadmap development

Bulgaria

  • Opportunity to Engineer your Future and to drive the world’s digital transformation with top industry clients
  • Personal development program that will allow you to be valued for your strengths
  • Wide range of professional trainings and workshops
  • Being part of a collaborative, fast-growing, and innovative design team
  • Established and accelerated growth toward different career paths, competencies, and roles
  • Broad projects variety and possible mobility between projects over the time
  • Collaboration in a multicultural environment and exchange of best practices with colleagues around the world
  • Varied social benefits, Sports, Transportation and Health programs
  • Work-life balance and flexible schedule, team buildings and sport opportunities
  • Modern office/collaboration spaces (incl. new Infinity Tower business center, Sofia)
  • Hybrid By Design - we provide you with the best productivity options from the 2 worlds. Meet, socialize and enjoy F2F time with your colleagues, while working from the modern EPAM's office for a few days per week and benefit from the EPAM's virtual working environment - making you able to be productive and work from remote for the rest of the week