Koniag Emerging Technologies, LLC is seeking a Senior Databricks Engineer with a Secret security clearance to design, develop, and deploy scalable data infrastructure using the Databricks platform. The ideal candidate will have 8+ years of dedicated Data Engineering experience and expert-level proficiency in PySpark and complex SQL.
Requirements
- Design, develop, and deploy scalable data infrastructure using the Databricks platform
- Drive key data engineering initiatives
- Collaborate with project managers, team members, and business owners
- Architect and implement scalable, highly available data pipelines
- Establish comprehensive data quality standards and implement monitoring dashboards
- Write clean, efficient, and production-ready code using PySpark and SQL
- Design and build robust data ingestion pipelines to reliably pull from various third-party external systems
- Implement automated deployments using Azure DevOps
- Develop and maintain custom Python libraries and modules
- Apply foundational and advanced data warehousing concepts
- Monitor, troubleshoot, and optimize Databricks clusters and Spark jobs
- Securely share data with external BI tools and develop dashboards natively within the Databricks environment
- Serve as a technical SME for the data engineering team
Benefits
- Health, dental and vision insurance
- 401K with company matching
- Flexible spending accounts
- Paid holidays
- Three weeks paid time off