AirflowApache SparkAWSAzureDatabricksDatabricks SQLdbtDelta LakeDelta Live TablesDockerEvent HubsGoogle Cloud PlatformKafkaKinesisKubernetesMLflowPythonSnowflakeSQLTerraformUnity Catalog
Preferred qualifications
Field of studyComputer Science, Data Engineering, Software Engineering (or related)
Study levelBachelor's or higher
Original posting
About ShyftLabs At ShyftLabs, we live and breathe data. Since 2020, we've been helping Fortune 500 companies unlock growth with cutting-edge digital solutions that transform industries and create measurable business impact. We're growing fast, and we're looking for passionate technical leaders who are excited to solve complex data challenges, build modern cloud platforms, and deliver innovative solutions for enterprise clients. The Opportunity ShyftLabs is seeking an experienced Senior / Lead Data Engineer to lead the design, architecture, and delivery of enterprise-scale data platforms for Fortune 500 organizations. This is a highly client-facing leadership role responsible for owning projects from discovery through production deployment. You'll partner directly with client stakeholders to understand business objectives, define technical strategy, architect scalable cloud solutions, and lead engineering teams through successful delivery. The ideal candidate combines deep hands-on expertise with Databricks, Apache Spark, Python, SQL, and modern cloud platforms with proven experience leading complex data modernization initiatives. You'll play a key role in shaping technical direction, mentoring engineers, establishing engineering best practices, and delivering scalable data products that enable analytics, AI, and machine learning.
What You'll Be Doing
Technical Leadership- Lead the architecture, design, and implementation of enterprise-scale data platforms from project inception through production deployment.
Own technical delivery across multiple client engagements while ensuring high-quality engineering standards.
Define solution architecture, technical roadmaps, and implementation strategies aligned with client business goals.
Conduct architecture reviews, code reviews, and establish engineering best practices across project teams.
Mentor and coach Data Engineers while fostering technical excellence and continuous learning.
Serve as the primary technical leader for complex engineering initiatives and critical project decisions.
Client Partnership- Partner directly with Fortune 500 clients to understand business requirements and translate them into scalable technical solutions.
Lead discovery workshops, architecture sessions, and technical planning meetings with both business and engineering stakeholders.
Present solution designs, delivery plans, and architectural recommendations to technical leadership and executive audiences.
Build trusted relationships with client teams while providing technical guidance throughout project execution.
Support pre-sales activities by contributing technical expertise, solution estimates, and implementation approaches when required.
- Design, develop, and optimize enterprise-grade data pipelines using the Databricks Unified Analytics Platform.
Build scalable ETL and ELT frameworks capable of processing large-scale structured and unstructured datasets.
Design and implement Lakehouse architectures using Delta Lake and Medallion design patterns.
Develop high-performance Spark applications for batch and real-time data processing.
Integrate data from enterprise applications, APIs, streaming platforms, and cloud storage solutions.
Ensure data quality, integrity, and reliability through automated validation, testing, and monitoring.
Cloud & DevOps- Architect cloud-native data platforms across AWS, Azure, or Google Cloud Platform.
Implement Infrastructure-as-Code using Terraform or similar technologies.
Build and maintain CI/CD pipelines supporting automated testing and deployment.
Optimize cloud infrastructure for scalability, reliability, security, and cost efficiency.
Monitor platform performance and proactively resolve operational issues.
Data Governance & Security- Implement enterprise data governance frameworks and security best practices.
Configure Unity Catalog, metadata management, lineage, and role-based access controls.
Ensure compliance with organizational security standards and regulatory requirements.
Promote data observability and operational excellence across production environments
Cross-Functional Collaboration- Partner closely with Product Managers, Data Scientists, Analytics Engineers, Machine Learning Engineers, and Software Engineers to deliver high-impact data products.
Enable AI and machine learning initiatives through scalable feature engineering pipelines and production-ready datasets.
Contribute reusable frameworks, accelerators, and engineering standards that improve delivery across client engagements.
What You'll Bring
Bachelor's or Master's degree in Computer Science, Data Engineering, Software Engineering, or a related technical discipline.
8+ years of experience designing and building enterprise-scale data platforms.
5+ years of hands-on experience with Databricks and Apache Spark.
Proven experience leading enterprise data engineering projects from architecture through production delivery.
Strong expertise in Python, SQL, and Spark for large-scale data processing.
Deep understanding of Delta Lake, Lakehouse architecture, and modern data platform design.
Experience working with AWS, Azure, or Google Cloud Platform.
Strong knowledge of ETL/ELT frameworks, distributed computing, and data modeling.
Experience implementing CI/CD pipelines and Infrastructure-as-Code.
Strong understanding of data governance, security, metadata management, and data quality practices.
Experience optimizing distributed data processing workloads for performance and cost.
Excellent communication and stakeholder management skills with experience working directly with enterprise clients.
Demonstrated ability to mentor engineers and lead technical initiatives
Nice to Have
Databricks Certified Professional Data Engineer certification.
Experience with Delta Live Tables, MLflow, Unity Catalog, and Databricks SQL.
Experience with Kafka, Kinesis, Event Hubs, or other streaming technologies.
Hands-on experience with Snowflake, dbt, Airflow, or modern data orchestration tools.
Experience with Kubernetes, Docker, and Terraform.
Knowledge of AI/ML data platforms, Feature Stores, or Retrieval-Augmented Generation (RAG) architectures.
Previous consulting or professional services experience delivering solutions for enterprise clients.
Experience within retail, e-commerce, financial services, logistics, healthcare, or ad-tech environments.
Salary Range
$140,000 – $180,000 (CAD)
Why You'll LoveWorking at ShyftLabs Lead Enterprise Transformations: Design and deliver modern data platforms that power analytics, AI, and digital transformation initiatives for Fortune 500 organizations. Technical Ownership: Drive architecture decisions, influence technical strategy, and lead projects from discovery through production. Growth & Leadership: Mentor talented engineers, shape engineering best practices, and continue developing your technical and leadership skills. Hybrid Flexibility: Work three days per week from our downtown Toronto office. Comprehensive Benefits: 100% employer-paid health, dental, and vision coverage for you and your dependents from day one, along with ongoing learning and professional development opportunities. Inclusion at ShyftLabs We're building something big, and we want you on the journey with us. If you're ready to solve complex data challenges, lead enterprise projects, and build innovative solutions that create measurable business impact, we'd love to hear from you. ShyftLabs is an equal opportunity employer committed to creating a safe, diverse, and inclusive workplace. We encourage applicants of all backgrounds, including ethnicity, religion, disability status, gender identity, sexual orientation, family status, age, and nationality, to apply. If you require accommodation during the interview process, please let us know and we'll be happy to support you.