Data Engineer (On-site)

Job Description - Data Engineer
Position: Data Engineer
Engagement: Full-time, 1-year contract (extendable)
Location: London, UK, on-site
Languages: English (working proficiency)
Expectations: On-site, 5 days a week

Role Overview
We are seeking a Data Engineer (Data Ingestion & Cloud Modernization) to design, build, and maintain scalable data ingestion pipelines across modern cloud platforms. The role focuses on implementing reliable batch and near-real-time data pipelines, modernizing legacy ETL processes, and enabling data platforms using Azure and Snowflake technologies.

This position is well-suited for mid-level data engineering professionals with hands-on experience in data ingestion, cloud-based data platforms, and Python-based pipeline development. The role requires strong collaboration with data platform, analytics, and application teams to ensure reliable, secure, and high-quality data delivery across the organization.

Key Responsibilities
Data Ingestion & Pipeline Engineering
• Design, build, and maintain robust data ingestion pipelines (batch and near-real-time) from diverse sources such as databases, APIs, files, and event streams.
• Migrate legacy ETL/ELT processes to modern Azure and Snowflake architectures through re-platforming and refactoring initiatives.
• Implement incremental data loads, CDC (Change Data Capture) patterns, schema evolution handling, and backfill/reprocessing strategies.
• Standardize ingestion workflows by developing reusable frameworks, templates, and best practices.

Azure & Snowflake Modernization
• Develop cloud-native ingestion solutions using Azure services such as:
o Azure Data Factory / Synapse Pipelines for orchestration.
o Azure Databricks and/or Spark for transformations.
o Azure Storage / ADLS Gen2 for landing and staging layers.
o Event-driven services (e.g., Event Hubs) where applicable.
• Build ingestion and loading patterns into Snowflake using:
o Snowflake stages, file formats, and COPY INTO commands.
o Snowflake Streams and Tasks where appropriate.
o Data modelling foundations for raw - curated data layers using dbt.

Real-Time Data Ingestion
• Build components to capture streaming data sources.
• Develop real-time transformation pipelines and ensure timely delivery of data to downstream consumer services.

Platform Enablement & Reusable Components
• Develop shared service components, Python libraries, and integration templates to accelerate delivery
across Data Engineering and Application teams.
• Follow integration best practices and ensure consistency across digital services and data pipelines.

Data Quality, Reliability & Observability
• Implement data validation and quality checks (completeness, freshness, duplicates, schema drift).
• Ensure pipelines are reliable and recoverable through idempotency, retry logic, re-run capabilities, and alerting mechanisms.
• Implement observability through logging, metrics, lineage metadata, and pipeline health dashboards.

Security, Governance & Ways of Working
• Apply security best practices including least privilege access, secrets management, encryption, and secure connectivity.
• Follow data governance standards such as naming conventions, data retention policies, classification, and documentation.
• Collaborate within agile delivery processes including code reviews, CI/CD pipelines, iterative release planning, and cross-team coordination.

Qualifications
• Education: Bachelor’s degree in computer science, Information Technology, Engineering, or a related field (or equivalent practical experience).
• Experience: 3–6 years of hands-on data engineering experience with a strong focus on data ingestion and pipeline development.
• Skills:
o Experience building production pipelines using Azure Data Factory, Databricks, or Synapse.
o Strong SQL skills and experience working with modern cloud data warehouses, ideally Snowflake.
o Proficiency in Python for data processing, automation, and pipeline utilities.
o Experience with data ingestion patterns such as batch processing, CDC, and streaming ingestion.
o Familiarity with cloud data architecture concepts and modern ELT practices.

Soft Skills
• Strong analytical and problem-solving abilities.
• Collaborative mindset with the ability to work across data, engineering, and application teams.
• Attention to detail with a focus on data reliability and quality.
• Proactive approach to improving data platform capabilities and automation.
• Effective communication and documentation skills.