CloudFactory is seeking a highly skilled and experienced Senior Data Engineer to join our dynamic team. As a global leader in combining people and technology to provide a cloud workforce solution, CloudFactory works on numerous machine learning and core business data processing projects. We are dedicated to creating economic and leadership opportunities for talented individuals in developing nations.
We are seeking a Lead Data Engineer with substantial experience in data warehousing to help mature our data platform.
As a Senior Data Engineer at CloudFactory, you will play a key role in developing and maintaining our data infrastructure to support our growing operations. You will have the opportunity to work with cutting-edge technologies and collaborate with a diverse and passionate team to scale our data processing capabilities. If you are passionate about making a difference in the world through meaningful work, enjoy building relationships, and thrive as part of a unified team, CloudFactory is the place for you.
While we offer remote work flexibility, we expect this role to be office-based two days a week.
Responsibilities
The successful candidate will translate user requirements into actionable insights by leveraging existing tools and technologies. Your responsibilities will include:
- Regular interaction with PMs/stakeholders to understand their requirements
- Investigation to ascertain if the requirements can be met using existing technology and data resources
- Data extraction using Python and other tools like Fivetran
- Data modelling using SQL and the concept of Star and Snowflake schema
- Assisting in cost and performance optimization choices within the data warehouse. Communicating reasoning and logic behind opinions to help make optimal choices
- Write data quality and other metadata collection systems within data pipelines to promote observability and recoverability of pipelines
- Write unit and integration tests to bolster system reliability
- Proactively identify the areas of improvement, propose solutions and manage technical debt
- Mentoring, guiding and line management of three/four team members.
Tech Stack
Our tech stack:
- Data extraction and orchestration tools - Fivetran, Prefect
- Data modelling tool - DBT
- Data Warehouse - Snowflake
- Data quality cheeks - dbt, Elementary
- Data visualisation - Tableau and Quicksight
- Data catalogue and lineage - Atlan, dbt
- Source code management - Github
- Infrastructure as Code - Terraform, Cloudformation
- Programming Language - Python
- AWS services (ECS, S3)
Interview Process
Candidates will be expected to complete an initial coding and analytical thinking test, followed by an interview with Data Architect and Senior Product Manager where they will be expected to talk through previous projects, including approaches taken and results achieved.
Successful candidates will then meet with the Engineering Director before a final interview with the CTO.
The process is expected to take 1-2 weeks.
Requirements
The ideal candidate will possess:
- Proficiency in requirement capture and technical design
- Mastery in Data pipeline development
- Proficiency with extracting data from source systems using Python
- Mastery in transforming data using SQL and data transformation tools
- Mastery of data modelling concepts and practices, preferably with Star schema
- Proficiency in writing data quality tests
- Proficiency in writing end-to-end tests using test scenarios approach including test data creation and management
- Proficiency in data pipeline observability techniques (e.g. monitoring, alerting, anomaly detection etc)
- Some experience in building reports and dashboards
- Data Warehouse administration
- Proficiency with Data Warehouse administration (e.g. roles, permissions, compute, performance)
- Data security and Governance
- Proficiency in Data security and Governance (e.g. RBAC, encryption, masking, data classification and tagging, time travel and data recovery) practices
- Some knowledge of using data catalogue and data lineage tools preferably Atlan and dbt documentation
- Performance and cost optimization
- Proficiency in performance and cost optimization of database queries using tools and techniques
- Proficiency with line management to manage a team of Data Engineers and Data Analysts/Analytics Engineers
- Excellent communication skills, able to translate complex data into understandable insights and narratives.
Benefits
Location dependent, but can include
- Office snacks provided Monday-Friday, lunches