Data Architect, Data and ETL Quality
Numerator | Engineering | Ottawa, Ontario
Numerator is a market intelligence firm that brings together omnichannel marketing, merchandising and sales data to make pursuing new possibilities simple for brand, retail and agency clients. Owned by Vista Equity Partners, Numerator is the only company in the marketplace to connect omnichannel purchase data (powered by the InfoScout, OmniPanel) and comprehensive path data to deliver an unmatched view of the consumer shopping and purchase experience.
The ideal candidate is an experienced data engineer and can serve as a thought leader on the team for organizing structured and unstructured data, managing data pipelines, building data models, and scaling our data warehouses. Our data provides the best understanding of consumer purchase behavior, that needs daily updates and availability across many countries.
Our platform leverages a data store that performs heavy computation at runtime, as opposed to pre-materializing our datasets. In our industry, this has proven to be a huge business advantage, but becomes a challenge as our concurrent user count increases. Here’s a comprehensive list of challenges that may present themselves as part of the role:
- What options exist to scale our data warehouse to support growing concurrency and report complexity?
- How would we process and organize a new stream of semi-structured data with 10m events/day?
- What solutions should be solved with a MPP data warehouse solutions versus a map-reduce system?
- Determine SLA strategy with Product & Business partners for new data sets
- When should we employ an ETL strategy versus and ELT strategy?
- Partner with the product teams to develop data testing strategies
- Create test plans and test cases
- Develop data quality automation tests and data pipelines
- Test large and complex data sets
- Ensure high level of data integrity
- Define strategies for continuous improvement of data quality
Along with a supporting cast of data engineers and application developers, this senior data warehouse lead will help provide answers and implement the solutions that allow Numerator to continue to build out the world's largest single-source set of purchase data across brands and retailers.
- BS or MS in Computer Science or equivalent work experience
- 4 + years of experience in the data warehouse space
- Certified AWS solution professional and Big Data Expert
- Expert in SQL, including advanced analytical queries
- Expert in one dynamic scripting language (Python preferred)
- Expert in one compiled OO language (Java, C++, or C#)
- Experience working with an MPP data warehouse (Redshift)
- Experience in ETL design and tools (Talend, Pentaho Kettle, Informatica)
- Experience integrating Mongo and Elastic Search
- Experience with schema design and data modeling
Exceptional candidates will have:
- Experience with Cloud Migrations
- Experience creating multi-cloud infrastructure automation using Chef and Ansible.
- Experience in data science and building models (R, Python/Pandas)
- Experience creating DW data models
- Experience partitioning and bucketing for high performance
- Experience in working with map-reduce systems (Hadoop, Apache Spark)
- Experience with NoSQL Systems (Mongo, Cassandra, DynamoDB)
- Experience using terraform and/or ansible for infrastructure deployment
- Experience using variety of Amazon Web Services (EC2, ELB, RDS)