About
News UK
TL;DR: To achieve operational excellence in applied data science delivery aim for: 0-day Handovers, 1-day Prototyping and to declare projects as Done when Completely Done. Work backwards from these goals, conducting a gap analysis between those and current team capabilities, to identify process, tooling and governance initiatives to reach them.
R&D vs. BAU
Successfully applying data science in an organisational setting is almost always a delicate balancing act.
It usually takes the shape of a business function that exhibits Research & Development (R&D) characteristics, trying to improve Business-As-Usual (BAU) processes. This can be a challenging environment to succeed in. There is high value but also high risk in trying to deliver data science-driven impact, and teams should aspire to be both effective and efficient.
In a way, the Google Trend on “data science” — and on relevant terms for that matter — depicts this value-risk balance, though one can give many interpretations to the time series. Data science (DS) has consistently over the years been a very attractive field and indeed DS functions have sprung up in practically almost every organisation worldwide during the last half decade. But the post Jan 2020 plateau in “data science” — by search volume at least — coinciding with the timing of the covid-19 pandemic schematically captures a vulnerability of data science, in the sense that in times of crisis, it may be seen as less appealing, although again other factors could be jointly at play here.
Indeed an R&D practice — by name or if perceived as such, especially if not established or proven — may be among the first to be put under the microscope by senior management in times of uncertainty. This is a key reason why these high-promise business DS functions need to be comfortably ‘bullet proof’ before tackling even more aspirational goals or expanding their remit.
Strategy overview
At News UK, we are privileged to be operating a successful Data Science practice, which has already delivered bottom line impact in a number of areas, across our key titles such as The Times and The Sunday Times, and The Sun. These include content recommendation engines, newsroom insight tools, and customer models to name a few. To accelerate its next level of growth, we decided to devise a strategy that solidifies delivery fundamentals, reduces operational risk factors, and sets strong foundations for the future — at which point our ambition for the function will further amplify.
We are sharing this iteration of our Data Science Strategy in the hope that other functions may benefit from adopting its principles and goals.
Three main pillars make up the strategy: People Development, Operational Excellence and instilling a Product Focus onto our projects. There is an element of hierarchy between these, as our people — their development and wellbeing — comes first, on top of which we form our goals to improve the way we work together (Operational Excellence), which is a further bedrock for how our solutions are built. You might reasonably notice that these three pillars resemble Dr. Harold Leavitt’s ‘People, Process, Technology’ framework, though this has only been a minor inspiration in developing the strategy. The people development and product focus pillars can take many shapes and forms, and the details can be very business specific or already mainstream. In contrast, the way we have articulated Operational Excellence is new, and can be generalised and more widely applied. It is this focus area that we elaborate on below. The term itself is not new — presumably coined by Dr. Joseph M. Juran in the 1970’s — but we use it here literally, and not in its original business context.
“0 / 1 / Done”
Our strategy’s Operational Excellence (OE) workstream aims to improve our internal ways of working and the coherence of how we deliver as a team. We do this by reviewing our technical capabilities, business knowledge, project governance and controls, and by looking for opportunities for automation to streamline and safeguard delivery.
We have three complementary and at the same time seemingly conflicting, aspirational goals.
We aim for:
- 0-day Handovers,
- 1-day Prototyping, and to call our work…
- Done when Completely Done, also interchangeably referred as Definition of Done
The code name is “0 / 1 / Done”. (It starts from 0 because… you know… Python.)
The first — and minor — thing to note is that “handovers” may not sound the most exciting concept to anchor a strategic goal to. However, it does not take much effort to appreciate the level of operational excellence that comes from aspiring for such a high level of flexibility when it comes to project ownership (practically unaided), and the de-risking in knowledge loss and business discontinuity it can support.
[ 0-day Handovers ]
In more detail, through 0-day Handovers we aim to be able to change project ownership with practically zero need for peer-to-peer support. The idea is that any team member can be on-boarded onto a new project after only being given a set of pointers through which they are able to independently pick up its essence and practical ownership. Docs, wiki sites, code repos and/or even video demonstrators could all be part of the enabling mix; they need to be so thorough, complete and up-to-date that no tacit knowledge transfer is otherwise necessary.
[ 1-day Prototyping ]
1-day Prototyping refers to our ability to be able to deliver a working prototype solution with usable output that performs above the existing business benchmark for problems that can be naturally formulated as “typical ML” supervised and unsupervised learning ones.
For instance, we might want to prototype a model to support an upcoming marketing campaign, and for it to perform better than a simple heuristic-based approach or a random selection. In our context, the space of problems we want to tackle in this manner is fairly wide. Marketing and advertising related customer models such as segmentations, predictive churn, lifetime value (LTV), engagement, cross-/up-sell, lookalike, acquisition models or forecasting ones could all be part of the mix.
The swift delivery of such prototypes allows us to (i) instantly bring to light and unblock downstream engineering dependencies whilst also (ii) quickly shift the conversation from how long complex modelling development takes to the BAU change management needed to embed the new capability and put it into action, a ‘last mile’ challenge ML projects frequently stumble onto. This way the Data Science team are not seen as a source of delay.
Clearly not all business projects are typical ML ones. For instance one cannot build a recommender engine overnight. We are also aware that a single-day model can always be further refined and improved, however this is something we can follow up on with less pressure after the business has a working prototype.
For a 1-day solution to be even considered, our Data Scientists need to have clear business requirements, and indeed this goal can guide conversations with Business Analysts and Product Owners to improve ways of working. The act of rapidly prototyping a solution also helps better explain how data science works and what business inputs it needs in advance of project initiation, as well as helping to reveal nuances in the problem that may provide early evidence of whether the project is likely to succeed or fail.
And perhaps not least, delivering 1-day prototypes via 1-day hackathons can at the same time be a fun, well-defined team building exercise.
[ Definition of Done ]
Definition of Done formalises the ML project delivery process by listing all steps it must undertake to ensure that solution development ticks all boxes, from data checks, modelling and validation to documentation and having explainability and monitoring in place, before final project sign-off.
This may not sound novel, but it is all too rarely followed in the world of applied data science. It also supports the handover goal. Furthermore, it is a vital check to avoid us hastily re-allocating data scientists to upcoming projects before everything is complete, while it provides Project Managers with guidance and clarity. It can extend to how we articulate ‘kill criteria’ for projects where we should responsibly ‘fail fast’. One of the initial checks is for Data Scientists to consult other teams — such as Analytics / Business Intelligence functions — before kicking off solution development, to confirm there has not already been separate, sufficient work on a given project. This way we encourage cross-functional collaboration and we protect against work duplication.
As part of this goal, we have a Project Completion Matrix, listing all our team’s projects against key project lifecycle stages and requirements. We have scored the corresponding cells with completion values between 0–100% to the best of our ability. This way we have mapped our technical debt, and have given ourselves an overall project completion score to aim to improve. The areas of most interest were the less come-across ones such as data drift handling and model explainability, rather than the unavoidable must-have’s of model selection and validation. Ultimately, all of these stages shall form a check in a Definition of Done list. In completing this matrix, we will naturally be looking at project agnostic solutions (tools) where appropriate, given that not all projects require bespoke solutions for certain areas.
Working backwards
Overall, our aim is to work backwards from the three ambitious goals of “0 / 1 / Done” to identify what is missing to achieve them and to then deliver more granular, bite-size initiatives. We list some of these below. These will naturally differ across organisations, given each Data Science team’s scope and level of operational maturity. Either way, these initiatives can be used to simply track and measure improvements in existing capabilities or as a refresher for a team.
0-day Handovers:
- Code development harmonisation (standards and conventions); to ensure easier, developer-agnostic code readability, maintainability and ultimately ownership transfer
- Automated setup scripts and procedures for our Cloud platform services; aiming primarily at easier Storage dataset and Compute instance creation, to support Data Scientist onboarding, set-up and project pipeline migration
- Team knowledge management space revamp; to turn the team’s project documentation wiki pages into an ‘one-stop shop’ for project information
1-day Prototyping:
- ‘Learn our key datasets’ training; to ensure every team member has good knowledge of critical product datasets covering key user, content and behavioural information to more speedily pick up on feature engineering and model development
- Feature Store; to further support on the above
- Training and knowledge sharing on: efficient Big Data querying, Agile and Lean principles, business domains and stakeholders; all of which support in their own right the overarching aim of speedy prototyping
- Tech stack review; to seek further opportunities to enhance tooling, such as by the use of AutoML
Definition of Done:
- Project lifecycle decision gates and checklist; for thorough governance in project delivery, including capturing ethics considerations
- Code review minimum check criteria articulation; to support in clarifying what a good code review (pre sign-off project requirement) consists of
- Explainability module creation (project-agnostic); to support in the delivery of the less frequently catered-for Explainability requirements
- Data Subject Access Request (DSAR) response plan; capturing the unearthing of a person’s digital data use by Machine Learning models in an organised and timely manner, as per their GDPR access right
- Completion of the …Project Completion Matrix; as described above
Measuring success
As a Data Science team, we have time-boxed the delivery of this strategy so it has momentum, and even though it is internal, we are advertising it more widely to other teams so we can ring-fence capacity for its delivery, and so we can hold ourselves more accountable for its progress. Our plan is to measure success for each OE goal.
0-day Handovers:
- Initiate handover of a project from one team member to another giving them zero notice and zero instructions other than the project’s documentation pointers. Monitor and measure the level of assistance they needed in the ownership transition process; the lower the better. Evaluate after a week.
1-day Prototyping:
- Provide the team with a typical ML business problem which is properly defined. Request that they prototype a solution within one day, as a full-day hackathon. Evaluate experiment.
Definition of Done:
- Get all projects across all stages in the Project Completion Matrix to an overall completion score >90%. Ensure project lifecycle checklist is understood and adopted by Project Managers responsible for DS projects.
Healthy tension
It’s worth facing into the healthy tension between the three Operational Excellence goals as well as how they complement each other.
- 1-day Prototyping invites our team members to be quick and somewhat sloppy (within some constraints) to get to value. At the same time, 0-day Handovers and Definition of Done invites them to be thorough and detailed.
- 1-day Prototyping and 0-day Handovers may not necessarily invite them to expand on a delivered solution but instead to ‘wrap it up’ even if partially complete, assuming the core deliverable has been handed over, while Definition of Done necessitates that all steps and project gates are followed with a justification requested when not so for certain stages.
- The pair of 1-day Prototyping and Definition of Done encourages the team to engage with and serve external stakeholders when 0-day Handover on its own is focused on the internal, within-team ways of working with a focus on empathy and support for fellow data scientists.
We hope you see value in this strategy as we do. Above all else, the key principle is to keep ambitions high. Even if in the end the vision is not achieved 100%, we believe that such high aspirations can significantly improve the maturity, well being and stature of a Data Science team within an organisation.
Special thanks to
for reviewing and sharing valuable feedback.
