Summary: This post describes how targeted, modest interventions improved budget analysis production for a parliamentary client. It outlines the problem, methodology, and results of optimizing policy analysis workflows given fixed resource constraints and insights from the assessment.
Background
Policy professionals face the same pressures as everybody else: we’re asked to produce more with less, without sacrificing quality. The Minister for Finance might have asked for you to advise her about the revenue impact of a tax change in the next hour. Your consulting team might have been engaged by a client to undertake an overly ambitious evaluation of the country’s social welfare system in a week. Or perhaps you’ve just been asked to model the impacts of a policy intervention that targets an area of the economy that data isn’t available for.
While I’ve grown accustomed to this as a policy professional, they became all the more tangible when I was hired to find ways to streamline a client’s policy analysis pipeline. Located within the parliament, the client was responsible for producing a series of briefings to familiarize parliamentarians with the national budget prior to the vote. By providing members of parliament (MPs) with an accessible summary of how money would be allocated in the coming financial year it was hoped they would be better able to represent their constituents, make more informed contributions to the debate and encourage public funds to be put to good use.
If you’ve worked on fiscal policy before, the process will feel familiar: once a year the team was provided with a set of budget estimates from the Ministry of Finances that described where and how money would be spent and sourced from for the upcoming financial year. The team’s job was to translate a set of detailed financial tables and forecasts into something that the average MP could understand – a set of accessible policy briefings.
The budget analysis pipeline: illustrative results chain
| Inputs and resources | Activities, workflows and processes | Outputs | Outcomes | Impact |
| Staff, time, software, data (etc) | Processes for inputting data, producing analysis and creating briefing material. | Briefing and analysis quality, quantity and alignment with the needs of parliamentarians. | More informed parliamentarians and improved budget oversight. | Better use of public resources and improved social and economic outcomes |
While, their work was familiar, the constraints they faced were not. Once a year a team of five people came together to produce over thirty technical briefings using budget data scattered across a series of poorly documented PDFs. Although there were guidelines outlining when parliament needs to approve the budget, these were not always adhered to, which often meant the briefings had to be produced in as little as five days.
Since team members primarily handled non-analytical duties throughout the year, their analysis skills had also frequently deteriorated – limiting what could be produced and increasing the chances of errors occurring. It also wasn’t possible to simply hire more staff, find more time or get the budget data in a better format. Instead, changes had to come from how inputs are combined to produce analysis and briefing material that would meet the needs of parliamentarians.
Optimizing the policy analysis pipeline
Setting aside questions about optimizing the impact of briefings for another post, my initial assessment suggested the problem could be framed as an optimization problem: how can the quantity and quality of analysis be maximized assuming the resources available are fixed?
In an attempt better their policy analysis pipeline from this perspective I devised a simple methodology for mapping out their workflow, with the aim of understanding how staff time, software and data were typically used to produce briefings. This mapping included:
- Document and analysis audit: where past briefing and analysis files were reviewed and categorized to understand the process used for entering, analyzing and presenting budget information to MPs.
- Structured discussion sessions: drawing from insights from the policy analysis audit, analysts were asked a set of targeted questions to better understand and quantify bottlenecks in their workflow and identify potential workarounds.
- Anonymous surveys: were also distributed after each session to provide analysts with an opportunity to share insights they might have been reluctant to share openly during a session.
Based on this, I was then able to construct a detailed listing of each step required to transform data from the budget tables into a final set of briefings distributed to MP. This included many of the typical steps that might be expected, such as manually inputting data into Microsoft Excel, producing a set of financial charts and descriptive analysis and transferring these to the briefing documents, which were reviewed by senior team members before being distributed to MPs.
But, the mapping also suggested several pain points. Firstly, manually entering data from PDF files into excel was time-consuming and error prone. Even though briefings followed a standard structure and used the same data, they were independently produced by analysts — resulting in data being entered, reshaped and analyzed for each briefing document. Analysts also indicated that a lot of time was spent on checking the analysis at each stage of the process, resulting in each analyst’s work being checked multiple times before a briefing was finalized. And because each set of analysis was produced independently, the final briefing documents didn’t follow a consistent structure or style.
Streamlining triage
To identify which parts of the analysis pipeline would be most likely to benefit from technical improvements a ‘streamlining triage’ was conducted. The triage was made up of a series of true / false questions drawn from the principles of ‘Robotic Process Automation’ (RPA) assessments. Steps that received more affirmative answers during the ‘streamlining triage’ were then shortlisted for a more detailed assessment. Questions included:
- Is the task repetitive, consistent and predictable?
- Can the task be defined using a set of logic and rules?
- Does the task benefit from the accurate processing of large amounts of data?
- Is the task unlikely to benefit from collaboration?
Task characteristics and automation suitability
| More suitable | Less suitable |
| Repetitive and predictable | Unpredictable and dynamic |
| Logical and rule-based | Requires judgement and creativity |
| Require processing of data and accurate calculations | Understanding of big picture patterns and context |
| Independent | Collaborative |
Adapted from: Agaton, B. and Swedberg, G., 2018. Evaluating and developing methods to assess business process suitability for robotic process automation. Department of Computer Science and Engineering; and Casey, K., 4/6/2019, “How to identify Robotic Process Automation (RPA) opportunities”, https://enterprisersproject.com/article/2019/6/rpa-robotic-process-automation-find-use-cases
Here be dragons: accounting for analytical Cheston fences
Of course, being an outsider viewing the problem from an optimization perspective I also knew that some practices that looked inefficient might serve an important purpose — they might be bureaucratic or analytical ‘Chesterton fences’ that shouldn’t be meddled with. Maybe the onerous checks reflect established hierarchies within the organization. Perhaps having analysts own a piece of analysis in its entirety encouraged specialized expertise to be built on a topic. And it’s possible that data being manually entered was less cumbersome than the automated conversion tools that were available.
Reflecting this risk, components of the pipeline that were deemed suitable for streamlining under the triage underwent a further round of assessments to better understand the benefits, costs and risks of moving away from the status quo. This was done by examining each step and providing a qualitative rating of the benefits, costs, risks and sustainability of making adjustments to the current approach. Once a set of scores was given to a task it was validated by stakeholders.
Streamlining viability assessment
| Criteria | Example |
| Benefits | Would the streamlining / automation of the tasks improve the accuracy, quality and timeliness of briefings and/or analysis? |
| Costs | Would making the required change be worthwhile when considering the cost of it being implemented relative to the potential benefits? |
| Risks | Does automating the task present risks that are potentially costly and likely to materialize? |
| Sustainability | Are changes to how the task is conducted likely to be easily implemented, maintained by the client and the practices retained over the longer-term? |
The table below provides an illustrative example of what the results looked like. Overall, out of the fifteen discrete steps identified during the mapping exercise, seven were highlighted as being potentially suitable under the streamlining triage. Upon deeper examination, three steps were deemed to be viable for streamlining and two were suggested as being unsuitable. Two steps were also rated as ‘unclear’, implying that the viability would hinge on the nature of the solution.
| Benefits | Costs | Risks | Sustainability | Overall | |
| Data entry | H | L | M-H | M | Y |
| Reshape and clean data | H | L-H | M | L | N |
| Verify accuracy of data | H | M-H | M-H | M | ~ |
| Produce standardized graphs | H | L-M | M | M-H | Y |
| Produce standardized analysis | H | M | M | M-H | Y |
| Insert content into briefings | M-H | M-H | M-H | L | ~ |
| Distribute briefings | L-M | L | L-M | L | N |
L = Low, L-M = Low to Moderate, M = Moderate, M-H = Moderate to High, H = High, ~ = unclear.
The results
Overall, parts of the pipeline that could be sustained by the client, had high benefits, low costs and presented acceptable levels of risk were then assessed in greater detail to determine the technical options available. Such as using off-the shelf analysis tools that performed the task on behalf of analysts, changing the workflow to reduce the resource requirements associated with it or building a bespoke tool that could be maintained by the client.
Interestingly, because the structure of the organization and makeup of the team were assumed to remain fixed in the future, many aspects of the pipeline were deemed to be unsuitable for full automation. For instance, although having the data read from PDFs automatically would save a lot of time, having analysts for enter, clean and check the data ensured there was personal accountability at a critical point of the process. Similarly, while it is relatively simple to automate graphs and tables being inserted into draft versions of the briefings, having analysts do this manually assured they were directly aware of what had been inserted into a briefing and its source. Finally, although it was possible to fully automate the production of briefings, the technical solution for doing so was likely to be too complex for the client to maintain and would risk undermining the analysis capacity of staff.
The implemented solution was elegantly simple: a structured Excel template that guided analysts through the budget analysis process from data entry to final output.This centralized approach to analysis also replaced the redundant workflow where each analyst independently entered and verified the same data for every briefing they were assigned. Entering all the data into unified table also made it easier to spot potential data entry errors using a simple set of sum-checks that verified whether the data was internally consistent. Standardizing the production of graphs and tables also made it easier to spot potential problems early in the process and ensured a consistent style was applied across all the briefings. Finally, although more feature-rich and modern analytical tools exist, the familiarity of Excel made it easier to use and maintain by the client.
Since being piloted, the tool has greatly improved the quality and accuracy of budget analysis that can be produced, with one analyst noting that “the tool feels like magic!”. Perhaps most significantly, the project demonstrated how finding targeted and modest interventions can deliver greater value than more ambitious initiatives. Although I expect this is true in any organization, in the public sector where institutional constraints and organizational culture are likely to determine what solutions stick, small and boring changes have the potential to yield significant gains. And even with the rapid advancement of artificial intelligence tools like ChatGPT, the fundamental insights from this assessment remain true: The core challenges in policy analysis aren’t primarily technological—they’re human.
A note how AI was used: AI tools were used to refine how some ideas have been communicated.
