Partial Understanding: The Bank of England and FCA's Survey of AI in UK Financial Services, November 2024

On November 21, 2024, the Bank of England and the Financial Conduct Authority published "Artificial intelligence in UK financial services 2024," the results of their Artificial Intelligence and Machine Learning Survey 2024. It continues their surveys of 2019 and 2022 of firms that the Bank or the FCA regulate, and, in view of generative AI's growth since the 2022 survey, it incorporated related questions. The survey received 118 responses, from banks, insurers, lenders, payments firms and others, and the results are anonymized and aggregated by sector. For the survey, AI is defined as "the simulation of human intelligence by machines, including the use of computer systems, which have the ability to perform tasks that demonstrate learning, decision-making, problem solving, and other tasks which previously required human intelligence." Parts of the report follow, not the whole.

How Far AI Has Gone In

75% of the firms that responded said they were already using AI, with a further 10% planning to use it over the next three years. In the 2022 survey, those figures were 58% and 14%. Foundation models, the report says, form 17% of all AI use cases.

Respondents report that 55% of all AI use cases have some degree of automated decision-making. Of those, 24% are semi-autonomous: they can make a range of decisions on their own, but "are designed to involve human oversight for critical or ambiguous decisions." Only 2% of use cases have fully autonomous decision-making. Firms rated 62% of their use cases as low materiality and 16% as high, materiality being, in the report's words, "a rating of the use case impact which could include quantitative and qualitative measures."

Third Parties, Understanding and Explainability

A third of all current AI use cases are third-party implementations, cases where most of the development or deployment of the application is done by a third party. That is significantly higher, the report says, than the 17% reported in 2022. The business areas with the highest shares are human resources, risk and compliance, and operations and IT. The top three providers named by firms account for 73% of all named cloud providers, 44% of model providers and 33% of data providers.

Of the firms using or planning to use AI, 46% reported having only "partial understanding" of the AI technologies they use, against 34% that said they have "complete understanding." "This is largely due to the use of third-party models," the report says, "where respondent firms noted a lack of complete understanding compared to models developed internally."

A high proportion of firms currently using AI, 81%, employ some kind of explainability method. Some respondents said they evaluate third-party AI systems with their existing frameworks, and some had additional conditions or considerations specific to AI.

Who Is Accountable

The most common governance measure specific to AI, used by 84% of firms currently using AI, is an accountable person or persons responsible for the AI framework. 72% of firms using or planning to use AI said they allocate accountability for AI use cases and their outputs to executive leadership. But accountability, the report says, "is often split with most firms reporting three or more accountable persons or bodies."

The survey ties partial understanding largely to models built by third parties, and finds accountability often split among several people or bodies. A model that is understood in part and answered for in pieces is where a mistake could pass with no one sure whose it was.

The Risks Firms Expect

Four of the top five risks firms perceive today are related to data: data privacy and protection, data quality, data security, and data bias and representativeness. The risks expected to increase the most over the next three years, the report says, are "third-party dependencies, model complexity, and embedded or 'hidden' models." Across all risks, the report says, the average level is slightly above medium, and respondents judge that the level of risk "will increase somewhat over the next three years."