Prudential Regulation Embedding Transformer (PRET) a domain-adapted model for prudential supervision

Download paper here

Analysis of unstructured text is a key aspect of everyday financial supervision operations run by regulators worldwide. The emergence of transformer-based language models have opened the possibility of improving the financial supervision process, by assisting supervisors with labour-intensive and time-consuming tasks of information retrieval across the wide range of complex corpora like financial rules and regulations. This paper introduces Prudential Regulation Embedding Transformer (PRET), a novel domain-adapted transformer encoder model tailored for information retrieval on topics relating to financial regulations. To train this model, we address the scarcity of high-quality training financial regulations text datasets with a dedicated pipeline to web-scrape and pre-process the Basel Framework into machine-readable format, which is then coupled with corresponding large language model (LLM) generated text as synthetic training data pairs for each rule in the Basel Framework. We evaluate the performance of our model on this domain-specific information retrieval task against commonly used state-of-the-art (SOTA) models. We show how our proposed model outperforms existing benchmarks while being substantially cheaper to train than previous methods. We discuss the implications of our findings for the design of better regulatory technology models across jurisdictions.

Recommended citation: Dragos Gorduza, Adam Muhtar Prudential Regulation Embedding Transformer (PRET) a domain-adapted model for prudential supervision