Helvia.ai participated in the FinNLP-MUFFIN Workshop on Financial Technology and AI, that took place as part of the IJCAI-2023 conference in Macau, China. Helvia.ai expert Lefteris Loukas presented the paper titled "Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance".

The paper explored the use of conversational GPT models for quick and easy few-shot text classification, aiming to tackle one of the most challenging aspects of conversational chatbots.

This work was sponsored by the by the LAW-GAME H2020 EU Project (GA no 101021714) and we were honored to have had the opportunity to showcase it at the conference.

The intent detection challenge

When designing conversational chatbots, one of the most common challenges is to make sure that they understand the user's questions properly. This is because people ask questions in various ways, using different words and phrases to express the same idea. This is where intent detection comes in. Intent detection is a classification task, which involves categorizing the user's question into different groups based on the intent behind it. Once the chatbot understands the intent behind the user’s question, it can then provide the user with the most relevant answer.

By using NLP techniques and machine learning algorithms, chatbots can analyze user questions and classify them into relevant intent categories, which allows them to provide personalized and efficient responses. However, intent classification can be proved particularly challenging, especially within specific contexts, when there is class overlap or few examples to train the models.

Despite its crucial role in designing chatbots that can provide accurate and helpful responses to user questions, intent detection is under-explored in various industries, due to the limited availability of relevant datasets. This study aims to bridge the gap between the industry and the latest advancement in academia in the field of intent classification.

The approach: Few-shot banking intent classification

In this work, we studied the intent classification task for a real-world and open dataset (Banking77), composed of customer service queries in the financial domain. The dataset has a large number of classes (77) with highly semantic overlaps between them.

In a business setting, it can be difficult to gather enough data to properly train machine learning models. To overcome this challenge, we simulated a "Few-Shot Setting" where we only had a small number of examples for each type of query (between 1 and 20). This approach is more feasible for smaller organizations that might not possess large quantities of annotated data..

Our methodology included the below:

  • In-Context Learning of GPT-3.5 and GPT-4 to adapt them to the banking intent detection problem.
  • Fine-tuning Masked Language Models (MLMs) for the banking domain by further training it on a smaller dataset.
  • Few-Shot Contrastive Learning (using SetFit) to train the model to learn from a few examples.
  • Human Expert Annotation for the curation of a representative subset that addresses the challenges of identified class overlaps.

Generative LLMs: a promising approach to the classification challenge across various domains

The results of our research showed that in-context learning with conversational LLMs provide a straightforward solution that can produce accurate responses, even where there is limited amount of training data available. We also demonstrated that generative LLMs, like GPT-3.5 and GPT-4, can perform better than MLM models in scenarios where little data is provided. As a drawback, closed-source and managed LLMs can provide substantial costs (for the case examined, the cost reached approximately 1,600$ for GPT-3.5 and GPT-4).

The findings of this study are not confined solely to the realm of finance; rather, they extend to a multitude of other fields where swift and precise outcomes are imperative, even in situations where there are limited examples available to draw from.

In future work, we plan to experiment with other generative open-source models and cost-effective methods of deploying LLMs.

You can access the full paper here. For any questions you might have, you can contact us at contact@helvia.ai.

Find out more about the LAW-GAME project at https://lawgame-project.eu/.

LAW-GAME project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101021714. Content reflects only the authors’ view and European Commission is not responsible for any use that may be made of the information it contains.