Why need LLM Evaluation?
Artificial intelligence technology has yielded exceptional tools, none more significant than large language models (LLMs). Language models have gained considerable attention for their ability to understand and process human-like language.
Large language models have become the foundation of AI systems that feature natural language processing (NLP) capabilities. As LLMs power many AI applications and technologies such as AI-powered chatbots, content creators, AI copilots, and speech recognition technologies, therefore LLM evaluation for performance, accuracy, and efficiency becomes crucial.
I strongly believe that there are some main reasons for which people do evaluation, which tend to be conflated together, but are actually very different, and each answer a separate question.
1) Is my model training well? Is my training method sound? – Non-regression testing
Non-regression testing is a concept which comes from the software industry, to make sure small changes have not broken the overall approach.
The idea is the following: when you add a new feature to your software, or fix a problem in the code base, have you broken something else? That’s what non-regression tests are for: making sure the expected, high-level behavior of your software is not suddenly broken by a (seemingly unrelated) change.
When you select a setup to train models, you want to test something very similar, and make sure that your changes (choosing different training data, architecture, parameters, etc) have not “broken” the expected performance for a model of these properties.
To give a concrete example, you would expect a 7B base LLM to get between 50 and 65 on (multiple choice) MMLU after training, and on the other hand, know that performance fluctuating between 20 and 30 indicates that no learning occurred.
For “non-regression” evaluation, you need to look at 1) evaluation scores trajectories (is the performance better now that when starting training), 2) evaluation scores ranges (is the performance within what’s expected). You actually… don’t care about the precise score themselves!
This evaluation is therefore not here to tell you anything about actual model capabilities, but instead just here to confirm that your training approach is “as sound” as the other training approach, and that your model behaves in similar ways. I believe that even some evaluations simply looking at changes in the perplexity (probabilities) of text could be sufficient for this step, but you usually want benchmarks which have a high “signal to noise” ratio, or in other words, you want to make sure that a big change in the score reflects a big shift in your model.
2) Which model is the best? Is my model better than your model? – Leaderboards and rankings
The next role of evaluation is simply to sort models to find and select the best architectures and approaches overall. If you have a leaderboard, take the best model, and it’s not working on your use case, it’s unlikely the next best model will work. In their paper about lessons learned on benchmarking and dataset design from the ImageNet era, the authors argue that, since scores are susceptible to instability, the only robust way to evaluate models is through rankings, and more specifically by finding broad groups of evaluations which provide consistent and stable rankings.
I believe looking for ranking stability is indeed an extremely interesting approach to model benchmarking, as we have shown that LLMs scores on automated benchmarks are extremely susceptible to minute changes in prompting, and that human evaluations are not more consistent – where rankings are actually more stable when using robust evaluation methods.
3) Where are we, as a field, in terms of model capabilities? Can my model do X?
“How do you know if models can do X?” is a question which comes up a lot, and I think it is a very valid one.
However, for any complex capability, we cannot at the moment just say “this model is the best at this”, but instead “this model is the best on this task that we hope is a good proxy for this capability, without any guarantee”.
We are strongly missing any kind of good definitions and framework on what a capability is for a machine learning model, especially for those surrounding reasoning and mind theory. However, this is not specific to machine learning! In human and animal studies, it is also quite hard to define what constitutes a “capability”, and metrics which try to provide precise scores (IQ and EQ for example) are hotly debated and controversial, with reason.
We might want to look at social sciences to think about evaluation of capabilities, as in these fields, people are used to thinking seriously about confounding factors in data gathering and analysis. However, I also believe it likely that 1) we cannot define these broad capabilities at all, since we cannot define them in humans and animals at the moment, 2) frameworks made with the human (or animal) in mind will not transfer well to models, as the underlying behaviors and assumptions are not the same.
What is LLM Evaluation?
Developers, researchers, and enterprise adopters increasingly rely on synthetic benchmarks and other evaluation tools to measure a model’s ability to navigate and process language nuances.
Overall, LLM Evaluation encompasses many domain, such as:
Model Capability Evaluation, mainly focusing on abilities such as mathematical calculations, logical reasoning, etc.
A well-defined concrete task, such as How well can my model classify spam from non spam emails?
A more abstract and general capability, such as How good is my model at math?
Model Safety Evaluation, mainly focus on whether the generated content conforms to human social values.
Model Security Evaluation, the main concern is that the model is prone to security risks such as instruction hijacking and data leakage.
LLM Evaluation is a concept derived from Software Unit Testing, essentially consisting of two core components:
Testing System: By inputting a set of static or dynamic data into the software and collecting the system’s response results.
Metrics Evaluation System: Quantify the test results through manual or machine automation to obtain a set of vector indicators.
The concept of LLM Evaluation encompasses a thorough and complex process necessary for assessing the functionalities and capabilities of large language models. It is within this evaluative framework that the strengths and limitations of a given model become clear, guiding developers towards refinements and deciding which models are best aligned with the project’s requirements.
Testing System
In order to achieve a complete evaluation, a variety of benchmarks was proposed, such as
GLUE (General Language Understanding Evaluation), GLUE tests an LLM’s understanding of language with nine different tasks, such as analyzing sentiments, answering questions, and figuring out if one sentence logically follows another. It gives a single score that summarizes the model’s performance across all these tasks, making it easier to see how different models compare.
SuperGLUE, As models began to beat human scores on GLUE, SuperGLUE was introduced. It’s a tougher set of tasks that pushes models to handle more complex language and reasoning.
HellaSwag, HellaSwag checks if an LLM can use common sense to predict what happens next in a given scenario. It challenges the model to pick the most likely continuation out of several options.
TruthfulQA, TruthfulQA is all about honesty. It tests whether a model can avoid giving false or misleading answers, which is essential for creating reliable AI.
MMLU (Massive Multitask Language Understanding), MMLU is vast, covering everything from science and math to the arts. It has over 15,000 questions across 57 different tasks. It’s designed to assess how well a model can handle a wide range of topics and complex reasoning.
ARC (AI2 Reasoning Challenge), Focuses on scientific reasoning.
BIG-bench, A collaborative project with many different tasks.
LAMBADA, Tests how well models can guess the last word in a paragraph.
SQuAD (Stanford Question Answering Dataset), Measures reading comprehension and ability to answer questions.
toxic-chat, this dataset contains toxicity annotations on 10K user prompts collected from the Vicuna online demo. It utilize a human-AI collaborative annotation framework to guarantee the quality of annotation while maintaining a feasible annotation workload. The details of data collection, pre-processing, and annotation can be found in our paper.
Meanwhile due to the powerful Multimodal Capability of large language models, for Testing Systems, the form of input dataset can be multimodal data streams, such as:
text
images
audio
video.
At the same time, similar to the concept of Vulnerability Fuzz in traditional software testing, the dataset is not only static, but may also be a dynamic dataset generated based on mutation algorithms or attack modules.
static dataset
dynamic dataset, generate by mutation algorithms or attack modules
Finally, based on the Degree of Interaction, the evaluation dataset can be divided into different types:
Single round interactive evaluation dataset.
Multi round interactive evaluation dataset.
Metrics Evaluation System
For LLM Evaluation, quantifying the evaluation results is a crucial step. Before quantification, the evaluation dataset and output results are just a pile of data that cannot be interpreted and utilized. Only by quantifying the evaluation results can the tested system be effectively evaluated and support subsequent research and operation decisions.
Basically, the Metrics Evaluation System consists of two main aspects:
01 Quantitative Methods
Human as a Judge, This is usually done by tasking humans with first, prompting models, then, grading a model answer or ranking several outputs according to guidelines. Using humans as judges allows to study more complex tasks, with more flexibility than automated metrics. It also prevents most contamination cases, since the written prompts are (hopefully) new. Lastly, it correlates well with human preference, since this is literally what is evaluated!
Automated Benchmarking, Quantitatively evaluate the test results using methods such as keyword matching, based on a pre-defined set of inputs and expected outputs.
Model as a Judge, two approach exist for grading: using generalist, high capability modelsusing small specialist models trained specifically to discriminate from preference data.
LLM as a Judge, LLM-as-a-Judge is a powerful solution that uses LLMs to evaluate LLM responses based on any specific criteria of your choice, which means using LLMs to carry out LLM (system) evaluation. The three types of LLM as judges include:
Single Output Scoring (without reference): A judge LLM is provided with a scoring rubric as the criteria and prompted to assign a score to LLM responses based on various factors such as input to LLM system, retrieval context in RAG pipelines, etc.
Single Output Scoring (with reference): Same as above, but sometimes LLM judges can get flaky. Having a reference, ideal, expected output helps the judge LLM to return consistent scores.
Pairwise Comparison: Given two LLM generated outputs, the judge LLM will pick which one is the better generation with respect to the input. This also requires a custom criteria to determine what is “better”.
02 Evaluation Metrics
Based on the quantitative scoring results, a set of vectorized evaluation indicators is summarized as a whole, with Metrics representing the performance profile of the tested system.
LLM Evaluation Metrics
LLM evaluation metrics such as answer correctness, semantic similarity, and hallucination, are metrics that score an LLM system’s output based on criteria you care about. They are critical to LLM evaluation, as they help quantify the performance of different LLM systems, which can just be the LLM itself.
Recognizing the diversity of applications that modern large language models serve, it becomes evident that a one-size-fits-all approach to LLM performance evaluation is impractical.
Rather, the large language model evaluation process must adapt to the intricacies of various use cases, employing tailored LLM evaluation metrics that accurately reflect the unique demands of each scenario.
Context-Specific Evaluation
When deploying LLMs in education, for instance, developers meticulously examine the age-appropriateness of the model’s responses, as well as their propensity to avoid toxic outputs. Similarly, consumer-facing applications may prioritize response relevance and the capacity of a model to sustain coherent and engaging interactions.
All these evaluation points are influenced significantly by the selection and structuring of the LLM prompts and contexts.
The following is a typical LLM Evaluation Metrics:
Answer Relevancy: Does the LLM provide information pertinent to the user’s query?
Correctness: Determines whether an LLM output is factually correct based on some ground truth.
Hallucination: Is the model prone to generating factually incorrect or illogical statements? Is the model prone to generating factually incorrect or illogical statements? What improvements can be made to reduce AI hallucinations?
Question-answering accuracy: How effectively can the LLM handle direct user inquiries?
Contextual Relevancy: Determines whether the retriever in a RAG-based LLM system is able to extract the most relevant information for your LLM as context.
Toxicity: Are the outputs clear of offensive or harmful content?
Bias detection: is an essential element of the current model evaluation techniques, identifying situations where the model might produce prejudiced outcomes. Effective LLM evaluation metrics aid in strategizing improvements, ensuring outputs from LLMs are fair and ethical.
Task-Specific Metrics: Includes metrics such as summarization, which usually contains a custom criteria depending on the use-case.
Bleu score: The BLEU (Bilingual Evaluation Understudy) score measures the similarity between a machine-generated text and a reference human translation. It evaluates how closely the machine output matches the human reference, often used in translation tasks.
Rouge score: The ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score is a set of metrics for evaluating automatic summarization and machine translations. It focuses on recall, assessing how much of the reference content is captured in the generated summary or translation.
In different context scenarios, there will be corresponding metrics focus points.
User Experience Metrics
Beyond these primary metrics, evaluating the overall user experience is crucial. This involves assessing how intuitive and user-friendly the LLM is, which includes:
Response Time: How quickly does the LLM generate responses?
User Satisfaction: Are users satisfied with the interactions? This can be measured through feedback and engagement metrics.
Error Recovery: How well does the LLM handle errors or misunderstandings? Effective error recovery mechanisms enhance user trust and reliability.
Guided by specific use cases, LLM system evaluation transcends mere number-crunching. It is an exercise in understanding the nuanced requirements of various applications, thereby shaping a more inclusive and responsible approach to AI development and implementation.
Performance Metrics
Security&Safety Metrics
Responsible Metrics: Includes metrics such as bias and toxicity, which determines whether an LLM output contains (generally) harmful and offensive content.
Custom LLM Evaluation Metrics
While most metrics are generic and necessarily, they are not sufficient to target specific use-cases. This is why you’ll want at least one custom task-specific metric to make your LLM evaluation pipeline production ready (as you’ll see later in the G-Eval section). For example, if your LLM application is designed to summarize pages of news articles, you’ll need a custom LLM evaluation metric that scores based on:
Whether the summary contains enough information from the original text.
Whether the summary contains any contradictions or hallucinations from the original text.
Moreover, if your LLM application has a RAG-based architecture, you’ll probably need to score for the quality of the retrieval context as well. The point is, an LLM evaluation metric assesses an LLM application based on the tasks it was designed to do.
An LLM application can simply be the LLM itself!
Standard way to do a LLM Evaluation
These four steps can streamline the process and enhance the accuracy of your evaluations.
Curate benchmark tasks: Design a set of language tasks that cover a spectrum from simple to complex, ensuring the benchmark captures the breadth of LLM capabilities.
Prepare datasets: Use diverse, representative datasets that have been carefully curated to avoid biases and evaluate the LLM’s performance on a level playing field.
Evaluate with metrics: Utilize established evaluation metrics such as perplexity, ROUGE, and diversity to assess the performance of the LLM objectively.
Analyze results: Interpret the data gathered to compare and contrast the performance of different LLMs, offering insights that could guide future improvements.
Upon completing these steps, you’ll have a thorough understanding of how LLMs perform under a variety of scenarios, which is essential for practical applications and further development.
Example use case about LLM evaluation
Let’s say you’re using an LLM in customer support for an online retail store. Here’s how you might evaluate it:
You’d start by setting up the LLM to answer common customer inquiries like order status, product details, and return policies. Then, you’d run simulations using a variety of real customer questions to see how the LLM handles them. For example, you might ask,
“What’s the return policy for an opened item?”
“Can I change the delivery address after placing an order?”
During the evaluation, you’d check if the LLM’s responses are accurate, clear, and helpful.
Does it fully understand the questions?
Does it provide complete and correct information?
If a customer asks something complex or ambiguous, does the LLM ask clarifying questions or jump to conclusions?
Does it produce toxic or harmful responses?
As you collect data from these simulations, you’re also building a valuable dataset. You can then use this data for LLM fine-tuning and RLHF to improve the model’s performance.
This cycle of constantly testing, gathering data, and making improvements helps the model work better. It makes sure the model can reliably help real customers, improving their experience and making things more efficient.
LLM Evaluation Datasets Taxonomy
As mentioned above, LLM Evaluation consists of three core components,
A collection of samples, given as input to the model to see what comes out as output. Samples are usually designed to try to emulate what you want to test the model on.
A metric, which is a way to compute a score for the model.
A collection of attack method, used for transforming and expanding seed datasets to enhance the adversarial nature of test datasets.
Therefore, a clear and well classified dataset classification taxonomy is crucial.
Safety
Misinformation
Terrorism
Violence
Sexual Content
Bias
Political Sensitivity
Crime
Insult
Hate Speech
Hallucination
Ethics
Privacy
Membership Inference
Consent Management
Data Sharing
Access Control
Data Collection
Data Deletion
Security
Data Poisoning
Hijacking
Model Inversion
API Vulnerability
Roleplay
Prompt Injection
Model Stealing
Exhaustion Attack
Pretending
Integrity
Scraping
Fraud
Spam
Authenticity
Copyright
Why we need an LLM Evaluation Framework?
To understand why LLM Evaluation Framework is pivotal, we must consider the rapidly expanding scope of their applications, often outpacing the capability of traditional feedback mechanisms to monitor their performance. The LLM Evaluation Framework is thus indispensable for several reasons.
Primarily, it provides a window into the model’s reliability and efficiency—key factors determining an AI’s ability to function in real-world settings. The absence of robust and current evaluation methods could lead to inaccuracies and inefficiencies going unchecked, which may culminate in unsatisfactory user experiences.
Secondly, security teams need the ability to tailor AI security policies to the specific needs of each application— whether internal or external-facing, handling highly sensitive data, or managing less critical information. The LLM Evaluation process must allow for LLM-agnostic security policies, ensuring security isn’t tied to individual model providers. Policies should be adjustable on a per-application basis, reflecting the varying levels of sensitivity and risk. As applications evolve or new data sources are introduced, the LLM Evaluation process must adapt seamlessly without requiring additional development work, ensuring ongoing protection.
Thirdly, a vendor with an independent LLM Security&Safety research team and global threat intelligence gathering capability can ensure that the LLM Evaluation process is constantly updated. The vendor needs access to vast amounts of data, feeding it into a strong and continually updated threat database. If this database isn’t updated daily, you’re leaving the door wide open for sophisticated attacks. A comprehensive, constantly updated threat intelligence database is essential for recognizing and countering AI-specific threats like prompt injection and data poisoning.
LLM Model Evals Versus LLM System Evals
While an LLM (Large Language Model) refers specifically to the model (eg., GPT-4) trained to understand and generate human language, an LLM system refers to the complete setup that includes not only the LLM itself but also additional components such as function tool calling (for agents), retrieval systems (in RAG), response caching, etc., that makes LLMs useful for real-world applications, such as customer support chatbots, autonomous sales agents, and text-to-SQL generators.
However, it’s important to note that an LLM system can sometimes simply be composed of the LLM itself, as is the case with ChatGPT. Here is an example RAG-based LLM system that performs the Text-SQL task:
Evaluating an LLM system, is therefore not as straightforward as evaluating an LLM itself. While both LLMs and LLM systems receive and generate textual outputs, the fact that there can be several components working in conjunction in an LLM system means you should apply LLM evaluation metrics more granularly to evaluate different parts of an LLM system for maximum visibility into where things are going wrong (or right).
For example, you can apply a contextual recall metric to evaluate the retrieval pipeline in the Text-SQL example above to assess whether it is able to retrieve all necessary tables needed to answer a particular user query.
Similarly, you can also apply a custom SQL correctness metric implemented via G-Eval to evaluate whether the generation pipeline generates the correct SQL based on the top-K data tables retrieved.
In summary, an LLM system is composed of multiple components that help make an LLM more effective in carrying out its task as shown in the Text-SQL example, and it is harder to evaluate because of its complex architecture.
Understanding the nuances between LLM evaluations and LLM system evaluations is critical for stakeholders looking to harness the full potential of large language models.
LLM model evaluations are designed to gauge the raw capability of the models, focusing on their ability to understand, generate, and manipulate language within the appropriate context.
LLM System evaluations are tailored to observe how these models perform within a predetermined framework, examining functionalities that are within the user’s influence.
Evaluating LLMs encompasses a broad spectrum of tasks and diverse predefined evaluation metrics to ensure objectivity and precision. For those pondering how to evaluate models and LLMs effectively, appreciating the differences and applications of these two types of evaluations is fundamental. Here we break down and compare the essential metrics used in model vs. system evaluations:
For developers and machine learning practitioners, the distinction between these evaluations is much more than academic, it directly influences their work and strategic approach toward improving LLM evaluation methods.
Foundational model builders consistently push the frontiers of what their LLM can do, testing it against divergent cases and refining its core functionalities.
System evaluators prioritize how to evaluate LLM effectiveness within specific contexts, often necessitating frequent iterations to enhance the user experience and overall system reliability.
There are different concerns between these two groups of people,
Model evaluators question, “How comprehensive and adaptable is the LLM?”
System evaluators ask, “How well does this LLM perform for the particular task at hand?”
Taking these differences into account enables targeted strategies for advancing LLMs. Therefore, evaluating large language models through both lenses ensures a comprehensive understanding of their capacities and limitations, thereby supporting the development of more efficient, ethical, and usable AI systems.
LLM evaluation frameworks and tools
SuperAnnotate
SuperAnnotate helps companies build their eval and fine-tuning datasets to improve model performance. Its fully customizable editor enables building datasets for any use case in any industry.
Amazon Bedrock
Amazon’s entry into the LLM space – Amazon Bedrock – includes evaluation capabilities. It’s particularly useful if you’re deploying models on AWS. SuperAnnotate integrates with Bedrock, allowing you to build data pipelines using SuperAnnotate’s editor and fine-tune models with Bedrock.
Nvidia Nemo
Nvidia Nemo is a cloud-based microservice designed to automatically benchmark both state-of-the art foundation and also custom models. It evaluates them using a variety of benchmarks, which include those from academic sources, customer submissions, or using LLMs as judges.
Azure AI Studio
Microsoft’s Azure AI Studio provides a comprehensive suite of tools for evaluating LLMs, including built-in metrics and customizable evaluation flows. It’s particularly useful if you’re already working within the Azure ecosystem.
Prompt Flow
Another Microsoft tool, Prompt Flow allows you to create and evaluate complex LLM workflows. It’s great for testing multi-step processes and iterating on prompts.
Weights & biases
Known for its experiment tracking capabilities, W&B has expanded into LLM evaluation. It’s a solid choice if you want to keep your model training and evaluation in one place.
LangSmith
Developed by Anthropic, LangSmith offers a range of evaluation tools specifically designed for language models. It’s particularly strong in areas like bias detection and safety testing.
TruLens
TruLens is an open-source framework that focuses on transparency and interpretability in LLM evaluation. It’s a good pick if you need to explain your model’s decision-making process.
Vertex AI Studio
Google’s Vertex AI Studio includes evaluation tools for LLMs. It’s well-integrated with other Google Cloud services, making it a natural choice for teams already using GCP.
DeepEval
Deep Eval is an open-source library that offers a wide range of evaluation metrics and is designed to be easily integratedinto existing ML pipelines.
Parea AI
Parea AI focuses on providing detailed analytics and insights into LLM performance. It’s particularly strong in areas like conversation analysis and user feedback integration.
Conclusion
Currently, we cannot really evaluate “general model capabilities” at the moment, first and foremost because we have not defined what that means.
However, LLM evaluation, as a research field, is very much in its infancy at the moment, and there is a lot to be done, which is very exciting! Inspiration can be grabbed from many fields, from machine learning interpretability to sociology, in order to define new metrics and tasks. Interdisciplinary work will likely open very new cool directions for the field!