How to evaluate AI applications

Evaluating AI applications focuses on assessing the performance of the application rather than the underlying model. This evaluation involves establishing a 'golden data set' of desired responses and defining specific metrics, such as accuracy, politeness, and latency, to measure the application's output against these benchmarks. The process can be conducted offline during development or online in real-time, using tools like Vertex AI's generative AI evaluation service, to ensure ongoing quality and user satisfaction.

Evaluating AI applications

• 00:00:44 Instead of evaluating the model itself, the focus should be on evaluating the application built on top of the model, as this is more relevant to most developers. Model benchmarks don't always translate to better application performance due to factors such as prompt templates, RAG, and fine-tuning.

Golden data set creation

• 00:01:30 To evaluate an AI application, a set of known good responses, or 'golden data set,' is needed. This set can be obtained from user interactions, manually created, or generated using AI, but the generated data must be validated by a human to ensure it accurately reflects the desired application behavior.

Defining evaluation metrics

• 00:03:55 Specific metrics should be defined to measure the aspects of the application that matter most, such as accuracy, relevance, politeness, fluency, and response time. Developers can also create custom metrics based on the application's domain and user needs, but consistency in scoring is crucial.

Evaluation tools and methods

• 00:05:10 Off-the-shelf evaluation tools like Vertex AI's generative AI evaluation service can automate evaluation by allowing users to select metrics, provide data, and receive results. Alternatively, developers can build custom pipelines to manage the evaluation process, including prompt selection, response evaluation, and result summarization.

Offline and Online Evaluation

• 00:06:21 Evaluation can be performed at various stages of the AI application's lifecycle, including offline evaluation during development and MLOps to ensure quality, as well as online evaluation in real-time before sending responses to users. Online evaluation allows for adjustments based on immediate response quality, improving application performance iteratively.