The overall process of building an evaluation pipeline looks like this:
  1. Select Your Dataset: Choose or upload datasets to serve as the basis for your evaluations, whether for scoring, regression testing, or bulk job processing.
  2. Build Your Pipeline: Start by visually constructing your evaluation pipeline, defining each step from input data processing to final evaluation.
  3. Run Evaluations: Execute your pipeline, observe the results in a spreadsheet-like interface, and make informed decisions based on comprehensive metrics and scores.

​
Creating a Pipeline

  1. Initiate a Batch Run: Start by creating a new batch run, which requires specifying a name and selecting a dataset.
  2. Dataset Selection: Upload a CSV/JSON dataset, or create a dataset from historical data using filters like time range, prompt template logs, scores, and metadata. Learn more here.
You now have a pipeline. Preview mode allows you to iterate with live feedback, allowing for adjustments in real-time.

​
Setting up the Pipeline

​
Adding Steps

Click ‘Add Step’ to start building your pipeline, with each column representing a step in the evaluation process. Steps execute in order left to right. That means that if a column depends on a previous column, make sure it appears to the right of the dependency.

​
Common Step Types

  • Prompt Template: Select a prompt template from the registry, set model parameters, LLM, arguments, and template version.
  • Custom API Endpoint: Define a URL to send and receive data, suitable for custom evaluators or external systems.
  • Human Input: Engage human graders by adding a step that allows for textual input.
  • String Comparison: Use this step to compare the outputs of two previous step, showing a visual diff when relevant.

​
Scoring

If the last step of your evaluation pipeline contains all booleans or numeric values, that will be consider the score for the row. Your full evaluation report will have a scorecard of the average of this last step. NOTE: All cells in the last column must be boolean or all must be numeric. If any cell deviates, the score will not be calculated

​
Executing Full Batch Runs

Transition from pipeline to full batch run to apply your pipeline across the entire dataset for comprehensive evaluation.