How to create evaluations in Solidroad (AutoQA)

Last updated: July 14, 2026

This guide walks you through creating an automated evaluation that scores AI-generated (simulated) conversations rather than live customer conversations. It's the quickest way to test a scorecard's accuracy or QA an agent's performance using Solidroad Simulations as your data source.

Before you start: You'll need Admin or QA Manager access to create evaluations, and at least one scorecard already set up in your workspace.

Step 1: Start a new evaluation

Evaluations are created from the Evaluations page under the Quality section of the left-hand navigation.

What to do:

  1. Go to Quality → Evaluations.

  2. Click Create in the top-right corner.

  3. Give your evaluation a Name (for example, "Trial").

Step 2: Choose Simulations as your data source

The Data source field decides which conversations feed into your evaluation. To score simulated conversations instead of live ones, select the Simulations option here.

What to do:

  1. Under Setup, open the Data source dropdown.

  2. Select your data source from the list.

Why it matters: Choosing Simulations lets you QA a scorecard or agent using AI-generated conversations, so you can validate your setup before it ever touches real customer interactions.

Step 3: Select a scorecard

The scorecard determines the criteria Solidroad uses to score each conversation.

What to do:

  1. Open the Scorecard dropdown.

  2. Search or scroll to find the scorecard you want to use (for example, Solidroad Customer Experience Evaluation).

  3. Select it from the list.

Step 4: Set your time filters

Time filters control which date range of conversations this evaluation includes, and how often it re-runs.

What to do:

  1. Click into Time Filters in the left-hand step menu.

  2. Under Filter by, leave this set to Created (or change it if you want to filter by a different date field).

  3. Set the Start date (for example, 1 week ago).

  4. Set the End date — choose Until cancelled for an ongoing evaluation, or a fixed date like Now to cap it.

Why it matters: An evaluation with no end date runs every day until you pause or cancel it, so it's worth deciding upfront whether you want a one-off check or an ongoing QA process.

Step 5: Add filters (optional)

Filters narrow down which conversations are included, beyond the time range.

What to do:

  1. Click + Add filter.

  2. Choose a condition, such as State, Channel source, Tags, Conversation rating score, or AI agent participated.

  3. Set the value for that condition.

Step 6: Configure agent selection

This section tells Solidroad which agent to score in each conversation, and how to choose between agents if more than one is involved.

What to do:

  1. Click into Agent selection.

  2. Under How many agents should be evaluated?, choose Evaluate one agent (Solidroad picks one participating agent automatically).

  3. Under Which agent types should Solidroad consider?, pick Human agents only, Human agents and selected bots, or Selected bots only.

  4. Optionally, add guidance in the How should Solidroad choose among participating agents? box — for example, "Whoever is most involved in a conversation".

Why it matters: This guidance is used to brief the scoring engine the same way you'd brief a human QA analyst, so it picks the right agent when a conversation has more than one participant.

Step 7: Review the Testing option

The Testing toggle lets you trial a scorecard's accuracy over a set period before relying on it fully.

What to do:

  1. Scroll to the Testing section.

  2. Toggle it on if you want to test and improve scorecard accuracy over time, or leave it off to run the evaluation live straightaway.

Step 8: Confirm and create

Once your setup is complete, review the Preview panel on the right — it shows your QA type, data source, scorecard, filters, and agent selection at a glance.

What to do:

  1. Click Create in the top-right corner.

  2. In the Confirm Evaluation dialog, review the message telling you how many conversations match your filters.

  3. Click Confirm.

Once confirmed, your evaluation goes Live immediately, and you'll see a Bulk evaluation created confirmation. From here, the evaluation's detail page shows who created it, when it next runs, its source, scorecard, and frequency (for example, Every day).

Now that your evaluation is live, Solidroad will automatically score matching simulated conversations against your scorecard and keep running on the schedule you set.

Once you've reviewed your first results, it's a good time to explore building or refining a scorecard for even more accurate scoring.

If you have any further questions or need assistance, please refer to our support resources or contact the Solidroad team via the Get Help tab within the platform.

Glossary

  • Evaluation: An automated or one-off QA run that scores conversations against a scorecard.

  • Data source: Where the conversations being scored come from — live conversations, uploads, or Solidroad Simulations.

  • Scorecard: The set of criteria and questions Solidroad uses to score a conversation.

  • Simulation: An AI-generated conversation used to test agents or scorecards without needing real customer interactions.