Skip to Content (Press Enter)
How is your experience with the OutSystems Community?
2024-10-31 16-25-26
João Palhares
17Views
1Comments
Has anyone evaluated OutSystems AI Agents using Azure/Microsoft AI Foundry Evaluators
Question
ODC

Hi everyone,

I want to evaluate an OutSystems AI Agent using external evaluators, specifically the built-in ones from Azure/Microsoft AI Foundry (such as Groundedness, Relevance, or Tool-Call-Success evaluators).

I am looking for the actual "how-to" from someone who has already built this integration, not just theoretical advice.

For those who have successfully connected an OutSystems agent to external evaluators, I have a few specific questions regarding your architecture and process:

  1. Evaluation Scope: Are you evaluating the agent as a whole system (orchestration, logic, and tool selection) or just the underlying LLM model's outputs?

  2. Dataset Preparation: Are you manually creating a golden dataset to feed into these Azure evaluators, or are you automating/generating synthetic data for it?

  3. Data Extraction & Flow: How are you extracting the interaction data (user queries, agent responses, context, and tool calls) from OutSystems and sending it to Azure?

  4. Execution Context: Are you running these evaluations asynchronously (using BPTs/Timers on logged data) to avoid impacting the agent's response time?

  5. General Question: How are you testing the agent, which approach are you using?

I want to understand the concrete technical steps you took to make this work. Any real-world examples would be greatly appreciated.

Thank you!

0
0
30 Jun
Copy Link
UserImage.jpg
Francisco Franco

I've done something similiar but not exact with your description but what i've done is:

  • Wrap CallAgent and store the query, response, retrieved context, tool calls/results and correlation ID.
  • Use an ODC Timer or Azure Function to send batches to Foundry asynchronously.
  • Convert the data to the OpenAI message format expected by Foundry (IN my case OpenAI).
  • Use Groundedness/Relevance for the final answer and Tool Call Accuracy/Success for the full agent flow.
  • Start with a manually validated golden dataset, then add production traces and synthetic cases.
2
0
15 Jul
Copy Link
Community GuidelinesBe kind and respectful, give credit to the original source of content, and search for duplicates before posting.