Tackling the AI Trust Problem: Insights from Aaron McCormack on building Trust with AI Quality
- Aaron
- 1 day ago
- 3 min read
Artificial intelligence is transforming industries, but ensuring its quality remains a challenge. Aaron McCormack, CTO and cofounder of sysWisdom.ai, is helping teams strengthen their AI quality capabilities to address the AI trust problem. Through speaking engagements, Aaron shares practical knowledge on what AI quality means, how to verify outcomes, detect drift, and monitoring for hallucinations. He also discusses scoring data for AI models and introduces a fun tool called SlopFilter. This post explores these insights to help teams build trust in their AI systems.
Reduce The AI Trust Problem By Prioritizing AI Quality
AI quality goes beyond accuracy or performance metrics. It involves ensuring AI systems behave reliably and safely over time. Aaron emphasizes that AI quality includes:
Consistency: The AI should produce stable results under similar conditions.
Robustness: The system must handle unexpected inputs or changes without failure.
Transparency: Understanding how AI makes decisions helps identify issues early.
Fault tolerance: The AI should gracefully manage errors or uncertainties.
Teams often confuse AI quality with just model accuracy, but Aaron points out that quality is a broader concept. It requires continuous monitoring and improvement, especially as AI interacts with real-world data that evolves.
What Outcomes Are Considered Slop?
Aaron introduces the concept of "slop" to describe outcomes that deviate from expected behavior but are not outright failures. Slop includes:
Minor inaccuracies that do not break the system
Slight drifts in model predictions over time
Unexpected but non-critical responses
Understanding slop helps teams set realistic quality thresholds. Instead of aiming for perfection, they focus on managing acceptable levels of variation. This approach supports building AI systems that tolerate some noise while maintaining overall reliability.
How Does Drift Occur in AI Systems?
Drift happens when the data or environment changes, causing AI models to perform worse. Aaron explains two main types of drift:
Data drift: The input data distribution shifts from what the model was trained on.
Concept drift: The relationship between input and output changes over time.
For example, a sentiment analysis model trained on social media posts from last year may struggle with new slang or topics today. Drift can lead to increased errors or unexpected outputs, reducing AI quality.
Aaron recommends continuous data monitoring and retraining models regularly to combat drift. Teams should also design systems to detect drift early and trigger alerts or updates.
Using Hallucinations to Understand Fault Tolerance
Hallucinations in AI refer to outputs that are plausible but incorrect or fabricated. While often seen as a problem, Aaron suggests using hallucinations as a tool to test fault tolerance.
By intentionally creating hallucinations, teams can:
Identify weaknesses in AI reasoning
Evaluate how the system handles uncertain or misleading inputs
Improve error detection and recovery mechanisms
This proactive approach helps build AI systems that remain reliable even when facing unexpected or faulty data.
Scoring Data for AI Models
High-quality data is the foundation of good AI. Aaron discusses how to score data to ensure it supports model performance and quality. Key factors include:
Relevance: Data should closely match the problem domain.
Accuracy: Labels and features must be correct and consistent.
Diversity: Data should cover a wide range of scenarios and edge cases.
Freshness: Recent data helps models stay current with trends.
Scoring data involves assigning metrics or labels that reflect these qualities. Teams can then prioritize cleaning, augmenting, or collecting new data based on scores. This process helps maintain a strong data pipeline that supports AI quality.
Introducing SlopFilter: A Fun Learning Tool
Aaron developed SlopFilter as a resource to help teams identify slop in AI outputs. SlopFilter is a tool that:
Detects and flags outputs that fall within slop thresholds
Provides insights into the types and frequencies of slop occurrences
Supports tuning AI systems to balance strictness and flexibility
Using SlopFilter, teams can identify where their AI systems tolerate variation and where improvements are needed. SlopFilter is a learning exercise that helps people step into the broader AI quality framework.
How to Engage with Aaron McCormack
Aaron is available for speaking engagements to help teams build their AI quality muscle. His sessions cover:
What AI quality means in practice
Detect AI sloppy outcomes
How does drift occuer in data
Using hallucinations for fault tolerance
Scoring data effectively
Organizations interested in booking Aaron can visit https://community.syswisdom.ai/. To learn more about Aaron’s background and expertise, check his LinkedIn profile at https://www.linkedin.com/in/amccormack01/.
Resources to Explore
Read the AI Quality Manifesto on GitHub: https://github.com/sysWisdom/AIQualityManifesto
Learn about SlopFilter and other tools from sysWisdom.ai’s community site
Building AI quality requires ongoing effort and practical tools. Aaron McCormack’s insights provide a clear path for teams to improve reliability, handle uncertainty, and maintain strong data practices. Engaging with experts like Aaron can accelerate this journey and help organizations build AI systems that truly deliver value.
Comments