What counts as evidence of an AI component
What counts as evidence of an AI component
When DecAIHub assesses whether a project has a working AI component, it looks for verifiable artifacts — not marketing copy, not promises, not brand associations. This page lists the kinds of evidence that strengthen the AI reality axis of the DecAI Fit score, ordered from strongest to weakest.
In short: evidence of AI means artifacts you can inspect, test, or independently verify — code, documentation, metrics with methodology, working demos, and third-party evaluations.
Tier-1 evidence (strongest)
Tier-1 sources are artifacts published or controlled by the project itself. For the AI reality axis, the strongest Tier-1 evidence includes:
- Public repository with AI/ML code — model training scripts, inference logic, data pipelines, preprocessing steps. A repository with commits, releases, and tests is far stronger than a placeholder repo with a README.
- Technical documentation — descriptions of model architecture, input/output formats, model versions, and update history. This answers "what model, how it works, and what version is running."
- Quality metrics with methodology — published benchmarks or evaluation results that specify what was measured, on what data, when, and with what version of the model. Metrics without this context are not evidence.
- Working demos or endpoints — a publicly accessible interface, API, or sandbox where the AI component can be tested. A demo that processes real inputs and returns AI-driven outputs is direct evidence.
- Audit reports or third-party evaluations — independent assessments of model quality, fairness, safety, or performance by a recognized entity.
Patterns that strengthen Tier-1 evidence
Not all Tier-1 artifacts are equally convincing. Several observable patterns make evidence more credible:
- Commit history depth. A repository with months or years of incremental commits, pull requests, code reviews, and versioned releases demonstrates sustained engineering effort. A single bulk upload carries far less weight.
- Test coverage and CI pipelines. Automated tests for model inference, data validation, and integration endpoints indicate a production-grade system rather than a proof-of-concept.
- Versioned models and changelogs. When documentation tracks which model version is deployed, what changed between versions, and why, it signals a living system with an active development cycle.
- Reproducible outputs. If a demo or API returns consistent, explainable results for the same inputs across sessions, this is stronger than a one-off showcase.
- Separation of AI logic from boilerplate. A codebase where AI/ML modules are clearly identifiable — distinct from frontend scaffolding, deployment scripts, or token contracts — makes it easier to verify what the AI component actually does.
Tier-2 evidence (context, not proof)
Tier-2 sources provide context but do not prove a working AI component on their own:
- Interviews or AMAs where the team describes the AI approach, architecture, or training process.
- Aggregator descriptions (CoinGecko, DeFiLlama, etc.) mentioning AI capabilities.
- Partnership announcements with AI providers or research labs.
Tier-2 evidence can support a Tier-1 finding but cannot replace it. A partnership with an AI company does not prove the project has its own working AI.
When Tier-2 evidence matters
Tier-2 sources become relevant when they corroborate Tier-1 artifacts or fill in gaps that Tier-1 alone leaves ambiguous:
- A technical interview where the team walks through the architecture described in their documentation adds credibility to the Tier-1 docs.
- An aggregator listing that matches the capabilities described in the project's own whitepaper signals consistency.
- A conference talk with a live demo that mirrors the behavior of the project's public API reinforces that the demo is not scripted.
Without underlying Tier-1 artifacts, however, even compelling Tier-2 sources remain context — valuable for understanding, insufficient for verification.
What does not count
Some artifacts look like evidence but do not demonstrate a working AI component. Common false signals include marketing copy, empty repositories, metrics without methodology, and API wrappers with no proprietary logic. For a detailed list, see What does not count as evidence of AI.
How DecAIHub records AI evidence: flags AI_0 through AI_4
The AI reality axis uses five binary flags to capture what evidence has been found:
| Flag | What it checks | Value |
|---|---|---|
| AI_0 | Is there at least a Tier-1 declaration of AI intent (roadmap, official update) | 1 / 0 |
| AI_1 | Is there a description of what the AI does | 1 / 0 |
| AI_2 | Are there demos or working examples | 1 / 0 |
| AI_3 | Are there quality metrics with context (what, on what data, when) | 1 / 0 |
| AI_4 | Is there a technical description, architecture, or model versions | 1 / 0 |
The more flags equal 1, the higher the ai_score, and the better the project's DecAI Fit. If none of the flags are set, ai_score = 0 and DecAI Fit cannot exceed 0.
How flag combinations map to confidence
Individual flags answer narrow questions, but their combination paints a fuller picture:
| Flags set | Typical interpretation |
|---|---|
| AI_0 only | Intent declared; no working AI verified yet |
| AI_0 + AI_1 | AI described in words; no demo, no metrics, no architecture |
| AI_0 + AI_1 + AI_2 | A working demo exists; still no metrics or technical depth |
| AI_0 + AI_1 + AI_2 + AI_4 | Demo and architecture present; quality measurement missing |
| All five (AI_0 through AI_4) | Strongest evidence profile: intent, description, demo, metrics, and architecture |
A project does not need to satisfy all five flags to receive a nonzero ai_score. Even AI_0 alone (intent with a Tier-1 source) sets ai_score = 1. However, higher flags contribute to a higher ceiling for DecAI Fit.
Why methodology matters for metrics
A standalone number — "95% accuracy," "low latency," "state-of-the-art" — is not evidence without context. DecAIHub considers metrics meaningful only when they answer four questions: what was measured (accuracy, F1, latency, perplexity), on what data (public benchmark, proprietary test set, production traffic), when (date of evaluation), and what version of the model produced the result. Without these, a metric is a claim, not evidence.
The four-question test
When evaluating any published metric, apply this checklist:
- What — Is the metric named and defined? "Accuracy" is ambiguous; "top-1 classification accuracy" is specific.
- On what data — Is the evaluation dataset identified? A public benchmark allows independent reproduction; a proprietary test set does not, but naming it is better than silence.
- When — Is there a date or release tag? Models change over time; a metric without a timestamp may not reflect the current system.
- What version — Is the model or system version stated? If a project has shipped multiple model iterations, the metric must be tied to a specific one.
If any of these four are missing, the metric drops from "evidence" to "claim" in DecAIHub's assessment. It may still appear in the project card as a noted claim, but it will not contribute to setting the AI_3 flag to 1.
How to use this checklist
When reading a project card, check whether the Evidence table references artifacts from the Tier-1 list above. If the card shows ai_score = 0 or flags an AI-washing concern, it means these artifacts were not found as of the verification date — this is not an accusation against the project.
When new Tier-1 artifacts appear, the card is updated and the score may change.
Practical steps for readers
- Start with the Evidence table. Look for links to repositories, documentation, or demos. If the Evidence table is empty or references only Tier-2/3 sources, the AI reality claim is unverified.
- Click through to the artifacts. A repository link is only as good as what is inside it. Open it and look for AI/ML code, not just a README or deployment scripts.
- Check the verification date. Evidence is a snapshot. A card verified six months ago may not reflect the project's current state, which is why DecAIHub applies a staleness penalty after 550 days.
- Cross-reference with the anti-pattern list. If you see patterns from What does not count as evidence of AI, weight the evidence accordingly.
What this checklist alone does not prove
- Satisfying all evidence criteria does not prove the AI is high quality or useful to end users. It proves artifacts exist and can be inspected.
- Tier-1 evidence from the project itself does not prove independence. Self-reported data should ideally be corroborated by third-party evaluation.
- A complete evidence record does not prove the token is necessary. AI reality is one axis of DecAI Fit; token necessity is another.
Frequently asked questions
What counts as the most complete evidence of AI in a crypto project?
A public repository containing AI/ML code with a meaningful commit history, combined with published quality metrics that include full methodology (what was measured, on what data, when, and which model version). When both of these exist and are supported by a working demo or API endpoint, the evidence profile is at its most complete. No single artifact is definitive on its own — the combination of code, metrics, and testable outputs is what distinguishes verified AI from unverified claims.
Can a project have real AI without open-source code?
Yes. Open-source code is the most transparent form of evidence, but it is not the only one. A project may demonstrate AI through detailed technical documentation, published benchmarks with methodology, working APIs that return verifiable AI-driven outputs, or independent third-party evaluations. However, without at least some inspectable artifact — code, demo, or evaluation — there is nothing for DecAIHub (or anyone else) to verify. The more closed a project is, the more it relies on Tier-2 corroboration (interviews, aggregator data, conference demonstrations) to build credibility.
Does a working demo prove the AI is real?
A working demo is strong evidence, but it is not conclusive proof by itself. A well-designed frontend can simulate AI-like behavior using hardcoded responses, pre-computed results, or hidden calls to a third-party API. DecAIHub treats demos as stronger evidence when they are accompanied by API documentation, model version identifiers, or reproducible outputs across different inputs. A demo that accepts arbitrary user inputs and returns varied, contextually appropriate outputs is more convincing than a guided showcase with predetermined examples.
What is the minimum evidence needed for a nonzero ai_score?
The minimum is a Tier-1 declaration of AI intent — a roadmap entry, official blog post, or documentation page that explicitly states AI-related plans. This sets the AI_0 flag to 1 and produces ai_score = 1. While this is the lowest possible nonzero score, it distinguishes projects that have officially committed to an AI direction from those that have not mentioned AI at all. To reach higher scores, the project needs to provide additional artifacts: descriptions of the AI component (AI_1), demos (AI_2), metrics with methodology (AI_3), and technical architecture details (AI_4).
External references
- CommonMark specification — the markup standard used across all DecAIHub wiki pages, including project cards and explainers.
- Model Cards for Model Reporting (Mitchell et al., 2019) — a foundational framework for documenting AI model performance, intended use, and limitations; closely aligned with the transparency principles behind DecAIHub's evidence criteria.
- NIST AI Risk Management Framework — a U.S. government framework for identifying and managing AI risks, including guidance on documentation, testing, and evaluation methodology.
No comments to display
No comments to display