Skip to main content

What is inference

What is inference

Inference is the process of running a trained AI model on new data to get a result. When a model classifies an image, generates a paragraph of text, predicts a risk score, or recommends an action, it is performing inference. If training is how the model learns, inference is how it works.

A useful analogy: training is studying for an exam -- reading textbooks, solving practice problems, absorbing patterns. Inference is taking the exam -- applying what was learned to new questions under real conditions. Every time an AI system produces an output in production, that is inference.

In short: training is studying for an exam; inference is taking the exam. Every time an AI system produces an output in production, that is inference.

How inference works step by step

Although inference looks instantaneous to the end user ("I typed a question and got an answer"), a concrete pipeline runs behind the scenes every time a model produces output.

Input

Everything starts with input data -- a text prompt, an image, a set of numbers, a blockchain transaction hash, an audio clip, or any other structured or unstructured data that the model was designed to process.

Preprocessing

Raw input is rarely fed directly to the model. A preprocessing step transforms it into the format the model expects. For a text model, this means tokenization -- splitting text into tokens (sub-word units) and converting them to numerical IDs. For an image model, this means resizing, normalizing pixel values, and sometimes augmenting the image. Preprocessing must exactly match the steps used during training; otherwise, the model receives data in a format it was never trained to handle.

Forward pass

The preprocessed input is passed through the model's architecture -- layer by layer, through all learned weights and activation functions. This is called the forward pass. During this step, the model applies the patterns it learned during training to the new input. No weights are updated; no learning happens. The model simply computes an output from the input using its existing knowledge.

For a large language model, the forward pass produces a probability distribution over possible next tokens. For a classifier, it produces probabilities for each class. For a generative image model, it produces a de-noised image step by step.

Output

The forward pass produces a raw output -- often a vector of numbers, probabilities, or a sequence of tokens. This raw output needs to be interpreted to become useful.

Postprocessing

Postprocessing converts the raw output into a human-readable or system-usable result. For an LLM, this means decoding token IDs back into text. For a classifier, it means mapping probability scores to class labels ("spam" vs "not spam"). For a scoring model, it means converting a raw value into a score range the application expects.

In summary:

Step What happens Example (text model)
Input Raw data arrives User types a question
Preprocessing Data is transformed into model format Text is tokenized into numerical IDs
Forward pass Model computes output from input using learned weights Probability distribution over vocabulary
Raw output Numerical result produced Token probabilities
Postprocessing Output is converted to usable form Token IDs decoded back into text

Types of inference

Not all inference is the same. Different deployment scenarios create different technical requirements and tradeoffs.

Real-time vs batch inference

Dimension Real-time inference Batch inference
When it runs On demand, per request On a schedule or when a batch accumulates
Latency requirement Low (milliseconds to seconds) Flexible (minutes to hours acceptable)
Use case Chatbots, live scoring, interactive agents Report generation, bulk classification, periodic analytics
Cost profile Higher per-request (hardware must be always ready) Lower per-item (hardware used efficiently in bursts)

Real-time inference is what end users typically experience -- they send a query and receive an answer immediately. Batch inference is common in backend pipelines where a model processes thousands of data points at once (e.g., scoring all transactions from the past hour for fraud risk).

Edge vs cloud inference

  • Cloud inference runs on centralized servers (or server clusters) in a data center. It offers powerful hardware (high-end GPUs, large memory) but introduces network latency and dependency on a central provider.
  • Edge inference runs on local devices -- smartphones, IoT sensors, embedded hardware. It reduces latency and keeps data on-device but is limited by the device's compute power and memory.

Many production systems combine both: lightweight models run on the edge for fast initial responses, while complex queries are forwarded to cloud-based models.

On-chain vs off-chain inference

This distinction is specific to crypto AI and is central to how DecAIHub evaluates projects:

  • Off-chain inference -- the model runs on traditional servers or GPU clusters. Results may or may not be posted to a blockchain afterward. This is the most common setup in crypto AI projects because running large models directly on a blockchain is computationally impractical.
  • On-chain inference -- the model (or a verifiable proof of its computation) executes within the blockchain's execution environment. True on-chain inference for non-trivial models is extremely rare due to the computational limitations of smart contract environments. Projects that claim on-chain inference typically use one of these approaches: running a very small model within a smart contract, posting a cryptographic proof (such as zkML) that the inference was performed correctly off-chain, or using an oracle-like mechanism where off-chain inference results are attested and recorded on-chain.

The distinction between on-chain and off-chain inference is directly relevant to the Onchain necessity axis in DecAI Fit.

Why inference matters in crypto AI

If a project claims to use artificial intelligence, inference must be happening somewhere. A model that was trained but never runs inference produces no outputs and delivers no value to users. Inference is the point where AI stops being a concept and starts being a product.

In DecAIHub project cards, the AI component field often mentions inference-related mechanisms: inference engines, decentralized inference, LLM inference endpoints, model serving. These are concrete indicators that a model is actually being used -- not just claimed.

Cost, speed, and the compute connection

Inference is not free. Running a model requires compute resources -- typically GPUs or specialized accelerators like TPUs and custom ASICs. The cost of inference depends on several factors:

Factor Impact on cost and speed
Model size (parameter count) Larger models require more memory and compute per request
Input length Longer inputs (e.g., long text prompts) increase processing time
Output length For autoregressive models (LLMs), each output token requires a forward pass
Batch size Processing multiple requests together can improve throughput but adds latency
Hardware More powerful GPUs reduce time but cost more
Quantization Reducing numerical precision (e.g., from 32-bit to 8-bit) cuts memory and compute at the cost of some accuracy

Projects optimize inference through several techniques:

  • Quantization -- reducing the numerical precision of weights to use less memory and compute.
  • Distillation -- training a smaller "student" model to mimic a larger "teacher" model, producing a faster model with acceptable quality loss.
  • Batching -- grouping multiple requests together to process them simultaneously and share compute overhead.
  • Caching -- storing and reusing results for repeated or similar queries.
  • Hardware selection -- choosing the right GPU, TPU, or accelerator for the workload.

The compute infrastructure that powers inference is a major area in crypto AI. Many projects build or coordinate GPU networks specifically to serve inference workloads, falling under the AI/Compute or AI/Inference categories.

Decentralized inference

In traditional AI, inference runs on centralized servers controlled by one company. Decentralized inference distributes model serving across a network of independent providers. This is a core pattern in the AI/Inference category of crypto AI projects.

Decentralized inference introduces a set of challenges that do not exist in centralized systems:

  • Output consistency -- different providers must produce identical (or sufficiently similar) outputs for the same input. Hardware differences, software versions, and numerical precision can cause divergence.
  • Verification -- how do you prove that the correct model was actually used, that the input was not tampered with, and that the output was computed honestly? Approaches include multi-provider consensus (running the same query on multiple nodes and comparing results), trusted execution environments (TEE) that provide hardware-level guarantees, and zero-knowledge proofs (zkML) that prove correct computation without revealing model weights or input data.
  • Latency -- routing requests to distributed providers adds network overhead compared to centralized servers.
  • Payment coordination -- providers need to be compensated fairly, which often involves on-chain payment mechanisms tied to verified inference work.

These mechanisms are described in detail on the Inference as an AI component page.

Where it appears in DecAIHub project cards

Inference surfaces in multiple parts of a DecAIHub project card:

  • AI component -- inference-related values such as "inference," "model serving," "LLM inference," or "decentralized inference" indicate that a project's core AI function involves running models in production.
  • Category -- projects in the AI/Inference category focus specifically on providing inference infrastructure or services.
  • Evidence table -- Tier-1 evidence for inference includes public API endpoints, inference benchmarks, latency measurements, and verifiable compute logs.
  • ai_score -- a project that demonstrates working inference with published evidence receives a higher ai_score than one that only describes a model without showing it runs.

A project can claim to have a model, but if there is no evidence that inference is happening in production, the model provides no value to end users.

What inference alone does not prove

  • Running inference on a third-party model via API (e.g. calling GPT or Claude) is technically inference but does not demonstrate proprietary AI capability. See What is an API wrapper.
  • Inference proves a model runs -- it does not prove the model is high quality or well-suited to the task. Quality requires benchmarks and metrics.
  • Inference does not prove that a token is necessary for the pipeline to function or that the project needs to be on-chain. These are separate axes in DecAI Fit.
  • Speed claims without published benchmarks or methodology are unverifiable. Claiming "fastest inference" requires reproducible evidence.
  • The existence of an inference endpoint does not prove the model behind it was built by the project rather than being a hosted copy of an open-source model.

Always check the evidence table and Tier-1/2/3 evidence for supporting artifacts.

Do not confuse with

  • Training produces the model. Inference uses it. They are sequential stages of the AI lifecycle.
  • Inference as an AI component describes how inference appears in DecAIHub project cards specifically. This page explains the concept itself.
  • AI/Inference category classifies projects that focus on inference infrastructure. The concept of inference applies to any project that runs a model, regardless of its category.
  • Compute provides the hardware. Inference is the workload that runs on that hardware. A compute network can serve both training and inference workloads.
  • AI model is the artifact (the trained weights). Inference is the process of using that artifact to produce outputs.

Frequently asked questions

What is the difference between training and inference?

Training is the process of teaching a model by exposing it to data and adjusting its weights to minimize errors. It is computationally expensive and typically happens once (or periodically for retraining). Inference is the process of using the trained model to produce outputs on new data. It happens every time a user interacts with the model. Training creates the model; inference uses it.

Why is inference speed important?

Speed determines user experience and operational cost. A chatbot that takes 30 seconds to respond is unusable in practice. A fraud detection system that takes minutes to score a transaction misses the window to act. Faster inference also means lower compute cost per request, which directly affects the economic viability of a service -- especially for decentralized networks where providers bear the hardware cost.

What is decentralized inference?

Decentralized inference distributes model serving across a network of independent compute providers instead of relying on a single company's servers. The key challenges are ensuring output consistency, verifying that the correct model was actually used, managing latency, and coordinating payments. Crypto AI projects in the AI/Inference category often build infrastructure specifically for this purpose.

How can you verify that inference is actually happening?

Look for Tier-1 evidence: public API endpoints that can be tested, inference latency benchmarks, compute logs, on-chain records of inference requests and results, or verifiable proofs (such as TEE attestations or zkML proofs). Marketing claims like "powered by AI" without any of these artifacts do not constitute evidence of real inference.

What is the cost of running inference?

Cost depends on model size, input/output length, hardware, and optimization techniques. Running a single query through a large language model can cost fractions of a cent to several cents depending on the provider and model size. For decentralized inference networks, costs are typically paid per query or per compute unit, often using the network's native token or a stablecoin. The economics of inference are a key factor in whether a project's AI pipeline is sustainable at scale.

See also

External references

No comments to display

Back to top