PEFT Methods

"PEFT vs LoRA" is a common search, but it compares a category with one of its members. This guide explains how the main parameter-efficient methods differ, what they cost to train in 2026, and how to pick one for a real project.

PEFT vs LoRA: What's the Difference in 2026?

PEFT is the category; LoRA is the most popular method inside it. Parameter-efficient fine-tuning (PEFT) is the family of techniques — and the Hugging Face library of the same name — that adapt a large model by training a small fraction of its weights. LoRA (low-rank adaptation) is one such method; QLoRA is LoRA over a 4-bit quantized base; DoRA, EVA and newer variants refine how the low-rank updates are applied. Asking "PEFT vs LoRA" is like asking "databases vs Postgres."

Method (peft v0.19, 2026)What it isWhen to pick it
LoRALow-rank adapter matrices on attention/MLP layersDefault for most SFT jobs
QLoRALoRA on a 4-bit quantized base modelSingle-GPU budgets; 7B fits in ~14GB, 70B in ~44GB
DoRAWeight-decomposed LoRA (use_dora=True)Quality bump at low ranks, small overhead
EVA initData-driven adapter initialization (init_lora_weights="eva")Faster convergence on domain data
BD-LoRA / TinyLoRA / GraLoRA2026 additions in peft v0.19 (tensor-parallel serving, tiny parameter counts)Specialized serving/efficiency needs
Prefix-tuning / AdaptersEarlier PEFT approachesLegacy; rarely chosen over LoRA today

Method list verified against the Hugging Face peft library v0.19.1 (April 2026 release). My own production fine-tune, the vision-language model behind the answer-sheet reader I built for an Istanbul school, was trained with LoRA and is served with vLLM. Small vision detectors rarely need adapters: Otocropper, the RT-DETRv2 question cropper I trained for the same school, was not a LoRA job.

What Is Parameter-Efficient Fine-Tuning (PEFT)?

Traditional fine-tuning updates ALL model parameters (billions of weights), requiring massive compute resources and long training times. Parameter-Efficient Fine-Tuning (PEFT) updates only a small subset of parameters (millions). In practice that means:

  • Lower GPU cost: a 7–8B model trains on one rented GPU for a few dollars instead of a multi-GPU node (July 2026 prices)
  • Faster runs: hours rather than days
  • Small artifacts: an adapter is megabytes, so one base model can serve several tasks
  • Quality close to full fine-tuning on most narrow tasks

The 3 Main PEFT Methods Explained

1. LoRA (Low-Rank Adaptation)

How it works: Adds small trainable matrices alongside frozen base model weights.

  • Parameters trained: 0.1-1% of total
  • Best for: General-purpose adaptation across most use cases
  • Complexity: Moderate (well-documented, widely supported)

2. Prefix-Tuning

How it works: Adds trainable "prefixes" (virtual tokens) to input sequences, leaving model weights completely frozen.

  • Parameters trained: 0.01-0.1% of total
  • Best for: Task-specific adaptation with minimal training data
  • Complexity: Low (simplest implementation)

3. Adapter Layers

How it works: Inserts small bottleneck neural networks between transformer layers.

  • Parameters trained: 1-5% of total
  • Best for: Multi-task scenarios where you need to switch between different adaptations
  • Complexity: High (requires modifying model architecture)

Head-to-Head Comparison

How the methods differ in practice. Parameter shares are typical ranges; your exact numbers depend on rank, target layers and model size.

MethodTrainable parametersExtra inference latencyUses context windowTypical use in 2026
LoRA / QLoRAAbout 0.1–1%None once merged into the base weightsNoThe default choice
Prefix tuningAbout 0.01–0.1%SmallYes, the virtual tokens take up sequence lengthRare
Adapter layersAbout 1–5%Small, extra layers on every forward passNoOlder multi-task setups
Full fine-tune100%NoneNoLarge behavior changes with a large budget

Short version: start with LoRA, or QLoRA if GPU memory is tight. Reach for the others only when you have a specific reason.

When to Use Each PEFT Method

Use LoRA When:

  • General text generation or classification tasks
  • GPU budget: a few dollars per run for 7–8B models with QLoRA
  • You need good quality with fast training
  • You want adapters you can merge or swap at serving time

Examples: Customer support chatbots, email classification, content generation, document summarization

Use Prefix-Tuning When:

  • You need many tiny task-specific variants of one frozen model
  • Simple, focused tasks (binary classification, sentiment analysis)
  • Your prompts are short, so the virtual tokens don't crowd out real input
  • Accuracy a little below LoRA is acceptable

Examples: Lead scoring, spam detection, simple categorization

Use Adapter Layers When:

  • Multiple related tasks requiring different adaptations
  • You already run an adapter-based stack and switching costs more than it saves
  • You have larger training datasets (50,000+ examples)
  • You need to dynamically switch between tasks

Examples: Multi-language support, multi-domain document processing, complex workflow automation

Cost Comparison: One Year, Three Tasks

Say you run three small tasks on a 7–8B model (email classification, lead scoring, ticket routing) and retrain each one quarterly: 12 training runs a year. At July 2026 rental rates:

  • Three full fine-tunes: each run needs a multi-GPU node, roughly $230–$800 of A100 time, so about $2,700–$9,600 of GPU time a year. You also host three full-size model copies.
  • Three LoRA/QLoRA adapters: each run costs $3–$10 of GPU time, so about $40–$120 a year. One base model serves all three adapters; vLLM can load several LoRA adapters on the same base model.

The serving saving usually matters more than the training saving: one base model in memory instead of three. Engineering time for data and evaluation is the same either way. The LoRA cost analysis has the per-run prices and a calculator.

A Typical PEFT Project, Week by Week

A realistic plan for a first fine-tune, from task definition to production:

Week 1: Task Analysis

  • Evaluate your use case and requirements
  • Determine best PEFT method
  • Calculate expected ROI

Weeks 2-3: Data Preparation

  • Clean and format training data
  • Create validation and test sets
  • Establish baseline performance metrics

Weeks 4-5: Training & Optimization

  • Initial PEFT training
  • Hyperparameter tuning
  • Performance benchmarking against requirements

Week 6: Deployment

  • Production environment setup
  • Load testing and optimization
  • Monitoring configuration

Ongoing: Continuous Improvement

  • Quarterly re-training with new data
  • Performance monitoring and alerting
  • Iterative improvements

Tools for PEFT in 2026

  • Hugging Face PEFT: the reference library for LoRA, QLoRA, DoRA, prefix tuning, prompt tuning and adapters. The method table above is checked against v0.19.
  • TRL: Hugging Face's training library; its SFTTrainer takes a PEFT config directly.
  • Unsloth: faster, lower-memory LoRA and QLoRA training for popular open models.
  • Axolotl: config-file driven fine-tuning on top of PEFT and TRL, useful when you run many experiments.
  • vLLM: serves a base model with several LoRA adapters loaded at once. It is what I use in production.

10 Questions to Ask Your AI Vendor About PEFT

  1. "Which PEFT method do you recommend and why?" (Should be based on your specific requirements)
  2. "What's the training cost breakdown?" (Get itemized costs)
  3. "How many parameters will be trained?" (Fewer = cheaper, but need quality guarantee)
  4. "What's the expected accuracy compared to full fine-tuning?" (Should be within 2-3%)
  5. "How long does training take?" (Hours, not days)
  6. "What's the inference latency?" (Should be <100ms for most applications)
  7. "Can you show me similar case studies?" (Real examples, not hypotheticals)
  8. "What's included in ongoing support?" (Re-training, monitoring, updates)
  9. "How often will we need to re-train?" (Typically quarterly)
  10. "How do you measure accuracy after launch?" (Ask what happens if it drops)

Not Sure Which Method Fits?

Describe the task, the data you have and where the model has to run. I'll tell you whether LoRA, QLoRA or plain prompting is the better start, and what the first run should cost.

Frequently Asked Questions

Is LoRA the same as PEFT?

No. PEFT (parameter-efficient fine-tuning) is the category of techniques, and the Hugging Face library, for adapting large models by training a small fraction of weights. LoRA is the most widely used method within PEFT; QLoRA and DoRA are refinements of it.

What is the difference between LoRA and QLoRA?

QLoRA is LoRA applied on top of a 4-bit quantized base model. It cuts the GPU memory needed for training: a 7B model fine-tunes in about 14GB and a 70B in about 44GB, at a small quality cost that is usually negligible for supervised fine-tuning.

Which PEFT method should I use in 2026?

Start with QLoRA if you are short on GPU memory, LoRA otherwise. Enable DoRA (use_dora=True in peft) when you need extra quality at low ranks. Prefix tuning and classic adapters are rarely chosen over LoRA today.