Prompting vs Looping: Why Most Engineers Are Burning Through AI Credits Wrong

The Pattern Nobody's Talking About

I've reviewed over 40 production LLM integrations in the past six months - everything from customer support bots to code generation tools - and nearly every single one makes the same architectural mistake. They treat the language model like a black box oracle: send a prompt, get a response, hope for the best. When the output isn't quite right, they add more instructions to the prompt. When it's still not right, they add examples. When it's still not right, they switch to a more expensive model.

This is prompting. It's simple, it's intuitive, and for complex tasks, it's almost always the wrong approach.

The alternative is looping - decomposing the task into discrete steps, validating outputs, maintaining state across multiple model calls, and using your application code to enforce structure that the model alone cannot guarantee. It's more engineering effort upfront, but it's also the difference between a demo that works 70% of the time and a production system that works 99% of the time.

The industry conversation around LLMs is obsessed with prompt engineering - how to phrase instructions, how to structure few-shot examples, how to cajole better reasoning out of the model. But the real leverage isn't in better prompts. It's in better architecture.

Prompting: When One Call Does Everything

Prompting is the default pattern. You construct a single request containing the full task description, any relevant context, examples if needed, and send it to the model. The model processes everything in one inference pass and returns a response. Done.

This works beautifully for tasks that are naturally atomic: summarising text, generating a short-form response, classifying sentiment, translating a sentence. The input is bounded, the output format is simple, and there's no state to maintain across calls.

const response = await anthropic.messages.create({
  model: "claude-sonnet-4",
  max_tokens: 1024,
  messages: [{
    role: "user",
    content: "Summarise this article in 3 bullet points: ..."
  }]
});

const summary = response.content[0].text;
// Done. No validation, no retry, no state.

The appeal is obvious: minimal code, low latency (one round trip), and straightforward debugging. If it doesn't work, you tweak the prompt and try again. The model does all the reasoning internally.

But this simplicity is deceptive. As soon as the task becomes multi-step, or the output needs to conform to a strict schema, or the result depends on external validation, prompting starts to break down. You end up with prompts that are thousands of tokens long, instructions that contradict each other, and outputs that are correct 80% of the time but fail silently the other 20%.

Looping: When You Take Control

Looping inverts the control flow. Instead of sending one massive prompt and hoping the model gets everything right in one shot, you break the task into stages. At each stage, your application code calls the model, validates the output, maintains state, and decides what to do next. The model becomes a component in a larger system rather than the system itself.

This is not just calling the API multiple times in sequence - that's still prompting, just slower. Looping means your code is making decisions based on the model's outputs and adjusting behaviour dynamically.

let result = null;
let attempts = 0;

while (!result && attempts < 3) {
  const response = await callModel(prompt);
  const parsed = tryParseJSON(response);

  if (parsed && validateSchema(parsed)) {
    result = parsed;
  } else {
    // Adjust prompt with specific error feedback
    prompt += `\n\nPrevious attempt failed validation: ${getErrorMessage(parsed)}`;
    attempts++;
  }
}

if (!result) throw new Error('Failed after retries');

Notice what's happening here. The application is not blindly accepting whatever the model returns. It's checking the structure, validating correctness, and providing targeted feedback when things go wrong. The model is no longer responsible for getting everything perfect on the first try - it's responsible for responding to feedback iteratively until the output is valid.

This fundamentally changes what you can build. Complex workflows like extracting structured data from unstructured text, generating valid code that compiles, or reasoning through multi-step problems all become tractable because you're not relying on the model's internal coherence alone. You're using code to enforce coherence externally.

Why Prompting Fails at Scale

The core issue with prompting for complex tasks is that language models are probabilistic. Even with temperature set to zero, the same prompt can produce different outputs across runs due to internal sampling variations and context-dependent reasoning paths. For creative tasks, this variability is a feature. For production systems, it's a bug.

Consider a simple task: extract all email addresses from a customer support transcript and return them as a JSON array. With prompting, you'd write a detailed instruction explaining the format, provide examples, and hope the model returns valid JSON. Most of the time, it will. Sometimes, it will wrap the JSON in markdown code fences. Sometimes, it will include explanatory text before or after the JSON. Sometimes, it will return an object instead of an array. Sometimes, it will hallucinate email addresses that don't exist in the transcript.

Each of these failure modes is rare individually - maybe 2-5% occurrence each - but cumulatively, your reliability drops to 85-90%. In a demo, that feels fine. In production, where you're processing thousands of requests per day, it means hundreds of failures that require manual intervention.

With looping, you add a validation layer. The model returns something. Your code checks if it's valid JSON. If not, you tell the model exactly what went wrong and ask again. If it's valid JSON but not an array, you tell it that. If it's an array but contains emails that don't appear in the transcript, you reject those entries. The model learns from feedback in-context and produces a corrected output.

This costs more in API calls - typically 2-4x depending on task complexity - but the reliability jump is worth it. You go from 87% success to 98% success, and the failure cases that remain are edge cases that genuinely require human review, not silent data corruption.

When to Prompt vs When to Loop

The choice isn't binary. Many production systems use both patterns at different stages of a pipeline. The question is: which pattern fits which part of your problem?

Use Prompting When:

The task is naturally atomic. If success or failure is determined entirely by the quality of a single output, and there's no way to programmatically validate correctness, prompting is fine. Example: generating marketing copy where "good" is subjective.

Latency matters more than correctness. If you need a response in under 2 seconds and can tolerate occasional errors, a single well-crafted prompt will outperform a multi-turn loop. Example: chatbot responses where users expect speed.

The output doesn't feed into downstream logic. If a human is going to review the result anyway, the cost of occasional malformed outputs is low. Example: draft email generation for user review.

Use Looping When:

The output must conform to a strict schema. If downstream code depends on specific fields existing and containing valid data, you need validation and retry logic. Example: extracting invoice line items from receipts.

The task has multiple dependent steps. If step 2 depends on the output of step 1, and you need to verify step 1 succeeded before proceeding, looping gives you checkpoints. Example: multi-step SQL query generation where each query must be syntactically valid.

External validation is possible. If you can programmatically check whether the model's output is correct (running code, querying a database, checking mathematical consistency), looping lets you give the model targeted feedback instead of hoping it self-corrects. Example: code generation where you can run tests.

The Hidden Cost: Context Window Exhaustion

One downside of looping that doesn't get enough attention is context window management. Every time you call the model in a loop, you're typically appending the previous conversation history to maintain coherence. After 5-6 iterations, you can easily hit 10,000+ tokens of conversation history for a task that would have been 2,000 tokens as a single prompt.

This matters for two reasons. First, cost - you're paying for input tokens on every call, and those tokens accumulate quickly. Second, performance - models perform worse as the context fills up, with attention degrading for information far back in the conversation history.

The solution is aggressive context pruning. Don't naively append the full conversation. Instead, maintain a sliding window of the most recent exchanges, and compress older context into a structured summary that the model can reference without re-reading the full text.

function buildContextWindow(conversationHistory, maxTokens) {
  const recent = conversationHistory.slice(-3); // Last 3 turns
  const older = conversationHistory.slice(0, -3);

  const summary = compressToKeyFacts(older);
  // "Previous attempts: invalid JSON (2x), missing field 'email' (1x)"

  return [
    { role: 'system', content: summary },
    ...recent
  ];
}

This approach keeps context growth linear rather than quadratic. You're not re-sending the full transcript of every failed attempt - you're sending a compressed summary of what's been tried and what went wrong, plus the most recent exchanges where the model is actively iterating.

In practice, this typically keeps context under 5,000 tokens even for loops that run 10+ iterations, compared to 20,000+ tokens with naive appending. The model's performance stays consistent, and your costs scale reasonably.

The Illusion of Determinism

Many engineers assume that setting temperature to 0 makes the model deterministic. It doesn't. Temperature controls the randomness of the sampling process, but it doesn't eliminate all sources of variation. Models use top-k and nucleus sampling, and even at temperature 0, tie-breaking between equally probable tokens can introduce variance.

More importantly, the model's reasoning process itself is path-dependent. The same prompt can trigger different reasoning chains depending on subtle context differences - like whether you included a trailing newline, or what the previous conversation was, or even what time of day the request was made (due to load-based routing to different instances with slightly different weights).

Looping gives you a way to enforce determinism at the application level even when the model isn't deterministic at the inference level. By validating outputs against a schema and rejecting anything that doesn't match, you're collapsing the model's probabilistic output space into a deterministic set of acceptable results.

This is why you see production LLM applications with elaborate validation pipelines, retry logic, and fallback strategies. They're not over-engineering - they're compensating for the fact that language models are fundamentally probabilistic systems being used in contexts that demand deterministic behaviour.

Real-World Example: Invoice Data Extraction

Let's ground this in a concrete scenario. You're building a system that extracts structured data from uploaded invoice PDFs. The model needs to identify vendor name, invoice number, date, line items, subtotal, tax, and total.

Prompting Approach

You write a detailed prompt explaining the task, provide examples of valid JSON outputs, and send the invoice text to the model. The model returns JSON most of the time. Sometimes it misses a field. Sometimes it hallucinates data. Sometimes the date format is inconsistent. You fix edge cases by adding more examples to the prompt. Reliability plateaus around 85%.

Looping Approach

You break the task into stages. Stage 1: extract vendor name and invoice number. Validate these exist and are non-empty. Stage 2: extract date and parse it into ISO 8601 format. If parsing fails, ask the model to reformat. Stage 3: extract line items. Validate that quantities are positive integers and amounts are valid numbers. Stage 4: validate that subtotal + tax = total. If not, ask the model to recalculate.

Each stage has specific validation logic. The model gets targeted feedback when it fails a validation. The final output is guaranteed to be schema-compliant because it passed every validation gate. Reliability hits 98%, and the 2% failures are cases where the invoice itself is malformed or illegible - not extraction errors.

async function extractInvoiceData(invoiceText) {
  let data = {};

  // Stage 1: Basic fields
  data.vendor = await extractWithRetry('vendor name', invoiceText,
    val => val.length > 0);

  data.invoiceNumber = await extractWithRetry('invoice number', invoiceText,
    val => /^[A-Z0-9-]+$/.test(val));

  // Stage 2: Date with format validation
  data.date = await extractWithRetry('invoice date', invoiceText,
    val => !isNaN(Date.parse(val)));

  // Stage 3: Line items with numeric validation
  data.lineItems = await extractWithRetry('line items as JSON array', invoiceText,
    items => items.every(i => i.quantity > 0 && i.amount > 0));

  // Stage 4: Totals with math validation
  const totals = await extractTotalsWithValidation(data.lineItems, invoiceText);

  return { ...data, ...totals };
}

The key insight: each validation gate is trivial to implement in code but nearly impossible to guarantee in a single prompt. The model can reason about what makes a valid invoice number, but it can't guarantee it will always output one. Your code can.

The Cost-Reliability Tradeoff

Let's be direct about costs. Looping is more expensive per task. A single-prompt system might average 1.2 API calls per task (1 call plus occasional retries when things break). A looped system might average 3.5 API calls per task. At Claude Sonnet pricing, that's the difference between $0.12 and $0.42 per 1,000 tasks for similar-length prompts.

But what's the cost of a failure? If 13% of your single-prompt tasks produce invalid outputs that require manual review, and manual review costs $2 per case, your effective cost per 1,000 tasks is actually $0.12 + (130 × $2) = $260.12. The looped system at $0.42 per 1,000 tasks with 2% failure rate costs $0.42 + (20 × $2) = $40.42.

This is the calculation most teams get wrong. They optimise API costs without accounting for the hidden cost of unreliability - customer support burden, data quality issues, engineering time spent debugging edge cases, and reputation damage from users encountering broken outputs.

There's also a less tangible benefit: predictability. A system with 85% reliability has unpredictable failure modes - you never know which 15% of cases will fail until they do. A system with 98% reliability has predictable failures - you know exactly which edge cases cause problems and can handle them explicitly.

Building Systems, Not Demos

The transition from prompting to looping is the transition from demo to production. Prompting is fine for exploration, prototyping, and tasks where correctness is subjective. But if you're building a product feature that users depend on, you need control.

Looping gives you that control. It lets you enforce schemas, validate outputs, provide feedback, and guarantee that the final result is usable by downstream code. It costs more in API calls but dramatically less in total ownership cost when you account for failures.

The deeper point is about how you think about LLMs in your architecture. If you treat them as magic black boxes, you'll struggle with reliability. If you treat them as probabilistic components in a deterministic system - where your code provides the deterministic scaffolding and the model provides the reasoning - you can build things that actually work at scale.

Most teams are still in the prompting mindset because that's what the tutorials teach and the demos showcase. The teams building production LLM features that actually ship and stay shipped have figured out looping. That's the pattern that separates the 70% success rate experiments from the 98% success rate products.