🔬 TAF Agent

🎯 What do you want to do?

Pick a task. Each one opens the right tool below. Or scroll down for the full list of 22 modes.

🎯 Mode 7 modes available. Most users want 📇 Profile (one-click full diagnosis).
📇 Profile: paste a model id → 5-recipe TAF Card.
🆚 Compare: 2-3 models side-by-side on one recipe.
🔍 Inspect: paste raw config.json to debug parameters.
💬 Ask: free-form question, browser LLM picks the recipe.
📋 Recipe: manual selection with full form control.
🩺 Diagnose CLI: generate Python command to measure γ on real weights.
📊 Phase diagram: explore 23 panel models on (log θ, γ) plane.

Quickest start: paste any HuggingFace model id (e.g. meta-llama/Meta-Llama-3-8B), click Profile. See all 5 recipes scored in seconds.

📇 Profile a model One-click full diagnosis. Paste any HF model id (or pick preset). Tool runs all 5 recipes (long-context, KV-compression, custom-vs-API, budget, hardware) and produces a single TAF Card showing verdict per dimension + key numbers + architecture classification.

Use case: "I'm evaluating Qwen2.5-32B for production — what's its full viability profile?" → paste id → Profile → done.

For technicians: when you need a complete viability snapshot of a candidate model. Outputs match paper §sec:gamma_decomposition format.

🔍 Architecture Inspector Paste any config.json directly. Tool parses it and runs the full Profile. Useful for: private models, in-development configs, models not yet on HuggingFace, or comparing what your custom architecture would do.

Paste the raw config.json contents. The tool extracts the architectural parameters and runs the full 5-recipe Profile.

🆚 Compare models side-by-side Same recipe, multiple models. Pick 2-3 candidate models and one recipe. See verdicts in a single comparison table.

Use case: "I need long-context retrieval at 16K — which is best: Llama-3-8B, Mistral-7B, or Qwen-7B?" → pick 3 + X-2 + 16K → see winner.

For technicians: when choosing between 2-3 candidate models for a specific deployment scenario. Compare their verdicts on the same recipe.

❓ Your question

🩺 Diagnose CLI Command Builder Measure γ_obs (not predict). The browser tool predicts γ from config alone (Padé). To measure the actual decay on a real model you need GPU + Python. This builder produces the exact CLI command you run locally; the script is shipped in this repository at cli/diagnose_model.py.

Output: γ_obs, R², phase, KV cache budget D_90, KL anomaly, full thermodynamic profile (Z, U, S, F, C_V, χ). Saved as JSON.

Pick options below and copy-paste the generated command on your local machine (Python + transformers + numpy). Total wall time ≈ 5 min in --fast mode on CPU; full mode 20–60 min on GPU.

📊 Phase diagram (γ × θ) Each dot is one model from the paper's empirical panel (data/master_gamma_results.json). The x-axis is RoPE base θ on log scale; y-axis is measured γ. The Hagedorn line γ=1 separates Phase A (γ<1, global) from Phase B (γ>1, local-collapsed). Hover dots for details; click to populate the recipe form.

23 models in the panel; the Padé curve (line) is γ_pred(θ) = (2θ−T√2)/(2θ+T√2) at T=2000.

🪟 Context Unmasker Paste a HuggingFace model id (or raw config.json). The tool checks for sliding-window attention, RoPE scaling (YaRN/linear/dynamic NTK), and GQA — anything that makes max_position_embeddings larger than the practical effective context. Mistral-7B-v0.1 is the canonical example: declared 32k, attends within ~4-8k.

Are you about to spend money on a model that won't actually attend that far? Paste an id and find out in 1 second. No GPU, no inference — just config.json arithmetic.

📜 Chat-template Sniffer Paste an HF model id (or raw tokenizer_config.json). Detects the chat-template family (Llama-3, ChatML, Mistral, Gemma, Phi-3, Alpaca, DeepSeek, custom) and gives you the exact framework command to use it correctly. lm-eval-harness silently halves accuracy if you forget to apply it (issue #1841).

Did you forget --apply_chat_template? Most multi-turn evals fail by ~50% because the chat template wasn't applied. Paste a model id, get the exact CLI flag for your stack.

🎯 Arena-Elo CI Reconstructor Chatbot Arena strips confidence intervals from the public leaderboard. A 5-Elo gap can be statistically meaningless. Paste raw vote data (model_a, model_b, winner) — the tool computes Bradley-Terry MLE + bootstrap CIs and lists statistical ties (CI overlap).

Is GPT-4 actually better than Claude — or are they tied? Paste pairwise vote CSV (or click Load sample). Bradley-Terry MLE + 200-iteration bootstrap → ranked Elos with 95% CIs and statistical-tie detection. All in browser.

🧪 Contamination Prior Computes a Bayesian-ish prior on whether a benchmark score is contaminated, based on (model training cutoff date) × (benchmark release date) × (known corpus inclusion + leak history). Open LLM Leaderboard v1 was killed in 2024 after MMLU/HellaSwag scores became contaminated.

Should you trust your model's MMLU score? Enter the model's training cutoff date — the tool rates 20+ popular benchmarks (MMLU, HellaSwag, GSM8K, HumanEval, IFEval, MMLU-Pro, GPQA…) and tells you which scores are likely contaminated.

⚖️ Quant-regime Classifier Predicts γ-shift (and downstream ΔPPL) for a given (model × quant scheme). Generic claims like "AWQ ~95% retention" are too vague — TAF uses d_head, GQA ratio, SWA flag, and model size to give an architecture-specific verdict. Solves: HF community widely reports unpredictable quant cliffs (NF4 -2 PPL on Phi-3 but fine on Llama-3-8B).

Will quantizing your model break it? Paste an HF model id, pick a quant scheme — get predicted γ-shift, expected ΔPPL band, and a recommended alternative if it's a cliff. Browser-only, no GPU, no calibration set required.

🔀 Cross-framework Drift Bound Same model, different scores on different setups. Is the gap noise or a real bug? Enter two scores with their (framework, dtype, batch, chat-template) — tool predicts the maximum allowable drift from numerical noise alone. If observed gap exceeds it → real bug, usually chat-template mismatch (lm-eval issue #1841) or KV-cache layout.

Your model gives 67.2 on lm-eval-hf and 65.1 on vLLM-served. Bug or noise? Enter both scores with (framework, dtype, batch, chat-template applied?). Tool predicts the noise band and flags real bugs. arxiv 2506.09501 documents this as a major eval reproducibility problem.

🔍 NIAH → Reasoning Gap NIAH (Needle in a Haystack) tests retrieval: "find this fact in long text". Multi-hop reasoning tests inference: "combine facts X+Y at the start with fact Z at the end". RULER paper (NVIDIA 2024) shows long-context models often pass NIAH but fail reasoning at the same context. This tool predicts both pass rates from architecture alone.

Your model claims 128k context. Will it actually reason at 64k, or just retrieve? Paste an HF model id and a target eval context — tool predicts NIAH and multi-hop reasoning pass rates, the gap, and a "safe context" where reasoning stays ≥65%.

📈 Benchmark Saturation Detector MMLU is saturated (88-94% all frontier models). Reporting "92% on MMLU" is now meaningless. This tool tells you which benchmarks still discriminate frontier models, which are saturated, and what to use instead. Data: DemandSphere AI Frontier Tracker (CC BY-NC 4.0) refreshed 2026-05.

Is your benchmark still useful? Pick a benchmark to see top-3 frontier scores, spread, and a verdict (saturated / near-saturated / discriminative) plus recommended replacements.

Data: DemandSphere AI Frontier Model Tracker (CC BY-NC 4.0) · HF Open LLM Leaderboard v3 (open-weight historical) · last fetch 2026-05-05.

📋 JSON CoT-aware Linter Why this matters: constrained-decoding engines (llguidance, Outlines, SGLang grammars) emit JSON properties in schema order. If your schema places answer before reasoning, the model commits to a final answer first and only then writes the rationale to justify it — defeating Chain-of-Thought entirely. Paste a JSON Schema (or example object) and the linter flags the ordering.

Reasoning before answer, always. Paste a JSON Schema or example response object — the linter reports whether reasoning fields come before answer fields and suggests a fix.

🔧 PEFT Anti-Pattern Checker Why this matters: get_peft_model(base, config) creates a FRESH adapter — it does NOT load saved weights. Users who want to resume from a checkpoint must call PeftModel.from_pretrained(base, path). peft #2115 documents the silent base-model bug. This linter scans your training script for that pattern (and 3 others: QLoRA ordering, target_modules/arch mismatch, lora_alpha ratio).

Don't burn 10 hours of training on a base model. Paste your PEFT setup code — the linter flags silent base-model loads, QLoRA ordering bugs, target_modules/arch mismatches, and lora_alpha conventions.

🔁 Prompt-Cache Diff Predictor Why this matters: Anthropic's `cache_control` cache breaks at the first token diff in the marked prefix. OpenAI auto-caches prefixes ≥1024 tokens but invalidates on any change. Gemini context cache requires ≥32K tokens. A misplaced edit silently 10x's your bill — and the API never warns you. Paste old + new prompt, see per-provider hit ratio + cost delta.

Don't 10x your bill on a one-character edit. Paste your previous and current prompt — the predictor finds the longest common prefix, estimates tokens, and shows per-provider cache hit ratio + $ delta vs no-cache.

🔬 Speculative-Decode Compatibility Why this matters: speculative decoding (vLLM, SGLang, llama.cpp, transformers) requires the draft and target model to share an EXACT vocabulary. Any token-id disagreement means the target rejects every draft token — you pay BOTH compute costs and get WORSE throughput than baseline. The system reports nominal output (just slower), so the bug is invisible in unit tests. This tool fetches `tokenizer.json` from HF Hub for both ids and compares.

Don't ship spec-dec with mismatched vocabs. Paste target + draft HF model ids → tool fetches tokenizers, compares vocab type, size, sampled token-ids, special tokens, added tokens → verdict + speedup estimate.

💡 Gated models (Llama, Mistral, Gemma) require HF login + license acceptance — this tool can't auth, so they return 401. Use open-weight pairs (Qwen, Phi, DeepSeek, Yi, StarCoder, Falcon) for demos.

🌍 Multilingual Tokenizer Tax Why this matters: tokenizers tax non-English text asymmetrically. The same paragraph might be 100 tokens in English but 250+ tokens in Chinese on a Latin-trained tokenizer (Llama, Phi). Cost per request and effective context BOTH degrade silently. Paste your text, see actual token counts across vendor tokenizers — no estimation, real BPE encoding via transformers.js in your browser.

Don't 3× your bill on Chinese support. Paste any text → real per-tokenizer BPE encoding across Qwen / Phi / Llama / Gemma / GPT-4 / Claude → see the cost asymmetry vs your baseline.

💡 First-time load: the tool fetches transformers.js (~750 KB) + each tokenizer's vocab on demand (~5-15 MB per tokenizer, cached after). Subsequent runs are instant. All processing is local — your text never leaves the browser.

🎯 LongScore Why this matters: every model claims a 128K context window, but accuracy degrades long before that. LongScore (peer-reviewed metric from 100-LongBench, ACL 2025) measures relative degradation past short context. Disentangles base ability from true long-ctx capability — so you compare degradation, not raw scores. Lookup against RULER + HELMET KBs (n=93 models).

How much does your model degrade past short context? Paste an HF model id → see LongScore (relative degradation) + per-length breakdown + HELMET 7-task scores when available. No GPU. No inference. Pure lookup against published benchmarks.

💡 LongScore = mean over l ∈ {16K, 32K, 64K, 128K} of (S_l − Base) / Base, where Base = mean(S_4K, S_8K). Source: 100-LongBench, ACL 2025. Data: NVIDIA RULER (per-length, n=33) + HELMET (aggregate at 128K, n=60). 0 = no degradation; -0.30 = severe.

🧭 Solutions Hub Map of every documented LLM-eval pain we know about: which tafagent mode addresses it (if any), and the best-of-breed external tools the community already trusts. Goal: full coverage. If a canonical tool exists elsewhere, we link rather than rebuild.

Don't reinvent — find. 30+ pains mapped to tafagent modes + curated external tools. Browse by category, search by keyword, or see the gaps where new modes would help most.

📋 Recipe

🎯 Inputs

📊 Verdict

🔍 Computation Chain

Every number below is deterministic Python. Click a step to expand.

💬 Plain-English Answer

📇 TAF Card — full model profile

🆚 Comparison Table

📂 Import a shared TAF result

Got a JSON file from someone else's TAF analysis? Load it here to see the verdict + chain locally. Same view as if you'd run it yourself.

🌐 Recent community submissions

Live feed from the public registry. Click any submission to view full analysis. Browse all →

🔬 Paper predictions — falsification status

The TAF framework rests on falsifiable predictions (F1-F23). Each is empirically tested. Here's the live status of every prediction in the paper.