What's broken

Container has CUDA 12.x drivers but the torch build expects another toolkit - imports work, .cuda() explodes at runtime.

• pytorch
• cuda

What breaks

GPU training/inference jobs die on first tensor move to device.

How to detect

python -c "import torch; print(torch.version.cuda, torch.cuda.is_available())" and nvidia-smi in the same image.

Mitigation

Pin torch index URL to the CUDA build you actually ship; bake and test the image on a GPU runner before promote.

Average prompt length jumps after a retrieval tweak; KV cache blows past GPU memory during traffic spikes.

• vllm
• cuda
• rag

What breaks

Intermittent 503s / restarts; p95 latency cliffs.

How to detect

Track prompt token percentiles + GPU mem; load test with p95 prompts, not averages.

Mitigation

Cap retrieved tokens, use chunk budgets, enable prefix caching where applicable, scale TP/replicas before launch weeks.

A 'harmless' pip upgrade changes default generation kwargs, tokenizer behavior, or remote code expectations.

• transformers
• pytorch

What breaks

API starts 500ing or returning empty/garbled completions after a dependency refresh.

How to detect

Lockfile diff + smoke test on generate/chat template in CI.

Mitigation

Pin transformers + tokenizers + torch; upgrade in a branch with golden prompt fixtures.

Someone upgrades to a 'better' embedder but the vector table still holds the old space.

• rag
• pgvector
• embeddings

What breaks

Retrieval looks plausible but relevance collapses; hallucinations rise.

How to detect

Store embedding_model_id + dim on every index; refuse query on mismatch.

Mitigation

Blue/green indexes; dual-write; cut over only after recall@k check.

Model card template changes; your hand-rolled prompt format silently diverges and quality tanks without exceptions.

• transformers
• llama
• vllm

What breaks

Accuracy/refusal rates shift; logs still look 'healthy'.

How to detect

Snapshot apply_chat_template output in CI; alert on diff.

Mitigation

Always render prompts via the tokenizer chat template; never hardcode role markers for instruct models.

Retriever returns overlapping chunks; prompt token count spikes; model truncates the actual question.

• rag
• langchain

What breaks

Answers ignore the user question; cost and latency jump together.

How to detect

Log prompt token counts; alert when p95 crosses budget.

Mitigation

Dedupe overlaps, hard cap context tokens, put the question at the end (or use a packed template).

Long silent or non-speech audio produces confident nonsense transcripts.

• whisper
• ffmpeg

What breaks

Meeting notes and subtitle pipelines insert fake sentences.

How to detect

VAD / energy gate before ASR; flag segments with absurd repetition.

Mitigation

Skip low-energy segments; use no_speech_threshold; post-filter repeated n-grams.

Autoscaler adds a node with an older NVIDIA driver; only some pods schedule into a black hole.

• kubernetes
• cuda
• nvidia

What breaks

Random GPU pod failures depending on placement.

How to detect

Node feature discovery / driver version labels; fail scheduling if mismatched.

Mitigation

Homogeneous GPU node groups; pin AMI; taints for unsupported driver versions.

Autoscaling replicas all download weights at once and get throttled.

• transformers
• huggingface_hub

What breaks

Rolling deploy hangs; pods flap CrashLoop during scale-out.

How to detect

Pre-bake weights into the image or a PVC; measure cold-start without Hub access.

Mitigation

Vendor weights into artifact storage; set HF_HUB_OFFLINE=1 in prod.

Phone uploads appear sideways to the model because EXIF rotation wasn’t applied.

• opencv
• pillow

What breaks

Classifier/OCR quality drops only on mobile-sourced images.

How to detect

Fixture set of EXIF-rotated JPEGs in eval; compare before/after fix.

Mitigation

Normalize with Pillow ImageOps.exif_transpose (or equivalent) before OpenCV.

Imports move between langchain / langchain_community / langchain_core faster than docs bookmarks.

• langchain
• python

What breaks

Deploys fail on ImportError after 'routine' dependency bump.

How to detect

Pinned versions + import tests for your graph entrypoints.

Mitigation

Treat LangChain as a thin orchestration layer; isolate behind your own interfaces.

Both packages installed; ImportError or GUI libs missing in slim Docker images.

• opencv
• python

What breaks

CV workers crash on import in production images that worked on laptops.

How to detect

pip freeze | grep opencv; enforce one package in dependency resolver.

Mitigation

Use headless in servers; never mix; add an import smoke test in CI image build.

Compile/warmup cost hits the first production request after deploy.

• pytorch

What breaks

SLO burn on deploy; users see multi-second spikes once per replica.

How to detect

Synthetic warmup in readinessProbe before traffic.

Mitigation

Compile during image bake or startup hook; don’t rely on first user request.

IDs and leading-zero codes become floats; joins miss; models train on NaNs after coerce.

• pandas
• sklearn

What breaks

Offline metrics look ok on sampled notebooks; prod features drift.

How to detect

Schema checks (Pandera/Great Expectations) on ingest.

Mitigation

Explicit dtype maps; never rely on inference for key columns.

GPU build missing a lib; ORT falls back to CPU and latency 10× without raising.

• onnxruntime
• cuda

What breaks

Throughput crater; GPU utilization near zero; bills look fine.

How to detect

Log session.get_providers() on boot; alert if CUDAExecutionProvider missing.

Mitigation

Fail fast if expected EP absent; smoke a timed inference in CI on GPU.

Old openai.ChatCompletion paths or proxy wrappers still linger in a utility module nobody owns.

• openai
• python

What breaks

Only some code paths fail after an image rebuild pulls the new SDK.

How to detect

ripgrep for ChatCompletion / openai.error; unit test the shared client wrapper.

Mitigation

One shared client module; ban direct SDK use in feature code via lint.