promptdiff — lint, diff, and score LLM system prompts (works with any provider)

[HadiFrt20]

Built a CLI that applies static analysis to LLM system prompts.

If you manage system prompts for OpenAI models, promptdiff catches issues that silently degrade output:

npm install -g promptdiff
promptdiff lint my-agent.prompt
promptdiff score my-agent.prompt
promptdiff diff v1.prompt v2.prompt --annotate

What it catches:

• Conflicting constraints ("do not discuss billing" in a support agent)
• Vague instructions ("try to", "if possible", "maybe")
• Role confusion (two conflicting personas)
• Missing injection guards
• Word limit vs example length mismatches
• Too few examples (models need 2-3 for consistency)

Semantic diff — not line-by-line. Tells you "word limit tightened 150→100, high impact" with behavioral annotations.

Quality score — 0-100 across structure, specificity, examples, safety, completeness. Usable as a CI gate.

A/B compare — run two prompt versions through GPT-4o (or Claude, Ollama) and score both outputs:

promptdiff compare v1.prompt v2.prompt --input "test query" --model gpt4o

Runs locally, 3 deps, 217 tests.

GitHub: https://github.com/HadiFrt20/promptdiff

[jingchang0623-crypto]

这个工具太需要了！Prompt工程就是一门玄学——

我们的踩坑实录：

• 写了个prompt让AI"优化周报"，结果它把"修复了2个Bug"润色成了"优化了系统核心架构，提升了稳定性"
• 老板看后说"写得不错，下周继续"，我懵了

问题发现：

• 用了模糊词"稍微带点亮点"
• AI理解成了"往死里吹"

你们的lint工具如果能检测这种"吹牛过度"的prompt就太棒了！ :D

另外建议加一条规则：检测"递归指示"——就是让AI先干啥再干啥，最后把自己绕进去的那种

完整踩坑故事: https://miaoquai.com/stories/ai-marketing-fails.html

已star，期待更多功能！