Add agent experience testing framework, expand .claude

.claude/architecture.md

@@ -0,0 +1,150 @@
+# Documentation Architecture
+
+## Directory Structure
+
+```
+mintlifydocs/
+├── docs.json              # Site configuration, navigation, theme, redirects
+├── CLAUDE.md              # AI assistant instructions (this file's parent)
+│
+├── get-started/           # Onboarding and account setup
+├── flash/                 # Flash SDK (Python functions on cloud GPUs)
+├── serverless/            # Serverless workers, endpoints, vLLM
+├── pods/                  # GPU/CPU instances
+├── storage/               # Network volumes, S3 API
+├── hub/                   # Runpod Hub and publishing
+├── public-endpoints/      # Public API endpoints
+├── instant-clusters/      # Multi-node GPU clusters
+├── sdks/                  # Python, JavaScript, Go, GraphQL SDKs
+├── runpodctl/             # CLI documentation
+├── api-reference/         # REST API reference
+├── integrations/          # Third-party integrations
+├── tutorials/             # Step-by-step guides
+├── references/            # Reference tables (GPU types, billing, etc.)
+├── community-solutions/   # Community-contributed content
+│
+├── snippets/              # Reusable content fragments
+│   ├── tooltips.jsx       # Tooltip component definitions
+│   └── *.mdx              # Reusable MDX snippets (e.g., pricing tables)
+│
+├── images/                # Static image assets
+├── logo/                  # Logo files
+├── styles/                # Custom CSS
+│
+├── scripts/               # Utility scripts
+│   └── validate-tooltips.js
+│
+└── helpers/               # Python scripts for generating content
+    ├── gpu_types.py       # Generates GPU reference tables
+    └── sls_cpu_types.py   # Generates CPU reference tables
+```
+
+## Configuration (docs.json)
+
+The `docs.json` file controls:
+
+- **Theme and styling**: Colors, fonts, code block themes
+- **Navigation**: Tab/group/page hierarchy
+- **SEO**: Meta tags, Open Graph images
+- **Redirects**: URL redirects for moved/renamed pages
+
+### Navigation Structure
+
+Pages are organized in a hierarchy:
+```
+tabs → groups → pages
+```
+
+Example:
+```json
+{
+  "tab": "Docs",
+  "groups": [
+    {
+      "group": "Serverless",
+      "pages": [
+        "serverless/overview",
+        "serverless/quickstart",
+        {
+          "group": "vLLM",
+          "pages": ["serverless/vllm/overview", "serverless/vllm/get-started"]
+        }
+      ]
+    }
+  ]
+}
+```
+
+Pages are referenced by file path without the `.mdx` extension.
+
+## MDX Files
+
+Each documentation page is an MDX file with:
+
+1. **Frontmatter** (required):
+   ```yaml
+   ---
+   title: "Page title"
+   sidebarTitle: "Shorter sidebar title"
+   description: "SEO description for the page."
+   ---
+   ```
+
+2. **Imports** (optional): React components, tooltips, snippets
+3. **Content**: Markdown with JSX components
+
+## Snippets
+
+Reusable content in `snippets/`:
+
+- **MDX snippets**: Embed with `import Table from '/snippets/pricing-table.mdx'`
+- **JSX components**: Import specific exports like tooltips
+
+### Tooltips
+
+Tooltips provide hover definitions for technical terms. Defined in `snippets/tooltips.jsx`.
+
+**Structure:**
+```jsx
+export const PodTooltip = () => {
+  return (
+    <Tooltip
+      headline="Pod"
+      tip="A dedicated GPU or CPU instance for containerized AI/ML workloads."
+      cta="Learn more about Pods"
+      href="/pods/overview"
+    >Pod</Tooltip>
+  );
+};
+```
+
+**Usage in MDX:**
+```mdx
+import { PodTooltip, TemplateTooltip } from "/snippets/tooltips.jsx";
+
+Deploy your first GPU <PodTooltip /> using a <TemplateTooltip />.
+```
+
+**Guidelines:**
+- Use for Runpod-specific terms users might not know.
+- Most tooltips have singular/plural variants (`PodTooltip`, `PodsTooltip`).
+- Group by category: Pods, Serverless, Storage, Products, Concepts, AI/ML, Flash.
+- Run `scripts/validate-tooltips.js` to check imports.
+
+## Adding New Pages
+
+1. Create `.mdx` file in the appropriate directory.
+2. Add frontmatter with `title`, `sidebarTitle`, and `description`.
+3. Add the page path to `docs.json` navigation.
+4. Import tooltips for technical terms.
+
+## Redirects
+
+When moving or renaming pages, add to `docs.json`:
+```json
+{
+  "redirects": [
+    { "source": "/old-path", "destination": "/new-path" }
+  ]
+}
+```

.claude/commands/test.md

@@ -0,0 +1,117 @@
+# /test - Run a documentation test
+
+Run a test from the testing framework to validate documentation quality.
+
+## Usage
+
+```
+/test <test-id>              # Run single test
+/test <test-id> local        # Run with local docs
+/test <category>             # Run all tests in category
+/test <category> local       # Run category with local docs
+/test smoke                  # Run smoke tests only
+```
+
+## Arguments
+
+- `<test-id>`: Single test ID (e.g., `pods-quickstart-terminal`, `flash-quickstart`)
+- `<category>`: Category name to run all tests in that section
+- `local`: (Optional) Use local MDX files instead of published docs
+- `smoke`: Run all smoke tests
+
+## Categories
+
+| Category | Tests | Description |
+|----------|-------|-------------|
+| `smoke` | 12 | Fast tests, no GPU deploys |
+| `flash` | 13 | Flash SDK tests |
+| `serverless` | 20 | Serverless endpoint tests |
+| `vllm` | 6 | vLLM deployment tests |
+| `pods` | 11 | Pod management tests |
+| `storage` | 11 | Network volume tests |
+| `templates` | 6 | Template tests |
+| `clusters` | 4 | Instant Cluster tests |
+| `sdk` | 8 | SDK and API tests |
+| `cli` | 6 | runpodctl tests |
+| `integrations` | 4 | Third-party integrations |
+| `public` | 3 | Public endpoint tests |
+| `tutorials` | 9 | End-to-end tutorials |
+
+## Single Test Execution
+
+When running a single test:
+
+1. **Read the test definition** from `tests/TESTS.md`
+2. **Do NOT use prior knowledge** - only use Runpod docs
+3. **Doc source mode**:
+   - Default: Use `mcp__runpod-docs__search_runpod_documentation`
+   - If `local`: Search and read `.mdx` files in this repository
+4. **Resource naming**: All resources MUST use `doc_test_` prefix
+5. **Attempt the goal** using available tools
+6. **Handle GPU availability** - see GPU Fallback section
+7. **Verify the Expected Outcome** from the test definition
+8. **Clean up** all `doc_test_*` resources
+9. **Generate report**: `python tests/scripts/report.py <test-id> <PASS|FAIL|PARTIAL> [--local]`
+10. **Complete the report** with actual results
+
+## Batch Execution
+
+When running a category (e.g., `/test serverless`):
+
+1. **Parse category** - Identify all test IDs in that section of TESTS.md
+2. **Show test list** - Display tests to be run and ask for confirmation
+3. **Run sequentially** - Execute each test following single test rules
+4. **Track results** - Record PASS/FAIL/PARTIAL for each
+5. **Clean up between tests** - Delete all `doc_test_*` resources before next test
+6. **Generate summary** - Create batch summary report at end
+
+### Batch Summary Format
+
+After running all tests in a batch, output:
+
+```markdown
+## Batch Summary: <category>
+
+| Test ID | Status | Notes |
+|---------|--------|-------|
+| test-1 | PASS | |
+| test-2 | FAIL | Missing docs for X |
+| test-3 | PARTIAL | Used fallback GPU |
+
+**Results:** X passed, Y failed, Z partial out of N tests
+**Doc Source:** Published / Local
+**Date:** YYYY-MM-DD HH:MM
+```
+
+Save the summary to:
+- `tests/reports/batch-<category>-<timestamp>.md`
+- `~/Dev/doc-tests/batch-<category>-<timestamp>.md`
+
+### Batch Options
+
+- **Stop on failure**: By default, continue through all tests. User can say "stop on first failure"
+- **Skip cleanup**: User can say "skip cleanup between tests" for speed (not recommended)
+
+## GPU Fallback Guidance
+
+| Queue Wait | Action |
+|------------|--------|
+| < 2 min | Keep waiting |
+| 2-5 min | Try fallback GPU |
+| > 5 min | Use fallback or mark blocked |
+
+**Fallback order**: L4 → A4000 → RTX 3090 (Community Cloud)
+
+**Status marking**:
+- PASS: Completed with documented GPU
+- PARTIAL: Completed with fallback GPU (doc improvement needed)
+- FAIL: Failed even with fallbacks
+
+## Examples
+
+```
+/test pods-quickstart-terminal       # Single test
+/test flash local                    # All Flash tests with local docs
+/test serverless                     # All Serverless tests
+/test smoke                          # Quick validation
+```

.claude/development.md

@@ -0,0 +1,114 @@
+# Development Guide
+
+## Local Development
+
+### Setup
+
+Install Mintlify globally:
+```bash
+npm i -g mintlify
+```
+
+Start the local development server:
+```bash
+mintlify dev
+```
+
+Most changes are reflected live without restarting the server.
+
+### Linting
+
+Install [Vale](https://vale.sh/docs/vale-cli/installation/), then lint files:
+```bash
+vale path/to/docs/
+vale path/to/*.mdx
+```
+
+Vale is configured with Google and Readability style guides via `.vale.ini`.
+
+### Python Code Formatting
+
+Format Python code examples in documentation:
+```bash
+pip install blacken-docs
+git ls-files -z -- '*.mdx' | xargs -0 blacken-docs
+```
+
+## Helper Scripts
+
+### Update GPU/CPU Reference Tables
+
+These scripts fetch current types from Runpod's GraphQL API:
+```bash
+cd helpers
+python gpu_types.py      # Updates GPU reference tables
+python sls_cpu_types.py  # Updates CPU reference tables
+```
+
+Requirements: `requests`, `tabulate`, `pandas` (see `helpers/requirements.txt`).
+
+### Validate Tooltips
+
+Check that all imported tooltips exist:
+```bash
+node scripts/validate-tooltips.js
+```
+
+This runs automatically in CI via `.github/workflows/validate-tooltips.yml`.
+
+## Publishing Workflow
+
+1. Create a pull request with changes.
+2. Request review from [@muhsinking](https://github.com/muhsinking).
+3. Changes deploy automatically to production after merge to `main` branch.
+
+## Common Tasks
+
+### Add a New Page
+
+1. Create `.mdx` file in the appropriate directory.
+2. Add frontmatter:
+   ```yaml
+   ---
+   title: "Full page title"
+   sidebarTitle: "Shorter title"
+   description: "SEO description."
+   ---
+   ```
+3. Add the page path to `docs.json` navigation.
+4. Import tooltips for Runpod-specific terms.
+
+### Add a New Tooltip
+
+1. Open `snippets/tooltips.jsx`.
+2. Add a new export in the appropriate category:
+   ```jsx
+   export const NewTermTooltip = () => {
+     return (
+       <Tooltip
+         headline="New Term"
+         tip="Definition of the new term."
+         cta="Learn more"
+         href="/path/to/docs"
+       >new term</Tooltip>
+     );
+   };
+   ```
+3. Create singular and plural variants if needed.
+
+### Move or Rename a Page
+
+1. Move/rename the `.mdx` file.
+2. Update `docs.json` navigation.
+3. Add a redirect in `docs.json`:
+   ```json
+   {
+     "redirects": [
+       { "source": "/old-path", "destination": "/new-path" }
+     ]
+   }
+   ```
+
+### Update a Pricing Table
+
+Edit `snippets/serverless-gpu-pricing-table.mdx` or run the helper scripts to regenerate from the API.