Phase 7 User Testing Guide¶

Purpose: Manual testing instructions to experience Phase 7 implementations as a real user Date: November 13, 2025 Status: Week 2 Complete - Ready for User Testing

What Can Be Tested Now?¶

⚠️ IMPORTANT: Most Phase 7 features are backend/API only and don't have UI yet. To test them, you'll need to: 1. Use API testing tools (Postman, curl, or browser DevTools) 2. Check database directly 3. Monitor logs and telemetry

What's Built (Week 1 + Week 2): - ✅ API endpoint: /api/teams/[teamId]/create-fix-pr - ✅ Quota management (database + logic) - ✅ Telemetry events (PostHog) - ✅ Structured logging (Vercel logs) - ✅ Error handling (GitHub API)

What's NOT Built Yet (Week 3): - ❌ Team dashboard UI - ❌ Quota display in UI - ❌ Usage analytics charts - ❌ Fix PR history table

Prerequisites¶

1. Local Development Setup¶

# Start development server
npm run dev

# Open browser
open http://localhost:3000

2. Database Setup¶

You need: - ✅ Neon database running (from Phase 5) - ✅ Teams table with quota fields - ✅ Test team in database

Verify Database Migration:

-- Check if quota fields exist
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'teams'
  AND column_name IN ('auto_fix_prs_used', 'auto_fix_quota_reset_at');

-- Expected: Both fields should exist

3. GitHub App Setup¶

You need: - ✅ GitHub App installed on a test repository - ✅ Installation ID in github_installations table - ✅ Test PR with vulnerabilities

4. PostHog Setup (Optional)¶

# Check PostHog env vars
echo $NEXT_PUBLIC_POSTHOG_KEY
echo $NEXT_PUBLIC_POSTHOG_HOST

# If empty, telemetry will log to console only

User Testing Scenarios¶

Scenario 1: Test Quota Management (NO UI)¶

What You're Testing: Monthly quota limits for auto-fix PRs

Testing Method: Database + API calls

Step 1: Check Current Quota¶

-- View team's current quota
SELECT
  id,
  name,
  plan,
  auto_fix_prs_used,
  auto_fix_quota_reset_at
FROM teams
WHERE id = 'your-team-id';

-- Expected for new team:
-- auto_fix_prs_used = 0
-- auto_fix_quota_reset_at = NULL

Step 2: Trigger Quota Check via API (Requires Auth)¶

Option A: Use Browser DevTools (if you're logged in)

Open browser DevTools (F12)
Go to Console tab
Run this code:

// Get your session
const session = await fetch('/api/auth/session').then(r => r.json());
console.log('Session:', session);

// Call create-fix-pr endpoint
const response = await fetch('/api/teams/YOUR_TEAM_ID/create-fix-pr', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    prUrl: 'https://github.com/owner/repo/pull/123'
  })
});

const result = await response.json();
console.log('Result:', result);

Option B: Use curl (harder - requires session cookie)

# This won't work without authentication
curl -X POST http://localhost:3000/api/teams/TEAM_ID/create-fix-pr \
  -H "Content-Type: application/json" \
  -d '{"prUrl": "https://github.com/owner/repo/pull/123"}'

# Expected: 401 Unauthorized (authentication required)

Step 3: Check Quota After Request¶

-- Check if usage incremented
SELECT
  auto_fix_prs_used,
  auto_fix_quota_reset_at
FROM teams
WHERE id = 'your-team-id';

-- Expected:
-- auto_fix_prs_used = 1 (incremented)
-- auto_fix_quota_reset_at = current timestamp

Step 4: Test Quota Limit¶

For Team Plan (limit: 10/month):

-- Manually set usage to 10
UPDATE teams
SET auto_fix_prs_used = 10
WHERE id = 'your-team-id';

Then call API again → should get 429 Quota Exceeded

Step 5: Test Monthly Reset¶

-- Manually set reset date to last month
UPDATE teams
SET auto_fix_quota_reset_at = CURRENT_DATE - INTERVAL '1 month'
WHERE id = 'your-team-id';

Then call API again → quota should auto-reset to 0

Scenario 2: Test Telemetry Events (NO UI)¶

What You're Testing: PostHog event capture

Testing Method: Console logs + PostHog dashboard

Step 1: Enable PostHog (if not already)¶

# Add to .env.local
NEXT_PUBLIC_POSTHOG_KEY=your_key
NEXT_PUBLIC_POSTHOG_HOST=https://eu.i.posthog.com

Step 2: Trigger API Request¶

Use Scenario 1 method to call the API

Step 3: Check Console Logs¶

Look for telemetry logs in terminal:

[Telemetry] fix_pr_requested { teamId: 'team-1', fixableCount: 5 }

or

[Telemetry] fix_pr_quota_exceeded { teamId: 'team-1', plan: 'team', used: 10, limit: 10 }

Step 4: Check PostHog Dashboard¶

Go to PostHog dashboard: https://eu.i.posthog.com
Navigate to Events
Look for:
fix_pr_requested
fix_pr_quota_exceeded
fix_pr_created (if job completes)

Properties to verify: - teamId - prUrl - fixableCount - plan

Scenario 3: Test Structured Logging (NO UI)¶

What You're Testing: JSON-formatted logs in Vercel

Testing Method: Vercel logs or local console

Step 1: Trigger API Errors¶

Call API with invalid data:

// In browser console
const response = await fetch('/api/teams/TEAM_ID/create-fix-pr', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    prUrl: 'invalid-url'  // Invalid URL format
  })
});

Step 2: Check Local Logs¶

In your terminal running npm run dev, look for JSON logs:

{
  "timestamp": "2025-11-13T13:15:00.000Z",
  "level": "ERROR",
  "service": "api",
  "message": "Fix PR creation failed",
  "teamId": "team-1",
  "step": "api_endpoint",
  "error": {
    "name": "Error",
    "message": "Invalid GitHub PR URL",
    "stack": "..."
  }
}

Step 3: Check Vercel Logs (Production Only)¶

Go to Vercel dashboard
Select your project
Go to Logs tab
Filter by "ERROR" level
Look for structured JSON logs

Scenario 4: Test Error Handling (NO UI)¶

What You're Testing: GitHub API error classification and retry logic

Testing Method: Simulate different error types

Test 4A: Rate Limit Error¶

How to Trigger: Make 60+ requests to GitHub API in 1 hour

Expected Behavior: - Error classified as rate_limit - Retry with exponential backoff (1s → 2s → 4s → 8s → 16s) - fix_pr_failed event with errorType: 'rate_limit'

Test 4B: Permission Error¶

How to Trigger: Use GitHub App without repo write permission

Expected Behavior: - Error classified as permission_denied - No retry (permanent error) - fix_pr_failed event with errorType: 'permission_denied'

Test 4C: Invalid PR URL¶

How to Trigger:

fetch('/api/teams/TEAM_ID/create-fix-pr', {
  method: 'POST',
  body: JSON.stringify({ prUrl: 'not-a-github-url' })
});

Expected Behavior: - 400 Bad Request - Error: "Invalid GitHub PR URL format"

What You CAN'T Test Yet¶

❌ No UI for Quota Display¶

Why: Team dashboard not built yet (Week 3)

Workaround: Check database directly

SELECT
  name as "Team",
  plan as "Plan",
  auto_fix_prs_used as "Used",
  auto_fix_quota_reset_at as "Reset Date"
FROM teams;

❌ No UI for Fix PR History¶

Why: Fix PR table UI not built yet (Week 3)

Workaround: Check fix_prs table directly

SELECT
  id,
  team_id,
  original_pr_number,
  fix_pr_number,
  status,
  created_at
FROM fix_prs
ORDER BY created_at DESC
LIMIT 10;

❌ No UI for Usage Analytics¶

Why: Analytics dashboard not built yet (Week 3)

Workaround: Query PostHog API or check console logs

Recommended Testing Flow¶

Quick Test (5 minutes)¶

Goal: Verify basics work

✅ Start dev server: npm run dev
✅ Check database has quota fields: \d teams in psql
✅ Call API endpoint with invalid URL → verify 400 error
✅ Check console for JSON logs

Medium Test (15 minutes)¶

Goal: Verify quota management

✅ Quick test +
✅ Create test team with plan: 'team'
✅ Call API 10 times → verify usage increments
✅ Call API 11^th time → verify 429 quota exceeded
✅ Reset quota date to last month → verify auto-reset

Full Test (30 minutes)¶

Goal: Verify telemetry and logging

✅ Medium test +
✅ Configure PostHog in .env.local
✅ Trigger various API scenarios
✅ Check PostHog dashboard for events
✅ Check Vercel logs for structured JSON
✅ Simulate GitHub API errors

Testing Checklist¶

Database Testing¶

Quota fields exist in teams table
auto_fix_prs_used defaults to 0
auto_fix_quota_reset_at is nullable
Usage increments correctly after API call
Monthly reset works (when date < current month)

API Testing¶

Endpoint requires authentication (401 without session)
Endpoint requires team membership (403 for non-members)
Invalid PR URL returns 400
Quota exceeded returns 429
Success returns 200 with job ID

Telemetry Testing¶

fix_pr_requested event captured
fix_pr_quota_exceeded event captured
Events appear in PostHog dashboard
Console logs show telemetry events

Logging Testing¶

Logs are JSON-formatted
Logs include timestamp and level
Errors include stack traces
Logs queryable in Vercel dashboard

Error Handling Testing¶

Rate limit errors classified correctly
Permission errors don't retry
Invalid input returns 400
Server errors return 500

Known Limitations (Can't Test)¶

No UI: Everything is API-only
No GitHub Integration: Needs real GitHub App setup
No PR Analysis: PRAnalyzer needs actual GitHub PR
No Job Processing: FixPRQueue runs in background (can't see visually)
No Real Fix PR Creation: Would need full GitHub integration

When Will UI Be Available?¶

Phase 7 Week 3: Team Features & Dashboard

Planned UI components: - Team dashboard (/teams/[teamId]/dashboard) - Quota widget (shows X/10 used) - Fix PR history table - Usage analytics charts - Team settings page

Until then: Use database queries and API testing tools

Summary¶

What You Can Test NOW (Week 2 Complete)¶

✅ API endpoint - Using curl/fetch/Postman ✅ Quota management - Via database queries ✅ Telemetry - Via console logs and PostHog ✅ Logging - Via Vercel logs ✅ Error handling - Via API error responses

What You CAN'T Test Yet (Week 3 Not Started)¶

❌ Team dashboard UI - No visual interface ❌ Quota display - No UI widget ❌ Fix PR history - No table view ❌ Usage charts - No analytics UI ❌ One-click testing - Requires manual API calls

Recommended Next Steps¶

If you want to test as a user: 1. ⏳ Wait for Week 3 (Team Dashboard UI) 2. ⏳ Or use API testing tools (Postman/curl)

If you want to verify code works: 1. ✅ Run automated tests: npm test 2. ✅ Check all 110+ tests pass 3. ✅ Review completion documents

Current Status: Week 2 Complete (API ready), Week 3 Not Started (No UI)

Recommendation: Proceed with Week 3 to build UI for user testing