WebTool & GitHub App Separation Strategy

Created: 2025-11-18 Status: Active - Protection Plan Priority: CRITICAL (interference already proven in production)

---

Executive Summary

CodeSlick has two distinct product surfaces with different users, performance requirements, and usage patterns:

Product	Users	Speed	Mode	Revenue
WebTool	Individual developers	<3s analysis, <10s fix	Interactive, single-issue	Free (API key required for AI)
GitHub App	Professional teams	10-30s OK	Batch, multi-file	Paid (€99-299/month)

Problem: Phase 7 GitHub work broke WebTool functionality (proven in version 20251117.14:30).

Root Cause: Shared modules (especially FixApplier) modified for GitHub batch processing inadvertently broke WebTool's interactive single-issue fixes.

Solution: Implement clear separation with protective measures to prevent future interference.

---

Current Architecture

✅ GOOD: API Routes Are Well Separated

WebTool Stack (3 primary routes):

/api/analyze          - Static code analysis (84 checks)
/api/generate-fix     - AI single-issue fix generation
/api/fix-all          - Pattern-based batch syntax fixes

GitHub App Stack (7 routes):

/api/github/webhook            - PR event receiver
/api/teams/[id]/create-fix-pr  - Queue auto-fix PR creation
/api/teams/[id]/apply-fix      - One-click commit from PR comment
/api/github/install/callback   - GitHub App installation
/api/teams/[id]/fix-prs        - Fetch team's fix PR history
/api/github/worker             - Legacy background processor
/api/github/analyze-worker     - Queued PR analysis

Finding: Zero overlap in API routes. Clean separation. ✅

---

⚠️ RISK: Shared Core Modules

CRITICAL RISK - Security Analyzers (src/lib/analyzers/)

Files: - JavaScriptAnalyzer.ts (17 security checks) - PythonAnalyzer.ts (19 security checks) - JavaAnalyzer.ts (18 security checks) - TypeScriptAnalyzer.ts (reuses JavaScript logic)

Used By: - WebTool: /api/analyze → Direct instantiation - GitHub: /api/github/webhook → Via PRAnalyzer → Direct instantiation

Risk Level: 🔴 CRITICAL

Why Critical: 1. Output format change breaks both: If someone changes vulnerabilities to issues, BOTH systems fail 2. Performance regression cascades: 2s slowdown = GitHub timeouts + WebTool perceived as slow 3. Detection rate drops silently: Missing vulnerabilities affects both user bases 4. 96 tests protect both: Test failure could be either system

Protection Needed: - Version analyzer outputs - Dual-system integration tests - Performance benchmarks (<2s per file) - Output format validation

---

HIGH RISK - FixApplier (src/lib/github/fix-applier.ts)

Used By: - WebTool: Never directly (BUT logic influenced generate-fix API) - GitHub: /api/github/webhook → Generates AI suggestions for PR comments

Evidence of Interference:

version.json line 82-83:
"Root cause: Commit 2084052 (Phase 7 GitHub PR work) introduced 30-line
 window approach that broke WebTool"

Risk Level: 🟠 HIGH (proven to cause interference)

What Happened: 1. GitHub PR work optimized for batch processing (30-line context windows) 2. This logic influenced /api/generate-fix endpoint 3. WebTool's single-issue fixes started returning FIXME comments instead of actual code 4. Required emergency fix on 2025-11-17

Current Status: Fixed, but no protection against recurrence

Protection Needed: - Separate fix generation logic for WebTool vs GitHub - Integration test: "WebTool generate-fix works after GitHub changes" - Clear comments: // GitHub-specific optimization - do not use in WebTool

---

MEDIUM RISK - AI Provider (src/lib/utils/advanced-qwen.ts)

Used By: - WebTool: /api/generate-fix - GitHub: /api/github/webhook → Via FixApplier

Risk Level: 🟡 MEDIUM

Potential Conflicts: - Timeout settings (WebTool needs 30s, GitHub can tolerate 60s) - Retry strategies (WebTool fails fast, GitHub retries 3x) - Token limits (WebTool prioritizes speed, GitHub can use larger context)

Protection Needed: - Timeout parameter: callAPI(prompt, { timeout: 30000 }) - Retry parameter: callAPI(prompt, { maxRetries: 1 }) for WebTool - Context size validation

---

LOW RISK - Pattern Fixer (src/lib/utils/pattern-fixer.ts)

Used By: - WebTool: /api/fix-all → Direct usage - GitHub: Not currently used (but could be added)

Risk Level: 🟢 LOW

Why Low: Pure utility, no state, simple string operations, well-tested

---

LOW RISK - Utilities (DiffGenerator, LanguageDetector)

Used By: Both systems

Risk Level: 🟢 LOW

Why Low: Mature code, pure functions, no mode-specific logic

---

Separation Strategy

Phase 1: Add Protective Comments (30 minutes)

Goal: Make it obvious which code serves which system

Actions: 1. Add headers to all shared modules:

/**
 * SHARED MODULE: Used by both WebTool and GitHub App
 *
 * ⚠️ CRITICAL: Changes to this module affect BOTH systems
 *
 * WebTool uses this for: [specific use case]
 * GitHub App uses this for: [specific use case]
 *
 * Before modifying:
 * - Run all 96 analyzer tests
 * - Test both /api/analyze and GitHub webhook
 * - Verify performance benchmarks (<2s per file)
 *
 * Last modified: [date]
 * Last verified: [date]
 */

1. Add inline warnings for risky areas:

   // WEBTOOL-CRITICAL: Do not change output format (breaks UI parsing)
   return {
     vulnerabilities: issues,
     // ...
   };
   
   // GITHUB-SPECIFIC: Batch optimization, not used by WebTool
   if (this.mode === 'github') {
     // ...
   }
   

2. Add route-level protection:

   // WebTool-only route - do not add GitHub logic here
   export async function POST(request: Request) {
     // ...
   }
   

Files to Update: - All 4 analyzers (JavaScriptAnalyzer, PythonAnalyzer, JavaAnalyzer, TypeScriptAnalyzer) - fix-applier.ts - advanced-qwen.ts - pattern-fixer.ts

---

Phase 2: Add Integration Tests (45 minutes)

Goal: Detect when changes to one system break the other

Test Suite: src/__tests__/integration/webtool-github-isolation.test.ts

describe('WebTool & GitHub Isolation', () => {
  describe('Analyzer Output Compatibility', () => {
    it('should return identical results for same code in both contexts', async () => {
      const testCode = `const x = eval(userInput);`;

      // WebTool flow
      const webToolResult = await analyzeCode(testCode, 'javascript');

      // GitHub flow
      const githubResult = await PRAnalyzer.analyzeFile({
        content: testCode,
        filename: 'test.js',
        language: 'javascript'
      });

      // CRITICAL: Same vulnerabilities detected
      expect(webToolResult.security.vulnerabilities.length).toBe(
        githubResult.issues.length
      );

      // Same severity scoring
      expect(webToolResult.security.vulnerabilities[0].severity).toBe(
        githubResult.issues[0].severity
      );
    });

    it('should maintain WebTool performance after GitHub changes', async () => {
      const testCode = readFileSync('fixtures/large-file.js', 'utf-8');

      const start = Date.now();
      await analyzeCode(testCode, 'javascript');
      const duration = Date.now() - start;

      // CRITICAL: Must stay under 3s
      expect(duration).toBeLessThan(3000);
    });
  });

  describe('Fix Generation Isolation', () => {
    it('WebTool generate-fix should work after GitHub changes', async () => {
      const response = await fetch('/api/generate-fix', {
        method: 'POST',
        body: JSON.stringify({
          code: 'const x = eval(userInput);',
          issue: { line: 1, message: 'Unsafe eval', severity: 'high' },
          language: 'javascript'
        })
      });

      const result = await response.json();

      // Should return actual code, not FIXME comments
      expect(result.success).toBe(true);
      expect(result.fixedCode).not.toContain('FIXME');
      expect(result.fixedCode).not.toContain('TODO');

      // Should be fast
      expect(result.estimatedTime).toBeLessThan(15);
    });
  });

  describe('Analyzer Regression Detection', () => {
    it('should maintain detection rate for known vulnerabilities', async () => {
      const testSuite = [
        { code: 'eval(x)', expected: 'Unsafe eval' },
        { code: 'innerHTML = x', expected: 'XSS' },
        { code: 'exec(cmd)', expected: 'Command injection' },
      ];

      for (const test of testSuite) {
        const result = await analyzeCode(test.code, 'javascript');
        const detected = result.security.vulnerabilities.some(v =>
          v.message.includes(test.expected)
        );

        expect(detected).toBe(true);
      }
    });
  });
});

Coverage: 15 tests protecting against cross-system interference

---

Phase 3: Version Analyzer Outputs (20 minutes)

Goal: Detect breaking changes to analyzer output format

Implementation:

1. Add version constant:

   // src/lib/analyzers/version.ts
   export const ANALYZER_OUTPUT_VERSION = '1.0.0';
   

2. Include in all analyzer outputs:

   return {
     version: ANALYZER_OUTPUT_VERSION,
     security: {
       vulnerabilities: issues
     },
     // ...
   };
   

3. Validate in consumers:

   const result = await analyzer.analyze(code);
   
   if (result.version !== ANALYZER_OUTPUT_VERSION) {
     throw new Error(
       `Analyzer version mismatch: expected ${ANALYZER_OUTPUT_VERSION}, got ${result.version}`
     );
   }
   

Benefit: Immediate detection of breaking changes during development

---

Phase 4: Performance Benchmarks (15 minutes)

Goal: Prevent performance regressions that affect WebTool UX

Implementation:

// src/__tests__/performance/analyzer-benchmarks.test.ts
describe('Analyzer Performance Benchmarks', () => {
  it('should analyze small file (<100 lines) in <500ms', async () => {
    const code = generateCode(100);
    const start = performance.now();
    await analyzer.analyze(code);
    const duration = performance.now() - start;

    expect(duration).toBeLessThan(500);
  });

  it('should analyze medium file (500 lines) in <2s', async () => {
    const code = generateCode(500);
    const start = performance.now();
    await analyzer.analyze(code);
    const duration = performance.now() - start;

    expect(duration).toBeLessThan(2000);
  });

  it('should analyze large file (1000 lines) in <3s', async () => {
    const code = generateCode(1000);
    const start = performance.now();
    await analyzer.analyze(code);
    const duration = performance.now() - start;

    expect(duration).toBeLessThan(3000);
  });
});

Run: As part of CI/CD before every deployment

---

Working Rules Going Forward

When Working on WebTool

Safe to modify freely: - /api/analyze route - /api/generate-fix route - /api/fix-all route - src/app/analyze/page.tsx - src/components/AnalysisResults/*

Require caution (run GitHub tests after): - Any analyzer (JavaScriptAnalyzer, PythonAnalyzer, JavaAnalyzer) - advanced-qwen.ts - pattern-fixer.ts

Never modify: - src/lib/github/* (GitHub-only modules) - /api/github/* routes - /api/teams/* routes

---

When Working on GitHub App

Safe to modify freely: - /api/github/* routes - /api/teams/* routes - src/lib/github/* modules (except fix-applier if it affects WebTool) - src/app/teams/* pages - src/components/dashboard/*

Require caution (run WebTool tests after): - Any analyzer modifications - fix-applier.ts (proven to interfere) - advanced-qwen.ts (shared AI provider)

Never modify: - WebTool routes (/api/analyze, /api/generate-fix, /api/fix-all) - WebTool UI (src/app/analyze/page.tsx)

---

Deployment Checklist

Before deploying ANY changes to shared modules:

1. ✅ Run full test suite: npm test
2. All 554+ tests must pass

3. Pay special attention to 96 analyzer tests

4. ✅ Run integration tests: npm test webtool-github-isolation

5. All 15 isolation tests must pass

6. ✅ Run performance benchmarks: npm test analyzer-benchmarks

7. All benchmarks must pass (<3s max)

8. ✅ Manual verification:

9. WebTool: Paste code → Analyze → Generate Fix → Verify actual code returned

10. GitHub: Open PR → Check webhook triggered → Verify AI suggestions posted

11. ✅ Monitor first 1 hour after deployment:

12. Check Vercel logs for errors
13. Verify WebTool /api/analyze response times
14. Check GitHub webhook success rate

---

Incident Response

If WebTool breaks after GitHub deployment:

1. Immediate: Revert deployment (Vercel rollback)
2. Investigate: Check version.json changelog for what changed
3. Fix: Apply fix with isolation tests
4. Verify: Run full isolation test suite
5. Deploy: With extra monitoring

If GitHub breaks after WebTool deployment:

1. Assess impact: Are PRs being analyzed?
2. Revert if critical: Teams are paying customers, can't tolerate downtime
3. Fix: Apply fix with isolation tests
4. Verify: Test on staging PR first
5. Deploy: Gradual rollout

---

Future Improvements (Optional)

Option 1: Separate Fix Generation Logic

Problem: fix-applier.ts is GitHub-specific but influenced WebTool

Solution: Create src/lib/webtool/fix-generator.ts specifically for WebTool

Benefit: Complete isolation, zero interference risk

Cost: Code duplication, maintenance overhead

Verdict: Consider if interference happens again

---

Option 2: Feature Flags for Mode-Specific Logic

Problem: Shared modules need different behavior

Solution:

// src/lib/config/runtime-mode.ts
export const getRuntimeMode = (): 'webtool' | 'github' => {
  // Detect from API route context
  if (headers.get('X-CodeSlick-Mode')) {
    return headers.get('X-CodeSlick-Mode') as 'webtool' | 'github';
  }
  return 'webtool'; // default
};

// Usage in analyzer
const mode = getRuntimeMode();
if (mode === 'github') {
  // Batch optimization
} else {
  // Interactive optimization
}

Benefit: Explicit mode separation in shared modules

Cost: Complexity, more test cases

Verdict: Implement if we add more mode-specific logic

---

Option 3: Separate Analyzer Instances

Problem: Single analyzer serves both systems

Solution: Two instances with different configs

// WebTool analyzer (speed-optimized)
const webToolAnalyzer = new JavaScriptAnalyzer({
  maxDepth: 5,
  timeout: 2000
});

// GitHub analyzer (thoroughness-optimized)
const githubAnalyzer = new JavaScriptAnalyzer({
  maxDepth: 10,
  timeout: 5000
});

Benefit: Each system gets optimized analyzer

Cost: Memory overhead, config management

Verdict: Not needed yet, keep shared for now

---

Metrics to Track

WebTool Health: - /api/analyze response time (target: <3s p95) - /api/generate-fix success rate (target: >95%) - Detection rate for known vulnerabilities (baseline: 100% of test suite)

GitHub Health: - Webhook processing time (target: <30s p95) - PR comment success rate (target: >98%) - Auto-fix PR creation success rate (target: >90%)

Shared Module Health: - Analyzer test pass rate (target: 100%) - Performance benchmarks pass rate (target: 100%) - Integration test pass rate (target: 100%)

Set up alerts: Vercel monitoring, PostHog events, Sentry errors

---

Summary

Current Status: ⚠️ MEDIUM-HIGH RISK

Evidence: Phase 7 GitHub work broke WebTool (version 20251117.14:30)

Root Cause: Shared modules modified without cross-system testing

Protection Plan: 1. ✅ Protective comments (30 min) 2. ✅ Integration tests (45 min) 3. ✅ Version analyzer outputs (20 min) 4. ✅ Performance benchmarks (15 min)

Total Time: ~2 hours

Benefit: Prevents future interference, enables confident development

Next Steps: Implement Phase 1-4, then proceed with WebTool bug fixes

---

Document Owner: Development Team Last Updated: 2025-11-18 Next Review: After implementing protection plan