CodeSlick GitHub App - Testing Guide¶

Simulating New User Experience¶

Last Updated: November 26, 2025 Purpose: Step-by-step guide to test GitHub App as a new user would experience it

Prerequisites¶

Before testing, ensure: - ✅ GitHub App is installed and configured - ✅ Webhook URL is set to production: https://codeslick.dev/api/github/webhook - ✅ Webhook secret is configured in environment variables - ✅ App permissions are correct (Read: Contents, Issues, Pull Requests / Write: Pull Requests)

Testing Approach: 3 Methods¶

Method 1: Quick Test with Pre-Made Vulnerable Files ⭐ RECOMMENDED¶

Time: 5 minutes Best For: Rapid validation that GitHub App is working

Step 1: Create Test Repository¶

# Create new repository on GitHub (via web UI or CLI)
gh repo create codeslick-test-1 --public --clone

cd codeslick-test-1

Step 2: Copy Test Files from CodeSlick Repo¶

# Copy pre-made vulnerable test files
cp /path/to/codeslick2/test-files/github-pr-test.js ./vulnerable-code.js
cp /path/to/codeslick2/test-files/github-pr-test.py ./vulnerable-code.py

# Or create simple vulnerable file manually
cat > vulnerable.js << 'EOF'
// CodeSlick Test - Multiple Vulnerabilities

// 1. SQL Injection (CRITICAL)
const userId = req.query.id;
const query = "SELECT * FROM users WHERE id = '" + userId + "'";
db.query(query);

// 2. XSS Vulnerability (HIGH)
document.getElementById('output').innerHTML = userInput;

// 3. Hardcoded Credentials (CRITICAL)
const apiKey = "sk-1234567890abcdef";
const password = "admin123";

// 4. Command Injection (CRITICAL)
const filename = req.body.file;
exec('cat ' + filename);

// 5. eval() Usage (CRITICAL)
eval(req.body.code);

// 6. Prototype Pollution (HIGH)
Object.assign(target, req.body);

// 7. Missing React key (MEDIUM)
users.map(user => <div>{user.name}</div>);

// 8. Direct state mutation (MEDIUM)
this.state.count = 5;
EOF

# Commit initial code
git add .
git commit -m "Initial commit with vulnerable code"
git push origin main

Step 3: Install CodeSlick GitHub App¶

Go to: https://github.com/apps/codeslick (or your GitHub App URL)
Click "Install"
Select "Only select repositories" → Choose codeslick-test-1
Click "Install & Authorize"

Step 4: Create Pull Request¶

# Create feature branch
git checkout -b fix/security-issues

# Make a small change (so PR has diff)
echo "// Fixed version coming soon" >> vulnerable.js

git add .
git commit -m "Starting security fixes"
git push origin fix/security-issues

# Create PR via GitHub CLI
gh pr create \
  --title "Security fixes" \
  --body "Testing CodeSlick automated analysis" \
  --base main \
  --head fix/security-issues

Step 5: Verify Automated Analysis¶

Expected Behavior (within 10-30 seconds):

Check Status:
PR should show "CodeSlick Security Analysis" check
Status: ⚠️ "Security issues detected"
PR Comments:
CodeSlick bot should post comment with analysis results
Should detect 8+ vulnerabilities in vulnerable.js
Each vulnerability includes:
- Severity (CRITICAL/HIGH/MEDIUM/LOW)
- Line number
- Description
- Remediation suggestions
- OWASP mapping
- CVSS score
Large File Warning (if applicable):
If file has 15+ vulnerabilities or 300+ lines
Warning comment should appear explaining why AI analysis was skipped

Example Comment Format:

## 🔒 CodeSlick Security Analysis

Analyzed 1 file and found **8 security vulnerabilities**

### 📊 Summary by Severity

- 🔴 **CRITICAL**: 4 issues
- 🟠 **HIGH**: 2 issues
- 🟡 **MEDIUM**: 2 issues

### vulnerable.js (8 issues)

#### 🔴 CRITICAL - SQL Injection (Line 5)
**Description**: Unsafe SQL query construction using string concatenation
**OWASP**: A03:2021 - Injection
**CVSS**: 9.8 (Critical)
**Fix**: Use parameterized queries or prepared statements

[View Details] [Apply AI Fix]

...

Method 2: Realistic User Scenario (Comprehensive Test)¶

Time: 15-20 minutes Best For: Full end-to-end validation

Step 1: Create Realistic Project¶

# Create new repository
gh repo create my-todo-app --public --clone
cd my-todo-app

# Initialize Node.js project
npm init -y
npm install express sqlite3

# Create realistic but vulnerable code
cat > server.js << 'EOF'
const express = require('express');
const sqlite3 = require('sqlite3');
const app = express();

// Vulnerability 1: No helmet (missing security headers)
app.use(express.json());

// Vulnerability 2: Hardcoded database credentials
const db = new sqlite3.Database('todos.db', 'admin', 'password123');

// Vulnerability 3: SQL Injection in GET endpoint
app.get('/todo/:id', (req, res) => {
  const query = `SELECT * FROM todos WHERE id = ${req.params.id}`;
  db.get(query, (err, row) => {
    if (err) throw err;
    res.json(row);
  });
});

// Vulnerability 4: Command Injection in file export
app.post('/export', (req, res) => {
  const filename = req.body.filename;
  require('child_process').exec(`cat todos.db > ${filename}.bak`);
  res.send('Exported');
});

// Vulnerability 5: XSS in template rendering
app.get('/search', (req, res) => {
  const term = req.query.q;
  res.send(`<h1>Results for: ${term}</h1>`);
});

app.listen(3000);
EOF

# Commit initial vulnerable version
git add .
git commit -m "Initial todo app implementation"
git push origin main

Step 2: Install GitHub App¶

Same as Method 1, Step 3

Step 3: Create Security Fix PR¶

# Create fix branch
git checkout -b security/fix-sql-injection

# Fix ONE vulnerability (to test incremental fixes)
cat > server.js << 'EOF'
const express = require('express');
const sqlite3 = require('sqlite3');
const app = express();

app.use(express.json());

const db = new sqlite3.Database('todos.db', 'admin', 'password123');

// FIXED: SQL Injection - using parameterized query
app.get('/todo/:id', (req, res) => {
  const query = 'SELECT * FROM todos WHERE id = ?';
  db.get(query, [req.params.id], (err, row) => {
    if (err) throw err;
    res.json(row);
  });
});

// Still vulnerable: Command Injection
app.post('/export', (req, res) => {
  const filename = req.body.filename;
  require('child_process').exec(`cat todos.db > ${filename}.bak`);
  res.send('Exported');
});

// Still vulnerable: XSS
app.get('/search', (req, res) => {
  const term = req.query.q;
  res.send(`<h1>Results for: ${term}</h1>`);
});

app.listen(3000);
EOF

git add .
git commit -m "Fix SQL injection vulnerability"
git push origin security/fix-sql-injection

# Create PR
gh pr create \
  --title "Security: Fix SQL injection in todo endpoint" \
  --body "Replaced string concatenation with parameterized query" \
  --base main \
  --head security/fix-sql-injection

Step 4: Verify Progressive Analysis¶

Expected Behavior: - ✅ SQL injection: Should be gone (fixed) - ⚠️ Command injection: Should still be detected - ⚠️ XSS: Should still be detected - ⚠️ Hardcoded credentials: Should still be detected

This validates that CodeSlick: - Detects remaining vulnerabilities - Doesn't report false positives on fixed code - Provides accurate line numbers after code changes

Method 3: Multi-Language Testing (Advanced)¶

Time: 30 minutes Best For: Validating multi-language support

Create Repository with Multiple Languages¶

gh repo create fullstack-app --public --clone
cd fullstack-app

# Create backend (Python - Django)
mkdir backend
cat > backend/views.py << 'EOF'
from django.shortcuts import render
from django.db import connection
from django.views.decorators.csrf import csrf_exempt

# Vulnerability 1: CSRF disabled
@csrf_exempt
def create_user(request):
    username = request.POST.get('username')

    # Vulnerability 2: SQL Injection
    query = f"INSERT INTO users (username) VALUES ('{username}')"
    with connection.cursor() as cursor:
        cursor.execute(query)

    return JsonResponse({'status': 'ok'})

# Vulnerability 3: DEBUG mode in production
DEBUG = True
EOF

# Create frontend (JavaScript - React)
mkdir frontend
cat > frontend/App.js << 'EOF'
import React from 'react';

function UserProfile({ user }) {
  // Vulnerability 1: Missing key in list
  const posts = user.posts.map(post => (
    <div>{post.title}</div>
  ));

  // Vulnerability 2: Direct state mutation
  handleUpdate = () => {
    this.state.user.name = 'Updated';
  }

  // Vulnerability 3: XSS via dangerouslySetInnerHTML
  return (
    <div dangerouslySetInnerHTML={{__html: user.bio}} />
  );
}

export default UserProfile;
EOF

# Create API layer (Java - Spring)
mkdir api
cat > api/UserController.java << 'EOF'
package com.example.api;

import org.springframework.web.bind.annotation.*;

@RestController
public class UserController {

    // Vulnerability 1: Missing authorization
    @GetMapping("/admin/users")
    public List<User> getAllUsers() {
        return userRepository.findAll();
    }

    // Vulnerability 2: SQL Injection in JPA
    @GetMapping("/search")
    public List<User> search(@RequestParam String name) {
        String query = "SELECT u FROM User u WHERE u.name = '" + name + "'";
        return entityManager.createQuery(query).getResultList();
    }

    // Vulnerability 3: Missing input validation
    @PostMapping("/user")
    public User createUser(@RequestBody User user) {
        return userRepository.save(user);
    }
}
EOF

# Commit and push
git add .
git commit -m "Initial fullstack implementation"
git push origin main

# Create PR with mixed changes
git checkout -b feature/user-management
echo "// Minor update" >> frontend/App.js
git add .
git commit -m "Update user profile component"
git push origin feature/user-management

gh pr create \
  --title "Feature: User management improvements" \
  --body "Testing multi-language analysis" \
  --base main \
  --head feature/user-management

Expected Multi-Language Results¶

Backend (Python): - 3 vulnerabilities detected (CSRF exempt, SQL injection, DEBUG=True)

Frontend (JavaScript): - 3 vulnerabilities detected (Missing key, state mutation, dangerouslySetInnerHTML)

API (Java): - 3 vulnerabilities detected (Missing auth, SQL injection, missing validation)

Total: 9 vulnerabilities across 3 languages

Verification Checklist¶

✅ GitHub App Installation¶

App appears in repository's "Installed GitHub Apps"
Webhook events are being delivered (check GitHub App settings → Advanced → Recent Deliveries)
Webhook responses are 200 OK (not 500 errors)

✅ PR Analysis¶

CodeSlick check appears in PR status checks
Check runs within 30 seconds of PR creation
Comment is posted with vulnerability details
Severity levels are correct (CRITICAL/HIGH/MEDIUM/LOW)
Line numbers match actual code locations
OWASP mappings are present
CVSS scores are displayed

✅ AI Features (if API key configured)¶

"Apply AI Fix" button appears for each vulnerability
Clicking button opens modal with AI-generated fix
Fix is context-aware and preserves surrounding code
Diff view shows before/after comparison
"Apply This Fix" commits the change to new branch

✅ Large File Handling¶

Files with 15+ vulnerabilities get warning comment
Files with 300+ lines get warning comment
Warning explains why AI was skipped
Static analysis results still shown
Alternatives suggested (split files, focus on critical, manual review)

✅ Edge Cases¶

Empty PRs (no files changed) → No analysis posted
Binary files → Skipped gracefully
Non-supported languages → Skipped with notice
Very large repos (100+ files) → Processes all files
Deleted files in PR → No errors
Renamed files → Analyzed correctly

Troubleshooting¶

Issue: No Comment Posted¶

Possible Causes: 1. Webhook not configured correctly 2. App permissions insufficient 3. Server error (check Vercel logs)

Debug Steps:

# Check webhook deliveries
# GitHub App Settings → Advanced → Recent Deliveries
# Look for 2XX response codes

# Check Vercel logs
vercel logs --follow

# Check database for team/user
# Ensure user has team with active subscription

Issue: Comment Posted But No Vulnerabilities¶

Possible Causes: 1. Code is actually secure (no vulnerabilities) 2. Language not supported 3. Analyzer not detecting patterns

Debug Steps:

# Test with known vulnerable code
cp test-files/github-pr-test.js ./test.js

# Verify analyzer detects issues locally
# Use WebTool at https://codeslick.dev

Issue: "Apply AI Fix" Not Working¶

Possible Causes: 1. User hasn't configured API key 2. File too large (>300 lines) 3. AI service timeout

Debug Steps: - Check ApiKeyManager has valid configuration - Verify file size is under 300 lines - Check Together AI/OpenRouter API status

Issue: Check Never Completes¶

Possible Causes: 1. Webhook timeout (>30s analysis) 2. Database connection issue 3. GitHub API rate limit

Debug Steps:

# Check Vercel function logs
vercel logs --follow

# Look for timeout errors (>30s execution)
# Large repos may need optimization

Performance Benchmarks¶

Expected Timing¶

Scenario	Expected Duration
Single file (10 vulnerabilities)	5-10 seconds
Multiple files (3-5 files)	10-20 seconds
Large PR (10+ files)	20-30 seconds
AI Fix generation	10-90 seconds (depends on file size)

Webhook Payload Size¶

PR Size	Payload	Processing
1 file changed	~5KB	<5s
5 files changed	~25KB	10-15s
20 files changed	~100KB	20-30s

Best Practices for Testing¶

1. Use Dedicated Test Repositories¶

Don't test on production repositories
Create throwaway repos: codeslick-test-1, codeslick-test-2
Delete test repos after validation

2. Test All Severity Levels¶

Create PRs with vulnerabilities of each severity: - CRITICAL: SQL injection, command injection, hardcoded credentials - HIGH: XSS, path traversal, prototype pollution - MEDIUM: Missing validation, weak crypto, CORS issues - LOW: Code quality, console.log in production

3. Test All Supported Languages¶

JavaScript: Node.js, React patterns
TypeScript: Type safety issues
Python: Django, Flask frameworks
Java: Spring Security, Log4j

4. Test Team Collaboration Features¶

Create team with multiple members
Assign different roles (owner, admin, member)
Verify permissions work correctly
Test usage limits and quota enforcement

Sample Test Cases¶

Test Case 1: SQL Injection Detection¶

File: vulnerable.js

const query = "SELECT * FROM users WHERE id = '" + userId + "'";

Expected: - ✅ Detected as CRITICAL - ✅ Line number correct - ✅ OWASP A03:2021 - Injection - ✅ CVSS 9.8

Test Case 2: React Missing Key¶

File: App.jsx

users.map(user => <div>{user.name}</div>)

Expected: - ✅ Detected as MEDIUM - ✅ React-specific check - ✅ Suggests adding key={user.id}

Test Case 3: Django CSRF Exempt¶

File: views.py

@csrf_exempt
def create_user(request):
    ...

Expected: - ✅ Detected as HIGH - ✅ Django framework check - ✅ OWASP A01:2021 - Broken Access Control

Test Case 4: Large File Warning¶

File: large.js (400+ lines OR 20+ vulnerabilities)

Expected: - ✅ Warning comment posted - ✅ Explains why AI skipped - ✅ Static analysis still shown - ✅ Suggests alternatives

Monitoring & Metrics¶

Key Metrics to Track¶

Analysis Success Rate: % of PRs successfully analyzed
Average Analysis Time: Median time from PR open to comment posted
False Positive Rate: User feedback on incorrect detections
Fix Application Rate: % of "Apply AI Fix" clicks that result in commits
User Engagement: % of users who configure API keys

Dashboard Queries¶

-- Analysis success rate (last 7 days)
SELECT
  COUNT(*) as total_prs,
  SUM(CASE WHEN analysis_completed THEN 1 ELSE 0 END) as successful,
  (SUM(CASE WHEN analysis_completed THEN 1 ELSE 0 END)::float / COUNT(*)) * 100 as success_rate
FROM pr_analyses
WHERE created_at > NOW() - INTERVAL '7 days';

-- Average analysis time
SELECT
  AVG(EXTRACT(EPOCH FROM (comment_posted_at - pr_opened_at))) as avg_seconds
FROM pr_analyses
WHERE analysis_completed = true
  AND created_at > NOW() - INTERVAL '7 days';

Production Validation¶

Before launching to real users:

Last Updated: November 26, 2025 Version: 1.0 Status: Production Testing Guide