Skip to content

Multi-Provider AI Support - Test Guide

Overview

This guide explains how to test the new multi-provider AI support implemented in CodeSlick.

What Was Implemented

1. Provider Configuration (src/lib/ai/provider-config.ts)

  • 5 AI Providers: OpenAI, Anthropic, Together.ai, Groq, Google Gemini
  • 20+ Models across all providers
  • Provider-specific pricing (automatic cost calculation)
  • API key validation (format checking, prefix detection)
  • Helper functions: validateApiKey(), detectProvider(), calculateCost()

2. Unified AI Client (src/lib/ai/unified-client.ts)

  • Single interface for all providers
  • Automatic routing to provider-specific API implementations
  • Token counting and cost calculation
  • Error handling with provider-specific error messages
  • Latency tracking for performance monitoring

3. Multi-Provider Usage Tracking (src/lib/tracking/usage-tracker.ts)

  • Provider field added to UsageRecord interface
  • Separate tracking per provider (usage:openai, usage:anthropic, etc.)
  • Backward compatible with existing Together.ai data
  • New method: getAllProvidersMetrics() for dashboard

4. Redesigned UI (src/components/ApiKeyManager.tsx)

  • Provider dropdown (5 providers)
  • Model dropdown (dynamically updates based on provider)
  • Pricing display (shows cost per 1M tokens)
  • Context window display
  • Smart validation (provider-specific rules)
  • Setup instructions with direct links to each provider

Testing the Implementation

Phase 1: Visual Verification (Browser)

  1. Start the development server: 

    npm run dev

  2. Open CodeSlick in your browser:

  3. Navigate to http://localhost:3000

  4. Test ApiKeyManager UI:

  5. Click "Use My API Key" button
  6. Verify you see "Multi-Provider AI Configuration" modal
  7. Test provider dropdown - should show: OpenAI, Anthropic, Together.ai, Groq, Google Gemini

  8. Change provider and verify:

    • Model dropdown updates with provider-specific models
    • Pricing info displays correctly
    • Setup instructions change
    • API key placeholder changes

  9. Test API Key Validation:

  10. Together.ai: Enter a 64-character hex string (valid) → should show green checkmark
  11. OpenAI: Enter sk-proj- + random chars → should show green checkmark
  12. Anthropic: Enter sk-ant-api03- + 88 chars → should show green checkmark
  13. Groq: Enter gsk_ + 52 chars → should show green checkmark
  14. Gemini: Enter AIzaSy + 33 chars → should show green checkmark
  15. Test invalid format → should show red error

Phase 2: Cost Calculation Verification

Run these calculations manually to verify pricing is correct:

// Together.ai Qwen 2.5 Coder
// 1M input tokens: $0.30
// 1M output tokens: $0.30
// Total for 2M tokens: $0.60

// OpenAI GPT-4 Turbo
// 1M input tokens: $10.00
// 1M output tokens: $30.00
// Total for 2M tokens: $40.00

// Anthropic Claude 3.5 Sonnet
// 1M input tokens: $3.00
// 1M output tokens: $15.00
// Total for 2M tokens: $18.00

// Groq Mixtral 8x7B
// 1M input tokens: $0.27
// 1M output tokens: $0.27
// Total for 2M tokens: $0.54

// Gemini 1.5 Flash 8B
// 1M input tokens: $0.075
// 1M output tokens: $0.30
// Total for 2M tokens: $0.375 (cheapest!)

Phase 3: Provider Comparison

Provider Best Model Input Cost Output Cost Context Best For
Gemini Flash 8B $0.075 $0.30 1M tokens Cheapest, largest context
Together.ai Qwen Coder $0.30 $0.30 32K tokens Code-specific tasks
Groq Mixtral 8x7B $0.27 $0.27 32K tokens Fastest inference
Anthropic Sonnet 3.5 $3.00 $15.00 200K tokens Safety & reasoning
OpenAI GPT-4 Turbo $10.00 $30.00 128K tokens Most capable

Phase 4: Manual Integration Test

To test the complete flow:

  1. Configure a provider:
  2. Open ApiKeyManager
  3. Select "Together.ai"
  4. Select "Qwen 2.5 Coder 32B"
  5. Enter your API key
  6. Click "Validate" → should show green checkmark

  7. Click "Save & Use" → should show success

  8. Verify localStorage:

  9. Open browser DevTools → Application → Local Storage
  10. Find codeslick_ai_config

  11. Should contain: 

    {
  "provider": "together",
  "apiKey": "your-key-here",
  "model": "Qwen/Qwen2.5-Coder-32B-Instruct"
}

  12. Test code fix generation:

  13. Paste code with an error
  14. Click "Analyze Code"
  15. Click "Generate Fix" on an issue
  16. Verify it uses your configured provider

Phase 5: Unit Test Coverage

The test file src/lib/ai/__tests__/multi-provider.test.ts covers:

Test Suite 1: Provider Configuration

  • ✓ All 5 providers configured
  • ✓ All 20+ models have pricing
  • ✓ Base URLs are correct
  • ✓ Key formats are defined

Test Suite 2: API Key Validation

  • ✓ Valid keys accepted for all providers
  • ✓ Invalid keys rejected
  • ✓ Provider detection from key prefix
  • ✓ Prefix matching works

Test Suite 3: Model Configuration

  • ✓ Default models exist
  • ✓ Model retrieval works
  • ✓ Non-existent models return null
  • ✓ Model pricing is defined

Test Suite 4: Cost Calculation

  • ✓ Together.ai costs: $0.60 per 2M tokens
  • ✓ OpenAI costs: $40.00 per 2M tokens
  • ✓ Anthropic costs: $18.00 per 2M tokens
  • ✓ Small requests: ~$0.00045

Test Suite 5: UnifiedAIClient

  • ✓ Client creation for all providers
  • ✓ Invalid provider throws error
  • ✓ Invalid model throws error
  • ✓ API routing works

Test Suite 6: Integration Scenarios

  • ✓ Complete user workflow
  • ✓ Cost comparison
  • ✓ Provider switching

Known Limitations

  1. No actual API calls in tests - Tests verify configuration and validation only
  2. Requires manual testing - UI interactions must be tested in browser
  3. Cost calculation is estimation - Actual costs may vary slightly

Next Steps

Backend Integration (Not Yet Implemented)

The following still need to be updated to use UnifiedAIClient:

  1. /api/generate-fix/route.ts - Individual fix generation
  2. /api/analyze/route.ts - Advanced AI analysis (if enabled)
  3. Usage tracking integration

Dashboard Updates (Not Yet Implemented)

  1. Provider breakdown in metrics dashboard
  2. Cost comparison charts
  3. Per-provider analytics

Success Criteria

✅ Phase 1 Complete: - Provider configuration created - UnifiedAIClient implemented - UsageTracker supports multi-provider - ApiKeyManager redesigned

⏳ Phase 2 Pending: - Backend integration - Dashboard updates - End-to-end testing with real API keys

Quick Verification Checklist

  • Dev server starts without errors
  • ApiKeyManager modal opens
  • All 5 providers appear in dropdown
  • Model dropdown updates when changing providers
  • Pricing displays correctly
  • API key validation works for each provider
  • Configuration saves to localStorage
  • No console errors

Troubleshooting

Problem: Modal doesn't open - Fix: Check browser console for errors, verify imports in page.tsx

Problem: Provider dropdown is empty - Fix: Verify provider-config.ts exports correctly

Problem: Validation always fails - Fix: Check regex patterns in provider-config.ts

Problem: Pricing shows $0.00 - Fix: Verify model pricing in provider-config.ts

Support

For issues or questions: 1. Check browser console for errors 2. Verify all files are saved 3. Restart dev server 4. Clear browser cache and localStorage

Status: ✅ Core infrastructure complete and ready for testing Last Updated: 2025-10-18