Choosing an AI integration partner is one of the highest-stakes vendor decisions a CTO makes. Get it right and you ship an AI-powered product that differentiates your business. Get it wrong and you waste 6 months and $50,000+ on a prototype that never reaches production.
We have been on both sides of this evaluation — as the agency being evaluated and as advisors helping clients evaluate other partners. This checklist comes from real experience with what matters and what does not.
Category 1: Technical Depth (8 Questions)
1. Do they understand the difference between prompt engineering, RAG, and fine-tuning?
Green flag: They can explain when each approach is appropriate and recommend the simplest option that meets your needs. Red flag: They default to "fine-tuning" or "custom model" for everything. This suggests they are either overselling or lack experience with modern LLM applications.
2. Can they show production AI applications (not just demos)?
Green flag: Live URLs, case studies with specific metrics, client references. Red flag: Only showing prototypes, notebooks, or "proof of concepts" that never went to production.
3. How do they handle AI cost management?
Green flag: They discuss token budgets, caching strategies, model routing, and include cost projections in proposals. Red flag: "We will optimize costs later" or no mention of API costs at all.
4. What is their testing strategy for AI features?
Green flag: They describe evaluation pipelines, golden datasets, semantic similarity checks, and regression testing for prompt changes. Red flag: "We test it manually" or reliance on traditional unit tests for non-deterministic AI outputs.
5. How do they handle AI failures and fallbacks?
Green flag: Every AI feature has a defined fallback behavior. They can explain their error handling and graceful degradation patterns. Red flag: No fallback strategy. "The AI will handle it" without contingency planning.
6. What LLM providers and models have they worked with?
Green flag: Experience with multiple providers (OpenAI, Anthropic, open-source models). They can articulate trade-offs between models. Red flag: Only experience with one provider, or inability to explain why they chose a specific model.
7. Do they understand embeddings and vector search?
Green flag: They can explain RAG architectures, embedding models, vector databases, and chunking strategies. Red flag: AI work limited to simple API calls without any retrieval augmentation.
8. Can they build the full product (web + AI) or just the AI?
Green flag: Integrated team that builds both the application and the AI features as one cohesive product. Red flag: AI-only consultancy that hands you a model and leaves you to integrate it.
Category 2: Process and Communication (6 Questions)
9. What is their development methodology?
Green flag: Agile with regular demos, sprint planning, and iterative delivery. Weekly or biweekly stakeholder updates. Red flag: Waterfall approach ("We will show you the finished product in 3 months") or no defined process.
10. How do they handle timezone differences?
Green flag: Defined overlap hours, async-first communication with structured updates, clear escalation paths. Red flag: "We will figure it out" or no acknowledgment that timezone management requires deliberate systems.
11. What does their communication stack look like?
Green flag: Structured tools — project management (Linear, Jira), async updates (Slack, email), video calls (scheduled cadence), documentation (Notion, Confluence). Red flag: "Just WhatsApp us whenever" with no structured reporting.
12. How do they handle scope changes?
Green flag: Written change request process with impact assessment (cost, timeline, technical). Changes are documented before implementation. Red flag: "Sure, we can add that" to every request without assessing impact.
13. What does handoff look like?
Green flag: Documentation, knowledge transfer sessions, code walkthroughs, deployment guides, and a defined support period. Red flag: "Here is the repository" with no documentation or knowledge transfer.
14. Who will actually work on your project?
Green flag: Named team members with LinkedIn profiles and relevant experience. You meet the people doing the work. Red flag: Anonymous team. "Our developers will handle it" without specifics.
Category 3: Portfolio and References (4 Questions)
15. Can they show similar projects in your industry?
Green flag: Case studies in your domain with specific technical details and measurable outcomes. Red flag: Generic portfolio with no industry-specific experience.
16. Do their projects have live URLs?
Green flag: Working, publicly accessible applications that you can test yourself. Red flag: Screenshots only, or "the project is under NDA" for every single case study.
17. Will they provide client references?
Green flag: Willing to connect you with past clients for honest feedback. Red flag: Refuses references or provides only written testimonials.
18. What is their team composition?
Green flag: Mix of frontend/backend developers, AI/ML specialists, designers, and project managers. Clear roles. Red flag: Single-person shop claiming to do everything, or unclear team structure.
Category 4: Security and Compliance (3 Questions)
19. How do they handle data security?
Green flag: Encryption at rest and in transit, secure API key management, access controls, and data retention policies. Red flag: No security documentation or "we use standard practices" without specifics.
20. Are they compliant with relevant regulations?
Green flag: Awareness of GDPR, SOC 2, HIPAA (if applicable), and data residency requirements. Willing to sign NDAs and DPAs. Red flag: "What is GDPR?" or dismissive attitude toward compliance.
21. What happens to your data after the project?
Green flag: Clear data deletion policy. Your data is deleted or returned after project completion. Red flag: No data handling policy or claims of indefinite data retention.
Category 5: Cost and Terms (2 Questions)
22. Is their pricing transparent?
Green flag: Detailed proposal with line items, milestone-based payments, and clear definition of what is included vs. extra. Red flag: Vague pricing, "it depends" without providing ranges, or hourly billing without estimates.
23. Who owns the intellectual property?
Green flag: Full IP transfer to you upon final payment. This is clearly stated in the contract. Red flag: Agency retains IP rights, licensing restrictions, or unclear ownership terms.
Scoring Framework
Rate each question 0-2:
- 0 = Red flag or unable to answer
- 1 = Adequate but not impressive
- 2 = Green flag, exceeds expectations
| Score Range | Assessment |
|---|---|
| 38-46 | Strong partner. Proceed with confidence. |
| 28-37 | Adequate with some gaps. Negotiate improvements. |
| 18-27 | Significant concerns. Consider alternatives. |
| Below 18 | Walk away. |
Final Note
No partner will score perfectly on every question. What matters is the overall pattern. A partner who scores well on technical depth but weakly on communication can be managed with more structure. A partner who scores poorly on technical depth cannot be managed — they simply lack the capability.
The most expensive mistake is choosing based on price alone. The cheapest option that fails costs more than the mid-range option that ships.
Evaluating AI partners? Schedule a discovery call — we are happy to answer all 23 questions.
Comments