How to evaluate an AI consultant: questions worth asking
There are a lot of AI consultants. Many of them are not actually engineers. Here are the questions that quickly tell you which is which.
The AI consulting market has filled up fast over the past eighteen months, and the average quality has dropped accordingly. There are excellent practitioners and there are people who watched a couple of webinars and bought a domain. The two are hard to distinguish from a website. Here are the questions we'd ask any consultant — including us — before signing anything.
First: who, specifically, will run my engagement, and have they done this before? If the answer involves a 'team' or an 'engagement manager,' that's a sign you're hiring an account person who will hand you off to a junior implementer. You want the most senior person in the room from day one and answering your texts in month nine.
Second: can you walk me through one project end-to-end, including what went wrong? Real engagements have texture. Scope changed. An integration was harder than expected. A model behaved oddly in the second month. A consultant who can't speak to that texture has either not done many engagements or is hiding something. Either is a problem.
Third: what's your stance on data handling? Vague answers — 'we use enterprise providers,' 'it's all very secure' — are red flags. You want specifics: what infrastructure, what audit trails, what happens to your data if you leave them. If you can't get a clear answer in two minutes, the consultant doesn't have one.
Fourth: what does the engagement look like after deployment? A consultant who goes quiet after launch is one who hasn't thought about the operational reality of running AI in production. The interesting work is mostly after deployment, not before. Ask explicitly about evals, drift, on-call, and quarterly reviews.
Fifth: what would make you tell me not to hire you? Anyone whose answer is 'oh, we're a fit for everyone' is a salesperson. Anyone whose answer is specific and thoughtful — 'if your processes aren't documented, we can't help,' or 'if you're hoping to replace headcount, that's not what we do' — is probably a real practitioner.