10 Questions to Ask When Hiring an AI Engineer

Hiring an AI engineer is hard when you are not an AI engineer yourself. The field is full of people who can call an API and produce a demo, and far fewer who can ship something that survives real users. These ten questions are the ones I would ask, designed to surface that difference without needing you to be technical.

1. Walk me through a system you took to production

The single most revealing question. A strong engineer will talk about reliability, edge cases, cost, and what broke and how they fixed it. Someone who has only built demos will describe the happy path and go quiet on what happened after launch. Listen for the messy middle, not the polished result.

2. How do you stop an LLM from hallucinating?

A good answer talks about grounding with retrieval, validation, and evaluation, not just a better prompt. If the answer is only prompt engineering, they have not run a system where wrong answers had consequences.

3. How do you control AI costs?

Token bills can surprise you. The answer should mention model routing, caching, and output limits. An engineer who has shipped real systems thinks about cost because they have been responsible for a bill.

4. When would you not use AI for a problem?

This is a credibility test. A practitioner knows that a normal script often beats an AI system, and will happily say so. Someone who answers AI everywhere is selling, not engineering.

5. How do you handle errors and failures?

Production is mostly error handling. Look for retries, fallbacks, validation, and monitoring. A vague answer here means a fragile system later.

6. How do you evaluate whether a change improved things?

Strong engineers measure. They build evaluation sets and compare outputs, rather than tweaking by feel. No evaluation means quality drifts and nobody notices.

7. What is your backend and systems experience?

AI features live inside real software. An engineer who also understands APIs, databases, and queues will build something that fits your stack and holds up. Pure prompt skills without engineering depth tend to produce brittle results.

8. How do you work with an existing team?

If you have engineers, the answer should cover code reviews, shared repos, and clear documentation, so your team can own the system after handoff. You want a collaborator, not a black box.

9. Can you show me code or a real project?

Talk is cheap. A real engineer can point to shipped work, a repository, or a product. Founders who have built and launched their own products, in particular, have felt every shortcut and learned from it.

10. What would you build first for us, and why?

A good engineer will not promise everything. They will pick one bounded, high-value problem and explain why it is the right place to start. That judgment is worth more than a long list of capabilities.

If you are evaluating AI engineers and want a straight, technical conversation rather than a sales pitch, I am happy to have one. Ask me any of these and I will answer plainly, with examples from systems I have shipped.