The seven questions we’d want asked of us and any vendor on your shortlist.
Choosing the wrong engineering training vendor is expensive in ways most procurement processes struggle to detect. If you’re evaluating engineering training vendors in financial services, most proposals will look credible on the surface. Course catalogues line up. Certifications line up. Past-client logos line up.
The challenge is that many firms optimise for course-delivery metrics, completion rates, satisfaction scores, certifications issued, not for engineering capability outcomes that show up in production three months later.
The seven questions below are the ones we’d want a serious buyer to ask us, and the ones we’d want them asking everyone else on the shortlist. They’re designed to separate vendors who teach engineering from vendors who have actually built and operated the systems they teach.
You can use them in a written RFI, a vendor presentation, or a thirty-minute screening call, and they apply equally whether the budget line reads engineering training, developer training, or technical L&D. The answers tend to become revealing very quickly.
How to evaluate who will actually deliver the engineering training and whether they have shipped real production work in the last three years.
The single most important question, and the one most vendors will route around. Ask for the named individuals who will deliver the engagement, not the firm. Then ask what production systems each of them has built or run in the last three years.
Our CEO Mike Clarke insists on a simple internal rule: the instructor named on a Mallon proposal is the same instructor in the room with your cohort. The mismatch between named lead and actual deliverer is far more common in this market than buyers realise.
Red flag: “We have a roster of certified instructors.” This usually means career trainers who teach the same material to many clients. The honest answer names the engineer, what they shipped, and what they currently work on between teaching engagements.
Engineering content goes stale fast. Annual reviews aren’t enough; the question is whether your instructors are still doing the work.
Engineering changes fast. Training content needs revising continually, not annually, not “as required.” Ask when each module was last updated and what changed. Ask whether the instructors are still doing the work the content describes, or whether they last did it three years ago.
Red flag: “Materials reviewed annually.” That’s the bare minimum and rarely enough. The stronger signal is a vendor whose instructors edit the content themselves between cohorts because they keep noticing things in the field.
How to assess engineering capability in an era when AI can produce code that passes most coursework by testing reasoning and trade-offs, not output.
This is a 2026 question that didn’t exist three years ago. Most assessment frameworks, multiple-choice, practical exercises, and build-a-thing can be passed by a learner who hasn’t understood anything but has Copilot or Claude open in another window.
The harder, more useful test:
Mike put it directly on a recent podcast: “time to first commit tells you the developer pushed some code. It doesn’t tell you if the code was any good.” The same principle drives every AI-era assessment we now design.
Can the learner explain why they made the trade-offs they made? Can they read unfamiliar code in your stack and predict where it will break under load? Can they walk through a code review and identify the question the senior engineer would ask?
Red flag: “We use a proctored assessment.” That only solves the cheating problem, not the comprehension problem. The honest answer involves live debugging in unfamiliar codebases, code reviews where the learner explains their thinking, and conversations about trade-offs.
How to measure long-term engineering capability after the training programme ends, when course completion stops being a useful signal.
The question worth asking is what changes for the learner sixty to ninety days after the engagement ends, when they’re back in their team, on their normal sprint cadence, with no instructor in the room.
Ask how the vendor designs for retention. Ask what evidence they have that retention actually happens. Ask whether the same instructor stays in contact with the cohort once the programme ends.
Red flag: “Comprehensive curriculum and post-course materials.” This usually means a SharePoint folder no one opens. The honest answer involves spaced revisit patterns, named on-the-job support, and follow-up check-ins from the instructor who ran the course.
The most cost-effective due diligence step in vendor evaluation, and the one most buyers skip.
Not a marketing video. Not a highlight reel. A genuine, untouched recording of a session run for a previous client (anonymised where necessary). A vendor confident in their delivery will say yes within an hour. A vendor whose delivery doesn’t survive honest viewing will find reasons not to share one.
Vendor pitches showcase the strongest instructor. The reality of multi-cohort delivery is that they cannot be in every room.
Ask what happens when they’re not who covers, what the quality gap looks like, and how the vendor manages variance.
Red flag: “All our instructors meet the same standard.” They don’t. No firms do. The honest answer acknowledges variance and explains how it’s managed.
How to assess a vendor’s editorial judgement by asking what they’ve removed from the curriculum, not what they’ve added.
A serious training vendor has opinions about what matters and what doesn’t, and those opinions get sharper over time. Ask what topics or modules the vendor has retired in the last two years, and why. The answers reveal whether they have editorial judgement or whether they teach whatever the market currently asks for.
Red flag: “We respond to client demand.” Translation: no editorial judgement. The honest answer names a topic they used to teach and no longer do, and explains the principle behind the decision.
If you’re already evaluating engineering training vendors and any of these answers are starting to concern you on a shortlist, we’d be happy to walk through how we’d answer them ourselves. Thirty minutes, no follow-up calls, no pitch.
A quick reference for shortlist screening. If three or more of these show up in a vendor’s responses, it’s worth asking the harder version of the question.
We’d rather work with buyers who ask hard questions. Programmes built around that kind of scrutiny tend to deliver clearer outcomes for both sides, and we don’t end up six months into an engagement realising the client and the vendor had different definitions of success.
If any of these answers are the conversation you wish your shortlist would have, we’d be a good fit.
