Software Engineer
Posted 2 Oct 2026
Advertised as “Senior Software Engineer - Paris”
BranchFactor Summary
The original posting with the fluff stripped out
- Expected experience
- 5+ yrs
- Management level
- Individual contributor
- Employment
- Hybrid
- Contract
- Full-time
No salary listed for this role.
About H:
When we released Holo4 on 28 September, we published every trajectory behind its public benchmark scores at trajectories.hcompany.ai. For OSWorld that's 369 desktop tasks, each run 3 times, with the steps, tokens and time of every attempt. This role builds and runs the evaluation framework that produces runs like these.
H builds computer-use agents and the models behind them. Developers use them through a managed API, and our forward deployed engineers take them into enterprise workflows.
What this team owns
The evaluation framework: orchestration, runtimes and observability. Researchers and forward deployed engineers bring the benchmarks, across web apps, desktop applications and the command line. Your job is to make the framework that runs them reliable, fast and cheap, and to make adding a new one quick. Research uses the results to choose checkpoints and decide whether a model ships. Product and the forward deployed engineers use them to measure agents on customer workflows. It carries roughly 50 benchmarks now. That number should be between 100 and 200 soon, and the framework has to keep up.
What you'd be doing
-
Integration support for researchers and forward deployed engineers bringing in a benchmark, with a shorter path each time.
-
Setting the standard for how a benchmark enters the framework, and building the checks that enforce it.
-
Scheduling and observability, so cluster capacity isn't left idle while evaluation jobs queue.
-
Reproducible results across trials, so a release decision rests on numbers that hold.
-
Whatever stack a benchmark calls for. One week that's cluster tuning; the next it's a browser extension or desktop environments.
-
Time with customers, from single developers to large companies, to find out what they want measured, then automating it so the results flow back into our harnesses and models.
The first few months
By 3 months you'll have helped researchers or forward deployed engineers integrate 5 benchmarks, and started fixing what slows the framework down. By 6 months one part of it is yours, for example scaling the runs, observability, or a group of related benchmarks, and a release will have gone out on your numbers. By 12 months you'll know the design and trade-offs of the whole evaluation system, and be the person the rest of H asks about evaluations.