…I feel like openssh has a much larger attack surface than a simple binary.

Right. This is just trading one set of security pitfalls with a second, much worse set of security pitfalls.