As a general rule, I would suggest that if you aren’t dealing with noise, things aren’t sounding distorted, and your headphones can go loud enough, you probably don’t need a headphone amplifier.
If you want to get technical about it, the output impedance from your device should be a minimum of 1/8th that of your headphones - or ideally even lower than that - to achieve a proper damping factor.
Insufficient damping can lead to inconsistencies in the frequency response or distortion - particularly at the low end.
I find that you lose definition, and things can sound ‘bloated’ in the bass region without proper damping.
If the output impedance is not specified, it’s safe to assume that it is going to be high (probably above 50 Ω).
The headphone output from a lot of instruments, interfaces, mixers, or speaker amplifiers does tend to be quite poor - even if that device performs well using the line/speaker outputs.
The Digitakt/Digitone headphone output is specified at 55 Ω for example, which means that it’s really only suited for driving 440 Ω or higher impedance headphones (55×8, following the rule above).
In that case, even though it may get loud enough with low impedance headphones, running the output through an amp would be beneficial.
Modern headphone designs are typically below 100 Ω, so that’s going to apply to most of them these days - though there are still some older studio models in the 300–600 Ω range.
A lot of audio interfaces do not have the spec listed for their headphone output, but you may be able to find measurements online.
The YouTuber Julian Krause has measured a lot of smaller interfaces for example:
The output from a lot of them is quite disappointing - either having too high of an output impedance, or not having the power to drive many headphones properly.
.
Amplifiers themselves do not have to be expensive.
There has been massive competition in recent years, which has brought high-end performance down to significantly lower price points now.
I recommend checking the Audio Science Review forum for measurements and recommendations.
A couple of hundred bucks can now get you what would have been state-of-the-art performance, at ten times the cost, just a few years ago.
In the budget range, Topping tends to be one of the better performing Chinese brands, while Schiit is often recommended for people that want something from a US company. But there are lots of options.
My preference would be to spend a little more than the absolute minimum and get something with a balanced audio input (TRS/XLR) rather than unbalanced (RCA) but that may not be useful/necessary depending on the setup.
