If you understand, what can be done with a pair of FM operators, I would say that you have already understood the most important part.
Even a pair of operators can create an overwhelming number of very different sounds. Most of those patches are very sensitive to small variations of the modulation and with a well thought-out modulation of pitch and timbre such a pair can sound like two or more different instruments playing together … not layered, of course.
Algorithms are making things only more complex and give us more options.
My approach to understand algorithms is to search for clusters, which make sense to me. This would be pairs, triplets, chains of two, chains of three, how many independent clusters I have, if there are any links between clusters, which I would like to use or not. Often I use the clusters as single voices, or ignore them at all.
We should consider that if even a pair of operators can generate very complex sounds, it is not making much sense to combine as many operators as possible in a chain. From my experience I can tell that a third operator added to a pair of operators can be very good used to add some grit or an other variation to the patch, but often enough only, if applied in very small dose.
My personal rules of thumb are:
- even two operators generate very complex sounds using very simple waveforms like sine or triangle
- a third instance of modulators creates even more complexity
- the more complexity we add, the more we enter the territory of chaos or noise