There are a couple of differences to note.
When modulated slowly or by DC, PM will adjust phase while FM will adjust frequency.
Another difference is that when the modulation is removed, in PM the carrier will be back to its original phase while in FM the carrier will be in an arbitrary phase (which is related to how long you have been modulating).
Yet another difference happens when you use non-sine oscilaltors.
But when the operators are sine waves and the modulation is in the audible range there is little sonic difference.