The error is very likely what is explained in the thread I linked above and only affects the USB audio compliant mode. Overbridge mode is not affected by this.
The issue, has nothing to do with PipeWire or the kernel itself. It is in the USB side of the devices because it has a not fully complaint behavior. Sadly, this can only be fixed by telling the USB driver to take that into account by setting some flags and, for that, recompiling the USB drivers included in the kernel by doing what is explained in this specific comment is required.
On top of that, the weird behavior causing the distorted audio only appears with some USB hosts, which made it even more difficult to diagnose.