That wouldn’t work I’m afraid, the headphone adaptor only supports mono sound input for a microphone on a TRRS jack (with stereo output in the other direction).
You have to use an interface of some kind, which is why the Go:Mixer looks like a really good solution, it’s just a bit fiddly and annoying with the levels. There aren’t many other options. The iRig Pro Duo is like twice the price. You might be able to use the camera connection kit with a cheaper USB interface - not sure I’m though, I have no experience of that.