It’s tolerable, and only occasionally loading samples onto the Tracker is what I do as well, but if Polyend could reliably do USB storage then I could cut new samples from the Line In or the FM radio of the Tracker and then copy them directly over to the Play (or the M:S, or similar), and I’d be updating samples a lot more often without worrying about bulk updates, and the Tracker would be a much more helpful device to have around.
There are conveniences that make a particular workflow easier, or faster, and then there are conveniences that are so powerful that they can change your workflow entirely, and you start doing things that you just wouldn’t do before.