It didn't occur to me at the time to try this (on rpi5, pulling files off a remote machine):

$ scp dave@desktop:'scptest/a??w.{txt,dat}' .
dave@desktop's password: ...

This works. The quotes prevent there being two scp invocations with associated password prompts.