Nobody is saying Stallman can't say having "consensual" sex with a 12 year old doesn't harm the kid. But nobody who isn't actually brain damaged should defend it nor is anybody actually forced to associate with him because of "free speech". It doesn't violate any free speech value to say you can believe whatever despicable things you want, you can advocate for any despicable thing you want, but nobody else is obligated to support you or be allied with you. It's not cancel culture. It's being responsible for the ignorance he spews culture. 

As for how the FSF is damaged for it, look at how many organizations have pulled their support. Nobody is bigger than an organization unless the organization allows itself to be subsumed to a cult of personality. It rarely works out well. Part of the reason that the FSF is useless is because they have allowed themselves to be so beholden to a useless repugnant toad like Stallman who pushes a majority of people away. If you want an idea to grow, it helps not to be led by somebody who intentionally spews reprehensible nonsense that pushes everybody else away. Stallman is a toxic excuse for a human being. You want Floss to have a chance to grow? Don't let it be led by a guy who intentionally says things that are considered morally repugnant by the majority of the public.