Flux fumes can cause acute respiratory and eye irritation and possibly worse. Asthmatics are especially affected.
If a substance is an admitted health hazard for people on a production line, why should you want to breathe it at all - when it is simple enough to filter or redirect the fumes? It is still a hazardous substance.
It states right on the spool of solder I'm using that "flux fumes may cause pulmonary irritation or damage."
I've talked to a few people (hobbyists) who complained that after a lengthy soldering session they had a sore throat, cough, and burning nasal passages. Can't say for sure that there isn't any long-term damage being done as well. Why chance it?
I'm sure that just like smoking tobacco, some people can be exposed for decades and be fine and not worry about it. It doesn't
have to make you ill, but the potential is there.
Flux fumes may not exactly be the most harmful thing we can be exposed to, but neither are they totally harmless.
I could also see filtering or venting the fumes simply because other people in the house don't want to smell it.
https://www.electro-tech-online.com/custompdfs/2006/05/indg248-1.pdf