This isn't very well thought out, but....
Generally, I dislike weed smokers. However, I find the ones I meet through climbing to be mostly good, likeable people.
I do have a big problem with people smoking weed in front of the kids. Cigarettes too, but more so with weed. I work in substance misuse services and meet a lot of young adults whose lives have been seriously interrupted by becoming habitual weed smokers whilst still at school. Anything I can do to reduce the chance of my kids going the same way will be pursued with some vigour - including stopping people smoking weed at the crag when I'm out with the kids.
Most of my less-than-healthy substance use came about initially through climbing. I'm not implying that climbing is a "gateway drug", but if I'd taken up these habits at the age of 14 rather than 17 then my weak will and social immaturity might well have led me into a boat load of trouble. There's no predicting which older kids/adults a child will latch on to as a role model, so in my mind everyone at the crag has some responsibility for the children around them. Hopefully my kids will be better prepared to deal with the real world, but I'd still like to reduce their exposure to illegal drug use.
I appreciate that the presence of children at the crag will annoy many people.
As I say, this isn't a well thought out post, but you get the idea