>I see no abuse of power by said manager at the point of expressing an interest.

I don't think you understand the perspective of women then. I don't know if this comment alone will be able to change that, but there are so many dynamics involved that I'm guessing you are unaffected or less affected by, and are therefore failing to take into account when judging what this situation would be like if you were in it. Things like the female-male power imbalance, the manager-employee power imbalance, the fact that this happens orders of magnitude more often to women than to men, etc.

I would urge you to read more accounts of women's experiences with workplace sexual harrassment, and/or to speak with women you know about it with an open mind. Their average experience is so different from yours in this respect that it can be hard to understand why something that seems innocuous to you is absolutely toxic to them.