It's scary finding out what others think.

It should be to some degree
You should still do it of course
But if we ignore our fear and the actual possibilities of something being "beyond our grasp and yet malicious", it will strike at random, with random severity

You see someone, battle scarred, in a horrible condition despite still having his youth, not taking care of himself etc, your intuition whispers to you that you should stay away from his ideas, potent as they might be

Similarly, when we speak to people, we take the context of what we know of them already, to determine if there is anything worth digging in there, or not

If they exhibit an unkempt, unhealthy overall mental state, we may chose to distance from their true thoughts

The opposite may apply as well

I don't think this is inherently wrong

Sometimes scars spark wisdom tho, so it's not cut and dry always