Your innermost beliefs are at variance with your speech and behaviour, which is dictated by survival considerations.
As a result, you suffer from severe internal conflict which warps your personality.
Society does not care if you are warped or not, so long as you talk the talk and walk the walk.
The inviolable spiritual law says that you will be at peace only if your thoughts, speech and actions are all aligned.
Therefore, those who need to progress on the spiritual path need to distance themselves from society, family, and all such structures that being them into proximity with other people.
When you are young, a balance that is tilted more towards society and less towards spirituality seems more rewarding and is probably essential.
As you grow older, the balance has to shift. You need to tilt more and more towards spirituality. That is the logic behind the four "ashramas" in Indian philosophy, Brahmacharya / student life, Grihasta / fulfilment of worldly desires, Vanaprastha / giving back to society while being still in it, and Sanyasashrama / giving up everything and focusing on Moksha, rather than Kama and Artha. Underlying the whole thing is Dharma, which one must not forsake throughout one's life.
But the template of "modern life", with its emphasis on "achievement", "wealth", and "success" keeps us trapped in the grihasta stage like a person on a hedonistic treadmill who is unable to step off.
So, where lies the answer? How does one strike the right balance? That is an eternal question that needs to be constantly reinterpreted in line with shifting ages and times...