Is it 'selfish' to choose financial security over having a family/kids?

This is something I've been thinking about for a while. I am 21, and my goal is to become financially secure. I don't yearn for riches, (perfectly fine taking the subway and living in an apartment or modest house) but I'd like to be able to travel the world, eat out and pursue hobbies I did not get to pursue as a kid due to lack of money. I grew up in a lower middle cl family (2 brothers) and all my mom did was work(2-3 jobs) , struggled to make ends meet and I feel like she kind of lost her self. Whenever I tell people I don't want kids or even marriage they give me looks and say I'm selfish, even some of my closest friends have told me this. Honestly do you think it is wrong or selfish to value material/monetary wealth over a family?