People often look at this kind of question as being an either/or decision. Yet the example of Tabithag above shows that in fact it is possible to do both.
I figure everyone has until they are around 35 to do as they please provided they get an education and start on a career path. Taking some time out can be done without having to abandon that career path.
If you are 45 and asking this question, my advice would be very different than if you are 25 and asking it.