This is a subject I've also battled with. I did the college thing and also took a semester abroad in Australia, which was a great experience. However, it was also something that made me want to travel even more! So after graduating with a degree in finance and having too much of an itch to sit behind a desk crunching numbers, I saved up and bought a one way to Hawaii to work in the scuba industry.
A lot of people gave me funny looks and said I was wasting my time, but I'm much happier now. All I did was follow my gut. Of course if you want something bad enough you'll have to sacrifice, work hard, and take the leap.
I was a bartender in an upscale lounge for a few years in college and I asked a lot of folks what they would do if they were my age again (23). They typically gave me a combination of three answers:
1. Don't be in a rush to get married.
2. Enjoy your freedom by traveling and trying new things.
3. Follow your gut and try the things you dream about - even if it doesn't work out you have 30-40 more years to work your 9-5 and hate your job.
From a Risk versus Reward perspective I think this is a no brainer, it does, however, depend a lot on how you spend your money.
Why? Consider the typical college grad - making 30k-40k. They'll end up spending most of it and having a lot of "stuff." They'll have a nice new car, great furniture, a mortgage, and a debt. If you prefer travel, simply avoid these things, and you'll have less debt (or none if you work your butt off in college), better memories, and no regrets. You'll also be a more rounded person who has more confidence and more to offer, even in the workplace. Tell your future employer that you safely navigated vietnam and used proper financial management to execute the trip. How is that not impressive? Show them that you had to work ethic to put in long hours to save up for such a trip and the courage to follow through. Demonstrating that you took the initiative and can think for yourself will help your job prospects (unless you want a mindless job).
Most people who say they want to travel young either will do it right away, or will get caught in the rat race and will be unable to get off the hamster wheel because they've accumulated three weeks vacation and have health insurance. "Safe" gets comfortable. This is also true with college, so get that degree before you get too comfortable to make the sacrifice needed to achieve your bachelors.
Do this before you regret not doing it. A great quote from fight club:
"You're not how much money you've got in the bank. You're not your job."
Experience is who you are, realize that and become who you want to be, not just a piece of paper.
Best of luck and hopefully you make a wise decision.