I go to NYC once a year for conferences and enjoy it. Last year I took my family and added a week to my schedule to show them the sights.
I'd suggest at least 7 days if you are going to try to squeeze the museums in. They can take a whole day and you don't want to feel rushed. That being said, 4-5 days is good if you don't go into too many of the museums. The subway can get you to most places quickly and at a low cost. Plan your days by sections of the city and it will be a lot easier. Also empire state building will take a couple of hours (up to 4) unless you buy the front of line pass, well worth it in my opinion.
I'd add Top of the Rock at Rockefeller Center as the view is much better than the Empire State Building view. Also, less crowded. I went in the morning and no wait!!! Just some suggestions.
Also don't miss the hidden parts of Chinatown where shopkeepers take you to show you their fake Coach and Gucci handbags and watches. Chinatown and Little Italy can be done quickly and together.