Living in the Southern Suburbs vs. Hermanus/ - Any advice?

My wife and I are busy exploring the possibility of living (for a certain period) in Cape Town. We have made a shortlist and are very enthusiastic about the Deep South (Fish Hoek, Simon's Town, Noordhoek), but we have also looked at a place on the coast like Hermanus.

I find it difficult to determine which location would be most interesting, especially for our son (2 years old). Are there any members who have been in a similar situation and made a choice? Or members who live or have lived in one of these places? Any advice/experience regarding safety, activity, people etc would be very welcome.

Many thanks!