This building was located at 41a Conduit Road and was known locally as the Mok Villa - named after the family who owned it - or by its official name of 'Fairview'. At the time of filming though it was home to the Foreign Correspondents' Club (F.C.C) and its connections to this film go a bit further than just it being one of the locations used.
Throughout the film the house is used to represent what is described at one point as 'Victoria Hospital' where Dr Han (Jennifer Jones) works as a doctor. The terraced gardens are featured more than the actual house though (all interior scenes were filmed back in the US on a sound stage) and we are led to believe that the hospital backs onto the famous windy hilltop.
In reality though the hilltop was in fact located in California at the former Fox Ranch (now part of Malibu Creek State Park) and the hillside behind the house was actually only about a third of the way up Hong Kong's Peak.
The other connection to this story is that the ornamental pavilion seen above in the buildings grounds was the real life location where the real life Han Suyin met her lover, Ian Morrison, who was a journalist from Australia and no doubt was a member of the F.C.C.
This splendid building was knocked down in the mid-1960's and was eventually replaced by a development called Realty Gardens in 1971 which still stands there today. The current development also has a small ornate pavilion in the grounds (the terracing remains fairly unscathed but the ornaments you see above are all gone) which for a while some people mistakenly believed to be the same one seen above. Alas, it is a modern replacement that was built on a slightly higher level to the original one. There is a great summary of the building over on Richard Wong's FLICKR page and it's necessary to read all the comments: https://flic.kr/p/8jNYZx
The thing I find most staggering about these grabs are the fact that there was a completely open view of the harbour back then. This view has slowly disappeared as more and more high rises have filled up the hillside.