Nowadays the name Four Oaks can be applied to anywhere in the north of Sutton, and most of the north of Sutton lies in the Four Oaks Ward of the City. But in 1800 Four Oaks was a tiny settlement of only twenty houses, scattered around a village green, and many of those were humble cottages where poor farm labourers lived.

Four Oaks proper lay just south of the junction of Belwell Lane and Four Oaks Road. It was an old hamlet, first settled in the Middle Ages, and we can guess how it got its name. To the east were the fields belonging to the village of Hill (over sixty houses strung out along Hill Village Road), with Sutton Park to the west; there were commons to the north and Four Oaks Park to the south. This left very little farmland for the Four Oaks people, who had either to sell their labour or engage in trade. One successful tradesman was the builder Solomon Smith, who lived in the big house now known as The Larches rated at 1 shilling and fivepence halfpenny (7½p), but poor widow Nanny Fisher lived in a tiny shack rated at one farthing (one tenth of 1p).

It was not until the 1890s that the first big houses for wealthy commuters were built in Four Oaks, at the same time as the Four Oaks Park Estate began to be developed. When the railway line to Lichfield was built in 1884, the race meetings at the Four Oaks Park racecourse were attracting big crowds, so the nearby station had plenty of sidings to accommodate special trains. Although the station was located at a place called Ley Hill, it was named Four Oaks, and from then on the name gradually took over the whole district. Four Oaks used to be a tiny hamlet in Hill and Little Sutton Quarter, but now Hill and Little Sutton come under Four Oaks.