In 1896, Adolph Sutro rebuilt the Cliff House from the ground up as a seven-story Victorian chateau, called by some “the Gingerbread Palace”, below his estate on the bluffs of Sutro Heights. This was the same year work began on the Sutro Baths in a small cove immediately north of the restaurant. The baths included six of the large indoor swimming pools, a museum, a skating rink and other pleasure grounds. Great throngs of San Franciscans arrived on steam trains, bicycles, carts and horse wagons on Sunday excursions. Sutro purchased some of the collection of stuffed animals, artwork, and historic items from Woodward’s Gardens to display at both the Cliff House and Sutro Baths.
The 1896 Cliff House survived the 1906 earthquake with little damage, but burned to the ground on the evening of September 7, 1907.