In 1858, 20,000 people came through Victoria from California on their way to the gold rush on the mainland. New businesses sprang up, and many foreigners stayed and settled in the region. For the first time, local First Nations did not comprise the majority of the population.

Many northern aboriginal people visited or moved to Victoria for commercial purposes, their camps stretching along the Inner Harbour. For example, in April 1859, there were 2,835 First Nations people camped near the city. Of these, approximately 600 were Songhees. The rest included Haida (405 people), Tsimshian (574), Stikine River Tlingit (223), Duncan Cowichan (111), Heiltsuk (126), Pacheedaht (62) and (44). Victoria was also a commercial centre for large numbers of First Nations people from Washington State.

The smallpox virus arrived in Victoria in 1862, carried by a man on a steamer from San Francisco. Most of the Songhees had been vaccinated, but the disease was devastating for the First Peoples visiting from the north coast. Police commissioner Joseph Pemberton ordered the removal of all aboriginal people in Victoria, except for those "employed by whites," and people returning home took the disease with them, along the coast and up the rivers to the interior, causing annihilation in village after village. Eventually, smaller numbers of northerners returned to Victoria, but the extent of their influence on the local society and economy was never the same.