This could be due to fig rust. The fungus Cerotelium fici causes fig rust, which attacks the leaves in late summer. Leaves that have been severely infected turn yellow-brown and drop. Numerous small, slightly raised, reddish brown spots will appear on the underside of the fallen leaves. These areas are frequently covered in a dusty golden-yellow swarm of rust spores.
Fig rust is usually not fatal, but repeated epidemics will weaken the plant. In any given year, heavy leaf drop from rust will reduce size and quality of the fruit.
Currently no fungicides are labelled for home garden figs so sanitation is important. Sanitation must be used to control fig rust. In the fall, please collect all infected leaves from the ground under the bushes and remove them from the area.
If you have any questions about growing figs in your backyard, please contact your county Extension agent.