Because they failed to either enrich themselves through robbing the colonies, or develop the colonies, or bring the culture there (ok people in Cote d'Ivoire indeed, eat baguettes and quiches but it looks more like a sad joke than like France).

Compare to the British colonies and dominions: brought British culture and legal system (even to territories where local population always dominated), developed them economically, and yes, robbed them too, making good profit.

French system, for comparison, was an utter failure. Only benefit it provided was a good network of anchorages and coal bunkers for the French Navy.