You're right that if you store your GUSD off Gemini and it gets stolen, then you're out of luck. I am not sure what happens if Gemini closes but my impression is that since they are a NYS trust they would have to their customers the 1:1 backing (rather than paying creditors with that money).
So if there's an exploit where GUSD can be stolen/duplicated, or if Gemini defaults the insurance is meaningless.
What it means is that if a bank where Gemini holds an account defaults, the government will pay Gemini up to $250k.