Yeah, it takes a few seconds on the screen, which adds up over time. I talked to somebody at Google and they said that man years are wasted each day by people waiting for Gmail to load. That's why Google is developing Google Gears, to cache all of the scripts/etc needed to get Gmail running. The problem is, at least for Safari users, is that Gears is still really buggy.
