To some degree there is a slight speed loss, but unless you use a slow filesystem (windows / mac / nfs) or a lot of extension(~20) you won't really notice it much. Some of the speed loss is fixed in 4.1 though, by introducing a setting to turn of modified time checking on ini files.
Basically, when we aim at generic solutions, we try to isolate tool functionalities and this leads to many extensions. Otherwise, when we implement a one-time client-dedicated solution, this usually grows bigger within one extension folder.
We have some implementations that have 20+ extensions. We haven't noticed any significant problems at this point, but that also much depends on site optimization and cache-layer. Also, the more dynamic content you have, the more important this issue I would expect to be.
