The way of accessing data from this cache will be more streamlined in 4.1. See http://issues.ez.no/13068 and related commits in svn.
It might be a good idea to have this in the session, yes, although I guess the performance gain will be only small. Can you post an enhancement request for it in the issue tracker please?
I've just added an enhancement request: http://issues.ez.no/IssueView.php?Id=13157&
As far as performance gain, you might be right, but think of a situation, where I need that for ten independent extensions, because each one makes use of such conversion. It would be much better to have a uniform tool for that to rely on.