(I'm guessing you are in the admin ...)
If you are viewing subfolders the folder icon appears. So go to a folder that has images in it, then you will see the thumbs The content class determines what your icon/thumb is
The template that deals with displaying the subitems of a node is admin/templates/children.tpl. And in there the template admin/templates/children_thunbnail.tpl is included if you selected the thumbnail view.
So have a look at these templates to solve your problem. It is working fine for me so I can't give you any tips where exactly to start.
You are right, it does seem to be using that template but instead of displaying the actual image it is display an icon that represents an image.
Maybe there is a settings file somewhere which says what icons are attached to what classes? Maybe such a file will also allows the actual image to be displayed instead?