Both problems depends on configuration of your eZ publish installation. Hard to say, what exacly causing this problems, too less information. I ve tested sorting and everything works fine, also I can add new roles to custom user groups.
I just tested to create a custom user group and it works fine to assign roles to it. But it needs to be placed in the user class group, you cannot have the class under the content group.
