IdP will remove membership to groups if the user no longer has the group in their group claims, which is good. However, it is also removing membership to groups that are created manually in the Users module. IdP should ensure that it is only removing membership from groups that it creates, perhaps by making sure the description is "IdP Auto Group Creation".