Hi @Greg Whitten ,
For the past versions of the IdP component, the component includes a dependency called IdP Customizations.
This component exposes 2 service actions that will be used by the IdP component if you enable the respective configuration like on the image below.

This component allows you to add and adjust what to do during the user mapping and group mapping that happens during the login flow used by the IdP component.
This component was created to add your custom logic without the need to change the IdP Component making it easier to upgrade to a new version without having to replicate any custom logic. This also allows you to adjust the behavior to your needs without introducing breaking changes to the rest of the community that is already used to the current behavior.
The IdP Customizations component will not be receiving new versions so it's safe for you to add your custom logic ( in your case, adjust how the group mapping works in order not to remove users that were added to "Manual" groups).
You can start by looking at the included "Default_Groups_Map" and "Default_User_Check" that mirror the current IdP behavior and then adjust it to your needs.
Make sure you use your own customized code that matches the corresponding "App Config" passed over by IdP

I hope this information will help you.