I understand this is mostly handled by the commissioner. Suggestion is if the team has an illegal roster for x time (Days), the system will randomly (Or sudo randomly) drop a player from their roster.
Thought of this idea as mostly a joke, but posted it cause why not.
There is a Wishlist item for keeping a running count of time teams spend as illegal. I wish there was more info available to commissioners regarding illegal states since it’s not always obvious with players moving on or off the IL.
Total time spent with invalid roster - Wishlist - The Ottoneu Fantasy Sports Community
It could be the last players acquired that would drop (or activated from 60-day IL).