There's no way to prevent it as some players simply decide not to re-sign and then don't lower their asking price so they don't end up signing elsewhere, which is a flaw in the Dynasty AI. If you're playing the PC version, you can overcome the problem by moving them back to their previous teams through
DBF editing. Unfortunately, there's no way around it on the console versions.