The word used for 'defiled' is also used for a seminal emmission in the course of normal sexual contact. Lev. 15:18 says,2. "Her former husband who sent her away is not allowed to take her back to be his wife after she has been defiled" (Deut 24:4). What defiled her? Notice that it was not the returning to her first husband that defiled her. Notice that it was joining herself to a second man that defiled her. Now, I ask you, men, if you truly loved the woman, why would you want to defile her? If you feel sorry for her because she is struggling to make her way after a divorce, then give her a job, but do not defile her.
18‘If a man lies with a woman so that there is a seminal emission, they shall both bathe in water and be unclean until evening.
A clarifying translation of Deut. 24:3 would read:
...her former husband who sent her away is not allowed to take her again to be his wife, since she has had sexual relations with the second husband; for that is an abomination before the LORD.
This prevents a husband from sharing his wife, effectively adultery.
What the law DOES NOT SAY is that she can't go to any man. It clearly limits her from returning to her first husband. Period.