Letting go of ex’s family may be harder than divorce