Tom Brady's dad reveals one big reason that factored into his son's surprising retirement decision