Whilst we're likely exiting in this case, clean up the loggers we add
so that in the case of certain server retries there is no possibility
multiple loggers stack up.
(Bitbake rev: 25b7bf6672be66bcbfe5760610dce7d3e866cdcc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>