Can you fire a Governor General?

The controversy over Julie Payette's role as Governor General has raised questions about her fate in that position. But what is the process for a governor general to vacate that role? Are there precedents? And what role could Buckingham Palace or the Queen play in such a move? CBC Explains