Can I remove things my neighbour has attached to my fence?

If you own the fence, your neighbour cannot attach anything to it without your permission - but can you remove the attachments yourself?