How random is a random playlist?

Does your music player's 'random' button actually do what it says?