My first choice is a Teensy with an Audio Adaptor Board for Teensy.
The Teensy Audio Library is super versatile. You can play multiples sound files at once, you can do FFT spectrum analysis, synthesising, delays name it. And you have a Visual Audio System Design Tool. If you want four outputs you can stack two shields.
The board is 14US$ but you'll need a compatible Teensy (20US$ for the 3.2)
What not to love?
Sparkfun MP3 Player Shield is compatible with any Arduino IDE compatible and the documentation is good. It is possible to play sample of quality, as it can decode Ogg Vorbis, AAC, WMA...
The shield is 25US$
There is the Arduino MP3 shield board. This board can read high quality audio and has a EarSpeaker spatial processing, that I'd like to try. It is compatible with Uno and Mega, and because those two board are not compatible with the USB MIDI library, this is usually not my choice. There is less documentation too for that board.
The shield cost 15US$
This module is really small and as it is controlled via Serial, it would be compatible with most Arduino boards. The documentation says nothing about compression but says it supported sampling rates up to 48kHz at 24 bit DAC output, so it might be a good audio quality. There is documentation here. The module has a 3 watts amplifier for small speakers.
This module is 9 US$
You can do audio directly with a Due, using the Arduino Audio Library but this is only 12bits audio. That said, this project seams to have done better, so, it is worth a look.
A Due cost 37US$
Adafruit Wave Shield for Arduino Kit has a great documentation, but it is 22KHz 12bit and only for 328-based Arduinos, so that is Duemilanove or Uno, so, not compatible either with the boards that are compatible with the USB MIDI library.
22US$
There is a few other boards that can be found, but have not much documentation. They are usually really cheap, from 2US$ to 12US$
There is a USB MP3 Audio Sound Player Decoder Module, that looks interesting as everything is on the board, including a little amp.
There is the Serial MP3 player or Serial UART TTL MP3 Module, you can find a library for it here
Audio module or BY8301-16P MP3 Module has a libray here
You can make some sounds with any Arduino and a buzzer. Here is a few exemples:
A pseudo-Theremin with a buzzer by Learn Adafruit
An instructables for playing a buzzer
A basic player with a SD reader and another one
A button piano with buzzer
Still a buzzer, but more explanation about synthesis
A library to read low quality wav
Play Melody Library