It is soooo worth it!
I went from Fez to Merzouga and then to Marrakesh, but the with the bus system over there you can usually get anywhere on an overnighter.
Use the company CTM http://www.ctm.co.ma/
They will be able to take you to Rissani, from there you have to get transport
Now here's the tricky part, you could organise with the place you are staying to pick you up from Rissani.
Or brave the chaos of all the taxi drivers in Rissani who will end up taking you to their friends hostel despite your polite pleas and will make you stick around while they do the big sell to you. Not fun! but quite an experience when you've had only a couple hours of sleep.
There will even be people on the bus offering lifts and the bus driver stops in Erfoud (which isn't an oficial stop) just to meet up with these "taxis" So just be aware of that.
When you get there you really only need to spend one or two nights. The main thing you want to do is take a camel ride out to the desert campsite. They serve you beautiful food and play music for you, fantastic stuff!
You can sleep under the stars or in the tents and the next morning you can climb the dunes to watch the sun rise!
The place I stayed at is fantastic. Auburge du Sud. The owner is Hassan and is a legend, he ended up taking us on a drive for three days visiting all sorts of amazing places along the way to Marrakesh, if you have time DO IT!
Let me know if you need anymore info!
Cheers