EXR was created by Industrial Light and Magic for use with the movie work they do. http://www.openexr.com/index.html
RichardAnnema at Splutterfish gives a good explanation what makes it better.
HDR is a format which allows storage of high dynamic range values in the form of RGBe pairs, compressed through the RLE method, and with a small reserved header.
What this means is that HDR's color fidelity is poor, as the red, green and blue channels all share the same exponent.
OpenEXR, in contrast, stores values in either half or full float per channel. Means it takes up more space if left as the raw data, but its color fidelity is much higher.
Technically its range is lower, but you would be hard-pressed to reach the top of the range.
It also means that HDR doesn't store an Alpha channel. That's pretty major if you want to composite anything.
OpenEXR does store Alpha. And G-Buffers. And any other channel you would like.
And with HDR you can only store so much information about the image in the header. OpenEXR, on the other hand, allows you to store limitless (or a lot, at least) amounts of information and such in string fields.
It's also backed by a lot more companies and products. HDR is fairly niche, whilst OpenEXR is gaining new grounds quickly - discreet added OpenEXR support in Combustion 3, for example - but I don't think it support HDR at all.
There's loads more advantages, but those are the major ones anyway.