That's pretty much my logic as well.
I just asked the parts manager who sold me my Aria this question a couple days ago, since it had come up here in another topic. He's been riding over 40 years, selling parts and gear (like helmets) for half that time. He shot it down. Maybe he knows what he's talking about, maybe he doesn't, but he has a lot to gain by confirming this. Namely, sale of another helmet to me every few years.
The shell getting brittle was more of an issue once upon a time, but the materials in most modern name brand helmets don't suffer from it, particularly the newer exotic materials like carbon fiber and kevlar weaves. Cars and airplanes are made out of this stuff, getting brittle sitting in a box on the shelf for a few years isn't likely.
The foam inside becomes compromised with use, not age. This is why throwing your Dunkin Donuts cup out the window is a crime- they sit there in the sun and rain for years and don't break down, ever. The stress of pulling it on and off will take it's toll eventually, but the odds of it ever affecting the outcome of an accident is slim.
The only thing you need to worry about wearing out is the liners and cheap pads, which will definitely lose their effectiveness with use, sometimes in as little as a year if you ride enough. Thankfully, in most decent helmets they are replaceable, and compared the the cost of a new helmet, quite cheap.
Synopsis was, if you bought a good helmet within the last few years, chances are you are going to start deteriorating long, long before it does.