It's because brake fluid is Hygroscopic (not a spelling error). It absorbs moisture out of the air, and over time this is a bad thing.
In extreme cases, it will cause internal corrosion to expensive parts such as the master cylinder, ABS pump, and brake calipers.
In the worst possible case, it can cause you to wreck. How? Because the water content gets to such a high percentage that in an emergency stop the hot brake caliper causes the water to boil. Brake fluid is incompressible (which is why we use it), but steam compresses quite well. Which means you no longer have as much braking force as you'd like, causing you to run into the vehicle or object in front of you.
So you want to change the fluid at least every 3 years, no matter how many miles you put on your vehicle. If you live on the coast, every 2 years wouldn't be a terrible thing to do.
This info also applies to motorcycles and mountain bikes, BTW. Any vehicle that uses DOT brake fluid.