In the minds of some, it is just an excuse to induce digital converts into buying a whole new line of lenses - with the magic word "Digital". There are a number or reasons why two separate and unequal lines of lenses are being made by Nikon, Canon and even third party makes such as Sigma.
The DX sensor on the Nikon dSLRs is much smaller than that of a 35mm frame - 23.7 x 15.7 mm vs 24 x 36mm. The DX line of lenses are optimized for this format. Price super-wides for 35mm and you will see that the extra coverage costs big bucks for lenses of equivalent speed. Much more economical to produce a highly corrected 17mm lens to cover the 28.4 circle of the DX sensor than to cover the 43.2 circle of the 35mm frame.
On the other side of the coin, DX - and all digital lenses - are also optimized for the angle of light striking the photosites in the sensor. Film will show a bit of drop-off in the corners as the angle of light increases with super wides. In fact filters with a centre gradient are available for the medium and large format SuperAngulons. The problem is much exacerbated with digital. The corner photosites function best when light-rays strike them perpendicular.
The requirements are substantially different from film. Finally, sensors are very shiny, and film is not. Kodak 14n owners had fantasies of using their full arsenal Nikon lenses without any cropping. Disaster - the sensors reflected light to the rear element of many lenses which then reflected back to the sensor, adding ghosts of lights and horrid smears from highlights. Where subjects had no bright highlights, the lenses just killed the contrast. Not all lenses, just some lenses. Lenses designed for digital are designed to avoid the problem by not having flat or near flat surfaces and by having deep anti-reflection coating.
The DX size is a whole new format - as different from 35mm as 35mm is from medium-format. Though there are clearly evolutionary links - the lines upon which dSLRs are built were long ago paid for by the 35mm cameras built on them - user demands require higher quality, and this means specialized lenses.