Matt,
Fast and small to me speaks to handling speed as well as lens speed and size.
Myself, I do not find moving from 35 to 50 enough of a change to make it worth having one. With an M6, you might look to a Cosina Voigtlander 75mm f2.5. With the right adapter, you have the right frame line, a small, sharp lens that will not cost you an arm and a leg.
I just up to 90mm or 85mm myself, but then I am a frugal speed freak who loves f2 lenses. 2.5 is a bit slow for me, but then the size is 2/3 of the stuff I carry. The 75 fits in a pocket, much like a 90mm f4 (old or C).
As you seem to like wide stuff (at least in MF), you might look at the 21mm or even the 15mm. You are into external viewfinder land with them, which might be an issue for you, don’t know your style.
Take a look at
, Cosina Voigtlander has some very good lenses these days, not quite as expensive as Leica and every bit as good as a generation ago no asp stuff.
Hopes this helps.
B2 (;->