From a budget point of view, I have ~£500 to spend (excluding ropes and QD's as already have this covered)
I used to do a lot of track days, but since climbing I haven't done a single one, prefering to spend nice weekends out on the rock. As a result I recently sold the seats and harnesses in order to pay for this gear.
Cheaper items such as slings etc I will add to my Birthday and Christmas list.
Id stick with one set and spend your $$ on a couple of friends/cams. But if you have the $$ then I'd get a set of each. Rocks/Walnuts sit differently in different cracks (as previously discussed).
Or get a set of nuts and some 0, 00 and 000 walnuts (as there are no small rocks below a 1)...