ok, big stones deliver more impact, but why sling them, if you can throw them as far without sling/staffsling? big stones deliver great impact, but it comes with bigger surface where it lands... it may give a big "klonk" on the shield but thats all, i think.
---Winkleried said that they didnt had armor in the sling-time, but i think they at least would have had a wooden shield. even the vikings had them. though a simple wooden shield can stop big stones, then it would give you a broken arm or something like that. but the smaller ones go through it, mate! I have cracked big branches form a tree with my sling...---
for the staffsling, you can use it as quarterstaff...
on the other hand small stones can be thrown very fast and with good range. if the enemy is closer than 50m, why would you try to sling a stone of half a kilo? when the time you have good wind-up they are holding their axes high in the sky to let it splice your head.
but then you would say; "what if your in a castle?" well quite simple, you empty the pot of boiling oil. Fry those critters
or (if you dont have oil, cause the lady made you some french fries yesterday) roll the stones from the tower or throw burning sticks