When it comes to scarves, blankets, and even hats, sizing is pretty straightforward. But when you're ready for your first sweater, things get a little more complicated. You see, there are really two sets of sizing on lots of sweaters: S, M, L, etc. and then there are the actual measurements.
The first type is not very useful. It will tell you what range of sizes an item comes in (e.g., a sweater that comes in XS, S, M, L, XL, and XXL has a range of six sizes, while a sweater that comes in S, M, L only has a range of three sizes), which can give you some clues about fit and patterning but these sizes really shouldn't be used to determine which size you're going to make.
What gets confusing is that S, M, L, etc. are relative sizes, so you can have one sweater with a small that's got a finished measurement of 32" and another with a finished measurement of 46". All that means is that it is the "small" of the range of sizes offered for that particular sweater.
What's really important are the actual finished measurements, generally given as a chest or bust measurement (I'll talk more about this in a minute). Note that unlike sewing patterns, knit and crochet patterns give actual measurements and do not include ease. So if you want your sweater to be a big boxy cardigan, you probably want to choose one with a finished measurement 4-6" larger than your body measurements (positive ease). Looking for a figure-hugging glam-girl sweater? Choose a measurement that's 2-4" smaller than your body measurements (this is called negative ease).
When we talk about "bust" or "chest" measurement, that's because for many people, that's the part of the body with the largest measurement. If this is not true for you -- say your hips are wider than your bust and you're making a tunic length sweater -- you should consider that when choosing which size to make.
Remember that when you're making a sweater, you're going to be putting a lot of time and care into crafting your garment, and you want it to be just right. It's not like grabbing a sweater off the shelf...you want to carefully consider which size will be perfect for you (or the lucky recipient of your hard work!).