Depends on your medium, but mainly I think it has to do with what fashion designers call a Color Story.
*
Medium first. I work with traditional media (watercolor), and there's this thing called a Limited Palette in which—as a fun challenge, and practical—you're only allowed three or four primary colors of paint, so you're limited by the colors that you can mix.
If the colors you choose to limit yourself to are lemon yellow, magenta, and cyan turquoise…then, the colors you can mix with those paints are still going to be candy-colored or neon. You can mix grays and browns with more difficulty, but it's precisely that difficulty that's going to make "candy neon" more tempting to stick with. You sure can mix realistic human complexions with this palette—but "fauvist" interpretations are just going to be easier (sensory stimulation warning if you decide to look up "fauvism" in art.)
If you go for something like the Velasquez Limited Palette, then the colors selection becomes more like: yellow ochre (which is a very earthy muddy yellow), burnt sienna (which is a reddish brown rather than a red), and ultramarine blue. Burnt sienna paint and ultramarine blue paint don't mix into a purple, even if sienna is technically red and ultramarine is obviously a blue. They're the wrong type of red and blue to mix purple, but the right type of red and blue to mix a gray. So, this becomes a suitable paint palette for designing architecture because there's so much brown and gray in the mixes…but maybe not so much for capturing sunsets in a tropical island paradise where the sky glows pink and purple. It'll be interpreted as moody with the Velasquez palette, whereas the yellow/magenta/cyan palette would also be an interpretation but that might go without notice if the colors are more true to life.
With digital, I have seen pixel artists set up a digital palette to pick colors from…but I myself am spoiled by layers and brush transparency settings, so the favored colours mix in that way, and then I can throw a filter over it or mess with the color saturation until it matches my mood or aesthetics.
*
Now for the Color Story. I think that a color story can be the natural result of a limited palette, but that you can also formulate a palette by thinking of the color story first.
What do you want everything to look like, and why?
Examples can be:
A.) Everything looks like candy. Vibrant, vivid, bright, youthful, modern, and at least a little loud to the eye.
B.) Everything looks like gemstones and precious metals or luxurious fabrics. There's still a lushness or saturation to all the colors, but it looks mature and also luxurious.
C.) Everything looks like pastels. It's option A plus marshmallow fluff to make everything soft and subtle and sweet or twee, less dynamic or assertive than A and more peaceful.
D.) Everything looks like a Zach Snyder or Wes Andersen movie where there's yellow filters over everything and it looks like a cat peed on the film. There's something retro or archaic in mood conveyed by the orange glow.
E.) Everything looks like later seasons of Game of Thrones, or a Shayamalan or Burton movie where there's usually a blue filter over everything. It's either always overcast outdoors, or it's always that time of the day that drivers don't know whether to turn their headlights on yet or keep them off. Everything is mournful and yet romantic, that's the mood conveyed by the blue filter.
So, when you notice the effect that color palettes have, then you can decide the mood that you want and work backwards.
Some more color theory that I personally use:
- light/dark value
- saturation
- temperature
Neon colors are different from pastels, even though they're both lighter in color value, because pastels are less saturated whereas neon colors are more saturated. Jewel tones often middle in dark/light value, but they're very saturated. Desaturated jewel tones create an "earthy" or "moody" color palette. Orange filters warm everything up, and blue filters cool everything down.
The other thing to be aware of is that color temperature makes it so that there are functionally two main kinds of red, two main kinds of blue, two main kinds of yellow—etcetera. Let's say that magenta is a kind of red (whereas let's say that true red is scarlet), and cyan is a kind of blue (whereas let's say that true blue is azure), and yellow can be either a lemon yellow or a yellow orange.
So if I sit down to make a color story, I'm thinking in terns of:
What sort of magenta is there going to be?
What sort of scarlet is there going to be?
What sort of lemon yellow is there going to be?
What sort of yellow orange is there going to be?
What sort of cyan is there going to be?
What sort of azure is there going to be?
What sort of greens?
What sort of oranges?
What sort of violets?
And maybe I'll decide that I like:
- terracotta pink
- mahogany red
- ecru almost-yellow
- ochre yellow
- woad teal
- midnight blue
- moss and olive greens
- metallic gold and brass/bronze
- plum purple and lavender
All of those together.
If there's a bright or floral magenta, lemon, tangerine, aqua blue-green, mint/lime green, or amethyst violet in whatever I'm illustrating that is supposed to have the above color story…then there had better be a good reason for it that I've argued with myself about and won because otherwise it would look "off", and making too many exceptions would lead to everything straying from the color story.
I hope this helps!