So a race of mine are born with a special magic that invades another person's dreams and is able to devour dreams with negative emotions tied to them (a nightmare essentially).
I'm trying to figure out how to make this balanced, so draw backs/special conditions, what would the more powerful side of this magic look like, and can it be countered and if so what counters it?
What's the benefit to the one who devours a dream? Do they have to use negative emotions?
Maybe you can make the difficulty be based on how lifelike the dream is and how long the "devouring" endures? Like maybe if you do the process for 5min, it's easier than doing it for 20? I'm guessing the most powerful user of this magic could devour a person's consiousness, or drive them insane at least. As long as this power is mostly negative. And perhaps everyone has it in them to counter the magic as long as they train themselves to be self aware in their dream in order to fight it?
They mostly eat nightmares for the magic enrichment (more nightmares eaten=stronger magic). I think how ill set it up is that at its basic level, this magic can devour simple/vague nightmares but have to ask the person for permision, and as they get stronger they'll be able to devour more types of dreams, be able to manipulate a person's mind, and maybe be able to forcibly wake/put someone to sleep.
Ah okay, well it sounds cool to me. I've never made a magic system before, so I don't really know how to help you (other than what I rambled about earlier). :P
Your race could be like a Succubus/Incubus race but with a twist. In our world and in your world you can have it as the same lore among the humans. They fear them for the same reasons and repel them, but in all truth humans fear what they don't understand and what the truth really is these are not really demons but guardian angels that have gotten a bad rap. They go around protecting people from demons that can be conjured or summoned from bad dreams.
How they got their bad rap could be up to you. Maybe someone f888ed up on assignment and someone or a village got killed by a summoned demon.
Maybe you could have creatures that counter them by creating those negative emotions inside people. I'm no expert on mythical creatures, but I think it could be ghouls? Or just demons or whatever.
Maybe have another race with opposite powers? Like, you have a race dedicated to protecting dreams/nightmares. I liked the idea Masterkey had, about the most powerful being able to devour a person's consciousness. So maybe this other race is about moderation, making sure that consciousness-devouring doesn't occur. I don't know how big of a part your nightmare-powered race plays in your story, but if you're looking for a plot, maybe a war of sorts between the two races over control/power of the world. This would create a sort of moderation vs extreme battle, instead of dark vs light or good vs evil.