So, the antagonist kind of spilled over two parts because I got caught up with villains and motivation in part 1 and never actually got around to the part about why we need antagonists in a story.
Why do we need antagonists?
In real life, we really don't need or want antagonists in our life. These people have a tendency to make life that little bit more frustrating and difficult than it needs to be. Ultimately we avoid these people or find ways to minimise our involvement with them. If writing were actually like real life our protagonist would find a way to avoid antagonistic characters fairly early on and then we would have one incredibly boring story.
This gives us our first answer as to why we need antagonists in our writing. Without conflict or tension there really isn't a lot to keep the reader interested. People don't want to read the book about the girl who goes to live with her aunt and they get along just fine and she goes to a new school where she makes friends and lives happily ever-after. They want to know about the tension between the characters and the set backs, which will hopefully be ultimately overcome but will be interesting in the meantime. By having antagonistic characters (whether they are villains or not) you can generate a lot more tension.
However, antagonists can serve other purposes as well.
Sometimes they allow you to offer up alternative suggestions or solutions to a problem that your protagonist would never consider. They allow the writer to explore the morality of an action from a point of view that might not be available if all of their characters are sugar and spice and all things nice. If a problem is completely blocking your protagonist, sometimes it takes an antagonistic character to find a way through.
They can help your protagonist to grow. By putting your protagonist through minor conflicts with the antagonist, you have the opportunity to help your protagonist grow and change. The antagonist acts as a catalyst for a development in the character that you have wanted to include but haven't really found a reason for them to grow in that direction.
As with villains, antagonists can be highly entertaining. These characters are generally not bound by the need to act in a likeable manner and can get away with saying and doing things that might not go down so well if your protagonist (or other heroic characters) did them.
They add human drama. This one isn't as important when the entire story is about interaction between characters, but if your ultimate complication is an asteroid colliding with the earth or some other natural disaster, you need antagonistic characters in order to create the human element. Imagine a story plot where scientists discover an asteroid coming toward the earth and they tell the government who tell the world and everybody listens gravely and nod their heads. No panic. No idiocy or looting. The governments then all join forces, organise a crack team of astronauts to fly up and blow the asteroid up. No arguments about costs or wanting more of this nations people involved or anything like that. Just, this is the team and off they go. The space ship goes up the astronauts land, they plant their bomb and they fly away. They all get along the entire time and the only dialogue are the congratulations they offer each other along the way. Not particularly entertaining.
I would love to know what role the antagonist plays in your writing and your views on great antagonists in writing.