I have it in mind that humanoid NPCs do not "rage against the dying of the light" as much as they should. Combat encounters frequently result in a full-on massacre, when realistically a single fatality would instill an idea in everyone's minds the gravity of the situation, resulting in either a terrified retreat or an unconditional surrender. Anyone who's wounded should begin to favor the preservation of their own mortality over combat success. Obviously these rules do not apply when the NPCs find themselves forced against their wills.

Does anyone have experience running this style of campaign? How does a combat scene play out with these restrictions? How would experience be awarded? Can the game still be rewarding and fun?

Beating an encounter can include the enemies fleeing or surrendering.

Role playing NPCs or enemies where murder or death are not the only options is an effective way to mitigate the tendency toward murder-hobo player parties.

An enemy that surrenders and/or flees counts as defeated. Make that clear from the outset session 0. They party gets the same credit as if they had killed the creature. Additionally, it offers up the opportunity for more loot and role play.

Additional Benefit

Demanding the opponents doff their armor or give up their weapons in their still serviceable form as terms of surrender.

Persuading or intimidating the opponents to reveal information before turning them loose. Such as the location of a horde or a clue in a side quest that would have otherwise been harder to obtain.

Additional Role Play

Make note of opponents that were defeated and allowed to live or escape. Make the circumstances of that come back around. If the player characters were benevolent or malicious, that information will spread. Perhaps they encounter the relatives, friends, or allies of an enemy they allowed to surrender. The treatment of the previous encounter should affect the subsequent interactions.

Run Away! using the optional chase rules

Determine the conditions that a group of enemies or NPCs would disengage or flee prior to the encounter. Figuring out what a show of significant force would be to the NPCs is a reasonable staring point. E.g. "If their best fighter goes down, they run away." or "Demonstrating a command of the elements inspires awe in the creatures and they lower their weapons."

It is particularly impactful if the characters have discovered the conditions that can manipulate the situation previously in the story, and can then apply their knowledge to great effect. E.g. "They might have discovered that Krog the Krogdarian is immune to fire, but fears lightning." or that "Meghanna, ironically titled the 'merciful', is particularly keen on avoiding a fight if not confronted in public."

When running away, it is a different mechanic than a fighting withdrawal. The Dungeon Master's Guide includes an optional rule for chases that makes for a different flavor should the characters find the need to give chase:

Strict application of the movement rules can turn a potentially exciting chase into a dull, predictable affair. Faster creatures always catch up to slower ones, while creatures with the same speed never close the distance between each other. This set of rules can make chase more exciting by introducing random elements.

It's All in the Roleplay

I don't mean this in the sense of character acting, but in projecting yourself into the mindset of your NPCs and making decisions for them. What do they want, how far are they willing to go, what skills can they throw at a problem, etc. Answering these questions will help you immensely in designing and running more realistic encounters. If you know a lot of your NPCs will try to run away, you'll probably need to familiarize yourself with the chase rules (or make a system to adjudicate chases, as the default system is lacking in choices for your players to make).

In my experience, it is very difficult to run an entire campaign where all enemies are inexperienced cowards who will cut and run at the first sign of trouble. It's hard to feel heroic when the only ones you fight could probably have been taken out by a few of bored teenagers. This is particularly true after level 4, when your power level enables you to be of real use to kings and governors. You need to vary the tenseness and the pacing of your fights by also including more professional enemies, either as supplements to the usual cannon fodder or as the occasional brute squad who came to see where the boss' lackeys got to.

Soldiers vs Joe Farmer

One thing that comes to mind when reading your question is that the attitude you describe will most likely be found in those who do not have formal combat training. Part of the reason soldiers train is so that when everything hits the fan, they have effective preconditioned responses to fall back on. Another difference between soldiers and farmers is that soldiers have made an implicit promise to follow orders in pursuit of a goal. Even if following orders might result in death, they will follow those orders to give their comrades a chance to complete the objective; the farmer may not have such a promise, but depending on his motive for starting the fight, he may still be willing to go through with a fight to make sure his community gets money for food, etc. The third difference is that soldiers are implicitly part of a unit, and they will take risks to protect each other; a group of farmers may or may not have any such cohesion, and as such will be more than will to cut and run or betray their fellows. Another thing to consider is the temperament of the enemy they fight. Soldiers are unlikely to break ranks, knowing that doing so no only compromises the mission, but may also cause more casualties than would be given if they simply kept at it. Farmers probably lack this discipline; they might cut and run if they think the foe will overpower them, they might try to surrender and bargain for mercy, or they might throw their "friends" under the bus.

What's my motivation?

The first thing you should ask when designing any encounter is what the motives of all NPCs are, how are they likely to accomplish those goals given their temperament and capabilities, and what are they willing to risk. Are there any ways that they might accomplish those goals without resorting to a mutual murder extravaganza? It might not hurt to just ask nicely first. How will they react when something they're not willing to risk is threatened? Do they surrender and beg, cut and run, or fight even harder? If a fight does break out and things go south, what escape routes are available?

Dirty Rotten Trickery

In the real world, there is no such thing as a fair fight. The underdogs are going to need an Ace in the Hole, and they're going to use any tactics they can to push the odds in their favor. Doing it right can lead to very interesting situations (see Tucker's Kobolds). Outnumbered? Use surprise hit and run attacks. Know the route your enemy is going to take? Stage an ambush. See that big rock over there? Attack from the top of it so your enemy can't get to you easily. Roll barrels down the street, swing logs on ropes through the middle of them, set the hay under their feet on fire, have Fred grapple the wizard so he can't cast using somatic components while Igor makes a few holes in his belly; just be disruptive and smart.

What are Your Player's Goals

If you're tasked with cleaning out a nest of small, weak monsters, you're probably not going to think too much about an enemies efforts to run or surrender; you will have completed your task. On the other hand, if your job is to quell tensions in a city devastated by famine, there are very real human consequences to breaking out the murder stick. You need to think about ways your players might try to defuse and work with their would-be enemies.

Example - Goblins: The Consumate Cowards

In the default lore, goblins are opportunists and cowards. They pick fights with those they know are weaker than they are, and they often strike from ambush. When overpowered, they won't hesitate to surrender and show the PCs in the back door to save their own hide. However, they're also clever. That goblin you captured? He's lying through his teeth and you're about to walk into the main force. Those goblins you have on the run? Watch out for pit traps. And for goodness sake, when you trip over a vine, throw yourselves sideways onto the ground and pray that poisoned dart misses you.

It's more complicated than that, and fighting to the last can often be realistic.

First, I have to disagree with your assessment that "...realistically a single fatality would instill an idea in everyone's minds the gravity of the situation, resulting in either a terrified retreat or an unconditional surrender." In real life, surrender and retreat happen frequently, but so does fighting far past a single fatality.

Most simply, if the characters fighting are acting on blind instinct then irrational fighting is just as valid and real as irrational fleeing. But even with rational units, fighting to the last (or nearly so) is hardly unheard of.

The movie 300 is highly fictionalized and dramatized, but it is based on a real battle in which Greek Defenders (despite the movie, more than just Spartans stood to fight to the end) held a pass to buy time for the other Greek cities to organize a resistance to Xerxes. They held until almost all of the defenders were killed and then the handful of survivors finally surrendered when they could no longer even buy time.

The band Sabaton has a song, "Resist and Bite", that only slightly dramatizes the role the Chasseurs Ardennais played in the beginning of WWII. A small company of soldiers held out against a division because their last orders were to hold the border at all costs. The Chasseurs were finally forced to surrender because they ran out of ammunition.

When people have something worth fighting for, observing their comrades fall will not deter them. It may even anger and inspire them. There are numerous instances in history of military units fighting nearly to the last man when they believe their deaths serve a greater purpose.

The other thing is that rational individuals will consider whether surrender or flight are even options. If you are facing an enemy with long range options or greater mobility that will not accept a surrender and wishes to arrange a complete destruction of your group, then many people may prefer to go down fighting. Fighting may be the most rational option.

The real answer in the context of D&D of course is that in D&D you usually see complete destruction of one side or another because that makes for a better game. But that facet of the game is not particularly unrealistic within the context of the story. Military units may choose to fight to the last and creatures like Orcs may have it in their nature to fight to the last.

There are rules for retreat, and surrender can be roleplayed.

If you want to see more retreats and surrenders in your game and you are the GM, there are rules for retreat. I won't belabor them since they are in the books and GcL already summarized better than I could, but they are there.

Similarly, surrender can be handled through role-play if desired.