Xjyuk

Data dependency

It's fairly easy to add more instructions per clock by making a chip "wider" - this has been the "SIMD" approach. The problem is that this doesn't help most use cases.

There are roughly two types of workload, independent and dependent. An example of an independent workload might be "given two sequences of numbers A1, A2, A3... and B1, B2,... etc, calculate (A1+B1) and (A2+B2) etc." This kind of workload is seen in computer graphics, audio processing, machine learning, and so on. Quite a lot of this has been given to GPUs, which are designed especially to handle it.

A dependent workload might be "Given A, add 5 to it and look that up in a table. Take the result and add 16 to it. Look that up in a different table."

The advantage of the independent workload is that it can be split into lots of different parts, so more transistors helps with that. For dependent workloads, this doesn't help at all - more transistors can only make it slower. If you have to get a value from memory, that's a disaster for speed. A signal has to be sent out across the motherboard, travelling sub-lightspeed, the DRAM has to charge up a row and wait for the result, then send it all the way back. This takes tens of nanoseconds. Then, having done a simple calculation, you have to send off for the next one.

Power management

Spare cores are turned off most of the time. In fact, on quite a lot of processors, you can't run all the cores all of the time without the thing catching fire, so the system will turn them off or downclock them for you.

Rewriting the software is the only way forwards

The hardware can't automatically convert dependent workloads into independent workloads. Neither can software. But a programmer who's prepared to redesign their system to take advantage of lots of cores just might.

In addition to the other answers, there is another element: chip yields. A modern processor has several billion transistors in them, each and every one of those transistors have to work perfectly in order for the whole chip to function properly.

By making multi-core processors, you can cleanly partition groups of transistors. If a defect exists in one of the cores, you can disable that core, and sell the chip at a reduced price according to the number of functioning cores. Likewise, you can also assemble systems out of validated components as in a SMP system.

For virtually every CPU you buy, it started life being made to be a top-end premium model for that processor line. What you end up with, depends on what portions of that chip are working incorrectly and disabled. Intel doesn't make any i3 processors: they are all defective i7, with all the features that separate the product lines disabled because they failed testing. However, the portions that are still working are still useful and can be sold for much cheaper. Anything worse becomes keychain trinkets.

And defects are not uncommon. Perfectly creating those billions of transistors is not an easy task. If you have no opportunities to selectively use portions of a given chip, the price of the result is going to go up, real fast.

With just a single über processor, manufacturing is all or nothing, resulting in a much more wasteful process. For some devices, like image sensors for scientific or military purposes, where you need a huge sensor and it all has to work, the costs of those devices are so enormous only state-level budgets can afford them.

Wow, I can tell this is being asked by someone under 30!

Going back in time, processors weren't able to run that fast. As a result, if you wanted to do more processing then you needed more processors. This could be with a maths coprocessor, or it could simply be with more of the same processor. The best example of this is the Inmos Transputer from the 80s, which was specifically designed for massively parallel processing with multiple processors plugged together. The whole concept hinged on the assumption that there was no better way to increase processing power than to add processors.

Trouble is, that assumption was temporarily incorrect. You can also get more processing power by making one processor do more calculations. Intel and AMD found ways to push clock speeds ever higher, and as you say, it's way easier to keep everything on one processor. The result was that until the mid 2000s, the fast single-core processor owned the market. Inmos died a death in the early 90s, and all their experience died with them.

The good times had to end though. Once clock speeds got up to GHz there really wasn't scope for going further. And back we went to multiple cores again. If you genuinely can't get faster, more cores is the answer. As you say though, it isn't always easy to use those cores effectively. We're a lot better these days, but we're still some way off making it as easy as the Transputer did.

Of course there are other options for improvement as well - you could be more efficient instead. SIMD and similar instruction sets get more processing done for the same number of clock ticks. DDR gets your data into and out of the processor faster. It all helps. But when it comes to processing, we're back to the 80s and multiple cores again.

You point out that a lot software doesn't use more than (x) cores. But this is entirely a limitation placed by the designers of that software. Home PCs having multiple cores is still new(ish) and designing multi-threaded software is also more difficult with traditional APIs and languages.

Your PC is also not just running that 1 program. It is doing a whole bunch of other things that can be put onto less active cores so your primary software isn't getting interrupted by them as much.

It's not currently possible to just increase the speed of a single core to match the throughput of 8 cores. More speed is likely going to have to come from new architecture.

As more cores are commonly available and APIs are designed with that assumption, programmers will start commonly using more cores. Efforts to make multi-threaded designs easier to make are on going. If you asked this question in a few years you would probably being saying "My games only commonly use 32 cores, so why does my CPU have 256?".

Good question, or at least one with an interesting answer. Summary:

• The cost of multiple cores scale close to linearly


• The cost of widening the pipeline scales quadratically


• Serious diminishing IPC returns from just widening the pipeline beyond 3 or 4-wide, even with out-of-order execution to find the ILP. Branch misses and cache misses are hard.

Costs are in die-area, (manufacturing cost) and/or power (which indirectly limits frequency).

Donald Knuth said in a 2008 interview

I might as well flame a bit about my personal unhappiness with the current trend toward multicore architecture. To me, it looks more or less like the hardware designers have run out of ideas, and that they’re trying to pass the blame for the future demise of Moore’s Law to the software writers by giving us machines that work faster only on a few key benchmarks!

Yes, if we could have miracle single-core CPUs with 8x the throughput on real programs, we'd probably still be using them. Maybe dual core to reduce context-switch costs when multiple programs are running; pre-emptive multitasking interrupting the massive out-of-order machinery such a CPU would require would probably hurt even more than it does now.

Or physically it would be single core (simple cache hierarchy) but support SMT (e.g. Intel's HyperThreading) so software could use it as 8 logical cores that dynamically compete for throughput resources. Or when only 1 thread is running / not stalled, it would get the full benefit.

So you'd use multiple threads when that was actually easier/natural (e.g. separate processes running at once), or for easily-parallelized problems with dependency chains that would prevent maxing out the IPC of this beast.

But unfortunately it's wishful thinking on Knuth's part that multi-core CPUs will ever stop being a thing at this point.

I think if they made a 1 core equivalent of an 8 core CPU, that one core would have 800% increase in IPC so you would get the full performance in all programs, not just those that are optimized for multiple cores.

Yes, that's true. If it was possible to build such a CPU at all, it would be very amazing. But I think it's literally impossible on the same semiconductor manufacturing process (i.e. same quality / efficiency of transistors). It's certainly not possible with the same power budget and die area as an 8-core CPU, even though you'd save on logic to glue cores together, and wouldn't need as much space for per-core private caches.

Even if you allow frequency increases (since the real criterion is work per second, not work per clock), making even a 2x faster CPU would be a huge challenge.

If it were possible at anywhere near the same power and die-area budget (thus manufacturing cost) to build such a CPU, yes CPU vendors would already be building them that way.

See the More Cores or Wider Cores? section of in Modern Microprocessors
A 90-Minute Guide! for the necessary background to understand this answer; it starts simple with how in-order pipelined CPUs work, then superscalar. Then explains how we hit the power-wall right around the P4 era, leading to a fundamental shift in scaling with smaller transistors, and going for higher ILP instead of frequency.

Making a pipeline wider (max instructions per clock) typically scales in cost as width-squared. That cost is measured in die area and/or power, for wider parallel dependency checking (hazard detection), and a wider out-of-order scheduler to find ready instructions to run. And more read / write ports on your register file and cache if you want to run instructions other than nop. Especially if you have 3-input instructions like FMA or add-with-carry (2 registers + flags).

There are also diminishing IPC returns for making CPUs wider; most workloads have limited small-scale / short-range ILP (Instruction-Level Parallelism) for CPUs to exploit, so making the core wider doesn't increase IPC (instructions per clock) if IPC is already limited to less than the width of the core by dependency chains, branch misses, cache misses, or other stalls. Sure you'd get a speedup in some unrolled loops with independent iterations, but that's not what most code spends most of its time doing. Compare/branch instructions make up 20% of the instruction mix in "typical" code, IIRC. (I think I've read numbers from 15 to 25% for various data sets.)

Also, a cache miss that stalls all dependent instructions (and then everything once ROB capacity is reached) costs more for a wider CPU. (The opportunity cost of leaving more execution units idle; more potential work not getting done.) Or a branch miss similarly causes a bubble.

To get 8x the IPC, we'd need at least an 8x improvement in branch-prediction accuracy and in cache hit rates. But cache hit rates don't scale well with cache capacity past a certain point for most workloads. And HW prefetching is smart, but can't be that smart. And at 8x the IPC, the branch predictors need to produce 8x as many predictions per cycle as well as having them be more accurate.

Current techniques for building out-of-order execution CPUs can only find ILP over short ranges. For example, Skylake's ROB size is 224 fused-domain uops, scheduler for non-executed uops is 97 unfused-domain. See Understanding the impact of lfence on a loop with two long dependency chains, for increasing lengths for a case where scheduler size is the limiting factor in extracting ILP from 2 long chains of instructions, if they get too long. And/or see this more general and introductory answer).

So finding ILP between two separate long loops is not something we can do with hardware. Dynamic binary-recompilation for loop fusion could be possible in some cases, but hard and not something CPUs can really do unless they go the Transmeta Crusoe route. (x86 emulation layer on top of a different internal ISA; in that case VLIW). But standard modern x86 designs with uop caches and powerful decoders aren't easy to beat for most code.

And outside of x86, all ISAs still in use are relatively easy to decode, so there's no motivation for dynamic-recompilation other than long-distance optimizations. TL:DR: hoping for magic compilers that can expose more ILP to the hardware didn't work out for Itanium IA-64, and is unlikely to work for a super-wide CPU for any existing ISA with a serial model of execution.

If you did have a super-wide CPU, you'd definitely want it to support SMT so you can keep it fed with work to do by running multiple low-ILP threads.

Since Skylake is currently 4 uops wide (and achieves a real IPC of 2 to 3 uops per clock, or even closer to 4 in high-throughput code), a hypothetical 8x wider CPU would be 32-wide!

Being able to carve that back into 8 or 16 logical CPUs that dynamically share those execution resources would be fantastic: non-stalled threads get all the front-end bandwidth and back-end throughput.

But with 8 separate cores, when a thread stalls there's nothing else to keep the execution units fed; the other threads don't benefit.

Execution is often bursty: it stalls waiting for a cache miss load, then once that arrives many instructions in parallel can use that result. With a super-wide CPU, that burst can go faster, and it can actually help with SMT.

But we can't have magical super-wide CPUs

So to gain throughput we instead have to expose parallelism to the hardware in the form of thread-level parallelism. Generally compilers aren't great at knowing when/how to use threads, other than for simple cases like very big loops. (OpenMP, or gcc's -ftree-parallelize-loops). It still takes human cleverness to rework code to efficiently get useful work done in parallel, because inter-thread communication is expensive, and so is thread startup.

TLP is coarse-grained parallelism, unlike the fine-grained ILP within a single thread of execution which HW can exploit.

BTW, all this is orthogonal to SIMD. Getting more work done per instruction always helps, if it's possible for your problem.

Let me draw an analogy:

If you have a monkey typing away at a typewriter, and you want more typing to get done, you can give the monkey coffee, typing lessons, and perhaps make threats to get it to work faster, but there comes a point where the monkey will be typing at maximum capacity.

So if you want to get more typing done, you have to get more monkeys.

multicores aren't usually multiscalar. and multiscalar cores aren't multicores. it would be sort of perfect finding a multiscalar architecture running at several megahert but in general it's bridges would be not consumer enabled but costly so the tendency is multicore programming at lower megahert rather than short instruction at high clock speeds. multiple instruction cores are cheaper and easier to command that's why it's a bad idea having a multiscalar architectures at several gigahert