Xjyuk

Many years ago I read an old sci-fi novel - when I say old, I suspect it was from about the 1950's or so.

The elements I can remember are an experiment where a cube is cooled to below absolute zero and that it then punches a hole into another dimension.

They try to get people to look through the hole but anyone who does goes insane. Then the 'inhabitants' of the other dimension take offence to the damage the experiment is doing in their world and start attacking our world in retaliation.

The Universe Between, a 1965 novel by Alan E. Nourse, which was also the answer to this old question. It was based on the short story "High Threshold" in Astounding Science Fiction, March 1951 and the novelette "The Universe Between" in Astounding Science Fiction, September 1951, which are available at the Internet Archive here and here.

ISFDB synopsis:

A scientist conducting experiments on low temperatures inadvertently trespasses on another universe.

My copy of the novel has gone missing. The following excerpt is from the short story "High Threshold":

Ned McEvoy took the ball from his briefcase and laid it on the desk before the thin man with his pince-nez. "What does that look like to you, Dr. Bamford?"

The man examined it closely, and looked up smiling. "It looks like a tennis ball that someone has turned inside out," he replied.

McEvoy chuckled. "And how would you go about turning a tennis ball inside out, Dr. Bamford?"

"Can't say I know, offhand." He looked quizzically at McEvoy. "What can I do for you, doctor?"

"You've heard the old story of the goats that were carried across the Andes on muleback—and all died of fright?"

"Of course."

"Well, we have the same problem in my laboratory. Only we have men dying of fright."

Dr. Bamford's eyebrows went up. "Adjustment?" he ventured.

"We think so. About six months ago we ran into a peculiar snag in the work I was supervising. The Institute of Physics has been concerned for several years with problems involving extremely low temperatures—nearly absolute zero temperatures. The work we originally planned called theoretically for an approach within six decimals of complete cessation of molecular motion. That involves a temperature of one millionth of one degree Kelvin. And we reached it."

McEvoy fingered his collar nervously, and shifted his weight in the chair. "Matter of fact, we did even better. Our pumps began acting up as we approached a thousandth of a degree. What happened, we think, was a reversal of the Franklinson effect in the extremes of high temperature, where the temperature doubles quite suddenly with a tremendous molecular expansion. Our temperature took a sudden startling drop."

"I'm not quite sure I follow you," said Dr. Bamford. "From one degree Kelvin, just where does the temperature drop?"

McEvoy scowled. "A good question," he said. "I don't know, to be quite frank. Zero Kelvin is a relative and hypothetical point at which all molecular motion ceases. Below zero Kelvin, if such were possible, one could reasonably expect negative molecular motion. That may have been what we obtained; we don't know. But we certainly observed a change. The tungsten block we were treating simply evaporated. Vanished. The temperature recording device vanished. All we could see in the vault was a small glowing hole in the corner of the room where the block had been. Nothing in it, Dr. Bamford—nothing. And the hole, seen at a distance, appeared very suspiciously like a—hypercube."

The doctor was silent for a long moment. "You investigated?"

"We surely did. We're still trying. It looks as if we have a four-dimensional projection in our three-dimensional space—a corner, or an edge, of four-dimensional space. We've tried everything, and we're getting nowhere. So far we've lost five crackerjack technicians investigating. We're no better off than when we started."