Steinberg: ‘The technology has changed quite a bit’

Follow @neilsteinbergThousands of Chicagoans pass this nondescript building near the Loop every day and never give it a second glance. That is intentional. Made of beige grooved concrete, it is identified only by a single small plaque: “AT&T.” But don’t let its modest exterior fool you.

“This building touches every single resident of the city,” said Jim Wilson, AT&T’s Area Manager Network Services.

Those who do pause might notice something peculiar: no windows on most floors. Why build a 538-foot-tall building where only the second and the top seven floors have windows? The short answer is, because what’s inside isn’t able to look out and nobody outside is supposed to look in. Those at AT&T refer to the place only by its address, which is . . . well, they’d rather I not say. Security.

A bit of online sleuthing will turn up the Holabird & Root-designed building easily enough, but you can understand their caution. Not only does this center handle much of the city’s phone and internet traffic, but all the 911 calls come through here. Pressed for something to call the place, AT&T officials say they refer to it as an “Office” or a “Mega-Office,” one of three in the city.

“This is one of the key switching stations for AT&T,” said Warren Salek, assistant vice president of the company’s Radio Access Engineering division, guiding a tour of the facility never seen by the public. “Some of the first electronic switching systems were installed right here in this building.”

Built in 1970, the building actually has just 27 stories, though it is tall as a 50-story building because each floor is double height, built to accommodate enormous banks of telephone equipment. Streams of multicolored wire lead to banks of routers, servers and switching systems. About 300 engineers and staff work here.

What happens in the building is both very simple and vastly complicated — every phone line has to connect to every other phone line in a fraction of a second. While cellphone towers connect wirelessly to cellphones, every call and internet search still ends up running across a wire.

Follow @neilsteinberg“There’s no true wireless,” said Wilson. “Everything starts here then goes out to cell towers. You’re walking down the street with your cellphone, you’re still being serviced from here, still starting and terminating out of here.”

“We’ve got cables strung throughout the city, and all those cables come back to offices like this one,” said Salek, standing in a narrow room strung with dozens of fat cables, black tubes, each holding 1,200 twisted pairs of copper wire, and orange tubes holding copper’s replacement, fiber optics, invented by Bell Labs in 1984 and capable of carrying thousands times the traffic. AT&T stopped installing copper in 1998 but replacing it all would be prohibitively expensive so it’s still in use, the same way the water department still uses century-old cast iron pipes because it can’t afford to dig them all up. Though AT&T had made an effort to yank unused copper out from Chicago’s labyrinthine underground clutter.

“There’s a lot of people who don’t have a home phone anymore,” Salek explained.

The wires enter the building at the cable vault, then are shunted to the seventh floor, each wire connected to a panel of silicon chips within metal cabinets.

“Each one of these cards is a customer’s line,” said Wilson. “This is where dial tone starts.”

For all our high-tech capacity, the lines are identified in an old-fashioned system — color-coded — as well as numbers, like “NT6XUSAAR19.” Wooden ladders also still play a role in our modern communications world.

As if the whole thing weren’t complicated enough, not only does AT&T use the space, but so do other telephone carriers that run their networks on AT&T equipment. Old timers remember when AT&T was first broken up, 30 years ago.

“When they split us up, they put tape on the floor, this side was Ameritech, that side was AT&T,” said John Narcissi, a network engineering area manager. “If my badge said AT&T, I was not allowed on that side of the tape.” Technicians from various companies still can’t work on each other’s equipment.

Each floor hums, literally, the sound of cooling fans each unit has to keep from overheating. This takes electricity, and lots of it.

“Overall, our entire network, we spend about a billion dollars on power every year,” said Salek.

What happens if the power goes out? The building has six enormous yellow Caterpillar generators, V-16 2-megawatt behemoths standing 6 feet tall. Seven seconds after the power fails, they kick in, and AT&T keeps 200,000 gallons of diesel fuel on hand, enough to run them 40 days. They fire them up once a month to make sure they’re ready.

“We realize people rely on our products and services,” said Salek. “We need to deliver a reliable, secure network at all times. We have an extensive disaster recovery program. We’re certified for disaster preparedness by Homeland Security.”

If the power fails, there would be no power break to the equipment. The whole building runs on batteries — 2,000 wet cells, accepting the fluctuating AC current that ComEd sends in, changing it to smooth DC to run the banks of electronics.

The footprint of the equipment has reduced dramatically over the decades, and there is room in the building now, even with equipment, and some has been “retired in place” — taken off line, but not removed because, well, it costs a lot of money to removed huge banks of equipment and it’s free to leave it there.

The system has to be constantly updated because demands on it are skyrocketing.

“We’ve had an 150,000 percent increase in traffic since smartphones were introduced in 2007,” said Narcissi.

“One hundred petabytes of data go across our network,” said Salek. “Over 60 percent of the traffic that goes across our wireless network is video.”

The latest development is equipment that changes purpose rather than having to be replaced.

“Where we’re heading is our software-defined network,” said Salek. “In the past, we added capacity by adding more network devices: routers, switches and other custom-built physical equipment.” Now rather than replace equipment, they can just re-program it.

“That model doesn’t scale quickly enough, though, so we are virtualizing those hardware devices — turning them into software apps running on commodity servers and other standardized hardware — lets us increase capacity much faster,” said AT&T spokesman Jim Kimberly. “And making those virtual network functions software-controlled allows customers to manage their own services in near-real time. AT&T’s goal is to virtualize 75 percent of its network by 2020, and we’re meeting or exceeding all of our initial benchmarks.”

“The technology,” Salek said, offering up the understatement of the day, “has changed quite a bit.”

Tweets by @neilsteinberg