Intel has a secret security lab full of old hardware to hunt for bugs

Intel has revealed some of the details of its Long-Term Retention (LTR) Lab, which is a hardware archive that allows engineers to probe older chips for security bugs. 

According to Intel, the need for a centralized LTR Lab became apparent a few years ago when its engineers were forced to search on eBay for an older chip in order to troubleshoot an issue affecting specific configurations. The chips they were searching for included Sandy Bridge CPUs, according to the Wall Street Journal. 

Intel released Sandy Bridge CPUs in 2011. Though Sandy Bridge was discontinued in 2013, Intel in 2018 was busy patching them and older generation chips with Microcode Updates (MCUs) to address the Spectre speculative execution flaws.

SEE: A winning strategy for cybersecurity (ZDNet special report)

Intel’s product line-up is already vast, but becomes more complicated when its chips are configured for various OEMs, operating-system makers, cloud-computing providers, firmware vendors and system integrators. It works with these partners to validate and deliver its Intel platform updates (IPUs).

Today, the LTR Lab houses chips for 2,800 platforms, with some of the oldest dating backing to 2012. The goal next year is to boost capacity to 6,000 platforms. Products include hardware for high-end servers, high performance computing, cloud providers, as well as desktops and laptops, and embedded products used by the Internet of Things ecosystem. 

“When the lab first started, the main goal was to create a centralized location for storing hardware; this later expanded to retaining thousands of live platforms along with design, software and documentation collateral,” Intel says in a blogpost. 

Given that chip design bugs like Spectre and Meltdown can persist for years and change in nature as researchers find new ways to exploit them, the hardware archive may help Intel engineers fix bugs faster in the future. 

Intel says it helps engineers “analyze security and functional issues on supported products more efficiently, while better enabling proactive research for the continuous improvement of products.”

The Wall Street Journal notes that Intel’s hardware archive is at a secret location in Costa Rica. Intel won’t reveal the exact location. The LTR Lab has been operational since 2019 and will expand to 27,000 square feet next year, up from 14,000 square feet. 

The lab receives about 1,000 requests from Intel engineers each month, allowing them to remotely troubleshoot security and functional issues on a specific CPU, operating-system version, microcode, and BIOS. It’s safer troubleshooting these bugs in a lab since testing on a live system could result in a crash and data loss.      

The lab is operational 24 hours a day, seven days a week and is capable of spinning up a specific configuration within minutes, according to Intel.  

Besides researching security flaws, Intel uses the lab test to explore the power and performance degradation of its products. 

SEE: Cloud security in 2021: A business guide to essential tools and best practices

Fawn Taylor, who leads Corporate Remediation Programs at Intel, says the LTR Lab has helped the company extend its support for CPUs and platforms beyond the official support period. 

“It’s a bit of a chicken and an egg problem,” says Taylor. “As Intel’s putting in our Long-Term Retention systems, we go back to our suppliers. There are certain tools and hardware that we get from suppliers and now we need them to extend some of their support for us in order to keep their environments and hardware alive longer than we planned. 

“So we’re kind of really challenging our suppliers. Intel’s obviously a supplier so when customers come to us looking for this level of support to keep the things we supply to them — our CPUs, our platforms — available years beyond the original sale, we’ll be ready. We will quickly become their supplier of choice because of what we’ve implemented.”