The 11 lines of code which broke the internet.

By Struan Caughey

One of the founding principles of the internet was openness, which has led to incredible communities that build upon each other’s works. Even large companies such as Facebook and Google rely on other people and companies’ work so that they do not have to write their code from scratch.

This system is brilliant— it allows for faster development of programs and means that people do not have to rewrite the same code. The name behind this principle is open-source code [1] and while generally accepted as an important and integral part of the internet there are some unintended consequences. Firstly you are relying on the code being efficient and secure. Secondly, this makes you reliant on the developers of that package to keep it up to date if any issues do occur. Lastly, this can also create chains where packages are dependent on packages resulting in the final programmer not knowing exactly what code their own program is executing.

Most programming languages contain all of the essential functions within their ‘standard library’, meaning that you do not have to rely on third-party authors' javascript, however, utilises third-party depositories instead of a standard library [2]. One of the largest of these is called npm [3]. People essentially place their code onto npm and then others can access it just through their code. One contributor to this platform was called Azer Koçulu. Koçulu was a high school graduate who taught himself how to code. In an email he sent to Quartz magazine he stated that “I owe everything I have to the people who never gave up with the open-source philosophy” [4]. With this belief, one of the core principles is pushing against the commercialisation of code and instead empowering creators such as Azer to have full control over their code.

Photo by Mohammad Rahmani on Unsplash

The issues started with a single project of Koçulu’s which was under a package named “kik” (unrelated to the code which would later cause further problems). The instant messaging app also named Kik based out of Canada decided that they too wanted to create a depository called kik, however, they would not be able to due to Koçulu’s existing project [5]. Initially, Kik reached out to Koçulu to ask if he would remove his project. He refused their request, as he perceived it as an overly aggressive response. This in turn resulted in Koçulu asking for $30,000 (USD) “for the hassle of giving up with my pet project for [a] bunch of corporate dicks” [6]. Kik then reached out directly to npm who sided with them, and agreed to turn over the package’s name to them from Koçulu. On this they said, “In this case, we believe that most users who would come across a kik package, would reasonably expect it to be related to kik.com,” [6].

As you would expect, this did not go down well with Koçulu, who had been an avid proponent of open-source philosophy and npm. He sent out an email stating that he was very disappointed and no longer wanted to be a part of npm— that he wanted all of his packages registered on npm to be taken down [6].

This ended up happening and sure enough, two days later coders around the world started getting the error message:

“Npm ERR! 404 ‘left-pad’ is not in the npm directory” [4].

Left pad was an incredibly simple piece of code written in full below:

[4]

All this code does is add characters to the left of the text to ensure that the length of a line stays consistent. For example, if I was to feed in “‘17’, 5, ‘0’” into the code it would print “00017”, adding padding to the left of the text [7].

It soon became evident what had occurred. This one small piece of code had been used in several other packages, which, in-turn, was propagated across all sorts of different programs, many of which did not explicitly use this package. One of these large packages which indirectly used it was called Babel which utilised left-pad; it was itself used by the likes of Facebook, Netflix, and Reddit amongst others. With left-pad gone, Babel became unable to install. For context, despite being relatively unknown, left-pad at its peak was downloaded over 4 million times per week [7].

Ironically, things came full circle when on March 24, 2016 Mike Roberts (head of messaging at Kik) found that his team was encountering the exact same issue due to their use of a package called LSCS, which through a long chain of dependencies relied on left-pad. In Mike’s piece on the self publishing site Medium, he also discusses the situation from Kik’s side as well as publishing all the emails between Kik’s patent agent, Koçulu and npm [6].

Immediately there was a scramble for a fix. The removal of this simple package was causing errors across the globe with people from Australia, Germany, the US, and more commenting on the left-pad npm page trying to find out what happened. Babel quickly replaced the package and within 2 hours npm took the un-un-published left-pad code. On the restoration of the code, npm said that “Un-un-publishing is an unprecedented action that we’re taking given the severity and widespread nature of breakage, and isn’t done lightly,” [4].

While in the end there were no long term issues resulting from this action, it did show there are some significant vulnerabilities within our current systems. While dependencies are vital, to ensure security of your code you should know what your code contains. If you think that issues like this must have been resolved you would be half right - npm has instituted new policies to make sure that unpublished code won’t cause widespread breakages [5], however, the system still has other issues, such as trust that your dependencies are secure. In November 2021 we saw in a catastrophic bug that this may not be the case.

On November 24th 2021 a 0-day bug was found (0-day meaning 0 days notice was given to fix it before the exploit went public). It was given a 10/10 criticality rating and is part of the language Java [8]. The flaw called Log4j immediately began being exploited [9]. On December 22, 2021, Tenable found that 10% of all assets that they assessed were vulnerable to the exploit and yet only 70% of organisations had begun looking at whether they are vulnerable [10]. While there have been several patches we are yet to see whether these can be navigated around.

One of the big issues which lie with log4j is again dependencies, as even if Java updates their code all the programs which are dependent on Java also have to update theirs and so on. Because of this the issues created by log4j could last for years and has been described by some in the industry as the “most serious (cybersecurity flaw) in decades” [11].

The world of tech is exciting and fast-paced but as it continues to grow new systems will have to be put in place to ensure that these kinds of domino effects cannot happen in the future. In no other industry would we accept industry-wide vulnerabilities as something that just happens and needs to be fixed, we instead need to learn to prevent this from occurring.

References

[1] Opensource. “What is open source?”, accessed 9/01/2022, https://opensource.com/resources/what-open-source

[2] Developedia.org. “Standard Library”, accessed 9/01/2022, https://devopedia.org/standard-library.

[3] W3shools. “What is npm”, accessed 9/01/2022, https://www.w3schools.com/whatis/whatis_npm.asp

[4] Collins, Keith. “How one programmer broke the internet by deleting a tiny piece of code”, 28/03/2016, https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/

[5] Miller, Paul. “How an irate developer briefly broke JavaScript”, 24/03/2016, https://www.theverge.com/2016/3/24/11300840/how-an-irate-developer-briefly-broke-javascript

[6] Roberts, Mike. 24/03/2016, https://medium.com/@mproberts/a-discussion-about-the-breaking-of-the-internet-3d4d2a83aa4d#.edmjtps48

[7] Mao, Steve. “left-pad”, Accessed 9/01/2022 https://www.npmjs.com/package/left-pad

[8] Apache Software Foundation, “CVE-2021-44228 Detail”, accessed 9/01/2022, https://nvd.nist.gov/vuln/detail/CVE-2021-44228

[9] Tung, Liam. “Log4j flaw hunt shows how complicated the software supply chain really is”, 7/01/2022, https://www.zdnet.com/article/log4j-flaw-hunt-shows-how-complicated-the-software-supply-chain-really-is/

[10] Yoran, Amit. “One in 10 Assets Assessed Are Vulnerable to Log4Shell”, 22/12/2021, https://www.tenable.com/blog/one-in-10-assets-assessed-are-vulnerable-to-log4shell

[11] Wayt, Theo. “Why is the Log4j cybersecurity flaw the ‘most serious’ in decades?”, 20/12/2021, https://nypost.com/2021/12/20/why-is-the-log4j-cybersecurity-flaw-the-most-serious-in-decades/