During the last two weeks of September, we found our way into the headlines due to a series of attacks on our web servers. Today, we wanted to give everyone a brief update on the status of our websites and a recap of what happened over the last two weeks.
jQuery Under Siege
Early on the morning of September 18th we were hit with a DDoS and went offline. We were down for a couple of hours. The sites were brought back up later that day on September 18th and all seemed well.
Later, during the afternoon of September 18th, we were contacted by a security company named RiskIQ reporting that their crawler had reported malware being served by our content sites. There were never any reports that the jQuery libraries nor the CDN were ever compromised. Immediately upon receiving that report, we completely destroyed and reimaged all of those machines, revoked and reissued all associated SSL certificates, and confirmed that there was no suspicious content being served at that point. Since then, our own team and security folks from Mozilla and MaxCDN have worked to analyze logs and attempt to confirm the impact of this attack.
On September 23rd, RiskIQ went public with their report which picked up steam throughout the day on various media outlets and Twitter. The next morning, September 24th, as DDoS attacks on our properties continued to increase both in frequency and magnitude, CVE-2014-6271, otherwise known as the ShellShock vulnerability, was issued. As we continued to respond to the media discussion and communicate to the community what had happened on September 18th, we were victimized again in a series of much more public attacks involving the repeated defacing of jquery.com.
Investigations into our systems have yet to find the initial attack vector. However, we did take some steps to make ourselves more secure. For instance, some of our WordPress installs were out of date, all of our servers were vulnerable to the recent shell vulnerabilities, NGINX was slightly out of date as well as maybe a few other patches etc. that needed to be made. The infrastructure team dove in and began making those changes and started building new, fully patched and secured servers to host our sites. It appears these changes were effective as the defacing stopped and we have not seen any evidence of intrusion since.
Later on September 24th, a massive and unrelenting DDoS attack began. It seemed as though it would come in waves, but did not stop until late on September 28th. Most of the time on September 26th and 27th was spent trying to implement various products and solutions in order to keep the servers alive. We fought day and night to try to keep the sites up. We have to commend Corey Frang, Adam Ulvi, the rest of the infrastructure team, and others; they worked through the nights and in alternating shifts to try to keep us on the internet. Without their efforts, we would not have had the short amounts of uptime we did. One significantly important step that we took was to reach out to CloudFlare, who generously and rapidly gave us access to their Enterprise service which has helped tremendously in mitigating these attacks.
Moving Forward
jQuery and the jQuery Foundation are important to the web ecosystem, as is evident from the amount of press and the number of concerned individuals and organizations that have reached out to ask questions about this attack. The jQuery Foundation works on a daily basis to maintain and improve our projects and the infrastructure around those projects. The goal of this work is to continue to make web developers’ jobs easier and make sure they have a voice in the world of standards and browsers. However, these objectives take a large quantity of resources. Whether those resources are provided by access to expertise of a company’s employees or services, or through financial support, we would be unable to continue this important work without the support of the open source community and our supporting members.
We have been asked several times throughout this ordeal about why we didn’t have XYZ service in place or why we didn’t have our security team keeping a closer eye on these types of risks. The simple answer is that our budgets are tight and resources are limited. Our infrastructure team, and most of our teams for that matter, are made up of volunteers who give their time for free to make sure things keep running. The Heartbleed and ShellShock vulnerabilities are recent examples of how badly things can go when open source projects are taken for granted and just assumed to be OK. Eventually something is going to fall through the cracks and those cracks become larger and more frequent when people are doing what they can in their spare time.
So how can you help? As an individual, get involved in one of our projects. We can always use help writing code, designing, maintaining servers, working on events and the list goes on. Take a look at contribute.jquery.org or come say hi on IRC in one of our many channels listed on irc.jquery.org. As an organization, we would love to hear about any service you may be willing to donate, any developers or other skilled professionals that you could spare for a few hours a week or if you can help financially. Send us a message at membership@jquery.org and let us know how you can help.
We haven’t wanted to say too much about these attacks as they have been happening because we remain a juicy target in the eyes of hackers who are continuing to attempt to infiltrate our servers even as of this writing. In sharing all of this information with the community now, we’ve tried to balance the need to explain what’s been happening with the potential backlash that could happen as a result of coming out publicly and saying that we believe we have the situation under control.
That said, we do at this point believe that we have the situation under control. For this, a huge thanks is due to the entire jQuery infrastructure team, who rolled up their sleeves and worked tirelessly on these issues to get us back to a good place. We will continue to be vigilant in ensuring the reliability and safety of all of our resources for our community of users.