What Is Happening To The jQuery Plugins Site?
For about a week, instead of hosting several hundred jQuery plugins and several thousand advertisements for laptop batteries, our plugins repository has been serving up a pretty pathetic message about spam and an allusion to some “new submission process.” This happened very suddenly, and we’re sorry to everyone who’s been inconvenienced. Please allow me a few minutes to explain what happened, where we’re headed, and how it impacts you. If you’re in a rush, here’s the short version.
The Backstory
Though the plugins site you’ve known and loved was a valuable tool when it was first set up, it gradually became something of a white elephant for the project. While powerful distribution tools like GitHub and npm have come to the fore, we’ve been stuck in an aging, CMS-oriented paradigm that frustrated developers and consumers of plugins alike. Many people moved onto alternative sources for finding and vetting plugins. Furthermore, the sites’s original implementors and maintainers had since moved on from active involvement within the jQuery project. While the team faced a steady stream of complaints about usability and a general lack of features, the site itself faced a veritable barrage of (several flavors of) spam. There was your standard, keyword-rich SEO garbage, but there was also something slightly more insidious: the batch posting of under-documented, demo-free plugins with links to paid download sites by third parties trawling for affiliate cash. While this wasn’t explcitly disallowed, it led to a terrible, confusing experience for users and gave the site the distinct sense that all was not on the up and up.
The Best Laid Plans…
Sensitive to all these problems, the team began drawing up requirements for a complete overhaul. We knew we wanted to get out of the business of accepting uploads, serving downloads, and generally legislating the plugin “release” process on our own servers. With our official projects already on GitHub, we knew we wanted to leverage the many wheels they’d already invented for distribution, versioning, and facilitating open, collaborative development. Another requirement was to create a standard schema for authors to clearly delineate dependencies, like which version(s) of jQuery a plugin supports, as well as other plugins and CSS assets it requires to be functional. We also wanted to introduce some quality control, with both passive mechanisms like requiring GitHub and a manifest file for distribution, as well as active ones like user ratings. Finally, we wanted a clean slate; with stricter requirements for submission, there could be no mass import of all the old plugins. We made a lot of progress preparing specifications, but hampered by a lack of resources and a number of other projects, we never were able to get too deep into the implementation phase. After all, the old site “worked!”
…Often Go Awry
As the glut of spam grew worse and multiple reports started showing up on the jQuery Core bug tracker, I wanted to at least take some steps to wipe a bit of the spam and egg off our face. With newly provisioned access to the administrative tools on the site, I teamed up with the Drupal Views Bulk Operations module and set out to identify and delete spammers and their posts. Within a day or two, nearly 90% of the alleged “content” was gone from the site. I continued to monitor the situation over the ensuing days, and deleted spam as it came in. Unfortunately, I likely cast too wide of a net, and threw out several perfectly good babies with the bathwater. Even more unfortunately, I didn’t back up the database before I began this process. At this point, you can probably see where this story is headed. Later in the week, while I was attempting to delete four spam items, I was left completely horrified when the results of the operation reported that the remaining 10% — every single plugin remaining in the database — had been purged. All that remained was a year-old backup. Of course, I realize that this is a dreadful outcome, and I take full responsibility for it. If it helps, I am very receptive to hate mail and tweets reprimanding me for being irresponsible, unprofessional, or just stupid.
At this juncture, we were left with two choices.
- Keep on kicking the can down the road: Restore from the old backup, losing a lot of data anyway, and have authors go through the painstaking process of re-uploading their plugins, even though the site was slated to be deprecated entirely anyway.
- Use this glaring mistake as the impetus to hunker down, cleave from the past, and finally implement the site we’ve talked about for so long.
We’ve chosen the latter.
Nuts And Bolts
Over the past few days, we’ve started converting our plans into action, building out an infrastructure that’s backed by GitHub. There are two requirements for listing a plugin on the new site:
- A valid package.json file
We’ve followed the lead of CommonJS and npm and created a schema for specifying dependencies, delivery, and other metadata of jQuery plugins. While the format is largely similar to those other projects, we’ve had to make some minor tweaks to account for some plugin-specific details. - At least one versioned release
This means having tagged your release point(s) with a valid semantic version number (semver) string.
We’ve pared down the submission and maintenance process to a single, one-time step: adding a post-receive hook to your plugin’s GitHub repository. Assuming your plugin meets the guidelines, a page will be created on the plugins site to present your usage and download information. We’ll keep track of new releases as you push them.
In The Interim
We recognize that the old site was still serving as an regular resource for a lot of people, especially newer jQuery users, who simply valued the existence of a central browsing location, despite its flaws. Though many experienced users had moved on to other sites, or relied on relationships with trusted authors and word of mouth, these can take time to develop. Until we’re able to launch the new system, we’re happy to direct you to several other directories and people who can help pick up the slack:
If, in the process of searching these or any other directories, you are directed back to the current site at plugins.jquery.com, typically a quick search for the author’s name and the plugin name will yield an alternative site where it was hosted. In addition, DailyJS just did a useful roundup of alternatives that goes into more detail.
Next Steps For Plugin Authors
If you’re a plugin developer who wants to make sure your plugins will be ready to go on day one (or even beforehand), you’ll want to make sure your plugins are up on GitHub, and then you’ll want to get started on creating your package.json files and making sure your versions are appropriately tagged. If you aren’t already familiar with Git and GitHub, then this is probably a very good time for you to take the plunge and get started. If you prefer another SCM system, you can look into setting up a mirror to git from svn or hg, or other project hosting sites, like bitbucket or gitorious. If you don’t develop your plugins as open source, or you don’t use any source control at all, we will not be able to accomodate you at this time.
How Can I Help?
Though the site is still a work in progress, you can track progress and even set up your own local development right now at github.com/jquery/plugins.jquery.com/. We’re still working on getting a public staging environment together.
Though the site is essentially “powered by” GitHub, there are obviously a number of moving parts in play. The site will actually be served from inside of WordPress, which is populated with the contents of the indexed plugins using a node.js tool. There are still many kinks to be ironed out, so if that sounds like a stack you want to jump on, please join us over at the repo. We’ll be using GitHub Issues for all feature requests, bugs, and discussion.
While we’ve put a great deal of thought into putting together our package.json schema, it’s still a living document, and if you have questions or comments on why it is the way that it is, head on over to this issue.
Errata
In an ideal world, this certainly wouldn’t have happened exactly as it did. Sadly, it did. We hope you’ll accept our apologies for the many ways this transition might make your job harder, give you some extra work you weren’t expecting, or just plain leave a bad taste in your mouth. If you are able to forgive us, bear with us, and maybe even lend a hand, we’re confident we’ll be able to deliver the modern, useful plugins site the community deserves.
TL;DR
So that was a lot of words, here’s the takeaway.
- We’d been planning on replacing the original, spam-ridden plugin site for quite some time
- In the process of deleting on the spam, all the plugins were deleted and we didn’t have a recent backup
- Instead of burning cycles keeping the old site on life support, we decided to make a clean break and kick development on the new site into high gear.
- The new site is powered by GitHub and a package manifest for plugins
- Plugins from the old site were never going to be automatically imported into the new one.
- The repo is at github.com/jquery/plugins.jquery.com
- We are very sorry, but also very excited!
Way to be transparent and take ownership of your mistakes. If only some of the “leaders” in our world followed this mantra.
yeah it’s happen sometimes.
very appreciate your honest
You’re excited about moving from an “aging CMS-oriented paradigm” to WORDPRESS?
Does. Not. Compute. Seems like a juvenile mistake and solution, through and through.
Fortunatly it happens!
It’s not a mistake, it’s an opportunity.
Can’t wait for the new one.
Have a good work on the new site.
Approved. The old system sucked. I can totally understand not wanting to back it up. Any code worth keeping should already be tracked in a VCS on the developer’s machine(s) (i.e., no actual good plugin code should be lost), and there was really no other data of any value. :P I look forward to seeing the new system come to fruition.
My prior process for using plugins was essentially: boss says “We can use X (link to author site)” => downloads X. I don’t think _any_ downloading happened through plugins.jquery.com, even the few times I was there.
Still, mass deletions with no backups (nor, apparently, previews)? Ouch.
A blessing in disguise…the previous site was a pain to try find plugins.
If I also I have trouble finding the plugins. But I think that you can better as people make their complaints of defects.
How do you include a list of optional files or dependencies, like the easing plugin? I’m only guessing the “files” should be essential, or should we just include everything?
I’m sure we’ve all made this type of mistake, no ‘stepping aside’ needed!
I agree about the archive.org idea, can’t hurt to look into it as a temporary fix. For that matter, I’m sure google has everything on their servers too (maybe even MSN).
Thank you jesus, is what I say. The old plugin site was not particularly usable, in my (honestly) humble opinion. I’m looking forward to a new site, and github integration would be great IMO.
I hope very much that you don’t go down the npm path (or use it’s code for your package manager), because it’s absolutely full of holes, and their submission process is WORSE than your old one.
To those advocating compatibility between jQuery plugins and Ender.js packages:
Compare the details of the proposed package.json spec to Ender’s package.json spec and you’ll see that they’re nearly equivalent. Of course, this shouldn’t come as a surprise because they’re both just using subsets of NPM‘s package.json spec, which is in turn based on (but not quite the same as) CommonJS‘s package.json spec.
So if you take the time to add an “ender” keyword and publish your jQuery plugin using NPM, it should show up in Ender. Couldn’t be easier, right? The problem is that it’ll still have a dependency on jQuery, which makes the entire exercise moot. There are a few different ways to address the issue.
The simplest would be to use CommonJS’s hash group dependency syntax, which unfortunately was not carried across into NPM. It provides the ability to specify alternate dependencies based on functionality, like so:
“dependencies”: {
“selector_engine”: {
“qwery”: “3.0.0”,
“jquery”: “1.6.0”
},
“dom_utility”: {
“bonzo”: “0.7.0”,
“jquery”: “1.6.0”
}
}
Note that the ordering of the group would be significant, with earlier entries having higher priority. In the example above, you would fall back to depending on jQuery if either of the Ender packages Qwery and Bonzo were missing.
The other way, which doesn’t require any modifications to NPM, is just to take advantage of forking. Since all jQuery plugins are moving to Github, it will become trivially simple to fork a given plugin and switch out the jQuery dependency for something else.
I personally would prefer the first approach because widespread adoption of standards tends to make the web a better place. That said, the second method is just fine. Either way, the move to Github can only help Ender and the overall JS ecosystem.
The old plugins section was less than amazing… now, I understand, when you accidentally delete/clear out an entire database on a major website and have no recent backup, you kind of have to apologize (after you swear and freak out for a bit); but seriously, I think it’s for the best. Many of the plugins were so hopelessly out of date and/or abandoned, that Google was a more effective way for me to find a jQuery plugin.
As someone else in the comments mentioned (or maybe a few people), most serious plugin developers will have their code sitting locally or online in some flavor of source control.
Being a light user of the plugins section, my personal priority would be for the jQuery UI team to release jQuery UI 1.9… that project has been moving quite slow. I am assuming that’s a mostly different group of people however, and certainly for the jQuery community as a whole, replacing the plugins site absolutely takes priority (especially since there isn’t a site at all right now, except on the archive site mentioned above perhaps).
rm -rf *
D:
Great to hear of the new system; already had my plugin on github so no major working points; was just wondering when will the call back system start working so I post my update with the new json file. In addition to that; in the file itself where it says list of files does this have to include any example code bundled within the repository?
No backup before batch cleanup => EPIC FAIL.
Sounds like NO BACKUPS AT ALL exist.
Conspiracy!
While the takedown of the jQuery plugins site was a complete management FAIL, since the old site could have been easily kept available, frozen, until the new one was available, I do respect the honest explanation.
Oops… didn’t read all of it.
Thanks for your honesty.
As long as “Powered by” GitHub doesn’t mean GitHub becomes a requirement this sounds pretty good. Would be nice if GitHub was just optional for versioning and bugtracking.
$(“#plugins”).remove();
Thanks for your honesty, Adam, and good luck with the new plugins site.
Please post a read-only archive of your year old plugins. There is lots of content in there that can be used as a template for new plugins.
Great job guys. I’m with you and don’t worry about inconveniences, because I think that people always tolerate it, if they know that a better future is awaiting. I’m also glad that you use GitHub. I’m gonna watch you there.
Writing code is like writing poem, everytime you come up with something new. can’t wait to see new stuff….
Thank you for the update, but most all thank you, all of you maintainers, coders and faithful users for all you do. I benefit so much from the site and plugins for the sites I work on. Looking forward to the spam free site.
One of the best attributes about the plugin page is it provided a way to “register” a plugin name to avoid namespace collisions.
How do we register our plugin names these days while awaiting the new site to come up?
I’m glad that I am not the only one who makes bad mistakes.
Here are so many good coders, we will help to fill the repository with new plugins.
I’ve collected all plugins, that I’ve downloaded in the last to month.
Wish you all a great new year
Sven
@Paul
“Github is nice for the really technical pple, but I dont understand how to download plugins anymore :((((((((((((((((((((((((”
If you’re not a technical person, you should not be developing JavaScript.
@The Team
The only reason one creates a backup before some non-standard operation is because you don’t want to go all the way back to the last daily backup in case of an accident… :/ Anyway, now you know how not to do things.
I am waiting :)
Adam, every dev worth his salt has forgotten a where clause at least once and blasted an entire table instead of a record or two. OK, maybe not so often on a production database without a recent backup… but we’ve all had those “Oh s#@t.” moments…
The jQuery plugins site has not been the place to download the finest plugins for a long time anyway – for well over a year.
I look forward to the new site!
insane.. i’m like.. is this real?
Although this is regrettable, I am so thankful for the so many people who work to benefit us — without being paid.
This is actually a great reminder for me to do a backup of my development laptop TODAY!
Is there any other place where we can get the plugins? I’m looking for the jFlick plugin to be more specific. Does anyone have it? That would be much appreciated if someone could post it somewhere or send it to me.
Thanks so much in advance!
Hello
Some plugins you can find on flcomponents.net. You can also submit plugins there or download for free.
Blaze
I am new to web development, but I have realized that without jQuery, web development cannot be imagine (at least I can’t).
Thank you very much guys, you rock!!!
Apparently an old database has been restored and is available under http://archive.plugins.jquery.com
This sounds like a pretty typical snafu, and I think you’ve probably gained some credibility for your honest and forthright explanation of the state of things and plans for a fix. Looking forward to see what this becomes!
We’re engineers, my friend. I don’t care what people say, but deleting data without taking a backup is NOT a crime. Please take it easy. And thank you for all the hard work you’re putting in. :)
Cheers!
Here is another start for the post:
“Hi! My name is Adam Sontag. I am responsible for deleting all the previous plugins on plugins.jquery.com. Here is why this is a good thing:”
:D :D
I think i should thank you for the error, I think that its a good thing that the whole or most of it plugin bank has been wiped , its gonna make some place for better new ones
To the people blaiming Drupal for this in the comments: please get informed. The problem had nothing to do with Drupal and was a human error. Read the post before commenting.
somebody has display-tag-ajax-1.2.js ? not found
Sounds great, awesome idea!
Any idea on what the URL for the GitHub hook will be?
You have a backup here it is http://web.archive.org/web/20110717030551/http://plugins.jquery.com/
This is a great site and I’m very sorry such thing happened. That being said, not having backups of the database was very, very stupid (I hope that never happens to me, karma’s a bitch). Anyway, what’s done is done. Good luck, I’m looking forward to the launch of the new site.
Could have happened to anybody.
I forgive you.
>>You have a backup here it is http://web.archive.org/web/20110717030551/http://plugins.jquery.com/
This backup only shows the plugin summaries.
Does anybody know when it is planed to release the new plugins website?
“To the people blaiming Drupal for this in the comments: please get informed. The problem had nothing to do with Drupal and was a human error. Read the post before commenting.”
-well put