What Is Happening To The jQuery Plugins Site?
For about a week, instead of hosting several hundred jQuery plugins and several thousand advertisements for laptop batteries, our plugins repository has been serving up a pretty pathetic message about spam and an allusion to some “new submission process.” This happened very suddenly, and we’re sorry to everyone who’s been inconvenienced. Please allow me a few minutes to explain what happened, where we’re headed, and how it impacts you. If you’re in a rush, here’s the short version.
The Backstory
Though the plugins site you’ve known and loved was a valuable tool when it was first set up, it gradually became something of a white elephant for the project. While powerful distribution tools like GitHub and npm have come to the fore, we’ve been stuck in an aging, CMS-oriented paradigm that frustrated developers and consumers of plugins alike. Many people moved onto alternative sources for finding and vetting plugins. Furthermore, the sites’s original implementors and maintainers had since moved on from active involvement within the jQuery project. While the team faced a steady stream of complaints about usability and a general lack of features, the site itself faced a veritable barrage of (several flavors of) spam. There was your standard, keyword-rich SEO garbage, but there was also something slightly more insidious: the batch posting of under-documented, demo-free plugins with links to paid download sites by third parties trawling for affiliate cash. While this wasn’t explcitly disallowed, it led to a terrible, confusing experience for users and gave the site the distinct sense that all was not on the up and up.
The Best Laid Plans…
Sensitive to all these problems, the team began drawing up requirements for a complete overhaul. We knew we wanted to get out of the business of accepting uploads, serving downloads, and generally legislating the plugin “release” process on our own servers. With our official projects already on GitHub, we knew we wanted to leverage the many wheels they’d already invented for distribution, versioning, and facilitating open, collaborative development. Another requirement was to create a standard schema for authors to clearly delineate dependencies, like which version(s) of jQuery a plugin supports, as well as other plugins and CSS assets it requires to be functional. We also wanted to introduce some quality control, with both passive mechanisms like requiring GitHub and a manifest file for distribution, as well as active ones like user ratings. Finally, we wanted a clean slate; with stricter requirements for submission, there could be no mass import of all the old plugins. We made a lot of progress preparing specifications, but hampered by a lack of resources and a number of other projects, we never were able to get too deep into the implementation phase. After all, the old site “worked!”
…Often Go Awry
As the glut of spam grew worse and multiple reports started showing up on the jQuery Core bug tracker, I wanted to at least take some steps to wipe a bit of the spam and egg off our face. With newly provisioned access to the administrative tools on the site, I teamed up with the Drupal Views Bulk Operations module and set out to identify and delete spammers and their posts. Within a day or two, nearly 90% of the alleged “content” was gone from the site. I continued to monitor the situation over the ensuing days, and deleted spam as it came in. Unfortunately, I likely cast too wide of a net, and threw out several perfectly good babies with the bathwater. Even more unfortunately, I didn’t back up the database before I began this process. At this point, you can probably see where this story is headed. Later in the week, while I was attempting to delete four spam items, I was left completely horrified when the results of the operation reported that the remaining 10% — every single plugin remaining in the database — had been purged. All that remained was a year-old backup. Of course, I realize that this is a dreadful outcome, and I take full responsibility for it. If it helps, I am very receptive to hate mail and tweets reprimanding me for being irresponsible, unprofessional, or just stupid.
At this juncture, we were left with two choices.
- Keep on kicking the can down the road: Restore from the old backup, losing a lot of data anyway, and have authors go through the painstaking process of re-uploading their plugins, even though the site was slated to be deprecated entirely anyway.
- Use this glaring mistake as the impetus to hunker down, cleave from the past, and finally implement the site we’ve talked about for so long.
We’ve chosen the latter.
Nuts And Bolts
Over the past few days, we’ve started converting our plans into action, building out an infrastructure that’s backed by GitHub. There are two requirements for listing a plugin on the new site:
- A valid package.json file
We’ve followed the lead of CommonJS and npm and created a schema for specifying dependencies, delivery, and other metadata of jQuery plugins. While the format is largely similar to those other projects, we’ve had to make some minor tweaks to account for some plugin-specific details. - At least one versioned release
This means having tagged your release point(s) with a valid semantic version number (semver) string.
We’ve pared down the submission and maintenance process to a single, one-time step: adding a post-receive hook to your plugin’s GitHub repository. Assuming your plugin meets the guidelines, a page will be created on the plugins site to present your usage and download information. We’ll keep track of new releases as you push them.
In The Interim
We recognize that the old site was still serving as an regular resource for a lot of people, especially newer jQuery users, who simply valued the existence of a central browsing location, despite its flaws. Though many experienced users had moved on to other sites, or relied on relationships with trusted authors and word of mouth, these can take time to develop. Until we’re able to launch the new system, we’re happy to direct you to several other directories and people who can help pick up the slack:
If, in the process of searching these or any other directories, you are directed back to the current site at plugins.jquery.com, typically a quick search for the author’s name and the plugin name will yield an alternative site where it was hosted. In addition, DailyJS just did a useful roundup of alternatives that goes into more detail.
Next Steps For Plugin Authors
If you’re a plugin developer who wants to make sure your plugins will be ready to go on day one (or even beforehand), you’ll want to make sure your plugins are up on GitHub, and then you’ll want to get started on creating your package.json files and making sure your versions are appropriately tagged. If you aren’t already familiar with Git and GitHub, then this is probably a very good time for you to take the plunge and get started. If you prefer another SCM system, you can look into setting up a mirror to git from svn or hg, or other project hosting sites, like bitbucket or gitorious. If you don’t develop your plugins as open source, or you don’t use any source control at all, we will not be able to accomodate you at this time.
How Can I Help?
Though the site is still a work in progress, you can track progress and even set up your own local development right now at github.com/jquery/plugins.jquery.com/. We’re still working on getting a public staging environment together.
Though the site is essentially “powered by” GitHub, there are obviously a number of moving parts in play. The site will actually be served from inside of WordPress, which is populated with the contents of the indexed plugins using a node.js tool. There are still many kinks to be ironed out, so if that sounds like a stack you want to jump on, please join us over at the repo. We’ll be using GitHub Issues for all feature requests, bugs, and discussion.
While we’ve put a great deal of thought into putting together our package.json schema, it’s still a living document, and if you have questions or comments on why it is the way that it is, head on over to this issue.
Errata
In an ideal world, this certainly wouldn’t have happened exactly as it did. Sadly, it did. We hope you’ll accept our apologies for the many ways this transition might make your job harder, give you some extra work you weren’t expecting, or just plain leave a bad taste in your mouth. If you are able to forgive us, bear with us, and maybe even lend a hand, we’re confident we’ll be able to deliver the modern, useful plugins site the community deserves.
TL;DR
So that was a lot of words, here’s the takeaway.
- We’d been planning on replacing the original, spam-ridden plugin site for quite some time
- In the process of deleting on the spam, all the plugins were deleted and we didn’t have a recent backup
- Instead of burning cycles keeping the old site on life support, we decided to make a clean break and kick development on the new site into high gear.
- The new site is powered by GitHub and a package manifest for plugins
- Plugins from the old site were never going to be automatically imported into the new one.
- The repo is at github.com/jquery/plugins.jquery.com
- We are very sorry, but also very excited!
Thanks for the honest explanation there. I know it was hard to admit. I think the plans for the new site sound great. Again – thank you!
Ender.js is much more suited to client side libs I think, and is compatible with package.json! http://ender.no.de/
Great plans though!
It would definitely be nice to figure out if we can do Ender support instead of having to reinvent the CLI wheel, for people who want it.
We have all done the same type of thing at least once! A blessing in disguise.
Having put a couple of plugins on the old plugins page and actually searched for some stuff in that same thing – I can only congratulate you (and us). From what I’ve read so far, I’m already in love with the upcoming plugins system. (yeah, maybe Ender compatible might be a thing?!)
Another fine side-effect of this »I accidentally … the DB«: All those non-maintained plugins are gone. finally. I’d rather read spam than sift through age old code authors have already forgotten about.
So yes, thank you!
This sounds like a tremendous and ambitious undertaking and I have to commend Mr. Sontag and the rest of the team for tackling this. Building it right into GitHub was an awesome move, well done. I’ll be happy to add my plugins once it ships.
You guys should change the “.local” to “.dev” in your instructions. ;)
Whatever you planned to do, just hurry up. I missed jquery plugins collection so bad :D
“Even more unfortunately, I didn’t back up the database before I began this process.”
In all frankness, this is the type of mistake that really shouldn’t be tolerated. I’m not talking about misconfiguring Drupal system or anything like that; I’m talking about the utter negligence of not even making a backup of an important system before performing bulk operations. While it’s a pretty funny story I guess in sort of a quirky way, it’s also pretty sad and pathetic. He should have taken responsibility and stepped away from the project.
As someone who just recently saw another increase in Spam activity and script kiddies I totally agree. Things have gotten so bad, that I (as many others) have had to tighten up security so much, real people find it difficult to get past the checks.
And things are getting even worse. Meanwhile bots went on to the strategy of posting non-ad (yet junky nonsense) entries to confuse spam-blockers. More sophisticated SEO buggers started to show up, who would post normal looking stuff, but hide the real links they wish to push in ICQ-Ids, Image-URLs and signatures, knowing it will show up under each newly posted material and bypass security, because signatures, user profiles and Messenger-Ids are not checked by spam-filters.
Not to mention the kids, that “play” with the spam-filters and make it a game to try to sneak past these. And unfortunately these kids got far more time to play than devs have time to clean up after them.
So: it all combined, fighting ~150 junk posts and more per day you eventually end up with no other choice but to either tighten up the checks beyond the level you usually consider to be fair (or sane), or take down the site entirely. Just to claim back your precious development time from fighting the spam.
Creating something new will keep the bots out for a while. I renamed, changed and updated scripts time and again – yet eventually in vein. The sad part of the story is: they will come back. Because there is money to be made. Plus unfortunately the bots have an easier way of getting their junk back online then real developers have.
As a funny side note: Newest (made by hand) spam of the week is a company advertising for a mass-posting spam-software. Most amusing part of the story is: at least THEIR software did never get past the spam checks. So I would truly recommend their product to any spammers out there.
Cheers!
True dfl, any time a backwards breaking bug enters into the codebase that coder should step aside.
Mistakes happen and luckily this project isn’t so ridiculous that its not tolerated.
dfl, I will be seeing Adam next week and will be sure to give him a stern talking to :p
Maintaining all the jQuery properties is hard work. There are a lot of servers and a lot of effort goes into it. A lot of *unpaid* effort with no thanks from most people ever. While steps are taken to ensure that things like this normally dont happen, sometimes stuff happens. I for one, thank Adam for all the hard work he does.
Though I should point out, as infra lead, that’s my responsibility – so if it’s anyone’s fault for not having a backup, that’d be mine.
@dfl I would’ve done it on purpose.
*shrugs*
What a wonderful *ahem* “accident”! I can’t count the number of times I’ve had a messy situation that I’ve been tempted to “accidentally” wipe out… Good for you! ;-)
good to know I’m not the only one who makes mistakes. Makes me feel way better about my recent trip to India (without a visa).
Kinda odd to go out and say “we’ve taken the repo offline because of spam” when it was in fact accidentally deleted…
That there has not been a backup of the db for at least a year on a site this prominent, dealing with a massive amount of user generated content, is shocking.
It might be true that the old system sucked and was spamfested and god knows what… but a good share of plugin maintainers have links pointing here as the primary source for their code contributions that were perfectly fine and not touched by the problems you mentioned.
Thus I find it slightly off to point at the shortcomings of the old repo in an attempt to downplay the major administration fubar.
Anyway – thank you for explaining the sudden and rather extreme “spam protection”… now I can stop scratching my head.
Done and done. Thanks for the update!
Helps if I paste the correct URL
Very saddened by all this guys, plugins was my daily haunt. :(
Perhaps repliers on here take heed and dont spam the blog ffs.
Not sure how github will ork, in comparison to the old system, but good luck.
> Ender.js is much more suited to client side libs I think, and is compatible with package.json! http://ender.no.de/
If your using package.json for these plugins there’s no good reason not to promote Ender.
Rather then writing your own package.json instructions just say it has to be ender compliant.
Not to beat a dead horse, but 5 years of plugins gone in an instant, not good. Like epically bad. Could you make the old backup available for posterity? I’m sure the new Github hotness will rock, but you can’t just write off a terrible mistake as an opportunity.
I understand why you’d want to use GitHub as a VCS and distribution system. However, even though you’ve apparently had a bad experience recently with Drupal, switching to WordPress seems like you’re moving one step forward and two steps back.
Not sure if you already thought of this, but thanks to the Internet Archive you can still access old versions of the plugins.jquery.com site from as recent as July 17th, 2011: http://web.archive.org/web/20110717030551/http://plugins.jquery.com/
The left-hand menu doesn’t work, but the links in the main content area do.
Anyways, maybe that could be useful for someone?
I’m excited for the new plugin site!
Oh wait, the links on the left-hand column do work after all, they just show a redirect page first.
jQuery team: Good idea: Contact the archive.org team and get the missing “backup” from them: http://web.archive.org/web/20110717030551/http://plugins.jquery.com/
I wouldn’t put the fault on a single person on the fact that the whole project has a site without any automatic daily or weekly backups !!!!
Does jQuery team use a hosting without daily backups and archiving of daily backups for at least a week ? Seriously ? It’s a pretty bad advertisement for your hoster too !
Raise money, donations, and use that to pay for or ask a hoster to sponsor you BACKED-UP hosting.
This is a serious issue that should be handled and answered by the jQuery board in charge. It’s a matter of trust into a seriously managed project.
– End of rant –
Well, happened, can’t blame an unpaid volunteer, but hey, jQuery team: take a lesson out of it, and now can you please confirm that all your sites are daily backed-up with weekly archives (including SaS hosted third party sites content) ?
This is possibly THE BEST thing that could of happen. All unmaintained plugins are gone, spam is gone, why complain?
Also, Github will serve as an additional filter for people who should not be writing plugins in the first place (it’s more difficult to set up git than to upload shite in a zip file).
This is a great news guys!!
oh adam… only a few people would be able to get away with this, and even fewer of them would do it in such a way :’)
Please.
Do not move your site to the GitHub
The site as it was, helped me find many useful plugins.
The site was fantastic, much better than GitHub.
It is easier to find jQuery plugins on Google than on GitHub.
If you start an empty database.
In less than three months you will have more than 1,000 plug-ins and will have removed all plugins without updates.
Think about it …
This is horrible… Github is nice for the really technical pple, but I dont understand how to download plugins anymore :((((((((((((((((((((((((
Guys, guys! Wait.. He is just telling that a NEW plugins website in under development and the source code is available at GitHub. The new website will be launching soon. It’s not like you are going to have to find plugins on github or whatsoever.
BEGIN;
DELETE FROM jquery_plugins WHERE 1=1;
COMMIT;
Happens to the best of us.
Amen. A fresh start is always good, especially after the mess Plugins have become.
Thanks for your honesty. A clean slate was painfully needed anyway.
what about:
http://web.archive.org/web/20110717030551/http://plugins.jquery.com/ maybe some data can be recovered.
well i saw Russ and Beat sorry for repost
… what an awful moment, right? In yo face, plugins database! The old plugins site had long since stopped being useful — I’m not sorry to see it go. Connecting it into github makes a lot more sense, given that it’ll ensure easy tracking of updates and that everything featured on the plugins site will be easily downloadable and available. Thanks for your hard work!
Will you provide the same experience and support for users that are not using Git(Hub)?
I’ve already a paid account on BitBucket and actually prefer Mercurial over Git since it has superior workflows, client and OS/IDE integration (at least on Windows). So I’m sure you understand that I’m not going to create a GitHub account (pay for it) and have a pretty bad user experience.
I would really want to see some repor sites as equitable solutions for developers, let devs choose which provider is best for their specific needs and preferences. Pushing us in the arms of an overhype company will (at least for me) not work.
Was your personal site hacked, Adam? I’m getting a redirect through a sweepstakes site, then a malicious software warning. Or is that another Adam J. Sontag, who programs in Boston?
Thanks for the honesty guys, good luck with the new site, can’t wait!
In your packaging format, you might want to make linking to a preview image required.
In my experience with the previous website, having no visual feedback whatsoever when browsing the plugins made it harder to search.
(Of course I understand that many plugins can not have their behaviour adequately explained in a static image.)
Very nice and honest. This is exactly how people should handle screwups! And this cloud has a pretty big silver lining.
I think this might be a blessing in disguise. I never used the old plugin site. I frequently came across spam and a lot old and unmaintained plugins (My own included). I hope this new Github system works out well.
I think it would be awesome if the next jQuery plugin site actually achieves the same level of usefulness as NodeJS’ NPM Repository search. It is instantaneous, clean, and fun.
“I say we take off and nuke the site from orbit. It’s the only way to be sure.”
Appreciate the honesty and have made a similar mistake myself. It’s the type of mistake you make only once. I’ll echo the sentiment of many here and agree that blowing the whole thing away isn’t the catastrophic event it may seem to be. A fresh start with a planned approach will be helpful for all.
Archive.org has saved me before and it might help here. I do think that it might be wise to resurrect the old, busted site if for no other reason than to help newcomers to jQuery.
Drupal? Why on earth would you bother with it in the first place. Good luck with the new site!
Shit happenz buddy ;)
We all make mistakes. But you could try to get some recent source from various developers. I could imagine that they could have some copys of the source ^^
I handpick some of the best and usable jQuery plugins and list them on http://bestofjquery.com/
I go through and check each plugin submitted and make sure they are solid and fit users needs.