Inaccurate Stats Have Been Corrected

It gives me no joy whatsoever to have to post this.

A little over 100 plugins recently were impacted by a stats gathering change. This means those plugins had their active install stats seemingly adjusted downward.

We understand this was painful for a number of developers and we held off on announcing this as we were still doing a bit of triage and making sure it was blocked. We are sorry about that confusion.

What happened?

Recently, it was pointed out that the active install counts of several plugins appeared to be inflated artificially. When we looked at the raw data, we found that this was correct for roughly 100+ plugins; fake update data was being sent to us.

This is not unusual, it’s happened before, although people are usually much more blatant about it, which is why it took a long time to notice it. In any case, we adjusted our stats mechanisms to ignore these, and so those 100+ plugins will have seen a drop of around ~8000 installs.

As the data was being faked before, this new count is more accurate. But it doesn’t change the old counts, and we can’t redo those counts as we don’t store that raw data for more than 2 days. 

@Otto42

Will this happen again?

Probably. This specific attack won’t, as the folks with server power on WordPress.orgWordPress.org The community site where WordPress code is created and shared by the users. This is where you can download the source code for WordPress core, plugins and themes as well as the central location for community conversations and organization. https://wordpress.org/ are outstanding. However about once every other year someone tries to do stuff like this. We usually catch on to them a little faster and blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. them. Now that we know about this one, we’ll add it to the list of things to monitor and block. But yes, people love munging with stats, they’ll certainly try it again.

Why didn’t you post right away?

We were asked not to while people were still working on stopping it, and then we didn’t want to while we were investigating the root cause. Basically we didn’t want to announce it until we had all the facts.

Can you tell us exactly what happened?

No, we cannot. We’ve learned that telling people exactly how we caught what they did, or even just what they did in details, leads to them doing it again in a slightly more clever way. Right now, they have no idea how we solved it, and that’s just fine.

How many users did I lose?

The Active Install count for affected plugins would be decreased by somewhere between 1 and 8 thousand. Depends on the pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party. And yes, we know that’s a galling number.

Were any of them valid users?

We can give you 100% assurance that no, they were not. The counts were inflated, so the number it shows now is much closer to the true active install count.

My plugin was impacted – am I in trouble?

No. If you were the culprit, your plugin would be already closed, your account banned, and you got a stern email from us about why you were banned for doing that, and you’re not welcome here anymore. If all of those aren’t true, we know you didn’t do it, and you have not a single thing to worry about.

Will a big drop in usage hurt my plugin popularity?

Not really, no. Please keep an eye on the big picture for a moment:

  • If you wants stats to be useful then they have to be accurate, right? Well, we fixed that.
  • The majority of end users don’t look at the charts that actually show the massive drop, they just look at the full usage count. And no, they don’t remember what you had yesterday.
  • You can point people here to explain “Someone else was a right prat and messed up stats for a lot of us.”

In the long run, this will even out and no one will notice. If you’re worried about your popularity, make sure you have a good readme that explains why someone wants the plugin and how to use it. That will help you much more than numbers or charts.

Did this impact historical data?

You mean from last month? Yes. Sadly. It’s been going on a while, like for most of the year. We don’t keep old stats like that in a manner that allows us to clean this up, so that’s why it looks like you had a big drop. At best we could force edit everyone impacted and drop them by X amount going back to when we think this started, but that doesn’t really change much, it just moves the weird needle back so it looks like a month or whatever ago, you had a massive drop.

There’s also the fact that the climb was a slow creep. We know the end volume of fake usage only because we saw the drop like you did. We could guess at how much it grew a month that was fake, but you run a higher risk of looking worse, like you were loosing 100s of users a month for a year.

Finally … asking us to manually edit your stats is a pretty terrible precedent. We don’t do that. We should never do that.

Why don’t you keep old data?

Two reasons: Privacy and size. We delete tracking data for your privacy, but also because with millions of sites out there, it’s heckin’ huge! Like “What comes after Petabytes?” huge. (Answer: exabyte, now you know.)

Can you undo this?

According to what I’ve been told, no. By blocking the fake data source, the stats automatically adjusted. The only way anyone would possibly be able to revert it would be to restore the fake data. We feel that is a terrible suggestion, as that would be intentionally lying to your users.

Who did this?

We are not about to disclose that. It’s being handled, and we are not in the business of dog-shaming people, nor encouraging mob-mentality to attack them.

If I didn’t do it, why am I being punished?

You’re not. Your plugin stats changed when we blocked the cause for the inaccurate counts. No one on WordPress.org has manually adjusted numbers. Basically we said “data like this is invalid” and when the counter recounted, which happens every day, those plugins were impacted.

This isn’t fair!

It’s equitable. Everyone who had their stats incorrectly inflated were corrected when we removed the data source.

I have some suggestions and ideas about how to fix this, where can I post those?

I am so glad you asked! The best way is to join us to be part of the ongoing solutions! And the easiest way to do that would be to come on over to help the META team. See, plugin reviews is just plugin reviews. But MetaMeta Meta is a term that refers to the inside workings of a group. For us, this is the team that works on internal WordPress sites like WordCamp Central and Make WordPress.? They do the heavy lifting of making the WordPress.org experience better for everyone. And, perhaps not shockingly at all, it’s mostly PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. and JS. Yes, that’s right, WordPress.org runs on WordPress!

Meta has a meeting every other week in #meta on Slack. You can keep tabs on all meetings via https://make.wordpress.org/meetings/

Also if you have a fully formed idea, that you think is a good proposal, head over to https://meta.trac.wordpress.org/ and make a ticket. If you have detailed screenshots and example code, all the better.

#statistics

2019 Insights

There’s been a lot of quiet change going on for Plugins, so now is as good a time as any to get into it!

If you’re interested in any details missing, leave a comment. I do ask you try not to speculate too much into the why’s and wherefores of what people do with plugins. I’ve been at this a while, and the one thing I can promise is people do weird things.

New Email System

We finally migrated off of the old system and on to HelpScoutHelp Scout A 3rd party service we use to process emails for plugin reviews. in March, which allows us the ability to sort and organize emails into teams. It also lets us properly filterFilter Filters are one of the two types of Hooks https://codex.wordpress.org/Plugin_API/Hooks. They provide a way for functions to modify data of other functions. They are the counterpart to Actions. Unlike Actions, filters are meant to work in an isolated manner, and should never have side effects such as affecting global variables and output. bad actors so not everyone has to deal with them. We make heavy use of automated filters now, which has let us do the impossible …

New Team Members

We onboarded two new team members in November and have been easing them in to the weird workload of Plugins. They’ve been instrumental in sorting out what filters and team assignments do and don’t work well for Plugins.

New Tools

I’ve been using a new bash script to expedite scanning plugins. While we’d love to use WPCSWPCS The collection of PHP_CodeSniffer rules (sniffs) used to format and validate PHP code developed for WordPress according to the WordPress Coding Standards. May also be an acronym referring to the Accessibility, PHP, JavaScript, CSS, HTML, etc. coding standards as published in the WordPress Coding Standards Handbook. (and I personally recommend it to for everyone), even with a heavily parred down version it hasn’t quite met our needs. The goal for next year is to move the bash script into a PHPPHP PHP (recursive acronym for PHP: Hypertext Preprocessor) is a widely-used open source general-purpose scripting language that is especially suited for web development and can be embedded into HTML. http://php.net/manual/en/intro-whatis.php. pluginPlugin A plugin is a piece of software containing a group of functions that can be added to a WordPress website. They can extend functionality or add new features to your WordPress websites. WordPress plugins are written in the PHP programming language and integrate seamlessly with WordPress. These can be free in the WordPress.org Plugin Directory https://wordpress.org/plugins/ or can be cost-based plugin from a third-party we can use to automate a lot more.

New Replies

Our saved replies (the standard ones you get for closures and reviews) have all been cleaned up, spellchecked, and formatted for easier reading. Now, when you get an alert that your plugin has been closed, we attempt to direct you on exactly how to resolve the issues. This is still a bit of a work in progress, but we’ve made great strides on consistent tone and softer language.

New Restrictions

Sadly as many people found out, we got dinged hard by some trademark owners, and are taking action against people who violate trademarks. Around 1000 plugins were closed due to that, and it’s one of those things we can’t protect you from. We’ve changed the plugin uploader for new submissions to blockBlock Block is the abstract term used to describe units of markup that, composed together, form the content or layout of a webpage using the WordPress editor. The idea combines concepts of what in the past may have achieved with shortcodes, custom HTML, and embed discovery into a single consistent API and user experience. a lot of that.

Remember the basic rule: If it’s not your company/product/library, don’t begin your plugin Display Name or permalink with it!

(Trademark owners: Please ask the developer to changes things before coming to us. Communication will help everyone.)

The Stats!

A lot of people like this part. Here’s the overall outlook from 2019:

Chart showing the Requested, Rejected, Closed, Approved, and Pending plugins each week for 2019.

And in a slightly more consumable summary table:

RequestedRejectedClosedApprovedPending
Most in a week194109480118718
Least in a week1292925527
Average1612511776623
Year to Date8048122160383836N/A

We’ve had 1000 more plugins submitted in 2019 than 2018, however the Rejected and Approved numbers only went up by 100.

So where are the extra 800 plugins? On average, pending plugins did go down but only by about 25 a week. Most of the missing counts are there, but they’re also in the dreaded “Closed” section.

A higher than expected number of developers have submitted plugins for review and then asked them to be closed within a 6 month timeframe. This has led to us pushing back on people and making notes in their accounts about that kind of behavior. There hasn’t yet been a common thread to why that’s happening, so we’re keeping an eye out.

HelpScout Overall

HelpScout also helpfully provides their own statistics for how much we used them. This is just since March when we switched over:

  • Customers: 6665
  • Conversations per Day: 35
  • Busiest Day: Thursday
  • Email Conversations: 12,829
  • Messages Received: 17,439
  • Replies Sent: 18,931
  • Emails Created: 6650
  • Resolved: 6642
  • Resolved on First Reply: 31%
  • Closed: 11,818

HelpScout Saved Replies

We make heavy use of Saved Replies to speed up reviews and processing. These were brought in to use in chunks, and I’m omitting the exact numbers. They won’t do you any good to know we sent 2,679 “Approval after send” emails when you realize we also only sent 628 “Intro to new Review”. All that means is we pulled in the Approval email first. Next year these stats will be more useful.

All that said, I think having a look at what the most common sorts of issues are might be a little enlightening. Everything is ordered from most use to least.

Closed and Warned

These emails are sent out when a plugin is closed or the developer needs to be warned about issues/behavior.

  • Closed: Trademark Abuse (All)
  • Closed: Removal Request Completed
  • Closed: Security Exploit
  • Warning: Sockpuppets
  • Warning: Trademark Violation
  • Notice: Closed Becuase Email Bounced
  • Warning: Security Issue (NOT CLOSED)
  • Closed: General Guideline Violation

Reviews

All these emails are sent when a plugin is being reviewed.

  • Approval: Approval after send
  • Review: End Of Review (goes at the end of all reviews)
  • Review: Intro to new review (all new reviews start here)
  • Review: Please sanitize, escape, and validate your POST calls
  • Review: Generic function/class/define names
  • Review: Incomplete Readme
  • Review: Including your own CURL code
  • Review: Not using wp_enqueue commands
  • Review: Calling remote files (js, css, images, etc)
  • Review: Including Libraries Already In WP CoreCore Core is the set of software required to run WordPress. The Core Development Team builds WordPress. (i.e. jquery)
  • Review: Calling file locations poorly (also hardcoding in paths)
  • Review: Including out of date libraries
  • Review: Undocumented use of a 3rd Party or external service
  • Review: Not using Nonces and/or checking permissions
  • Review: Using file_get_contents on remote files
  • Review: Calling core loading files directly (wp-config, wp-load, wp-blog- etc etc)
  • Review: Display Name infringes on trademarks (slug is fine)
  • Review: Using esc_ to sanitize (not esc_url)

Pended

A pended plugin is one we stop before even reviewing the code. This usually happens because someone’s infringing on trademarks, or using a personal account to submit a company owned plugin.

  • Pended: Name Infringes on Trademarks (slug and name need to be changed)
  • Pended: Never replied to previous review (was rejected)
  • Pended: Not Official Owner

Rejected

This should give you an idea of why plugins are rejected. Top of the list? People who don’t reply.

  • Rejected: Review never completed within 6 months
  • Rejected: Not Your Plugin (Tried to upload vs host)
  • Rejected: Generic for plugins we’re just not hosting
  • Rejected: Framework or Library Plugins
  • Rejected: New/renamed version of their own plugin

Miscellanous

The rest of the emails are lumped together. You’ll notice we have prefixes to what each email is. That helps us find them faster.

  • Notice: Plugin Restored
  • Reply: Plugin Slug Renamed
  • Reply: Rescan (Plugins must be checked before being reopened)
  • Thank You: Security Report
  • Thank You: Guideline Report
  • Reply: Don’t call people ‘sir’
  • Thank You: Generic, Will Review
  • Notice: AutoReply Sucks
  • Notice: Already Mailed Review
  • Approved: Resend Approval
  • Question: Why Close?
  • Reply: Cannot Rename Plugins (for people who email RIGHT after approval)

#statistics, #year-in-review