Theme Reviews, Guidelines and Community

I’ve accidentally started a long discussion due to a small and fairly insignificant bug that turned out to be more than enough for several major theme authors, reviewers and theme lead devs to chime in and share their thoughts. I usually try to avoid those scenarios, but every now and then the tension needs to be released.

These are my observations, thoughts and ideas based on my experience over the past 2.5 years with the theme directory.

Theme Reviews in 2011-2012

I have joined the WPTRT in 2011, after doing a dozen reviews as a “candidate reviewer” before getting my status approved. It was exciting and challenging to join, there was a lot of work to be done, authors were occasionally getting anxious waiting for weeks and weeks to get their themes reviewed every single time.

Since then, numerous things happened. There were several batches of updates in the guidelines, procedural enhancements for getting new reviewers on board, and the average number of “waiting” themes in the queues have been shifting in different directions.

mid-2011 vs. end of 2013

For the past 2 years I have noticed few things in the overall process that have been updated one way or the other. I can’t confirm which of them were good or bad, or should we have other updates in the guidelines, but what I see as a user/developer and reviewer is the following:

  • now it’s sufficient to complete a single test review to get a reviewer status, whilst it took more than 10 “exams” (review-wise) two years ago
  • there were several small updates in the guidelines and few more things added to Theme-Check, nothing quite as major
  • the brilliant idea of the Theme Review Incentive program led to the first long-term position of the directory where themes are now getting approved in just a few days, and having just a couple of themes waiting for a review in every moment. Huge props to Chip and the admin team for that.

However, during the past 2 and a half years WordPress have been drastically improved in the CMS aspect and even in the Application Framework direction, while the theme regulations have been mostly focused on blogging.

My motivation

I have joined the team just as many other reviewers did back then – I was curious what are those mysterious coding practices that the theme directory demands and I wanted my newly submitted theme to be reviewed quickly, so my help was accelerating that process. It wasn’t trivial to get in, I had several people monitoring my actions and reviews and sending feedback, but eventually I got in and I learned tons of new things about theme development, best practices, the Settings API and much more. Even if you know development and you tend to know the WordPress concepts, there are tons of specifics and APIs that you might be unaware of, or handy functions making use of the templating engine and specific concepts just like child themes.

Subjectivity and “retro”

Two of my biggest concerns are subjective guidelines/regulations and not being on the top of the wave. There are way too many things to check for and any reviewer couldn’t quite verify every single one of them. One might say that each reviewer should be able to point out all the rules – I think that we pay too much attention to details that are insignificant and lose time, people, resources and themes due to that.

That leads to various issues, such as:

  • Themes getting rejected several times due to tiny details – this is a real culprit for enthusiast authors who refuse to apply anymore.
  • Some reviewers avoid to do many reviews – it is too much time and energy consuming to run through hundreds of guidelines instead of focusing on important details only.
  • Reviewers stepping on each others’ toes – the existence of “sufficient”-alike words in the unit test data requirements or the review document leads to people trying to re-review a theme after a reviewer has already submitted a comment. In my observation that would be one of the following two problems – the first reviewer is inexperienced and should be revised or so, or the second reviewer is being disrespectful and too pedantic for no apparent reason. Mistakes happen or one might miss a detail, but that’s more of an exception.
  • Theme authors with business themes refuse to submit them to the repository or fail to get in as all of the guidelines focus on the blogging site and forget that WordPress is currently a CMS and millions of people use WordPress for their business sites, eCommerce stores, portfolio sites, portals and so forth, and the repository is surprisingly empty when it comes to the number of non-blogging themes.

Credits

There’s another subject that people avoid to talk about (and I won’t get into long and detailed explanation as it would lead to flames, wars, trolling or whatever), but credit links and texts are also dodgy and quite non-established. People share different opinions on what is allowed for credit links (and where) depending on what they do for a living, and I’ve been working in the IT industry for more than 10 years now and definitely try to see the bigger picture for the sake of the community.

What I would consider as an appropriate credit link is – any link in footer and/or theme options page that links to the author, if the author is 1) a freelancer, or 2) a company, related to the IT industry – which includes web development, design, hosting, some PaaS/SaaS, theme market, podcast website and everything else online that is being a part of the community and trying to give back. I like the idea of giving back for the sake of giving back, but the truth is that people should pay bills, buy groceries and provide for families before they can afford to give back, and if the link is important and is some sort of approval for their work – sure, why not? It would benefit both the repository, the developer/company and the users, that is a triple win.

Some ideas

As I wrote earlier, the process has been pretty flexible and vivid, sometimes a review for a new theme could take a day, or more than two months. The new incentive is great, I also keep finding more and more people willing to help with theme reviews – I gave talks for the Theme Directory, themes and reviews at WordSesh last year, WordCamp Sofia this year and we got more than 20 people reviewing themes at the last 2 contribution days over the past 3 weeks which is literally awesome. It actually turns out that there are not enough themes for new reviewers to review, and just a year ago there were 100-200 themes waiting in the queues to be reviewed!

In order to boost the WordPress repository and push it in the right way, bring new reviewers, clean the process, invite new authors and embrace variety for themes, I would consider few ideas:

  1. design a new process with priorities – for example, outlining 20-30 regulations that are absolutely required and would definitely reject a theme. List 100 recommended items, non of them would be critical, but if more than 10 or 15 of these are missing, then the theme would be rejected too (otherwise only a note in the ticket would suffice).
  2. simplify and specify the requirements – I would refer to the existing guidelines as “the comprehensive list of all possible scenarios” but stick to a shorter document with the important things. Let the theme authors worry about just a bunch of details instead of trying to think about hundreds of them. Make the process easy for reviewers too, less hassle, quicker reviews, more joy for everyone in the process. Focus on security, critical issues and several important functions and make everything else “nice to have”.
  3. rely on ratings and feedback – as long as a theme does not expose security holes or introduce some nasty code (or the UI is broken on standard pages) – let it in. Focus on ratings and feedback. Give people the access to secure and fairly well-developed themes, where 80% of the work is done and reviews take 20% of the time. Let people test and judge proven themes and decide if they want to deviate or downvote themes that make no sense. Create a “top rating” filter or something for the best themes, but don’t limit thousands of decent themes.
  4. implement some control mechanisms for reviewers and review actions – what would be the process of getting new reviewers on board and dealing with old reviewers? I haven’t been active for some time due to the process complications, but I would love to take part in a bigger, process-driven update and deal with more and more themes. But if I fail to stick to the basic guidelines or I haven’t done reviews in 9-12 months, let’s flush the list and keep it active.
  5. business category – go outside and ask your friends who use WordPress but don’t do development if they would build their business website with a theme from WordPress.org. I would guess that less than 10% are willing to spend enough time to find a theme that eventually looks good for anything else but a blog, or has a non-blog screenshot and definition in the first place. Build a business category for business themes, or even get further and accept WooCommerce/Easy Digital Downloads themes, One Page themes and others that people keep searching for over and over. Make the directory useful not only for newbie bloggers, but for real WordPress users. Show this new directory to all those people who claim that WordPress is still a blogging platform and wouldn’t work for a real website.
  6. listen to the users and the authors – ask the users and the authors what would they like to see in the directory, and get improved. Take notes on that, see what the regular WordPress user wants and looks for. Help them find what they need or educate them on what is good for them. Either way we tend to forget how does the regular user thinks just as developers often forget how would the average user use their product.
  7. more exposure for the directory – whenever this is ready, spend some time sharing this directory. The blogging constraints above are the reason for people to spend all of their time in theme markets. This is definitely not a bad thing, but there are so many because the original free directory doesn’t provide enough non-blogging content and all the other free themes directories are hacked. Let’s fix that.

8 thoughts on “Theme Reviews, Guidelines and Community”

  1. Ray Gulick says: November 13, 2013 at 4:45 pm

    Great to hear someone on the inside of the review process acknowledge that it’s skewed heavily toward blog themes! I hope this changes dramatically over the next several months.

  2. Mario Peshev says: November 13, 2013 at 5:21 pm

    Thanks for your comment, Ray.

    I like the latest updates from the admins team – http://make.wordpress.org/themes/2013/11/12/theme-unit-tests-recommended/ – so I trust that the business use could be discussed as well in a few months, when the reviewing process is better optimized.

  3. Chip Bennett says: November 14, 2013 at 4:03 pm

    Personally, I don’t think the overall Theme Review Guidelines are at all biased toward blogging or against “business” Themes or any other. (I would agree that the Theme Unit Tests, specifically, are targeted at blog-based Themes.)

    The Guidelines include code quality, separation of presentation and functionality, proper implementation of Theme features, proper use of template tags and hooks, support for core-generated CSS classes, proper implementation of template files, security and privacy, policy matters (licensing, trademarks, Theme name, credit links, up-sell Themes), documentation, obsolescence, and accessibility. None of those things are specific to any particular use case for Themes. They apply equally to blogs and to “CMS”/business sites.

    The one aspect that stood out from that list was the Theme Unit Tests, which helped justify why the TUT criteria really probably should be recommended rather than required.

    Some of the other issues, such as tickets being closed for minor issues, have largely been addressed by infrastructure changes that have allowed us to change our process, such that reviews are continued in the same ticket, with more feedback/interaction between developer and reviewer.

    I have less of a problem with multiple reviewers commenting on a given ticket. We have a large group of reviewers, with a wide spectrum of review experience and development expertise – and we are a team. Working together should be the norm. What is important, however, is ensuring that we conduct ourselves respectfully (especially when commenting in other reviewers’ tickets), and that in all cases, we communicate to the developer the things that are *required* in order to get the Theme approved. If those aren’t happening, please CC one of the admins on the ticket, so that we can intervene.

    Back to “business”/CMS Themes: the only real difference for most of these Themes is that they omit support for a blog posts index. This has been brought up often, and the response is always the same: implementing support for a blog posts index is so trivial that it simply doesn’t warrant an exception to the Guidelines – and users that have no need for blog posts would be completely unaffected, while allowing that same Theme to be used by a much larger pool of end users.

    Credit links are tough; the issues should be obvious. They’re a huge magnet for spam, and for sub-par Themes being submitted just to get footer credit links out into the wild. Is it conceivable that the current stance prevents a small percentage of legitimate sites being used as a credit link? Sure. But the alternative is a deluge of obviously inappropriate sites. And in the end: our focus is on what is best for the *end users*, and Theme submission should first and foremost come from a desire to contribute to the community. While it is appropriate to give credit for such contributions, it should never come at the expense of protecting end users. It’s a touchy subject, and a fine line to walk trying to handle credit links correctly.

    We already do have a “short list” of requirements, and clarifying the Guidelines and making them more approachable is an ongoing process.

    The WPORG ratings/reviews/feedback system is simply broken, and is hardly reliable or useful to developers, much less to the Theme Review Team.

    The primary control system we have in place currently is that Themes marked as “approved” undergo a final “quality audit” by one of the admins, before being resolved as “live” and pushed to the directory. While it introduces a bit of delay, it has been incredibly helpful in identifying deficiencies and inconsistencies in reviews – which we then turn into opportunities to educate/clarify/emphasize certain aspects of the Guidelines. So far, it has been the best balance we’ve found between a smaller team of better/more-consistently trained reviewers, and ensuring that the review queue stays manageable. To give an idea of scale: we’ve had almost 350 new Themes added to the directory in the past four months. We never could have done that in the past.

    There’s nothing in the Guidelines that prohibit “business” Themes, or one-page Themes, or WooCommerce/EDD Themes, etc. If there are minor things that need exceptions (such as allowing the screenshot to display a static front page), developers can always ask for such exceptions.

    If you get feedback from users, please pass it along. Whenever I speak at a WordCamp, I always solicit any Theme/Theme Review/Theme Directory feeback from both developers and end users. It’s invaluable, and the more we get, the more we can improve.

  4. Mario Peshev says: November 14, 2013 at 4:38 pm

    Hey Chip,

    I’d like to thank you for your thorough comment, it’s highly appreciated.

    I believe that the latest changes for the Theme Unit Test data requirements being diminished to “recommended” is crucial and would help a lot, just as the Incentive did a few months ago. Without cardinal decisions though things would have been exactly where they were a year ago.

    Given these updates in question, I would like to see how is that going for both developers, reviewers and the end themes getting submitted. It is a massive change and we have yet to see how would it reflect the existing process and the end quality of some themes that don’t follow the Theme Unit Test requirements anymore.

    In my opinion those requirements were keeping users from submitting non-blog themes as their (the authors’) initial purpose was not to comply with every blog use case when that is not the major goal of a theme. Talking about subjectivity, I remember a comment by the reviewer of my first theme who said that readability was an issue due to the center text alignment of my theme. I don’t mind the opinion (and it was not a ‘required’ mark) but that sort of attitude is what needs to be avoided IMO, as the theme got several thousand downloads by users who were unable to find other center-aligned themes.

    Encouragement as part of the community feeling is the essential element that might get some people think that TRT is “an army of dictators” instead of “mentors” when it comes to getting a theme approved. I hope that would make sense, analogy-wise.

    Credit links is a tough matter, which is the reason I didn’t want to go deeper. I have observed various situations where companies pay freelance designers and developers to build themes for their sites (or just for free) and want to release them to the public. The designer and the freelancer are not from the same company, the site is built for the company, which pays the money. It would definitely make sense for me to get that link.

    The same goes with PaaS and SaaS businesses and hosting vendors (pretty much most of my clients) who run multisites or services and built dozens of themes. For me (that’s completely unbiased opinion as I leave commercial links to various places most of the times when I’m allowed to remove them, see the footer here) the link to a legitimate and recognized service in the community is just a statement who is in charge of it, especially when it’s not a one-man/company work.

    Right now I’m happy with the majority of the updates and hadn’t expected as much to get changed in the right direction, so I’ll just be around and continue to help and share the mission to theme developers around the world. The jibber-jabber here is striving to improve the process more and more, and it would be an ongoing process as awesome changes could be applied all the time, no matter how good a product is.

  5. Donna Fontenot says: November 15, 2013 at 11:58 pm

    “more exposure for the directory”… Making the directory better and more user-friendly would be a good direction to take before promoting it more. Most of the Previews are useless, and while I appreciate the attempt at the tag filter interface, it’s pretty lame.

  6. Mario Peshev says: November 16, 2013 at 12:01 am

    Hey Donna, thanks for commenting here.

    There is a Meta Trac for similar requests – http://meta.trac.wordpress.org/report/1. Otto has been working on this since WordCamp San Francisco this year, feel free to submit any feature and UI-related ideas there.

Leave a Reply

Your email address will not be published. Required fields are marked *