UPDATE: Here’s the transcript!
You know those posts I write from time to time where I show you how I’ve screwed up something here in the hopes that you won’t repeat my mistakes? Yeah, we’re about to have another episode in that series…
So this has been a rough year for this blog. In February the blog was hacked, and it took me about 3 months to finally get rid of all the gremlins, with the help of Sucuri. Apparently, I had let hackers get access to this blog because I hadn’t kept my plugins and WordPress updated quickly enough. Typically I would wait a week or two till I had several plugins that needed to be updated, then update them all at once. BIG mistake, as I learned the hard way. Often, one of the main reasons that a plugin updates is to address an existing vulnerability! So always update your plugins as soon as possible!
Anyway, one of the things I did from Feb-May when we were dealing with the malware is I went apeshit getting security plugins. Anything that promised to restrict this or block that, I got it. At once time I think I had like 5 different security plugins.
So on Wednesday, July 23st I noticed that traffic fell about 30% over the previous day. Now this isn’t a huge deal and happens from time to time. I was traveling then from Y’all Connect and hadn’t been posting as much over the last few days, so I assumed that was the reason for the decline in traffic. I should have dug into Google Analytics at this point to figure out what the problem was, but I just assumed there wasn’t a problem.
Then the next day on Thursday the 24th, traffic fell another 20%, then another 20% on Friday. Something was definitely wrong.
On Friday the 25th I finally dug into Google Analytics and found the problem. My search traffic was falling like a rock:
Search traffic on Monday, July the 22nd was 543 visitors, but by Saturday, July 26th that number had fallen to a lowly 34 visitors.
What the hell happened?
It seems that one of those many security plugins I had added that I wanted to block and restrict everything was actually blocking Google from crawling my site! During #Blogchat on July 21st, a few members had mentioned the advantages of creating a Sitemap for your blog, and how that would help you with search. So I did that on Sunday night via a plugin, and apparently, that somehow triggered the security plugin to start blocking Google from crawling the site (I am assuming here, this started happening almost immediately after I added the Sitemap).
I discovered this by going into my Google Webmaster Tools dashboard and I discovered the crawl errors (BTW you should set up an account for your site as well. It might seem like overkill but it’s worth it). I also started digging into my crawl stats and realized that while there was a huge spike in errors right after I added the Sitemap, that there had been a smaller amount that had consistently been there for months. Apparently, since I had started adding all those security plugins.
Sooooo….what I did was I started disabling plugins one by one and attempting to ‘Fetch as Google’ after each one. After disabling the first security plugin, suddenly the crawl errors disappeared! I did this on Saturday, July 27th. I immediately noticed that search traffic began to bounce back. By last Friday (Aug 2nd), search traffic was all the way up to 471 visitors for the day, the most search visitors ever for here on a Friday. I saw good search traffic yesterday as well, and I am hoping this continues from here out!
But the point to all of this is that:
1 – You need to be careful about adding plugins. Sometimes they can actually hurt your blog’s performance, so you need to understand exactly what they are doing. It’s tempting to just add a bunch of plugins to get access to cool new features, but they can sometimes cause more problems than they fix. And if you do add them make sure you keep them updated.
2 – Keep a close eye on your blog’s traffic and understand why changes are happening. If you don’t have it already, add Google Analytics to your blog. This will help you understand the traffic on your blog and track changes.
So tonight at #Blogchat we are going to discuss how to keep your blog clean and less cluttered. I am a digital packrat (and real-life one too), but I have learned the hard way this year to streamline everything as much as possible. You can follow #Blogchat tonight on Twitter starting at 8pm Central! And here’s the transcript so go ahead and save this for later!
Sounds like a nightmare! Glad you got everything back under control. Yikes!! Looking forward to your next blog and learning more. Now go and enjoy the rest of the weekend 🙂
Thank you Lori I *hope* I have everything fixed! There is SO much to learn! Hope you have a great weekend as well!
Take away all you know about your GA, it makes perfect sense that some security plug-in would actually stop Google from crawling because “security” is meant to build a wall around your site. So a badly built security plug-in would block “good” attempts at access too. Logically, if you have five such plug-ins, you would increase your chances of running into a badly-built one. I think that it would be interesting to your people on #blogchat to know which plug-in was doing this, and if this is a plug-in made by someone reputable, it is probably also worth contacting them and telling them what happened because they may be grateful for the information. On the other hand, if they’re just a hit-or-miss plug-in company that is fly-by-night, other bloggers should know to to avoid them.
We all love when we learn from your errors, Mack. 🙂
Linda the reason why I didn’t name the plugin is because it’s a well-respected security plugin (that’s why I added it in the first place), and I’m not certain it was the problem. It’s possible that it was actually a problem with the Sitemap plugin that I added that somehow triggered the security plugin to change its settings. I’m not sure so I didn’t want to call out one plugin it when it might have been something else.
The one thing I know for sure is when I disabled the security plugin, that Google could then crawl the site as normal.
BTW it wasn’t Sucuri, which is fabulous and well worth the $89 a year it costs.
Great tips Matt! See you on #BlogChat!
I’m sorry …. Mack not Matt.