342 – Thomas J Raef on protecting your website. Security mini series 3/4

[00:00:00] Nathan Wrigley: Hello, and welcome once again to the WP Builds podcast, you've reached episode number 342 and titled Thomas J Raef on protecting your website security mini series three or four. It was published on Thursday, the 21st of September, 2023. My name's Nathan Wrigley, and I'll be joined by Thomas in a few moments.

But before then a few little bits of housekeeping, we are well in the middle of the page builder summit. It began on Monday. Hopefully you've been attending. It's completely free, but if you want to buy the power pack, which allows you to view all of the fabulous content from that event, then probably this is the best time to do it. Because we're on Thursday, there's only one day left, the networking sessions tomorrow and whatever remains of the presentations for today. So head over to pagebuildersummit.com and you can sign up and for a very small investments, you will be able to make your way through all of the different presentations, whenever you like in the future. It's called the power pack, pagebuildersummit.com.

The WP Builds podcast is brought to you today by GoDaddy Pro. GoDaddy Pro the home of a managed WordPress hosting that includes free domain, SSL and 24 7 support. Bundle that with The Hub by GoDaddy pro to unlock more free benefits to manage multiple sites in one place, invoice clients and get 30% off new purchases. You can find out more by heading to go.me forward slash WP Builds. That's go.me forward slash WP Builds. And sincere honest, thanks to GoDaddy Pro for their support. Keeping the lights on over at WP Builds.

Okay, what have we got for you today? It's the third of our four-part security mini series. Prior to that, the other episodes featured Calvin Alkan as well as Akshat Choudhary. This is Thomas from, we watch your website giving his impression of the state of WordPress security.

I did the spiel before, but I'm going to do it again in order to not put anybody's noses out of joint and to make it as fair as possible. I recorded four episodes with Calvin, Akshat, Thomas, and in a future episode, Dan Knauss from SolidWP. They all have that chance to explain the malware madness as we'll get into a little bit later. How WordPress security plugins do or don't, depending on your point of view, protect your WordPress website.

Thomas is giving his opinion today and we go into loads of ground. We talk about such subjects as security plugin, and cleanup processes, previous attacks, challenges, and vulnerabilities, security features, Thomas' expertise, protecting your WordPress by blocking IPs, the motivations of hackers, and so much more.

So there'll be one more episode in this series, but if you have a comment, I would really appreciate it. If rather than going on social media, you actually went to the WP Builds website. Search for this episode, number 342. And leave us a comment there. That will be lovely. I hope that you enjoy it.

I am joined on the podcast today by Thomas Raef. How you doing, Thomas?

[00:03:41] Thomas J. Raef: doing well. Thank you, Nathan.

[00:03:42] Nathan Wrigley: Very nice to have you with us. Thomas is joining us all the way across the Atlantic from where I am. He's in Chicago. I am in the UK.

We are going to talk about the the very thorny topic of security and some recent news that there has been in the security landscape ecosystem, if you like, in the WordPress world. It's going to touch upon a... Few topic areas. We'll get into all of that, but I do want to just preface this in that it's gonna be part of almost like a three part podcast series, so there will be different opinions in this three part podcast series.

Thomas is about to explain what he has found and back it up with some data, but there are some other. Other podcast episodes to listen to, and I will point in the show notes to all of those so that you can get the different perspectives. I just want to make that clear. I hope that's okay, Thomas. Given that we're going to be talking about something deeply technical today I'm going to be, firstly, I'm going to be in over my head.

So I'm hoping that you'll be able to guide somebody like me through it. But also I think it's terribly important that we get to understand why we should. Trust what you say. What are your credentials? So that we've got some kind of understanding, some orientation for why, what you say matters. So would you mind just giving us your backstory?

It could be a WordPress backstory. It might be that you've, what jobs you've worked in and so on. So yeah, just let us know a little bit about you and how come we should trust what you say in the realms of security.

[00:05:09] Thomas J. Raef: Okay. Yeah. I got years ago, oh five, 2005. I started creating a it's a firewall device that I was going to market to small businesses. It was a small business it consultant back then. And so it was Linux based, but one of the things I wanted to do is make sure that PCs inside of the firewall protection we're not going to get hit with viruses because so many of these small businesses were not paying for antivirus protection at that time.

So anyway so that led me down to this Linux program called Provoxy and what it allowed you to do was basically filter. what was coming to the browser from the internet. Because this this device I was creating was like a gateway. So basically I started like looking at what kind of malware the hackers were distributing on websites so that I could block all that before it got to the end users browser.

And so that led me into a whole bunch of research and I found this website back then called badwarebusters. org. It's no longer around. Somebody said that the domain's available now,

[00:06:40] Nathan Wrigley: It's a great domain.

[00:06:42] Thomas J. Raef: Yeah, it was but it was started by a guy named Max Weinstein, who is a Harvard grad. I think he now, I believe he still works for Sophos the antivirus company now.

And so he started it, he had Google's backing Google actually if people were in any of the Google forums and they said something about malware on their website, Google would send them to Badware Busters where a bunch of us volunteers would try and help you for no charge just so we could learn more, we were experienced with malware we could help you out.

And one of the early in that group with me was David Sid who started Securi along with Tony Perez. So David and I were the top two. Volunteers in the group. So after you help somebody, they give you like a star rating. And David and I had, I don't know, I don't even know how many stars, but we were like way up there.

So anyway that led, that got me more into website malware. And so I just decided, you know what, I'm going to drop the whole security device and just focus on website malware. It seems like there's a huge need. That, like I said, that was like 2006, somewhere around there, and I started my business called it WeWatchYourWebsite because I lacked originality, it's people are like what do you do?

My company is

[00:08:23] Nathan Wrigley: website.

[00:08:23] Thomas J. Raef: What do you do? Pretty much, probably self explanatory, so anyway but I had I was blogging like every day on stuff I was finding and back then Joomla was just as popular as WordPress, which like most people today haven't even heard of Joomla, but so anyway, so I'm blogging every day about new infections that I found and.

One day I get a call from a guy and he says my boss's personal blog just got hit by what you wrote about yesterday. I'm very technical. What can you tell me? So we spent like an hour and a half on the phone. And the guy says, thanks, hangs up, calls me back the next day. He's Hey, the information you gave me was spot on.

And he's I see that you host with us. I'm like, Who are you? He's I'm Alex Lundquist. I'm a level three tech here at Bluehost. My boss is Matt Heaton, the founder of Bluehost, and he'd like to know if you'd like us to send you some business.

[00:09:29] Nathan Wrigley: Nice.

[00:09:30] Thomas J. Raef: yeah!

[00:09:32] Nathan Wrigley: No, thank you. Nice.

[00:09:33] Thomas J. Raef: Yeah, no I think I got laundry to do later today or something. So yeah, they just started, so they were telling all their tech support people, all their terms of service people, anybody calls in with malware, just send them to, we watch your website. So anyway, so that got the ball rolling. I had to automate.

As, as much as my process as I possibly could to handle this deluge of of business. And so since then till current day my services removed malware from over five and a half million websites.

[00:10:16] Nathan Wrigley: wow.

[00:10:17] Thomas J. Raef: So and I'm not saying we've seen it all because we see new stuff every day. Even this morning I was looking at stuff and new malware out there.

Yeah, like I I've stood the test of time and like I said, we, we literally have. removed malware from over five and a half million websites. And one of my things has always been even when I was a kid, my dad had me mowing the grass in our front yard and I had to know.

How that lawnmower worked, I pull this thing, it starts up. So he's take it apart, put it back together. So I did, so I'm the type that I need to know how things happen. So that led me to root cause analysis with infected websites. So I would look at log files until my eyes were bloodshot.

And. But that also gives me a great understanding of how hackers work how they try and hide things the reconnaissance attacks and so on and so forth. So anyway, so that's, to me, it validates, the fact that, people should. Or at least be curious as to what I have to say.

[00:11:42] Nathan Wrigley: yeah. That's great. Thank you. That really did paint a, an excellent picture of why we should listen to you. So I appreciate that. And just before we get into it, I think a lot of people listening to this podcast, my, my intuition is that we have a real broad spectrum of people listening to this podcast.

So there'll be people. Staying, hanging on every word that you say, understanding everything that you say, and equally there'll be people who, every time you use a new acronym, or you say something slightly off piste, will be, hang on a minute wait, Thomas what did that mean? So I think we should probably deal with those people first, and just, let's just paint the picture of what these people these people who are creating malware, whatever other name it may go by.

I want to just delve deeply in not deeply, fairly quickly, actually, into what the objective is there. Why are these people doing it? Because you hear this story all the time. My website, it's just an unimportant website. Nobody's coming after me because I don't do anything of any great importance.

I've got a local shop and it's frequented by 20 people a week. It's no big deal. Of course, the picture is not like that. Nobody is... Literally coming after your website. So the idea that because you believe it's small fry, the attackers think it's small fry might not necessarily correlate. So just give us an idea.

What are the incentives for people doing this? And I say people in the broadest sense of the word, because I imagined that in many cases, it's. It was long ago a person and it became a robot possibly, years, weeks, months ago. So just give us an idea of the incentives. What are these entities up to?

[00:13:22] Thomas J. Raef: It all boils down to money. And it's not that they're trying to steal credit cards from your e comm site, because like you said, maybe somebody just has a small, postcard type website, they just want to post some pictures, whatever hackers don't have any need to attack my website. Like you said but they can make money off of everybody's website. They can, they get paid a lot of times by unscrupulous online marketers. And what they do is they drive, they get, these hackers get paid to drive traffic to websites, which helps them increase their, their search engine rankings.

So what they'll do is anybody that comes to your website, they can redirect traffic from your site to one that they're getting paid to drive traffic to. Now that's a small thing, but they can also use any website to attack others. They're really good at hiding, this whole cloak and dagger.

They're really good at hiding themselves. They may be sitting in a cafe and they could be launching attacks on, websites and they don't want anybody to know where they're located. So they'll bounce those attacks off of an infected website.

And, a lot of people don't understand that the, how. massive a market that is for hackers, but they will actually infect a website, take control of it, and then group that together with maybe a hundred other websites and sell it on the dark web to somebody who wants to launch.

Bigger attacks on, a whole bunch of websites. There's, but the bottom line is they have so many ways of making money. It's like that guy years ago, Robert Allen, I think was the author's name, multiple streams of income.

[00:15:37] Nathan Wrigley: Yeah.

yeah.

[00:15:38] Thomas J. Raef: hackers all read it and, they've got more ways to make money than you could possibly imagine.

[00:15:44] Nathan Wrigley: Yeah. So broadly speaking, it's not really the website that they're bothered about. It's the fact that the website is sitting on. a computer, for want of a better word. It's sitting on resources. It's sitting on a machine which it can, if infected correctly, repurpose into other nefarious functions.

Like you said, it might be going out and visiting other websites so that we get fraudulent analysis of how many visits it's had. It could be, becoming part of a botnet, which is going around trying to take down other websites because somebody has potentially paid for a website to be removed from the internet for a period of time, those kinds of things.

So the front end, your WordPress website really isn't the point. It's the machine sitting behind that and the horsepower Is that broadly correct?

[00:16:36] Thomas J. Raef: Yeah. You nailed it on the head there. It's the resources that they want.

[00:16:40] Nathan Wrigley: Yeah okay, perfect. In the WordPress world then, we're quite familiar with logging into WordPress. And, when we log into WordPress Vanilla installer WordPress does a certain bunch of things. But, typically, people are... Not entirely satisfied with what WordPress will do in its vanilla state, the core of WordPress.

And so they throw in themes and they throw in various different plugins and it all gets lumped upon one thing upon another and I guess this is where things begin to potentially go wrong in that the more complexity we add to our WordPress website, the greater the footprint is for these attackers to, to get a foothold, a beach head, if you like.

Again, is that broadly true?

[00:17:27] Thomas J. Raef: Yes. Yeah. The more. plugins that you put on a WordPress site the greater your attack surface becomes. And it's, it's a fact of life.

[00:17:44] Nathan Wrigley: Yeah. Yeah. You are basically just putting more code in there and the quality or the recent update to that particular piece of code, that plugin could, and I guess that's one of the things we're going to maybe talk about in a minute. It's possible to leverage something wrong in one plugins code to then gain a beachhead into something else.

So it's not true to say that, just because I've got these plugins and they're all up to date. That doesn't mean that there isn't something lurking in there that will be discovered next week, which will then allow them to. Do something else, maybe leverage a problem in another plugin.

And so these sort of things get stacked upon one, one another. And the more that you've got, the more likely you are to get to get well, problems, let's just leave it at that. So we're recording this in the latter part of July, 2023. I'm not entirely sure when it will air, but that will give you at least some context of when we were recording it.

There's a, there's been a. Brouhaha as we say in the UK, there's been a storm of interest and the title that's been applied to it posthumously is malware madness. I suppose it fits perfectly. And we're going to talk around that. What, what has been going on? How has it been happening?

And you've got a. Post on your website, which I will link to in the show notes where you lay out some very specific numbers that, they're remarkably specific in that, there's no zeros anywhere. It all ends in a, a nine or a seven or a one or whatever. So you've obviously got some telemetry somewhere feeding all this back to you.

First of all, before we get into malware madness what is malware?

[00:19:28] Thomas J. Raef: Malware is it's short for malicious software. So it, anything malicious on your website that hackers have. put there with the intention of causing somebody some ill will. But yeah, basically malware is just short for malicious software.

[00:19:51] Nathan Wrigley: Okay. If you if you're listening to this and the website that Thomas runs, it's called wewatchyourwebsite. com, as he said, and the article was posted on the 14th of July, 2023. So just over a couple of weeks ago, and it's called how we identified nearly 150, 000 hacked WordPress sites in 60 days.

I guess probably you should paint that picture for us because that's the story that we're going to be talking about really. Okay. Obviously it can be read by anybody, but are you able to just paraphrase what was going on? What provoked you to write that passage? What was in there? Basically

[00:20:30] Thomas J. Raef: Um, yeah, basically what it was it was a timing of, 60 days. And I had seen where some security companies in the WordPress space had outlined their perspective of the the Elementor attack, basically the Elementor exploit on the Elementor add ons. exploit, I should be more specific, the plugin.

And I dug deep into our information because we monitor a lot of websites around the world and so I was able to gather all this information and fine tune it, run it through I'm a Python programmer so I ran it through pandas and spits out all this information. So essentially, back in, the the Elementor, Add ons, plug in exploit was announced in May, I think it was May 11th, as I recall.

And, but we had seen traffic by hackers looking for Elementor add on plug ins back in February of this year. So it's like hackers knew something and why the delay between February and May, I'm not, I don't know. I can speculate that maybe they were just trying to build up their list of sites that had this Elementor add on plug in.

But that's just purely speculation on my part. But once WordFence, and I have high respect for WordFence Once they announced the the exploit on their site hackers went crazy and they just started infecting sites because, hackers know what's going on in the market.

They read blogs, they read everything. So my guess is that when WordFence announced this, hackers know that WordFence and others They have a, like a free version and a paid version and the pay the people that pay money for the the plugin, they get the security rules and protection right away.

Like by the time it's announced, they already have the protection. The free, the people who use the free version. Which, and I don't have numbers of their distribution, but I imagine that there's probably more people using the free version of security plugins than there are people paying for them based on what we saw, because, like I said, the hackers knew that These people, majority of people weren't going to get the updated security rules for another 30 days yet.

So they went to town and they just started infecting sites like crazy. And unless people updated, which very few people update, on a daily basis. It's been one of my fights for a long time, but, yeah, so hackers took advantage of the timing and just unloaded all their resources on attacking websites.

They were attacking websites that didn't even have the Elementor add on plugin in it. They were just like, okay, anything that's a WordPress site, anything that's a website, we're just going to launch an attack on. And if it takes, if it doesn't we'll get enough of them. To make it worth our while.

Like I said, in that 60 day time span from beginning of May till the end of June we saw just under 150, 000 infected websites.

[00:24:48] Nathan Wrigley: so I'll just be clear. The plugin which is an add-on for the elemental plugin, so Element or the page builder, extremely popular in the WordPress space installed on multiple millions of websites. I'm led to understand there are Bolt on bits and pieces that you can add to Elementor.

And one of them is called essential add ons for Elementor. And that was the plugin in question. Now you explain in your piece, the numbers involved and there were lots and lots of numbers that you list out there. What I want to know is, and again, this is just asking about your credentials really, is how are you getting the data that you're getting?

I know that you provide a service, but do you have a, I don't know, a plugin installed or are you in communication with the server logs or did the, your clients hand them over to you? How do you how do we know that the data that you've got is reliable, I guess. good

[00:25:55] Thomas J. Raef: question. What we do is It's a, my story initially about my background, we hosted on, I hosted my blue, my website on Bluehost and that was a shared hosting account. But since then, the market has shifted to, there's more and more people on VPSs on dedicated servers on the cloud servers like, Vultr, DigitalOcean, those types of people.

Transcribed by https: otter. ai And so what we did is we created a service that we can install on a server that can't be tampered with by hackers because I'll try and explain this as cleanly as I

[00:26:49] Nathan Wrigley: Thank you, yeah.

[00:26:51] Thomas J. Raef: Our service is installed under root. Which is the, the master of every, of all permissions and everything else.

So you need root access in order to tamper with our system. installed on a server. A hacker comes into a WordPress site and they're pretty far removed from having root access. They have access to the files for that website and that's typically about it. Most of them, most hosting providers now isolate websites, to the point where you can't go from if I hack into one website on this server, I can't go to other websites on that same server without hacking them individually, if that makes sense.

So our stuff is installed as root, so you need root access in order to tamper with it. And that's part of the March Madness thing, which we'll touch on later. But. Is that our system can't be, you can't alter it. Hackers can't get at it and gum it up to the point where they can do whatever they want and not have to worry about being caught.

So we, we do a number of things. We monitor for file changes PHP and JavaScript files, any file in a website. We monitor on a live. Real time basis. So the second a file is changed or added to your website, our system knows about it. And then our system grabs it and analyzes it to see if it's malicious or not.

But we also stream your log files again in real time to our servers where they can be analyzed. And they get analyzed again in real time. We're And we're also watching your database so we know when people like an admin logs in, did they go to the theme editor and did they edit the header file? If so, that header file will now show up in our file monitoring system and so it's being grabbed and analyzed to see if anything malicious had happened.

So anyway, we're watching, we are watching your website and everything going on with it. And our Like our log file system this blows my mind and blows a lot of other people's minds too. But we can handle up to 20 million log entries per second.

[00:29:36] Nathan Wrigley: Good grief.

[00:29:38] Thomas J. Raef: Which when you do the math is like 1. 7 trillion a day.

[00:29:42] Nathan Wrigley: Yeah, that's

something.

[00:29:44] Thomas J. Raef: I mean we're seeing, what's going on. And, logs don't lie. A file change or added, that that's not a lie. We can see how it happens in the logs. So yeah, all this information is gathered from our system and people say, Oh, you should make these, access to your database.

A little more open so people can do research. I'm not into opening up, the information that we've got. It's just, not because I want to keep it all to myself, but just because I just don't want it to fall into the wrong hands. That's all. I'm

[00:30:26] Nathan Wrigley: can I just explore this just a little bit more deeply because I'm you've really piqued my interest here So in the WordPress world the way that we typically would interact with security products now That might be on like a hosting level. So we would tick a box say in our Hosting control panel, whatever that is and something's going on there, but we're also quite familiar with downloading and installing Plugins so that there's a whole suite of security plugins for WordPress and I want to tease out the distinction.

So you mentioned root and some people may have not been able to understand what's going on there, but with a WordPress plugin, you log into the UI of WordPress, you upload a plugin file and you activate it and you are now you're now using that suite of plugins. Yours is going to be a little bit different.

So I'm interested how you get that software onto these computers, because I'm guessing that at the beginning your offering is a little bit more is a little bit more difficult to implement. You can't just buy it, chuck in a plugin, click activate, and you're off to the races. I imagine there's a little bit more from your end, making sure that you've set that all up correctly because of.

The nature of the fact that it's got root access and it's living below the level of the WordPress websites, if you like.

[00:31:51] Thomas J. Raef: Yeah. We require root access. In order for us to install our system. And again, it's one of those things where we don't make it like a, we don't give you an installation script and you just put, throw that on your server and run it and hope for the best, it's it, we install it and then we test it.

So we do need root access and our system needs. constant root access to your web server in order to perform all of its functions and not be tampered

[00:32:29] Nathan Wrigley: Yeah, I think the sentence that I got most from there was you install it. So it's not for non technical people to just purchase something, upload it, click activate. This is I go to you, you get the keys for the root access. And then I leave you to do that work and to install it on my system on my behalf.

[00:32:53] Thomas J. Raef: Correct.

[00:32:55] Nathan Wrigley: That's interesting because that really does speak to the conversation that we're going to have possibly in a moment. Okay. So the data that you're gathering, it's streaming out of these websites in real time, so that gives you some confidence that the data that you've got is accurate. In other words, some of the stuff that we may get on in a minute is that things can be tampered with, but if yours is streaming to your service.

You can be confident, or at least you claim to be confident that what you, the data that you're looking at has not in fact been tampered with. It is immutable, it's arrived at your service, and you've got that locked down, right?

[00:33:33] Thomas J. Raef: Correct.

[00:33:34] Nathan Wrigley: okay, because that's an important part of the whole jigsaw puzzle as well.

And another thing I want to ask, just before we get into the whole thing, is you must have heuristics running. You must have something to say hang on a minute, this is a bit suspicious. Because if 1. 7, I think trillion bits of data potentially could arrive in a day, how do you make any sense out of that?

So presumably there are bits of your infrastructure, which is saying, wait. This is happening a lot or wait, this is new. We haven't seen this before. Is that part of what you do as well?

[00:34:04] Thomas J. Raef: Yes. Yeah. We're gathering live data on new attacks, new IP addresses that are attacking and a lot of people say, Oh, I don't, I don't block by IP address because it's unreliable. I beg to differ. I can show you the numbers, but yeah that's for another story. But so yeah we're gathering information and.

Some real quick, some incredibly interesting things are like IP addresses and the user agent, the. Just to explain what a user agent is, every browser has a user agent that it like when you go to a website, it records various things about about you, and one of them is like the the hardware that you're coming from, are you coming from a Mac or a Windows, what version and also what browser you're using and what version of that browser.

And but some of the interesting information is what, when hackers come to a website the combination of IP address and user agent is like, just, you're like, wow, really?

[00:35:19] Nathan Wrigley: Yeah.

[00:35:20] Thomas J. Raef: So it's, yeah, we find stuff, yeah, through our heuristics you can call it artificial intelligence, machine learning, it's got a whole bunch of different names.

But. It's all part of what we call our Paul Bunyan system, as Paul Bunyan was a logger I think, I believe a mythical logger, but I thought, I had heard that story from my dad, years and years ago. And so I was like, Oh, logging, what should we call this system?

Ah, Paul Bunyan.

[00:35:54] Nathan Wrigley: Like it.

[00:35:56] Thomas J. Raef: But yeah, Paul is built with all this knowledge. And and learning ability built into it. So that, yeah, as things are streaming, we can detect new potential points of interest. Like I said, with the essential the essential Elementor, we started seeing, The hackers scanning for the readme.

txt file, which is part of the, that plugin back in February. And you're like, why would somebody be scanning for a readme. txt file

[00:36:40] Nathan Wrigley: you, can I just ask, did you spot that at the time? Did you see that spike at the time or was it only upon looking back that connection was made or did you see that spike and think, Oh, that's curious, but nobody seems to be doing anything with it.

[00:36:53] Thomas J. Raef: Yeah we saw it back in February, but because like I said, I've been reading log files. People say, Oh you must have a really interesting life. You read log files,

Get my little glass of brandy and, sit by the fireside reading log files.

[00:37:13] Nathan Wrigley: bedtime story for the

[00:37:15] Thomas J. Raef: Yeah. But like I said, my, my mind just wants to understand why, and how. And so years ago when I started seeing different IP addresses looking for readme. txt file and some other innocuous files in a plug in, I was like, why would they be doing that?

And then you keep watching over days and weeks and stuff. And you're like, ah, they knew something was that there was a, there was an exploit in there and they're building up their list. Yeah. So it's one of the rules that, that I, that we created was looking for read me that, any Web access traffic that's looking for readme.

txt file because it's like, why would they be doing that?

[00:38:04] Nathan Wrigley: So again, just to clarify that I'm keeping up with you in this case, they're looking for the the file readme. txt, which is embedded as a part of the essential add ons for Elementor plugin. And so that spike began in February, 2023. And because that's public facing, it's really just a. Oh, if we can find that file, we know that plugin is there.

And so the supposition then is the hackers, the malware creators, knew there was an exploit to be exploited in that plugin. They'd figured that part of the puzzle out. And we're supposing that the quickest way to discover which websites have got that plugin installed is to just go and capture data about readme.

txt associated with it.

[00:38:49] Thomas J. Raef: Correct

[00:38:50] Nathan Wrigley: Okay. And so they built that up and it ramped up over time. And then what happened?

[00:38:57] Thomas J. Raef: and then I said when I don't Recall, I don't want to get this wrong, but I don't recall if it was WordFence that announced that exploit or if it was PatchStack. But one of, somebody announced the, that there was an exploit that had been patched, but you had to update in order to be protected. But one of them announced it, I believe it was on May 11th.

And from that point on once hackers knew the cat was out of the bag, they're just like, okay, time to light this up. And they just turned on their automated systems and started hacking.

[00:39:37] Nathan Wrigley: I see. Okay, that bit of the jigsaw puzzle I hadn't gotten. That's a very human part, isn't it? That's really interesting, because that is literally human beings who've been grabbing this readme. txt data, compiling what we suppose is tons and tons of information about which websites can be hacked.

Suddenly they see that other people know. That this exists. And so the window is closing. The door is starting to be shut. So it's time to just throw caution to the wind. No more time to wait. Get on and exploit the exploit, whatever it is right away, because pretty soon everybody's going to be updated and it'll be gone.

And all of that gathering of data will have been worthless. Interesting. Is it a little window into the hacker's mind here? That's fascinating. Yeah.

[00:40:25] Thomas J. Raef: Yeah. They're, like I said they're incredibly smart people. I hate to give them credit, but they are they're incredibly

[00:40:33] Nathan Wrigley: Yeah, so around this we'll get on to the whole other bits and pieces around this was there reasonable disclosure around this in other words when word fence or it could have been other people I'm going to say word fence because you thought it might have been word fence when they disclosed it.

Presumably the patch for the essential add ons for Elementor plugin was out at that point because typically a security vendor would want to give what's it called reasonable disclosure, something disclosure. Yeah. So they give them 90 days or something like that. And then they will then release the data that they've got into the wild once the patch has been secured.

Is that the case in this case? It was available, but maybe not everybody got updated.

[00:41:18] Thomas J. Raef: Yeah, it was available, but it wasn't pushed out, to everybody that had the that had that plug in. You had to, go into your site and do the plug in update in order to get it.

[00:41:33] Nathan Wrigley: Right, right. So this is the typical scenario that anybody listening to this podcast knows about, if you don't go into your website back end for a month or more, you'll come back in and virtually everything will need updating the and so it was so it is probably the case in many of these websites.

Okay, so this is all happening in. May, what then begins to happen after the beginning of May? What sort of dominoes do we start seeing falling?

[00:42:02] Thomas J. Raef: Well, At that point. Hackers were, it was almost like throwing some wounded fish into a pond of piranhas or something, they just

[00:42:16] Nathan Wrigley: I love that analogy.

[00:42:19] Thomas J. Raef: they just started going crazy. And so at that same time I'm not sure if it was the same group. I couldn't really. tell from the information that we gather.

They were also, they had also stolen the whole bunch of authentication cookies from websites or actually from desktop computers. So when you log into a WordPress site, it creates a, an authentication cookie on your. on your local computer. So that, if people have ever done this before, you log in to your to your WordPress.

And then when you're done, you just close the close out that the browser tab. You don't necessarily log out. You just close out the tab. Then if you go, Oh, wait, I forgot. I wanted to go back in and do something. You open that up again, go to WP admin and boom, it just logs you right in.

How does it do that? It's because you still have the authentication cookie on your computer, in your browser. But that can be stolen and I don't recall what the default is with WordPress. You'd think I would know this off the top of my head, but I think like a typical authentication cookie is like good for 30 days.

So anytime during that 30 days after you're in your system and you don't log out, hackers can steal the authentication cookie. And then put it into their browser, go to your WordPress admin, and it just bypasses the 2FA, two factor authentication. It bypasses everything. You just, bam up comes your WP admin dashboard.

[00:44:16] Nathan Wrigley: So is there, can we draw a direct line? And is there evidence for a direct line between the hack or rather Elementor plugin. And the stealing of authentication cookies. And as you've described that basically is all bets are off. You are now logging in as more or less anybody who's been on that website.

If you can get the, all the authentication cookies, you could be an admin or any kind of user. Is there a, can we draw that line or is it just that there was a serendipitous coincidence that started to happen in the same timeframe that the. The hack that you've just mentioned took place.

[00:44:56] Thomas J. Raef: Yeah I believe it, it, there must be some correlation, but what it is exactly I couldn't pinpoint. But one other thing that was happening. When people were getting into the through the essential add ons for Elementor exploit is we started seeing file changes in the various security plugins that people were using, like changes to the files and changes in the databases.

[00:45:29] Nathan Wrigley: Okay. So I guess at this point we should just circle back and talk about what your services. So your service is outside of WordPress. It's somewhere completely different. Whereas the plugins that we're describing, they live inside the WP file structure. If you're not familiar with WordPress's file structure, if you FTP into your site, you can, if you follow the right path, you can get to all of these plugins.

So you're saying that they were. updating, modifying the security plugins because they'd gained enough of a beachhold, enough of a foothead to have that permission.

[00:46:05] Thomas J. Raef: Correct.

[00:46:06] Nathan Wrigley: Sorry, I interrupted. I just

Make that clear. Yeah.

[00:46:10] Thomas J. Raef: yeah, no, I'm glad you're clarifying things for people. That's good. I like that. Yeah we started digging into, what are these file changes, 'cause it, it appeared that it was from the attack the attacker's IP address. And again, stepping back for a second, these IP addresses that the hackers were attacking from were infected websites.

So they were hiding behind, other infected websites, but anyway, so yeah, when you see a website and I'll just use this as a example, a GoDaddy website, logging in with admin rights to your website, you got to wonder, why, and but because the hackers had already compromised the website on a GoDaddy account.

But yeah so we were seeing. that they were using the theme editor, they were using plugin editor other core functionality of WordPress and they were modifying files and they're like. What the heck is going on? What they were doing is they were like a lot of these security plug ins have Some people call it whitelisting.

Some people call it like an allowed list. Things like that. So there's Typically with a malware scanning, there's what they call false positives. A false positive is a file that is detected by the scanner to be malicious, but really isn't. A lot of these security plugins will create These these bypass, these lists of files that shouldn't be scanned because they will show up as a false positive.

And what hackers were doing is they were uploading their malicious backdoor shell scripts, which give them access to your files and your database without having to log in again as admin. And they were uploading these backdoors and then adding those backdoor files to the exclusion list. Of the malware scanner

[00:48:37] Nathan Wrigley: Right.

[00:48:37] Thomas J. Raef: That it gives them longevity on that website.

[00:48:41] Nathan Wrigley: So, Logging in, making making the little hole that they've got bigger, basically, suddenly we can now, if we can, so we've logged in, we've detected that there's some kind of security plug in on here, aha, we know about that security plug in, we know where the files are that we would need to access in order to make the hole a bit wider allow these certain files, permit this range of IP addresses or what have you.

Thank you. That's interesting as well. Okay yeah I'm keeping up. And then what happened?

[00:49:16] Thomas J. Raef: And then, yeah, like I said they just started, they were using those infected websites to infect other websites. So they had uploaded code that allows them to like they, some people call it a C and C type file. It's a command and control. File. So what that does is it, once they copy that file up to your website and run it, it connects to one of their servers, grabs instructions, and then carries out those instructions, typically as attacks on other websites.

So we started seeing that. So they, the hackers were Uploading these files, adding them to the exclusion list of the malware scanners so they wouldn't be detected, and then they were running them, which, like I said, gives them like total use of your website of your resources as we talked in the beginning.

That's what typically hackers are looking for, and just carrying out massive attacks on other sites. It was just, it went from... The exploit wasn't even known about, until May 11th to just this full blown malware attack and we're, with the sites and servers that we monitor, we're only seeing, a small portion of it, obviously the internet is much bigger than what we cover.

Darn it. But

[00:51:02] Nathan Wrigley: Yeah, that's a fact of life, I'm afraid.

[00:51:05] Thomas J. Raef: Yeah.

[00:51:05] Nathan Wrigley: have the whole pie. So I guess what makes this story interesting is firstly, I should say, if you want really to get the deep dive on this, the article on the website, again, it'll be linked in the show notes, you've got the numbers there.

But in some cases, the numbers are fairly large, we're talking in the hundreds of thousands and what have you. Given that you are confident that your data is correct, and given that you, you put your trust in that, I guess the interesting thing here that makes this story kind of special is the fact that it was able to bypass these attacks, you.

Contend, at least anyway, was able to bypass some of the software that people have installed specifically to prevent this kind of thing happening. A k a, a malware piece of scanner type software or a or a security plugin, is that again, is that really why this story is of greatest interest?

[00:52:10] Thomas J. Raef: Um, Yes. And I have hesitancy because, Anybody who's in, in this, the market of fighting hackers I have a level of respect for how they do, the fact that their system is different than mine that, everybody's different. But yes, it was interesting that. I their default rules, if you will, for these security plugins were not preventing this.

And to the best of my knowledge, they weren't even seeing it because I can only believe that, they would have been doing something like I said, we're like other. security vendors. We have a free version and a paid version. Our free version just watches and it'll let you know, Hey, your website's infected.

It won't tell you how or what or anything like that. It'll just tell you, Hey, your website's infected. And people install that you know, all the time. And that's it. So it's, we're watching this happen in front of us, live and we're like, wow, and our, I see our system sending out notifications that, Hey this site's been infected.

Your site's been infected, Hey, on the paid versions, they, Hey, your site was infected, but it was cleaned. Here's how it happened. Here's what we did. So on and so forth. But yeah to see this unfolding in front of you it was just amazing. But then they said, yeah, the fact that it was bypassing so many of these security plugin systems was like, okay.

I, the only way I could explain it would be there. A lot of them are tuned to very specific attacks. So blocking the exploit for this essential add on for Elementor the rules for blocking that type of attack may not have any effect on an attack for a different plugin. In other words, the rules are so specific to that exploit

That it doesn't really do anything else for any, it, it doesn't have something where it just blocks file inclusion infections.

[00:55:06] Nathan Wrigley: right. There's no magic bullet here that you can't just write the perfect script, which will block anything of this nature. In the case of these pieces of software, they have to get very specific because that's the threat that they're facing now. And the way to mitigate that is to write some kind of challenge against that specific threat.

I guess. I guess it, I guess the scenario that you just described would be better if you could just come up with some golden bullet that would stop all of these things, but I'm also guessing that isn't how it happens. Do you know, in the case of of the security plugins that that you've mentioned, do you know if they've managed to get on top of this since you wrote this piece?

[00:55:50] Thomas J. Raef: Oh, yes. And they people like Mark Maunder from WordFence his. The way his company is structured, the rule was there ready for the paid people. The non paid people, the people were using the free version of WordFence got the rule automatically, I believe it was June 20th.

Yeah, there yeah, I'm at a loss here.

[00:56:33] Nathan Wrigley: Yeah, it's okay. I can understand what you're maybe challenged with thinking there, but okay. Yeah, I get it and in terms of the kind of, okay, so you've just described a whole broad swathe of attacks or rather this particular one, but a whole broad swathe of things that it did your solution, it feels like rather than trying to fix Hacking, to stop it from occurring, you're saying, okay we'll look for it when it's happened.

And then if you use our services we'll mend what got broken. So you're not trying to prevent things happening. And I hear the word hardening quite a lot as a sort of thing, which seems to be what people in all the Facebook groups I've been frequenting that are talking about this story, that they're making the drawing the line between, Okay, in this scenario, the best you can hope for is to harden your website, shut down the things which you can, but don't necessarily expect the malware scanner in this case to be able to pick that up.

[00:57:36] Thomas J. Raef: Yes we do, we, there's a lot of protection that we do and a lot of proactive things that we do. Like I said before the scenario of a GoDaddy, a website hosted with GoDaddy trying to log into your WordPress site. I. I've asked this at the cPanel conference in 2017 down in Fort Lauderdale, Florida.

Anybody tell me why, how that could be a possible use case? No. That, that possibility does not exist.

[00:58:17] Nathan Wrigley: Right.

[00:58:19] Thomas J. Raef: What we've done is we've built up a list of IP addresses for hosting providers around the world. And because we're working on the server level, this is something we can do at the I don't want to get into the whole seven layers of the network connectivity and stuff, but we do it at layer four.

So

[00:58:46] Nathan Wrigley: Oh, the glorious TCP IP stack. Oh,

[00:58:49] Thomas J. Raef: Yes. Yes. Okay. Yeah. The OSI model. Yeah.

[00:58:55] Nathan Wrigley: Oh.

[00:58:56] Thomas J. Raef: you knew that would come in handy

[00:58:58] Nathan Wrigley: Oh, yeah.

[00:58:58] Thomas J. Raef: just knowing about

[00:58:59] Nathan Wrigley: time I've managed to say this in the seven years I've been doing this podcast. Great. Anyway, sorry, I interrupted.

[00:59:06] Thomas J. Raef: No, that's good. So anyway, so it happens fast, cause we might be blocking 120, 000 ranges of IP addresses. Yeah. Yeah. But. when we put our protection on let's say you do a GT metrics on your website before we add our protection on and then you run it again after we add our protection, you're not going to see any difference at all because it happened so fast and it happened so.

so much earlier in the network, in the internet stack of communication that you don't notice it at all. And that blocks, like in this case, it would have blocked, it did block all these elementary attacks. And it also blocked the follow up attacks from sites that were compromised.

That, hackers were trying to use those compromised sites to attack other sites while our system blocked all that too. And it does that without... Without even trying, we don't have to change anything. We add, if somebody's using SEMrush or, some of those type analytical tools that will analyze your website.

We have whitelists also, so we'll whitelist those IP address ranges. But the other thing that we do is when we see these attacks come from, again, I'm not picking on GoDaddy, I'm just using them as a reference. If we see an attack coming from a GoDaddy, hosted with GoDaddy, our system will gather that information, removes anything that could be, harmful privacy violation, but sends that information to GoDaddy's abuse department.

So they can now look at, okay they're right. Nobody should be using one of our websites to attack other sites. So let's find out. Which website on this at this IP address is actually infected and then notify the the website owners that they have to take some action here to get this cleaned up. So it just takes one, it's something that's automated in our process. And it's something that it just takes one more digital asset out of the inventory of hackers.

[01:01:38] Nathan Wrigley: Yeah, sadly, the it feels like stories like these are not going away. Actually it's probably quite beneficial to your industry that that these attacks are not going away. Otherwise, you're gonna struggle to earn a living, but it would appear that, if we were to run this episode again and we were to talk to you in a year's time, we'd probably have another story is the sad truth.

I wish that these hackers had better things to do with their time, but... This is probably a very, in some cases, very lucrative. Let's call it a career because I imagine for them it probably is. I've really enjoyed this episode. I did say at the beginning, or at least I think I said it at the beginning, that we're going to give some of these other companies a bit of a right to reply as well.

But I've really enjoyed getting into that story with you and having your understanding of what's gone on and how it's all worked. I will link in the show notes to your piece on your website. But before we say goodbye to you, Thomas, is there anything you wanted to add? Did we miss anything out? Or if not, you could just leave us where we can get in touch with you.

[01:02:45] Thomas J. Raef: Okay. Yeah. One thing I do want to say, and, maybe it's unnecessary, but I said, I, at one point through all of this I had a conversation with Mark Maunder, the CEO of WordFence. And we talked, and exchanged laughs and learn more about each other and so on and so forth. One of the things I learned about this guy is as far as I know, he might be the only CEO in the security space that I know of, that's also a CISSP, which is like a high ranking certification for security people.

So the guy knows what he's doing, from a, even from a security standpoint, he's not just a CEO. So anyway, that's my mad respect for Mark Monder.

[01:03:41] Nathan Wrigley: Yeah, unfortunately the narrative of these stories doesn't really work unless you... Unless you've got some context to hang your coat on, really, does it? Unfortunately, you've got to, in some cases, use the word fence. So you might have to use the word out essential add ons for Elementor.

And it's, it really is. You're not really having a go at the people. It's just the code. So I appreciate your your, yeah, you're saying that right at the end. That's really helpful. Okay. So where can we find you, Thomas?

[01:04:11] Thomas J. Raef: You can find me at wewatchyourwebsite. com. You can also email me at T R A E F. Last name is Raef, it's Fear spelled backwards.

[01:04:27] Nathan Wrigley: Ha! I love that! Ha!

[01:04:30] Thomas J. Raef: so you can email me at Traife at wewatchyourwebsite. com You can find me on Twitter at, it's just at WeWatch and yeah, that's

you can find me on Skype a lot of

[01:04:49] Nathan Wrigley: Ooh! Ooh, now there's a word I haven't heard for a while! Ha! That's great!

[01:04:55] Thomas J. Raef: I can give you my MySpace account.

[01:04:57] Nathan Wrigley: oh yeah. By the way, just for just to round it out a little bit, I've spelled my name backwards and it's Yelgirl.

[01:05:05] Thomas J. Raef: Right.

[01:05:05] Nathan Wrigley: It's not nearly as fun as fear. Yeah,

[01:05:11] Thomas J. Raef: yeah. Yell, yell gear,

[01:05:13] Nathan Wrigley: yeah, something like that, yeah.

[01:05:15] Thomas J. Raef: Not not them. them.

[01:05:17] Nathan Wrigley: No not good. Okay, Thomas, really appreciate it. Lovely to chat to you today. Hopefully you'll get some comments or some people reaching out and telling you what they thought of the podcast. Appreciate it. Thanks for your

[01:05:28] Thomas J. Raef: Appreciate it. Thank you very much for the time.

[01:05:30] Nathan Wrigley: Well, I hope that you enjoyed that. Absolutely fabulous having Thomas on talking all about WordPress security and plugins and malware and the effectiveness, or not, of different solutions. As I said at the top of the show, if you've got any commentary, please head over to WP Builds.com. Leave us a comment there. After all WordPress has this fabulous commenting system. Search for episode number 342, and leave us a comment. We would really appreciate it. As I said, this was the third of four parts in our little mini series. We'll have Dan coming up in a couple of weeks.

The other thing to mention is like I did at the top, if you are interested in page building, and WordPress websites, and design and improving your agency processes and all of that pagebuildersummit.com, there's still a few presentations left, and you can pick up the power pack for a very modest fee, and anything that you missed, you can watch over and over again, forever. To your heart's delight.

The WP Builds podcast was brought to you today by GoDaddy Pro. GoDaddy Pro the home of managed WordPress hosting that includes free domain, SSL and 24 7 support. Bundle that with The Hub by GoDaddy Pro to unlock more free benefits to manage multiple sites in one place, invoice clients and get 30% of new purchases. You can find out more at go.me forward slash WP Builds. And sincere thanks to GoDaddy Pro for helping keep the lights on, on the WP Builds podcast.

Okay, we'll be back next week. It'll be a chat with David Waumsley and. But don't forget. We've also got our, this week in WordPress show. WP Builds.com forward slash live every Monday, 2:00 PM. UK time. Hopefully we'll be there. You can leave as a comment with the international panelists and the international people commenting. It's really good fun. So hopefully you can join us for that. If not have a good week, stay safe. Here comes some cheesy music. Bye-bye for now.