Why AI Gives Cyber Attackers the Early Advantage With IEEE

[00:00:00] The leading issue of agentic AI in businesses right now is ensuring agents act with compliance guidelines. And Denodo applies guardrails across your entire data estate. By aligning your company's data infrastructure under one system, these guardrails perform consistently across your platform. So start scaling your business and start with Denodo. Simply visit denodo.com to learn more.

[00:00:32] Can cybersecurity teams keep pace when attackers can generate convincing phishing messages, find targets and test new methods and do it all at machine speed? Well, today I'm going to be joined by Professor Stephen Fernall from the University of Nottingham for a conversation about the changing economics of cyber defence.

[00:00:56] And if his name sounds familiar, well, if you listen to Tech Talks daily and have done for a long time, four years ago I was speaking with him about assessing website password practices. And today, the world's a slightly different place with the explosion of AI. So we will talk about how AI is creating an opportunity for attackers while also producing additional work for defenders.

[00:01:21] And why phishing education built around yesterday's warning signs is losing relevance. But maybe pass keys could finally reduce our dependence on passwords. Be interesting to find out more four years after our last conversation. And we will also examine the gap between boards calling cybersecurity a priority and their business actually acting on it.

[00:01:48] And then we also have the limits of annual awareness training. Yeah, it's enough to make you shudder, isn't it, when you get that dreaded compliance training message arriving in your inbox. But maybe the practical value of cyber essentials could offer something different. And I also want to learn more about a game called Hacker Whacker that proves cyber education doesn't always need to be another compulsory video and box ticking exercise. Really looking forward to getting him back on.

[00:02:17] So let me introduce you to him now. So thank you for joining me on the show today. Can you tell everyone listening just a little about who you are and what you do? So I'm Stephen Fennell. I'm a professor of cybersecurity at the University of Nottingham. And that basically involves me doing research on cybersecurity, teaching around the subject and doing bits of public engagement in that context. Well, it's a pleasure to have you join me today.

[00:02:44] And you're talking to me at a time where reports are emerging about the first fully autonomous cyber attack. So from your perspective and everything that you're seeing and hearing, is it just hype in the media and when we're doom scrolling down our news feeds? Or are we witnessing a fundamental paradigm shift in how threats operate now? I certainly wouldn't say it's hype now.

[00:03:07] I mean, I think the recent incident with Hugging Face shows that things can happen that are fundamentally unexpected from even the perspective of those who've produced the technology. You know, in that particular case, the AI was meant to be sandboxed. It managed to get out of that environment. And that wasn't something that those that had developed it were expecting to happen.

[00:03:27] And that in itself is somewhat worrying that even those that are producing the technology aren't fully, let's say, in control of the effects that it could have. And that was a situation where it was meant to be in a controlled environment and then wasn't. And there was a real world thing that it decided to do. And so, OK, we're not quite at the stage of the machines are going to take over. But it does give a sign that this is something we've got to be careful of.

[00:03:56] And perhaps there needs to be a greater level of scrutiny and responsibility over the technologies that we become quite comfortable in using because they're beneficial. But in this particular case, we're not understanding the full extent of what the technology can do for itself.

[00:04:15] You've made a great point there, especially around the scrutiny on the technology, because I think if we take one area, fishing, for example, used to be relatively easy to spot due to bad grammar or obvious spoof domains, etc. But how are you seeing generative AI turbocharging social engineering now and why are traditional employee awareness programs? Maybe they're struggling to keep up as well because the game has got a little bit more sophisticated, hasn't it?

[00:04:41] Yes. I mean, as you say, I mean, the stereotypical fishing incidents of the past would very often sort of jump out at you because it claimed to be from a bank that you didn't bank with saying you needed to verify an account that you didn't have, etc., etc.

[00:04:55] Or, again, as you mentioned, the use of poor grammar, which sometimes, it must be said, was a deliberate thing on the part of the attackers on the base that if somebody doesn't spot the grammatical issues and still thinks it's believable, then perhaps you've got an easier mark at the other end who basically is less aware. But yeah, I think that, as you say, turbocharging is possibly a very good term for what the generative AI has managed to do for the fishing sector.

[00:05:23] But if you ask generative AI to generate you an example of a message that could be used to target XYZ community and to try and extract such and such information out of them, it will merrily chug out something instantly that could be potentially a bit of tuning, perhaps, deployed. And if you name a particular organization that there's information about online, it will be able to pepper the, you know, it can make a spear phishing message for you.

[00:05:51] So it's taken a lot of the effort out that otherwise an attacker might have to do some manual reconnaissance, etc. Now you can get a ready package thing. And yeah, that amplifies the challenge for the user awareness side of things, because I think already we have a problem with user awareness around phishing. It's the most common form of incident that gets reported in breaches surveys, for example. Organizations saying they're coming into contact with it. Staff do fall for it.

[00:06:20] And, you know, fundamentally, that's what the attacks are designed to do. You know, it becomes more and more difficult for the human recipient to distinguish unless they're very alert and very clued up on what is legitimate to ask for and how things would occur, let's say, if they're being requested to do something. And they're also being put under potential psychological pressure by the buttons that the message you're trying to put.

[00:06:48] But it's making it even more difficult to try and tune people into this is what a scam looks like. You know, we've maybe encultured this thing of, you know, it's banking messages, it's things like that. And that, you know, where the mindset of the potential victim is tuned to what the attacks looked like 10 years ago, let's say.

[00:07:09] And I'm curious, if we look at the current arms race between AI-driven attack tools and AI-driven defense mechanisms, it's that infamous game of cat and mouse that we've always seen. But who has the upper hand right now? Is it the attackers or the defenders? I appreciate that's possibly an entire episode on its own. But what are you seeing there? Well, I suppose that the fact is that the technology is there, unfortunately, for both communities.

[00:07:34] And if you think about the two parties involved, the use of it for the attackers, it's giving them an advantage. It's enabling something they want to do. Obviously, the defenders want to defend as well. But equally there, the use of the technology is more in the context of potentially even creating work for them. You know, so the attackers are utilizing it to break through. The defenders in that context are needing to be responsive to what's going on.

[00:08:02] And so I think from a certain point of view, the attackers are the ones setting the agenda because they're the ones leveraging the technology to create the turbo boosted, to use that previous term, or new attack vectors. And so the defenders are having to act in response. And for them, yeah, it's basically creating additional work, whereas for the attackers, it's creating additional opportunity, shall we say. And I've been recording tech podcasts for 11 years now.

[00:08:29] And since then, even before I began, we've been hearing about a passwordless future for years. But one of the things we've noticed in the last 18 months is pass keys do seem to be gaining traction. So do you think, are we getting closer to finally declaring the password dead? Or is that just wishful thinking on my part? I think we actually could be for once. I mean, you can find stories going back a couple of decades declaring passwords are dead or they're going to be dead by a couple of years time. And of course, they weren't.

[00:08:58] But now, exactly as you say, with pass keys, we are seeing that technology getting adopted and being able to be utilised by end users on more platforms and services. They're not ubiquitous yet. And passwords are not going to suddenly disappear off the face of the authentication landscape. But pass keys are gaining traction. They are now the thing that's recommended, for example, by the UK's National Cyber Security Centre as something that ought to be adopted.

[00:09:26] So they weren't saying the same thing last year. This year, they have transitioned to saying that. So that indicates the maturity of the technology, the belief in it, the availability. But it's not yet ubiquitous. So we can't yet say goodbye to passwords and all the concerns that they have still been bringing for us. Yeah, as you said, it's not quite ubiquitous yet.

[00:09:49] So if we'd have a business leader listening and they want to move toward pass keys and modern user authentication, especially because their board might be demanding it as well. But what is the biggest technical and human friction points that they should prepare for before embarking on this project? Well, I think fundamentally, yes, it is another technology change.

[00:10:10] So you've got to be prepared for that to be rolled out on the platforms for the various devices that the user community are going to be wanting to access from to be able to utilize it, making the relevant provisions on the technology side. So in that sense, I guess it's similar to what we've previously seen with rollouts of things like multifactor authentication or indeed other technology deployments. The organization itself has to be ready.

[00:11:03] These are one time codes and all of this. But, you know, it sort of feels fairly hot on the heels of MFA having been rolled out as this is now what we need to do to be secure. This is how authentication needs to work. And now it's something different relatively quickly off the back of that. And just expanding on that as well.

[00:11:25] Any tips on how organizations can implement some of those stringent intrusion detection and access controls without frustrating their workforce or killing productivity? Because it's that friction that can derail the project sometimes. Yeah, so I think, I mean, there's no magic wand solution here, but I think it is it requires thought about what are users going to come into contact with and where where is the technology potentially going to create a barrier for them? I mean, it's not just in terms of intrusion detection.

[00:11:55] Let's say I think more widely with cybersecurity thinking what friction it potentially introduces for users and even not just in terms of technology, but procedures and processes. What are we required to do? How naturally does that fit into the workflow of what we're trying to do from a productivity perspective? So recognizing, you know, doing almost a usability assessment of the technologies and the practices that we're suggesting be introduced.

[00:12:21] I mean, MFA was a good example, let's say, in that context, because it does require a behavioral change for the user every time they log in. It's going to introduce some complications. It's going to require them perhaps to be using their device, installing an app and the authenticator app on their device in order to participate in the process. It's more involved than just typing the password or getting the saved password to populate that they would previously done.

[00:12:48] And so every intervention, every activity that involves authentication now takes that little bit longer. And so the very least needs to be explained to people as to why that's beneficial to the organization and arguably to them as well, that they are now going to be encountering this little extra bit of activity. And sometimes in something like that, it's unavoidable.

[00:13:10] But in other cases where you've got an option for things to happen automatically versus it to require a user intervention, think about what the trade-offs of that are. I mean, if we think about pushing updates, for example. Well, OK, if we push updates at some point, it has the potential to automatically reset somebody's application or reset their whole system for the app to install. So that could annoy people.

[00:13:34] But equally, if we leave it to the user to press the button when the update is available, well, that update might not ever get applied or may be sometime down the line. And so it's thinking about the trade-off between automating and encouraging the user in many cases. Yeah, I suspect a lot of people would be listening to you there with a little smile when they hit that remind me later option.

[00:14:00] Yeah, later always feels quite attractive. Unfortunately, it doesn't say how much later you'd like it to be. But looking across your work there, you highlight that as well, a massive disconnect between security best practices and what a business actually executes in their day-to-day activities. So where are organizations failing most often from what you're seeing? And why is this gap still widening?

[00:14:28] So I think one of the key areas, and we've mentioned already user awareness, that seems to be a perennial problem in terms of receiving less attention. And yet you see, and again, if I refer to the UK cybersecurity breaches survey, one of the consistent results, if you look across the areas of security that they ask organizations, have you got key steps in these areas?

[00:14:51] Because user awareness, it's typically less than 20% of organizations say they've done staff awareness training on cybersecurity in the last 12 months. And even those that have done it, you sort of have a question about how well have they done it. But if the majority haven't done anything in the last 12 months, that's not necessarily going to leave their staff in a very aware position. You then look at another result from the same survey. What actions did you take in response to your most disruptive breach or attack?

[00:15:19] And the most common response is additional people facing things. So additional training, awareness, and things of that nature. So not many do it by default, but it seems a very common response when something happens. So that's one area. I mean, another couple of areas from the same survey. Incident response and planning seems to get relatively little attention. Risk management, risk assessment also.

[00:15:44] So I think that the areas where there's more prominent attention seem to be the things where, for one of them, there's a product for it. Or there's a technical solution that can be deployed. So lots of the respondents have network security, anti-malware protection. They do backups, et cetera. But when it comes to the things that it's not just, OK, you've got to install this and enable it, then there's less.

[00:16:10] And I think there is a potentially in some organizations a mindset that they'd like to get something in place and then it just happens and they can forget about it. Whereas these things like user awareness, they require, unfortunately, ongoing attention. And I say even where there is training in place. And I think about this from the perspective of the IT security or cybersecurity training that I completed very recently in my own organization. OK, what did that involve?

[00:16:40] That involved me watching some things, reading some things, answering some questions. And then I'm trained for the next 12 months until I have to do probably exactly the same thing again. And, OK, I'd like to think I had a reasonable chance of passing the cybersecurity training. From the outset anyway. But how much more aware has it made me? How much more capable has it made me to do something if something happens?

[00:17:05] Possibly not that much because it's talked about things rather than put me in a situation where something is happening and see what I do. And I've not had a chance to talk to anybody else about it. It's just been me doing the online training. And if you think that's perhaps the general picture about how the training and awareness is handled. Then, OK, it, for want of a better term, ticks the box. So we're compliant. We have provided organizationally training for our staff.

[00:17:34] So if we have a data breach and the Information Commissioner's Office would like to know, did we suitably tell people about data protection and securing data? Yes, we have training. But was it effective? Well, we're not so sure when it comes to that. And so it's not necessarily entering into the spirit of it. And that's just one area of security. I say that if you look across the breaches server, there's various others.

[00:17:59] And I suppose a finding that always catches my attention is one of the early questions in that server. They ask the businesses and the charities who are the respondents. They ask, you know, to what extent is security a high or fairly high priority for your organization versus no priority, etc.? And about 70% of the respondents say it's a high priority for our senior managers, board of directors, trustees. They're talking about the senior people. So 70%, you know, keep that figure in mind.

[00:18:27] And then you look at, you know, to what extent these different areas of security, the NCSC's 10 steps is used as a reference point in the survey. To what extent are those things being attended to? Only a couple of those are over 70% of the organization saying they do it. And so that extent to which it's a high priority sort of sits with a question mark as a result. Yeah. And your example there is possibly one of my biggest frustrations with corporate life.

[00:18:54] And looking at, I think, maybe security is often seen as an IT problem rather than a cultural one very often. And many business leaders need to shift their organization away from that checkbox compliance that HR and IT work together on to ensure everyone's got that tick against their name. And maybe better build a genuine proactive security culture. I don't know, that's probably a bit woo-woo of me and idealistic. But it is possible with a different approach, right?

[00:19:24] Absolutely. I mean, I think, yes, that's what we ought to be trying to aim towards. You go from, let's say, awareness to behavioral change to organizational culture. But you don't go just by sending out an email saying, oh, watch out for phishing messages or sending out a mock phishing message and telling people if they were fooled or not. Or 10 minutes, 20 minutes of online training every 12 months. That isn't going to get you there.

[00:19:52] You need to have the thing more ingrained into the practices and behaviors of the organization, thinking about the value of the systems, the data, the devices that people are taking around. And again, those considerations for, well, people, let's say, using mobile technology. To what extent do people think about the value of what they're carrying around on their devices?

[00:20:18] I mean, take it back a little while, mobile phones prior to Bionetrics coming along as an authentication mechanism. Very often you would find phones that were not protected at the point of picking them up and trying to get access to them because people didn't want to have to type a passcode along. Or they might have a four-digit PIN that was 1234 or something like this. Now it's easier to have that protection.

[00:20:44] So hopefully in most cases there is something that protects it. But that device is very often the pathway into a lot of data and a lot of systems that that individual, if it's their personal device or their work device, has in terms of their organizational life. And so even that, again, looking at past surveys, proportion of organizations have policies that include guidance for users on use of mobile devices.

[00:21:14] It's not all of them. And it's a significant sort of shortfall. And you then think, well, what proportion of those organizations have got staff who've got mobile devices? Well, probably all of them. And what proportion of those staff with mobile devices are using them in some way for work purposes, whether it's getting email, putting in appointments, something that the organization might think, well, this shouldn't be open and public? Again, probably everybody.

[00:21:41] To what extent have we as an organization taken any interest in the extent to which that device then is protected? Not so sure. And so you get all these little points of potential disconnection in there. When many people listening hear about enterprise security frameworks, it can feel overwhelming, especially for some of the mid-market or smaller businesses that have a much smaller budget.

[00:22:07] So what would you say are the essential non-negotiables for every business that they must be getting right today? So I think a good reference point there is cyber essentials. Again, a national cybersecurity center scheme, which is talking about some of the key things that you need to do to protect your online connected systems. So having some sort of border protection, having a regime to apply the updates, having anti-malware protection, etc.

[00:22:33] These are essentially basic elements of organizational cyber hygiene. And so having those points, and that's why cyber essentials has been put forward and it's advocated as, if you like, a minimum baseline that organizations can certify against to say that they've achieved it. Because doing those things will remove quite a lot of the sort of low-hanging fruit opportunities for attackers and for other forms of vulnerability.

[00:23:02] So, yeah, as you say, from the perspective of a small organization, it is daunting. And some of the research that we've been doing for the last couple of years has been specifically focusing on the support available to small and medium enterprises around cybersecurity. And, yeah, it's a challenge because they don't have the time, the resource, the expertise. And, you know, you can go off and read something about it and you can understand that, OK, I now know I ought to have this.

[00:23:29] But how do I do it? How do I put that into practice is still something that can be very much out of reach without further help. You've helped inform the UK government cybersecurity policy. So I'm curious from your experiences there, how do you view that relationship between government regulation, private sector responsibility? And do you think regulation is keeping pace with the rapid tech shifts that we're seeing? They've always had a reputation for playing catch up.

[00:23:58] Is that changing? Yeah, so as you say, I've been involved in a couple of working groups that have helped to inform certain elements of guidance and policy around it. And I think things are improving. And certainly there is more attention towards these issues now than perhaps there was. We've got some very relevant acts and guidance and things that have emerged. So, for example, the software security code of practice, et cetera.

[00:24:25] We've got things now around consumer connected devices, which we didn't have just a few years ago. And there's attention around sort of in the wider sort of context, not yet in terms of regulation, but security by design. There are recommendations around that.

[00:24:44] So you get the sort of the transition from things becoming codes of practice through to, you know, sort of things to voluntarily adopt through to legislation that people are required to comply with. And I think there is more positive attention in that respect. But, yeah, it still is a thing of, you know, the technology, the use of the technology happens more quickly than the legislative response to it or the government policy response.

[00:25:13] And I think that's the nature of the pace of technology just by default. I suppose there is a, you know, an ongoing question that we can ask ourselves about as technology developers and consumers that every generation of technology that comes out gets adopted because of the benefits that it offers.

[00:25:34] And it's only somewhat latterly that the security implications, the vulnerabilities tend to then get highlighted and suddenly everybody's using a technology that isn't properly safeguarded. I mean, we can roll back to thinking about the use of wireless networks, for example, when, you know, every organizations, domestic users buying Wi-Fi access points, those access points initially weren't encrypted by default.

[00:26:00] And so, you know, you could find quite readily unprotective wireless access points that you could eavesdrop on or you could hook into and use somebody else's provision. Yeah, that changed. But was it unpredictable when that technology was released that that would happen? Probably not. You know, that could, you know, you leave something open, somebody's going to misuse it. That's not the intended use of it, but that's what happens.

[00:26:27] Similarly, you know, the various platforms on which malware has managed to find its way to people, you know, basically any service that becomes popular, email, messaging, social media, becomes a channel through which malware can potentially propagate. But on each occasion, it almost seems to come as a surprise to, oh, malware's there now. Really?

[00:26:53] And so, yeah, smart devices, Internet of Things, all of it, you know, they get released in the initial form in a more vulnerable configuration than perhaps they latterly become. And security becomes that not quite afterthought, but that thing that wasn't there in the first place that then people realize, okay, there is a security issue to answer.

[00:27:17] And it would seem entirely predictable if you were thinking with a secure by default, secure by design mindset on it. And if we look at, let's say, Gen Z and Gen Alpha as well, how important do you see education in making sure that the next generation of cyber remains cyber aware? And are there any innovative ways that companies, educational institutions or governments can ensure that education around this is taken seriously?

[00:27:47] I would imagine this is an area you're passionate about too. Yes, I mean, fundamentally, yes. I feel that it's very important that the cybersecurity literacy becomes part of the wider digital literacy that, you know, all the generations that are coming into it will then encounter. And the nature of what that looks like, the messaging will also need to evolve to keep pace with whatever the form of the technology they're encountering. I mean, there's some fundamental principles that have always remained the same.

[00:28:17] Let's say our things around confidentiality, integrity, availability, but that's not the message that you would present to the younger generation in that form. But it comes down to those sort of things. What have you got to think about in your use of the technology? Why are the threats relevant to you? How could you be harmed? How could your data device, et cetera, be harmed? So making it sort of personalized and recognizable for people. I mean, I think how you do it depends on the age group and the context that you've got available to you.

[00:28:46] I mean, one of the things that we've done at the University of Nottingham in our cyber games lab, we've created a number of little provocations of interest around cybersecurity to try and just get that initial engagement and familiarity with things. So we've got taking it from a physical device that we've called Hacker Whacker, which is a bit like a whack-a-mole game. Pictures appear on the screen.

[00:29:10] You've got to hit a light, a lit up button, depending on whether it's a picture of a threat or a safeguard or indeed an asset if we played a more complicated three light version. And now we've taken those same concepts into a game called Cyber Defense Dice, where basically you've got a set of red dice, a set of blue dice, and you've got attacker versus defender.

[00:29:29] And they can play a little game learning something about the relationships between, well, if this type of attack happens, you need this type of defense in place to safeguard against it. And that opens up a conversation. We've played it with young people at events. We've had young people come into the university for an event called Step Into Cyber that we've run and had 140 of them all around tables playing this.

[00:29:55] But we've also done this at security events, and we've shown it to people already in business, for example, as a potential provocation of interest, a conversation starter for use in organizations. Because if you can get people engaged in a way that they feel maybe is a little bit more fun and interactive than sitting and doing the stereotypical watch this video, answer these questions, and see you in 12 months sort of online training, you've got something as a touch point.

[00:30:24] I mean, another touch point is if something bad has already happened, that tends to get people's attention. So the real-life experiences of security tend to also be a good jumping-off point. But we don't want that to happen for everybody. So having something that makes it more fun and engaging is one way of trying to bring it to a wider audience, let's say. Absolutely love that. And kudos to you for looking at it this way.

[00:30:51] And for anyone listening, maybe they want to bring Hacker Whacker to one of their security events or just learn more about anything that we've covered today. Where would you like me to point everyone listening? Well, I think in relation to the last of it, if they go to www.cybergameslab.org, that will take them to our page for the various gamified activities that we've produced. And more generally, sort of have a look at our stuff at the University of Nottingham in the cybersecurity group.

[00:31:19] But I think also, I've mentioned a few times the NCSC, if you're actually looking for actual actionable cybersecurity advice, what you ought to do as an organization, go look at the guidance on cyber essentials, the 10 steps to cybersecurity. And also, they don't call it this now, but cyber aware. If you do a search for cyber aware and the NCSC, you'll come to their top tips for individuals. Awesome. Well, I will add links to absolutely everything that you mentioned now.

[00:31:48] I encourage everyone listening, go over to techtalksnetwork.com. There will be a blog post associated with this episode and all the links to everything that we've mentioned today and everything we've talked about. And I urge everyone listening to check that out and let me know your thoughts, your experiences. But more than anything, Stephen, thank you for joining me today and bringing all this to life in a language that everyone can understand. Appreciate your time.

[00:32:13] We're back on the podcast today and a very clear message that cybersecurity culture cannot be created through one annual course, a mock phishing email or a policy that employees forget until something goes wrong. And businesses need controls that fit very real workflows, explanations that can help understand the tradeoffs and repeated opportunities to practice safer behavior.

[00:32:39] And pass keys, yes, they might reduce authentication friction, but organizations must also prepare users for another change after years of multi-factor authentication or MFA. And smaller businesses, they can also use cyber essentials as a manageable baseline while seeking help with implementation.

[00:33:03] So, again, massive thank you to Stephen for translating these issues into a language that every organization can use. And you can find his Cyber Games Lab at cybergameslab.org. Practical Ed's NCSC guidance through Cyber Essentials and its 10 Steps. But don't worry, you don't have to remember all this. Go to techtalksnetwork.com. They will be a blog post associated with this episode and there'll be links to everything that we've mentioned today.

[00:33:33] But before I let you walk off into the sunset, which security behavior would your organization struggle to demonstrate if you were asked to test it today? Remember, over at techtalksnetwork.com, you can send me an audio message. Love to hear from you. But I'm afraid we're out of time today. Thanks for listening as always. Bye for now. Bye.