How to Detect Rootkits In Windows 10 (In-Depth Guide)

Rootkits are used by hackers to hide persistent, seemingly undetectable malware within your device that will silently steal data or resources, sometimes over the course of multiple years. They can also be used in keylogger fashion where your keystrokes and communications are surveilled providing the onlooker with privacy information.  

This particular hacking method saw more relevance pre-2006, prior to Microsoft Vista requiring vendors to digitally sign all computer drivers. The Kernel Patch Protection (KPP) caused malware writers to change their attack methods and only recently as of 2018 with the Zacinlo ad fraud operation, did rootkits re-enter the spotlight.

The rootkits pre-dating 2006 were all specifically operating system-based. The Zacinlo situation, a rootkit from the Detrahere malware family, gave us something even more dangerous in the form of a firmware-based rootkit. Regardless, rootkits are only around one percent of all malware output seen annually. 

Even so, because of the danger they can present, it would be prudent to understand how detecting rootkits that may have already infiltrated your system works.

Detecting Rootkits in Windows 10 (In-Depth)

Zacinlo had actually been in play for almost six years before being discovered targeting the Windows 10 platform. The rootkit component was highly configurable and protected itself from processes it deemed dangerous to its functionality and was capable of intercepting and decrypting SSL communications.

It would encrypt and store all of its configuration data within the Windows Registry and, while Windows was shutting down, rewrite itself from memory to disk using a different name, and update its registry key. This helped it to evade detection by your standard antivirus software.

This goes to show that a standard antivirus or antimalware software is not enough for detecting rootkits. Although, there are a few top tier antimalware programs that will alert you to suspicions of a rootkit attack. 

The 5 Key Attributes Of a Good Antivirus Software

Most of the prominent antivirus programs today will perform all five of these notable methods for detecting rootkits.

• Signature-based Analysis – The antivirus software will compare logged files with known signatures of rootkits. The analysis will also look for behavioral patterns that mimic certain operating activities of known rootkits, such as aggressive port use.
• Interception Detection – The Windows operating system employs pointer tables to run commands that are known to prompt a rootkit to act. Since rootkits attempt to replace or modify anything considered a threat, this will tip off your system to their presence.
• Multi-Source Data Comparison – Rootkits, in their attempt to remain hidden, may alter certain data presented in a standard examination. The returned results of high and low-level system calls can give away the presence of a rootkit. The software may also compare the process memory loaded into the RAM with the content of the file on the hard disk.
• Integrity Check – Every system library possesses a digital signature that is created at the time the system was considered “clean”. Good security software can check the libraries for any alteration of the code used to create the digital signature.
• Registry Comparisons – Most antivirus software programs have these on a preset schedule. A clean file will be compared with a client file, in real-time, to determine if the client is or contains an unrequested executable (.exe).

Performing Rootkit Scans

Performing a rootkit scan is the best attempt for detecting rootkit infection. Most often your operating system cannot be trusted to identify a rootkit on its own and presents a challenge to determine its presence. Rootkits are master spies, covering their tracks at almost every turn and capable of remaining hidden in plain sight.

If you suspect a rootkit virus attack has taken place on your machine, a good strategy for detection would be to power down the computer and execute the scan from a known clean system. A surefire way to locate a rootkit within your machine is through a memory dump analysis. A rootkit cannot hide the instructions it gives your system as it executes them in the machine’s memory.

Using WinDbg For Malware Analysis

Microsoft Windows has provided its own multi-function debugging tool that can be used to perform debugging scans on applications, drivers, or the operating system itself. It will debug kernel-mode and user-mode code, help analyze crash dumps, and examine the CPU registers.

Some Windows systems will come with WinDbg already bundled in. Those without will need to download it from the Microsoft Store. WinDbg Preview is the more modern version of WinDbg, providing easier on the eyes visuals, faster windows, complete scripting, and the same commands, extensions, and workflows as the original.

At the bare minimum, you can use WinDbg to analyze a memory or crash dump, including a Blue Screen Of Death (BSOD). From the results, you can look for indicators of a malware attack. If you feel that one of your programs may be hindered by the presence of malware, or is using more memory than is required, you can create a dump file and use WinDbg to help analyze it.

A complete memory dump can take up significant disk space so it may be better to perform a Kernel-Mode dump or Small Memory dump instead. A Kernel-Mode dump will contain all memory usage information by the kernel at the time of the crash. A Small Memory dump will contain basic information on varying systems like drivers, the kernel, and more, but is tiny in comparison.

Small Memory dumps are more useful in analyzing why a BSOD has occurred. For detecting rootkits, a complete or kernel version will be more helpful.

Creating A Kernel-Mode Dump File

A Kernel-Mode dump file can be created in three ways:

• Enable the dump file from Control Panel to allow the system to crash on its own
• Enable the dump file from Control Panel to force the system to crash
• Use a debugger tool to create one for you

We’ll be going with choice number three. 

To perform the necessary dump file, you only need to enter the following command into the Command window of WinDbg.

Replace FileName with an appropriate name for the dump file and the “?” with an f. Make sure that the “f” is lowercase or else you’ll create a different kind of dump file.

Once the debugger has run its course (the first scan will take considerable minutes), a dump file will have been created and you’ll be able to analyze your findings.

Understanding what it is your looking for, such as volatile memory (RAM) usage, to determine the presence of a rootkit takes experience and testing. It is possible, though not recommended for a novice, to test malware discovering techniques on a live system. To do this will again take expertise and in-depth knowledge on workings of WinDbg so as not to accidentally deploy a live virus into your system.

There are safer, more beginner-friendly ways to uncover our well-hidden enemy.

Additional Scanning Methods

Manual detection and behavioral analysis are also reliable methods for detecting rootkits. Attempting to discover the location of a rootkit can be a major pain so, instead of targeting the rootkit itself, you can instead look for rootkit-like behaviors.

You can look for rootkits in downloaded software bundles by using Advanced or Custom install options during installation. What you’ll need to look for are any unfamiliar files listed in the details. These files should be discarded, or you can do a quick search online for any references to malicious software.

Firewalls and their logging reports are an incredibly effective way to discover a rootkit. The software will notify you if your network is under scrutiny, and should quarantine any unrecognizable or suspicious downloads prior to installation. 

If you suspect that a rootkit may already be on your machine, you can dive into the firewall logging reports and look for any out of the ordinary behavior.

Reviewing Firewall Logging Reports

You’ll want to review your current firewall logging reports, making an open-source application like IP Traffic Spy with firewall log filtering capabilities, a very useful tool. The reports will show you what is necessary to see should an attack occur. 

If you have a large network with a standalone egress filtering firewall, IP Traffic Spy will not be necessary. Instead, you should be able to see the inbound and outbound packets to all devices and workstations on the network via the firewall logs.

Whether you’re in a home or small business setting, you can use the modem provided by your ISP or, if you own one, a personal firewall or router to pull up the firewall logs. You’ll be able to identify the traffic for each device connected to the same network. 

It may also be beneficial to enable Windows Firewall Log files. By default, the log file is disabled meaning no information or data is written.

• To create a log file, open up the Run function by pressing the Windows key + R.
• Type wf.msc into the box and press Enter.

• In the Windows Firewall and Advanced Security window highlight “Windows Defender Firewall with Advanced Security on Local Computer” in the left side menu. On the far right side menu under “Actions” click Properties.

• In the new dialog window, navigate over to the “Private Profile” tab and select Customize, which can be found in the “Logging” section.

• The new window will allow you to select how big of a log file to write, where you’d like the file sent, and whether to log only dropped packets, successful connection, or both.

• Dropped packets are those that Windows Firewall has blocked on your behalf.
• By default, Windows Firewall log entries will only store the last 4MB of data and can be found in the %SystemRoot%\System32\LogFiles\Firewall\Pfirewall.log
• Keep in mind that increasing the size limit on data usage for logs can impact your computer’s performance.
• Press OK when finished.
• Next, repeat the same steps you just went through in the “Private Profile” tab, only this time in the “Public Profile” tab.
  • Logs will now be generated for both public and private connections. You can view the files in a text editor like Notepad or import them into a spreadsheet.
  • You can now export the logs’ files into a database parser program like IP Traffic Spy to filter and sort the traffic for easy identification.

Keep an eye out for anything out of the ordinary in the log files. Even the slightest system fault can indicate a rootkit infection. Something along the lines of excessive CPU or bandwidth usage when you’re not running anything too demanding, or at all, can be a major clue.