Detecting insiders on GNU/Linux servers

Yet another auditd ruleset

Intrusion detection on Linux systems is an information security subject that does not get as much light as intrusion detection on other operating systems such as Microsoft Windows. One theory I heard of that may explain this situation is that attackers usually target more Windows workstations of employees inside organizations to gain a foothold rather than Linux servers. I found that theory to be false in multiple incident response cases over the years. Attackers often target web servers or other internet-facing services, which are mostly running under the GNU/Linux operating system (sometimes under BSD) as a starting point in conducting attacks against organizations.

An attacker, once he gains initial foothold on a GNU/Linux system, usually follow one or more of the below purposes:

• performing reconnaissance
• evading restricted environments
• elevating his privileges
• obtaining secrets
• persisting

Meanwhile he pursue these, he also wants to avoid detection, so he will try to cover his traces. In order to achieve these goals though, his methods almost always enter in one or more of the following categories of actions:

• read a file
• write a file
• execute a binary
• load a kernel module

This is partly inherent to the “everything is a file” philosophy of GNU/Linux systems and it makes detection rely on these actions.

The present article aims at detecting some malicious actions with auditd rules. This is not the only method to achieve it. However, auditd is a mature, kernel-implemented way of detection and is available in most, if not all, GNU/Linux distributions. It seemed to me a good choice to start detecting at a low cost. As detection is much of a cat-and-mouse game, the hereby presented methods will not be enough to detect all ways of elevating one’s privileges or persist on a system, nor will they address future ways of performing above listed actions.

Moreover, the mantra for these kind of detection rules should be “if this rule match, the behaviour is suspicious”, as much as possible I am trying not to raise alerts when a legit user is typing command lines on a GNU/Linux machine that a system administrator could type. These are usually time-consuming alerts for a SOC that are impossible to distinguish from legit behaviour. Here are some example of such commands that I have seen detected in existing signature rulesets:

whoami
ss -ltpn
ls /etc/cron.d
cat /etc/passwd

The above are, not enough to raise alerts. It is however good-to-have material if some other action triggered a signature for a given user but cannot be considered a malicious behaviour by itself.

I hope the presented rules will give you ideas that you may not find in other sets available online, they are mostly based on incident response and penetration testing experience.

The ruleset is also made for a GNU/Linux server, not a user workstation, keep it in mind. A server has less way tools installed and less user activity while the number of daemons it is running may be greater than a user workstation.

All the rules presented here are available on a github repository.

Reconnaissance

Sniffing the network

Sniffing the network is a typical reconnaissance action that you want to detect on an internal network. While a network administrator may run a tcpdump command from time to time, it should be considered a suspicious activity as this is a common way for attackers to gather network information such as IP addresses, protocols used at link-layer etc.

There are two actions you want to spot:

• installing a network sniffer
• using a network sniffer

tcpdump, a well-known sniffer, is usually is installed on /usr/bin, so a write to that location can detect its installation.

Associated auditd rules:

# tcpdump install/modification
-w /usr/bin/tcpdump -p wa -k detect_tcpdump_install

# tcpdump use
-a always,exit -F path=/usr/bin/tcpdump -F perm=x -k detect_tcpdump_exec

Attackers also make intensive use of another well-known network sniffer: wireshark. While unusual to execute on the target server (for better opsec and safety tcpdump will be used and the resulting pcap will be downloaded and analyzed offline) , you still want to detect it.

Associated auditd rules:

# wireshark install/modification
-w /usr/bin/wireshark -p wa -k detect_wireshark_install
-w /usr/bin/tshark -p wa -k detect_tshark_install

# tshark/wireshark use
-a always,exit -F path=/usr/bin/wireshark -F perm=x -k detect_wireshark_exec
-a always,exit -F path=/usr/bin/tshark -F perm=x -k detect_tshark_exec

Scanning the network

Apart from passive actions an attacker could conduct on the network, he usually will also perform active actions such as scanning targets. The go-to tool for that is the Network Mapper: nmap.

Associated auditd rules:

# nmap install/modification
-w /usr/bin/nmap -p wa -k detect_nmap_install

# nmap use
-w /usr/bin/nmap -p x -k detect_nmap

Privilege escalation

Local user or group addition

Adding a user to the system or adding an existing user inside a privileged group could constitute a privilege escalation due to poor file permissions on key Linux files. Exploiting a privileged binary having write permissions onto one of these key files could also be a way of achieving the same thing. But this could also be categorized as persistence, I chose to put it in this section arbitrarily.

On a GNU/Linux system this translates by modifications of /etc/passwd, /etc/shadow, /etc/group or /etc/gshadow files.

Associated auditd rules:

# User database modification
-w /etc/passwd -p wa -k detect_modify_passwd
-w /etc/shadow -p wa -k detect_modify_shadow

# Group database modification
-w /etc/group -p wa -k detect_modify_group
-w /etc/gshadow -p wa -k detect_modify_gshadow

sudoers addition

Same goes for the users listed in the sudoers file, an attacker having write permissions on /etc/sudoers could add himself or a group he is in into the file. For the same reason, it could be used as a persistence mean as well.

Associated auditd rules:

# Sudoers addition
-w /etc/sudoers -p wa -k detect_modify_sudoers
-w /etc/sudoers.d/ -p wa -k detect_modify_sudoers_d

set*id() syscalls

System calls to setuid() or setgid() is a common privilege escalation technique used in numerous exploits. You want to detect when this is done towards a privileged UID such as root’s but from an supposedly unprivileged one.

Associated auditd rules:

# Set*id() function family calls to privileged UID (root) 
-a always,exit -S setuid -S setgid -S setreuid -S setregid -F a0=0 -F uid!=0 -k detect_root_setuid_setgid

Note that this rule could (and should) be extended to any UID on your perimeter that is a privileged user.

This rule can be tested with a simple C program:

#include <stdio.h>
#include <unistd.h>

int main()
{
        if (setuid(0) == -1)
                printf("setuid failed\n");
        printf("%d\n", getuid());
}

Unprivileged user shell:

$ gcc -o setuid setuid.c
$ ./setuid 
setuid failed
1000

Auditor:

# ausearch --start this-week -k detect_setuid
time->Mon Apr 21 05:07:02 2025
type=PROCTITLE msg=audit(1745226422.803:4276): proctitle="./setuid"
type=SYSCALL msg=audit(1745226422.803:4276): arch=c000003e syscall=105 success=no exit=-1 a0=0 a1=7ffc11084808 a2=7ffc11084818 a3=7f87c451f758 items=0 ppid=8444 pid=611300 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts2 ses=2 comm="setuid" exe="/tmp/setuid" subj=unconfined key="detect_setuid_setgid"

Code injection

Code injection in a running process is a classic way for an attacker to escalate privileges through vulnerable binaries with the help of shellcodes. Under GNU/Linux systems, a well-known system call serve the purpose of writing another process memory: ptrace(). But this is not the only method, both the procfs pseudo filesystem and the process_vm_writev() system call can also be used.

Whatever the technique used, what you want to detect are memory writes to another process.

Associated auditd rules:

# Process injection
# Memory write with ptrace(PTRACE_POKETEXT, ...)
-a always,exit -S ptrace -F a0=4 -k detect_injection_ptrace_poketext

# Memory write with process_vm_writev()
-a always,exit -S process_vm_writev -k detect_injection_vm_writev

Note that it I did not add a rule for procfs writes, this is because the path to the memory map of a process is unpredictable as it contains the process pid. Such as: /proc/<pid>/map. Unfortunately auditd does not allow for rules with globbing in paths.

A few programs to test:

1. ptrace() (from another article of this blog)

Unprivileged user shell:

$ gcc -o ptrace ptrace.c
$ ./ptrace /usr/bin/yes
Writing PID: 617889 Addr: 0x69274810 Buf: 0x096aff31
Writing PID: 617889 Addr: 0x69274818 Buf: 0x4dd68948
Writing PID: 617889 Addr: 0x69274820 Buf: 0x076a5a41
Writing PID: 617889 Addr: 0x69274828 Buf: 0x5178c085
Writing PID: 617889 Addr: 0x69274830 Buf: 0x58296a50
Writing PID: 617889 Addr: 0x69274838 Buf: 0x0f5e016a
Writing PID: 617889 Addr: 0x69274840 Buf: 0x97483b78
Writing PID: 617889 Addr: 0x69274848 Buf: 0x007fbb01
Writing PID: 617889 Addr: 0x69274850 Buf: 0x106ae689
Writing PID: 617889 Addr: 0x69274858 Buf: 0x4859050f
Writing PID: 617889 Addr: 0x69274860 Buf: 0x74c9ff49
Writing PID: 617889 Addr: 0x69274868 Buf: 0x6a006a58
Writing PID: 617889 Addr: 0x69274870 Buf: 0x0ff63148
Writing PID: 617889 Addr: 0x69274878 Buf: 0x79c08548
Writing PID: 617889 Addr: 0x69274880 Buf: 0x0f5f016a
Writing PID: 617889 Addr: 0x69274888 Buf: 0x48050f5a
Writing PID: 617889 Addr: 0x69274890 Buf: 0x0000e6ff

Auditor:

# ausearch --start this-week -k detect_injection_ptrace_poketext
time->Mon Apr 21 05:20:08 2025
type=PROCTITLE msg=audit(1745227208.397:4420): proctitle=2E2F707472616365002F7573722F62696E2F796573
type=SYSCALL msg=audit(1745227208.397:4420): arch=c000003e syscall=101 success=yes exit=0 a0=4 a1=96da1 a2=7f6669274890 a3=e6ff items=0 ppid=8444 pid=617888 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts2 ses=2 comm="ptrace" exe="/tmp/ptrace" subj=unconfined key="detect_injection_ptrace_poketext"

2. process_vm_writev() (stolen from that good article)

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/uio.h>

int main(int argc, char *argv[])
{
        int pid = atoi(argv[1]);

        // Initialize local and remote iovec structs used to perform the syscall
        struct iovec local[1];
        struct iovec remote[1];

        // Place our data in the local iovec 
        local[0].iov_base = "aaa";
        local[0].iov_len = 3;

        // Point the remote iovec to the address in the remote process
        remote[0].iov_base = (void *)0x12345678;
        remote[0].iov_len = 3;

        // Write the local data to the remote address
        process_vm_writev(pid, local, 1, remote, 1, 0);
}

Unprivileged user shell (note that trying to inject init is a bad idea and will fail):

$ gcc -o process_vm_writev process_vm_writev.c
$ ./process_vm_writev 1

Auditor:

ausearch --start this-week -k detect_injection_vm_writev
time->Mon Apr 21 05:13:54 2025
type=PROCTITLE msg=audit(1745226834.016:4290): proctitle=2E2F70726F636573735F766D5F7772697465760031
type=SYSCALL msg=audit(1745226834.016:4290): arch=c000003e syscall=311 success=no exit=-1 a0=1 a1=7fff5691eaf0 a2=1 a3=7fff5691eae0 items=0 ppid=8444 pid=614707 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts2 ses=2 comm="process_vm_writ" exe="/tmp/process_vm_writev" subj=unconfined key="detect_injection_vm_writev"

Code compilation

In order to use exploits, attacker often compile it on the system it will be executed on. Installing compilers on servers is a bad idea as it will serve that purpose while having no legit use (distributions packages binaries, not code). What we want is detecting both installation of compilers and use of these.

Associated auditd rules:

# Compiler install
-w /usr/bin/gcc -p wa -k detect_compiler_gcc_install
-w /usr/bin/g++ -p wa -k detect_compiler_gcc_plusplus_install
-w /usr/bin/clang -p wa -k detect_compiler_clang_install
-w /usr/bin/clang++ -p wa -k detect_compiler_clang_plusplus_install

# Compiler usage
-a always,exit -F path=/usr/bin/gcc -F perm=x -k detect_compiler_gcc
-a always,exit -F path=/usr/bin/g++ -F perm=x -k detect_compiler_gcc_plusplus
-a always,exit -F path=/usr/bin/clang -F perm=x -k detect_compiler_clang
-a always,exit -F path=/usr/bin/clang++ -F perm=x -k detect_compiler_clang_plusplus

Restricted environments evasion

Container escape

There are numerous documented container evasions out there and it is difficult to detect them all. A well-known one to escape a privileged container is to mount the host’s filesystem. One usually achieve this by locating the host root filesystem and executing the mount system call with the corresponding device in /dev.

As an example, we can take the setup listed in that article. Basically it is a docker container on a host with LVM. The host filesystem is on the /dev/dm-0 block, which is predictable as it is the first exposed device by LVM through the device mapper. There are no reasonable reason I can see for re-mounts of the host root partition from such a container, a specific rule can be written.

Associated auditd rules:

# Container evasion via host root FS mount: mount("/dev/dm-0", ...)
-a always,exit -S mount -F a0=/dev/dm-0 -F auid!=0 -k detect_container_evasion_privileged_mount

Once again, the absence of support for globbing in the rules prevents the writing of a more generic rule (for all LVM-style devices for example). And the device the root partition might be on is not necessarily dm-0.

Lateralization

RDP

In the Microsoft Windows world, RDP is a well-known living-off-the-land protocol widely used by attackers to jump from a machine to another. It is not rare during incident response to find a GNU/Linux machine accessing another one in the Active Directory domain this way.

Associated auditd rules:

# RDP tools install
-w /usr/bin/xfreerdp -p wa -k detect_rdp_xfreerdp_install_1
-w /usr/local/bin/xfreerdp -p wa -k detect_rdp_xfreerdp_install_2

# RDP use
-w /usr/bin/xfreerdp -p x -k detect_rdp_xfreerdp_1
-w /usr/local/bin/xfreerdp -p x -k detect_rdp_xfreerdp_2

As xfreerdp is not the only tool one can use to RDP from a GNU/Linux system, you may want to investigate if there are others on your perimeter.

Persistence

SSH

Adding an SSH key in order to come back on a system is a common way of achieving persistence. If the user adding the key is also different from the SSH configuration folder owner, this sounds suspicious. root could decide to add keys for other users but once again, you want to know if a sysadmin is doing that.

Associated auditd rules:

# Add root SSH keys as non-root user
-w /root/.ssh/authorized_keys -p wa -F auid!=0 -k detect_add_SSH_root_key 

Cron task

Adding a scheduled task is a way an attacker can ensure its binary or script, if it dies, will be executed again, either on host reboot or periodically while the host is up. The historic way of doing so on GNU/Linux systems is via a cron job.

Associated auditd rules:

# Non-root users adding privileged cron jobs
-w /etc/crontab -p wa -F auid!=0 -k detect_add_cron_job_crontab
-w /etc/anacrontab -p wa -F auid!=0 -k detect_add_cron_job_anacrontab
-w /etc/cron.d/ -p war -F auid!=0 -k detect_add_cron_job_d
-w /etc/cron.hourly/ -p war -F auid!=0 -k detect_add_cron_job_hourly
-w /etc/cron.daily/ -p war -F auid!=0 -k detect_add_cron_job_daily
-w /etc/cron.weekly/ -p war -F auid!=0 -k detect_add_cron_job_weekly
-w /etc/cron.monthly/ -p war -F auid!=0 -k detect_add_cron_job_monthly
-w /etc/cron.yearly/ -p war -F auid!=0 -k detect_add_cron_job_yearly

Systemd task

A more modern approach is to create a systemd service (a.k.a. task). The unit files for these services have different locations, some of which cannot be found in the official documentation. A useful tool I found to know what paths should be checked is systemd-analyze:

# uname -a
Linux kali 6.8.11-amd64 #1 SMP PREEMPT_DYNAMIC Kali 6.8.11-1kali2 (2024-05-30) x86_64 GNU/Linux
# systemd-analyze --system unit-paths
/etc/systemd/system.control
/run/systemd/system.control
/run/systemd/transient
/run/systemd/generator.early
/etc/systemd/system
/etc/systemd/system.attached
/run/systemd/system
/run/systemd/system.attached
/run/systemd/generator
/usr/local/lib/systemd/system
/usr/lib/systemd/system
/run/systemd/generator.late

Associated auditd rules:

# Non-root users adding "system" systemd unit files
-w /etc/systemd/system/ -p wa -F auid!=0 -k detect_add_systemd_etc_system
-w /lib/systemd/system/ -p wa -F auid!=0 -k detect_add_systemd_lib
-w /usr/lib/systemd/system/ -p wa -F auid!=0 -k detect_add_systemd_usr_lib
-w /usr/local/lib/systemd/system/ -p wa -F auid!=0 -k detect_add_systemd_usr_local_lib

udev rules

Adding udev rules is a nice persistence mean and is often overlooked by analysts writing detection signatures. Non-root users adding such rules is highly suspicious while root adding some does not happen so often either so you want to catch all added udev rules.

Associated auditd rules:

# Udev rules
-w /etc/udev/rules.d/ -p wa -k detect_udev_etc 
-w /usr/lib/udev/rules.d/  -p wa -k detect_udev_usr_lib

Detection evasion

Shared memory (/dev/shm)

To avoid writing to disks on a host, attackers try to keep malwares in RAM. One way of doing this on a GNU/Linux system is to use shared memory, exposed to userland in /dev/shm. The shm_open()function is implemented to write to that path on Linux. Unfortunately, this is a standard library function, not a system call, the auditd rule I found relevant to detect it while not raising too many false positives was on the target /dev/shm path.

Associated auditd rules:

# Shared memory use
-w /dev/shm -p wa -k detect_shm_dev_write

A simple way to test is to use the shm_open() function to write data in a shared memory space:

#include <sys/mman.h>
#include <sys/stat.h>        /* For mode constants */
#include <fcntl.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

int main()
{
        char *shm_name = "/shm_test";
        char *shm_data = "ciao";
        size_t shm_len = strlen(shm_data);

        int fd = shm_open(shm_name, O_RDWR | O_CREAT, S_IRUSR | S_IWRITE);

        if (fd == -1)
        {
                printf("shm_open failed: %s\n", strerror(errno));
                return 1;
        }

        if (ftruncate(fd, shm_len) == -1)
        {
                printf("ftruncate failed: %s\n", strerror(errno));
                return 1;
        }

        void *addr = mmap(NULL, shm_len, PROT_WRITE, MAP_SHARED, fd, 0);

        if (addr == MAP_FAILED)
        {
                printf("mmap failed: %s\n", strerror(errno));
                return 1;
        }

        memcpy(addr, shm_data, shm_len);

        if (munmap(addr, shm_len) == -1)
        {
                printf("munmap failed: %s\n", strerror(errno));
                return 1;
        }

        if (shm_unlink(shm_name) == -1)
        {
                printf("shm_unlink failed: %s\n", strerror(errno));
                return 1;
        }

        return 0;
}

Unprivileged user shell:

$ gcc -o shm_open shm_open.c
$ ./shm_open

Auditor

# ausearch --start this-week -k detect_shm
time->Mon Apr 21 05:01:38 2025
type=PROCTITLE msg=audit(1745226098.316:4259): proctitle="./shm_open"
type=PATH msg=audit(1745226098.316:4259): item=1 name="/dev/shm/shm_test" inode=34 dev=00:19 mode=0100600 ouid=1000 ogid=1000 rdev=00:00 nametype=DELETE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1745226098.316:4259): item=0 name="/dev/shm/" inode=1 dev=00:19 mode=041777 ouid=0 ogid=0 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1745226098.316:4259): cwd="/tmp"
type=SYSCALL msg=audit(1745226098.316:4259): arch=c000003e syscall=87 success=yes exit=0 a0=7ffde153fa60 a1=2f a2=fffffffffffffff8 a3=7faa26833980 items=2 ppid=8444 pid=608617 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts2 ses=2 comm="shm_open" exe="/tmp/shm_open" subj=unconfined key="detect_shm_dev_write"

Conclusion

auditd is a low-cost solution to begin host detection on GNU/Linux systems. Rules are simple to write and can be deployed easily thanks to how well-spread the auditd daemon is on Linux distributions. However it has practical limitations:

• Rules over standard library functions cannot be written as it is a kernel mechanism and only system calls can be watched
• It does not support globbing in paths or system call arguments, making writing of generic signatures impossible
• It does not detect all kind of changes made to files. For example, echo <whatever> >> /etc/sudoers will not trigger the sudoers rule of this article (see here for why)

I hope you found new ideas in these lines, make attackers' lives harder, use auditd and implement detection.

References

• https://github.com/Neo23x0/auditd
• https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/security_guide/sec-defining_audit_rules_and_controls#sec-Defining_Audit_Rules_with_auditctl
• https://www.akamai.com/blog/security-research/the-definitive-guide-to-linux-process-injection
• https://github.com/linux-audit/audit-documentation/wiki
• https://documentation.suse.com/fr-fr/sles/12-SP5/html/SLES-all/cha-audit-setup.html
• cron(8)
• auditd.conf(5)
• https://wiki.archlinux.org/title/Udev
• https://ch4ik0.github.io/en/posts/leveraging-Linux-udev-for-persistence/
• shm_open(3)