Build a post-patch Linux verification CSV with Ansible

The patch window is over. dnf exited cleanly. The package versions look better. The scanner will probably calm down once the next scan runs.

But if you are the person closing the patch ticket, the useful question is not only:

Did the packages update?

It is:

Are these hosts actually running the fixed code now?

For one box, you can SSH in and poke around. Across a fleet, that gets old fast. In the real world, I am more likely to use Ansible, collect a narrow set of facts, and turn the results into a CSV that can be sorted, filtered, attached to a ticket, or handed to a security lead.

This is the workflow I would use for a RHEL-family fleet after a patch window.

What the CSV should answer

The CSV should not try to be a full vulnerability scanner. It should answer the last-mile operational questions that decide whether a ticket can close.

Useful columns:

host,checked_at_utc,os_pretty,running_kernel,latest_kernel_core,reboot_required,needs_restarting_rc,stale_library_count,stale_library_pids,listening_pids,notes

Those fields tell you:

• which host was checked
• when it was checked
• what OS and kernel are live
• whether a newer kernel is installed than the one currently booted
• whether dnf needs-restarting -r thinks a reboot is required
• whether processes still map deleted shared libraries
• which PIDs need follow-up
• whether any suspicious PIDs appear to own listening sockets

That is enough to split the work into practical buckets:

• clean enough to close
• patched but reboot pending
• patched but service restart pending
• needs application owner review
• needs deeper investigation

The manual commands behind the collection

On a single host, I still start with plain commands:

hostname
date -u
cat /etc/os-release
uname -a

For kernel state:

uname -r
rpm -q --last kernel-core 2>/dev/null | head -1
dnf needs-restarting -r
echo $?

For stale shared library mappings:

sudo grep -H '(deleted)' /proc/[0-9]*/maps 2>/dev/null | \
  grep -E '/(usr/)?lib(64)?/.*\.so'

For listeners:

sudo ss -tulpn

The problem is not that these commands are hard. The problem is that they are annoying to run consistently across enough hosts to trust the answer.

That is where Ansible fits.

An Ansible playbook that collects in parallel and builds a CSV

Here is a practical playbook shape for more than a handful of hosts.

The important detail: do not have every host append to the same CSV while Ansible is running in parallel. That creates a local write race. Instead, let each host write one small JSON artifact on the control machine, then build the CSV in a localhost-only play after collection finishes.

---
- name: Prepare local report workspace
  hosts: localhost
  gather_facts: false
  become: false

  vars:
    artifact_dir: "./artifacts/post-patch"

  tasks:
    - name: Ensure artifact directory exists
      ansible.builtin.file:
        path: "{{ artifact_dir }}"
        state: directory
        mode: "0755"

    - name: Find rows from a previous run
      ansible.builtin.find:
        paths: "{{ artifact_dir }}"
        patterns: "*.json"
      register: previous_rows

    - name: Remove rows from a previous run
      ansible.builtin.file:
        path: "{{ item.path }}"
        state: absent
      loop: "{{ previous_rows.files }}"

- name: Collect post-patch verification evidence
  hosts: linux
  gather_facts: false
  become: true
  strategy: free

  vars:
    artifact_dir: "./artifacts/post-patch"

  tasks:
    - name: Capture hostname
      ansible.builtin.command: hostname
      register: host_name
      changed_when: false

    - name: Capture UTC timestamp
      ansible.builtin.command: date -u +%Y-%m-%dT%H:%M:%SZ
      register: checked_at
      changed_when: false

    - name: Read os-release
      ansible.builtin.slurp:
        src: /etc/os-release
      register: os_release_raw

    - name: Capture running kernel
      ansible.builtin.command: uname -r
      register: running_kernel
      changed_when: false

    - name: Capture full uname for ticket evidence
      ansible.builtin.command: uname -a
      register: uname_full
      changed_when: false

    - name: Find newest installed kernel-core package
      ansible.builtin.shell: "rpm -q --last kernel-core 2>/dev/null | head -1 | awk '{print $1}'"
      register: latest_kernel_core
      changed_when: false
      failed_when: false

    - name: Check whether dnf-utils or yum-utils is installed
      ansible.builtin.shell: "rpm -q dnf-utils yum-utils >/dev/null 2>&1"
      register: needs_restarting_available
      changed_when: false
      failed_when: false

    - name: Check reboot requirement with dnf needs-restarting
      ansible.builtin.command: dnf needs-restarting -r
      register: needs_restarting_reboot
      changed_when: false
      failed_when: false
      when: needs_restarting_available.rc == 0

    - name: Find deleted shared libraries still mapped by processes
      ansible.builtin.shell: |
        set -o pipefail
        grep -H '(deleted)' /proc/[0-9]*/maps 2>/dev/null |
          grep -E '/(usr/)?lib(64)?/.*\.so' || true
      args:
        executable: /bin/bash
      register: stale_libs
      changed_when: false
      failed_when: false

    - name: Extract PIDs with stale mapped libraries
      ansible.builtin.set_fact:
        stale_library_pids: >-
          {{
            stale_libs.stdout_lines
            | map('regex_search', '^/proc/([0-9]+)/maps:', '\1')
            | select('defined')
            | flatten
            | unique
            | list
          }}

    - name: Capture listening sockets
      ansible.builtin.command: ss -tulpn
      register: listening_sockets
      changed_when: false
      failed_when: false

    - name: Extract listening PIDs
      ansible.builtin.set_fact:
        listening_pids: >-
          {{
            listening_sockets.stdout
            | regex_findall('pid=([0-9]+)')
            | unique
            | list
          }}

    - name: Compute CSV fields
      ansible.builtin.set_fact:
        os_pretty: "{{ (os_release_raw.content | b64decode | regex_search('PRETTY_NAME=\"?([^\"\\n]+)\"?', '\\1') | first) | default('unknown') }}"
        reboot_required: "{{ (needs_restarting_reboot.rc | default(0) | int) != 0 }}"
        stale_library_count: "{{ stale_library_pids | length }}"
        stale_listening_pids: "{{ stale_library_pids | intersect(listening_pids) }}"
        notes: >-
          {{
            [
              'uname=' ~ uname_full.stdout,
              'stale_listening_pids=' ~ (stale_library_pids | intersect(listening_pids) | join('|'))
            ] | join('; ')
          }}

    - name: Build local row object
      ansible.builtin.set_fact:
        post_patch_row:
          host: "{{ host_name.stdout }}"
          checked_at_utc: "{{ checked_at.stdout }}"
          os_pretty: "{{ os_pretty }}"
          running_kernel: "{{ running_kernel.stdout }}"
          latest_kernel_core: "{{ latest_kernel_core.stdout | default('') }}"
          reboot_required: "{{ reboot_required }}"
          needs_restarting_rc: "{{ needs_restarting_reboot.rc | default('not_available') }}"
          stale_library_count: "{{ stale_library_count }}"
          stale_library_pids: "{{ stale_library_pids }}"
          listening_pids: "{{ stale_listening_pids }}"
          notes: "{{ notes }}"

    - name: Write one JSON row per host locally
      ansible.builtin.copy:
        dest: "{{ artifact_dir }}/{{ inventory_hostname | regex_replace('[^A-Za-z0-9_.-]', '_') }}.json"
        content: "{{ post_patch_row | to_nice_json }}\n"
        mode: "0644"
      delegate_to: localhost
      become: false

- name: Build CSV on the control machine
  hosts: localhost
  gather_facts: false
  become: false

  vars:
    artifact_dir: "./artifacts/post-patch"
    report_path: "./post_patch_verification.csv"

  tasks:
    - name: Find collected host rows
      ansible.builtin.find:
        paths: "{{ artifact_dir }}"
        patterns: "*.json"
      register: row_files

    - name: Read collected host rows
      ansible.builtin.slurp:
        src: "{{ item.path }}"
      loop: "{{ row_files.files }}"
      register: row_json

    - name: Decode collected host rows
      ansible.builtin.set_fact:
        post_patch_rows: "{{ (post_patch_rows | default([])) + [item.content | b64decode | from_json] }}"
      loop: "{{ row_json.results }}"

    - name: Write final CSV
      ansible.builtin.copy:
        dest: "{{ report_path }}"
        mode: "0644"
        content: |
          host,checked_at_utc,os_pretty,running_kernel,latest_kernel_core,reboot_required,needs_restarting_rc,stale_library_count,stale_library_pids,listening_pids,notes
          {% for row in post_patch_rows | default([]) | sort(attribute='host') -%}
          "{{ row.host | string | replace('"', '""') }}","{{ row.checked_at_utc | string | replace('"', '""') }}","{{ row.os_pretty | string | replace('"', '""') }}","{{ row.running_kernel | string | replace('"', '""') }}","{{ row.latest_kernel_core | string | replace('"', '""') }}","{{ row.reboot_required | string | replace('"', '""') }}","{{ row.needs_restarting_rc | string | replace('"', '""') }}","{{ row.stale_library_count | string | replace('"', '""') }}","{{ row.stale_library_pids | default([]) | join('|') | replace('"', '""') }}","{{ row.listening_pids | default([]) | join('|') | replace('"', '""') }}","{{ row.notes | string | replace('"', '""') }}"
          {% endfor -%}

Run it against the group you patched:

ansible-playbook -i inventory.ini post_patch_verification.yml --limit patched_linux -f 50

Ansible already runs hosts in parallel up to the fork limit. The -f 50 flag tells the control node it can work on up to 50 hosts at a time, and strategy: free lets a fast host keep moving instead of waiting for a slower host to finish the same task. Tune -f to match your control node, network, and target environment. -f 25 might be plenty. -f 100 might be fine in a lab and rude in production.

The key is that the collection play can fan out safely because each host writes its own artifact. The only single-writer step is the final localhost play that builds the CSV.

The result is a local CSV, plus the per-host JSON rows that created it:

post_patch_verification.csv
artifacts/post-patch/app-01.json
artifacts/post-patch/web-02.json

Now you have something you can open, sort, filter, and attach.

What the CSV might look like

Example output:

host,checked_at_utc,os_pretty,running_kernel,latest_kernel_core,reboot_required,needs_restarting_rc,stale_library_count,stale_library_pids,listening_pids,notes
"app-01","2026-06-28T16:30:00Z","Rocky Linux 9.4","5.14.0-427.el9.x86_64","kernel-core-5.14.0-427.20.1.el9_4.x86_64","true","1","0","","","uname=Linux app-01 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64; stale_listening_pids="
"web-02","2026-06-28T16:31:12Z","AlmaLinux 9.4","5.14.0-427.20.1.el9_4.x86_64","kernel-core-5.14.0-427.20.1.el9_4.x86_64","false","0","2","1842|1843","1842","uname=Linux web-02 5.14.0-427.20.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64; stale_listening_pids=1842"
"batch-03","2026-06-28T16:32:44Z","Red Hat Enterprise Linux 9.4","5.14.0-427.20.1.el9_4.x86_64","kernel-core-5.14.0-427.20.1.el9_4.x86_64","false","0","0","","","uname=Linux batch-03 5.14.0-427.20.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC x86_64; stale_listening_pids="

The interesting rows jump out quickly:

• app-01 has reboot debt.
• web-02 has stale mapped libraries, and one stale PID appears to be listening.
• batch-03 looks clean from this narrow post-patch check.

That is already more useful than “patch job succeeded.”

How to triage the CSV

I usually think about the rows in this order.

1. Reboot required

Filter:

reboot_required = true

Those hosts may have the fixed kernel package installed, but they are not booted into it yet.

The follow-up is not “rerun dnf.” The follow-up is a reboot plan:

• can this host reboot now?
• does it need to be drained first?
• is it part of a cluster?
• who owns the workload?
• when does the exception expire?

2. Stale libraries in listening processes

Filter:

stale_library_count > 0
listening_pids is not empty

These are the rows I care about most after a patch window.

If a process still maps a deleted shared library and that same PID owns a listening socket, that is a strong signal that the package update did not finish the runtime remediation.

The next step is to map the PID back to a service:

pid=1842
ps -fp "$pid"
sudo readlink -f "/proc/$pid/exe"
sudo tr '\0' ' ' < "/proc/$pid/cmdline"; echo
systemctl status "$pid"

Then restart the smallest safe unit:

sudo systemctl restart nginx

And verify the old mapping disappeared.

3. Stale libraries in non-listening processes

Filter:

stale_library_count > 0
listening_pids is empty

This still deserves attention, but it may not be the same urgency as a public listener. It could be an agent, a batch process, a local daemon, or a process that is about to exit anyway.

The row should not disappear into the void. It should get an owner and a decision:

• restart during the next service window
• allow process to exit naturally
• accept temporarily with expiration
• investigate because the process is unexpected

4. Clean rows

Filter:

reboot_required = false
stale_library_count = 0

Those are the easiest rows to close, at least for this specific verification pass.

That does not mean the host has no risk. It means this post-patch runtime check did not find obvious reboot debt or deleted shared library mappings.

Make package checks targeted when you have a known CVE

The generic CSV is good for after-window hygiene. If you are chasing a known CVE, add package-specific checks.

For OpenSSL:

- name: Capture openssl package evidence
  ansible.builtin.shell: |
    rpm -q openssl-libs
    rpm -q --changelog openssl-libs | grep -i 'CVE-' | head -20
  register: openssl_evidence
  changed_when: false
  failed_when: false

For glibc:

- name: Capture glibc package evidence
  ansible.builtin.shell: |
    rpm -q glibc
    rpm -q --changelog glibc | grep -i 'CVE-' | head -20
  register: glibc_evidence
  changed_when: false
  failed_when: false

For kernel advisories:

- name: Capture installed kernel evidence
  ansible.builtin.shell: |
    uname -a
    rpm -q --last kernel-core | head -5
  register: kernel_evidence
  changed_when: false

You can add those fields to the CSV, or write per-host evidence files under an artifacts directory. For CSV, keep it narrow. For evidence bundles, keep the richer command output.

Add per-host evidence files when the CSV is not enough

CSV is good for sorting. It is not great for long command output.

The playbook above already keeps per-host JSON rows so the CSV can be rebuilt. If you also need human-readable proof for a ticket, add a text artifact per host:

artifacts/
  post-patch/
    app-01.json
    web-02.json
  post_patch_verification.csv
  app-01.txt
  web-02.txt
  batch-03.txt

The CSV gives the summary. The text file gives the proof.

For example:

- name: Write per-host evidence locally
  ansible.builtin.copy:
    dest: "./artifacts/{{ inventory_hostname | regex_replace('[^A-Za-z0-9_.-]', '_') }}.txt"
    mode: "0644"
    content: |
      host={{ host_name.stdout }}
      checked_at_utc={{ checked_at.stdout }}
      os={{ os_pretty }}
      uname={{ uname_full.stdout }}
      running_kernel={{ running_kernel.stdout }}
      latest_kernel_core={{ latest_kernel_core.stdout | default('') }}
      needs_restarting_rc={{ needs_restarting_reboot.rc | default('not_available') }}

      stale_library_pids={{ stale_library_pids | join('|') }}
      stale_listening_pids={{ stale_listening_pids | join('|') }}

      stale_libraries:
      {{ stale_libs.stdout | default('none') }}

      listening_sockets:
      {{ listening_sockets.stdout | default('none') }}
  delegate_to: localhost
  become: false

Now the ticket can say:

See CSV for fleet summary. See per-host artifact for command output.

That is much easier to defend than a screenshot pile.

Where this still falls short

This Ansible workflow is useful, but it is still a script.

It does not fully solve:

• mapping every deleted library back to the exact package build that supplied it
• correlating package state with vendor advisory data
• distinguishing vulnerable-but-not-live from vulnerable-and-running at scale
• adding CISA KEV and EPSS prioritization
• tracking restart and reboot debt over time
• producing a clean report a champion can forward internally

But it is a strong step up from closing patch tickets based only on package install success.

It gives you a CSV that can answer:

Which hosts are patched but still need runtime follow-up?

That is the question that matters after the patch window.

Where oxharden fits

oxharden’s Patch Truth Snapshot is built around this same applied-vs-live gap.

The manual Ansible version asks:

• what package is installed?
• what kernel is running?
• which processes still map stale libraries?
• which services likely need restart?
• which hosts still need reboot?
• what evidence can I keep?

The Snapshot packages that into a report for one representative RHEL-family host, then gives you a path to expand across more systems when the first report finds something worth chasing.

You can absolutely start with the CSV. If that CSV shows reboot debt, stale mapped libraries, or listener-owned stale PIDs, you have already found the reason this check should become repeatable.

Patched is what the package database says. Fixed is what the runtime proves.